Predictions are not correct
#1
by
Arman73k
- opened
Hey, I'm using the latest model for sentiment analysis. However, it seems the classifier is not trained properly. The predictions do not make sense:
Here is the warning:
Some layers of TFRobertaForSequenceClassification were not initialized from the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Here is the code to reproduce the results:
from transformers import AutoModelForSequenceClassification
from transformers import TFAutoModelForSequenceClassification
from transformers import AutoTokenizer, AutoConfig
import numpy as np
from scipy.special import softmax
MODEL = "cardiffnlp/twitter-roberta-base-sentiment-latest"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
config = AutoConfig.from_pretrained(MODEL)
model = TFAutoModelForSequenceClassification.from_pretrained(MODEL)
text = "covid cases are increasing fast!"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
scores = output.logits.numpy().squeeze()
scores = softmax(scores)
print(scores)
RESULTS:
[0.26674476 0.41626564 0.31698957]
This does not match the results reported in the document.
UPDATE:
The problem seems only to exist for Tensorflow. Pytorch successfully loads the classifier layer weights, and the predictions seem to be reasonable.
Arman73k
changed discussion status to
closed