Predictions are not correct

#1
by Arman73k - opened

Hey, I'm using the latest model for sentiment analysis. However, it seems the classifier is not trained properly. The predictions do not make sense:

Here is the warning:

Some layers of TFRobertaForSequenceClassification were not initialized from the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Here is the code to reproduce the results:

from transformers import AutoModelForSequenceClassification
from transformers import TFAutoModelForSequenceClassification
from transformers import AutoTokenizer, AutoConfig
import numpy as np
from scipy.special import softmax

MODEL = "cardiffnlp/twitter-roberta-base-sentiment-latest"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
config = AutoConfig.from_pretrained(MODEL)

model = TFAutoModelForSequenceClassification.from_pretrained(MODEL)
text = "covid cases are increasing fast!"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

scores = output.logits.numpy().squeeze()
scores = softmax(scores)
print(scores)

RESULTS:
[0.26674476 0.41626564 0.31698957]

This does not match the results reported in the document.

UPDATE:
The problem seems only to exist for Tensorflow. Pytorch successfully loads the classifier layer weights, and the predictions seem to be reasonable.

Arman73k changed discussion status to closed

Sign up or log in to comment