cardiffnlp

/

twitter-roberta-base-sentiment-latest

like 514

Hey, I'm using the latest model for sentiment analysis. However, it seems the classifier is not trained properly. The predictions do not make sense:

Here is the warning:

Some layers of TFRobertaForSequenceClassification were not initialized from the model checkpoint at cardiffnlp/twitter-roberta-base-sentiment-latest and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Here is the code to reproduce the results:

from transformers import AutoModelForSequenceClassification
from transformers import TFAutoModelForSequenceClassification
from transformers import AutoTokenizer, AutoConfig
import numpy as np
from scipy.special import softmax

MODEL = "cardiffnlp/twitter-roberta-base-sentiment-latest"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
config = AutoConfig.from_pretrained(MODEL)

model = TFAutoModelForSequenceClassification.from_pretrained(MODEL)
text = "covid cases are increasing fast!"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

scores = output.logits.numpy().squeeze()
scores = softmax(scores)
print(scores)

RESULTS:
[0.26674476 0.41626564 0.31698957]

This does not match the results reported in the document.

UPDATE:
The problem seems only to exist for Tensorflow. Pytorch successfully loads the classifier layer weights, and the predictions seem to be reasonable.