Predictions from BertForSequenceClassification model keep changing across runs
Author: varun-nathan
Pull request: no
Labels: name: Need more information

name: wontfix

I used the code in run_classifier.py to train a model for intent detection which is a multi-class classification problem. After training the model, when I used it for prediction, I found the predictions to be changing from one run to another. I'm trying to understand the reason for the same and how I can avoid this behavior.

Let's say that model is the fine-tuned sequence classification model. I get different results (logits) every time I run the following: 
logits = model1(input_ids, segment_ids, input_mask, labels=None)

Why is that so?



Comments:
You probably forgot to deactivate the DropOut modules with `model.eval()`

Oh ok, I get it. That's helpful. Thanks!

I tried that and now I seem to be getting the same predictions for any input. In other words, the logits don't change with change in input. Not sure why this is happening?

Maybe a bug in your code?
If you can share a simple self-contained example exhibiting the behavior we can have a look.

That's unlikely as I tried the same code with other problems and got reasonable results. The problems on which I tried this code include MRPC task, sentiment prediction on IMDB dataset and intent detection on smalltalk data. The results are logical and reasonable.
The dataset on which I get this behaviour reported above has about 20 examples for each of the 4 intents viz. restaurant_search(0), booking_table(1), greet(2) and thanks(3). So, I was thinking if this could be due to the data having less number of examples. The sequence classifier class is basically bert model + a single hidden layer neural network with output layer as the number of labels. I feel that there's not enough data to train the last (classifier) layer. Here's my test data:
test_df = pd.DataFrame({'text': ["hey", "indian hotels"],
                        'label': [2, 0]})
I print the logits for each of the 2 test samples which are as follows:
tensor([[ 28.9354,  28.3292,  20.1560, -20.7804]])
tensor([[ 28.9354,  28.3292,  20.1560, -20.7804]])

The logits tensor doesn't change for any input text.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi @varun-nathan, were you ever able to solve this?  I'm having a very similar issue with ReformerForSequenceClassification.

Hi @varun-nathan and @jstremme, were either of you able to find the issue? I'm experiencing the same thing with ElectraForSequenceClassification!

> Hi @varun-nathan and @jstremme, were either of you able to find the issue? I'm experiencing the same thing with ElectraForSequenceClassification!

Hi @mollha, I managed to solve this but can't remember exactly what I did.  Did you try `model.eval()`? Also, make sure you are training on more than just a few sample records and using a large enough model in terms of neurons per layer etc. 

Thanks! I was already using model.eval(), but my dataset size was too small (around 1000). After increasing to 15000 I am getting much better results.

Excellent!