BertForSequenceClassification producing same output during evaluation 
Author: drjosephliu
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
I'm doing mutli-class classification over 50 classes on the Reuters 5050 dataset. The task is to identify the author that wrote the text. The texts can get up to 1700 tokens long but I'm truncating them down to 512. 

I've noticed that during evaluation, the model 99% of the time outputs the same label regardless of the input. I say 99% because once in a while it outputs something different. This **does not** happen during training.

Things I've tried:

- Removed `~/.cache/torch/transformers/` directory
- Tested learning rates of `[2e-5, 1e-5, 5e-6, 1e-6, 1e-7, 1e-8]`
- Reducing batch size to 1
- Setting:
```
hidden_dropout_prob=0.5
'use_cached_eval_features': False,
'no_cache': True,
'overwrite_output_dir': True,
'reprocess_input_data': True,
```

**Tokenising and encoding:**

```
MAX_LEN = 512
def get_encodings(texts):
    token_ids = []
    attention_masks = []
    for text in texts:
        token_id = tokenizer.encode(text, 
                                    add_special_tokens=True,
                                    max_length=MAX_LEN,
                                    pad_to_max_length=True)
        token_ids.append(token_id)
    return token_ids

def get_attention_masks(padded_encodings):
    attention_masks = []
    for encoding in padded_encodings:
        attention_mask = [int(token_id > 0) for token_id in encoding]
        attention_masks.append(attention_mask)
    return attention_masks

train_encodings = get_encodings(train_df.text.values)
train_attention_masks = get_attention_masks(train_encodings)

test_encodings = get_encodings(test_df.text.values)
test_attention_masks = get_attention_masks(test_encodings)
```


**Packing into datasets and dataloaders:**

```
batch_size = 1

train_data = TensorDataset(X_train, train_masks, y_train)
train_sampler = RandomSampler(train_data)
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

validation_data = TensorDataset(X_test, test_masks, y_test)
validation_sampler = SequentialSampler(validation_data)
validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)
```


**Model setup:**

```
config = BertConfig.from_pretrained(
    'bert-base-uncased',
    num_labels = 50,
    output_attentions = False,
    output_hidden_states = False,
    max_position_embeddings=MAX_LEN,
    hidden_dropout_prob=0.5,
    args={
        'use_cached_eval_features': False,
        'no_cache': True,
        'overwrite_output_dir': True,
        'reprocess_input_data': True,
    }
)

model = BertForSequenceClassification(config)

model.to(device)


optimizer = AdamW(model.parameters(),
                  lr = 1e-5
                  eps = 1e-8 
                )

epochs = 4

total_steps = len(train_dataloader) * epochs

scheduler = get_linear_schedule_with_warmup(optimizer, 
                                            num_warmup_steps = 0, 
                                            num_training_steps = total_steps)
```

**Training:**

```
loss_values = []

print("Training...")
for epoch_i in range(0, epochs):

    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')

    t0 = time.time()
    total_loss = 0
    model.train()

    for step, batch in enumerate(train_dataloader):

        if step % 40 == 0 and not step == 0:
            elapsed = format_time(time.time() - t0)
            
            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))

        b_texts = batch[0].to(device)
        b_attention_masks = batch[1].to(device)
        b_authors = batch[2].to(device)

        model.zero_grad()        

        outputs = model(b_texts, 
                        token_type_ids=None, 
                        attention_mask=b_attention_masks, 
                        labels=b_authors)
        
        loss = outputs[0]

        total_loss += loss.item()

        loss.backward()

        # Clip the norm of the gradients to 1.0.
        # This is to help prevent the "exploding gradients" problem.
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        
        optimizer.step()

        scheduler.step()

    avg_train_loss = total_loss / len(train_dataloader)            
    
    # Store the loss value for plotting the learning curve.
    loss_values.append(avg_train_loss)

    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(format_time(time.time() - t0)))
```


**Evaluation:**

   ```
 print("")
    print("Running Validation...")

    t0 = time.time()

    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    model.eval()

    # Tracking variables 
    eval_loss, eval_accuracy = 0, 0
    nb_eval_steps, nb_eval_examples = 0, 0

    for batch in validation_dataloader:
        
        b_texts = batch[0].to(device)
        b_attention_masks = batch[1].to(device)
        b_authors = batch[2].to(device)
        
        # Telling the model not to compute or store gradients, saving memory and
        # speeding up validation
        with torch.no_grad():        

            # Forward pass, calculate logit predictions.
            # This will return the logits rather than the loss because we have
            # not provided labels.
            # token_type_ids is the same as the "segment ids", which 
            # differentiates sentence 1 and 2 in 2-sentence tasks.
            # The documentation for this `model` function is here: 
            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
            outputs = model(b_texts, 
                            token_type_ids=None, 
                            attention_mask=b_attention_masks)
        
        # Get the "logits" output by the model. The "logits" are the output
        # values prior to applying an activation function like the softmax.
        logits = outputs[0]

        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        author_ids = b_authors.to('cpu').numpy()
        print("Pred: {}, label: {}".format(np.argmax(logits).flatten(),
                                           author_ids.flatten()))
        
        # Calculate the accuracy for this batch of test sentences.
        tmp_eval_accuracy = flat_accuracy(logits, author_ids)
        
        # Accumulate the total accuracy.
        eval_accuracy += tmp_eval_accuracy

        # Track the number of batches
        nb_eval_steps += 1

    # Report the final accuracy for this validation run.
    print("  Accuracy: {0:.2f}".format(eval_accuracy/nb_eval_steps))
    print("  Validation took: {:}".format(format_time(time.time() - t0)))

```

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
Hi, have you solved this problem?

I have a similar problem. I'm doing a multi-classification task over 8 classes. My task is to classify the entity type of short texts(usually < 32, much shorter than yours). These classes are something like school name, company name, job title, etc. Besides, I'm using the tf2 version model `TFBertForSequenceClassification` for my task.

However, at most of the time(99% just as you describe), my fine-tuned model gives a same output distribution over my 8 classes whatever text I feed.

Yes, turns out I was not constructing the model properly. All I had to do was call `
model = BertForSequenceClassification.from_pretrained(config)` instead.