Very bad performances with BertModel on sentence classification
Author: seo-95
Pull request: no

## ❓ Questions & Help

I'm trying to use the raw BertModel for predictions over a dataset containing a set of dialogues. In the original work I had 3 different losses but I've noticed that the losses are very high and not going down epoch after epoch. So I started to take only one loss (intent classification) and I've tried to overfit a small portions of the training dataset with a set of 50 samples. Anyway the results have not changed.

I've tried 2 solutions for the intent classification:

- Linear layer on top of the [CLS] embedding -> loss after 100 epochs = 2.4
-  2layer-LSTM to encode the bert hiddens of the last layer + linear-> loss after 100 epochs = 1.1

The input shape is:
`[CLS] what's the weather like [SEP] [PAD] .... [PAD]`


I've thought also to use biLSTM but at this point I think that something goes wrong... the sentences are very simple "check the weather for tomorrow" (weather intent) and contains only 3 intents to classify.
- The BertModel is the raw one pretrained with "bert-base-cased".
- The batch size is 1 because I had memory issue with BERT. i'm working with dialogue granularity and so I have an 3D-input of shape DIALOGUE_NUM x SENTENCE_NUM x SENTENCE_LEN while BERT expects a 2D-input tensor. By putting batch size of 1 I've found a work around to the problem.
- The optimizer is Adam with a learning rate of 0.001, by increasing it at 0.01 performances got worse.
- The loss is the Cross-Entropy loss to which I pass the output logits.

The BERT paper said that the finetuning process can achieve great performances within few epochs... Anyone has an idea why I cannot achieve these?

Comments:
I still trying but the system seems to be in underfitting, i do not understand why it performs so poor

I've never used BERT for sentence classification tasks, but in regard to the batch size and memory constraints, you could use gradient accumulation to have a bigger effective batch size (see [examples/run_squad.py]](https://github.com/huggingface/pytorch-transformers/blob/b33a385091de604afb566155ec03329b84c96926/examples/run_squad.py#L137-L153)).
I didn't fully understand your input's shape, but it seems like you should drop dialogue and sentence number dimensions and work with shape (batch_size, sentence_length) as BERT expects. What information do you have on the first two dimensions?

I need to work with dialogue granularity because I have different predictions to do: prediction of the action to perform on the KB (insert/fetch), prediction of the intent and prediction of the end-of-dialogue(eod). The first two prediction are done on the first sentence of the dialogue, while eod prediction is done by concatenating the current dialogue with the first sentence of the next one, in this way the model can notice a sort of discontinuity and so an eod. The system is end-to-end, I perform the joint prediction of this 3 labels.
The loss of the eod classifier is computed for each sentence in the dialogue, the other two loss only once per "batch".
BERT receives always a 2D tensor [NUM_SENTENCES x SEQ_LEN] so I don't think this could be a problem. My losses (CrossEntropyLoss) are quite high after 50 epochs:
- 0.54 for eod
- 0.45 for action
- 1.1 for intent

So I've tried to overfit with the intent prediction only a smaller dataset of 20 sample but the results are the same. I've tried with less samples but the situation doesn't change...
I perform the gradient accumulation as following: 

```
_GRADIENT_RATE = 16
for e in enumerate(_N_EPOCHS):
     train_losses  = []
     model.train()
     for idx, (batch, intent) in training_generator:
          logits = model(batch)
          loss = criterion(logits, target)
          loss.backward()
         if idx % _GRADIENT_RATE == 0 or idx == dataset.__len__()-1:
                     optimizer.step()
                     optimizer.zero_grad()
```
I cannot understand why my model is underfitted, I also thought about some errors with the loading of the pretrained model but I have already checked it.


```
class BertClass(nn.Module):
     def __init__():
              .....

    def build_nn(self):
        self._bert = BertModel.from_pretrained('bert_base_cased')
        self._intent_classifier = nn.Linear(768, 3)

   def forward(self, input, ...):
         .... computing attention and segment mask ...
        bert_hiddens, bert_cls_out = self._bert(input, segment_mask, attention_mask)
        logits = self._intent_classifier(bert_cls_out)
        return logits
```

I also modify the learning rate multiplying it by 0.1 after epochs 10, 20, 40

Ok I solved. Of course was my mistake.. this is my first real deep learning project and I have to learn a lot of things. Anyway my error was with learning rate, it was 2 order of magnitude greater wrt the ones suggested in the paper. Thank you for the support