BART finetune.py: model not learning anything
Author: alexgaskell10
Pull request: no
Labels: name: Examples

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
./examples/summarization/finetune.sh
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Finetuning on PubMed dataset.

## To reproduce

Steps to reproduce the behavior:

1. set DATA_DIR & OUT_DIR
2. Run command:
python finetune.py \
    --model_name_or_path=facebook/bart-large-cnn \
    --learning_rate=3e-5 \
    --gpus 1 \
    --do_predict \
    --do_train \
    --n_val 1000 \
    --val_check_interval 0.1 \
    --sortish_sampler \
    --max_target_length=80 \
    --val_max_target_length=200 \
    --test_max_target_length=200 \
    --max_source_length=850 \
    --train_batch_size=1 \
    --eval_batch_size=1 \
    --data_dir=$DATA_DIR \
    --output_dir=$OUT_DIR \
    --logger=wandb \
    $@
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The model is not learning anything. The generated eval summaries are identical throughout training and identical across training instances with different hyperparams. Appears as though backprop is not happening but there is not error message. (Probably I am missing something simple)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Linux-5.0.0-37-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
Seems to be coming from the --warmup_steps arg. If I set to 0, the model seems to learn as expected. If it is > 0 the model doesn't learn anything.

Interesting. Could you link me to that dataset/instructions?
I just reran cnn_dm with warmup_steps=0 and my loss curve looks very similar in wandb. pink is no warmup:

![image](https://user-images.githubusercontent.com/6045025/85876097-38e05680-b7a3-11ea-9c39-7d9c572cd4be.png)

Anyways, I will set default warmup_steps=0.


Hmmm interesting, I agree yours both look normal. I am using Pubmed dataset and truncating input docs at 850 input tokens. I'm pretty sure my model wasn't learning anything when warmuo_steps>0 for the following reasons:
- The generated summaries were identical across training
- The target summaries begin with '\<S\>' token but I believe the CNN data the model was finetuned on didn't. As a result, the model learns to begin output summaries with '\<S\>' very quickly. However, for each case with warmup_steps > 0 the generated val summaries didn't begin with this token
- Loss curves below (orange: warmup_steps>0, grey: warmup_steps=0):
- When warmup_steps>0 model followed identical loss curves even after a changed hparams like learning rate

<img width="561" alt="Screenshot 2020-06-26 at 16 56 37" src="https://user-images.githubusercontent.com/51463426/85877481-22e88b00-b7cf-11ea-8b21-11320c0737b4.png">

I played around with it quite a bit and this pattern repeated for any warmp_steps>0 (including =1).

That is pretty compelling evidence.
I will change the default to 0.
Nothing else changed between runs?



Yes- I have tried across runs changing only this hparam.

Interesting. I fixed master. Would love to hear your experimental results/developer experience as you continue!

For sure, I'll pop in this library lots over the coming weeks I'm sure. Great library btw