Truncated last sentence after bart finetuning on custom dataset.
Author: sajastu
Pull request: no
Labels: name: wontfix

   
- `transformers` version: 3.0.2
- Platform:
- Python version: 3.6
- PyTorch version (GPU?): 1.4
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Single GPU

### Who can help
@sshleifer

## Information

Model I am using (Bert, XLNet ...): BART

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Finetuning BART-large-xsum on my custom dataset with differing configs: min_length=590, max_length=620.
2. Doing inference on the trained model.
3. Sentences (specifically, last sentence) that BART produces oftentimes (~90% of the cases) is incomplete. 

## Expected behavior

I would expect to have a complete and neat output, without having truncated outputs. Should mention that when I do the inference on the raw bart-large-cnn (or -xsum) checkpoint, I do not see this problem and all the outputs are complete. It sounds to me that finetuned-bart on custom dataset is not able to emit <EOS> token.

I also checked this thread: https://github.com/huggingface/transformers/issues/5674 which faces the same problem, but couldn't find the answer.

Comments:
What was your training command?


I used the `finetune_tiny_bart.sh` script in the seq2seq examples. @sshleifer 

If that helps to figure out the source of the problem, as I know the position_embeddings of bart-large-cnn model is 1026 (with addition of SOS, and EOS tokens). Since my task is long summarization, I changed it to 2050, and let the model learn the whole on my custom dataset; Additionally, as I mentioned earlier, I have also increased the `min_length` and `max_length` in the BART config class. But the problem still remains.

I've never really trained with such large length parameters, but we have been seeing similar problems for many models. I think these are the lines causing the issue, will try to get a fix soon.
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.py#L144

I have been facing this issue in the new versions of finetune.sh as well... even for T5...
eg.
In only the first quarter century after the breakup of the Soviet Union, Azerbaijan has impressed the Caucasus region and the world with its progress. Although it still must work diligently to enhance citizen inputs into its governance structures, continue to expand its productive capacity beyond the energy sectors, and distribute its new wealth equitably among its entire population, the country has faced the complex challenges of independence with a mostly steady hand. Much has been achieved in rediscovery of a proud national identity, new resource abundance, sound transportation infrastructure, and a thriving capital city that is now a vibrant modern regional hub. Among the most important next steps for policy priority over the coming decades will be in sustaining the progress already made with continuing "greener" approaches to development, and increasing diversification of the economy beyond just the oil and natural gas sectors. Initiatives already in place have started along this road, but will need to be strengthened over

I would love to replicate (need data) or have one of you test on the branch with my proposed fix:

https://github.com/huggingface/transformers/pull/6654

```bash
git fetch
git checkout batch-parity-cleaner
```

That branch is broken right now, I will comment when it's fixed.

Should work now!

Hey,
Was trying out your branch, the earlier version atleast ran fine.
After pulling the latest like you mentioned...getting back this:
  f"Mbart is using sequence lengths {self.max_source_length}, {self.max_target_length}. "
Validation sanity check: 0it [00:00, ?it/s]Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.

I assume this has to do with the translation code? Any suggestions how to get around this?

Fixed on the branch, sorry about that!

So...should I try now?

Yah!

Still the same :(

OK. Which dataset are you using? I can't really debug without being able to see what a batch looks like when it goes into the model.

I am using a custom dataset but you can try with BillSum as well and you should be able to reproduce the issue.
And btw here I was talking about this particular issue:
> Hey,
> Was trying out your branch, the earlier version atleast ran fine.
> After pulling the latest like you mentioned...getting back this:
> f"Mbart is using sequence lengths {self.max_source_length}, {self.max_target_length}. "
> Validation sanity check: 0it [00:00, ?it/s]Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
> Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
> Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
> Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
> Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
> Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
> Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
> Keyword arguments {'src_lang': None, 'tgt_lang': None} not recognized.
> 
> I assume this has to do with the translation code? Any suggestions how to get around this?

I am not able to start training itself with your branch.
Let me know if you need anymore info.

Yes a training command I can paste into my terminal to run on billsum and reproduce your failure.

I just tried this and got the same:
`./finetune.sh \
    --data_dir "../../../BillSum"
    --train_batch_size=2 
    --eval_batch_size=8 
    --output_dir="/content/models/t5_narrative_512/" 
    --num_train_epochs 2 
    --model_name_or_path="t5-base" --n_val 1000 
    --val_check_interval 0.5 
    --max_source_length=512 --max_target_length=150 --val_max_target_length=150 --test_max_target_length=150 `

@patil-suraj we think these should be fixed both for t5 and bart, right?

Yes, AFAIK these issues are fixed now. @amanpreet692  could you try this with the latest master branch ?

I used distilbart:

`tokenizer_dbart = BartTokenizer.from_pretrained('sshleifer/distilbart-cnn-6-6')`
`model_dbart = BartForConditionalGeneration.from_pretrained('sshleifer/distilbart-cnn-6-6')`

The last sentence of the summary obtained from the model is sometimes truncated.

Is this expected? @sshleifer 

@patil-suraj Sorry I got back to this only now, I checked out the latest from repo today and ran finetune.sh for finetuning and could still see this issue,
eg. 
Research on inventory risk management based on abc analysis. The traditional ABC analysis is a kind of management method from the ABC curve. ABC curve is also called Pareto (Pareto) curve. The basic idea is, " vital few and the majority of the general ". In all the inventory, the cumulative percentage of species ranged from 5% to 15% and the average amount of funds occupied the cumulative percentages of 60% ~ 80% items identified as A class; the cumulative proportion of funds is 20% ~ 30% of the goods, identified as B class; and the rest as class C. The different objects use different management methods and means. In the China's enterprises,

The command I used is the same as above, only I removed the fp16 parameter from the script.

Hi @amanpreet692,
Could you post the arguments you are passing to `generate` ? for ex. `num_beams, max_length, length_penalty` etc


Hey, I haven't tinker with the arguments to generate so I guess they should be the same as in config for distilbart:
"early_stopping": true,
"length_penalty": 2.0,
"max_length": 142,
"min_length": 56,
"no_repeat_ngram_size": 3,
"num_beams": 4
Let me know if you need anything else.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.