Truncated Outputs by t5 fine-tuned models
Author: manojpreveen
Pull request: no
Labels: name: t5

I fine-tuned t5-small over CNN/DM dataset using the finetune_t5.sh script. The outputs produced by the saved fine-tuned model is okayish but it's getting cut i.e., producing incomplete sentence at the end.

Example : 

Artcile:

(CNN)The only thing crazier than a guy in snowbound Massachusetts boxing up the powdery white stuff and offering it for sale online? People are actually buying it. For $89, self-styled entrepreneur Kyle Waring will ship you 6 pounds of Boston-area snow in an insulated Styrofoam box -- enough for 10 to 15 snowballs, he says.Kyle Waring died last week. But not if you live in New England or surrounding states. "We will not ship snow to any states in the northeast!" says Waring's website, ShipSnowYo.com. "We're in the business of expunging snow!" His website and social media accounts claim to have filled more than 133 orders for snow -- more than 30 on Tuesday alone, his busiest day yet. With more than 45 total inches, Boston has set a record this winter for the snowiest month in its history. Most residents see the huge piles of snow choking their yards and sidewalks as a nuisance, but Waring saw an opportunity. According to Boston.com, it all started a few weeks ago, when Waring and his wife were shoveling deep snow from their yard in Manchester-by-the-Sea, a coastal suburb north of Boston. He joked about shipping the stuff to friends and family in warmer states, and an idea was born. His business slogan: "Our nightmare is your dream!" At first, ShipSnowYo sold snow packed into empty 16.9-ounce water bottles for $19.99, but the snow usually melted before it reached its destination. So this week, Waring began shipping larger amounts in the Styrofoam cubes, which he promises will arrive anywhere in the U.S. in less than 20 hours. He also has begun selling a 10-pound box of snow for $119. Many of his customers appear to be companies in warm-weather states who are buying the snow as a gag, he said. Whether Waring can sustain his gimmicky venture into the spring remains to be seen. But he has no shortage of product. "At this rate, it's going to be July until the snow melts," he told Boston.com. "But I've thought about taking this idea and running with it for other seasonal items. Maybe I'll ship some fall foliage."

Summary produced by t5-small fine-tuned over CNN/DM : 

Kyle Waring will ship you 6 pounds of snow in an insulated Styrofoam box for $89 . The self-styled entrepreneur says he will not ship snow to any states in the northeast . Waring's website and social media accounts claim to have filled more than 133 orders for snow . "We're in the business of expunging snow!" Waring says . He has begun selling a 10-pound box of snow for $119 . His business slogan: "Our nightmare is your

At first I thought this might be because the model hasn't converged as I just ran for 1 epoch but it's producing similar truncated outputs even for t5-small fine-tuned over cnn/dm for 5 epochs.Also this problem is not related to min_length or max_length parameters I think, as it produced similar outputs for all combinations of those two parameters. Tried changing --max_source_length, --max_target_length, --val_max_target_length, --test_max_target_length(these 4 parameters are present in finetune.py) parameter's values too from their default values before fine-tuning but no use.

What might be the reason for this truncation? Is this a problem of the fine-tuning code used to fine-tune pretrained models as pre-trained models don't produce this kind of outputs.

Comments:
Hi, what are the arguments for `.generate` method ? you can control the generation length using `max_length` and  `min_length` parameter. And if you want to see if there's something wrong with the fine-tuning code, then take the default t5-small model (it's already trained for summerization) and generate summaries  using it and compare with your model. This should give you some idea.

The arguments for the .generate method are (input_ids=input_ids, attention_mask=attention_mask, early_stopping= True, length_penalty = 2.0, max_length = 142, min_length = 56, no_repeat_ngram_size = 3, num_beams = 4), where input_ids and attention_mask are the corresponding tensors obtained through tokenizer.encode(). The config.json file of my saved t5-small model fine-tuned on cnn/dm is same as the default t5-small model(I cross-checked that).

I set the parameters I'm passing while running run_eval.py the same for both fine-tuned t5-small and default t5-small but the former produces truncated outputs whereas the later produces complete outputs.

Yeah I can control the generation length by the min_length and max_length parameters but in default t5-small model whatever be the above two parameters it always produced complete sentences whose length are within that range but in case of the fine-tuned model its giving truncated outputs for all combinations of these two parameters.

That's why I strongly felt that there was some problem with the fine-tuning code.

cc @sshleifer 

Since the default --max_source_length is 1024 and some articles in CNN are bigger than that, thought that the truncation of the input sentences was messing up the fine-tuned model and tried fine-tuning t5-small over xsum.

The xsum articles are relatively smaller and none of them exceeds 1024 tokens.
Used --max_target_length=60 -- val_max_target_length=60 --test_max_target_length=100 in finetune.py as they are mentioned as reasonable setting for XSUM.

Ran the script finetune_t5.sh for xsum, i.e.,
python finetune.py \
--data_dir=xsum \
--model_name_or_path=t5-small \
--learning_rate=3e-5 \
--train_batch_size=8 \
--eval_batch_size=4 \
--output_dir=xsum_results \
--max_source_length=1024 \
--val_check_interval=0.1 --n_val=200 \
--do_train --do_predict \
 $@

The outputs produced by the best_tfmr model for the test.souce dataset of xsum is still truncated as given by the test_generations.txt

Eg.

Article : (1st article of test.source dataset of xsum)

The London trio are up for best UK act and best album, as well as getting two nominations in the best song category."We got told like this morning 'Oh I think you're nominated'", said Dappy."And I was like 'Oh yeah, which one?' And now we've got nominated for four awards. I mean, wow!"Bandmate Fazer added: "We thought it's best of us to come down and mingle with everyone and say hello to the cameras. And now we find we've got four nominations."The band have two shots at the best song prize, getting the nod for their Tynchy Stryder collaboration Number One, and single Strong Again.Their album Uncle B will also go up against records by the likes of Beyonce and Kanye West.N-Dubz picked up the best newcomer Mobo in 2007, but female member Tulisa said they wouldn't be too disappointed if they didn't win this time around."At the end of the day we're grateful to be where we are in our careers."If it don't happen then it don't happen - live to fight another day and keep on making albums and hits for the fans."Dappy also revealed they could be performing live several times on the night.The group will be doing Number One and also a possible rendition of the War Child single, I Got Soul.The charity song is a  re-working of The Killers' All These Things That I've Done and is set to feature artists like Chipmunk, Ironik and Pixie Lott.This year's Mobos will be held outside of London for the first time, in Glasgow on 30 September.N-Dubz said they were looking forward to performing for their Scottish fans and boasted about their recent shows north of the border."We just done Edinburgh the other day," said Dappy."We smashed up an N-Dubz show over there. We done Aberdeen about three or four months ago - we smashed up that show over there! Everywhere we go we smash it up!"

Output :(output produced by the best_tfmr for the 1st article of test.source dataset of xsum)

N-Dubz have announced they have been nominated for the UK's best song prize. They have been told 'Oh yeah, which one?' - and now they've got four nominations. "We're going to be the best newcomer


Similarly, all the others outputs too are truncated at the end similar to the previous case(fine-tuned over CNN/DM) .
Once the default model goes under fine-tuning, it's unable to finish the summary with EOS token but getting cut abruptly.  

I think part of the problem may be that t5 tokenizer is not adding EOS token.
@patrickvonplaten 

```python
ipdb> tok_bart = AutoTokenizer.from_pretrained('facebook/bart-large-cnn')
ipdb> tok_bart('sentence')
{'input_ids': [0, 19530, 4086, 2], 'attention_mask': [1, 1, 1, 1]}
ipdb> tok_t5 = AutoTokenizer.from_pretrained('t5-small')
ipdb> tok_t5('sentence')
{'input_ids': [7142], 'attention_mask': [1]}
```
So maybe the model is training on targets without EOS, and eventually learns to stop generating it?

Thanks a lot for checking this @sshleifer! Yeah, I agree - I think T5 should add the EOS token to the end.
Is there a reason why T5 does not add the EOS token? @thomwolf @mfuntowicz @n1t0 ?

Yes, in case of T5 we manually need to add ` 
` at the end of text. I think this same issue is causing [this](https://discuss.huggingface.co/t/generate-very-short-summaries/277/5)  

@patil-suraj /others Have you ran clean experiments with and without adding `
`?

I don't want to merge #5866 this without more evidence that it is helpful, and [my first experiment](https://github.com/huggingface/transformers/pull/5866) did not result in any change. 

To those of you on many of these related issues, sorry for spamming.


Hi @sshleifer, in all of my T5 experiments I didn't use the bos token `
` at all, all of those experiments gave expected results (even better in some cases). But `
` is very important, without it the model generates really weird text, and its very easy to forget. So adding `
`  automatically is really  important. `
` won't matter

Ok, I'll merge the change. You won't need to add it anymore.

@tromedlov22 Did you ever figure out what the issue was? I have the same problem, doesn't seem to be an issue with the tokenizer adding eos since it's doing that. 

#5866 This solved my issue. If you still facing the issue, post your sample output maybe along with the input and the hyper-params you using.

hey guys, I am facing the same issue with truncation . My input:
When I first entered high school I was very nervous as it was a new school for me and it was a big adjustment
. I was overwhelmed with work and mentally wasn't staying optimistic as I found it hard to manage my time and make friends. I felt like I wasn't good enough, and this caused me to treat myself like I wasn't worthy of being at such a place
. In terms of behavior to others, I would say it made me more shy while still adapting to the new environment
.

Output:

when I first entered high school I was very nervous as it was a new school for me. I felt like I wasn't good enough to manage my time and make friends. it made me more shy while still adapting to


Generate
tokens_input,
                                min_length= 0,
                                max_length=50,
                                num_beams=4,
                                early_stopping=True,
                                no_repeat_ngram_size=3,
                                num_return_sequences=2,