Summarization of long text with T5 seems to output random memory content
Author: db1981
Pull request: no

Hello everyone,
I'm trying to summarizing a long text (~1800 words) using the T5 model. I set the max-length and min_length parameters as well, and when I do so, the output seems to contain random memory content...

Here my code:

#len_text=1755
    inputs = tokenizer.encode("summarize: " + proc_text, return_tensors="pt", max_length= (len_text + 1), truncation=True)
    outputs = model.generate(
    inputs, 
    max_length=round(len_text/3), #~590 words
    min_length=round(len_text/5),  #~350 words
    no_repeat_ngram_size=2,
    length_penalty=2.0, 
    num_beams=4, 
    early_stopping=False)

And here the output (~150 words only):

the key is to deploy predictive maintenance on assets where it makes sense. a combination of machine learning and data driven analytics can be used to plan, analyze, plan and expand across an enterprise to gain real savings and improvements. to achieve high operational efficiency and availability, ensuring that all assets are performing at peak performance with high availability and the lowest possible maintenance costs, companies are provided with some compelling options to manage their assets. the cost of adopting the wrong strategy can actually introduce failures in themselves. many companies in the process industry and energy sectors are still- - ­­.­ s­-­­»  - n­r­h­n hh gra­­,­[­&­...­_­/­*­—­s...  '­–­ [­“­?­;­e­ and­(­**­”­ (­---­]­" & – / _ »­­:­ "­ *­',--..,.-...[[__[*-_-"-&-s-–&&_-(-//-n'-,n-d-'rs/d&m––re»â«_â­â-ââ–â? â€[“?.”123467891012—:’...;‘’’–‘‘'&’-«­‘­’&–——‘–-—–==– ‘‘-’—-e, ‘’­l’ — e–’_—_–_&—&/–... ‘­o ’ ‘–/— ‘—’.– “­ ‘& ‘ re­re i–“– (–,–e’,’ (‘—e&,—...—/& (&.&...’/ ‘...– =–./’=- ‘- «– „– «-»–»- (— (-)-‘&'–«–*–: ‘e-a­ii­a–n–o–s–en­y­en–i—=­= ‘=&=?&*&# ;&e : r– and eng­t­d&‘ ‘ ‘ee,,&n&y-r&o-in–y–d’; ‘n’e ‘s&

I attached the full text I'm attempting to summarize as well.

Thanks!

[brief.txt](https://github.com/huggingface/transformers/files/6007898/brief.txt)


Comments:
Hey @db1981,

could you please post a fully reproducible code snippet in the following format:

```python
from transformers import T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained("...")

...
```

so that we can help you better?

Hi @patrickvonplaten thanks for you reply! Below the full code!

from transformers import T5ForConditionalGeneration, T5Tokenizer

```python
model_str = "t5-base"

model = T5ForConditionalGeneration.from_pretrained(model_str)
tokenizer = T5Tokenizer.from_pretrained(model_str)

full_path = "./brief.txt"

with open(full_path) as file: # Use file to refer to the file object
    text = file.read()

    proc_text = text.strip().replace("\n","")
    len_text = len(proc_text.split())

    inputs = tokenizer.encode("summarize: " + proc_text, return_tensors="pt", max_length= (len_text + 1), truncation=True)
    outputs = model.generate(
    inputs, 
    max_length=round(len_text/3), 
    min_length=round(len_text/5), 
    no_repeat_ngram_size=2,
    length_penalty=2.0, 
    num_beams=4, 
    early_stopping=False)

    final = tokenizer.decode(outputs[0], skip_special_tokens=True)

    print("%d words: %s" % (len(final.split()), final))


Yeah actually your input text is too long here I think -> T5 was only trained to handle up to 512 tokens (which corresponds to less than 512 words), so T5 will definitely not perform well for > 1500 words

@patrickvonplaten correct, T5 was originally trained to handle up to 512 tokens. But recently the Transformers library was updated to handle longer texts, using the Longformer apparoach. I thought that behind the scenes a longer text would have triggered automagically Longformer...I'll now give. it a shot with that class directly.

Thanks! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.