Perform max_input_tokens truncation with Summarization Pipeline
Author: brandenchan
Pull request: no

# 🚀 Feature request

I'd like to be able to set a `max_input_tokens` and configure a `truncation_strategy` in `SummarizationPipeline`.

Please let me know if I am missing something that already allows for this!

## Motivation

I initialize and call a summarization pipeline as follows

```
model = AutoModelForSeq2SeqLM.from_pretrained(pretrained_model_name_or_path=model_name_or_path, revision=model_version)
summarizer = pipeline("summarization", model=model, tokenizer=tokenizer, device=use_gpu)
summaries = summarizer(
            contexts,
            min_length=self.min_length,
            max_length=self.max_length,
            return_text=True,
            clean_up_tokenization_spaces=self.clean_up_tokenization_spaces,
        )
```

Currently when I pass a text that is longer than `"google/pegasus-xsum"`'s 512 token limit, I get the following warning
```
Token indices sequence length is longer than the specified maximum sequence length for this model (768 > 512). Running this sequence through the model will result in indexing errors
```
and my program crashes. I'd like to just be able to set a max input tokens and truncation strategy either when I init or call the `pipeline` (which in turns inits a `SummarizationPipeline`).

## Your contribution

If I am missing something or there already exists a way to get around this problem please let me know! If it wouldn't take too much effort to implement this, I might consider opening a PR.

Comments:
I think the argument you're looking for is `truncation`! Could you try passing `truncation=True` or `truncation="longest_first"` to your pipeline call?

Hi @LysandreJik this doesn't seem to be working for me. When I call 
```
summarizer = pipeline("summarization", model=model, tokenizer=tokenizer, device=use_gpu, truncation=True)
```
I get 
```
<ipython-input-8-315444041dc9> in <module>
      5 
      6 #Summarize
----> 7 summarizer = TransformersSummarizer(model_name_or_path="google/pegasus-xsum")
      8 
      9 p_summarizer = Pipeline()

~/Code/haystack/haystack/summarizer/transformers.py in __init__(self, model_name_or_path, model_version, tokenizer, max_length, min_length, use_gpu, clean_up_tokenization_spaces, separator_for_single_summary, generate_single_summary)
     87             tokenizer = model_name_or_path
     88         model = AutoModelForSeq2SeqLM.from_pretrained(pretrained_model_name_or_path=model_name_or_path, revision=model_version)
---> 89         self.summarizer = pipeline("summarization", model=model, tokenizer=tokenizer, device=use_gpu, truncation=True)
     90         self.max_length = max_length
     91         self.min_length = min_length

~/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/transformers/pipelines.py in pipeline(task, model, config, tokenizer, framework, revision, use_fast, **kwargs)
   3307                     break
   3308 
-> 3309     return task_class(model=model, tokenizer=tokenizer, modelcard=modelcard, framework=framework, task=task, **kwargs)

~/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/transformers/pipelines.py in __init__(self, *args, **kwargs)
   2353     def __init__(self, *args, **kwargs):
   2354         kwargs.update(task="summarization")
-> 2355         super().__init__(*args, **kwargs)
   2356 
   2357         self.check_model_type(

TypeError: __init__() got an unexpected keyword argument 'truncation'
```
and when I put the `truncation` argument into the call
```
        summaries = self.summarizer(
            contexts,
            min_length=self.min_length,
            max_length=self.max_length,
            return_text=True,
            clean_up_tokenization_spaces=self.clean_up_tokenization_spaces,
            truncation=True
        )
```
I get
```
TypeError                                 Traceback (most recent call last)

~/Code/haystack/haystack/pipeline.py in run(self, **kwargs)
    121                     logger.debug(f"Running node `{node_id}` with input `{node_input}`")
--> 122                     node_output, stream_id = self.graph.nodes[node_id]["component"].run(**node_input)
    123                 except Exception as e:

~/Code/haystack/haystack/summarizer/base.py in run(self, documents, generate_single_summary, **kwargs)
     36         if documents:
---> 37             results["documents"] = self.predict(documents=documents, generate_single_summary=generate_single_summary)
     38 

~/Code/haystack/haystack/summarizer/transformers.py in predict(self, documents, generate_single_summary)
    131             clean_up_tokenization_spaces=self.clean_up_tokenization_spaces,
--> 132             truncation=True
    133         )

~/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, return_tensors, return_text, clean_up_tokenization_spaces, *documents, **generate_kwargs)
   2438                 attention_mask=inputs["attention_mask"],
-> 2439                 **generate_kwargs,
   2440             )

~/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context

~/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, decoder_start_token_id, use_cache, num_beam_groups, diversity_penalty, prefix_allowed_tokens_fn, **model_kwargs)
    502             # add encoder_outputs to model_kwargs
--> 503             model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
    504 

~/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/transformers/generation_utils.py in _prepare_encoder_decoder_kwargs_for_generation(self, input_ids, model_kwargs)
     85         }
---> 86         model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
     87         return model_kwargs

~/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(

TypeError: forward() got an unexpected keyword argument 'truncation'


During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)

<ipython-input-4-315444041dc9> in <module>
     10 p_summarizer.add_node(component=es_retriever, name="Retriever", inputs=["Query"])
     11 p_summarizer.add_node(component=summarizer, name="Summarizer", inputs=["Retriever"])
---> 12 res = p_summarizer.run(query="Who is the father of Arya Stark??", top_k_retriever=10)
     13 
     14 pprint(res)

~/Code/haystack/haystack/pipeline.py in run(self, **kwargs)
    123                 except Exception as e:
    124                     tb = traceback.format_exc()
--> 125                     raise Exception(f"Exception while running node `{node_id}` with input `{node_input}`: {e}, full stack trace: {tb}")
    126                 queue.pop(node_id)
    127                 next_nodes = self.get_next_nodes(node_id, stream_id)

Exception: Exception while running node `Summarizer` with input `{'query': 'Who is the father of Arya Stark??', 'documents': [{'text': "\n===In the Riverlands===\nThe Stark army reaches the Twins, a bridge stronghold controlled by Walder Frey, who agrees to allow the army to cross the river and to commit his troops in return for Robb and Arya Stark marrying two of his children.\nTyrion Lannister suspects his father Tywin, who decides Tyrion and his barbarians will fight in the vanguard, wants him killed. As Tyrion, Bronn, and the prostitute Shae swap stories, Tyrion reveals he was married to a woman his father revealed was a prostitute, and made Tyrion watch as his guardsmen raped her.\nAs a Stark force approaches, Tyrion is trampled in the rush and regains consciousness to find the battle over. Tywin discovers the Stark host was only 2,000 men, not the 20,000 he was led to expect.\nRobb, having divided his forces, defeats Jaime Lannister's army with his remaining 18,000 men and captures Jaime.", 'id': '824a2362-004b-4234-a2fe-d0b82fdca52f', 'score': 11.656843, 'probability': 0.8110895501528345, 'question': None, 'meta': {'name': '450_Baelor.txt'}, 'embedding': None}, {'text': '\n===On the Kingsroad===\nCity Watchmen search the caravan for Gendry but are turned away by Yoren. Gendry tells Arya Stark that he knows she is a girl, and she reveals she is actually Arya Stark after learning that her father met Gendry before he was executed.', 'id': 'a04e3059-a941-4aa1-96e4-da0429c1a617', 'score': 11.3836775, 'probability': 0.8058019827683869, 'question': None, 'meta': {'name': '224_The_Night_Lands.txt'}, 'embedding': None}, {'text': '\n===\'\'A Game of Thrones\'\'===\nSansa Stark begins the novel by being betrothed to Crown Prince Joffrey Baratheon, believing Joffrey to be a gallant prince. While Joffrey and Sansa are walking through the woods, Joffrey notices Arya sparring with the butcher\'s boy, Mycah. A fight breaks out and Joffrey is attacked by Nymeria (Arya\'s direwolf) after Joffrey threatens to hurt Arya. Sansa lies to King Robert about the circumstances of the fight in order to protect both Joffrey and her sister Arya.  Since Arya ran off with her wolf to save it, Sansa\'s wolf is killed instead, estranging the Stark daughters.\nDuring the Tourney of the Hand to honour her father Lord Eddard Stark, Sansa Stark is enchanted by the knights performing in the event.  At the request of his mother, Queen Cersei Lannister, Joffrey spends a portion of the tourney with Sansa, but near the end he commands his guard Sandor Clegane, better known as The Hound, to take her back to her quarters. Sandor explains how his older brother, Gregor, aka "Mountain that Rides" pushed his face into a brazier of hot coals, for playing with one of his wooden toys.\nAfter Eddard discovers the truth of Joffrey\'s paternity, he tells Sansa that they will be heading back to Winterfell. Sansa is devastated and wishes to stay in King\'s Landing, so she runs off to inform Queen Cersei of her father\'s plans, unwittingly providing Cersei with the information needed to arrest her father. After Robert dies, Sansa begs Joffrey to show mercy on her father and he agrees, if Ned will swear an oath of loyalty, but executes him anyway, in front of Sansa.  Sansa is now effectively a hostage in King\'s Landing and finally sees Joffrey\'s true nature, after he forces her to look at the tarred head of her now-deceased father.', 'id': '1d2bb694-88fb-44eb-972d-6a72dd0009a1', 'score': 11.194147, 'probability': 0.8020677650524326, 'question': None, 'meta': {'name': '332_Sansa_Stark.txt'}, 'embedding': None}, {'text': "\n===Season 2===\nGendry travels North with Yoren and other Night's Watch recruits, including Arya Stark (disguised as an orphan boy named 'Arry), Lommy Greenhands, Hot Pie and Jaqen H'ghar. During their journey, they are stopped by the Goldcloaks of the City Watch, who demand that Yoren hand Gendry over to them - King Joffrey has ordered that all of his father Robert's bastards be killed, but Yoren turns the Goldcloaks away. Later, Gendry forces Arya to reveal her true identity, and is surprised to learn she is in fact Ned Stark's daughter. After the Goldcloaks get help from Ser Amory Lorch and his men, they ambush the travelling party. In the chaos, Yoren is killed. Gendry's life is then saved by Arya, who convinces the Goldcloaks that Lommy, who was killed during the attack, was in fact Gendry. Gendry and the rest of the recruits are then escorted to Harrenhal, the ruined castle-turned-prison. Ser Gregor Clegane oversees order here, and arbitrarily has many of the prisoners tortured and killed. Gendry is nearly tortured and killed but is saved by the arrival of Lord Tywin Lannister, who chides Clegane's men for their reckless treatment of the prisoners. Thanks to Jaqen H'ghars help, Arya, Gendry and Hot Pie are able to escape Harrenhal.", 'id': '689dac66-1347-43ea-8456-ab728566f9aa', 'score': 11.098732, 'probability': 0.8001674895900547, 'question': None, 'meta': {'name': '191_Gendry.txt'}, 'embedding': None}, {'text': '\n====Season 1====\nArya accompanies her father Ned and her sister Sansa to King\'s Landing. Before their departure, Arya\'s half-brother Jon Snow gifts Arya a sword which she dubs "Needle". On the Kingsroad, Arya is sparring with a butcher\'s boy, Mycah, when Sansa\'s betrothed Prince Joffrey Baratheon attacks Mycah, prompting Arya\'s direwolf Nymeria to bite Joffrey. Arya shoos Nymeria away so she is not killed, but is furious when Sansa later refuses to support her version of events. Mycah is later killed by Joffrey\'s bodyguard Sandor "The Hound" Clegane, earning him Arya\'s hatred. Ned arranges for Arya to have sword lessons with the Braavosi Syrio Forel, who later defends her from Ser Meryn Trant after Joffrey ascends to the throne and kills the Stark household. Arya flees the Red Keep, accidentally killing a stable boy in her escape, hiding out as a beggar in the streets of King\'s Landing. Ned is eventually taken to the Great Sept of Baelor to face judgment; he spots Arya in the crowd, and alerts the Night\'s Watch recruiter Yoren to her presence. Yoren prevents Arya from witnessing Ned\'s execution and has her pose as a boy, "Arry", to avoid detection as she joins Yoren\'s recruits traveling north to Castle Black.', 'id': '6947d45a-f420-4608-b396-774972193849', 'score': 10.634479, 'probability': 0.7907264571073125, 'question': None, 'meta': {'name': '43_Arya_Stark.txt'}, 'embedding': None}, {'text': '\n===In King\'s Landing===\nAfter Varys tells him that Sansa Stark\'s life is also at stake, Eddard "Ned" Stark agrees to make a false confession and swear loyalty to King Joffrey Baratheon.\nArya Stark finds a crowd gathering to watch her father be judged, and climbs onto the statue of Baelor the Blessed. Ned notices Arya and alerts Night\'s Watch recruiter Yoren. Before Sansa, Cersei Lannister, Joffrey and the Small Council, Ned confesses to treason and swears fealty to Joffrey. Instead of sparing Ned as promised, Joffrey orders him to be executed. Seeing that Arya has been rescued by Yoren, Ned accepts his fate and is beheaded.', 'id': 'eb0b5450-a583-428b-b412-266a70c48e30', 'score': 10.627409, 'probability': 0.7905801782386174, 'question': None, 'meta': {'name': '450_Baelor.txt'}, 'embedding': None}, {'text': "\n==== ''A Storm of Swords'' and ''A Feast for Crows'' ====\nPrior to the Red Wedding, Roose Bolton presents Robb Stark with a piece of Theon's skin, revealing that Ramsay has been flaying him; though disgusted, Robb acquiesces to Theon's further captivity, as Theon's father Balon has recently died and Theon's absence presents a succession crisis for the Ironborn. Following Robb Stark's death, King Tommen Baratheon legitimizes Ramsay as a Bolton. The Lannisters pass off Jeyne Poole as Arya Stark and send her north to be betrothed to Ramsay, with only the Lannisters and Boltons aware she is not the real Arya Stark.", 'id': '2e1f4f84-036c-4a2e-956b-090b20e32b25', 'score': 10.533444, 'probability': 0.788628898071797, 'question': None, 'meta': {'name': '487_Ramsay_Bolton.txt'}, 'embedding': None}, {'text': '\n===House Frey===\n* \'\'\'Walder Frey\'\'\' (seasons 1, 3, 6–7) portrayed by David Bradley. David Bradley Lord Walder Frey, nicknamed the "Late Lord Frey", is the head of House Frey, Lord of the Crossing and bannerman to House Tully. He is known for outliving his many wives (now on his 8th) and siring over 100 children (both bastard and trueborn). Because the use of the Twins became a strategic necessity for Robb\'s host, Walder was able to negotiate marriage contracts for his children to Robb and Arya Stark. But during Season 2 Robb broke his word and married Lady Talisa. For this slight, and willing to take advantage of the war\'s changing fortunes, he conspires with Tywin Lannister and Roose Bolton to betray Robb Stark at the wedding of his liege Edmure Tully, which he insists in return for support of his men. Frey hosts the infamous "Red Wedding" at which Robb Stark, his wife and mother are all murdered, refusing to spare Robb even as Catelyn holds Lady Frey hostage and threatens to slit her throat, which she does. He is subsequently granted Riverrun and its lands (though the title Lord Paramount of the Riverlands passes to Harrenhal and House Baelish) and expresses delight to take another young wife, but his house is irredeemably tarnished by the betrayal and House Tully\'s vassals refuse to submit to his rule. In Season 6, he is outraged when he hears of the Blackfish recapture\' of Riverrun and blames his sons Lothar and Black Walder for allowing him to escape. He then orders them to retake the castle using Edmure Tully as a hostage. Though they successfully retake Riverrun with the help of a Lannister host led by Jaime Lannister, Walder is ambushed shortly afterwards by Arya Stark, who slits his throat in revenge for the Red Wedding. In Season 7, Arya uses Walder\'s face to deceive and poison the rest of his family.\n* \'\'\'Lothar Frey\'\'\' (seasons 3, 6) portrayed by Tom Brooke in season 3, and by Daniel Tuite in season 6. One of Lord Walder Frey\'s many sons, nicknamed “Lame Lothar” because of his twisted leg. He and his half-brother Black Walder are sent by their father to Riverrun to propose a marriage between Lord Edmure Tully and Roslin Frey as terms for House Frey rejoining Robb Stark\'s campaign against the Lannisters. He is one of the first to commence the "Red Wedding", stabbing Talisa Stark in the womb several times and killing her and her unborn child. In the sixth season, he is ordered by Walder to retake Riverrun from Brynden Tully. Though they succeed with Lannister help, he is killed by Arya Stark, who subsequently bakes him into a pie.\n* \'\'\'Black Walder Rivers\'\'\' (seasons 3, 6) portrayed by Tim Plester. One of Lord Walder Frey\'s many bastard sons, nicknamed “Black Walder” for his dark demeanor. He and his half-brother Lame Lothar are sent by their father to Riverrun to propose a marriage between Lord Edmure Tully and Roslin Frey as terms for House Frey rejoining Robb Stark\'s campaign against the Lannister. He kills Catelyn Stark at the Red Wedding, after she slits Lady Frey\'s throat in retaliation for her son\'s death. In the sixth season, he takes part in the siege of Riverrun. Though the Freys reclaim the castle with the help of a Lannister host, Black Walder is killed shortly afterwards along with Lothar by Arya Stark, who bakes them both into a pie.', 'id': '6a8e1b47-e9ca-441c-b8ca-a4d259b58ece', 'score': 10.513748, 'probability': 0.7882182073903166, 'question': None, 'meta': {'name': '349_List_of_Game_of_Thrones_characters.txt'}, 'embedding': None}, {'text': '\n==== \'\'A Game of Thrones\'\' ====\nArya adopts a direwolf cub, which she names Nymeria after a legendary warrior queen. She travels with her father, Eddard, to King\'s Landing when he is made Hand of the King. Before she leaves, her half-brother Jon Snow has a smallsword made for her as a parting gift, which she names "Needle" after her least favorite ladylike activity.\nWhile taking a walk together, Prince Joffrey and her sister Sansa happen upon Arya and her friend, the low-born butcher apprentice Mycah, sparring in the woods with broomsticks.  Arya defends Mycah from Joffrey\'s torments and her direwolf Nymeria helps Arya fight off Joffrey, wounding his arm in the process.  Knowing that Nymeria will likely be killed in retribution, Arya chases her wolf away; but Sansa\'s direwolf Lady is killed in Nymeria\'s stead and Mycah is hunted down and killed by Sandor Clegane, Joffrey\'s bodyguard.\nIn King\'s Landing, her father discovers Arya\'s possession of Needle, but instead of confiscating it he arranges for fencing lessons under the Braavosi swordmaster Syrio Forel, who teaches her the style of fighting known as "water dancing".  After her father\'s arrest, Syrio is killed protecting her and Arya narrowly escapes capture.  She later witnesses the public execution of her father before falling under the protection of the Night\'s Watch recruiter Yoren.', 'id': '1642d35f-2d57-4c42-988f-a2b7afc3ac6c', 'score': 10.344947, 'probability': 0.7846745387990397, 'question': None, 'meta': {'name': '43_Arya_Stark.txt'}, 'embedding': None}, {'text': '\n== Character description ==\nGendry was conceived and born in King\'s Landing after Robert\'s Rebellion ended and is one of sixteen (twenty in the television series) bastard children of King Robert Baratheon,.  He is portrayed as tall and very muscled, having blue eyes and thick black hair, very similar to his biological father Robert and uncle Renly in their youth (Brienne of Tarth once almost mistook him for Renly for a moment).  He is stubborn and easily confused.\nDespite being one of the only four surviving biological children of King Robert (along with Mya Stone, Edric Storm and Bella Rivers), Gendry never knew who his father was.  His mother was reported to have been a worker at an alehouse who died when Gendry was still a young boy, and all he remembers of her was that she had blond hair. Later on, Tobho Mott, a master armourer from Qohor, was offered double the customary fee by a "lord" with concealed identity to take Gendry in as a smith apprentice, but accepted him for free after being impressed by the boy\'s physique.  Gendry turns out to be a talented apprentice, and likes to spend time polishing a bull head helmet that he proudly made for himself, which earned him the nickname "Bull" by Arya Stark.', 'id': 'bb5166b7-2e87-4095-ae0d-5fa5097f197e', 'score': 9.938516, 'probability': 0.7759666295293488, 'question': None, 'meta': {'name': '191_Gendry.txt'}, 'embedding': None}]}`: forward() got an unexpected keyword argument 'truncation', full stack trace: Traceback (most recent call last):
  File "/home/branden/Code/haystack/haystack/pipeline.py", line 122, in run
    node_output, stream_id = self.graph.nodes[node_id]["component"].run(**node_input)
  File "/home/branden/Code/haystack/haystack/summarizer/base.py", line 37, in run
    results["documents"] = self.predict(documents=documents, generate_single_summary=generate_single_summary)
  File "/home/branden/Code/haystack/haystack/summarizer/transformers.py", line 132, in predict
    truncation=True
  File "/home/branden/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/transformers/pipelines.py", line 2439, in __call__
    **generate_kwargs,
  File "/home/branden/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/home/branden/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/transformers/generation_utils.py", line 503, in generate
    model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(input_ids, model_kwargs)
  File "/home/branden/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/transformers/generation_utils.py", line 86, in _prepare_encoder_decoder_kwargs_for_generation
    model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
  File "/home/branden/Code/anaconda3/envs/haystack/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'truncation'
```



Is there a way you could share your environment/reproducible code example so that I can take a look? On a recent version, running this fails:

```py
from transformers import pipeline

sum = pipeline("summarization")
sum("hey" * 10000)
```
with the following error:
```
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```
so I do indeed get an out of range error. However, adding the `truncation` flag:

```py
sum("hey" * 10000, truncation=True)
```
works!

Hmmm ok I need to look more closely at my code first. Closing for now.

I think the key here was to put the truncation key in the summarization call instead of the pipeline call - 
```
sum = pipeline("summarization")
sum("hey" * 10000, truncation=True)
```

instead of 
```
sum = pipeline("summarization", truncation=True)
sum("hey" * 10000)
```