Repository: deepset-ai/haystack

Issue:
TransformersSummarizer crashes if given long input
If the TransformersSummarizer is given an input that is longer than the model's max_seq_len, an error will be thrown. Instead, I think a warning message should be printed to console and the input text should be truncated so that the Node can still run.
TransformersSummarizer crashes if given long input
If the TransformersSummarizer is given an input that is longer than the model's max_seq_len, an error will be thrown. Instead, I think a warning message should be printed to console and the input text should be truncated so that the Node can still run.


Discussion:
@brandenchan You are talking about model tokens length. By default, the transformer model triggers an error if the token length is greater than 512.


Yup that's right. But the idea is that we could implement something that will shorten the input text that is going into the Summarizer. This way, we get a summary on the first 512 tokens, a warning message is printed to the console and the pipeline doesn't crash
Exactly same issue we faced in Obsei and currently truncating it to [`max_position_embeddings`](https://github.com/obsei/obsei/blob/master/obsei/analyzer/classification_analyzer.py#L44) instead of fixed 512 chars.

Currently we have [open issue](https://github.com/obsei/obsei/issues/164) to truncate input based on tokens instead of chars. Following idea suggested by @akar5h . I think similar can be done here as well.

> So would it not be more optimal that instead of splitting texts, we split the tokenizer output ( tokenizer.batch_encode_plus output) like input_ids , attention mask into sequences of length 512 and feed these splits to the model ?

Adding few more information to help contributor to understand more (again suggested by @akar5h) -

> https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb#scrollTo=n9qywopnIrJH&uniqifier=1
> Scroll down to Preprocessing the training data , . Here we can see how huggingface tokenizer provides a way to split longer length documents as an inbuilt operation .
@lalitpagaria Thanks for sharing. I will look into this solution.

Right now we are using the transformer pipeline module I think they aren't flexible to modify. I am looking into the documentation If we can provide `truncation=True` to the Summarizer module.

@brandenchan Can we rewrite the TransformersSummarizer class without using the Pipeline module.  The pipeline module is just an abstraction.
@brandenchan You are talking about model tokens length. By default, the transformer model triggers an error if the token length is greater than 512.


Yup that's right. But the idea is that we could implement something that will shorten the input text that is going into the Summarizer. This way, we get a summary on the first 512 tokens, a warning message is printed to the console and the pipeline doesn't crash
Exactly same issue we faced in Obsei and currently truncating it to [`max_position_embeddings`](https://github.com/obsei/obsei/blob/master/obsei/analyzer/classification_analyzer.py#L44) instead of fixed 512 chars.

Currently we have [open issue](https://github.com/obsei/obsei/issues/164) to truncate input based on tokens instead of chars. Following idea suggested by @akar5h . I think similar can be done here as well.

> So would it not be more optimal that instead of splitting texts, we split the tokenizer output ( tokenizer.batch_encode_plus output) like input_ids , attention mask into sequences of length 512 and feed these splits to the model ?

Adding few more information to help contributor to understand more (again suggested by @akar5h) -

> https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb#scrollTo=n9qywopnIrJH&uniqifier=1
> Scroll down to Preprocessing the training data , . Here we can see how huggingface tokenizer provides a way to split longer length documents as an inbuilt operation .
@lalitpagaria Thanks for sharing. I will look into this solution.

Right now we are using the transformer pipeline module I think they aren't flexible to modify. I am looking into the documentation If we can provide `truncation=True` to the Summarizer module.

@brandenchan Can we rewrite the TransformersSummarizer class without using the Pipeline module.  The pipeline module is just an abstraction.

Resolving patch:
diff --git a/haystack/summarizer/transformers.py b/haystack/summarizer/transformers.py
--- a/haystack/summarizer/transformers.py
+++ b/haystack/summarizer/transformers.py
@@ -1,5 +1,5 @@
 import logging
-from typing import List, Optional
+from typing import List, Optional, Set
 
 from transformers import pipeline
 from transformers.models.auto.modeling_auto import AutoModelForSeq2SeqLM
@@ -99,8 +99,10 @@ def __init__(
         self.clean_up_tokenization_spaces = clean_up_tokenization_spaces
         self.separator_for_single_summary = separator_for_single_summary
         self.generate_single_summary = generate_single_summary
+        self.print_log: Set[str] = set()
 
-    def predict(self, documents: List[Document], generate_single_summary: Optional[bool] = None) -> List[Document]:
+    def predict(self, documents: List[Document], generate_single_summary: Optional[bool] = None,
+                truncation: bool = True) -> List[Document]:
         """
         Produce the summarization from the supplied documents.
         These document can for example be retrieved via the Retriever.
@@ -110,6 +112,7 @@ def predict(self, documents: List[Document], generate_single_summary: Optional[b
                                         If set to "True", all docs will be joined to a single string that will then
                                         be summarized.
                                         Important: The summary will depend on the order of the supplied documents!
+        :param truncation: Truncate to a maximum length accepted by the model
         :return: List of Documents, where Document.text contains the summarization and Document.meta["context"]
                  the original, not summarized text
         """
@@ -130,12 +133,25 @@ def predict(self, documents: List[Document], generate_single_summary: Optional[b
             # Different order of same documents produce different summary.
             contexts = [self.separator_for_single_summary.join(contexts)]
 
+        encoded_input = self.summarizer.tokenizer(contexts, verbose=False)
+        for input_id in encoded_input['input_ids']:
+            tokens_count: int = len(input_id)
+            if tokens_count > self.summarizer.tokenizer.model_max_length:
+                truncation_warning = "One or more of your input document texts is longer than the specified " \
+                                     f"maximum sequence length for this summarizer model. "\
+                                     f"Generating summary from first {self.summarizer.tokenizer.model_max_length}"\
+                                     f" tokens."
+                if truncation_warning not in self.print_log:
+                    logger.warning(truncation_warning)
+                    self.print_log.add(truncation_warning)
+
         summaries = self.summarizer(
             contexts,
             min_length=self.min_length,
             max_length=self.max_length,
             return_text=True,
             clean_up_tokenization_spaces=self.clean_up_tokenization_spaces,
+            truncation=True,
         )
 
         result: List[Document] = []


Test patch: