Repository: deepset-ai/haystack

Issue:
Prompt Truncation Bug for models with smaller max_seq_length - truncation of all prompt
Bug was discovered by @recrudesce @rolandtannous and @danielbichuetti

It seems that when the prompt truncation PR was merged for OpenAI #4179  this made it so that if the specified maximum length in the PromptNode definition is equal to or larger than the models own max sequence length, then the truncation can also truncate the prompt itself + documents too e.g. for models that have a significantly lower token limit like flan models. So the Prompt is empty, and documents are also truncated.

**To Reproduce**
```
from haystack.nodes import EmbeddingRetriever, PromptNode, PromptTemplate
from haystack.document_stores import WeaviateDocumentStore
from haystack.pipelines import Pipeline
import sys

document_store = WeaviateDocumentStore(similarity="cosine", embedding_dim=768)
lfqa_prompt = PromptTemplate(
    name="lfqa",
    prompt_text="Generate a comprehensive, summarized answer to the given question using the provided paragraphs and reply as if you are Yoda.  \n\n Paragraphs: $documents \n\n Question: $query \n\n Answer:",
)

retriever = EmbeddingRetriever(
    document_store=document_store,
    embedding_model="flax-sentence-embeddings/all_datasets_v3_mpnet-base",
    model_format="sentence_transformers",
    top_k=20,
)

prompt_node = PromptNode(
    model_name_or_path="google/flan-t5-xl",
    default_prompt_template=lfqa_prompt,
    top_k=9,
    max_length=512,
)

pipe = Pipeline()
pipe.add_node(component=retriever, name="retriever", inputs=["Query"])
pipe.add_node(component=prompt_node, name="prompt_node", inputs=["retriever"])

output = pipe.run(query=sys.argv[1])

full_sentence = list(filter(lambda x: x.endswith("."), output["results"]))

longest_output = max(full_sentence, key=len)

print("**Answer:** " + longest_output
```

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?




Discussion:
I think that we should not truncate from Prompt to Answer, but in the opposite direction. And we warn the user: 
“Your prompt uses Z tokens, this LLM has a max seq. length of X tokens, you requested an Answer token space of Y tokens, which would exacerbate the model limits by I tokens. You may get no Answer. Please reduce your documents size or numbers, and/or your requested max Answer size.”

Going the opposite direction means that we are reserving space for an Answer and not minding about the prompt, documents, and the query itself. Any LLM without a prompt would be unusable, and the consequence is no answer or just a random-generated answer (noise).
Hey everyone, @zoltan-fedor also brought up a really good point in this issue https://github.com/deepset-ai/haystack/issues/4388. For at least the flan-T5 models it doesn't seem to make sense to have this token limit enforced. Check out this [comment](https://github.com/deepset-ai/haystack/issues/4388#issue-1621001303) from the linked issue for more info. 
Yes. FLAN limit set by the tokenizer is not really a hard limit because of the attention mech. There is a post about this on the HF forum.

It's a common fit as the memory usage increase exponentially.

@sjrl but anyway, even if not enforcing the recommended value which fits most scenarios, there are other models and use cases, e.g., even for OpenAI if we set the answer length to a value of 4000. So, maybe changing the way we warn users or the message would be advisable. 

Resolving patch:
diff --git a/haystack/nodes/prompt/invocation_layer/hugging_face.py b/haystack/nodes/prompt/invocation_layer/hugging_face.py
--- a/haystack/nodes/prompt/invocation_layer/hugging_face.py
+++ b/haystack/nodes/prompt/invocation_layer/hugging_face.py
@@ -48,8 +48,16 @@ def __init__(
         all PromptModelInvocationLayer instances, this instance of HFLocalInvocationLayer might receive some unrelated
         kwargs. Only kwargs relevant to the HFLocalInvocationLayer are considered. The list of supported kwargs
         includes: trust_remote_code, revision, feature_extractor, tokenizer, config, use_fast, torch_dtype, device_map.
-        For more details about these kwargs, see
+        For more details about pipeline kwargs in general, see
         Hugging Face [documentation](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.pipeline).
+
+        This layer supports two additional kwargs: generation_kwargs and model_max_length.
+
+        The generation_kwargs are used to customize text generation for the underlying pipeline. See Hugging
+        Face [docs](https://huggingface.co/docs/transformers/main/en/generation_strategies#customize-text-generation)
+        for more details.
+
+        The model_max_length is used to specify the custom sequence length for the underlying pipeline.
         """
         super().__init__(model_name_or_path)
         self.use_auth_token = use_auth_token
@@ -79,6 +87,7 @@ def __init__(
                 "torch_dtype",
                 "device_map",
                 "generation_kwargs",
+                "model_max_length",
             ]
             if key in kwargs
         }
@@ -89,6 +98,7 @@ def __init__(
 
         # save generation_kwargs for pipeline invocation
         self.generation_kwargs = model_input_kwargs.pop("generation_kwargs", {})
+        model_max_length = model_input_kwargs.pop("model_max_length", None)
 
         torch_dtype = model_input_kwargs.get("torch_dtype")
         if torch_dtype is not None:
@@ -121,6 +131,19 @@ def __init__(
         # max_length must be set otherwise HFLocalInvocationLayer._ensure_token_limit will fail.
         self.max_length = max_length or self.pipe.model.config.max_length
 
+        # we allow users to override the tokenizer's model_max_length because models like T5 have relative positional
+        # embeddings and can accept sequences of more than 512 tokens
+        if model_max_length is not None:
+            self.pipe.tokenizer.model_max_length = model_max_length
+
+        if self.max_length > self.pipe.tokenizer.model_max_length:
+            logger.warning(
+                "The max_length %s is greater than model_max_length %s. This might result in truncation of the "
+                "generated text. Please lower the max_length (number of answer tokens) parameter!",
+                self.max_length,
+                self.pipe.tokenizer.model_max_length,
+            )
+
     def invoke(self, *args, **kwargs):
         """
         It takes a prompt and returns a list of generated texts using the local Hugging Face transformers model
@@ -194,9 +217,10 @@ def _ensure_token_limit(self, prompt: Union[str, List[Dict[str, str]]]) -> Union
 
         :param prompt: Prompt text to be sent to the generative model.
         """
+        model_max_length = self.pipe.tokenizer.model_max_length
         n_prompt_tokens = len(self.pipe.tokenizer.tokenize(prompt))
         n_answer_tokens = self.max_length
-        if (n_prompt_tokens + n_answer_tokens) <= self.pipe.tokenizer.model_max_length:
+        if (n_prompt_tokens + n_answer_tokens) <= model_max_length:
             return prompt
 
         logger.warning(
@@ -204,14 +228,14 @@ def _ensure_token_limit(self, prompt: Union[str, List[Dict[str, str]]]) -> Union
             "answer length (%s tokens) fit within the max token limit (%s tokens). "
             "Shorten the prompt to prevent it from being cut off",
             n_prompt_tokens,
-            self.pipe.tokenizer.model_max_length - n_answer_tokens,
+            max(0, model_max_length - n_answer_tokens),
             n_answer_tokens,
-            self.pipe.tokenizer.model_max_length,
+            model_max_length,
         )
 
         tokenized_payload = self.pipe.tokenizer.tokenize(prompt)
         decoded_string = self.pipe.tokenizer.convert_tokens_to_string(
-            tokenized_payload[: self.pipe.tokenizer.model_max_length - n_answer_tokens]
+            tokenized_payload[: model_max_length - n_answer_tokens]
         )
         return decoded_string
 


Test patch:
diff --git a/test/prompt/test_prompt_node.py b/test/prompt/test_prompt_node.py
--- a/test/prompt/test_prompt_node.py
+++ b/test/prompt/test_prompt_node.py
@@ -302,6 +302,30 @@ def test_stop_words(prompt_model):
     assert "capital" in r[0] or "Germany" in r[0]
 
 
+@pytest.mark.unit
+def test_prompt_node_model_max_length(caplog):
+    prompt = "This is a prompt " * 5  # (26 tokens with t5 flan tokenizer)
+
+    # test that model_max_length is set to 1024
+    # test that model doesn't truncate the prompt if it is shorter than
+    # the model max length minus the length of the output
+    # no warning is raised
+    node = PromptNode(model_kwargs={"model_max_length": 1024})
+    assert node.prompt_model.model_invocation_layer.pipe.tokenizer.model_max_length == 1024
+    with caplog.at_level(logging.WARNING):
+        node.prompt(prompt)
+        assert len(caplog.text) <= 0
+
+    # test that model_max_length is set to 10
+    # test that model truncates the prompt if it is longer than the max length (10 tokens)
+    # a warning is raised
+    node = PromptNode(model_kwargs={"model_max_length": 10})
+    assert node.prompt_model.model_invocation_layer.pipe.tokenizer.model_max_length == 10
+    with caplog.at_level(logging.WARNING):
+        node.prompt(prompt)
+        assert "The prompt has been truncated from 26 tokens to 0 tokens" in caplog.text
+
+
 @pytest.mark.unit
 @patch("haystack.nodes.prompt.prompt_node.PromptModel")
 def test_prompt_node_streaming_handler_on_call(mock_model):