Repository: deepset-ai/haystack

Issue:
Use finish_reason from OpenAI output to warn users if max_tokens is too short
**Is your feature request related to a problem? Please describe.**
Sometimes answers returned by the `PromptNode` or the `OpenAIAnswerGenerator` are cutoff and this can happen because the maximum amount of answer tokens can be set too short depending on the expected response. I would like a warning from Haystack suggesting I increase the number of maximum answer tokens to get better results.

**Describe the solution you'd like**
Fortunately, OpenAI returns the field `finish_reason` in their output (can be seen [here](https://platform.openai.com/docs/api-reference/making-requests)) and example reproduced below

```python
{
    "id": "cmpl-GERzeJQ4lvqPk8SkZu4XMIuR",
    "object": "text_completion",
    "created": 1586839808,
    "model": "text-davinci:003",
    "choices": [
        {
            "text": "\n\nThis is indeed a test",
            "index": 0,
            "logprobs": null,
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 5,
        "completion_tokens": 7,
        "total_tokens": 12
    }
}
```
When the `finish_reason` is `length` I think it would be helpful to log a warning to the user that `max_tokens` should probably be increased to produce better answers.

**Describe alternatives you've considered**
Increase the default `max_tokens` limit to handle more possible use cases, but this can be disadvantageous since this would limit the size of the context that can be sent with the query request. 

**Additional context**
I'm not sure what are all the possible return values for `finish_reason`. So far I can only find mention of `stop` and `length` in their docs [here](https://platform.openai.com/docs/guides/completion/best-practices).



Discussion:
Hey @sjrl...

Your proposal certainly makes sense!

I've researched this a bit...
The `finish_reason` parameter is relatively poorly documented:
as you also point out, it is reported only in a specific section (about the "inserting text" experimental feature).

if you deem it useful and adequate, I can open a PR...

Resolving patch:
diff --git a/haystack/nodes/answer_generator/openai.py b/haystack/nodes/answer_generator/openai.py
--- a/haystack/nodes/answer_generator/openai.py
+++ b/haystack/nodes/answer_generator/openai.py
@@ -69,7 +69,7 @@ def __init__(
         :param model: ID of the engine to use for generating the answer. You can select one of `"text-ada-001"`,
                      `"text-babbage-001"`, `"text-curie-001"`, or `"text-davinci-002"`
                      (from worst to best and from cheapest to most expensive). For more information about the models,
-                     refer to the [OpenAI Documentation](https://beta.openai.com/docs/models/gpt-3).
+                     refer to the [OpenAI Documentation](https://platform.openai.com/docs/models/gpt-3).
         :param max_tokens: The maximum number of tokens allowed for the generated Answer.
         :param top_k: Number of generated Answers.
         :param temperature: What sampling temperature to use. Higher values mean the model will take more risks and
@@ -199,6 +199,15 @@ def predict(
                 )
             raise openai_error
 
+        number_of_truncated_answers = sum(1 for ans in res["choices"] if ans["finish_reason"] == "length")
+        if number_of_truncated_answers > 0:
+            logger.warning(
+                "%s out of the %s answers have been truncated before reaching a natural stopping point."
+                "Consider increasing the max_tokens parameter to allow for longer answers.",
+                number_of_truncated_answers,
+                top_k,
+            )
+
         generated_answers = [ans["text"] for ans in res["choices"]]
         answers = self._create_answers(generated_answers, input_docs)
         result = {"query": query, "answers": answers}
@@ -222,7 +231,7 @@ def _build_prompt(self, query: str, documents: List[Document]) -> Tuple[str, Lis
         n_docs_tokens = [self._count_tokens(doc.content) for doc in documents]
         logger.debug("Number of tokens in documents: %s", n_docs_tokens)
 
-        # for length restrictions of prompt see: https://beta.openai.com/docs/api-reference/completions/create#completions/create-max_tokens
+        # for length restrictions of prompt see: https://platform.openai.com/docs/api-reference/completions/create#completions/create-max_tokens
         leftover_token_len = self.MAX_TOKENS_LIMIT - n_instruction_tokens - self.max_tokens
 
         # Add as many Documents as context as fit into the model
diff --git a/haystack/nodes/prompt/prompt_node.py b/haystack/nodes/prompt/prompt_node.py
--- a/haystack/nodes/prompt/prompt_node.py
+++ b/haystack/nodes/prompt/prompt_node.py
@@ -379,7 +379,7 @@ def supports(cls, model_name_or_path: str) -> bool:
 class OpenAIInvocationLayer(PromptModelInvocationLayer):
     """
     PromptModelInvocationLayer implementation for OpenAI's GPT-3 InstructGPT models. Invocations are made using REST API.
-    See [OpenAI GPT-3](https://beta.openai.com/docs/models/gpt-3) for more details.
+    See [OpenAI GPT-3](https://platform.openai.com/docs/models/gpt-3) for more details.
 
     Note: kwargs other than init parameter names are ignored to enable reflective construction of the class
     as many variants of PromptModelInvocationLayer are possible and they may have different parameters.
@@ -399,13 +399,13 @@ def __init__(
         kwargs. Only the kwargs relevant to OpenAIInvocationLayer are considered. The list of OpenAI-relevant
         kwargs includes: suffix, temperature, top_p, presence_penalty, frequency_penalty, best_of, n, max_tokens,
         logit_bias, stop, echo, and logprobs. For more details about these kwargs, see OpenAI
-        [documentation](https://beta.openai.com/docs/api-reference/completions/create).
+        [documentation](https://platform.openai.com/docs/api-reference/completions/create).
 
         """
         super().__init__(model_name_or_path, max_length)
         if not isinstance(api_key, str) or len(api_key) == 0:
             raise OpenAIError(
-                f"api_key {api_key} must be a valid OpenAI key. Visit https://beta.openai.com/ to get one."
+                f"api_key {api_key} must be a valid OpenAI key. Visit https://openai.com/api/ to get one."
             )
         self.api_key = api_key
         self.url = "https://api.openai.com/v1/completions"
@@ -443,7 +443,7 @@ def invoke(self, *args, **kwargs):
         :return: The responses are being returned.
 
         Note: Only kwargs relevant to OpenAI are passed to OpenAI rest API. Others kwargs are ignored.
-        For more details, see OpenAI [documentation](https://beta.openai.com/docs/api-reference/completions/create).
+        For more details, see OpenAI [documentation](https://platform.openai.com/docs/api-reference/completions/create).
         """
         prompt = kwargs.get("prompt")
         if not prompt:
@@ -492,6 +492,15 @@ def invoke(self, *args, **kwargs):
                 )
             raise openai_error
 
+        number_of_truncated_completions = sum(1 for ans in res["choices"] if ans["finish_reason"] == "length")
+        if number_of_truncated_completions > 0:
+            logger.warning(
+                "%s out of the %s completions have been truncated before reaching a natural stopping point."
+                "Consider increasing the max_tokens parameter to allow for longer completions.",
+                number_of_truncated_completions,
+                payload["n"],
+            )
+
         responses = [ans["text"].strip() for ans in res["choices"]]
         return responses
 


Test patch:
diff --git a/test/nodes/test_prompt_node.py b/test/nodes/test_prompt_node.py
--- a/test/nodes/test_prompt_node.py
+++ b/test/nodes/test_prompt_node.py
@@ -1,4 +1,5 @@
 import os
+import logging
 from typing import Optional, Union, List, Dict, Any, Tuple
 
 import pytest
@@ -248,6 +249,20 @@ def test_open_ai_prompt_with_params():
     assert len(r) == 1 and len(r[0]) > 0
 
 
+@pytest.mark.integration
+@pytest.mark.skipif(
+    not os.environ.get("OPENAI_API_KEY", None),
+    reason="Please export an env var called OPENAI_API_KEY containing the OpenAI API key to run this test.",
+)
+def test_open_ai_warn_if_max_tokens_is_too_short(caplog):
+    pm = PromptModel("text-davinci-003", api_key=os.environ["OPENAI_API_KEY"])
+    pn = PromptNode(pm)
+    optional_davinci_params = {"temperature": 0.5, "max_tokens": 2, "top_p": 1, "frequency_penalty": 0.5}
+    with caplog.at_level(logging.WARNING):
+        _ = pn.prompt("question-generation", documents=["Berlin is the capital of Germany."], **optional_davinci_params)
+        assert "Consider increasing the max_tokens parameter to allow for longer completions." in caplog.text
+
+
 @pytest.mark.integration
 @pytest.mark.parametrize("prompt_model", ["hf", "openai"], indirect=True)
 def test_stop_words(prompt_model):