Repository: deepset-ai/haystack

Issue:
PromptNode should handle OpenAIRateLimitError with back-offs and retry inference attempt before giving up
**Is your feature request related to a problem? Please describe.**
While using PromptNode with the OpenAI InstructGPT model I encountered the following error:

```
>           raise openai_error
E           haystack.errors.OpenAIRateLimitError: API rate limit exceeded: {
E             "error": {
E               "message": "The server is currently overloaded with other requests. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if the error persists.",
E               "type": "server_error",
E               "param": null,
E               "code": null
E             }
E           }
haystack/nodes/prompt/prompt_node.py:448: OpenAIRateLimitError
```
As a PromptNode user using the OpenAI IntructGPT model, I would like the PromptNode to handle OpenAIRateLimitErrors for me. I don't want to create custom retry back-off loops; this should be done automatically. If, after a specified number of attempts, there are still OpenAIRateLimitErrors, it is ok to receive an exception. 

**Describe the solution you'd like**
A built-in mechanism for handling OpenAIRateLimitError, just like I am used to in other Haystack components. 

**Describe alternatives you've considered**
I considered adding backoff, but it is not practical, and the error code repeats a lot. In pipelines, it is much harder to do this as well.

**Additional context**
N/A


Discussion:

Resolving patch:
diff --git a/haystack/environment.py b/haystack/environment.py
--- a/haystack/environment.py
+++ b/haystack/environment.py
@@ -11,6 +11,9 @@
 HAYSTACK_EXECUTION_CONTEXT = "HAYSTACK_EXECUTION_CONTEXT"
 HAYSTACK_DOCKER_CONTAINER = "HAYSTACK_DOCKER_CONTAINER"
 
+# Any remote API (OpenAI, Cohere etc.)
+HAYSTACK_REMOTE_API_BACKOFF_SEC = "HAYSTACK_REMOTE_API_BACKOFF_SEC"
+HAYSTACK_REMOTE_API_MAX_RETRIES = "HAYSTACK_REMOTE_API_MAX_RETRIES"
 
 env_meta_data: Dict[str, Any] = {}
 
diff --git a/haystack/nodes/prompt/prompt_node.py b/haystack/nodes/prompt/prompt_node.py
--- a/haystack/nodes/prompt/prompt_node.py
+++ b/haystack/nodes/prompt/prompt_node.py
@@ -1,5 +1,6 @@
 import json
 import logging
+import os
 import re
 from abc import ABC, abstractmethod
 from string import Template
@@ -10,10 +11,12 @@
 from transformers import pipeline, AutoModelForSeq2SeqLM, StoppingCriteria, StoppingCriteriaList, AutoTokenizer
 
 from haystack import MultiLabel
+from haystack.environment import HAYSTACK_REMOTE_API_BACKOFF_SEC, HAYSTACK_REMOTE_API_MAX_RETRIES
 from haystack.errors import OpenAIError, OpenAIRateLimitError
 from haystack.modeling.utils import initialize_device_settings
 from haystack.nodes.base import BaseComponent
 from haystack.schema import Document
+from haystack.utils.reflection import retry_with_exponential_backoff
 
 logger = logging.getLogger(__name__)
 
@@ -420,6 +423,10 @@ def __init__(
             if key in kwargs
         }
 
+    @retry_with_exponential_backoff(
+        backoff_in_seconds=int(os.environ.get(HAYSTACK_REMOTE_API_BACKOFF_SEC, 5)),
+        max_retries=int(os.environ.get(HAYSTACK_REMOTE_API_MAX_RETRIES, 5)),
+    )
     def invoke(self, *args, **kwargs):
         """
         Invokes a prompt on the model. It takes in a prompt and returns a list of responses using a REST invocation.


Test patch: