Repository: deepset-ai/haystack

Issue:
Results of intermediate `PromptNodes` in a Multi-`PromptNode`-Pipelines are burried
**Describe the bug**
Currently it's not possible to access easily the output of multiple `PromptNodes` that are chained together. 
E.g. the code from the docs using a "question-generation" and a "question-answering" PromptNode in sequence only returns the answers as `results`, but the questions (which are a intermediate result) are burried inside `PromptNode`'s `invocation_context`:

```python
from haystack.nodes.prompt import PromptTemplate, PromptNode, PromptModel

# This is to set up the OpenAI model:
from getpass import getpass

api_key_prompt = "Enter OpenAI API key:" 
api_key = getpass(api_key_prompt)

# Specify the model you want to use:
prompt_open_ai = PromptModel(model_name_or_path="text-davinci-003", api_key=api_key)

# This sets up the default model:
prompt_model = PromptModel()

# Now let make one PromptNode use the default model and the other one the OpenAI model:
node_default_model = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="questions")
node_openai = PromptNode(prompt_open_ai, default_prompt_template="question-answering")

pipeline = Pipeline()
pipeline.add_node(component=node_default_model, name="prompt_node1", inputs=["Query"])
pipe.add_node(component=node_openai, name="prompt_node_2", inputs=["prompt_node1"])
output = pipe.run(query="not relevant", documents=[Document("Berlin is the capital of Germany")])
output["results"]
```
would produce something like this.
```python
["Berlin"]
```
while the question to the answer is burried under `output["meta"]["invocation_context"]["questions"]`


**Expected behavior**
`questions` and `answers` can be easily accessed together, e.g. by exposing `PromptNode`'s `output_variable` on the root level of node_output.

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?


Discussion:
We'll do this @tstadel thanks for bringing this request to our attention 

Resolving patch:
diff --git a/haystack/nodes/prompt/prompt_node.py b/haystack/nodes/prompt/prompt_node.py
--- a/haystack/nodes/prompt/prompt_node.py
+++ b/haystack/nodes/prompt/prompt_node.py
@@ -909,9 +909,14 @@ def run(
             **invocation_context,
         )
 
+        final_result: Dict[str, Any] = {}
         if self.output_variable:
             invocation_context[self.output_variable] = results
-        return {"results": results, "invocation_context": invocation_context}, "output_1"
+            final_result[self.output_variable] = results
+
+        final_result["results"] = results
+        final_result["invocation_context"] = invocation_context
+        return final_result, "output_1"
 
     def run_batch(
         self,


Test patch:
diff --git a/test/nodes/test_prompt_node.py b/test/nodes/test_prompt_node.py
--- a/test/nodes/test_prompt_node.py
+++ b/test/nodes/test_prompt_node.py
@@ -377,6 +377,7 @@ def test_complex_pipeline_yaml(tmp_path):
     response = result["results"][0]
     assert any(word for word in ["berlin", "germany", "population", "city", "amazing"] if word in response.casefold())
     assert len(result["invocation_context"]) > 0
+    assert len(result["questions"]) > 0
     assert "questions" in result["invocation_context"] and len(result["invocation_context"]["questions"]) > 0
 
 
@@ -415,6 +416,7 @@ def test_complex_pipeline_with_shared_prompt_model_yaml(tmp_path):
     response = result["results"][0]
     assert any(word for word in ["berlin", "germany", "population", "city", "amazing"] if word in response.casefold())
     assert len(result["invocation_context"]) > 0
+    assert len(result["questions"]) > 0
     assert "questions" in result["invocation_context"] and len(result["invocation_context"]["questions"]) > 0
 
 
@@ -462,6 +464,7 @@ def test_complex_pipeline_with_shared_prompt_model_and_prompt_template_yaml(tmp_
     response = result["results"][0]
     assert any(word for word in ["berlin", "germany", "population", "city", "amazing"] if word in response.casefold())
     assert len(result["invocation_context"]) > 0
+    assert len(result["questions"]) > 0
     assert "questions" in result["invocation_context"] and len(result["invocation_context"]["questions"]) > 0
 
 
@@ -541,6 +544,7 @@ def run_batch(
     response = result["results"][0]
     assert any(word for word in ["berlin", "germany", "population", "city", "amazing"] if word in response.casefold())
     assert len(result["invocation_context"]) > 0
+    assert len(result["questions"]) > 0
     assert "questions" in result["invocation_context"] and len(result["invocation_context"]["questions"]) > 0
 
 
@@ -601,4 +605,5 @@ def test_complex_pipeline_with_all_features(tmp_path):
     response = result["results"][0]
     assert any(word for word in ["berlin", "germany", "population", "city", "amazing"] if word in response.casefold())
     assert len(result["invocation_context"]) > 0
+    assert len(result["questions"]) > 0
     assert "questions" in result["invocation_context"] and len(result["invocation_context"]["questions"]) > 0