Repository: deepset-ai/haystack
Pull request: fix: Check Agent's prompt template variables and prompt resolver parameters are aligned
### Proposed Changes:
**What?**

* Introduced a method check_prompt_template that validates the PromptTemplate against the parameters supplied by the prompt parameter resolver
* Implemented logging of a detailed error message if ReAct-based agents' PromptTemplate does not contain `{transcript}` variable
* A debug message is now generated when the prompt parameter resolver supplies a parameter not utilized in the agent's template

**Why?**

* Fixes https://github.com/deepset-ai/haystack/issues/5130
* Supersedes https://github.com/deepset-ai/haystack/pull/5134 (if we agree on this approach)
* This fix mitigates potential complications and misunderstandings associated with the Agent's prompt template and prompt variables resolved by the resolver
* Enhances the user experience by generating more informative debug and error messages

### How did you test it?
* New unit tests were added to assess the functionality of the check_prompt_template method 
* Manual tests were also performed using `examples/agent_multihop_qa.py` to verify the correct functioning of the method

### Notes for the reviewer
* This change does not affect the overall functioning of the Agent class
* Please pay special attention to the way the parameters provided by the prompt parameter resolver are handled and the way the error and debug messages are logged
* The specified fix differs from a proposed fix to inject `{template}` into PromptTemplate automatically. It is a common practice to have such a template variable in other Agent implementations (i.e. `{scratchpad}` etc.), and users should not be burdened too much to add `{transcript}` in their templates. Otherwise, if we inject `{transcript}`, we make an exception and assumptions about the Agent's prompt template that might not hold in the future. There may still be opportunities for further improvements, and your suggestions are appreciated and welcomed.

Problem statement:
Haystack v1.17 introduces {transcript} at end of ReAct Agents
We noticed that the latest Haystack release introduces a setup where a user now needs to add a `{transcript}` at the end of a prompt, which we used to not require to do manually. This would break any custom agent prompt out there as of v1.17

One other concern/understanding issue @bilgeyucel brought up which is somewhat related to this is that we aren't clear on the distinction of a 'transcript' vs a memory. Any clarification there would be great too.

Hints from issue discussion:
Requiring `{transcript}` was added here: https://github.com/deepset-ai/haystack/pull/4931

Commit message: Check Agent's prompt template parameters and prompt resolver parameters are aligned

Patch to review:
diff --git a/haystack/agents/base.py b/haystack/agents/base.py
index 4b0eeba8ef..3a9a32809a 100644
--- a/haystack/agents/base.py
+++ b/haystack/agents/base.py
@@ -401,17 +401,15 @@ def _plan(self, query, current_step):
         # first resolve prompt template params
         template_params = self.prompt_parameters_resolver(query=query, agent=self, agent_step=current_step)
 
-        # if prompt node has no default prompt template, use agent's prompt template
-        if self.prompt_node.default_prompt_template is None:
-            prepared_prompt = next(self.prompt_template.fill(**template_params))
-            prompt_node_response = self.prompt_node(
-                prepared_prompt, stream_handler=AgentTokenStreamingHandler(self.callback_manager)
-            )
-        # otherwise, if prompt node has default prompt template, use it
-        else:
-            prompt_node_response = self.prompt_node(
-                stream_handler=AgentTokenStreamingHandler(self.callback_manager), **template_params
-            )
+        # check for template parameters mismatch
+        self.check_prompt_template(template_params)
+
+        # invoke via prompt node
+        prompt_node_response = self.prompt_node.prompt(
+            prompt_template=self.prompt_template,
+            stream_handler=AgentTokenStreamingHandler(self.callback_manager),
+            **template_params,
+        )
         return prompt_node_response
 
     def create_agent_step(self, max_steps: Optional[int] = None) -> AgentStep:
@@ -427,3 +425,32 @@ def prepare_data_for_memory(self, **kwargs) -> dict:
         return {
             k: v if isinstance(v, str) else next(iter(v)) for k, v in kwargs.items() if isinstance(v, (str, Iterable))
         }
+
+    def check_prompt_template(self, template_params: Dict[str, Any]) -> None:
+        """
+        Verifies that the Agent's prompt template is adequately populated with the correct parameters
+        provided by the prompt parameter resolver.
+
+        If template_params contains a parameter that is not specified in the prompt template, a warning is logged
+        at DEBUG level. Sometimes the prompt parameter resolver may provide additional parameters that are not
+        used by the prompt template. However, if the prompt parameter resolver provides a 'transcript'
+        parameter that is not used in the prompt template, an error is logged.
+
+        :param template_params: The parameters provided by the prompt parameter resolver.
+
+        """
+        unused_params = set(template_params.keys()) - set(self.prompt_template.prompt_params)
+
+        if "transcript" in unused_params:
+            logger.error(
+                "The 'transcript' parameter is missing from the Agent's prompt template. All ReAct agents "
+                "that go through multiple steps to reach a goal require this parameter. Please append {transcript} "
+                "to the end of the Agent's prompt template to ensure its proper functioning."
+            )
+
+        elif unused_params:
+            logger.debug(
+                "The Agent's prompt template does not utilize the following parameters provided by the "
+                "prompt parameter resolver: %s. Note that these parameters are available for use if needed.",
+                list(unused_params),
+            )
diff --git a/test/agents/test_agent.py b/test/agents/test_agent.py
index 971e58621c..e194573e53 100644
--- a/test/agents/test_agent.py
+++ b/test/agents/test_agent.py
@@ -2,18 +2,17 @@
 import os
 import re
 from typing import Tuple
+from unittest import mock
 from unittest.mock import Mock, patch
-
-from events import Events
-
-from haystack.agents.types import AgentTokenStreamingHandler, AgentToolLogger
 from test.conftest import MockRetriever, MockPromptNode
-from unittest import mock
+
 import pytest
+from events import Events
 
 from haystack import BaseComponent, Answer, Document
 from haystack.agents import Agent, AgentStep
 from haystack.agents.base import Tool, ToolsManager
+from haystack.agents.types import AgentTokenStreamingHandler, AgentToolLogger
 from haystack.nodes import PromptModel, PromptNode, PromptTemplate
 from haystack.pipelines import ExtractiveQAPipeline, DocumentSearchPipeline, BaseStandardPipeline
 
@@ -356,3 +355,66 @@ def test_agent_token_streaming_handler():
 
     assert result == "test"
     mock_callback.assert_called_once_with("test")  # assert that the mock callback was called with "test"
+
+
+@pytest.mark.unit
+def test_agent_prompt_template_parameter_has_transcript(caplog):
+    mock_prompt_node = Mock(spec=PromptNode)
+    prompt = PromptTemplate(prompt="I now have {query} as a template parameter but also {transcript}")
+    mock_prompt_node.get_prompt_template.return_value = prompt
+
+    agent = Agent(prompt_node=mock_prompt_node)
+    agent.check_prompt_template({"query": "test", "transcript": "some fake transcript"})
+    assert "The 'transcript' parameter is missing from the Agent's prompt template" not in caplog.text
+
+
+@pytest.mark.unit
+def test_agent_prompt_template_has_no_transcript(caplog):
+    mock_prompt_node = Mock(spec=PromptNode)
+    prompt = PromptTemplate(prompt="I only have {query} as a template parameter but I am missing transcript")
+    mock_prompt_node.get_prompt_template.return_value = prompt
+    agent = Agent(prompt_node=mock_prompt_node)
+    agent.check_prompt_template({"query": "test", "transcript": "some fake transcript"})
+    assert "The 'transcript' parameter is missing from the Agent's prompt template" in caplog.text
+
+
+@pytest.mark.unit
+def test_agent_prompt_template_unused_parameters(caplog):
+    caplog.set_level(logging.DEBUG)
+    mock_prompt_node = Mock(spec=PromptNode)
+    prompt = PromptTemplate(prompt="I now have {query} and {transcript} as template parameters")
+    mock_prompt_node.get_prompt_template.return_value = prompt
+    agent = Agent(prompt_node=mock_prompt_node)
+    agent.check_prompt_template({"query": "test", "transcript": "some fake transcript", "unused": "test"})
+    assert (
+        "The Agent's prompt template does not utilize the following parameters provided by the "
+        "prompt parameter resolver: ['unused']" in caplog.text
+    )
+
+
+@pytest.mark.unit
+def test_agent_prompt_template_multiple_unused_parameters(caplog):
+    caplog.set_level(logging.DEBUG)
+    mock_prompt_node = Mock(spec=PromptNode)
+    prompt = PromptTemplate(prompt="I now have strange {param_1} and {param_2} as template parameters")
+    mock_prompt_node.get_prompt_template.return_value = prompt
+    agent = Agent(prompt_node=mock_prompt_node)
+    agent.check_prompt_template({"query": "test", "unused": "test"})
+    assert (
+        "The Agent's prompt template does not utilize the following parameters provided by the "
+        "prompt parameter resolver: ['unused', 'query']" in caplog.text
+    )
+
+
+@pytest.mark.unit
+def test_agent_prompt_template_missing_parameters(caplog):
+    # in check_prompt_template we don't check that all prompt template parameters are filled
+    # prompt template resolution will do that and flag the missing parameters
+    # in check_prompt_template we check if some template parameters are not used
+    caplog.set_level(logging.DEBUG)
+    mock_prompt_node = Mock(spec=PromptNode)
+    prompt = PromptTemplate(prompt="I now have {query} and {transcript} as template parameters")
+    mock_prompt_node.get_prompt_template.return_value = prompt
+    agent = Agent(prompt_node=mock_prompt_node)
+    agent.check_prompt_template({"transcript": "test"})
+    assert not caplog.text


Code review comments:
diff_hunk: @@ -427,3 +425,32 @@ def prepare_data_for_memory(self, **kwargs) -> dict:
         return {
             k: v if isinstance(v, str) else next(iter(v)) for k, v in kwargs.items() if isinstance(v, (str, Iterable))
         }
+
+    def check_prompt_template(self, template_params: Dict[str, Any]) -> None:
+        """
+        Verifies that the Agent's prompt template is adequately populated with the correct parameters
+        provided by the prompt parameter resolver.
+
+        If template_params contains a parameter that is not specified in the prompt template, a warning is logged
+        at DEBUG level. Sometimes the prompt parameter resolver may provide additional parameters that are not
+        used by the prompt template. However, if the prompt parameter resolver provides a 'transcript'
+        parameter that is not used in the prompt template, an error is logged.
+
+        :param template_params: The parameters provided by the prompt parameter resolver.
+
+        """
+        unused_params = set(template_params.keys()) - set(self.prompt_template.prompt_params)
+
+        if "transcript" in unused_params:
+            logger.error(
+                "The 'transcript' parameter is missing from the Agent's prompt template. All ReAct agents "
+                "that go through multiple steps to reach a goal require this parameter. Please append {transcript} "
+                "to the end of the Agent's prompt template to ensure its proper functioning."
+            )
path: haystack/agents/base.py
text: @user1:
How about lowering it to warning and appending the missing `{transcript}` to the prompt? If we're raising this error we already know for sure that the transcript is necessary for the agent to work, so we might as well just add it.


@author:
Ok, makes sense.