Repository: holoviz/panel

Issue:
PanelCallbackHandler design principle
yesterday I changed the PanelCallBackHandler handler to also output the final response from the agent.

![image](https://github.com/holoviz/panel/assets/42288570/444a258c-71e4-4158-a55a-1225ecf1db3a)

The change is marked below.

![image](https://github.com/holoviz/panel/assets/42288570/f8d86452-85f8-4a39-a1d8-b7d34c3dd375)

I can see that Streamlits Callback handler does not write the 'output'. The user/ developer have to manually do this. 

![image](https://github.com/holoviz/panel/assets/42288570/d79028f8-e30e-4e44-a814-b2e063bee409)

I'm thinking that maybe it was a mistake by me to change this. What do you think @ahuang11 ?


Discussion:
Right if users type return / yield in the callback, it would duplicate the results.

At the same time, all of the streaming from Langchain does not require any yield/return so if users use return, its also semi inconsistent. 

One last point to consider is the tool usually reports the final answer.
I'm more and more convinced it was a mistake. The CallbackHandler should be seen as something you attach to debug and can remove again. You should not change what else you do.

The code in the `callback` function should be

```python
pn_cb = pn.chat.langchain.PanelCallbackHandler(instance=instance)
response = state.pandas_df_agent.run(contents, callbacks=[pn_cb])
return response
```

not

```python
pn_cb = pn.chat.langchain.PanelCallbackHandler(instance=instance)
response = state.pandas_df_agent.run(contents, callbacks=[pn_cb])
```

I will make a PR to revert including updating the example notebooks. Then we can discuss from there.

Resolving patch:
diff --git a/panel/chat/langchain.py b/panel/chat/langchain.py
--- a/panel/chat/langchain.py
+++ b/panel/chat/langchain.py
@@ -22,7 +22,8 @@
 class PanelCallbackHandler(BaseCallbackHandler):
     """
     The Langchain `PanelCallbackHandler` itself is not a widget or pane, but is useful for rendering
-    and streaming output from Langchain Tools, Agents, and Chains as `ChatMessage` objects.
+    and streaming the *chain of thought* from Langchain Tools, Agents, and Chains
+    as `ChatMessage` objects.
 
     Reference: https://panel.holoviz.org/reference/chat/PanelCallbackHandler.html
 
@@ -67,13 +68,20 @@ def _update_active(self, avatar: str, label: str):
         if f"- {label}" not in self._active_user:
             self._active_user = f"{self._active_user} - {label}"
 
+    def _reset_active(self):
+        self._active_user = self._input_user
+        self._active_avatar = self._input_avatar
+        self._message = None
+
     def _stream(self, message: str):
-        return self.instance.stream(
-            message,
-            user=self._active_user,
-            avatar=self._active_avatar,
-            message=self._message,
-        )
+        if message.strip():
+            return self.instance.stream(
+                message,
+                user=self._active_user,
+                avatar=self._active_avatar,
+                message=self._message,
+            )
+        return self._message
 
     def on_llm_start(self, serialized: Dict[str, Any], *args, **kwargs):
         model = kwargs.get("invocation_params", {}).get("model_name", "")
@@ -99,9 +107,7 @@ def on_llm_end(self, response: LLMResult, *args, **kwargs):
                 respond=False,
             )
 
-        self._active_user = self._input_user
-        self._active_avatar = self._input_avatar
-        self._message = None
+        self._reset_active()
         return super().on_llm_end(response, *args, **kwargs)
 
     def on_llm_error(self, error: Union[Exception, KeyboardInterrupt], *args, **kwargs):
@@ -117,10 +123,12 @@ def on_tool_start(
         self, serialized: Dict[str, Any], input_str: str, *args, **kwargs
     ):
         self._update_active(DEFAULT_AVATARS["tool"], serialized["name"])
+        self._stream(f"Tool input: {input_str}")
         return super().on_tool_start(serialized, input_str, *args, **kwargs)
 
     def on_tool_end(self, output: str, *args, **kwargs):
         self._stream(output)
+        self._reset_active()
         return super().on_tool_end(output, *args, **kwargs)
 
     def on_tool_error(
@@ -136,9 +144,7 @@ def on_chain_start(
         return super().on_chain_start(serialized, inputs, *args, **kwargs)
 
     def on_chain_end(self, outputs: Dict[str, Any], *args, **kwargs):
-        if 'output' in outputs: # The chain is finished. Report the result
-            self.instance.disabled = self._disabled_state
-            self._stream(outputs['output'])
+        self.instance.disabled = self._disabled_state
         return super().on_chain_end(outputs, *args, **kwargs)
 
     def on_retriever_error(


Test patch:
diff --git a/panel/tests/chat/test_langchain.py b/panel/tests/chat/test_langchain.py
--- a/panel/tests/chat/test_langchain.py
+++ b/panel/tests/chat/test_langchain.py
@@ -30,7 +30,7 @@ async def callback(contents, user, instance):
         tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, callbacks=[callback_handler]
     )
     instance.send("2 + 2")
-    assert len(instance.objects) == 4
+    assert len(instance.objects) == 3
     assert instance.objects[1].object == "Action: Python REPL\nAction Input: print(2 + 2)"
     assert instance.objects[2].object == "Final Answer: 4"
-    assert instance.objects[3].object == "4"
+    assert not instance.disabled