Repository: run-llama/llama_index
Pull request: Add chat summary memory buffer
# Description

This PR introduces a new ChatSummaryMemoryBuffer to limit the chat history to a certain token length, and iteratively summarize all messages that do not fit in the memory buffer. This can be useful if you want to limit costs and latency (assuming the summarization prompt uses and generates fewer tokens than including the entire history). The original ChatMemoryBuffer gives you the option to truncate the history after a certain number of tokens, which is useful to limit costs and latency, but also removes potentially relevant information from the chat history. The new ChatSummaryMemoryBuffer aims to makes this a bit more flexible, so the user has more control over which chat_history is retained.

Fixes #12845

## New Package?

Did I fill in the `tool.llamahub` section in the `pyproject.toml` and provide a detailed README.md for my new integration or package?

- [X] Yes
- [ ] No

## Version Bump?

Did I bump the version in the `pyproject.toml` file of the package I am updating? (Except for the `llama-index-core` package)

- [ ] Yes
- [x] No

## Type of Change

Please delete options that are not relevant.
- [X ] New feature (non-breaking change which adds functionality)
- 
## How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

- [X] Added new unit/integration tests
- [X] Added new notebook (that tests end-to-end)
- [X] I stared at the code and made sure it makes sense

## Suggested Checklist:

- [X] I have performed a self-review of my own code
- [X] I have commented my code, particularly in hard-to-understand areas
- [X] I have made corresponding changes to the documentation
- [ ] I have added Google Colab support for the newly added notebooks.
- [X] My changes generate no new warnings
- [X] I have added tests that prove my fix is effective or that my feature works
- [X] New and existing unit tests pass locally with my changes
- [X] I ran `make format; make lint` to appease the lint gods


Problem statement:
[Question]: Is it possible integrate langchain ConversationBufferMemory ?
### Question Validation

- [X] I have searched both the documentation and discord for an answer.

### Question

Is it possible to integrate langchain `ConversationBufferMemory` with llama-index chat engine?
I'm checking on this because current llama-index `ChatMemoryBuffer` is not sufficient.

Hints from issue discussion:
<!-- Answer -->
Integrating langchain `ConversationBufferMemory` with the llama-index chat engine is indeed feasible. This process involves creating an adapter or bridge that allows llama-index to leverage langchain's `ConversationBufferMemory` for enhanced memory management. Specifically, you would need to either extend the `BaseMemory` class or develop a new class that adheres to the `BaseMemory` interface expected by llama-index, but internally utilizes langchain's memory management features.

To start, examine the `BaseMemory` class in llama-index for an understanding of the required interface and then look into how `ConversationBufferMemory` operates within langchain. This approach will enable you to incorporate langchain's advanced memory management capabilities into llama-index, potentially overcoming limitations you've encountered with the current `ChatMemoryBuffer`.

If you're addressing specific limitations of the `ChatMemoryBuffer`, such as token limit constraints, message role handling, or serialization issues, a tailored solution focusing on those aspects would be more effective. Understanding the exact features or capabilities you need will help in providing a more precise integration strategy.

For a successful integration, closely follow the implementation patterns and interface requirements outlined in llama-index's memory management system, adapting langchain's `ConversationBufferMemory` functionalities within that framework.

## Sources
- [How can I use Conversational memory with llama-index?](https://github.com/run-llama/llama_index/issues/1386)
- [Is llamaindex supports conversationBufferMemory?](https://github.com/run-llama/llama_index/issues/6775)
- [Support to add  conversation history  in the prompt](https://github.com/run-llama/llama_index/issues/6574)
- [llama-index-core/llama_index/core/memory/__init__.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/memory/__init__.py)
- [llama-index-core/tests/memory/test_chat_memory_buffer.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/tests/memory/test_chat_memory_buffer.py)
- [llama-index-core/llama_index/core/memory/chat_memory_buffer.py](https://github.com/run-llama/llama_index/blob/main/llama-index-core/llama_index/core/memory/chat_memory_buffer.py)

<!-- Dosu Comment Footer -->
<details>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>

@sura2k its not possible currently, but could maybe be possible.

I'm curious what ChatMemoryBuffer lacks that you need?
@logan-markewich sorry for late reply. I'm actually having a requirement to use something similar to  `ConversationSummaryBufferMemory`
We would also like something along the lines of ConversationSummaryBufferMemory to be implemented in LlamaIndex. The reason is that for our use case, we want to keep latency and costs down for our end users. Using the full history up to the context length increases costs and latency for each follow-up question due to increased number of tokens. Limiting the memory to the last X messages works, but is suboptimal, because the older messages may still have relevance. We would like older messages < X to be summarized, then appended by the full text of the last X messages.

@logan-markewich I can contribute to this if you are okay with adding this into LlamaIndex


Yea go for it

Commit message: Fix test comment

Patch to review:
diff --git a/docs/docs/examples/memory/ChatSummaryMemoryBuffer.ipynb b/docs/docs/examples/memory/ChatSummaryMemoryBuffer.ipynb
new file mode 100644
index 0000000000..3d61c7f292
--- /dev/null
+++ b/docs/docs/examples/memory/ChatSummaryMemoryBuffer.ipynb
@@ -0,0 +1,246 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "c62c1447-0afb-4fad-8dc6-389949c3496e",
+   "metadata": {},
+   "source": [
+    "# Chat Summary Memory Buffer\n",
+    "In this demo, we use the new *ChatSummaryMemoryBuffer* to limit the chat history to a certain token length, and iteratively summarize all messages that do not fit in the memory buffer. This can be useful if you want to limit costs and latency (assuming the summarization prompt uses and generates fewer tokens than including the entire history). The original *ChatMemoryBuffer* gives you the option to truncate the history after a certain number of tokens, which is useful to limit costs and latency, but also removes potentially relevant information from the chat history. The new *ChatSummaryMemoryBuffer* aims to makes this a bit more flexible, so the user has more control over which chat_history is retained."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c00a753b-df2c-4164-90c3-76b8a15f74c9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install llama-index-llms-openai\n",
+    "%pip install llama-index"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c1f24186-b86e-4580-b7b4-072e719d424f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "os.environ[\"OPENAI_API_KEY\"] = \"YOUR_API_KEY\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "db66e3bc-9791-497b-9e7d-386765dccf74",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llama_index.core.memory import ChatSummaryMemoryBuffer\n",
+    "from llama_index.core.llms import ChatMessage, MessageRole\n",
+    "from llama_index.llms.openai import OpenAI as OpenAiLlm\n",
+    "import tiktoken"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "68e26f76-f819-4e1a-bc47-f2ea855ee189",
+   "metadata": {},
+   "source": [
+    "First, we simulate some chat history that will not fit in the memory buffer in its entirety."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6402621a-4131-465c-b92d-1d9a8e7ee985",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chat_history = [\n",
+    "    ChatMessage(role=\"user\", content=\"What is LlamaIndex?\"),\n",
+    "    ChatMessage(\n",
+    "        role=\"assistant\",\n",
+    "        content=\"LlamaaIndex is the leading data framework for building LLM applications\",\n",
+    "    ),\n",
+    "    ChatMessage(role=\"user\", content=\"Can you give me some more details?\"),\n",
+    "    ChatMessage(\n",
+    "        role=\"assistant\",\n",
+    "        content=\"\"\"LlamaIndex is a framework for building context-augmented LLM applications. Context augmentation refers to any use case that applies LLMs on top of your private or domain-specific data. Some popular use cases include the following: \n",
+    "        Question-Answering Chatbots (commonly referred to as RAG systems, which stands for \"Retrieval-Augmented Generation\"), Document Understanding and Extraction, Autonomous Agents that can perform research and take actions\n",
+    "        LlamaIndex provides the tools to build any of these above use cases from prototype to production. The tools allow you to both ingest/process this data and implement complex query workflows combining data access with LLM prompting.\"\"\",\n",
+    "    ),\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f057a791-9d8e-43e5-b40a-6675b28f6fd0",
+   "metadata": {},
+   "source": [
+    "We set up the summarizer_llm, and create a *ChatSummaryMemoryBuffer* instance."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "67dbb654-4a66-43cf-9f6a-daead87a1084",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = \"gpt-4-0125-preview\"\n",
+    "summarizer_llm = OpenAiLlm(model_name=model, max_tokens=256)\n",
+    "tokenizer_fn = tiktoken.encoding_for_model(model).encode\n",
+    "memory = ChatSummaryMemoryBuffer.from_defaults(\n",
+    "    chat_history=chat_history,\n",
+    "    summarizer_llm=summarizer_llm,\n",
+    "    token_limit_full_text=2,\n",
+    "    tokenizer_fn=tokenizer_fn,\n",
+    ")\n",
+    "\n",
+    "history = memory.get()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e4e333c-6c33-4e01-b1a8-750d21800076",
+   "metadata": {},
+   "source": [
+    "When printing the history, we can observe that older messages have been summarized."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0821eb89-4164-4a06-b66c-ea2632706e11",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[ChatMessage(role=<MessageRole.SYSTEM: 'system'>, content='The user inquired about LlamaIndex, a leading data framework for developing LLM applications. The assistant explained that LlamaIndex is used for building context-augmented LLM applications, giving examples such as Question-Answering Chatbots, Document Understanding and Extraction, and Autonomous Agents. It was mentioned that LlamaIndex provides tools for ingesting and processing data, as well as implementing complex query workflows combining data access with LLM prompting.', additional_kwargs={})]\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(history)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fae3efe0-6889-49f7-9f21-6ced186f0609",
+   "metadata": {},
+   "source": [
+    "Let's add some new chat history."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7ddb295c-c5c0-4faf-b0e5-a451d1d26d60",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "new_chat_history = [\n",
+    "    ChatMessage(role=\"user\", content=\"Why context augmentation?\"),\n",
+    "    ChatMessage(\n",
+    "        role=\"assistant\",\n",
+    "        content=\"LLMs offer a natural language interface between humans and data. Widely available models come pre-trained on huge amounts of publicly available data. However, they are not trained on your data, which may be private or specific to the problem you're trying to solve. It's behind APIs, in SQL databases, or trapped in PDFs and slide decks. LlamaIndex provides tooling to enable context augmentation. A popular example is Retrieval-Augmented Generation (RAG) which combines context with LLMs at inference time. Another is finetuning.\",\n",
+    "    ),\n",
+    "    ChatMessage(role=\"user\", content=\"Who is LlamaIndex for?\"),\n",
+    "    ChatMessage(\n",
+    "        role=\"assistant\",\n",
+    "        content=\"LlamaIndex provides tools for beginners, advanced users, and everyone in between. Our high-level API allows beginner users to use LlamaIndex to ingest and query their data in 5 lines of code. For more complex applications, our lower-level APIs allow advanced users to customize and extend any module—data connectors, indices, retrievers, query engines, reranking modules—to fit their needs.\",\n",
+    "    ),\n",
+    "]\n",
+    "memory.put(new_chat_history[0])\n",
+    "memory.put(new_chat_history[1])\n",
+    "memory.put(new_chat_history[2])\n",
+    "memory.put(new_chat_history[3])\n",
+    "history = memory.get()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9cedc25a-cf6c-45c2-977d-c3fef14f1ea5",
+   "metadata": {},
+   "source": [
+    "The history will now be updated with a new summary, containing the latest information."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2e03428c-069f-4c17-9882-17f3fd0dcf4c",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[ChatMessage(role=<MessageRole.SYSTEM: 'system'>, content='The user asked about LlamaIndex and why context augmentation is important. The assistant explained that LlamaIndex is for building context-augmented LLM applications, which are necessary because LLMs are not trained on specific private or problem-specific data. LlamaIndex provides tools for beginners and advanced users to ingest and query data easily, as well as customize modules for more complex applications.', additional_kwargs={})]\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(history)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f402643f-74ee-4c8f-902f-59296d4d8edf",
+   "metadata": {},
+   "source": [
+    "Using a longer *token_limit_full_text* allows the user to control the balance between retaining the full chat history and summarization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "91d34b9b-04c3-4453-85ab-8afe59a7e763",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[ChatMessage(role=<MessageRole.SYSTEM: 'system'>, content='The user inquired about LlamaIndex, and the assistant explained that it is a data framework for creating context-augmented LLM applications. These applications can include question-answering chatbots, document understanding and extraction, and autonomous agents for research and actions. LlamaIndex provides tools for building these applications from prototype to production, enabling data processing, complex query workflows, and LLM prompting.', additional_kwargs={}), ChatMessage(role=<MessageRole.USER: 'user'>, content='Why context augmentation?', additional_kwargs={}), ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content=\"LLMs offer a natural language interface between humans and data. Widely available models come pre-trained on huge amounts of publicly available data. However, they are not trained on your data, which may be private or specific to the problem you're trying to solve. It's behind APIs, in SQL databases, or trapped in PDFs and slide decks. LlamaIndex provides tooling to enable context augmentation. A popular example is Retrieval-Augmented Generation (RAG) which combines context with LLMs at inference time. Another is finetuning.\", additional_kwargs={}), ChatMessage(role=<MessageRole.USER: 'user'>, content='Who is LlamaIndex for?', additional_kwargs={}), ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, content='LlamaIndex provides tools for beginners, advanced users, and everyone in between. Our high-level API allows beginner users to use LlamaIndex to ingest and query their data in 5 lines of code. For more complex applications, our lower-level APIs allow advanced users to customize and extend any module—data connectors, indices, retrievers, query engines, reranking modules—to fit their needs.', additional_kwargs={})]\n"
+     ]
+    }
+   ],
+   "source": [
+    "memory = ChatSummaryMemoryBuffer.from_defaults(\n",
+    "    chat_history=chat_history + new_chat_history,\n",
+    "    summarizer_llm=summarizer_llm,\n",
+    "    token_limit_full_text=256,\n",
+    "    tokenizer_fn=tokenizer_fn,\n",
+    ")\n",
+    "print(memory.get())"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "llama-index-py3.9",
+   "language": "python",
+   "name": "llama-index-py3.9"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/llama-index-core/llama_index/core/memory/__init__.py b/llama-index-core/llama_index/core/memory/__init__.py
index 9ef3af4f6f..0a49dbe85d 100644
--- a/llama-index-core/llama_index/core/memory/__init__.py
+++ b/llama-index-core/llama_index/core/memory/__init__.py
@@ -1,4 +1,5 @@
 from llama_index.core.memory.chat_memory_buffer import ChatMemoryBuffer
+from llama_index.core.memory.chat_summary_memory_buffer import ChatSummaryMemoryBuffer
 from llama_index.core.memory.types import BaseMemory
 
-__all__ = ["BaseMemory", "ChatMemoryBuffer"]
+__all__ = ["BaseMemory", "ChatMemoryBuffer", "ChatSummaryMemoryBuffer"]
diff --git a/llama-index-core/llama_index/core/memory/chat_summary_memory_buffer.py b/llama-index-core/llama_index/core/memory/chat_summary_memory_buffer.py
new file mode 100644
index 0000000000..01fbe2f758
--- /dev/null
+++ b/llama-index-core/llama_index/core/memory/chat_summary_memory_buffer.py
@@ -0,0 +1,245 @@
+from typing import Any, Callable, Dict, List, Optional
+
+from pydantic.fields import PrivateAttr
+
+from llama_index.core.base.llms.types import ChatMessage, MessageRole
+from llama_index.core.bridge.pydantic import Field, root_validator
+from llama_index.core.llms.llm import LLM
+from llama_index.core.memory.types import DEFAULT_CHAT_STORE_KEY, BaseMemory
+from llama_index.core.storage.chat_store import BaseChatStore, SimpleChatStore
+from llama_index.core.utils import get_tokenizer
+
+DEFAULT_TOKEN_LIMIT_FULL_TEXT_RATIO = 0.75
+DEFAULT_TOKEN_LIMIT_FULL_TEXT = 2000
+SUMMARIZE_PROMPT = "The following is a conversation between the user and assistant. Write a concise summary about the contents of this conversation."
+
+
+# TODO: Add option for last N user/assistant history interactions instead of token limit
+class ChatSummaryMemoryBuffer(BaseMemory):
+    """Buffer for storing chat history that uses the full text for the latest
+    {token_limit_full_text}.
+
+    All older messages are iteratively summarized using the {summarizer_llm} provided, with
+    the max number of tokens defined by the {summarizer_llm}.
+
+    User can specify whether initial tokens (usually a system prompt)
+    should be counted as part of the {token_limit_full_text}
+    using the parameter {count_initial_tokens}.
+
+    This buffer is useful to retain the most important information from a
+    long chat history, while limiting the token count and latency
+    of each request to the LLM.
+    """
+
+    token_limit_full_text: int
+    count_initial_tokens: bool = False
+    summarizer_llm: Optional[LLM] = None
+    summarize_prompt: Optional[str] = None
+    tokenizer_fn: Callable[[str], List] = Field(
+        # NOTE: mypy does not handle the typing here well, hence the cast
+        default_factory=get_tokenizer,
+        exclude=True,
+    )
+
+    chat_store: BaseChatStore = Field(default_factory=SimpleChatStore)
+    chat_store_key: str = Field(default=DEFAULT_CHAT_STORE_KEY)
+
+    _token_count: int = PrivateAttr(default=0)
+
+    @root_validator(pre=True)
+    def validate_memory(cls, values: dict) -> dict:
+        """Validate the memory."""
+        # Validate token limits
+        token_limit_full_text = values.get("token_limit_full_text", -1)
+        if token_limit_full_text < 1:
+            raise ValueError(
+                "Token limit for full-text messages must be set and greater than 0."
+            )
+
+        # Validate tokenizer -- this avoids errors when loading from json/dict
+        tokenizer_fn = values.get("tokenizer_fn", None)
+        if tokenizer_fn is None:
+            values["tokenizer_fn"] = get_tokenizer()
+
+        return values
+
+    @classmethod
+    def from_defaults(
+        cls,
+        chat_history: Optional[List[ChatMessage]] = None,
+        summarizer_llm: Optional[LLM] = None,
+        chat_store: Optional[BaseChatStore] = None,
+        chat_store_key: str = DEFAULT_CHAT_STORE_KEY,
+        token_limit_full_text: Optional[int] = None,
+        tokenizer_fn: Optional[Callable[[str], List]] = None,
+        summarize_prompt: Optional[str] = None,
+        count_initial_tokens: bool = False,
+    ) -> "ChatSummaryMemoryBuffer":
+        """Create a chat memory buffer from an LLM
+        and an initial list of chat history messages.
+        """
+        if summarizer_llm is not None:
+            context_window = summarizer_llm.metadata.context_window
+            token_limit_full_text = token_limit_full_text or int(
+                context_window * DEFAULT_TOKEN_LIMIT_FULL_TEXT_RATIO
+            )
+        elif token_limit_full_text is None:
+            token_limit_full_text = DEFAULT_TOKEN_LIMIT_FULL_TEXT
+
+        chat_store = chat_store or SimpleChatStore()
+        chat_history = chat_history or []
+        chat_store.set_messages(chat_store_key, chat_history)
+
+        summarize_prompt = summarize_prompt or SUMMARIZE_PROMPT
+        return cls(
+            summarizer_llm=summarizer_llm,
+            token_limit_full_text=token_limit_full_text,
+            # TODO: Check if we can get the tokenizer from the llm
+            tokenizer_fn=tokenizer_fn or get_tokenizer(),
+            summarize_prompt=summarize_prompt,
+            chat_store=chat_store,
+            chat_store_key=chat_store_key,
+            count_initial_tokens=count_initial_tokens,
+        )
+
+    @classmethod
+    def class_name(cls) -> str:
+        """Get class name."""
+        return "ChatSummaryMemoryBuffer"
+
+    def to_string(self) -> str:
+        """Convert memory to string."""
+        return self.json()
+
+    def to_dict(self, **kwargs: Any) -> dict:
+        """Convert memory to dict."""
+        return self.dict()
+
+    @classmethod
+    def from_string(cls, json_str: str) -> "ChatMemoryBuffer":
+        raise NotImplementedError("This is not yet supported.")
+
+    @classmethod
+    def from_dict(
+        cls, data: Dict[str, Any], **kwargs: Any
+    ) -> "ChatSummaryMemoryBuffer":
+        raise NotImplementedError("This is not yet supported.")
+
+    def get(self, initial_token_count: int = 0, **kwargs: Any) -> List[ChatMessage]:
+        """Get chat history."""
+        chat_history = self.get_all()
+        if len(chat_history) == 0:
+            return []
+
+        # Give the user the choice whether to count the system prompt or not
+        if self.count_initial_tokens:
+            if initial_token_count > self.token_limit_full_text:
+                raise ValueError("Initial token count exceeds token limit")
+            self._token_count = initial_token_count
+
+        (
+            chat_history_full_text,
+            chat_history_to_be_summarized,
+        ) = self._split_messages_summary_or_full_text(chat_history)
+
+        if self.summarizer_llm is None or len(chat_history_to_be_summarized) == 0:
+            # Simply remove the message that don't fit the buffer anymore
+            updated_history = chat_history_full_text
+        else:
+            updated_history = [
+                self._summarize_oldest_chat_history(chat_history_to_be_summarized),
+                *chat_history_full_text,
+            ]
+
+        self.reset()
+        self._token_count = 0
+        self.set(updated_history)
+
+        return updated_history
+
+    def get_all(self) -> List[ChatMessage]:
+        """Get all chat history."""
+        return self.chat_store.get_messages(self.chat_store_key)
+
+    def put(self, message: ChatMessage) -> None:
+        """Put chat history."""
+        # ensure everything is serialized
+        self.chat_store.add_message(self.chat_store_key, message)
+
+    def set(self, messages: List[ChatMessage]) -> None:
+        """Set chat history."""
+        self.chat_store.set_messages(self.chat_store_key, messages)
+
+    def reset(self) -> None:
+        """Reset chat history."""
+        self.chat_store.delete_messages(self.chat_store_key)
+
+    def get_token_count(self) -> int:
+        """Returns the token count of the memory buffer (excluding the last assistant response)."""
+        return self._token_count
+
+    def _token_count_for_messages(self, messages: List[ChatMessage]) -> int:
+        """Get token count for list of messages."""
+        if len(messages) <= 0:
+            return 0
+
+        msg_str = " ".join(str(m.content) for m in messages)
+        return len(self.tokenizer_fn(msg_str))
+
+    def _split_messages_summary_or_full_text(
+        self, chat_history: List[ChatMessage]
+    ) -> (List[ChatMessage], List[ChatMessage]):
+        """Determine which messages will be included as full text,
+        and which will have to be summarized by the summarizer_llm.
+        """
+        chat_history_full_text = []
+        message_count = len(chat_history)
+        while (
+            message_count > 0
+            and self._token_count + self._token_count_for_messages([chat_history[-1]])
+            <= self.token_limit_full_text
+        ):
+            # traverse the history in reverse order, when token limit is about to be
+            # exceeded, we stop, so remaining messages are summarized
+            self._token_count += self._token_count_for_messages([chat_history[-1]])
+            chat_history_full_text.insert(0, chat_history.pop())
+            message_count -= 1
+
+        chat_history_to_be_summarized = chat_history.copy()
+        return chat_history_full_text, chat_history_to_be_summarized
+
+    def _summarize_oldest_chat_history(
+        self, chat_history_to_be_summarized: List[ChatMessage]
+    ) -> ChatMessage:
+        """Use the summarizer_llm to summarize the messages that do not fit into the
+        buffer.
+        """
+        # Only summarize if there is new information to be summarized
+        if (
+            len(chat_history_to_be_summarized) == 1
+            and chat_history_to_be_summarized[0].role == MessageRole.SYSTEM
+        ):
+            return chat_history_to_be_summarized[0]
+
+        summarize_prompt = ChatMessage(
+            role=MessageRole.SYSTEM,
+            content=self._get_prompt_to_summarize(chat_history_to_be_summarized),
+        )
+        # TODO: Maybe it is better to pass a list of history to llm
+        return ChatMessage(
+            role=MessageRole.SYSTEM,
+            content=self.summarizer_llm.chat([summarize_prompt]).message.content,
+        )
+
+    def _get_prompt_to_summarize(
+        self, chat_history_to_be_summarized: List[ChatMessage]
+    ):
+        """Ask the LLM to summarize the chat history so far."""
+        # TODO: This probably works better when question/answers are considered together.
+        prompt = '"Transcript so far: '
+        for msg in chat_history_to_be_summarized:
+            prompt += msg.role + ": "
+            prompt += msg.content + "\n\n"
+        prompt += '"\n\n'
+        prompt += self.summarize_prompt
+        return prompt
diff --git a/llama-index-core/tests/memory/test_chat_summary_memory_buffer.py b/llama-index-core/tests/memory/test_chat_summary_memory_buffer.py
new file mode 100644
index 0000000000..e5fdcb5f50
--- /dev/null
+++ b/llama-index-core/tests/memory/test_chat_summary_memory_buffer.py
@@ -0,0 +1,287 @@
+from typing import List, Sequence, Any
+import pickle
+
+import pytest
+from pydantic.fields import PrivateAttr
+
+from llama_index.core.base.llms.types import ChatResponse
+from llama_index.core.llms import ChatMessage, MessageRole, MockLLM
+from llama_index.core.memory.chat_summary_memory_buffer import (
+    ChatSummaryMemoryBuffer,
+)
+from llama_index.core.utils import get_tokenizer
+
+tokenizer = get_tokenizer()
+
+
+def _get_role_alternating_order(i: int):
+    if i % 2 == 0:
+        return MessageRole.USER
+    return MessageRole.ASSISTANT
+
+
+USER_CHAT_MESSAGE = ChatMessage(role=MessageRole.USER, content="first message")
+USER_CHAT_MESSAGE_TOKENS = len(tokenizer(str(USER_CHAT_MESSAGE.content)))
+LONG_USER_CHAT_MESSAGE = ChatMessage(
+    role=MessageRole.USER,
+    content="".join(
+        ["This is a message that is longer than the proposed token length"] * 10
+    ),
+)
+LONG_RUNNING_CONVERSATION = [
+    ChatMessage(role=_get_role_alternating_order(i), content=f"Message {i}")
+    for i in range(5)
+]
+LONG_USER_CHAT_MESSAGE_TOKENS = len(tokenizer(str(LONG_USER_CHAT_MESSAGE.content)))
+
+
+class MockSummarizerLLM(MockLLM):
+    _i: int = PrivateAttr()
+    _responses: List[ChatMessage] = PrivateAttr()
+    _max_tokens: int = PrivateAttr()
+
+    def __init__(self, responses: List[ChatMessage], max_tokens: int = 512) -> None:
+        self._i = 0  # call counter, determines which response to return
+        self._responses = responses  # list of responses to return
+        self._max_tokens = max_tokens  # Max tokens for summary
+
+        super().__init__()
+
+    def chat(self, messages: Sequence[ChatMessage], **kwargs: Any) -> ChatResponse:
+        del messages  # unused
+
+        # For this mockLLM, we assume tokens are separated by spaces
+        max_tokens = self._max_tokens
+        if self._max_tokens > len(self._responses[self._i].content):
+            max_tokens = len(self._responses[self._i].content)
+        response_tokens = " ".join(
+            self._responses[self._i].content.split(" ")[0:max_tokens]
+        )
+
+        response = ChatResponse(
+            message=ChatMessage(role=MessageRole.ASSISTANT, content=response_tokens),
+        )
+        self._i += 1
+        return response
+
+    def set_max_tokens(self, max_tokens):
+        self._max_tokens = max_tokens
+
+
+FIRST_SUMMARY_RESPONSE = "First, the user asked what an LLM was, and the assistant explained the basic ideas."
+SECOND_SUMMARY_RESPONSE = (
+    "The conversation started about LLMs. It then continued about LlamaIndex."
+)
+
+
+@pytest.fixture()
+def summarizer_llm():
+    return MockSummarizerLLM(
+        responses=[
+            ChatMessage(
+                content=FIRST_SUMMARY_RESPONSE,
+                role=MessageRole.ASSISTANT,
+            ),
+            ChatMessage(
+                content=SECOND_SUMMARY_RESPONSE,
+                role=MessageRole.ASSISTANT,
+            ),
+        ]
+    )
+
+
+def test_put_get(summarizer_llm) -> None:
+    # Given one message with fewer tokens than token_limit_full_text
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        chat_history=[USER_CHAT_MESSAGE], summarizer_llm=summarizer_llm
+    )
+
+    # When I get the chat history from the memory
+    history = memory.get()
+
+    # Then the history should contain the full message
+    assert len(history) == 1
+    assert history[0].content == USER_CHAT_MESSAGE.content
+
+
+def test_put_get_summarize_long_message(summarizer_llm) -> None:
+    # Given one message with more tokens than token_limit_full_text
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        chat_history=[LONG_USER_CHAT_MESSAGE],
+        token_limit_full_text=2,
+        summarizer_llm=summarizer_llm,
+    )
+
+    # When I get the chat history from the memory
+    history = memory.get()
+
+    # Then the history should contain the summarized message
+    assert len(history) == 1
+    assert history[0].content == FIRST_SUMMARY_RESPONSE
+
+
+def test_put_get_summarize_part_of_conversation(summarizer_llm) -> None:
+    # Given a chat history where only 2 responses fit in the token_limit_full_text
+    tokens_most_recent_messages = sum(
+        [
+            len(tokenizer(str(LONG_RUNNING_CONVERSATION[-i].content)))
+            for i in range(1, 3)
+        ]
+    )
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        chat_history=LONG_RUNNING_CONVERSATION.copy(),
+        token_limit_full_text=tokens_most_recent_messages,
+        summarizer_llm=summarizer_llm,
+    )
+
+    # When I get the chat history from the memory
+    history = memory.get()
+
+    # Then the history should contain the full message for the latest two and
+    # a summary for the older messages
+    assert len(history) == 3
+    assert history[0].content == FIRST_SUMMARY_RESPONSE
+    assert history[0].role == MessageRole.SYSTEM
+    assert history[1].content == "Message 3"
+    assert history[2].content == "Message 4"
+
+    # When I add a new message to the history
+    memory.put(ChatMessage(role=MessageRole.USER, content="Message 5"))
+
+    # Then the history should re-summarize
+    history = memory.get()
+    assert len(history) == 3
+    assert history[0].content == SECOND_SUMMARY_RESPONSE
+    assert history[0].role == MessageRole.SYSTEM
+    assert history[1].content == "Message 4"
+    assert history[2].content == "Message 5"
+
+
+def test_get_when_initial_tokens_less_than_limit_returns_history() -> None:
+    # Given some initial tokens much smaller than token_limit and message tokens
+    initial_tokens = 5
+
+    # Given a user message
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        token_limit_full_text=1000, chat_history=[USER_CHAT_MESSAGE]
+    )
+
+    # When I get the chat history from the memory
+    history = memory.get(initial_tokens)
+
+    # Then the history should contain the message
+    assert len(history) == 1
+    assert history[0] == USER_CHAT_MESSAGE
+
+
+def test_get_when_initial_tokens_exceed_limit_raises_value_error() -> None:
+    # Given some initial tokens exceeding token_limit
+    initial_tokens = 50
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        chat_history=[USER_CHAT_MESSAGE],
+        token_limit_full_text=initial_tokens - 1,
+        count_initial_tokens=True,
+    )
+
+    # When I get the chat history from the memory
+    with pytest.raises(ValueError) as error:
+        memory.get(initial_tokens)
+
+    # Then a value error should be raised
+    assert str(error.value) == "Initial token count exceeds token limit"
+
+
+def test_set() -> None:
+    memory = ChatSummaryMemoryBuffer.from_defaults(chat_history=[USER_CHAT_MESSAGE])
+
+    memory.put(USER_CHAT_MESSAGE)
+
+    assert len(memory.get()) == 2
+
+    memory.set([USER_CHAT_MESSAGE])
+    assert len(memory.get()) == 1
+
+
+def test_max_tokens_without_summarizer() -> None:
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        chat_history=[USER_CHAT_MESSAGE], token_limit_full_text=5
+    )
+
+    memory.put(USER_CHAT_MESSAGE)
+    assert len(memory.get()) == 2
+
+    # do we limit properly
+    memory.put(USER_CHAT_MESSAGE)
+    memory.put(USER_CHAT_MESSAGE)
+    assert len(memory.get()) == 2
+
+    # In ChatSummaryMemoryBuffer, we overwrite the actual chat history
+    assert len(memory.get_all()) == 2
+
+    # does get return in the correct order?
+    memory.put(ChatMessage(role=MessageRole.USER, content="test message2"))
+    assert memory.get()[-1].content == "test message2"
+    assert len(memory.get()) == 2
+
+
+def test_max_tokens_with_summarizer(summarizer_llm) -> None:
+    max_tokens = 1
+    summarizer_llm.set_max_tokens(max_tokens)
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        summarizer_llm=summarizer_llm,
+        chat_history=[USER_CHAT_MESSAGE],
+        token_limit_full_text=5,
+    )
+
+    # do we limit properly
+    memory.put(USER_CHAT_MESSAGE)
+    memory.put(USER_CHAT_MESSAGE)
+    memory_results = memory.get()
+    assert len(memory_results) == 3
+    # Oldest message is summarized
+    assert memory_results[0].content == " ".join(
+        FIRST_SUMMARY_RESPONSE.split(" ")[0:max_tokens]
+    )
+    assert memory_results[0].role == MessageRole.SYSTEM
+
+    # In ChatSummaryMemoryBuffer, we overwrite the actual chat history
+    assert len(memory.get_all()) == 3
+
+    # does get return in the correct order?
+    memory.put(ChatMessage(role=MessageRole.USER, content="test message2"))
+    memory_results = memory.get()
+    assert memory_results[-1].content == "test message2"
+    assert len(memory_results) == 3
+    # Oldest message is summarized based on the latest information
+    assert memory_results[0].content == " ".join(
+        SECOND_SUMMARY_RESPONSE.split(" ")[0:max_tokens]
+    )
+
+
+def test_sting_save_load() -> None:
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        chat_history=[USER_CHAT_MESSAGE], token_limit_full_text=5
+    )
+
+    json_str = memory.to_string()
+
+    with pytest.raises(NotImplementedError):
+        ChatSummaryMemoryBuffer.from_string(json_str)
+
+
+def test_dict_save_load() -> None:
+    memory = ChatSummaryMemoryBuffer.from_defaults(
+        chat_history=[USER_CHAT_MESSAGE], token_limit_full_text=5
+    )
+
+    json_dict = memory.to_dict()
+
+    with pytest.raises(NotImplementedError):
+        ChatSummaryMemoryBuffer.from_dict(json_dict)
+
+
+def test_pickle() -> None:
+    """Unpickleable tiktoken tokenizer should be circumvented when pickling."""
+    memory = ChatSummaryMemoryBuffer.from_defaults()
+    bytes_ = pickle.dumps(memory)
+    assert isinstance(pickle.loads(bytes_), ChatSummaryMemoryBuffer)


Code review comments:
diff_hunk: @@ -0,0 +1,245 @@
+from typing import Any, Callable, Dict, List, Optional
+
+from pydantic.fields import PrivateAttr
+
+from llama_index.core.base.llms.types import ChatMessage, MessageRole
+from llama_index.core.bridge.pydantic import Field, root_validator
+from llama_index.core.llms.llm import LLM
+from llama_index.core.memory.types import DEFAULT_CHAT_STORE_KEY, BaseMemory
+from llama_index.core.storage.chat_store import BaseChatStore, SimpleChatStore
+from llama_index.core.utils import get_tokenizer
+
+DEFAULT_TOKEN_LIMIT_FULL_TEXT_RATIO = 0.75
+DEFAULT_TOKEN_LIMIT_FULL_TEXT = 2000
+SUMMARIZE_PROMPT = "The following is a conversation between the user and assistant. Write a concise summary about the contents of this conversation."
+
+
+# TODO: Add option for last N user/assistant history interactions instead of token limit
+class ChatSummaryMemoryBuffer(BaseMemory):
+    """Buffer for storing chat history that uses the full text for the latest
+    {token_limit_full_text}.
+
+    All older messages are iteratively summarized using the {summarizer_llm} provided, with
+    the max number of tokens defined by the {summarizer_llm}.
+
+    User can specify whether initial tokens (usually a system prompt)
+    should be counted as part of the {token_limit_full_text}
+    using the parameter {count_initial_tokens}.
+
+    This buffer is useful to retain the most important information from a
+    long chat history, while limiting the token count and latency
+    of each request to the LLM.
+    """
+
+    token_limit_full_text: int
path: llama-index-core/llama_index/core/memory/chat_summary_memory_buffer.py
text: @user1:
also for consistency, thoughts on calling this just `token_limit` ?

@author:
I agree. Updated.

diff_hunk: @@ -0,0 +1,245 @@
+from typing import Any, Callable, Dict, List, Optional
+
+from pydantic.fields import PrivateAttr
+
+from llama_index.core.base.llms.types import ChatMessage, MessageRole
+from llama_index.core.bridge.pydantic import Field, root_validator
+from llama_index.core.llms.llm import LLM
+from llama_index.core.memory.types import DEFAULT_CHAT_STORE_KEY, BaseMemory
+from llama_index.core.storage.chat_store import BaseChatStore, SimpleChatStore
+from llama_index.core.utils import get_tokenizer
+
+DEFAULT_TOKEN_LIMIT_FULL_TEXT_RATIO = 0.75
+DEFAULT_TOKEN_LIMIT_FULL_TEXT = 2000
+SUMMARIZE_PROMPT = "The following is a conversation between the user and assistant. Write a concise summary about the contents of this conversation."
+
+
+# TODO: Add option for last N user/assistant history interactions instead of token limit
path: llama-index-core/llama_index/core/memory/chat_summary_memory_buffer.py
text: @user1:
side note, slightly off topic but: does N messages make sense? What if a single user or assistant message is huge? Also another consideration, for function calls, openai (and others) require pairs of an assistant and tool message to be in memory

@author:
If a single user or assistant message is huge, then indeed, this is not such a great idea. On the other hand, the user has knowledge about the prompts and the expected length of the responses (perhaps something along the lines of "Answer in a single sentence: what is LlamaIndex?") and it could be a use case to consider only the last 3 or 5 "interactions", say.

The reasons I considered it is to ensure that USER/ASSISTANT messages are always considered in pairs, and to give some more flexibility to the end user. But it's more of a nice to have for the future if there is appetite for that flexibility.

diff_hunk: @@ -0,0 +1,245 @@
+from typing import Any, Callable, Dict, List, Optional
+
+from pydantic.fields import PrivateAttr
+
+from llama_index.core.base.llms.types import ChatMessage, MessageRole
+from llama_index.core.bridge.pydantic import Field, root_validator
+from llama_index.core.llms.llm import LLM
+from llama_index.core.memory.types import DEFAULT_CHAT_STORE_KEY, BaseMemory
+from llama_index.core.storage.chat_store import BaseChatStore, SimpleChatStore
+from llama_index.core.utils import get_tokenizer
+
+DEFAULT_TOKEN_LIMIT_FULL_TEXT_RATIO = 0.75
+DEFAULT_TOKEN_LIMIT_FULL_TEXT = 2000
+SUMMARIZE_PROMPT = "The following is a conversation between the user and assistant. Write a concise summary about the contents of this conversation."
+
+
+# TODO: Add option for last N user/assistant history interactions instead of token limit
+class ChatSummaryMemoryBuffer(BaseMemory):
+    """Buffer for storing chat history that uses the full text for the latest
+    {token_limit_full_text}.
+
+    All older messages are iteratively summarized using the {summarizer_llm} provided, with
+    the max number of tokens defined by the {summarizer_llm}.
+
+    User can specify whether initial tokens (usually a system prompt)
+    should be counted as part of the {token_limit_full_text}
+    using the parameter {count_initial_tokens}.
+
+    This buffer is useful to retain the most important information from a
+    long chat history, while limiting the token count and latency
+    of each request to the LLM.
+    """
+
+    token_limit_full_text: int
+    count_initial_tokens: bool = False
+    summarizer_llm: Optional[LLM] = None
path: llama-index-core/llama_index/core/memory/chat_summary_memory_buffer.py
text: @user1:
for consistency with the rest of the framework, lets call this field just `llm`

@author:
I agree. Updated.

diff_hunk: @@ -0,0 +1,245 @@
+from typing import Any, Callable, Dict, List, Optional
+
+from pydantic.fields import PrivateAttr
+
+from llama_index.core.base.llms.types import ChatMessage, MessageRole
+from llama_index.core.bridge.pydantic import Field, root_validator
+from llama_index.core.llms.llm import LLM
+from llama_index.core.memory.types import DEFAULT_CHAT_STORE_KEY, BaseMemory
+from llama_index.core.storage.chat_store import BaseChatStore, SimpleChatStore
+from llama_index.core.utils import get_tokenizer
+
+DEFAULT_TOKEN_LIMIT_FULL_TEXT_RATIO = 0.75
+DEFAULT_TOKEN_LIMIT_FULL_TEXT = 2000
+SUMMARIZE_PROMPT = "The following is a conversation between the user and assistant. Write a concise summary about the contents of this conversation."
+
+
+# TODO: Add option for last N user/assistant history interactions instead of token limit
+class ChatSummaryMemoryBuffer(BaseMemory):
+    """Buffer for storing chat history that uses the full text for the latest
+    {token_limit_full_text}.
+
+    All older messages are iteratively summarized using the {summarizer_llm} provided, with
+    the max number of tokens defined by the {summarizer_llm}.
+
+    User can specify whether initial tokens (usually a system prompt)
+    should be counted as part of the {token_limit_full_text}
+    using the parameter {count_initial_tokens}.
+
+    This buffer is useful to retain the most important information from a
+    long chat history, while limiting the token count and latency
+    of each request to the LLM.
+    """
+
+    token_limit_full_text: int
+    count_initial_tokens: bool = False
+    summarizer_llm: Optional[LLM] = None
+    summarize_prompt: Optional[str] = None
+    tokenizer_fn: Callable[[str], List] = Field(
+        # NOTE: mypy does not handle the typing here well, hence the cast
+        default_factory=get_tokenizer,
+        exclude=True,
+    )
+
+    chat_store: BaseChatStore = Field(default_factory=SimpleChatStore)
+    chat_store_key: str = Field(default=DEFAULT_CHAT_STORE_KEY)
+
+    _token_count: int = PrivateAttr(default=0)
+
+    @root_validator(pre=True)
+    def validate_memory(cls, values: dict) -> dict:
+        """Validate the memory."""
+        # Validate token limits
+        token_limit_full_text = values.get("token_limit_full_text", -1)
+        if token_limit_full_text < 1:
+            raise ValueError(
+                "Token limit for full-text messages must be set and greater than 0."
+            )
+
+        # Validate tokenizer -- this avoids errors when loading from json/dict
+        tokenizer_fn = values.get("tokenizer_fn", None)
+        if tokenizer_fn is None:
+            values["tokenizer_fn"] = get_tokenizer()
+
+        return values
+
+    @classmethod
+    def from_defaults(
+        cls,
+        chat_history: Optional[List[ChatMessage]] = None,
+        summarizer_llm: Optional[LLM] = None,
+        chat_store: Optional[BaseChatStore] = None,
+        chat_store_key: str = DEFAULT_CHAT_STORE_KEY,
+        token_limit_full_text: Optional[int] = None,
+        tokenizer_fn: Optional[Callable[[str], List]] = None,
+        summarize_prompt: Optional[str] = None,
+        count_initial_tokens: bool = False,
+    ) -> "ChatSummaryMemoryBuffer":
+        """Create a chat memory buffer from an LLM
+        and an initial list of chat history messages.
+        """
+        if summarizer_llm is not None:
+            context_window = summarizer_llm.metadata.context_window
+            token_limit_full_text = token_limit_full_text or int(
+                context_window * DEFAULT_TOKEN_LIMIT_FULL_TEXT_RATIO
+            )
+        elif token_limit_full_text is None:
+            token_limit_full_text = DEFAULT_TOKEN_LIMIT_FULL_TEXT
+
+        chat_store = chat_store or SimpleChatStore()
+        chat_history = chat_history or []
+        chat_store.set_messages(chat_store_key, chat_history)
+
+        summarize_prompt = summarize_prompt or SUMMARIZE_PROMPT
+        return cls(
+            summarizer_llm=summarizer_llm,
+            token_limit_full_text=token_limit_full_text,
+            # TODO: Check if we can get the tokenizer from the llm
+            tokenizer_fn=tokenizer_fn or get_tokenizer(),
+            summarize_prompt=summarize_prompt,
+            chat_store=chat_store,
+            chat_store_key=chat_store_key,
+            count_initial_tokens=count_initial_tokens,
+        )
+
+    @classmethod
+    def class_name(cls) -> str:
+        """Get class name."""
+        return "ChatSummaryMemoryBuffer"
+
+    def to_string(self) -> str:
+        """Convert memory to string."""
+        return self.json()
+
+    def to_dict(self, **kwargs: Any) -> dict:
+        """Convert memory to dict."""
+        return self.dict()
+
+    @classmethod
+    def from_string(cls, json_str: str) -> "ChatMemoryBuffer":
+        raise NotImplementedError("This is not yet supported.")
+
+    @classmethod
+    def from_dict(
+        cls, data: Dict[str, Any], **kwargs: Any
+    ) -> "ChatSummaryMemoryBuffer":
+        raise NotImplementedError("This is not yet supported.")
path: llama-index-core/llama_index/core/memory/chat_summary_memory_buffer.py
text: @user1:
from_dict() / from_string() could be implemented by just passing in the LLM when calling it?

@author:
I'm not sure if I understood you correctly, but I added a basic implementation. If this is not what you are looking for, could you make the code change?