Repository: Azure-Samples/azure-search-openai-demo
Pull request #728 (2023-10-04 04:11:47)

Urllib3 update for security issue
## Purpose

Updating urllib3 since dependabot couldnt do it.

## Does this introduce a breaking change?
<!-- Mark one with an "x". -->
```
[ ] Yes
[X] No
```

## Pull Request Type
What kind of change does this Pull Request introduce?

<!-- Please check the one that applies to this PR using "x". -->
```
[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[X] Other... Please describe:
```

## How to Test
*  pytest
* scripts/prepdocs.sh


Patch:
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
index 71b2d3a..bf063db 100644
--- a/.devcontainer/devcontainer.json
+++ b/.devcontainer/devcontainer.json
@@ -1,6 +1,6 @@
 {
     "name": "Azure Search OpenAI Demo",
-    "image": "mcr.microsoft.com/devcontainers/python:3.10",
+    "image": "mcr.microsoft.com/devcontainers/python:3.11",
     "features": {
         "ghcr.io/devcontainers/features/node:1": {
             "version": "16",
diff --git a/.github/dependabot.yaml b/.github/dependabot.yaml
index db648e4..b8645e7 100644
--- a/.github/dependabot.yaml
+++ b/.github/dependabot.yaml
@@ -6,19 +6,36 @@ updates:
     directory: "/"
     schedule:
       interval: "weekly"
+    groups:
+      github-actions:
+        patterns:
+          - "*"
 
   # Maintain dependencies for npm
   - package-ecosystem: "npm"
     directory: "/app/frontend"
     schedule:
       interval: "weekly"
+    groups:
+      node-packages:
+        patterns:
+          - "*"
 
   # Maintain dependencies for pip
   - package-ecosystem: "pip"
     directory: "/app/backend"
     schedule:
       interval: "weekly"
+    groups:
+      python-requirements:
+        patterns:
+          - "*"
+
   - package-ecosystem: "pip"
     directory: "/scripts"
     schedule:
       interval: "weekly"
+    groups:
+      python-requirements:
+        patterns:
+          - "*"
diff --git a/README.md b/README.md
index 2563e6e..3c62c64 100644
--- a/README.md
+++ b/README.md
@@ -52,7 +52,9 @@ The repo includes sample data so it's ready to try end to end. In this sample ap
 
 * **Azure account**. If you're new to Azure, [get an Azure account for free](https://azure.microsoft.com/free/cognitive-search/) and you'll get some free Azure credits to get started.
 * **Azure subscription with access enabled for the Azure OpenAI service**. You can request access with [this form](https://aka.ms/oaiapply). If your access request to Azure OpenAI service doesn't match the [acceptance criteria](https://learn.microsoft.com/legal/cognitive-services/openai/limited-access?context=%2Fazure%2Fcognitive-services%2Fopenai%2Fcontext%2Fcontext), you can use [OpenAI public API](https://platform.openai.com/docs/api-reference/introduction) instead. Learn [how to switch to an OpenAI instance](#switching-from-an-azure-openai-endpoint-to-an-openai-instance).
-* **Azure account permissions**: Your Azure account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [Role Based Access Control Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview), [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator), or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner). If you don't have subscription-level permissions, you must be granted [RBAC](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview) for an existing resource group and [deploy to that existing group](#existing-resource-group).
+* **Azure account permissions**:
+  * Your Azure account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [Role Based Access Control Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview), [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator), or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner). If you don't have subscription-level permissions, you must be granted [RBAC](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview) for an existing resource group and [deploy to that existing group](#existing-resource-group).
+  * Your Azure account also needs `Microsoft.Resources/deployments/write` permissions on the subscription level.
 
 ## Azure deployment
 
@@ -96,7 +98,7 @@ A related option is VS Code Remote Containers, which will open the project in yo
 First install the required tools:
 
 * [Azure Developer CLI](https://aka.ms/azure-dev/install)
-* [Python 3.9+](https://www.python.org/downloads/)
+* [Python 3.9, 3.10, or 3.11](https://www.python.org/downloads/)
   * **Important**: Python and the pip package manager must be in the path in Windows for the setup scripts to work.
   * **Important**: Ensure you can run `python --version` from console. On Ubuntu, you might need to run `sudo apt install python-is-python3` to link `python` to `python3`.
 * [Node.js 14+](https://nodejs.org/en/download/)
@@ -136,11 +138,29 @@ If you already have existing Azure resources, you can re-use those by setting `a
 
 #### Existing OpenAI resource
 
+##### Azure OpenAI:
+
 1. Run `azd env set AZURE_OPENAI_SERVICE {Name of existing OpenAI service}`
 1. Run `azd env set AZURE_OPENAI_RESOURCE_GROUP {Name of existing resource group that OpenAI service is provisioned to}`
 1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing ChatGPT deployment}`. Only needed if your ChatGPT deployment is not the default 'chat'.
 1. Run `azd env set AZURE_OPENAI_EMB_DEPLOYMENT {Name of existing GPT embedding deployment}`. Only needed if your embeddings deployment is not the default 'embedding'.
 
+When you run `azd up` after and are prompted to select a value for `openAiResourceGroupLocation`, make sure to select the same location as the existing OpenAI resource group.
+
+##### Openai.com OpenAI:
+
+1. Run `azd env set OPENAI_HOST openai`
+2. Run `azd env set OPENAI_ORGANIZATION {Your OpenAI organization}`
+3. Run `azd env set OPENAI_API_KEY {Your OpenAI API key}`
+4. Run `azd up`
+
+You can retrieve your OpenAI key by checking [your user page](https://platform.openai.com/account/api-keys) and your organization by navigating to [your organization page](https://platform.openai.com/account/org-settings).
+Learn more about creating an OpenAI free trial at [this link](https://openai.com/pricing).
+Do *not* check your key into source control.
+
+When you run `azd up` after and are prompted to select a value for `openAiResourceGroupLocation`, you can select any location as it will not be used.
+
+
 #### Existing Azure Cognitive Search resource
 
 1. Run `azd env set AZURE_SEARCH_SERVICE {Name of existing Azure Cognitive Search service}`
@@ -148,6 +168,8 @@ If you already have existing Azure resources, you can re-use those by setting `a
 1. If that resource group is in a different location than the one you'll pick for the `azd up` step,
   then run `azd env set AZURE_SEARCH_SERVICE_LOCATION {Location of existing service}`
 1. If the search service's SKU is not standard, then run `azd env set AZURE_SEARCH_SERVICE_SKU {Name of SKU}`. The free tier won't work as it doesn't support managed identity. ([See other possible values](https://learn.microsoft.com/azure/templates/microsoft.search/searchservices?pivots=deployment-language-bicep#sku))
+1. If you have an existing index that is set up with all the expected fields, then run `azd env set AZURE_SEARCH_INDEX {Name of existing index}`. Otherwise, the `azd up` command will create a new index.
+
 
 #### Other existing Azure resources
 
@@ -184,20 +206,6 @@ either you or they can follow these steps:
 
 ## Enabling optional features
 
-#### Using a non-Azure OpenAI instance
-
-To use an existing non-Azure OpenAI account, follow these steps before running `azd up`
-
-1. Run `azd env set OPENAI_HOST openai`
-2. Run `azd env set OPENAI_ORGANIZATION {Your OpenAI organization}`
-3. Run `azd env set OPENAI_API_KEY {Your OpenAI API key}`
-4. Run `azd up`
-
-You can retrieve your OpenAI key by checking [your user page](https://platform.openai.com/account/api-keys) and your organization by navigating to [your organization page](https://platform.openai.com/account/org-settings).
-Learn more about creating an OpenAI free trial at [this link](https://openai.com/pricing).
-Do *not* check your key into source control.
-
-
 ### Enabling Application Insights
 
 To enable Application Insights and the tracing of each request, along with the logging of errors, set the `AZURE_USE_APPLICATION_INSIGHTS` variable to true before running `azd up`
@@ -376,7 +384,6 @@ The ask tab uses the approach programmed in [retrievethenread.py](https://github
 - It queries Azure Cognitive Search for search results for the user question (optionally using the vector embeddings for that question).
 - It then combines the search results and user question, and asks ChatGPT API to answer the question based on the sources.
 
-There are also two other /ask approaches with a slightly different approach, but they aren't currently working due to [langchain compatibility issues](https://github.com/Azure-Samples/azure-search-openai-demo/issues/541).
 </details>
 
 <details><a id="azd-up-explanation"></a>
diff --git a/app/backend/app.py b/app/backend/app.py
index 1153bbf..fd50f37 100644
--- a/app/backend/app.py
+++ b/app/backend/app.py
@@ -29,15 +29,13 @@ from quart import (
 from quart_cors import cors
 
 from approaches.chatreadretrieveread import ChatReadRetrieveReadApproach
-from approaches.readdecomposeask import ReadDecomposeAsk
-from approaches.readretrieveread import ReadRetrieveReadApproach
 from approaches.retrievethenread import RetrieveThenReadApproach
 from core.authentication import AuthenticationHelper
 
 CONFIG_OPENAI_TOKEN = "openai_token"
 CONFIG_CREDENTIAL = "azure_credential"
-CONFIG_ASK_APPROACHES = "ask_approaches"
-CONFIG_CHAT_APPROACHES = "chat_approaches"
+CONFIG_ASK_APPROACH = "ask_approach"
+CONFIG_CHAT_APPROACH = "chat_approach"
 CONFIG_BLOB_CONTAINER_CLIENT = "blob_container_client"
 CONFIG_AUTH_CLIENT = "auth_client"
 CONFIG_SEARCH_CLIENT = "search_client"
@@ -92,11 +90,8 @@ async def ask():
     request_json = await request.get_json()
     auth_helper = current_app.config[CONFIG_AUTH_CLIENT]
     auth_claims = await auth_helper.get_auth_claims_if_enabled(request.headers)
-    approach = request_json["approach"]
     try:
-        impl = current_app.config[CONFIG_ASK_APPROACHES].get(approach)
-        if not impl:
-            return jsonify({"error": "unknown approach"}), 400
+        impl = current_app.config[CONFIG_ASK_APPROACH]
         # Workaround for: https://github.com/openai/openai-python/issues/371
         async with aiohttp.ClientSession() as s:
             openai.aiosession.set(s)
@@ -114,11 +109,8 @@ async def chat():
     request_json = await request.get_json()
     auth_helper = current_app.config[CONFIG_AUTH_CLIENT]
     auth_claims = await auth_helper.get_auth_claims_if_enabled(request.headers)
-    approach = request_json["approach"]
     try:
-        impl = current_app.config[CONFIG_CHAT_APPROACHES].get(approach)
-        if not impl:
-            return jsonify({"error": "unknown approach"}), 400
+        impl = current_app.config[CONFIG_CHAT_APPROACH]
         # Workaround for: https://github.com/openai/openai-python/issues/371
         async with aiohttp.ClientSession() as s:
             openai.aiosession.set(s)
@@ -143,11 +135,8 @@ async def chat_stream():
     request_json = await request.get_json()
     auth_helper = current_app.config[CONFIG_AUTH_CLIENT]
     auth_claims = await auth_helper.get_auth_claims_if_enabled(request.headers)
-    approach = request_json["approach"]
     try:
-        impl = current_app.config[CONFIG_CHAT_APPROACHES].get(approach)
-        if not impl:
-            return jsonify({"error": "unknown approach"}), 400
+        impl = current_app.config[CONFIG_CHAT_APPROACH]
         response_generator = impl.run_with_streaming(
             request_json["history"], request_json.get("overrides", {}), auth_claims
         )
@@ -255,50 +244,27 @@ async def setup_clients():
 
     # Various approaches to integrate GPT and external knowledge, most applications will use a single one of these patterns
     # or some derivative, here we include several for exploration purposes
-    current_app.config[CONFIG_ASK_APPROACHES] = {
-        "rtr": RetrieveThenReadApproach(
-            search_client,
-            OPENAI_HOST,
-            AZURE_OPENAI_CHATGPT_DEPLOYMENT,
-            OPENAI_CHATGPT_MODEL,
-            AZURE_OPENAI_EMB_DEPLOYMENT,
-            OPENAI_EMB_MODEL,
-            KB_FIELDS_SOURCEPAGE,
-            KB_FIELDS_CONTENT,
-        ),
-        "rrr": ReadRetrieveReadApproach(
-            search_client,
-            OPENAI_HOST,
-            AZURE_OPENAI_CHATGPT_DEPLOYMENT,
-            OPENAI_CHATGPT_MODEL,
-            AZURE_OPENAI_EMB_DEPLOYMENT,
-            OPENAI_EMB_MODEL,
-            KB_FIELDS_SOURCEPAGE,
-            KB_FIELDS_CONTENT,
-        ),
-        "rda": ReadDecomposeAsk(
-            search_client,
-            OPENAI_HOST,
-            AZURE_OPENAI_CHATGPT_DEPLOYMENT,
-            OPENAI_CHATGPT_MODEL,
-            AZURE_OPENAI_EMB_DEPLOYMENT,
-            OPENAI_EMB_MODEL,
-            KB_FIELDS_SOURCEPAGE,
-            KB_FIELDS_CONTENT,
-        ),
-    }
-    current_app.config[CONFIG_CHAT_APPROACHES] = {
-        "rrr": ChatReadRetrieveReadApproach(
-            search_client,
-            OPENAI_HOST,
-            AZURE_OPENAI_CHATGPT_DEPLOYMENT,
-            OPENAI_CHATGPT_MODEL,
-            AZURE_OPENAI_EMB_DEPLOYMENT,
-            OPENAI_EMB_MODEL,
-            KB_FIELDS_SOURCEPAGE,
-            KB_FIELDS_CONTENT,
-        )
-    }
+    current_app.config[CONFIG_ASK_APPROACH] = RetrieveThenReadApproach(
+        search_client,
+        OPENAI_HOST,
+        AZURE_OPENAI_CHATGPT_DEPLOYMENT,
+        OPENAI_CHATGPT_MODEL,
+        AZURE_OPENAI_EMB_DEPLOYMENT,
+        OPENAI_EMB_MODEL,
+        KB_FIELDS_SOURCEPAGE,
+        KB_FIELDS_CONTENT,
+    )
+
+    current_app.config[CONFIG_CHAT_APPROACH] = ChatReadRetrieveReadApproach(
+        search_client,
+        OPENAI_HOST,
+        AZURE_OPENAI_CHATGPT_DEPLOYMENT,
+        OPENAI_CHATGPT_MODEL,
+        AZURE_OPENAI_EMB_DEPLOYMENT,
+        OPENAI_EMB_MODEL,
+        KB_FIELDS_SOURCEPAGE,
+        KB_FIELDS_CONTENT,
+    )
 
 
 def create_app():
@@ -307,7 +273,7 @@ def create_app():
         AioHttpClientInstrumentor().instrument()
     app = Quart(__name__)
     app.register_blueprint(bp)
-    app.asgi_app = OpenTelemetryMiddleware(app.asgi_app)
+    app.asgi_app = OpenTelemetryMiddleware(app.asgi_app)  # type: ignore[method-assign]
 
     # Level should be one of https://docs.python.org/3/library/logging.html#logging-levels
     default_level = "INFO"  # In development, log more verbosely
diff --git a/app/backend/approaches/approach.py b/app/backend/approaches/approach.py
index 894f3f1..c160e52 100644
--- a/app/backend/approaches/approach.py
+++ b/app/backend/approaches/approach.py
@@ -1,11 +1,11 @@
-from abc import ABC, abstractmethod
-from typing import Any
+from abc import ABC
+from typing import Any, Optional
 
 from core.authentication import AuthenticationHelper
 
 
 class Approach(ABC):
-    def build_filter(self, overrides: dict[str, Any], auth_claims: dict[str, Any]) -> str:
+    def build_filter(self, overrides: dict[str, Any], auth_claims: dict[str, Any]) -> Optional[str]:
         exclude_category = overrides.get("exclude_category") or None
         security_filter = AuthenticationHelper.build_security_filters(overrides, auth_claims)
         filters = []
@@ -14,9 +14,3 @@ class Approach(ABC):
         if security_filter:
             filters.append(security_filter)
         return None if len(filters) == 0 else " and ".join(filters)
-
-
-class AskApproach(Approach):
-    @abstractmethod
-    async def run(self, q: str, overrides: dict[str, Any], auth_claims: dict[str, Any]) -> dict[str, Any]:
-        ...
diff --git a/app/backend/approaches/chatreadretrieveread.py b/app/backend/approaches/chatreadretrieveread.py
index d4037e1..f525475 100644
--- a/app/backend/approaches/chatreadretrieveread.py
+++ b/app/backend/approaches/chatreadretrieveread.py
@@ -1,5 +1,5 @@
 import json
-from typing import Any, AsyncGenerator
+from typing import Any, AsyncGenerator, Optional
 
 import openai
 from azure.search.documents.aio import SearchClient
@@ -56,9 +56,9 @@ If you cannot generate a search query, return just the number 0.
         self,
         search_client: SearchClient,
         openai_host: str,
-        chatgpt_deployment: str,
+        chatgpt_deployment: Optional[str],  # Not needed for non-Azure OpenAI
         chatgpt_model: str,
-        embedding_deployment: str,
+        embedding_deployment: Optional[str],  # Not needed for non-Azure OpenAI or for retrieval_mode="text"
         embedding_model: str,
         sourcepage_field: str,
         content_field: str,
@@ -228,9 +228,8 @@ If you cannot generate a search query, return just the number 0.
             history, overrides, auth_claims, should_stream=False
         )
         chat_resp = await chat_coroutine
-        chat_content = chat_resp.choices[0].message.content
-        extra_info["answer"] = chat_content
-        return extra_info
+        chat_resp.choices[0]["extra_args"] = extra_info
+        return chat_resp
 
     async def run_with_streaming(
         self, history: list[dict[str, str]], overrides: dict[str, Any], auth_claims: dict[str, Any]
@@ -238,7 +237,13 @@ If you cannot generate a search query, return just the number 0.
         extra_info, chat_coroutine = await self.run_until_final_call(
             history, overrides, auth_claims, should_stream=True
         )
-        yield extra_info
+        yield {
+            "choices": [
+                {"delta": {"role": self.ASSISTANT}, "extra_args": extra_info, "finish_reason": None, "index": 0}
+            ],
+            "object": "chat.completion.chunk",
+        }
+
         async for event in await chat_coroutine:
             # "2023-07-01-preview" API version has a bug where first response has empty choices
             if event["choices"]:
@@ -249,7 +254,7 @@ If you cannot generate a search query, return just the number 0.
         system_prompt: str,
         model_id: str,
         history: list[dict[str, str]],
-        user_conv: str,
+        user_content: str,
         few_shots=[],
         max_tokens: int = 4096,
     ) -> list:
@@ -259,23 +264,21 @@ If you cannot generate a search query, return just the number 0.
         for shot in few_shots:
             message_builder.append_message(shot.get("role"), shot.get("content"))
 
-        user_content = user_conv
         append_index = len(few_shots) + 1
 
         message_builder.append_message(self.USER, user_content, index=append_index)
 
         for h in reversed(history[:-1]):
+            if message_builder.token_length > max_tokens:
+                break
             if bot_msg := h.get("bot"):
                 message_builder.append_message(self.ASSISTANT, bot_msg, index=append_index)
             if user_msg := h.get("user"):
                 message_builder.append_message(self.USER, user_msg, index=append_index)
-            if message_builder.token_length > max_tokens:
-                break
 
-        messages = message_builder.messages
-        return messages
+        return message_builder.messages
 
-    def get_search_query(self, chat_completion: dict[str, any], user_query: str):
+    def get_search_query(self, chat_completion: dict[str, Any], user_query: str):
         response_message = chat_completion["choices"][0]["message"]
         if function_call := response_message.get("function_call"):
             if function_call["name"] == "search_sources":
diff --git a/app/backend/approaches/readdecomposeask.py b/app/backend/approaches/readdecomposeask.py
deleted file mode 100644
index 258d00c..0000000
--- a/app/backend/approaches/readdecomposeask.py
+++ /dev/null
@@ -1,292 +0,0 @@
-import re
-from typing import Any, Optional, Sequence
-
-import openai
-from azure.search.documents.aio import SearchClient
-from azure.search.documents.models import QueryType
-from langchain.agents import AgentExecutor, Tool
-from langchain.agents.react.base import ReActDocstoreAgent
-from langchain.callbacks.manager import CallbackManager
-from langchain.llms.openai import AzureOpenAI, OpenAI
-from langchain.prompts import BasePromptTemplate, PromptTemplate
-from langchain.tools.base import BaseTool
-
-from approaches.approach import AskApproach
-from langchainadapters import HtmlCallbackHandler
-from text import nonewlines
-
-
-class ReadDecomposeAsk(AskApproach):
-    def __init__(
-        self,
-        search_client: SearchClient,
-        openai_host: str,
-        openai_deployment: str,
-        openai_model: str,
-        embedding_deployment: str,
-        embedding_model: str,
-        sourcepage_field: str,
-        content_field: str,
-    ):
-        self.search_client = search_client
-        self.openai_deployment = openai_deployment
-        self.openai_model = openai_model
-        self.embedding_deployment = embedding_deployment
-        self.embedding_model = embedding_model
-        self.sourcepage_field = sourcepage_field
-        self.content_field = content_field
-        self.openai_host = openai_host
-
-    async def search(
-        self, query_text: str, overrides: dict[str, Any], auth_claims: dict[str, Any]
-    ) -> tuple[list[str], str]:
-        has_text = overrides.get("retrieval_mode") in ["text", "hybrid", None]
-        has_vector = overrides.get("retrieval_mode") in ["vectors", "hybrid", None]
-        use_semantic_captions = True if overrides.get("semantic_captions") and has_text else False
-        top = overrides.get("top", 3)
-        filter = self.build_filter(overrides, auth_claims)
-
-        # If retrieval mode includes vectors, compute an embedding for the query
-        if has_vector:
-            embedding_args = {"deployment_id": self.embedding_deployment} if self.openai_host == "azure" else {}
-            embedding = await openai.Embedding.acreate(**embedding_args, model=self.embedding_model, input=query_text)
-            query_vector = embedding["data"][0]["embedding"]
-        else:
-            query_vector = None
-
-        # Only keep the text query if the retrieval mode uses text, otherwise drop it
-        if not has_text:
-            query_text = ""
-
-        if overrides.get("semantic_ranker") and has_text:
-            r = await self.search_client.search(
-                query_text,
-                filter=filter,
-                query_type=QueryType.SEMANTIC,
-                query_language="en-us",
-                query_speller="lexicon",
-                semantic_configuration_name="default",
-                top=top,
-                query_caption="extractive|highlight-false" if use_semantic_captions else None,
-                vector=query_vector,
-                top_k=50 if query_vector else None,
-                vector_fields="embedding" if query_vector else None,
-            )
-        else:
-            r = await self.search_client.search(
-                query_text,
-                filter=filter,
-                top=top,
-                vector=query_vector,
-                top_k=50 if query_vector else None,
-                vector_fields="embedding" if query_vector else None,
-            )
-        if use_semantic_captions:
-            self.results = [
-                doc[self.sourcepage_field] + ":" + nonewlines(" . ".join([c.text for c in doc["@search.captions"]]))
-                async for doc in r
-            ]
-        else:
-            results = [doc[self.sourcepage_field] + ":" + nonewlines(doc[self.content_field][:500]) async for doc in r]
-        return results, "\n".join(results)
-
-    async def lookup(self, q: str) -> Optional[str]:
-        r = await self.search_client.search(
-            q,
-            top=1,
-            include_total_count=True,
-            query_type=QueryType.SEMANTIC,
-            query_language="en-us",
-            query_speller="lexicon",
-            semantic_configuration_name="default",
-            query_answer="extractive|count-1",
-            query_caption="extractive|highlight-false",
-        )
-
-        answers = await r.get_answers()
-        if answers and len(answers) > 0:
-            return answers[0].text
-        if await r.get_count() > 0:
-            return "\n".join([d["content"] async for d in r])
-        return None
-
-    async def run(self, q: str, overrides: dict[str, Any], auth_claims: dict[str, Any]) -> dict[str, Any]:
-        search_results = None
-
-        async def search_and_store(q: str) -> Any:
-            nonlocal search_results
-            search_results, content = await self.search(q, overrides, auth_claims)
-            return content
-
-        # Use to capture thought process during iterations
-        cb_handler = HtmlCallbackHandler()
-        cb_manager = CallbackManager(handlers=[cb_handler])
-
-        if self.openai_host == "azure":
-            llm = AzureOpenAI(
-                deployment_name=self.openai_deployment,
-                temperature=overrides.get("temperature", 0.3),
-                openai_api_key=openai.api_key,
-            )
-        else:
-            llm = OpenAI(
-                model_name=self.openai_model,
-                temperature=overrides.get("temperature", 0.3),
-                openai_api_key=openai.api_key,
-            )
-        tools = [
-            Tool(
-                name="Search",
-                func=lambda _: "Not implemented",
-                coroutine=search_and_store,
-                description="useful for when you need to ask with search",
-                callbacks=cb_manager,
-            ),
-            Tool(
-                name="Lookup",
-                func=lambda _: "Not implemented",
-                coroutine=self.lookup,
-                description="useful for when you need to ask with lookup",
-                callbacks=cb_manager,
-            ),
-        ]
-
-        prompt_prefix = overrides.get("prompt_template")
-        prompt = PromptTemplate.from_examples(
-            EXAMPLES,
-            SUFFIX,
-            ["input", "agent_scratchpad"],
-            prompt_prefix + "\n\n" + PREFIX if prompt_prefix else PREFIX,
-        )
-
-        class ReAct(ReActDocstoreAgent):
-            @classmethod
-            def create_prompt(cls, tools: Sequence[BaseTool]) -> BasePromptTemplate:
-                return prompt
-
-        agent = ReAct.from_llm_and_tools(llm, tools)
-        chain = AgentExecutor.from_agent_and_tools(agent, tools, verbose=True, callback_manager=cb_manager)
-        result = await chain.arun(q)
-
-        # Replace substrings of the form <file.ext> with [file.ext] so that the frontend can render them as links, match them with a regex to avoid
-        # generalizing too much and disrupt HTML snippets if present
-        result = re.sub(r"<([a-zA-Z0-9_ \-\.]+)>", r"[\1]", result)
-
-        return {"data_points": search_results or [], "answer": result, "thoughts": cb_handler.get_and_reset_log()}
-
-
-# Modified version of langchain's ReAct prompt that includes instructions and examples for how to cite information sources
-EXAMPLES = [
-    """Question: What is the elevation range for the area that the eastern sector of the
-Colorado orogeny extends into?
-Thought: I need to search Colorado orogeny, find the area that the eastern sector
-of the Colorado orogeny extends into, then find the elevation range of the
-area.
-Action: Search[Colorado orogeny]
-Observation: <info1.pdf> The Colorado orogeny was an episode of mountain building (an orogeny) in
-Colorado and surrounding areas.
-Thought: It does not mention the eastern sector. So I need to look up eastern
-sector.
-Action: Lookup[eastern sector]
-Observation: <info2.txt> (Result 1 / 1) The eastern sector extends into the High Plains and is called
-the Central Plains orogeny.
-Thought: The eastern sector of Colorado orogeny extends into the High Plains. So I
-need to search High Plains and find its elevation range.
-Action: Search[High Plains]
-Observation: <some_file.pdf> High Plains refers to one of two distinct land regions
-Thought: I need to instead search High Plains (United States).
-Action: Search[High Plains (United States)]
-Observation: <filea.pdf> The High Plains are a subregion of the Great Plains. <another-ref.docx> From east to west, the
-High Plains rise in elevation from around 1,800 to 7,000 ft (550 to 2,130
-m).
-Thought: High Plains rise in elevation from around 1,800 to 7,000 ft, so the answer
-is 1,800 to 7,000 ft.
-Action: Finish[1,800 to 7,000 ft <filea.pdf>]""",
-    """Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons"
-character Milhouse, who Matt Groening named after who?
-Thought: The question simplifies to "The Simpsons" character Milhouse is named after
-who. I only need to search Milhouse and find who it is named after.
-Action: Search[Milhouse]
-Observation: <info7.pdf> Milhouse Mussolini Van Houten is a recurring character in the Fox animated
-television series The Simpsons voiced by Pamela Hayden and created by Matt
-Groening.
-Thought: The paragraph does not tell who Milhouse is named after, maybe I can look up
-"named after".
-Action: Lookup[named after]
-Observation: <historyref2.txt> (Result 1 / 1) Milhouse was named after U.S. president Richard Nixon, whose
-middle name was Milhous.
-Thought: Milhouse was named after U.S. president Richard Nixon, so the answer is
-Richard Nixon.
-Action: Finish[Richard Nixon <historyref2.txt>]""",
-    """Question: Which documentary is about Finnish rock groups, Adam Clayton Powell or The
-Saimaa Gesture?
-Thought: I need to search Adam Clayton Powell and The Saimaa Gesture, and find which
-documentary is about Finnish rock groups.
-Action: Search[Adam Clayton Powell]
-Observation: <info9.pdf> Could not find [Adam Clayton Powell]. Similar: ['Adam Clayton Powell
-III', 'Seventh Avenue (Manhattan)', 'Adam Clayton Powell Jr. State Office
-Building', 'Isabel Washington Powell', 'Adam Powell', vAdam Clayton Powell
-(film)', 'Giancarlo Esposito'].
-Thought: To find the documentary, I can search Adam Clayton Powell (film).
-Action: Search[Adam Clayton Powell (film)]
-Observation: <data123.txt> Adam Clayton Powell is a 1989 American documentary film directed by
-Richard Kilberg. The film is about the rise and fall of influential
-African-American politician Adam Clayton Powell Jr.[3][4] It was later aired
-as part of the PBS series The American Experience.
-Thought: Adam Clayton Powell (film) is a documentary about an African-American
-politician, not Finnish rock groups. So the documentary about Finnish rock
-groups must instead be The Saimaa Gesture.
-Action: Finish[The Saimaa Gesture <data123.txt>]""",
-    """Question: What profession does Nicholas Ray and Elia Kazan have in common?
-Thought: I need to search Nicholas Ray and Elia Kazan, find their professions, then
-find the profession they have in common.
-Action: Search[Nicholas Ray]
-Observation: <files-987.png> Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 - June 16,
-1979) was an American film director, screenwriter, and actor best known for
-the 1955 film Rebel Without a Cause.
-Thought: Professions of Nicholas Ray are director, screenwriter, and actor. I need
-to search Elia Kazan next and find his professions.
-Action: Search[Elia Kazan]
-Observation: <files-654.txt> Elia Kazan was an American film and theatre director, producer, screenwriter
-and actor.
-Thought: Professions of Elia Kazan are director, producer, screenwriter, and actor.
-So profession Nicholas Ray and Elia Kazan have in common is director,
-screenwriter, and actor.
-Action: Finish[director, screenwriter, actor <files-987.png><files-654.txt>]""",
-    """Question: Which magazine was started first Arthur's Magazine or First for Women?
-Thought: I need to search Arthur's Magazine and First for Women, and find which was
-started first.
-Action: Search[Arthur's Magazine]
-Observation: <magazines-1850.pdf> Arthur's Magazine (1844-1846) was an American literary periodical published
-in Philadelphia in the 19th century.
-Thought: Arthur's Magazine was started in 1844. I need to search First for Women
-next.
-Action: Search[First for Women]
-Observation: <magazines-1900.pdf> First for Women is a woman's magazine published by Bauer Media Group in the
-USA.[1] The magazine was started in 1989.
-Thought: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First
-for Women), so Arthur's Magazine was started first.
-Action: Finish[Arthur's Magazine <magazines-1850.pdf><magazines-1900.pdf>]""",
-    """Question: Were Pavel Urysohn and Leonid Levin known for the same type of work?
-Thought: I need to search Pavel Urysohn and Leonid Levin, find their types of work,
-then find if they are the same.
-Action: Search[Pavel Urysohn]
-Observation: <info4444.pdf> Pavel Samuilovich Urysohn (February 3, 1898 - August 17, 1924) was a Soviet
-mathematician who is best known for his contributions in dimension theory.
-Thought: Pavel Urysohn is a mathematician. I need to search Leonid Levin next and
-find its type of work.
-Action: Search[Leonid Levin]
-Observation: <datapoints_aaa.txt> Leonid Anatolievich Levin is a Soviet-American mathematician and computer
-scientist.
-Thought: Leonid Levin is a mathematician and computer scientist. So Pavel Urysohn
-and Leonid Levin have the same type of work.
-Action: Finish[yes <info4444.pdf><datapoints_aaa.txt>]""",
-]
-SUFFIX = """\nQuestion: {input}
-{agent_scratchpad}"""
-PREFIX = (
-    "Answer questions as shown in the following examples, by splitting the question into individual search or lookup actions to find facts until you can answer the question. "
-    "Observations are prefixed by their source name in angled brackets, source names MUST be included with the actions in the answers."
-    "All questions must be answered from the results from search or look up actions, only facts resulting from those can be used in an answer. "
-)
-"Answer questions as truthfully as possible, and ONLY answer the questions using the information from observations, do not speculate or your own knowledge."
diff --git a/app/backend/approaches/readretrieveread.py b/app/backend/approaches/readretrieveread.py
deleted file mode 100644
index 0b7eaf8..0000000
--- a/app/backend/approaches/readretrieveread.py
+++ /dev/null
@@ -1,193 +0,0 @@
-from typing import Any
-
-import openai
-from azure.search.documents.aio import SearchClient
-from azure.search.documents.models import QueryType
-from langchain.agents import AgentExecutor, Tool, ZeroShotAgent
-from langchain.callbacks.manager import CallbackManager, Callbacks
-from langchain.chains import LLMChain
-from langchain.llms.openai import AzureOpenAI, OpenAI
-
-from approaches.approach import AskApproach
-from langchainadapters import HtmlCallbackHandler
-from lookuptool import CsvLookupTool
-from text import nonewlines
-
-
-class ReadRetrieveReadApproach(AskApproach):
-    """
-    Attempt to answer questions by iteratively evaluating the question to see what information is missing, and once all information
-    is present then formulate an answer. Each iteration consists of two parts:
-     1. use GPT to see if we need more information
-     2. if more data is needed, use the requested "tool" to retrieve it.
-    The last call to GPT answers the actual question.
-    This is inspired by the MKRL paper[1] and applied here using the implementation in Langchain.
-
-    [1] E. Karpas, et al. arXiv:2205.00445
-    """
-
-    template_prefix = (
-        "You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. "
-        "Answer the question using only the data provided in the information sources below. "
-        "For tabular information return it as an html table. Do not return markdown format. "
-        "Each source has a name followed by colon and the actual data, quote the source name for each piece of data you use in the response. "
-        'For example, if the question is "What color is the sky?" and one of the information sources says "info123: the sky is blue whenever it\'s not cloudy", then answer with "The sky is blue [info123]" '
-        'It\'s important to strictly follow the format where the name of the source is in square brackets at the end of the sentence, and only up to the prefix before the colon (":"). '
-        'If there are multiple sources, cite each one in their own square brackets. For example, use "[info343][ref-76]" and not "[info343,ref-76]". '
-        "Never quote tool names as sources."
-        "If you cannot answer using the sources below, say that you don't know. "
-        "\n\nYou can access to the following tools:"
-    )
-
-    template_suffix = """
-Begin!
-
-Question: {input}
-
-Thought: {agent_scratchpad}"""
-
-    CognitiveSearchToolDescription = "useful for searching the Microsoft employee benefits information such as healthcare plans, retirement plans, etc."
-
-    def __init__(
-        self,
-        search_client: SearchClient,
-        openai_host: str,
-        openai_deployment: str,
-        openai_model: str,
-        embedding_deployment: str,
-        embedding_model: str,
-        sourcepage_field: str,
-        content_field: str,
-    ):
-        self.search_client = search_client
-        self.openai_deployment = openai_deployment
-        self.openai_model = openai_model
-        self.embedding_deployment = embedding_deployment
-        self.embedding_model = embedding_model
-        self.sourcepage_field = sourcepage_field
-        self.content_field = content_field
-        self.openai_host = openai_host
-
-    async def retrieve(self, query_text: str, overrides: dict[str, Any], auth_claims: dict[str, Any]) -> Any:
-        has_text = overrides.get("retrieval_mode") in ["text", "hybrid", None]
-        has_vector = overrides.get("retrieval_mode") in ["vectors", "hybrid", None]
-        use_semantic_captions = True if overrides.get("semantic_captions") and has_text else False
-        top = overrides.get("top", 3)
-        filter = self.build_filter(overrides, auth_claims)
-
-        # If retrieval mode includes vectors, compute an embedding for the query
-        if has_vector:
-            embedding_args = {"deployment_id": self.embedding_deployment} if self.openai_host == "azure" else {}
-            embedding = await openai.Embedding.acreate(**embedding_args, model=self.embedding_model, input=query_text)
-            query_vector = embedding["data"][0]["embedding"]
-        else:
-            query_vector = None
-
-        # Only keep the text query if the retrieval mode uses text, otherwise drop it
-        if not has_text:
-            query_text = ""
-
-        # Use semantic ranker if requested and if retrieval mode is text or hybrid (vectors + text)
-        if overrides.get("semantic_ranker") and has_text:
-            r = await self.search_client.search(
-                query_text,
-                filter=filter,
-                query_type=QueryType.SEMANTIC,
-                query_language="en-us",
-                query_speller="lexicon",
-                semantic_configuration_name="default",
-                top=top,
-                query_caption="extractive|highlight-false" if use_semantic_captions else None,
-                vector=query_vector,
-                top_k=50 if query_vector else None,
-                vector_fields="embedding" if query_vector else None,
-            )
-        else:
-            r = await self.search_client.search(
-                query_text,
-                filter=filter,
-                top=top,
-                vector=query_vector,
-                top_k=50 if query_vector else None,
-                vector_fields="embedding" if query_vector else None,
-            )
-        if use_semantic_captions:
-            results = [
-                doc[self.sourcepage_field] + ":" + nonewlines(" -.- ".join([c.text for c in doc["@search.captions"]]))
-                async for doc in r
-            ]
-        else:
-            results = [doc[self.sourcepage_field] + ":" + nonewlines(doc[self.content_field][:250]) async for doc in r]
-        content = "\n".join(results)
-        return results, content
-
-    async def run(self, q: str, overrides: dict[str, Any], auth_claims: dict[str, Any]) -> dict[str, Any]:
-        retrieve_results = None
-
-        async def retrieve_and_store(q: str) -> Any:
-            nonlocal retrieve_results
-            retrieve_results, content = await self.retrieve(q, overrides, auth_claims)
-            return content
-
-        # Use to capture thought process during iterations
-        cb_handler = HtmlCallbackHandler()
-        cb_manager = CallbackManager(handlers=[cb_handler])
-
-        acs_tool = Tool(
-            name="CognitiveSearch",
-            func=lambda _: "Not implemented",
-            coroutine=retrieve_and_store,
-            description=self.CognitiveSearchToolDescription,
-            callbacks=cb_manager,
-        )
-        employee_tool = EmployeeInfoTool("Employee1", callbacks=cb_manager)
-        tools = [acs_tool, employee_tool]
-
-        prompt = ZeroShotAgent.create_prompt(
-            tools=tools,
-            prefix=overrides.get("prompt_template_prefix") or self.template_prefix,
-            suffix=overrides.get("prompt_template_suffix") or self.template_suffix,
-            input_variables=["input", "agent_scratchpad"],
-        )
-        if self.openai_type == "azure":
-            llm = AzureOpenAI(
-                deployment_name=self.openai_deployment,
-                temperature=overrides.get("temperature", 0.3),
-                openai_api_key=openai.api_key,
-            )
-        else:
-            llm = OpenAI(
-                model_name=self.openai_model,
-                temperature=overrides.get("temperature", 0.3),
-                openai_api_key=openai.api_key,
-            )
-
-        chain = LLMChain(llm=llm, prompt=prompt)
-        agent_exec = AgentExecutor.from_agent_and_tools(
-            agent=ZeroShotAgent(llm_chain=chain), tools=tools, verbose=True, callback_manager=cb_manager
-        )
-        result = await agent_exec.arun(q)
-
-        # Remove references to tool names that might be confused with a citation
-        result = result.replace("[CognitiveSearch]", "").replace("[Employee]", "")
-
-        return {"data_points": retrieve_results or [], "answer": result, "thoughts": cb_handler.get_and_reset_log()}
-
-
-class EmployeeInfoTool(CsvLookupTool):
-    employee_name: str = ""
-
-    def __init__(self, employee_name: str, callbacks: Callbacks = None):
-        super().__init__(
-            filename="data/employeeinfo.csv",
-            key_field="name",
-            name="Employee",
-            description="useful for answering questions about the employee, their benefits and other personal information",
-            callbacks=callbacks,
-        )
-        self.func = lambda _: "Not implemented"
-        self.coroutine = self.employee_info
-        self.employee_name = employee_name
-
-    async def employee_info(self, name: str) -> str:
-        return self.lookup(name)
diff --git a/app/backend/approaches/retrievethenread.py b/app/backend/approaches/retrievethenread.py
index cbd644e..689a3d4 100644
--- a/app/backend/approaches/retrievethenread.py
+++ b/app/backend/approaches/retrievethenread.py
@@ -1,15 +1,15 @@
-from typing import Any
+from typing import Any, Optional
 
 import openai
 from azure.search.documents.aio import SearchClient
 from azure.search.documents.models import QueryType
 
-from approaches.approach import AskApproach
+from approaches.approach import Approach
 from core.messagebuilder import MessageBuilder
 from text import nonewlines
 
 
-class RetrieveThenReadApproach(AskApproach):
+class RetrieveThenReadApproach(Approach):
     """
     Simple retrieve-then-read implementation, using the Cognitive Search and OpenAI APIs directly. It first retrieves
     top documents from search, then constructs a prompt with them, and then uses OpenAI to generate an completion
@@ -41,9 +41,9 @@ info4.pdf: In-network institutions include Overlake, Swedish and others in the r
         self,
         search_client: SearchClient,
         openai_host: str,
-        chatgpt_deployment: str,
+        chatgpt_deployment: Optional[str],  # Not needed for non-Azure OpenAI
         chatgpt_model: str,
-        embedding_deployment: str,
+        embedding_deployment: Optional[str],  # Not needed for non-Azure OpenAI or for retrieval_mode="text"
         embedding_model: str,
         sourcepage_field: str,
         content_field: str,
@@ -131,9 +131,10 @@ info4.pdf: In-network institutions include Overlake, Swedish and others in the r
             n=1,
         )
 
-        return {
+        extra_info = {
             "data_points": results,
-            "answer": chat_completion.choices[0].message.content,
             "thoughts": f"Question:<br>{query_text}<br><br>Prompt:<br>"
             + "\n\n".join([str(message) for message in messages]),
         }
+        chat_completion.choices[0]["extra_args"] = extra_info
+        return chat_completion
diff --git a/app/backend/core/authentication.py b/app/backend/core/authentication.py
index d31964d..ac6e201 100644
--- a/app/backend/core/authentication.py
+++ b/app/backend/core/authentication.py
@@ -4,7 +4,7 @@ import json
 import logging
 import os
 from tempfile import TemporaryDirectory
-from typing import Any
+from typing import Any, Optional
 
 import aiohttp
 from msal import ConfidentialClientApplication
@@ -28,11 +28,11 @@ class AuthenticationHelper:
     def __init__(
         self,
         use_authentication: bool,
-        server_app_id: str,
-        server_app_secret: str,
-        client_app_id: str,
-        tenant_id: str,
-        token_cache_path: str = None,
+        server_app_id: Optional[str],
+        server_app_secret: Optional[str],
+        client_app_id: Optional[str],
+        tenant_id: Optional[str],
+        token_cache_path: Optional[str] = None,
     ):
         self.use_authentication = use_authentication
         self.server_app_id = server_app_id
@@ -143,7 +143,7 @@ class AuthenticationHelper:
             return None
 
     @staticmethod
-    async def list_groups(graph_resource_access_token: str) -> [str]:
+    async def list_groups(graph_resource_access_token: dict) -> list[str]:
         headers = {"Authorization": "Bearer " + graph_resource_access_token["access_token"]}
         groups = []
         async with aiohttp.ClientSession(headers=headers) as session:
@@ -171,7 +171,7 @@ class AuthenticationHelper:
 
         return groups
 
-    async def get_auth_claims_if_enabled(self, headers: dict) -> dict[str:Any]:
+    async def get_auth_claims_if_enabled(self, headers: dict) -> dict[str, Any]:
         if not self.use_authentication:
             return {}
         try:
diff --git a/app/backend/core/messagebuilder.py b/app/backend/core/messagebuilder.py
index dc21961..9409ae7 100644
--- a/app/backend/core/messagebuilder.py
+++ b/app/backend/core/messagebuilder.py
@@ -1,3 +1,5 @@
+import unicodedata
+
 from .modelhelper import num_tokens_from_messages
 
 
@@ -14,10 +16,13 @@ class MessageBuilder:
     """
 
     def __init__(self, system_content: str, chatgpt_model: str):
-        self.messages = [{"role": "system", "content": system_content}]
+        self.messages = [{"role": "system", "content": self.normalize_content(system_content)}]
         self.model = chatgpt_model
         self.token_length = num_tokens_from_messages(self.messages[-1], self.model)
 
     def append_message(self, role: str, content: str, index: int = 1):
-        self.messages.insert(index, {"role": role, "content": content})
+        self.messages.insert(index, {"role": role, "content": self.normalize_content(content)})
         self.token_length += num_tokens_from_messages(self.messages[index], self.model)
+
+    def normalize_content(self, content: str):
+        return unicodedata.normalize("NFC", content)
diff --git a/app/backend/data/employeeinfo.csv b/app/backend/data/employeeinfo.csv
deleted file mode 100644
index b3a4c88..0000000
--- a/app/backend/data/employeeinfo.csv
+++ /dev/null
@@ -1,4 +0,0 @@
-name,title,insurance,insurancegroup
-Employee1,Program Manager,Northwind Health Plus,Family
-Employee2,Software Engineer,Northwind Health Plus,Single
-Employee3,Software Engineer,Northwind Health Standard,Family
diff --git a/app/backend/langchainadapters.py b/app/backend/langchainadapters.py
deleted file mode 100644
index 8487c37..0000000
--- a/app/backend/langchainadapters.py
+++ /dev/null
@@ -1,83 +0,0 @@
-from typing import Any, Dict, List, Optional, Union
-
-from langchain.callbacks.base import BaseCallbackHandler
-from langchain.schema import AgentAction, AgentFinish, LLMResult
-
-
-def ch(text: Union[str, object]) -> str:
-    s = text if isinstance(text, str) else str(text)
-    return s.replace("<", "&lt;").replace(">", "&gt;").replace("\r", "").replace("\n", "<br>")
-
-
-class HtmlCallbackHandler(BaseCallbackHandler):
-    html: str = ""
-
-    def get_and_reset_log(self) -> str:
-        result = self.html
-        self.html = ""
-        return result
-
-    def on_llm_start(self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) -> None:
-        """Print out the prompts."""
-        self.html += "LLM prompts:<br>" + "<br>".join(ch(prompts)) + "<br>"
-
-    def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
-        """Do nothing."""
-        pass
-
-    def on_llm_error(self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any) -> None:
-        self.html += f"<span style='color:red'>LLM error: {ch(error)}</span><br>"
-
-    def on_chain_start(self, serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) -> None:
-        """Print out that we are entering a chain."""
-        class_name = serialized["name"]
-        self.html += f"Entering chain: {ch(class_name)}<br>"
-
-    def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> None:
-        """Print out that we finished a chain."""
-        self.html += "Finished chain<br>"
-
-    def on_chain_error(self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any) -> None:
-        self.html += f"<span style='color:red'>Chain error: {ch(error)}</span><br>"
-
-    def on_tool_start(
-        self,
-        serialized: Dict[str, Any],
-        input_str: str,
-        color: Optional[str] = None,
-        **kwargs: Any,
-    ) -> None:
-        """Print out the log in specified color."""
-        pass
-
-    def on_tool_end(
-        self,
-        output: str,
-        color: Optional[str] = None,
-        observation_prefix: Optional[str] = None,
-        llm_prefix: Optional[str] = None,
-        **kwargs: Any,
-    ) -> None:
-        """If not the final action, print out observation."""
-        self.html += (
-            f"{ch(observation_prefix)}<br><span style='color:{color}'>{ch(output)}</span><br>{ch(llm_prefix)}<br>"
-        )
-
-    def on_tool_error(self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any) -> None:
-        self.html += f"<span style='color:red'>Tool error: {ch(error)}</span><br>"
-
-    def on_text(
-        self,
-        text: str,
-        color: Optional[str] = None,
-        **kwargs: Optional[str],
-    ) -> None:
-        """Run when agent ends."""
-        self.html += f"<span style='color:{color}'>{ch(text)}</span><br>"
-
-    def on_agent_action(self, action: AgentAction, color: Optional[str] = None, **kwargs: Any) -> Any:
-        self.html += f"<span style='color:{color}'>{ch(action.log)}</span><br>"
-
-    def on_agent_finish(self, finish: AgentFinish, color: Optional[str] = None, **kwargs: Any) -> None:
-        """Run on agent end."""
-        self.html += f"<span style='color:{color}'>{ch(finish.log)}</span><br>"
diff --git a/app/backend/lookuptool.py b/app/backend/lookuptool.py
deleted file mode 100644
index 7d0ce7b..0000000
--- a/app/backend/lookuptool.py
+++ /dev/null
@@ -1,27 +0,0 @@
-import csv
-from pathlib import Path
-from typing import Union
-
-from langchain.agents import Tool
-from langchain.callbacks.manager import Callbacks
-
-
-class CsvLookupTool(Tool):
-    data: dict[str, str] = {}
-
-    def __init__(
-        self,
-        filename: Union[str, Path],
-        key_field: str,
-        name: str = "lookup",
-        description: str = "useful to look up details given an input key as opposite to searching data with an unstructured question",
-        callbacks: Callbacks = None,
-    ):
-        super().__init__(name, self.lookup, description, callbacks=callbacks)
-        with open(filename, newline="") as csvfile:
-            reader = csv.DictReader(csvfile)
-            for row in reader:
-                self.data[row[key_field]] = "\n".join([f"{i}:{row[i]}" for i in row])
-
-    def lookup(self, key: str) -> str:
-        return self.data.get(key, "")
diff --git a/app/backend/requirements.in b/app/backend/requirements.in
index 3cb828a..85044a4 100644
--- a/app/backend/requirements.in
+++ b/app/backend/requirements.in
@@ -1,7 +1,6 @@
 azure-identity
 quart
 quart-cors
-langchain
 openai[datalib]
 tiktoken
 azure-search-documents==11.4.0b6
diff --git a/app/backend/requirements.txt b/app/backend/requirements.txt
index d897447..b5b2680 100644
--- a/app/backend/requirements.txt
+++ b/app/backend/requirements.txt
@@ -9,16 +9,11 @@ aiofiles==23.2.1
 aiohttp==3.8.5
     # via
     #   -r requirements.in
-    #   langchain
     #   openai
 aiosignal==1.3.1
     # via aiohttp
-annotated-types==0.5.0
-    # via pydantic
 anyio==3.7.1
-    # via
-    #   langchain
-    #   watchfiles
+    # via watchfiles
 asgiref==3.7.2
     # via opentelemetry-instrumentation-asgi
 async-timeout==4.0.3
@@ -73,8 +68,6 @@ cryptography==41.0.4
     #   azure-storage-blob
     #   msal
     #   pyjwt
-dataclasses-json==0.6.1
-    # via langchain
 deprecated==1.2.14
     # via opentelemetry-api
 et-xmlfile==1.1.0
@@ -87,8 +80,6 @@ frozenlist==1.4.0
     # via
     #   aiohttp
     #   aiosignal
-greenlet==2.0.2
-    # via sqlalchemy
 h11==0.14.0
     # via
     #   hypercorn
@@ -124,21 +115,11 @@ jinja2==3.1.2
     # via
     #   flask
     #   quart
-jsonpatch==1.33
-    # via langchain
-jsonpointer==2.4
-    # via jsonpatch
-langchain==0.0.305
-    # via -r requirements.in
-langsmith==0.0.41
-    # via langchain
 markupsafe==2.1.3
     # via
     #   jinja2
     #   quart
     #   werkzeug
-marshmallow==3.20.1
-    # via dataclasses-json
 msal==1.24.1
     # via
     #   -r requirements.in
@@ -154,14 +135,8 @@ multidict==6.0.4
     # via
     #   aiohttp
     #   yarl
-mypy-extensions==1.0.0
-    # via typing-inspect
-numexpr==2.8.7
-    # via langchain
 numpy==1.26.0
     # via
-    #   langchain
-    #   numexpr
     #   openai
     #   pandas
     #   pandas-stubs
@@ -260,9 +235,7 @@ opentelemetry-util-http==0.41b0
     #   opentelemetry-instrumentation-urllib3
     #   opentelemetry-instrumentation-wsgi
 packaging==23.1
-    # via
-    #   marshmallow
-    #   opentelemetry-instrumentation-flask
+    # via opentelemetry-instrumentation-flask
 pandas==2.1.1
     # via openai
 pandas-stubs==2.1.1.230928
@@ -273,12 +246,6 @@ priority==2.0.0
     # via hypercorn
 pycparser==2.21
     # via cffi
-pydantic==2.4.2
-    # via
-    #   langchain
-    #   langsmith
-pydantic-core==2.10.1
-    # via pydantic
 pyjwt[crypto]==2.8.0
     # via msal
 python-dateutil==2.8.2
@@ -288,10 +255,8 @@ python-dotenv==1.0.0
 pytz==2023.3.post1
     # via pandas
 pyyaml==6.0.1
-    # via
-    #   langchain
-    #   uvicorn
-quart==0.19.2
+    # via uvicorn
+quart==0.19.3
     # via
     #   -r requirements.in
     #   quart-cors
@@ -302,8 +267,6 @@ regex==2023.8.8
 requests==2.31.0
     # via
     #   azure-core
-    #   langchain
-    #   langsmith
     #   msal
     #   msrest
     #   openai
@@ -318,10 +281,6 @@ six==1.16.0
     #   python-dateutil
 sniffio==1.3.0
     # via anyio
-sqlalchemy==2.0.21
-    # via langchain
-tenacity==8.2.3
-    # via langchain
 tiktoken==0.5.1
     # via -r requirements.in
 tqdm==4.66.1
@@ -333,15 +292,9 @@ typing-extensions==4.8.0
     #   azure-core
     #   azure-storage-blob
     #   opentelemetry-sdk
-    #   pydantic
-    #   pydantic-core
-    #   sqlalchemy
-    #   typing-inspect
-typing-inspect==0.9.0
-    # via dataclasses-json
 tzdata==2023.3
     # via pandas
-urllib3==2.0.5
+urllib3==2.0.6
     # via requests
 uvicorn[standard]==0.23.2
     # via -r requirements.in
diff --git a/app/frontend/package-lock.json b/app/frontend/package-lock.json
index 9c8bdf4..380af00 100644
--- a/app/frontend/package-lock.json
+++ b/app/frontend/package-lock.json
@@ -8,7 +8,8 @@
       "name": "frontend",
       "version": "0.0.0",
       "dependencies": {
-        "@azure/msal-react": "^2.0.2",
+        "@azure/msal-browser": "^3.1.0",
+        "@azure/msal-react": "^2.0.4",
         "@fluentui/react": "^8.110.7",
         "@fluentui/react-components": "^9.33.0",
         "@fluentui/react-icons": "^2.0.206",
@@ -24,7 +25,7 @@
         "@types/react": "^18.2.14",
         "@types/react-dom": "^18.2.8",
         "@vitejs/plugin-react": "^4.0.2",
-        "prettier": "^3.0.0",
+        "prettier": "^3.0.3",
         "typescript": "^5.1.6",
         "vite": "^4.4.2"
       },
@@ -46,30 +47,28 @@
       }
     },
     "node_modules/@azure/msal-browser": {
-      "version": "3.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-3.0.2.tgz",
-      "integrity": "sha512-a2lIzWGV8EJhxVTHu2Vd5sRdK6DTisueq2HCXt49CQqc1hG/j9pK+ds+hJi+QLlZHPo3FPMwiTCHEA9sNgX3HA==",
-      "peer": true,
+      "version": "3.2.0",
+      "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-3.2.0.tgz",
+      "integrity": "sha512-le2qutddMiq0i3ErQaLKuwP1DpNgdd9iXPs3fSCsLuBrdGg9B4/j4ArCAHCwgxA82Ydj9BcqtMIL5BSWwU+P5A==",
       "dependencies": {
-        "@azure/msal-common": "14.0.2"
+        "@azure/msal-common": "14.1.0"
       },
       "engines": {
         "node": ">=0.8.0"
       }
     },
     "node_modules/@azure/msal-common": {
-      "version": "14.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-14.0.2.tgz",
-      "integrity": "sha512-yvVP63oXe20JFzvooTVxiX9UorxcL8VFoVwEtKuGlQnEnNGzrOpQZ6lKa1lqwGqxMtwKpW7oAFQhbbskG5RbYA==",
-      "peer": true,
+      "version": "14.1.0",
+      "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-14.1.0.tgz",
+      "integrity": "sha512-xphmhcfl5VL+uq5//VKMwQn+wfEZLMKNpFCcMi8Ur8ej5UT166g6chBsxgMzc9xo9Y24R9FB3m/tjDiV03xMIA==",
       "engines": {
         "node": ">=0.8.0"
       }
     },
     "node_modules/@azure/msal-react": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-react/-/msal-react-2.0.2.tgz",
-      "integrity": "sha512-EQn735OKtnxYf9JbQX94UG+Op14MugW379pI+gbKmcK8s5CaK2lClb9Zr+fA2isL3MOj8rOQsGGXSaIx9lSsAQ==",
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/@azure/msal-react/-/msal-react-2.0.4.tgz",
+      "integrity": "sha512-BujRm5FBDWYXyr3pnmubS4dIqZMlurYVtV2AyztoeAFUd+nh3XQZD9knHBqTyu53IDjhCCvUPUke/jSkv5WGlg==",
       "dependencies": {
         "@rollup/plugin-typescript": "^11.1.0",
         "rollup": "^3.20.2"
@@ -78,7 +77,7 @@
         "node": ">=10"
       },
       "peerDependencies": {
-        "@azure/msal-browser": "^3.0.2",
+        "@azure/msal-browser": "^3.2.0",
         "react": "^16.8.0 || ^17 || ^18"
       }
     },
@@ -3217,9 +3216,9 @@
       }
     },
     "node_modules/postcss": {
-      "version": "8.4.25",
-      "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.4.25.tgz",
-      "integrity": "sha512-7taJ/8t2av0Z+sQEvNzCkpDynl0tX3uJMCODi6nT3PfASC7dYCWV9aQ+uiCf+KBD4SEFcu+GvJdGdwzQ6OSjCw==",
+      "version": "8.4.31",
+      "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.4.31.tgz",
+      "integrity": "sha512-PS08Iboia9mts/2ygV3eLpY5ghnUcfLV/EXTOW1E2qYxJKGGBUtNjN76FYHnMs36RmARn41bC0AZmn+rR0OVpQ==",
       "dev": true,
       "funding": [
         {
@@ -3245,9 +3244,9 @@
       }
     },
     "node_modules/prettier": {
-      "version": "3.0.0",
-      "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.0.0.tgz",
-      "integrity": "sha512-zBf5eHpwHOGPC47h0zrPyNn+eAEIdEzfywMoYn2XPi0P44Zp0tSq64rq0xAREh4auw2cJZHo9QUob+NqCQky4g==",
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.0.3.tgz",
+      "integrity": "sha512-L/4pUDMxcNa8R/EthV08Zt42WBO4h1rarVtK0K+QJG0X187OLo7l699jWw0GKuwzkPQ//jMFA/8Xm6Fh3J/DAg==",
       "dev": true,
       "bin": {
         "prettier": "bin/prettier.cjs"
@@ -3600,24 +3599,22 @@
       }
     },
     "@azure/msal-browser": {
-      "version": "3.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-3.0.2.tgz",
-      "integrity": "sha512-a2lIzWGV8EJhxVTHu2Vd5sRdK6DTisueq2HCXt49CQqc1hG/j9pK+ds+hJi+QLlZHPo3FPMwiTCHEA9sNgX3HA==",
-      "peer": true,
+      "version": "3.2.0",
+      "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-3.2.0.tgz",
+      "integrity": "sha512-le2qutddMiq0i3ErQaLKuwP1DpNgdd9iXPs3fSCsLuBrdGg9B4/j4ArCAHCwgxA82Ydj9BcqtMIL5BSWwU+P5A==",
       "requires": {
-        "@azure/msal-common": "14.0.2"
+        "@azure/msal-common": "14.1.0"
       }
     },
     "@azure/msal-common": {
-      "version": "14.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-14.0.2.tgz",
-      "integrity": "sha512-yvVP63oXe20JFzvooTVxiX9UorxcL8VFoVwEtKuGlQnEnNGzrOpQZ6lKa1lqwGqxMtwKpW7oAFQhbbskG5RbYA==",
-      "peer": true
+      "version": "14.1.0",
+      "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-14.1.0.tgz",
+      "integrity": "sha512-xphmhcfl5VL+uq5//VKMwQn+wfEZLMKNpFCcMi8Ur8ej5UT166g6chBsxgMzc9xo9Y24R9FB3m/tjDiV03xMIA=="
     },
     "@azure/msal-react": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-react/-/msal-react-2.0.2.tgz",
-      "integrity": "sha512-EQn735OKtnxYf9JbQX94UG+Op14MugW379pI+gbKmcK8s5CaK2lClb9Zr+fA2isL3MOj8rOQsGGXSaIx9lSsAQ==",
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/@azure/msal-react/-/msal-react-2.0.4.tgz",
+      "integrity": "sha512-BujRm5FBDWYXyr3pnmubS4dIqZMlurYVtV2AyztoeAFUd+nh3XQZD9knHBqTyu53IDjhCCvUPUke/jSkv5WGlg==",
       "requires": {
         "@rollup/plugin-typescript": "^11.1.0",
         "rollup": "^3.20.2"
@@ -6055,9 +6052,9 @@
       "integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA=="
     },
     "postcss": {
-      "version": "8.4.25",
-      "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.4.25.tgz",
-      "integrity": "sha512-7taJ/8t2av0Z+sQEvNzCkpDynl0tX3uJMCODi6nT3PfASC7dYCWV9aQ+uiCf+KBD4SEFcu+GvJdGdwzQ6OSjCw==",
+      "version": "8.4.31",
+      "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.4.31.tgz",
+      "integrity": "sha512-PS08Iboia9mts/2ygV3eLpY5ghnUcfLV/EXTOW1E2qYxJKGGBUtNjN76FYHnMs36RmARn41bC0AZmn+rR0OVpQ==",
       "dev": true,
       "requires": {
         "nanoid": "^3.3.6",
@@ -6066,9 +6063,9 @@
       }
     },
     "prettier": {
-      "version": "3.0.0",
-      "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.0.0.tgz",
-      "integrity": "sha512-zBf5eHpwHOGPC47h0zrPyNn+eAEIdEzfywMoYn2XPi0P44Zp0tSq64rq0xAREh4auw2cJZHo9QUob+NqCQky4g==",
+      "version": "3.0.3",
+      "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.0.3.tgz",
+      "integrity": "sha512-L/4pUDMxcNa8R/EthV08Zt42WBO4h1rarVtK0K+QJG0X187OLo7l699jWw0GKuwzkPQ//jMFA/8Xm6Fh3J/DAg==",
       "dev": true
     },
     "prop-types": {
diff --git a/app/frontend/package.json b/app/frontend/package.json
index 2ec0e22..644d4e7 100644
--- a/app/frontend/package.json
+++ b/app/frontend/package.json
@@ -12,7 +12,8 @@
     "watch": "tsc && vite build --watch"
   },
   "dependencies": {
-    "@azure/msal-react": "^2.0.2",
+    "@azure/msal-react": "^2.0.4",
+    "@azure/msal-browser": "^3.1.0",
     "@fluentui/react": "^8.110.7",
     "@fluentui/react-components": "^9.33.0",
     "@fluentui/react-icons": "^2.0.206",
@@ -28,7 +29,7 @@
     "@types/react": "^18.2.14",
     "@types/react-dom": "^18.2.8",
     "@vitejs/plugin-react": "^4.0.2",
-    "prettier": "^3.0.0",
+    "prettier": "^3.0.3",
     "typescript": "^5.1.6",
     "vite": "^4.4.2"
   }
diff --git a/app/frontend/src/api/api.ts b/app/frontend/src/api/api.ts
index 444d4de..cf97ac5 100644
--- a/app/frontend/src/api/api.ts
+++ b/app/frontend/src/api/api.ts
@@ -1,6 +1,6 @@
 const BACKEND_URI = "";
 
-import { AskRequest, AskResponse, ChatRequest } from "./models";
+import { AskRequest, ChatAppResponse, ChatAppResponseOrError, ChatRequest } from "./models";
 import { useLogin } from "../authConfig";
 
 function getHeaders(idToken: string | undefined): Record<string, string> {
@@ -17,13 +17,12 @@ function getHeaders(idToken: string | undefined): Record<string, string> {
     return headers;
 }
 
-export async function askApi(options: AskRequest): Promise<AskResponse> {
+export async function askApi(options: AskRequest): Promise<ChatAppResponse> {
     const response = await fetch(`${BACKEND_URI}/ask`, {
         method: "POST",
         headers: getHeaders(options.idToken),
         body: JSON.stringify({
             question: options.question,
-            approach: options.approach,
             overrides: {
                 retrieval_mode: options.overrides?.retrievalMode,
                 semantic_ranker: options.overrides?.semanticRanker,
@@ -40,12 +39,12 @@ export async function askApi(options: AskRequest): Promise<AskResponse> {
         })
     });
 
-    const parsedResponse: AskResponse = await response.json();
+    const parsedResponse: ChatAppResponseOrError = await response.json();
     if (response.status > 299 || !response.ok) {
         throw Error(parsedResponse.error || "Unknown error");
     }
 
-    return parsedResponse;
+    return parsedResponse as ChatAppResponse;
 }
 
 export async function chatApi(options: ChatRequest): Promise<Response> {
@@ -55,7 +54,6 @@ export async function chatApi(options: ChatRequest): Promise<Response> {
         headers: getHeaders(options.idToken),
         body: JSON.stringify({
             history: options.history,
-            approach: options.approach,
             overrides: {
                 retrieval_mode: options.overrides?.retrievalMode,
                 semantic_ranker: options.overrides?.semanticRanker,
diff --git a/app/frontend/src/api/models.ts b/app/frontend/src/api/models.ts
index 4be7499..002e3df 100644
--- a/app/frontend/src/api/models.ts
+++ b/app/frontend/src/api/models.ts
@@ -1,9 +1,3 @@
-export const enum Approaches {
-    RetrieveThenRead = "rtr",
-    ReadRetrieveRead = "rrr",
-    ReadDecomposeAsk = "rda"
-}
-
 export const enum RetrievalMode {
     Hybrid = "hybrid",
     Vectors = "vectors",
@@ -27,18 +21,35 @@ export type AskRequestOverrides = {
 
 export type AskRequest = {
     question: string;
-    approach: Approaches;
     overrides?: AskRequestOverrides;
     idToken?: string;
 };
 
-export type AskResponse = {
-    answer: string;
+export type ResponseMessage = {
+    content: string;
+    role: string;
+}
+
+export type ResponseExtraArgs = {
     thoughts: string | null;
     data_points: string[];
+}
+
+export type ResponseChoice = {
+    index: number;
+    message: ResponseMessage;
+    extra_args: ResponseExtraArgs;
+};
+
+export type ChatAppResponseOrError = {
+    choices?: ResponseChoice[];
     error?: string;
 };
 
+export type ChatAppResponse = {
+    choices: ResponseChoice[];
+};
+
 export type ChatTurn = {
     user: string;
     bot?: string;
@@ -46,7 +57,6 @@ export type ChatTurn = {
 
 export type ChatRequest = {
     history: ChatTurn[];
-    approach: Approaches;
     overrides?: AskRequestOverrides;
     idToken?: string;
     shouldStream?: boolean;
diff --git a/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx b/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
index cd01a08..8f54582 100644
--- a/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
+++ b/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
@@ -4,7 +4,7 @@ import DOMPurify from "dompurify";
 import styles from "./AnalysisPanel.module.css";
 
 import { SupportingContent } from "../SupportingContent";
-import { AskResponse } from "../../api";
+import { ChatAppResponse } from "../../api";
 import { AnalysisPanelTabs } from "./AnalysisPanelTabs";
 
 interface Props {
@@ -13,17 +13,17 @@ interface Props {
     onActiveTabChanged: (tab: AnalysisPanelTabs) => void;
     activeCitation: string | undefined;
     citationHeight: string;
-    answer: AskResponse;
+    answer: ChatAppResponse;
 }
 
 const pivotItemDisabledStyle = { disabled: true, style: { color: "grey" } };
 
 export const AnalysisPanel = ({ answer, activeTab, activeCitation, citationHeight, className, onActiveTabChanged }: Props) => {
-    const isDisabledThoughtProcessTab: boolean = !answer.thoughts;
-    const isDisabledSupportingContentTab: boolean = !answer.data_points.length;
+    const isDisabledThoughtProcessTab: boolean = !answer.choices[0].extra_args.thoughts;
+    const isDisabledSupportingContentTab: boolean = !answer.choices[0].extra_args.data_points.length;
     const isDisabledCitationTab: boolean = !activeCitation;
 
-    const sanitizedThoughts = DOMPurify.sanitize(answer.thoughts!);
+    const sanitizedThoughts = DOMPurify.sanitize(answer.choices[0].extra_args.thoughts!);
 
     return (
         <Pivot
@@ -43,7 +43,7 @@ export const AnalysisPanel = ({ answer, activeTab, activeCitation, citationHeigh
                 headerText="Supporting content"
                 headerButtonProps={isDisabledSupportingContentTab ? pivotItemDisabledStyle : undefined}
             >
-                <SupportingContent supportingContent={answer.data_points} />
+                <SupportingContent supportingContent={answer.choices[0].extra_args.data_points} />
             </PivotItem>
             <PivotItem
                 itemKey={AnalysisPanelTabs.CitationTab}
diff --git a/app/frontend/src/components/Answer/Answer.tsx b/app/frontend/src/components/Answer/Answer.tsx
index 7444b26..4d9cc95 100644
--- a/app/frontend/src/components/Answer/Answer.tsx
+++ b/app/frontend/src/components/Answer/Answer.tsx
@@ -4,12 +4,12 @@ import DOMPurify from "dompurify";
 
 import styles from "./Answer.module.css";
 
-import { AskResponse, getCitationFilePath } from "../../api";
+import { ChatAppResponse, getCitationFilePath } from "../../api";
 import { parseAnswerToHtml } from "./AnswerParser";
 import { AnswerIcon } from "./AnswerIcon";
 
 interface Props {
-    answer: AskResponse;
+    answer: ChatAppResponse;
     isSelected?: boolean;
     isStreaming: boolean;
     onCitationClicked: (filePath: string) => void;
@@ -29,7 +29,8 @@ export const Answer = ({
     onFollowupQuestionClicked,
     showFollowupQuestions
 }: Props) => {
-    const parsedAnswer = useMemo(() => parseAnswerToHtml(answer.answer, isStreaming, onCitationClicked ), [answer]);
+    const messageContent = answer.choices[0].message.content;
+    const parsedAnswer = useMemo(() => parseAnswerToHtml(messageContent, isStreaming, onCitationClicked ), [answer]);
 
     const sanitizedAnswerHtml = DOMPurify.sanitize(parsedAnswer.answerHtml);
 
@@ -45,7 +46,7 @@ export const Answer = ({
                             title="Show thought process"
                             ariaLabel="Show thought process"
                             onClick={() => onThoughtProcessClicked()}
-                            disabled={!answer.thoughts}
+                            disabled={!answer.choices[0].extra_args.thoughts?.length}
                         />
                         <IconButton
                             style={{ color: "black" }}
@@ -53,7 +54,7 @@ export const Answer = ({
                             title="Show supporting content"
                             ariaLabel="Show supporting content"
                             onClick={() => onSupportingContentClicked()}
-                            disabled={!answer.data_points?.length}
+                            disabled={!answer.choices[0].extra_args.data_points?.length}
                         />
                     </div>
                 </Stack>
diff --git a/app/frontend/src/pages/chat/Chat.tsx b/app/frontend/src/pages/chat/Chat.tsx
index 47c09b2..ed33139 100644
--- a/app/frontend/src/pages/chat/Chat.tsx
+++ b/app/frontend/src/pages/chat/Chat.tsx
@@ -5,7 +5,7 @@ import readNDJSONStream from "ndjson-readablestream";
 
 import styles from "./Chat.module.css";
 
-import { chatApi, RetrievalMode, Approaches, AskResponse, ChatRequest, ChatTurn } from "../../api";
+import { chatApi, RetrievalMode, ChatAppResponse, ChatAppResponseOrError, ChatRequest, ChatTurn } from "../../api";
 import { Answer, AnswerError, AnswerLoading } from "../../components/Answer";
 import { QuestionInput } from "../../components/QuestionInput";
 import { ExampleList } from "../../components/Example";
@@ -41,18 +41,24 @@ const Chat = () => {
     const [activeAnalysisPanelTab, setActiveAnalysisPanelTab] = useState<AnalysisPanelTabs | undefined>(undefined);
 
     const [selectedAnswer, setSelectedAnswer] = useState<number>(0);
-    const [answers, setAnswers] = useState<[user: string, response: AskResponse][]>([]);
-    const [streamedAnswers, setStreamedAnswers] = useState<[user: string, response: AskResponse][]>([]);
+    const [answers, setAnswers] = useState<[user: string, response: ChatAppResponse][]>([]);
+    const [streamedAnswers, setStreamedAnswers] = useState<[user: string, response: ChatAppResponse][]>([]);
 
-    const handleAsyncRequest = async (question: string, answers: [string, AskResponse][], setAnswers: Function, responseBody: ReadableStream<any>) => {
+    const handleAsyncRequest = async (question: string, answers: [string, ChatAppResponse][], setAnswers: Function, responseBody: ReadableStream<any>) => {
         let answer: string = "";
-        let askResponse: AskResponse = {} as AskResponse;
+        let askResponse: ChatAppResponse = {} as ChatAppResponse;
 
         const updateState = (newContent: string) => {
             return new Promise(resolve => {
                 setTimeout(() => {
                     answer += newContent;
-                    const latestResponse: AskResponse = { ...askResponse, answer };
+                    const latestResponse: ChatAppResponse = { ...askResponse,
+                        choices: [{ ...askResponse.choices[0],
+                                    message: { content: answer,
+                                        role: askResponse.choices[0].message.role
+                                    } }
+                                ]
+                    };
                     setStreamedAnswers([...answers, [question, latestResponse]]);
                     resolve(null);
                 }, 33);
@@ -61,7 +67,8 @@ const Chat = () => {
         try {
             setIsStreaming(true);
             for await (const event of readNDJSONStream(responseBody)) {
-                if (event["data_points"]) {
+                if (event["choices"] && event["choices"][0]["extra_args"] && event["choices"][0]["extra_args"]["data_points"]) {
+                    event["choices"][0]["message"] = event["choices"][0]["delta"];
                     askResponse = event;
                 } else if (event["choices"] && event["choices"][0]["delta"]["content"]) {
                     setIsLoading(false);
@@ -71,7 +78,13 @@ const Chat = () => {
         } finally {
             setIsStreaming(false);
         }
-        const fullResponse: AskResponse = { ...askResponse, answer };
+        const fullResponse: ChatAppResponse = { ...askResponse,
+            choices: [{ ...askResponse.choices[0],
+                        message: { content: answer,
+                            role: askResponse.choices[0].message.role
+                        } }
+                    ]
+        };
         return fullResponse;
     };
 
@@ -88,10 +101,12 @@ const Chat = () => {
         const token = client ? await getToken(client) : undefined;
 
         try {
-            const history: ChatTurn[] = answers.map(a => ({ user: a[0], bot: a[1].answer }));
+            const history: ChatTurn[] = answers.map(a => ({
+                user: a[0],
+                bot: a[1].choices[0].message.content
+            }));
             const request: ChatRequest = {
                 history: [...history, { user: question, bot: undefined }],
-                approach: Approaches.ReadRetrieveRead,
                 shouldStream: shouldStream,
                 overrides: {
                     promptTemplate: promptTemplate.length === 0 ? undefined : promptTemplate,
@@ -112,14 +127,14 @@ const Chat = () => {
                 throw Error("No response body");
             }
             if (shouldStream) {
-                const parsedResponse: AskResponse = await handleAsyncRequest(question, answers, setAnswers, response.body);
+                const parsedResponse: ChatAppResponse = await handleAsyncRequest(question, answers, setAnswers, response.body);
                 setAnswers([...answers, [question, parsedResponse]]);
             } else {
-                const parsedResponse: AskResponse = await response.json();
+                const parsedResponse: ChatAppResponseOrError = await response.json();
                 if (response.status > 299 || !response.ok) {
                     throw Error(parsedResponse.error || "Unknown error");
                 }
-                setAnswers([...answers, [question, parsedResponse]]);
+                setAnswers([...answers, [question, parsedResponse as ChatAppResponse]]);
             }
         } catch (e) {
             setError(e);
diff --git a/app/frontend/src/pages/oneshot/OneShot.tsx b/app/frontend/src/pages/oneshot/OneShot.tsx
index 89cd1a2..18caf04 100644
--- a/app/frontend/src/pages/oneshot/OneShot.tsx
+++ b/app/frontend/src/pages/oneshot/OneShot.tsx
@@ -3,7 +3,7 @@ import { Checkbox, ChoiceGroup, IChoiceGroupOption, Panel, DefaultButton, Spinne
 
 import styles from "./OneShot.module.css";
 
-import { askApi, Approaches, AskResponse, AskRequest, RetrievalMode } from "../../api";
+import { askApi, ChatAppResponse, AskRequest, RetrievalMode } from "../../api";
 import { Answer, AnswerError } from "../../components/Answer";
 import { QuestionInput } from "../../components/QuestionInput";
 import { ExampleList } from "../../components/Example";
@@ -15,7 +15,6 @@ import { TokenClaimsDisplay } from "../../components/TokenClaimsDisplay";
 
 export function Component(): JSX.Element {
     const [isConfigPanelOpen, setIsConfigPanelOpen] = useState(false);
-    const [approach, setApproach] = useState<Approaches>(Approaches.RetrieveThenRead);
     const [promptTemplate, setPromptTemplate] = useState<string>("");
     const [promptTemplatePrefix, setPromptTemplatePrefix] = useState<string>("");
     const [promptTemplateSuffix, setPromptTemplateSuffix] = useState<string>("");
@@ -31,7 +30,7 @@ export function Component(): JSX.Element {
 
     const [isLoading, setIsLoading] = useState<boolean>(false);
     const [error, setError] = useState<unknown>();
-    const [answer, setAnswer] = useState<AskResponse>();
+    const [answer, setAnswer] = useState<ChatAppResponse>();
 
     const [activeCitation, setActiveCitation] = useState<string>();
     const [activeAnalysisPanelTab, setActiveAnalysisPanelTab] = useState<AnalysisPanelTabs | undefined>(undefined);
@@ -51,7 +50,6 @@ export function Component(): JSX.Element {
         try {
             const request: AskRequest = {
                 question,
-                approach,
                 overrides: {
                     promptTemplate: promptTemplate.length === 0 ? undefined : promptTemplate,
                     promptTemplatePrefix: promptTemplatePrefix.length === 0 ? undefined : promptTemplatePrefix,
@@ -95,10 +93,6 @@ export function Component(): JSX.Element {
         setRetrievalMode(option?.data || RetrievalMode.Hybrid);
     };
 
-    const onApproachChange = (_ev?: React.FormEvent<HTMLElement | HTMLInputElement>, option?: IChoiceGroupOption) => {
-        setApproach((option?.key as Approaches) || Approaches.RetrieveThenRead);
-    };
-
     const onUseSemanticRankerChange = (_ev?: React.FormEvent<HTMLElement | HTMLInputElement>, checked?: boolean) => {
         setUseSemanticRanker(!!checked);
     };
@@ -140,21 +134,6 @@ export function Component(): JSX.Element {
         setUseGroupsSecurityFilter(!!checked);
     };
 
-    const approaches: IChoiceGroupOption[] = [
-        {
-            key: Approaches.RetrieveThenRead,
-            text: "Retrieve-Then-Read"
-        },
-        {
-            key: Approaches.ReadRetrieveRead,
-            text: "Read-Retrieve-Read"
-        },
-        {
-            key: Approaches.ReadDecomposeAsk,
-            text: "Read-Decompose-Ask"
-        }
-    ];
-
     return (
         <div className={styles.oneshotContainer}>
             <div className={styles.oneshotTopSection}>
@@ -208,46 +187,16 @@ export function Component(): JSX.Element {
                 onRenderFooterContent={() => <DefaultButton onClick={() => setIsConfigPanelOpen(false)}>Close</DefaultButton>}
                 isFooterAtBottom={true}
             >
-                <ChoiceGroup
+
+                <TextField
                     className={styles.oneshotSettingsSeparator}
-                    label="Approach"
-                    options={approaches}
-                    defaultSelectedKey={approach}
-                    onChange={onApproachChange}
+                    defaultValue={promptTemplate}
+                    label="Override prompt template"
+                    multiline
+                    autoAdjustHeight
+                    onChange={onPromptTemplateChange}
                 />
 
-                {(approach === Approaches.RetrieveThenRead || approach === Approaches.ReadDecomposeAsk) && (
-                    <TextField
-                        className={styles.oneshotSettingsSeparator}
-                        defaultValue={promptTemplate}
-                        label="Override prompt template"
-                        multiline
-                        autoAdjustHeight
-                        onChange={onPromptTemplateChange}
-                    />
-                )}
-
-                {approach === Approaches.ReadRetrieveRead && (
-                    <>
-                        <TextField
-                            className={styles.oneshotSettingsSeparator}
-                            defaultValue={promptTemplatePrefix}
-                            label="Override prompt prefix template"
-                            multiline
-                            autoAdjustHeight
-                            onChange={onPromptTemplatePrefixChange}
-                        />
-                        <TextField
-                            className={styles.oneshotSettingsSeparator}
-                            defaultValue={promptTemplateSuffix}
-                            label="Override prompt suffix template"
-                            multiline
-                            autoAdjustHeight
-                            onChange={onPromptTemplateSuffixChange}
-                        />
-                    </>
-                )}
-
                 <SpinButton
                     className={styles.oneshotSettingsSeparator}
                     label="Retrieve this many search results:"
diff --git a/infra/main.bicep b/infra/main.bicep
index 2452874..d0db43e 100644
--- a/infra/main.bicep
+++ b/infra/main.bicep
@@ -58,7 +58,7 @@ param chatGptDeploymentName string // Set in main.parameters.json
 param chatGptDeploymentCapacity int = 30
 param chatGptModelName string = (openAiHost == 'azure') ? 'gpt-35-turbo' : 'gpt-3.5-turbo'
 param chatGptModelVersion string = '0613'
-param embeddingDeploymentName string = 'embedding'
+param embeddingDeploymentName string // Set in main.parameters.json
 param embeddingDeploymentCapacity int = 30
 param embeddingModelName string = 'text-embedding-ada-002'
 
@@ -142,7 +142,7 @@ module backend 'core/host/appservice.bicep' = {
     tags: union(tags, { 'azd-service-name': 'backend' })
     appServicePlanId: appServicePlan.outputs.id
     runtimeName: 'python'
-    runtimeVersion: '3.10'
+    runtimeVersion: '3.11'
     appCommandLine: 'python3 -m gunicorn main:app'
     scmDoBuildDuringDeployment: true
     managedIdentity: true
diff --git a/infra/main.parameters.json b/infra/main.parameters.json
index 5cbcc19..d18a4b8 100644
--- a/infra/main.parameters.json
+++ b/infra/main.parameters.json
@@ -59,6 +59,9 @@
     "chatGptDeploymentName": {
       "value": "${AZURE_OPENAI_CHATGPT_DEPLOYMENT=chat}"
     },
+    "embeddingDeploymentName": {
+      "value": "${AZURE_OPENAI_EMB_DEPLOYMENT=embedding}"
+    },
     "openAiHost":{
       "value": "${OPENAI_HOST=azure}"
     },
diff --git a/locustfile.py b/locustfile.py
index 563d6b7..3bcf62d 100644
--- a/locustfile.py
+++ b/locustfile.py
@@ -26,7 +26,6 @@ class ChatUser(HttpUser):
                         )
                     }
                 ],
-                "approach": "rrr",
                 "overrides": {
                     "retrieval_mode": "hybrid",
                     "semantic_ranker": True,
@@ -47,7 +46,6 @@ class ChatUser(HttpUser):
                     },
                     {"user": "Does my plan cover eye exams?"},
                 ],
-                "approach": "rrr",
                 "overrides": {
                     "retrieval_mode": "hybrid",
                     "semantic_ranker": True,
diff --git a/pyproject.toml b/pyproject.toml
index 26d8bcc..def1afe 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -16,3 +16,14 @@ source = ["scripts", "app"]
 
 [tool.coverage.report]
 show_missing = true
+
+[tool.mypy]
+check_untyped_defs = true
+python_version = 3.9
+
+[[tool.mypy.overrides]]
+module = [
+    "msal.*",
+    "msal_extensions.*",
+]
+ignore_missing_imports = true
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 4108af1..f5408b2 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -12,3 +12,4 @@ pytest-playwright
 pre-commit
 locust
 pip-tools
+mypy
diff --git a/scripts/adlsgen2setup.py b/scripts/adlsgen2setup.py
index 0c4bd59..00de15e 100644
--- a/scripts/adlsgen2setup.py
+++ b/scripts/adlsgen2setup.py
@@ -3,6 +3,7 @@ import asyncio
 import json
 import logging
 import os
+from typing import Any, Optional
 
 import aiohttp
 from azure.core.credentials_async import AsyncTokenCredential
@@ -24,7 +25,7 @@ class AdlsGen2Setup:
         storage_account_name: str,
         filesystem_name: str,
         security_enabled_groups: bool,
-        data_access_control_format: dict[str, any],
+        data_access_control_format: dict[str, Any],
         credentials: AsyncTokenCredential,
     ):
         """
@@ -49,7 +50,7 @@ class AdlsGen2Setup:
         self.credentials = credentials
         self.security_enabled_groups = security_enabled_groups
         self.data_access_control_format = data_access_control_format
-        self.graph_headers = None
+        self.graph_headers: Optional[dict[str, str]] = None
 
     async def run(self):
         async with self.create_service_client() as service_client:
@@ -143,7 +144,7 @@ class AdlsGen2Setup:
         return group_id
 
 
-async def main(args: any):
+async def main(args: Any):
     async with AzureDeveloperCliCredential() as credentials:
         with open(args.data_access_control) as f:
             data_access_control_format = json.load(f)
diff --git a/scripts/manageacl.py b/scripts/manageacl.py
index 6b49ae4..fa02bd2 100644
--- a/scripts/manageacl.py
+++ b/scripts/manageacl.py
@@ -2,11 +2,11 @@ import argparse
 import asyncio
 import json
 import logging
-from typing import Union
+from typing import Any, Union
 
 from azure.core.credentials import AzureKeyCredential
 from azure.core.credentials_async import AsyncTokenCredential
-from azure.identity import AzureDeveloperCliCredential
+from azure.identity.aio import AzureDeveloperCliCredential
 from azure.search.documents.aio import SearchClient
 from azure.search.documents.indexes.aio import SearchIndexClient
 from azure.search.documents.indexes.models import (
@@ -143,14 +143,16 @@ class ManageAcl:
             await search_index_client.create_or_update_index(index_definition)
 
 
-async def main(args: any):
+async def main(args: Any):
     # Use the current user identity to connect to Azure services unless a key is explicitly set for any of them
     azd_credential = (
         AzureDeveloperCliCredential()
         if args.tenant_id is None
         else AzureDeveloperCliCredential(tenant_id=args.tenant_id, process_timeout=60)
     )
-    search_credential = azd_credential if args.search_key is None else AzureKeyCredential(args.search_key)
+    search_credential: Union[AsyncTokenCredential, AzureKeyCredential] = azd_credential
+    if args.search_key is not None:
+        search_credential = AzureKeyCredential(args.search_key)
 
     command = ManageAcl(
         service_name=args.search_service,
diff --git a/scripts/prepdocs.py b/scripts/prepdocs.py
index c555928..a718758 100644
--- a/scripts/prepdocs.py
+++ b/scripts/prepdocs.py
@@ -7,11 +7,12 @@ import os
 import re
 import tempfile
 import time
+from typing import Any, Optional, Union
 
 import openai
 import tiktoken
 from azure.ai.formrecognizer import DocumentAnalysisClient
-from azure.core.credentials import AzureKeyCredential
+from azure.core.credentials import AzureKeyCredential, TokenCredential
 from azure.identity import AzureDeveloperCliCredential
 from azure.search.documents import SearchClient
 from azure.search.documents.indexes import SearchIndexClient
@@ -60,7 +61,7 @@ MAX_SECTION_LENGTH = 1000
 SENTENCE_SEARCH_LIMIT = 100
 SECTION_OVERLAP = 100
 
-open_ai_token_cache = {}
+open_ai_token_cache: dict[str, Any] = {}
 CACHE_KEY_TOKEN_CRED = "openai_token_cred"
 CACHE_KEY_CREATED_TIME = "created_time"
 CACHE_KEY_TOKEN_TYPE = "token_type"
@@ -118,7 +119,7 @@ def remove_blobs(filename):
     blob_container = blob_service.get_container_client(args.container)
     if blob_container.exists():
         if filename is None:
-            blobs = blob_container.list_blob_names()
+            blobs = iter(blob_container.list_blob_names())
         else:
             prefix = os.path.splitext(os.path.basename(filename))[0]
             blobs = filter(
@@ -177,8 +178,8 @@ def get_document_text(filename):
         for page_num, page in enumerate(form_recognizer_results.pages):
             tables_on_page = [
                 table
-                for table in form_recognizer_results.tables
-                if table.bounding_regions[0].page_number == page_num + 1
+                for table in (form_recognizer_results.tables or [])
+                if table.bounding_regions and table.bounding_regions[0].page_number == page_num + 1
             ]
 
             # mark all positions of the table spans in the page
@@ -289,7 +290,9 @@ def filename_to_id(filename):
     return f"file-{filename_ascii}-{filename_hash}"
 
 
-def create_sections(filename, page_map, use_vectors, embedding_deployment: str = None, embedding_model: str = None):
+def create_sections(
+    filename, page_map, use_vectors, embedding_deployment: Optional[str] = None, embedding_model: Optional[str] = None
+):
     file_id = filename_to_id(filename)
     for i, (content, pagenum) in enumerate(split_text(page_map, filename)):
         section = {
@@ -397,7 +400,7 @@ def create_search_index():
 
 
 def update_embeddings_in_batch(sections):
-    batch_queue = []
+    batch_queue: list = []
     copy_s = []
     batch_response = {}
     token_count = 0
@@ -469,9 +472,9 @@ def remove_from_index(filename):
         r = search_client.search("", filter=filter, top=1000, include_total_count=True)
         if r.get_count() == 0:
             break
-        r = search_client.delete_documents(documents=[{"id": d["id"]} for d in r])
+        removed_docs = search_client.delete_documents(documents=[{"id": d["id"]} for d in r])
         if args.verbose:
-            print(f"\tRemoved {len(r)} sections from index")
+            print(f"\tRemoved {len(removed_docs)} sections from index")
         # It can take a few seconds for search results to reflect changes, so wait a bit
         time.sleep(2)
 
@@ -494,8 +497,8 @@ def read_files(
     path_pattern: str,
     use_vectors: bool,
     vectors_batch_support: bool,
-    embedding_deployment: str = None,
-    embedding_model: str = None,
+    embedding_deployment: Optional[str] = None,
+    embedding_model: Optional[str] = None,
 ):
     """
     Recursively read directory structure under `path_pattern`
@@ -530,7 +533,10 @@ def read_files(
 
 
 def read_adls_gen2_files(
-    use_vectors: bool, vectors_batch_support: bool, embedding_deployment: str = None, embedding_model: str = None
+    use_vectors: bool,
+    vectors_batch_support: bool,
+    embedding_deployment: Optional[str] = None,
+    embedding_model: Optional[str] = None,
 ):
     datalake_service = DataLakeServiceClient(
         account_url=f"https://{args.datalakestorageaccount}.dfs.core.windows.net", credential=adls_gen2_creds
@@ -549,7 +555,7 @@ def read_adls_gen2_files(
                     file_client = filesystem_client.get_file_client(path)
                     file_client.download_file().readinto(temp_file)
 
-                    acls = None
+                    acls: Optional[dict[str, list]] = None
                     if args.useacls:
                         # Parse out user ids and group ids
                         acls = {"oids": [], "groups": []}
@@ -560,7 +566,7 @@ def read_adls_gen2_files(
                         # ACL Format: user::rwx,group::r-x,other::r--,user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r--
                         acl_list = acl_list.split(",")
                         for acl in acl_list:
-                            acl_parts = acl.split(":")
+                            acl_parts: list = acl.split(":")
                             if len(acl_parts) != 3:
                                 continue
                             if len(acl_parts[1]) == 0:
@@ -706,14 +712,15 @@ if __name__ == "__main__":
         if args.tenantid is None
         else AzureDeveloperCliCredential(tenant_id=args.tenantid, process_timeout=60)
     )
-    default_creds = azd_credential if args.searchkey is None or args.storagekey is None else None
     adls_gen2_creds = azd_credential if args.datalakekey is None else AzureKeyCredential(args.datalakekey)
-    search_creds = default_creds if args.searchkey is None else AzureKeyCredential(args.searchkey)
+    search_creds: Union[TokenCredential, AzureKeyCredential] = azd_credential
+    if args.searchkey is not None:
+        search_creds = AzureKeyCredential(args.searchkey)
     use_vectors = not args.novectors
     compute_vectors_in_batch = not args.disablebatchvectors and args.openaimodelname in SUPPORTED_BATCH_AOAI_MODEL
 
     if not args.skipblobs:
-        storage_creds = default_creds if args.storagekey is None else args.storagekey
+        storage_creds = azd_credential if args.storagekey is None else args.storagekey
     if not args.localpdfparser:
         # check if Azure Form Recognizer credentials are provided
         if args.formrecognizerservice is None:
@@ -721,9 +728,9 @@ if __name__ == "__main__":
                 "Error: Azure Form Recognizer service is not provided. Please provide formrecognizerservice or use --localpdfparser for local pypdf parser."
             )
             exit(1)
-        formrecognizer_creds = (
-            default_creds if args.formrecognizerkey is None else AzureKeyCredential(args.formrecognizerkey)
-        )
+        formrecognizer_creds: Union[TokenCredential, AzureKeyCredential] = azd_credential
+        if args.formrecognizerkey is not None:
+            formrecognizer_creds = AzureKeyCredential(args.formrecognizerkey)
 
     if use_vectors:
         if args.openaihost != "openai":
diff --git a/scripts/requirements.txt b/scripts/requirements.txt
index 0329999..a8a5d54 100644
--- a/scripts/requirements.txt
+++ b/scripts/requirements.txt
@@ -13,7 +13,7 @@ async-timeout==4.0.3
 attrs==23.1.0
     # via aiohttp
 azure-ai-formrecognizer==3.3.0
-    # via -r scripts/requirements.in
+    # via -r requirements.in
 azure-common==1.1.28
     # via
     #   azure-ai-formrecognizer
@@ -27,15 +27,15 @@ azure-core==1.29.4
     #   azure-storage-file-datalake
     #   msrest
 azure-identity==1.14.0
-    # via -r scripts/requirements.in
+    # via -r requirements.in
 azure-search-documents==11.4.0b6
-    # via -r scripts/requirements.in
+    # via -r requirements.in
 azure-storage-blob==12.18.2
     # via
-    #   -r scripts/requirements.in
+    #   -r requirements.in
     #   azure-storage-file-datalake
 azure-storage-file-datalake==12.13.1
-    # via -r scripts/requirements.in
+    # via -r requirements.in
 certifi==2023.7.22
     # via
     #   msrest
@@ -88,7 +88,7 @@ numpy==1.26.0
 oauthlib==3.2.2
     # via requests-oauthlib
 openai[datalib]==0.28.1
-    # via -r scripts/requirements.in
+    # via -r requirements.in
 openpyxl==3.1.2
     # via openai
 pandas==2.1.1
@@ -101,8 +101,8 @@ pycparser==2.21
     # via cffi
 pyjwt[crypto]==2.8.0
     # via msal
-pypdf==3.16.2
-    # via -r scripts/requirements.in
+pypdf==3.16.3
+    # via -r requirements.in
 python-dateutil==2.8.2
     # via pandas
 pytz==2023.3.post1
@@ -125,9 +125,9 @@ six==1.16.0
     #   isodate
     #   python-dateutil
 tenacity==8.2.3
-    # via -r scripts/requirements.in
+    # via -r requirements.in
 tiktoken==0.5.1
-    # via -r scripts/requirements.in
+    # via -r requirements.in
 tqdm==4.66.1
     # via openai
 types-pytz==2023.3.1.1
@@ -140,7 +140,7 @@ typing-extensions==4.8.0
     #   azure-storage-file-datalake
 tzdata==2023.3
     # via pandas
-urllib3==2.0.5
+urllib3==2.0.6
     # via requests
 yarl==1.9.2
     # via aiohttp


Test patch:
diff --git a/.github/workflows/python-test.yaml b/.github/workflows/python-test.yaml
index bdb89ee..4ed5652 100644
--- a/.github/workflows/python-test.yaml
+++ b/.github/workflows/python-test.yaml
@@ -37,10 +37,12 @@ jobs:
             pip install -r requirements-dev.txt
         - name: Lint with ruff
           run: ruff .
+        - name: Check types with mypy
+          run: python3 -m mypy scripts/ app/
         - name: Check formatting with black
           run: black . --check --verbose
         - name: Run Python tests
-          run: python3 -m pytest --cov
+          run: python3 -m pytest --cov --cov-fail-under=78
         - name: Run E2E tests with Playwright
           run: |
             playwright install --with-deps
diff --git a/tests/conftest.py b/tests/conftest.py
index 206b98f..5f2c262 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -40,20 +40,18 @@ def mock_openai_embedding(monkeypatch):
 def mock_openai_chatcompletion(monkeypatch):
     class AsyncChatCompletionIterator:
         def __init__(self, answer):
-            self.num = 2
-            self.answer = answer
+            self.responses = [
+                {"object": "chat.completion.chunk", "choices": []},
+                {"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]},
+                {"object": "chat.completion.chunk", "choices": [{"delta": {"content": answer}}]},
+            ]
 
         def __aiter__(self):
             return self
 
         async def __anext__(self):
-            if self.num == 2:
-                self.num -= 1
-                # Emulate the first response being empty - bug with "2023-07-01-preview"
-                return openai.util.convert_to_openai_object({"choices": []})
-            elif self.num == 1:
-                self.num -= 1
-                return openai.util.convert_to_openai_object({"choices": [{"delta": {"content": self.answer}}]})
+            if self.responses:
+                return self.responses.pop(0)
             else:
                 raise StopAsyncIteration
 
@@ -70,7 +68,9 @@ def mock_openai_chatcompletion(monkeypatch):
         if "stream" in kwargs and kwargs["stream"] is True:
             return AsyncChatCompletionIterator(answer)
         else:
-            return openai.util.convert_to_openai_object({"choices": [{"message": {"content": answer}}]})
+            return openai.util.convert_to_openai_object(
+                {"object": "chat.completion", "choices": [{"message": {"role": "assistant", "content": answer}}]}
+            )
 
     monkeypatch.setattr(openai.ChatCompletion, "acreate", mock_acreate)
 
diff --git a/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
index 4379dc6..87ebda3 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
@@ -1,5 +1,15 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n '}"
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n '}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
+    ],
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
index 7778198..c400f9f 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
index 7778198..c400f9f 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_hybrid/client0/result.json b/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_hybrid/client1/result.json b/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json b/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
index b04d12b..2a92fc5 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': 'You are a cat.'}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': 'You are a cat.'}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json b/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
index b04d12b..2a92fc5 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': 'You are a cat.'}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': 'You are a cat.'}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json b/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
index b354871..2d64cd3 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n Meow like a cat.\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n Meow like a cat.\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json b/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
index b354871..2d64cd3 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n Meow like a cat.\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n Meow like a cat.\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
index 3cb1cce..4ad2e5c 100644
--- a/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
@@ -1,2 +1,3 @@
-{"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}
-{"choices": [{"delta": {"content": "The capital of France is Paris."}}]}
+{"choices": [{"delta": {"role": "assistant"}, "extra_args": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris."}}]}
diff --git a/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
index 3cb1cce..4ad2e5c 100644
--- a/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
@@ -1,2 +1,3 @@
-{"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}
-{"choices": [{"delta": {"content": "The capital of France is Paris."}}]}
+{"choices": [{"delta": {"role": "assistant"}, "extra_args": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris."}}]}
diff --git a/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
index 75e296a..cc6d520 100644
--- a/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
@@ -1,2 +1,3 @@
-{"data_points": [], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"}
-{"choices": [{"delta": {"content": "The capital of France is Paris."}}]}
+{"choices": [{"delta": {"role": "assistant"}, "extra_args": {"data_points": [], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris."}}]}
diff --git a/tests/snapshots/test_app/test_chat_text/client0/result.json b/tests/snapshots/test_app/test_chat_text/client0/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_text/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text/client1/result.json b/tests/snapshots/test_app/test_chat_text/client1/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_text/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json b/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
index 77d891a..ddcb422 100644
--- a/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
@@ -1,5 +1,15 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
+    ],
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
index 0ca3460..2ce4879 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
index 0ca3460..2ce4879 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json b/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json b/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_vector/client0/result.json b/tests/snapshots/test_app/test_chat_vector/client0/result.json
index 140529a..4117f95 100644
--- a/tests/snapshots/test_app/test_chat_vector/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_vector/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>None<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>None<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_vector/client1/result.json b/tests/snapshots/test_app/test_chat_vector/client1/result.json
index 140529a..4117f95 100644
--- a/tests/snapshots/test_app/test_chat_vector/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_vector/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>None<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>None<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/test_app.py b/tests/test_app.py
index 256bdbf..843dbcb 100644
--- a/tests/test_app.py
+++ b/tests/test_app.py
@@ -45,18 +45,11 @@ async def test_ask_request_must_be_json(client):
     assert result["error"] == "request must be json"
 
 
-@pytest.mark.asyncio
-async def test_ask_with_unknown_approach(client):
-    response = await client.post("/ask", json={"approach": "test"})
-    assert response.status_code == 400
-
-
 @pytest.mark.asyncio
 async def test_ask_rtr_text(client, snapshot):
     response = await client.post(
         "/ask",
         json={
-            "approach": "rtr",
             "question": "What is the capital of France?",
             "overrides": {"retrieval_mode": "text"},
         },
@@ -72,7 +65,6 @@ async def test_ask_rtr_text_filter(auth_client, snapshot):
         "/ask",
         headers={"Authorization": "Bearer MockToken"},
         json={
-            "approach": "rtr",
             "question": "What is the capital of France?",
             "overrides": {
                 "retrieval_mode": "text",
@@ -96,7 +88,6 @@ async def test_ask_rtr_text_semanticranker(client, snapshot):
     response = await client.post(
         "/ask",
         json={
-            "approach": "rtr",
             "question": "What is the capital of France?",
             "overrides": {"retrieval_mode": "text", "semantic_ranker": True},
         },
@@ -111,7 +102,6 @@ async def test_ask_rtr_text_semanticcaptions(client, snapshot):
     response = await client.post(
         "/ask",
         json={
-            "approach": "rtr",
             "question": "What is the capital of France?",
             "overrides": {"retrieval_mode": "text", "semantic_captions": True},
         },
@@ -126,7 +116,6 @@ async def test_ask_rtr_hybrid(client, snapshot):
     response = await client.post(
         "/ask",
         json={
-            "approach": "rtr",
             "question": "What is the capital of France?",
             "overrides": {"retrieval_mode": "hybrid"},
         },
@@ -144,18 +133,11 @@ async def test_chat_request_must_be_json(client):
     assert result["error"] == "request must be json"
 
 
-@pytest.mark.asyncio
-async def test_chat_with_unknown_approach(client):
-    response = await client.post("/chat", json={"approach": "test"})
-    assert response.status_code == 400
-
-
 @pytest.mark.asyncio
 async def test_chat_text(client, snapshot):
     response = await client.post(
         "/chat",
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {"retrieval_mode": "text"},
         },
@@ -171,7 +153,6 @@ async def test_chat_text_filter(auth_client, snapshot):
         "/chat",
         headers={"Authorization": "Bearer MockToken"},
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {
                 "retrieval_mode": "text",
@@ -195,7 +176,6 @@ async def test_chat_text_semanticranker(client, snapshot):
     response = await client.post(
         "/chat",
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {"retrieval_mode": "text", "semantic_ranker": True},
         },
@@ -210,7 +190,6 @@ async def test_chat_text_semanticcaptions(client, snapshot):
     response = await client.post(
         "/chat",
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {"retrieval_mode": "text", "semantic_captions": True},
         },
@@ -225,7 +204,6 @@ async def test_chat_prompt_template(client, snapshot):
     response = await client.post(
         "/chat",
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {"retrieval_mode": "text", "prompt_template": "You are a cat."},
         },
@@ -240,7 +218,6 @@ async def test_chat_prompt_template_concat(client, snapshot):
     response = await client.post(
         "/chat",
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {"retrieval_mode": "text", "prompt_template": ">>> Meow like a cat."},
         },
@@ -255,7 +232,6 @@ async def test_chat_hybrid(client, snapshot):
     response = await client.post(
         "/chat",
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {"retrieval_mode": "hybrid"},
         },
@@ -270,7 +246,6 @@ async def test_chat_vector(client, snapshot):
     response = await client.post(
         "/chat",
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {"retrieval_mode": "vector"},
         },
@@ -288,18 +263,11 @@ async def test_chat_stream_request_must_be_json(client):
     assert result["error"] == "request must be json"
 
 
-@pytest.mark.asyncio
-async def test_chat_stream_with_unknown_approach(client):
-    response = await client.post("/chat_stream", json={"approach": "test"})
-    assert response.status_code == 400
-
-
 @pytest.mark.asyncio
 async def test_chat_stream_text(client, snapshot):
     response = await client.post(
         "/chat_stream",
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {"retrieval_mode": "text"},
         },
@@ -315,7 +283,6 @@ async def test_chat_stream_text_filter(auth_client, snapshot):
         "/chat_stream",
         headers={"Authorization": "Bearer MockToken"},
         json={
-            "approach": "rrr",
             "history": [{"user": "What is the capital of France?"}],
             "overrides": {
                 "retrieval_mode": "text",
diff --git a/tests/test_chatapproach.py b/tests/test_chatapproach.py
index 5ce4145..4dc40d2 100644
--- a/tests/test_chatapproach.py
+++ b/tests/test_chatapproach.py
@@ -21,3 +21,82 @@ def test_get_search_query_returns_default():
     query = chat_approach.get_search_query(json.loads(payload), default_query)
 
     assert query == default_query
+
+
+def test_get_messages_from_history():
+    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+
+    messages = chat_approach.get_messages_from_history(
+        system_prompt="You are a bot.",
+        model_id="gpt-35-turbo",
+        history=[
+            {
+                "user": "What happens in a performance review?",
+                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+            },
+            {"user": "What does a Product Manager do?"},
+        ],
+        user_content="What does a Product Manager do?",
+    )
+    assert messages == [
+        {"role": "system", "content": "You are a bot."},
+        {"role": "user", "content": "What happens in a performance review?"},
+        {
+            "role": "assistant",
+            "content": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+        },
+        {"role": "user", "content": "What does a Product Manager do?"},
+    ]
+
+
+def test_get_messages_from_history_truncated():
+    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+
+    messages = chat_approach.get_messages_from_history(
+        system_prompt="You are a bot.",
+        model_id="gpt-35-turbo",
+        history=[
+            {
+                "user": "What happens in a performance review?",
+                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+            },
+            {"user": "What does a Product Manager do?"},
+        ],
+        user_content="What does a Product Manager do?",
+        max_tokens=10,
+    )
+    assert messages == [
+        {"role": "system", "content": "You are a bot."},
+        {"role": "user", "content": "What does a Product Manager do?"},
+    ]
+
+
+def test_get_messages_from_history_truncated_longer():
+    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+
+    messages = chat_approach.get_messages_from_history(
+        system_prompt="You are a bot.",
+        model_id="gpt-35-turbo",
+        history=[
+            {
+                "user": "What happens in a performance review?",
+                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+            },
+            {
+                "user": "Is there a dress code?",
+                "bot": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]",
+            },
+            {"user": "What does a Product Manager do?"},
+        ],
+        user_content="What does a Product Manager do?",
+        max_tokens=30,
+    )
+    assert messages == [
+        {"role": "system", "content": "You are a bot."},
+        {"role": "user", "content": "Is there a dress code?"},
+        {
+            "role": "assistant",
+            "content": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]",
+        },
+        {"role": "user", "content": "What does a Product Manager do?"},
+    ]
diff --git a/tests/test_messagebuilder.py b/tests/test_messagebuilder.py
index 6eaaad2..85a5497 100644
--- a/tests/test_messagebuilder.py
+++ b/tests/test_messagebuilder.py
@@ -22,3 +22,26 @@ def test_messagebuilder_append():
     ]
     assert builder.model == "gpt-35-turbo"
     assert builder.token_length == 17
+
+
+def test_messagebuilder_unicode():
+    builder = MessageBuilder("a\u0301", "gpt-35-turbo")
+    assert builder.messages == [
+        # 1 token, 1 token, 1 token, 1 token
+        {"role": "system", "content": "á"}
+    ]
+    assert builder.model == "gpt-35-turbo"
+    assert builder.token_length == 4
+
+
+def test_messagebuilder_unicode_append():
+    builder = MessageBuilder("a\u0301", "gpt-35-turbo")
+    builder.append_message("user", "a\u0301")
+    assert builder.messages == [
+        # 1 token, 1 token, 1 token, 1 token
+        {"role": "system", "content": "á"},
+        # 1 token, 1 token, 1 token, 1 token
+        {"role": "user", "content": "á"},
+    ]
+    assert builder.model == "gpt-35-turbo"
+    assert builder.token_length == 8