Repository: Azure-Samples/azure-search-openai-demo
Pull request #741 (2023-10-05 17:42:26)

Clarify roles
## Purpose

Clarifying based on discussion in https://github.com/Azure-Samples/azure-search-openai-demo/issues/4


## Does this introduce a breaking change?
<!-- Mark one with an "x". -->
```
[ ] Yes
[X] No
```

## Pull Request Type
What kind of change does this Pull Request introduce?

<!-- Please check the one that applies to this PR using "x". -->
```
[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[X] Documentation content changes
[ ] Other... Please describe:
```


charris-msft: The solution is ugly (not your fault), but the update looks good to me.

Patch:
diff --git a/README.md b/README.md
index 2ab7e05..afaabe3 100644
--- a/README.md
+++ b/README.md
@@ -52,7 +52,9 @@ The repo includes sample data so it's ready to try end to end. In this sample ap
 
 * **Azure account**. If you're new to Azure, [get an Azure account for free](https://azure.microsoft.com/free/cognitive-search/) and you'll get some free Azure credits to get started.
 * **Azure subscription with access enabled for the Azure OpenAI service**. You can request access with [this form](https://aka.ms/oaiapply). If your access request to Azure OpenAI service doesn't match the [acceptance criteria](https://learn.microsoft.com/legal/cognitive-services/openai/limited-access?context=%2Fazure%2Fcognitive-services%2Fopenai%2Fcontext%2Fcontext), you can use [OpenAI public API](https://platform.openai.com/docs/api-reference/introduction) instead. Learn [how to switch to an OpenAI instance](#switching-from-an-azure-openai-endpoint-to-an-openai-instance).
-* **Azure account permissions**: Your Azure account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [Role Based Access Control Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview), [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator), or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner). If you don't have subscription-level permissions, you must be granted [RBAC](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview) for an existing resource group and [deploy to that existing group](#existing-resource-group).
+* **Azure account permissions**:
+  * Your Azure account must have `Microsoft.Authorization/roleAssignments/write` permissions, such as [Role Based Access Control Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview), [User Access Administrator](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#user-access-administrator), or [Owner](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#owner). If you don't have subscription-level permissions, you must be granted [RBAC](https://learn.microsoft.com/azure/role-based-access-control/built-in-roles#role-based-access-control-administrator-preview) for an existing resource group and [deploy to that existing group](#existing-resource-group).
+  * Your Azure account also needs `Microsoft.Resources/deployments/write` permissions on the subscription level.
 
 ## Azure deployment
 
@@ -136,11 +138,29 @@ If you already have existing Azure resources, you can re-use those by setting `a
 
 #### Existing OpenAI resource
 
+##### Azure OpenAI:
+
 1. Run `azd env set AZURE_OPENAI_SERVICE {Name of existing OpenAI service}`
 1. Run `azd env set AZURE_OPENAI_RESOURCE_GROUP {Name of existing resource group that OpenAI service is provisioned to}`
 1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing ChatGPT deployment}`. Only needed if your ChatGPT deployment is not the default 'chat'.
 1. Run `azd env set AZURE_OPENAI_EMB_DEPLOYMENT {Name of existing GPT embedding deployment}`. Only needed if your embeddings deployment is not the default 'embedding'.
 
+When you run `azd up` after and are prompted to select a value for `openAiResourceGroupLocation`, make sure to select the same location as the existing OpenAI resource group.
+
+##### Openai.com OpenAI:
+
+1. Run `azd env set OPENAI_HOST openai`
+2. Run `azd env set OPENAI_ORGANIZATION {Your OpenAI organization}`
+3. Run `azd env set OPENAI_API_KEY {Your OpenAI API key}`
+4. Run `azd up`
+
+You can retrieve your OpenAI key by checking [your user page](https://platform.openai.com/account/api-keys) and your organization by navigating to [your organization page](https://platform.openai.com/account/org-settings).
+Learn more about creating an OpenAI free trial at [this link](https://openai.com/pricing).
+Do *not* check your key into source control.
+
+When you run `azd up` after and are prompted to select a value for `openAiResourceGroupLocation`, you can select any location as it will not be used.
+
+
 #### Existing Azure Cognitive Search resource
 
 1. Run `azd env set AZURE_SEARCH_SERVICE {Name of existing Azure Cognitive Search service}`
@@ -148,6 +168,8 @@ If you already have existing Azure resources, you can re-use those by setting `a
 1. If that resource group is in a different location than the one you'll pick for the `azd up` step,
   then run `azd env set AZURE_SEARCH_SERVICE_LOCATION {Location of existing service}`
 1. If the search service's SKU is not standard, then run `azd env set AZURE_SEARCH_SERVICE_SKU {Name of SKU}`. The free tier won't work as it doesn't support managed identity. ([See other possible values](https://learn.microsoft.com/azure/templates/microsoft.search/searchservices?pivots=deployment-language-bicep#sku))
+1. If you have an existing index that is set up with all the expected fields, then run `azd env set AZURE_SEARCH_INDEX {Name of existing index}`. Otherwise, the `azd up` command will create a new index.
+
 
 #### Other existing Azure resources
 
@@ -184,20 +206,6 @@ either you or they can follow these steps:
 
 ## Enabling optional features
 
-#### Using a non-Azure OpenAI instance
-
-To use an existing non-Azure OpenAI account, follow these steps before running `azd up`
-
-1. Run `azd env set OPENAI_HOST openai`
-2. Run `azd env set OPENAI_ORGANIZATION {Your OpenAI organization}`
-3. Run `azd env set OPENAI_API_KEY {Your OpenAI API key}`
-4. Run `azd up`
-
-You can retrieve your OpenAI key by checking [your user page](https://platform.openai.com/account/api-keys) and your organization by navigating to [your organization page](https://platform.openai.com/account/org-settings).
-Learn more about creating an OpenAI free trial at [this link](https://openai.com/pricing).
-Do *not* check your key into source control.
-
-
 ### Enabling Application Insights
 
 To enable Application Insights and the tracing of each request, along with the logging of errors, set the `AZURE_USE_APPLICATION_INSIGHTS` variable to true before running `azd up`
diff --git a/app/backend/approaches/chatreadretrieveread.py b/app/backend/approaches/chatreadretrieveread.py
index d4037e1..b6efd60 100644
--- a/app/backend/approaches/chatreadretrieveread.py
+++ b/app/backend/approaches/chatreadretrieveread.py
@@ -228,9 +228,8 @@ If you cannot generate a search query, return just the number 0.
             history, overrides, auth_claims, should_stream=False
         )
         chat_resp = await chat_coroutine
-        chat_content = chat_resp.choices[0].message.content
-        extra_info["answer"] = chat_content
-        return extra_info
+        chat_resp.choices[0]["extra_args"] = extra_info
+        return chat_resp
 
     async def run_with_streaming(
         self, history: list[dict[str, str]], overrides: dict[str, Any], auth_claims: dict[str, Any]
@@ -238,7 +237,13 @@ If you cannot generate a search query, return just the number 0.
         extra_info, chat_coroutine = await self.run_until_final_call(
             history, overrides, auth_claims, should_stream=True
         )
-        yield extra_info
+        yield {
+            "choices": [
+                {"delta": {"role": self.ASSISTANT}, "extra_args": extra_info, "finish_reason": None, "index": 0}
+            ],
+            "object": "chat.completion.chunk",
+        }
+
         async for event in await chat_coroutine:
             # "2023-07-01-preview" API version has a bug where first response has empty choices
             if event["choices"]:
@@ -249,7 +254,7 @@ If you cannot generate a search query, return just the number 0.
         system_prompt: str,
         model_id: str,
         history: list[dict[str, str]],
-        user_conv: str,
+        user_content: str,
         few_shots=[],
         max_tokens: int = 4096,
     ) -> list:
@@ -259,21 +264,19 @@ If you cannot generate a search query, return just the number 0.
         for shot in few_shots:
             message_builder.append_message(shot.get("role"), shot.get("content"))
 
-        user_content = user_conv
         append_index = len(few_shots) + 1
 
         message_builder.append_message(self.USER, user_content, index=append_index)
 
         for h in reversed(history[:-1]):
+            if message_builder.token_length > max_tokens:
+                break
             if bot_msg := h.get("bot"):
                 message_builder.append_message(self.ASSISTANT, bot_msg, index=append_index)
             if user_msg := h.get("user"):
                 message_builder.append_message(self.USER, user_msg, index=append_index)
-            if message_builder.token_length > max_tokens:
-                break
 
-        messages = message_builder.messages
-        return messages
+        return message_builder.messages
 
     def get_search_query(self, chat_completion: dict[str, any], user_query: str):
         response_message = chat_completion["choices"][0]["message"]
diff --git a/app/backend/approaches/retrievethenread.py b/app/backend/approaches/retrievethenread.py
index 668a52f..3f6c85a 100644
--- a/app/backend/approaches/retrievethenread.py
+++ b/app/backend/approaches/retrievethenread.py
@@ -131,9 +131,10 @@ info4.pdf: In-network institutions include Overlake, Swedish and others in the r
             n=1,
         )
 
-        return {
+        extra_info = {
             "data_points": results,
-            "answer": chat_completion.choices[0].message.content,
             "thoughts": f"Question:<br>{query_text}<br><br>Prompt:<br>"
             + "\n\n".join([str(message) for message in messages]),
         }
+        chat_completion.choices[0]["extra_args"] = extra_info
+        return chat_completion
diff --git a/app/backend/core/messagebuilder.py b/app/backend/core/messagebuilder.py
index dc21961..9409ae7 100644
--- a/app/backend/core/messagebuilder.py
+++ b/app/backend/core/messagebuilder.py
@@ -1,3 +1,5 @@
+import unicodedata
+
 from .modelhelper import num_tokens_from_messages
 
 
@@ -14,10 +16,13 @@ class MessageBuilder:
     """
 
     def __init__(self, system_content: str, chatgpt_model: str):
-        self.messages = [{"role": "system", "content": system_content}]
+        self.messages = [{"role": "system", "content": self.normalize_content(system_content)}]
         self.model = chatgpt_model
         self.token_length = num_tokens_from_messages(self.messages[-1], self.model)
 
     def append_message(self, role: str, content: str, index: int = 1):
-        self.messages.insert(index, {"role": role, "content": content})
+        self.messages.insert(index, {"role": role, "content": self.normalize_content(content)})
         self.token_length += num_tokens_from_messages(self.messages[index], self.model)
+
+    def normalize_content(self, content: str):
+        return unicodedata.normalize("NFC", content)
diff --git a/app/frontend/package-lock.json b/app/frontend/package-lock.json
index 070fd73..24e564f 100644
--- a/app/frontend/package-lock.json
+++ b/app/frontend/package-lock.json
@@ -8,7 +8,8 @@
       "name": "frontend",
       "version": "0.0.0",
       "dependencies": {
-        "@azure/msal-react": "^2.0.2",
+        "@azure/msal-browser": "^3.1.0",
+        "@azure/msal-react": "^2.0.4",
         "@fluentui/react": "^8.110.7",
         "@fluentui/react-components": "^9.33.0",
         "@fluentui/react-icons": "^2.0.206",
@@ -46,30 +47,28 @@
       }
     },
     "node_modules/@azure/msal-browser": {
-      "version": "3.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-3.0.2.tgz",
-      "integrity": "sha512-a2lIzWGV8EJhxVTHu2Vd5sRdK6DTisueq2HCXt49CQqc1hG/j9pK+ds+hJi+QLlZHPo3FPMwiTCHEA9sNgX3HA==",
-      "peer": true,
+      "version": "3.2.0",
+      "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-3.2.0.tgz",
+      "integrity": "sha512-le2qutddMiq0i3ErQaLKuwP1DpNgdd9iXPs3fSCsLuBrdGg9B4/j4ArCAHCwgxA82Ydj9BcqtMIL5BSWwU+P5A==",
       "dependencies": {
-        "@azure/msal-common": "14.0.2"
+        "@azure/msal-common": "14.1.0"
       },
       "engines": {
         "node": ">=0.8.0"
       }
     },
     "node_modules/@azure/msal-common": {
-      "version": "14.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-14.0.2.tgz",
-      "integrity": "sha512-yvVP63oXe20JFzvooTVxiX9UorxcL8VFoVwEtKuGlQnEnNGzrOpQZ6lKa1lqwGqxMtwKpW7oAFQhbbskG5RbYA==",
-      "peer": true,
+      "version": "14.1.0",
+      "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-14.1.0.tgz",
+      "integrity": "sha512-xphmhcfl5VL+uq5//VKMwQn+wfEZLMKNpFCcMi8Ur8ej5UT166g6chBsxgMzc9xo9Y24R9FB3m/tjDiV03xMIA==",
       "engines": {
         "node": ">=0.8.0"
       }
     },
     "node_modules/@azure/msal-react": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-react/-/msal-react-2.0.2.tgz",
-      "integrity": "sha512-EQn735OKtnxYf9JbQX94UG+Op14MugW379pI+gbKmcK8s5CaK2lClb9Zr+fA2isL3MOj8rOQsGGXSaIx9lSsAQ==",
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/@azure/msal-react/-/msal-react-2.0.4.tgz",
+      "integrity": "sha512-BujRm5FBDWYXyr3pnmubS4dIqZMlurYVtV2AyztoeAFUd+nh3XQZD9knHBqTyu53IDjhCCvUPUke/jSkv5WGlg==",
       "dependencies": {
         "@rollup/plugin-typescript": "^11.1.0",
         "rollup": "^3.20.2"
@@ -78,7 +77,7 @@
         "node": ">=10"
       },
       "peerDependencies": {
-        "@azure/msal-browser": "^3.0.2",
+        "@azure/msal-browser": "^3.2.0",
         "react": "^16.8.0 || ^17 || ^18"
       }
     },
@@ -3600,24 +3599,22 @@
       }
     },
     "@azure/msal-browser": {
-      "version": "3.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-3.0.2.tgz",
-      "integrity": "sha512-a2lIzWGV8EJhxVTHu2Vd5sRdK6DTisueq2HCXt49CQqc1hG/j9pK+ds+hJi+QLlZHPo3FPMwiTCHEA9sNgX3HA==",
-      "peer": true,
+      "version": "3.2.0",
+      "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-3.2.0.tgz",
+      "integrity": "sha512-le2qutddMiq0i3ErQaLKuwP1DpNgdd9iXPs3fSCsLuBrdGg9B4/j4ArCAHCwgxA82Ydj9BcqtMIL5BSWwU+P5A==",
       "requires": {
-        "@azure/msal-common": "14.0.2"
+        "@azure/msal-common": "14.1.0"
       }
     },
     "@azure/msal-common": {
-      "version": "14.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-14.0.2.tgz",
-      "integrity": "sha512-yvVP63oXe20JFzvooTVxiX9UorxcL8VFoVwEtKuGlQnEnNGzrOpQZ6lKa1lqwGqxMtwKpW7oAFQhbbskG5RbYA==",
-      "peer": true
+      "version": "14.1.0",
+      "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-14.1.0.tgz",
+      "integrity": "sha512-xphmhcfl5VL+uq5//VKMwQn+wfEZLMKNpFCcMi8Ur8ej5UT166g6chBsxgMzc9xo9Y24R9FB3m/tjDiV03xMIA=="
     },
     "@azure/msal-react": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/@azure/msal-react/-/msal-react-2.0.2.tgz",
-      "integrity": "sha512-EQn735OKtnxYf9JbQX94UG+Op14MugW379pI+gbKmcK8s5CaK2lClb9Zr+fA2isL3MOj8rOQsGGXSaIx9lSsAQ==",
+      "version": "2.0.4",
+      "resolved": "https://registry.npmjs.org/@azure/msal-react/-/msal-react-2.0.4.tgz",
+      "integrity": "sha512-BujRm5FBDWYXyr3pnmubS4dIqZMlurYVtV2AyztoeAFUd+nh3XQZD9knHBqTyu53IDjhCCvUPUke/jSkv5WGlg==",
       "requires": {
         "@rollup/plugin-typescript": "^11.1.0",
         "rollup": "^3.20.2"
diff --git a/app/frontend/package.json b/app/frontend/package.json
index 2ec0e22..281764a 100644
--- a/app/frontend/package.json
+++ b/app/frontend/package.json
@@ -12,7 +12,8 @@
     "watch": "tsc && vite build --watch"
   },
   "dependencies": {
-    "@azure/msal-react": "^2.0.2",
+    "@azure/msal-react": "^2.0.4",
+    "@azure/msal-browser": "^3.1.0",
     "@fluentui/react": "^8.110.7",
     "@fluentui/react-components": "^9.33.0",
     "@fluentui/react-icons": "^2.0.206",
diff --git a/app/frontend/src/api/api.ts b/app/frontend/src/api/api.ts
index 290dc60..cf97ac5 100644
--- a/app/frontend/src/api/api.ts
+++ b/app/frontend/src/api/api.ts
@@ -1,6 +1,6 @@
 const BACKEND_URI = "";
 
-import { AskRequest, AskResponse, ChatRequest } from "./models";
+import { AskRequest, ChatAppResponse, ChatAppResponseOrError, ChatRequest } from "./models";
 import { useLogin } from "../authConfig";
 
 function getHeaders(idToken: string | undefined): Record<string, string> {
@@ -17,7 +17,7 @@ function getHeaders(idToken: string | undefined): Record<string, string> {
     return headers;
 }
 
-export async function askApi(options: AskRequest): Promise<AskResponse> {
+export async function askApi(options: AskRequest): Promise<ChatAppResponse> {
     const response = await fetch(`${BACKEND_URI}/ask`, {
         method: "POST",
         headers: getHeaders(options.idToken),
@@ -39,12 +39,12 @@ export async function askApi(options: AskRequest): Promise<AskResponse> {
         })
     });
 
-    const parsedResponse: AskResponse = await response.json();
+    const parsedResponse: ChatAppResponseOrError = await response.json();
     if (response.status > 299 || !response.ok) {
         throw Error(parsedResponse.error || "Unknown error");
     }
 
-    return parsedResponse;
+    return parsedResponse as ChatAppResponse;
 }
 
 export async function chatApi(options: ChatRequest): Promise<Response> {
diff --git a/app/frontend/src/api/models.ts b/app/frontend/src/api/models.ts
index 5a2f9e8..002e3df 100644
--- a/app/frontend/src/api/models.ts
+++ b/app/frontend/src/api/models.ts
@@ -25,13 +25,31 @@ export type AskRequest = {
     idToken?: string;
 };
 
-export type AskResponse = {
-    answer: string;
+export type ResponseMessage = {
+    content: string;
+    role: string;
+}
+
+export type ResponseExtraArgs = {
     thoughts: string | null;
     data_points: string[];
+}
+
+export type ResponseChoice = {
+    index: number;
+    message: ResponseMessage;
+    extra_args: ResponseExtraArgs;
+};
+
+export type ChatAppResponseOrError = {
+    choices?: ResponseChoice[];
     error?: string;
 };
 
+export type ChatAppResponse = {
+    choices: ResponseChoice[];
+};
+
 export type ChatTurn = {
     user: string;
     bot?: string;
diff --git a/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx b/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
index cd01a08..8f54582 100644
--- a/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
+++ b/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
@@ -4,7 +4,7 @@ import DOMPurify from "dompurify";
 import styles from "./AnalysisPanel.module.css";
 
 import { SupportingContent } from "../SupportingContent";
-import { AskResponse } from "../../api";
+import { ChatAppResponse } from "../../api";
 import { AnalysisPanelTabs } from "./AnalysisPanelTabs";
 
 interface Props {
@@ -13,17 +13,17 @@ interface Props {
     onActiveTabChanged: (tab: AnalysisPanelTabs) => void;
     activeCitation: string | undefined;
     citationHeight: string;
-    answer: AskResponse;
+    answer: ChatAppResponse;
 }
 
 const pivotItemDisabledStyle = { disabled: true, style: { color: "grey" } };
 
 export const AnalysisPanel = ({ answer, activeTab, activeCitation, citationHeight, className, onActiveTabChanged }: Props) => {
-    const isDisabledThoughtProcessTab: boolean = !answer.thoughts;
-    const isDisabledSupportingContentTab: boolean = !answer.data_points.length;
+    const isDisabledThoughtProcessTab: boolean = !answer.choices[0].extra_args.thoughts;
+    const isDisabledSupportingContentTab: boolean = !answer.choices[0].extra_args.data_points.length;
     const isDisabledCitationTab: boolean = !activeCitation;
 
-    const sanitizedThoughts = DOMPurify.sanitize(answer.thoughts!);
+    const sanitizedThoughts = DOMPurify.sanitize(answer.choices[0].extra_args.thoughts!);
 
     return (
         <Pivot
@@ -43,7 +43,7 @@ export const AnalysisPanel = ({ answer, activeTab, activeCitation, citationHeigh
                 headerText="Supporting content"
                 headerButtonProps={isDisabledSupportingContentTab ? pivotItemDisabledStyle : undefined}
             >
-                <SupportingContent supportingContent={answer.data_points} />
+                <SupportingContent supportingContent={answer.choices[0].extra_args.data_points} />
             </PivotItem>
             <PivotItem
                 itemKey={AnalysisPanelTabs.CitationTab}
diff --git a/app/frontend/src/components/Answer/Answer.tsx b/app/frontend/src/components/Answer/Answer.tsx
index 7444b26..4d9cc95 100644
--- a/app/frontend/src/components/Answer/Answer.tsx
+++ b/app/frontend/src/components/Answer/Answer.tsx
@@ -4,12 +4,12 @@ import DOMPurify from "dompurify";
 
 import styles from "./Answer.module.css";
 
-import { AskResponse, getCitationFilePath } from "../../api";
+import { ChatAppResponse, getCitationFilePath } from "../../api";
 import { parseAnswerToHtml } from "./AnswerParser";
 import { AnswerIcon } from "./AnswerIcon";
 
 interface Props {
-    answer: AskResponse;
+    answer: ChatAppResponse;
     isSelected?: boolean;
     isStreaming: boolean;
     onCitationClicked: (filePath: string) => void;
@@ -29,7 +29,8 @@ export const Answer = ({
     onFollowupQuestionClicked,
     showFollowupQuestions
 }: Props) => {
-    const parsedAnswer = useMemo(() => parseAnswerToHtml(answer.answer, isStreaming, onCitationClicked ), [answer]);
+    const messageContent = answer.choices[0].message.content;
+    const parsedAnswer = useMemo(() => parseAnswerToHtml(messageContent, isStreaming, onCitationClicked ), [answer]);
 
     const sanitizedAnswerHtml = DOMPurify.sanitize(parsedAnswer.answerHtml);
 
@@ -45,7 +46,7 @@ export const Answer = ({
                             title="Show thought process"
                             ariaLabel="Show thought process"
                             onClick={() => onThoughtProcessClicked()}
-                            disabled={!answer.thoughts}
+                            disabled={!answer.choices[0].extra_args.thoughts?.length}
                         />
                         <IconButton
                             style={{ color: "black" }}
@@ -53,7 +54,7 @@ export const Answer = ({
                             title="Show supporting content"
                             ariaLabel="Show supporting content"
                             onClick={() => onSupportingContentClicked()}
-                            disabled={!answer.data_points?.length}
+                            disabled={!answer.choices[0].extra_args.data_points?.length}
                         />
                     </div>
                 </Stack>
diff --git a/app/frontend/src/pages/chat/Chat.tsx b/app/frontend/src/pages/chat/Chat.tsx
index 7fd6ef3..ed33139 100644
--- a/app/frontend/src/pages/chat/Chat.tsx
+++ b/app/frontend/src/pages/chat/Chat.tsx
@@ -5,7 +5,7 @@ import readNDJSONStream from "ndjson-readablestream";
 
 import styles from "./Chat.module.css";
 
-import { chatApi, RetrievalMode, AskResponse, ChatRequest, ChatTurn } from "../../api";
+import { chatApi, RetrievalMode, ChatAppResponse, ChatAppResponseOrError, ChatRequest, ChatTurn } from "../../api";
 import { Answer, AnswerError, AnswerLoading } from "../../components/Answer";
 import { QuestionInput } from "../../components/QuestionInput";
 import { ExampleList } from "../../components/Example";
@@ -41,18 +41,24 @@ const Chat = () => {
     const [activeAnalysisPanelTab, setActiveAnalysisPanelTab] = useState<AnalysisPanelTabs | undefined>(undefined);
 
     const [selectedAnswer, setSelectedAnswer] = useState<number>(0);
-    const [answers, setAnswers] = useState<[user: string, response: AskResponse][]>([]);
-    const [streamedAnswers, setStreamedAnswers] = useState<[user: string, response: AskResponse][]>([]);
+    const [answers, setAnswers] = useState<[user: string, response: ChatAppResponse][]>([]);
+    const [streamedAnswers, setStreamedAnswers] = useState<[user: string, response: ChatAppResponse][]>([]);
 
-    const handleAsyncRequest = async (question: string, answers: [string, AskResponse][], setAnswers: Function, responseBody: ReadableStream<any>) => {
+    const handleAsyncRequest = async (question: string, answers: [string, ChatAppResponse][], setAnswers: Function, responseBody: ReadableStream<any>) => {
         let answer: string = "";
-        let askResponse: AskResponse = {} as AskResponse;
+        let askResponse: ChatAppResponse = {} as ChatAppResponse;
 
         const updateState = (newContent: string) => {
             return new Promise(resolve => {
                 setTimeout(() => {
                     answer += newContent;
-                    const latestResponse: AskResponse = { ...askResponse, answer };
+                    const latestResponse: ChatAppResponse = { ...askResponse,
+                        choices: [{ ...askResponse.choices[0],
+                                    message: { content: answer,
+                                        role: askResponse.choices[0].message.role
+                                    } }
+                                ]
+                    };
                     setStreamedAnswers([...answers, [question, latestResponse]]);
                     resolve(null);
                 }, 33);
@@ -61,7 +67,8 @@ const Chat = () => {
         try {
             setIsStreaming(true);
             for await (const event of readNDJSONStream(responseBody)) {
-                if (event["data_points"]) {
+                if (event["choices"] && event["choices"][0]["extra_args"] && event["choices"][0]["extra_args"]["data_points"]) {
+                    event["choices"][0]["message"] = event["choices"][0]["delta"];
                     askResponse = event;
                 } else if (event["choices"] && event["choices"][0]["delta"]["content"]) {
                     setIsLoading(false);
@@ -71,7 +78,13 @@ const Chat = () => {
         } finally {
             setIsStreaming(false);
         }
-        const fullResponse: AskResponse = { ...askResponse, answer };
+        const fullResponse: ChatAppResponse = { ...askResponse,
+            choices: [{ ...askResponse.choices[0],
+                        message: { content: answer,
+                            role: askResponse.choices[0].message.role
+                        } }
+                    ]
+        };
         return fullResponse;
     };
 
@@ -88,7 +101,10 @@ const Chat = () => {
         const token = client ? await getToken(client) : undefined;
 
         try {
-            const history: ChatTurn[] = answers.map(a => ({ user: a[0], bot: a[1].answer }));
+            const history: ChatTurn[] = answers.map(a => ({
+                user: a[0],
+                bot: a[1].choices[0].message.content
+            }));
             const request: ChatRequest = {
                 history: [...history, { user: question, bot: undefined }],
                 shouldStream: shouldStream,
@@ -111,14 +127,14 @@ const Chat = () => {
                 throw Error("No response body");
             }
             if (shouldStream) {
-                const parsedResponse: AskResponse = await handleAsyncRequest(question, answers, setAnswers, response.body);
+                const parsedResponse: ChatAppResponse = await handleAsyncRequest(question, answers, setAnswers, response.body);
                 setAnswers([...answers, [question, parsedResponse]]);
             } else {
-                const parsedResponse: AskResponse = await response.json();
+                const parsedResponse: ChatAppResponseOrError = await response.json();
                 if (response.status > 299 || !response.ok) {
                     throw Error(parsedResponse.error || "Unknown error");
                 }
-                setAnswers([...answers, [question, parsedResponse]]);
+                setAnswers([...answers, [question, parsedResponse as ChatAppResponse]]);
             }
         } catch (e) {
             setError(e);
diff --git a/app/frontend/src/pages/oneshot/OneShot.tsx b/app/frontend/src/pages/oneshot/OneShot.tsx
index 447a562..18caf04 100644
--- a/app/frontend/src/pages/oneshot/OneShot.tsx
+++ b/app/frontend/src/pages/oneshot/OneShot.tsx
@@ -3,7 +3,7 @@ import { Checkbox, ChoiceGroup, IChoiceGroupOption, Panel, DefaultButton, Spinne
 
 import styles from "./OneShot.module.css";
 
-import { askApi, AskResponse, AskRequest, RetrievalMode } from "../../api";
+import { askApi, ChatAppResponse, AskRequest, RetrievalMode } from "../../api";
 import { Answer, AnswerError } from "../../components/Answer";
 import { QuestionInput } from "../../components/QuestionInput";
 import { ExampleList } from "../../components/Example";
@@ -30,7 +30,7 @@ export function Component(): JSX.Element {
 
     const [isLoading, setIsLoading] = useState<boolean>(false);
     const [error, setError] = useState<unknown>();
-    const [answer, setAnswer] = useState<AskResponse>();
+    const [answer, setAnswer] = useState<ChatAppResponse>();
 
     const [activeCitation, setActiveCitation] = useState<string>();
     const [activeAnalysisPanelTab, setActiveAnalysisPanelTab] = useState<AnalysisPanelTabs | undefined>(undefined);
diff --git a/infra/main.bicep b/infra/main.bicep
index 2452874..1ac2e72 100644
--- a/infra/main.bicep
+++ b/infra/main.bicep
@@ -58,7 +58,7 @@ param chatGptDeploymentName string // Set in main.parameters.json
 param chatGptDeploymentCapacity int = 30
 param chatGptModelName string = (openAiHost == 'azure') ? 'gpt-35-turbo' : 'gpt-3.5-turbo'
 param chatGptModelVersion string = '0613'
-param embeddingDeploymentName string = 'embedding'
+param embeddingDeploymentName string // Set in main.parameters.json
 param embeddingDeploymentCapacity int = 30
 param embeddingModelName string = 'text-embedding-ada-002'
 
diff --git a/infra/main.parameters.json b/infra/main.parameters.json
index 5cbcc19..d18a4b8 100644
--- a/infra/main.parameters.json
+++ b/infra/main.parameters.json
@@ -59,6 +59,9 @@
     "chatGptDeploymentName": {
       "value": "${AZURE_OPENAI_CHATGPT_DEPLOYMENT=chat}"
     },
+    "embeddingDeploymentName": {
+      "value": "${AZURE_OPENAI_EMB_DEPLOYMENT=embedding}"
+    },
     "openAiHost":{
       "value": "${OPENAI_HOST=azure}"
     },


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 206b98f..5f2c262 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -40,20 +40,18 @@ def mock_openai_embedding(monkeypatch):
 def mock_openai_chatcompletion(monkeypatch):
     class AsyncChatCompletionIterator:
         def __init__(self, answer):
-            self.num = 2
-            self.answer = answer
+            self.responses = [
+                {"object": "chat.completion.chunk", "choices": []},
+                {"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]},
+                {"object": "chat.completion.chunk", "choices": [{"delta": {"content": answer}}]},
+            ]
 
         def __aiter__(self):
             return self
 
         async def __anext__(self):
-            if self.num == 2:
-                self.num -= 1
-                # Emulate the first response being empty - bug with "2023-07-01-preview"
-                return openai.util.convert_to_openai_object({"choices": []})
-            elif self.num == 1:
-                self.num -= 1
-                return openai.util.convert_to_openai_object({"choices": [{"delta": {"content": self.answer}}]})
+            if self.responses:
+                return self.responses.pop(0)
             else:
                 raise StopAsyncIteration
 
@@ -70,7 +68,9 @@ def mock_openai_chatcompletion(monkeypatch):
         if "stream" in kwargs and kwargs["stream"] is True:
             return AsyncChatCompletionIterator(answer)
         else:
-            return openai.util.convert_to_openai_object({"choices": [{"message": {"content": answer}}]})
+            return openai.util.convert_to_openai_object(
+                {"object": "chat.completion", "choices": [{"message": {"role": "assistant", "content": answer}}]}
+            )
 
     monkeypatch.setattr(openai.ChatCompletion, "acreate", mock_acreate)
 
diff --git a/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
index 4379dc6..87ebda3 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
@@ -1,5 +1,15 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n '}"
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n '}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
+    ],
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
index 7778198..c400f9f 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
index 7778198..c400f9f 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
index 6e0acac..fba6d28 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_hybrid/client0/result.json b/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_hybrid/client1/result.json b/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json b/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
index b04d12b..2a92fc5 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': 'You are a cat.'}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': 'You are a cat.'}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json b/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
index b04d12b..2a92fc5 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': 'You are a cat.'}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': 'You are a cat.'}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json b/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
index b354871..2d64cd3 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n Meow like a cat.\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n Meow like a cat.\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json b/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
index b354871..2d64cd3 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n Meow like a cat.\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n Meow like a cat.\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
index 3cb1cce..4ad2e5c 100644
--- a/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
@@ -1,2 +1,3 @@
-{"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}
-{"choices": [{"delta": {"content": "The capital of France is Paris."}}]}
+{"choices": [{"delta": {"role": "assistant"}, "extra_args": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris."}}]}
diff --git a/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
index 3cb1cce..4ad2e5c 100644
--- a/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
@@ -1,2 +1,3 @@
-{"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}
-{"choices": [{"delta": {"content": "The capital of France is Paris."}}]}
+{"choices": [{"delta": {"role": "assistant"}, "extra_args": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris."}}]}
diff --git a/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
index 75e296a..cc6d520 100644
--- a/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
@@ -1,2 +1,3 @@
-{"data_points": [], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"}
-{"choices": [{"delta": {"content": "The capital of France is Paris."}}]}
+{"choices": [{"delta": {"role": "assistant"}, "extra_args": {"data_points": [], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris."}}]}
diff --git a/tests/snapshots/test_app/test_chat_text/client0/result.json b/tests/snapshots/test_app/test_chat_text/client0/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_text/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text/client1/result.json b/tests/snapshots/test_app/test_chat_text/client1/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_text/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json b/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
index 77d891a..ddcb422 100644
--- a/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
@@ -1,5 +1,15 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
+    ],
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
index 0ca3460..2ce4879 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
index 0ca3460..2ce4879 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: Caption: A whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: Caption: A whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json b/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json b/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
index ca1821d..8ab51eb 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_vector/client0/result.json b/tests/snapshots/test_app/test_chat_vector/client0/result.json
index 140529a..4117f95 100644
--- a/tests/snapshots/test_app/test_chat_vector/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_vector/client0/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>None<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>None<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_vector/client1/result.json b/tests/snapshots/test_app/test_chat_vector/client1/result.json
index 140529a..4117f95 100644
--- a/tests/snapshots/test_app/test_chat_vector/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_vector/client1/result.json
@@ -1,7 +1,17 @@
 {
-    "answer": "The capital of France is Paris.",
-    "data_points": [
-        "Benefit_Options-2.pdf: There is a whistleblower policy."
+    "choices": [
+        {
+            "extra_args": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>None<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris.",
+                "role": "assistant"
+            }
+        }
     ],
-    "thoughts": "Searched for:<br>None<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+    "object": "chat.completion"
 }
\ No newline at end of file
diff --git a/tests/test_chatapproach.py b/tests/test_chatapproach.py
index 5ce4145..4dc40d2 100644
--- a/tests/test_chatapproach.py
+++ b/tests/test_chatapproach.py
@@ -21,3 +21,82 @@ def test_get_search_query_returns_default():
     query = chat_approach.get_search_query(json.loads(payload), default_query)
 
     assert query == default_query
+
+
+def test_get_messages_from_history():
+    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+
+    messages = chat_approach.get_messages_from_history(
+        system_prompt="You are a bot.",
+        model_id="gpt-35-turbo",
+        history=[
+            {
+                "user": "What happens in a performance review?",
+                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+            },
+            {"user": "What does a Product Manager do?"},
+        ],
+        user_content="What does a Product Manager do?",
+    )
+    assert messages == [
+        {"role": "system", "content": "You are a bot."},
+        {"role": "user", "content": "What happens in a performance review?"},
+        {
+            "role": "assistant",
+            "content": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+        },
+        {"role": "user", "content": "What does a Product Manager do?"},
+    ]
+
+
+def test_get_messages_from_history_truncated():
+    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+
+    messages = chat_approach.get_messages_from_history(
+        system_prompt="You are a bot.",
+        model_id="gpt-35-turbo",
+        history=[
+            {
+                "user": "What happens in a performance review?",
+                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+            },
+            {"user": "What does a Product Manager do?"},
+        ],
+        user_content="What does a Product Manager do?",
+        max_tokens=10,
+    )
+    assert messages == [
+        {"role": "system", "content": "You are a bot."},
+        {"role": "user", "content": "What does a Product Manager do?"},
+    ]
+
+
+def test_get_messages_from_history_truncated_longer():
+    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+
+    messages = chat_approach.get_messages_from_history(
+        system_prompt="You are a bot.",
+        model_id="gpt-35-turbo",
+        history=[
+            {
+                "user": "What happens in a performance review?",
+                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+            },
+            {
+                "user": "Is there a dress code?",
+                "bot": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]",
+            },
+            {"user": "What does a Product Manager do?"},
+        ],
+        user_content="What does a Product Manager do?",
+        max_tokens=30,
+    )
+    assert messages == [
+        {"role": "system", "content": "You are a bot."},
+        {"role": "user", "content": "Is there a dress code?"},
+        {
+            "role": "assistant",
+            "content": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]",
+        },
+        {"role": "user", "content": "What does a Product Manager do?"},
+    ]
diff --git a/tests/test_messagebuilder.py b/tests/test_messagebuilder.py
index 6eaaad2..85a5497 100644
--- a/tests/test_messagebuilder.py
+++ b/tests/test_messagebuilder.py
@@ -22,3 +22,26 @@ def test_messagebuilder_append():
     ]
     assert builder.model == "gpt-35-turbo"
     assert builder.token_length == 17
+
+
+def test_messagebuilder_unicode():
+    builder = MessageBuilder("a\u0301", "gpt-35-turbo")
+    assert builder.messages == [
+        # 1 token, 1 token, 1 token, 1 token
+        {"role": "system", "content": "á"}
+    ]
+    assert builder.model == "gpt-35-turbo"
+    assert builder.token_length == 4
+
+
+def test_messagebuilder_unicode_append():
+    builder = MessageBuilder("a\u0301", "gpt-35-turbo")
+    builder.append_message("user", "a\u0301")
+    assert builder.messages == [
+        # 1 token, 1 token, 1 token, 1 token
+        {"role": "system", "content": "á"},
+        # 1 token, 1 token, 1 token, 1 token
+        {"role": "user", "content": "á"},
+    ]
+    assert builder.model == "gpt-35-turbo"
+    assert builder.token_length == 8