Repository: Azure-Samples/azure-search-openai-demo
Pull request #816 (2023-10-17 17:11:14)

Add front-matter needed for Sample browser
## Purpose

This sample is not currently discoverable in the Azure Samples browser:
https://learn.microsoft.com/en-us/samples/browse/

We'd like all the OpenAI chat sample apps to be indexed there, but that requires adding some YAML to README.md.

Instructions here:

https://review.learn.microsoft.com/en-us/help/contribute/samples/process/onboarding?branch=main#add-metadata-to-readme

Unfortunately, the YAML will get rendered in the README.md. We've given feedback to the Learn.com that we don't want to have it visible, but for now, we don't see a way to hide it.


## Does this introduce a breaking change?
<!-- Mark one with an "x". -->
```
[ ] Yes
[X] No
```

## Pull Request Type
What kind of change does this Pull Request introduce?

<!-- Please check the one that applies to this PR using "x". -->
```
[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[X] Documentation content changes
[ ] Other... Please describe:
```

## How to Test

* View readme


Patch:
diff --git a/.gitattributes b/.gitattributes
index 526c8a3..99f84ac 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,1 +1,2 @@
-*.sh text eol=lf
\ No newline at end of file
+*.sh text eol=lf
+*.jsonlines text eol=lf
diff --git a/.github/workflows/frontend.yaml b/.github/workflows/frontend.yaml
new file mode 100644
index 0000000..14d04cb
--- /dev/null
+++ b/.github/workflows/frontend.yaml
@@ -0,0 +1,18 @@
+name: Frontend linting
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+    prettier:
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Run prettier on frontend
+              run: |
+                cd ./app/frontend
+                npm install
+                npx prettier --check .
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 00e7f48..85af29b 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -14,3 +14,8 @@ repos:
     rev: 23.9.1
     hooks:
     -   id: black
+-   repo: https://github.com/pre-commit/mirrors-prettier
+    rev: v3.0.3
+    hooks:
+    - id: prettier
+      types_or: [css, javascript, ts, tsx, html]
diff --git a/README.md b/README.md
index a09d5e8..4cc545d 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,20 @@
+---
+name: ChatGPT + Enterprise data
+description: Chat with your data using OpenAI and Cognitive Search.
+languages:
+- azdeveloper
+- typescript
+- python
+- bicep
+products:
+- azure
+- azure-cognitive-search
+- azure-openai
+- azure-app-service
+page_type: sample
+urlFragment: azure-search-openai-demo
+---
+
 # ChatGPT + Enterprise data with Azure OpenAI and Cognitive Search
 
 ## Table of Contents
@@ -8,7 +25,7 @@
   - [Cost estimation](#cost-estimation)
   - [Project setup](#project-setup)
     - [GitHub Codespaces](#github-codespaces)
-    - [VS Code Remote Containers](#vs-code-remote-containers)
+    - [VS Code Dev Containers](#vs-code-dev-containers)
     - [Local environment](#local-environment)
   - [Deploying from scratch](#deploying-from-scratch)
   - [Deploying with existing Azure resources](#deploying-with-existing-azure-resources)
@@ -28,7 +45,7 @@
   - [Troubleshooting](#troubleshooting)
 
 [![Open in GitHub Codespaces](https://img.shields.io/static/v1?style=for-the-badge&label=GitHub+Codespaces&message=Open&color=brightgreen&logo=github)](https://github.com/codespaces/new?hide_repo_select=true&ref=main&repo=599293758&machine=standardLinux32gb&devcontainer_path=.devcontainer%2Fdevcontainer.json&location=WestUs2)
-[![Open in Remote - Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Remote%20-%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/azure-samples/azure-search-openai-demo)
+[![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/azure-samples/azure-search-openai-demo)
 
 This sample demonstrates a few approaches for creating ChatGPT-like experiences over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access the ChatGPT model (gpt-35-turbo), and Azure Cognitive Search for data indexing and retrieval.
 
@@ -87,11 +104,16 @@ You can run this repo virtually by using GitHub Codespaces, which will open a we
 
 [![Open in GitHub Codespaces](https://img.shields.io/static/v1?style=for-the-badge&label=GitHub+Codespaces&message=Open&color=brightgreen&logo=github)](https://github.com/codespaces/new?hide_repo_select=true&ref=main&repo=599293758&machine=standardLinux32gb&devcontainer_path=.devcontainer%2Fdevcontainer.json&location=WestUs2)
 
-#### VS Code Remote Containers
+#### VS Code Dev Containers
 
-A related option is VS Code Remote Containers, which will open the project in your local VS Code using the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers):
+A related option is VS Code Dev Containers, which will open the project in your local VS Code using the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers):
 
-[![Open in Remote - Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Remote%20-%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/azure-samples/azure-search-openai-demo)
+1. Start Docker Desktop (install it if not already installed)
+1. Open the project:
+    [![Open in Dev Containers](https://img.shields.io/static/v1?style=for-the-badge&label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode)](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/azure-samples/azure-search-openai-demo)
+1. In the VS Code window that opens, once the project files show up (this may take several minutes), open a terminal window
+1. Run `azd auth login`
+1. Now you can follow the instructions in [Deploying from scratch](#deploying-from-scratch) below
 
 #### Local environment
 
diff --git a/app/backend/approaches/chatreadretrieveread.py b/app/backend/approaches/chatreadretrieveread.py
index 50abc1c..75da318 100644
--- a/app/backend/approaches/chatreadretrieveread.py
+++ b/app/backend/approaches/chatreadretrieveread.py
@@ -1,4 +1,5 @@
 import json
+import logging
 from typing import Any, AsyncGenerator, Optional, Union
 
 import aiohttp
@@ -113,12 +114,12 @@ If you cannot generate a search query, return just the number 0.
 
         # STEP 1: Generate an optimized keyword search query based on the chat history and the last question
         messages = self.get_messages_from_history(
-            self.query_prompt_template,
-            self.chatgpt_model,
-            history,
-            user_query_request,
-            self.query_prompt_few_shots,
-            self.chatgpt_token_limit - len(user_query_request),
+            system_prompt=self.query_prompt_template,
+            model_id=self.chatgpt_model,
+            history=history,
+            user_content=user_query_request,
+            max_tokens=self.chatgpt_token_limit - len(user_query_request),
+            few_shots=self.query_prompt_few_shots,
         )
 
         chatgpt_args = {"deployment_id": self.chatgpt_deployment} if self.openai_host == "azure" else {}
@@ -201,12 +202,15 @@ If you cannot generate a search query, return just the number 0.
         else:
             system_message = prompt_override.format(follow_up_questions_prompt=follow_up_questions_prompt)
 
+        response_token_limit = 1024
+        messages_token_limit = self.chatgpt_token_limit - response_token_limit
         messages = self.get_messages_from_history(
-            system_message,
-            self.chatgpt_model,
-            history,
-            original_user_query + "\n\nSources:\n" + content,
-            max_tokens=self.chatgpt_token_limit,  # Model does not handle lengthy system messages well. Moving sources to latest user conversation to solve follow up questions prompt.
+            system_prompt=system_message,
+            model_id=self.chatgpt_model,
+            history=history,
+            # Model does not handle lengthy system messages well. Moving sources to latest user conversation to solve follow up questions prompt.
+            user_content=original_user_query + "\n\nSources:\n" + content,
+            max_tokens=messages_token_limit,
         )
         msg_to_display = "\n\n".join([str(message) for message in messages])
 
@@ -221,7 +225,7 @@ If you cannot generate a search query, return just the number 0.
             model=self.chatgpt_model,
             messages=messages,
             temperature=overrides.get("temperature") or 0.7,
-            max_tokens=1024,
+            max_tokens=response_token_limit,
             n=1,
             stream=should_stream,
         )
@@ -290,8 +294,8 @@ If you cannot generate a search query, return just the number 0.
         model_id: str,
         history: list[dict[str, str]],
         user_content: str,
+        max_tokens: int,
         few_shots=[],
-        max_tokens: int = 4096,
     ) -> list:
         message_builder = MessageBuilder(system_prompt, model_id)
 
@@ -302,15 +306,16 @@ If you cannot generate a search query, return just the number 0.
         append_index = len(few_shots) + 1
 
         message_builder.append_message(self.USER, user_content, index=append_index)
+        total_token_count = message_builder.count_tokens_for_message(message_builder.messages[-1])
 
-        for h in reversed(history[:-1]):
-            if message_builder.token_length > max_tokens:
+        newest_to_oldest = list(reversed(history[:-1]))
+        for message in newest_to_oldest:
+            potential_message_count = message_builder.count_tokens_for_message(message)
+            if (total_token_count + potential_message_count) > max_tokens:
+                logging.debug("Reached max tokens of %d, history will be truncated", max_tokens)
                 break
-            if bot_msg := h.get("bot"):
-                message_builder.append_message(self.ASSISTANT, bot_msg, index=append_index)
-            if user_msg := h.get("user"):
-                message_builder.append_message(self.USER, user_msg, index=append_index)
-
+            message_builder.append_message(message["role"], message["content"], index=append_index)
+            total_token_count += potential_message_count
         return message_builder.messages
 
     def get_search_query(self, chat_completion: dict[str, Any], user_query: str):
diff --git a/app/backend/core/messagebuilder.py b/app/backend/core/messagebuilder.py
index 9409ae7..b071511 100644
--- a/app/backend/core/messagebuilder.py
+++ b/app/backend/core/messagebuilder.py
@@ -18,11 +18,12 @@ class MessageBuilder:
     def __init__(self, system_content: str, chatgpt_model: str):
         self.messages = [{"role": "system", "content": self.normalize_content(system_content)}]
         self.model = chatgpt_model
-        self.token_length = num_tokens_from_messages(self.messages[-1], self.model)
 
     def append_message(self, role: str, content: str, index: int = 1):
         self.messages.insert(index, {"role": role, "content": self.normalize_content(content)})
-        self.token_length += num_tokens_from_messages(self.messages[index], self.model)
+
+    def count_tokens_for_message(self, message: dict[str, str]):
+        return num_tokens_from_messages(message, self.model)
 
     def normalize_content(self, content: str):
         return unicodedata.normalize("NFC", content)
diff --git a/app/backend/requirements.in b/app/backend/requirements.in
index 85044a4..635191e 100644
--- a/app/backend/requirements.in
+++ b/app/backend/requirements.in
@@ -5,7 +5,7 @@ openai[datalib]
 tiktoken
 azure-search-documents==11.4.0b6
 azure-storage-blob
-uvicorn[standard]
+uvicorn
 aiohttp
 azure-monitor-opentelemetry
 opentelemetry-instrumentation-asgi
diff --git a/app/backend/requirements.txt b/app/backend/requirements.txt
index b5b2680..f6f4bdf 100644
--- a/app/backend/requirements.txt
+++ b/app/backend/requirements.txt
@@ -12,8 +12,6 @@ aiohttp==3.8.5
     #   openai
 aiosignal==1.3.1
     # via aiohttp
-anyio==3.7.1
-    # via watchfiles
 asgiref==3.7.2
     # via opentelemetry-instrumentation-asgi
 async-timeout==4.0.3
@@ -89,15 +87,12 @@ h2==4.1.0
     # via hypercorn
 hpack==4.0.0
     # via h2
-httptools==0.6.0
-    # via uvicorn
 hypercorn==0.14.4
     # via quart
 hyperframe==6.0.1
     # via h2
 idna==3.4
     # via
-    #   anyio
     #   requests
     #   yarl
 importlib-metadata==6.8.0
@@ -250,12 +245,8 @@ pyjwt[crypto]==2.8.0
     # via msal
 python-dateutil==2.8.2
     # via pandas
-python-dotenv==1.0.0
-    # via uvicorn
 pytz==2023.3.post1
     # via pandas
-pyyaml==6.0.1
-    # via uvicorn
 quart==0.19.3
     # via
     #   -r requirements.in
@@ -279,8 +270,6 @@ six==1.16.0
     #   azure-core
     #   isodate
     #   python-dateutil
-sniffio==1.3.0
-    # via anyio
 tiktoken==0.5.1
     # via -r requirements.in
 tqdm==4.66.1
@@ -294,16 +283,10 @@ typing-extensions==4.8.0
     #   opentelemetry-sdk
 tzdata==2023.3
     # via pandas
-urllib3==2.0.6
+urllib3==2.0.7
     # via requests
-uvicorn[standard]==0.23.2
+uvicorn==0.23.2
     # via -r requirements.in
-uvloop==0.17.0
-    # via uvicorn
-watchfiles==0.20.0
-    # via uvicorn
-websockets==11.0.3
-    # via uvicorn
 werkzeug==3.0.0
     # via
     #   flask
diff --git a/app/frontend/.prettierignore b/app/frontend/.prettierignore
new file mode 100644
index 0000000..fc355bc
--- /dev/null
+++ b/app/frontend/.prettierignore
@@ -0,0 +1,2 @@
+# Ignore JSON
+**/*.json
diff --git a/app/frontend/index.html b/app/frontend/index.html
index 4d96bfb..a7aa7e9 100644
--- a/app/frontend/index.html
+++ b/app/frontend/index.html
@@ -1,13 +1,13 @@
-<!DOCTYPE html>
+<!doctype html>
 <html lang="en">
-  <head>
-    <meta charset="UTF-8" />
-    <link rel="icon" type="image/x-icon" href="/favicon.ico" />
-    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>GPT + Enterprise data | Sample</title>
-  </head>
-  <body>
-    <div id="root"></div>
-    <script type="module" src="/src/index.tsx"></script>
-  </body>
+    <head>
+        <meta charset="UTF-8" />
+        <link rel="icon" type="image/x-icon" href="/favicon.ico" />
+        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+        <title>GPT + Enterprise data | Sample</title>
+    </head>
+    <body>
+        <div id="root"></div>
+        <script type="module" src="/src/index.tsx"></script>
+    </body>
 </html>
diff --git a/app/frontend/package-lock.json b/app/frontend/package-lock.json
index c4deeb9..b5a6e36 100644
--- a/app/frontend/package-lock.json
+++ b/app/frontend/package-lock.json
@@ -18,7 +18,8 @@
         "ndjson-readablestream": "^1.0.7",
         "react": "^18.2.0",
         "react-dom": "^18.2.0",
-        "react-router-dom": "^6.16.0"
+        "react-router-dom": "^6.16.0",
+        "scheduler": "^0.20.2"
       },
       "devDependencies": {
         "@types/dompurify": "^3.0.3",
@@ -3066,7 +3067,6 @@
       "version": "0.20.2",
       "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.20.2.tgz",
       "integrity": "sha512-2eWfGgAqqWFGqtdMmcL5zCMK1U8KlXv8SQFGglL3CEtd0aDVDWgeF/YoCmvln55m5zSk3J/20hTaSBeSObsQDQ==",
-      "peer": true,
       "dependencies": {
         "loose-envify": "^1.1.0",
         "object-assign": "^4.1.1"
@@ -5395,7 +5395,6 @@
       "version": "0.20.2",
       "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.20.2.tgz",
       "integrity": "sha512-2eWfGgAqqWFGqtdMmcL5zCMK1U8KlXv8SQFGglL3CEtd0aDVDWgeF/YoCmvln55m5zSk3J/20hTaSBeSObsQDQ==",
-      "peer": true,
       "requires": {
         "loose-envify": "^1.1.0",
         "object-assign": "^4.1.1"
diff --git a/app/frontend/package.json b/app/frontend/package.json
index 8a533ee..d5b90ff 100644
--- a/app/frontend/package.json
+++ b/app/frontend/package.json
@@ -22,7 +22,8 @@
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
     "react-router-dom": "^6.16.0",
-    "ndjson-readablestream": "^1.0.7"
+    "ndjson-readablestream": "^1.0.7",
+    "scheduler": "^0.20.2"
   },
   "devDependencies": {
     "@types/dompurify": "^3.0.3",
diff --git a/app/frontend/src/api/api.ts b/app/frontend/src/api/api.ts
index a12f8f3..f2fe048 100644
--- a/app/frontend/src/api/api.ts
+++ b/app/frontend/src/api/api.ts
@@ -4,13 +4,13 @@ import { ChatAppResponse, ChatAppResponseOrError, ChatAppRequest } from "./model
 import { useLogin } from "../authConfig";
 
 function getHeaders(idToken: string | undefined): Record<string, string> {
-    var headers : Record<string, string> = {
+    var headers: Record<string, string> = {
         "Content-Type": "application/json"
     };
     // If using login, add the id token of the logged in account as the authorization
     if (useLogin) {
         if (idToken) {
-            headers["Authorization"] = `Bearer ${idToken}`
+            headers["Authorization"] = `Bearer ${idToken}`;
         }
     }
 
diff --git a/app/frontend/src/api/models.ts b/app/frontend/src/api/models.ts
index 3cc3f65..d66f23d 100644
--- a/app/frontend/src/api/models.ts
+++ b/app/frontend/src/api/models.ts
@@ -22,12 +22,12 @@ export type ChatAppRequestOverrides = {
 export type ResponseMessage = {
     content: string;
     role: string;
-}
+};
 
 export type ResponseContext = {
     thoughts: string | null;
     data_points: string[];
-}
+};
 
 export type ResponseChoice = {
     index: number;
@@ -47,7 +47,7 @@ export type ChatAppResponse = {
 
 export type ChatAppRequestContext = {
     overrides?: ChatAppRequestOverrides;
-}
+};
 
 export type ChatAppRequest = {
     messages: ResponseMessage[];
diff --git a/app/frontend/src/authConfig.ts b/app/frontend/src/authConfig.ts
index 37326c3..bb4f5cc 100644
--- a/app/frontend/src/authConfig.ts
+++ b/app/frontend/src/authConfig.ts
@@ -12,17 +12,17 @@ interface AuthSetup {
      */
     msalConfig: {
         auth: {
-            clientId: string, // Client app id used for login
-            authority: string, // Directory to use for login https://learn.microsoft.com/azure/active-directory/develop/msal-client-application-configuration#authority
-            redirectUri: string, // Points to window.location.origin. You must register this URI on Azure Portal/App Registration.
-            postLogoutRedirectUri: string, // Indicates the page to navigate after logout.
-            navigateToLoginRequestUrl: boolean // If "true", will navigate back to the original request location before processing the auth code response.
-        },
+            clientId: string; // Client app id used for login
+            authority: string; // Directory to use for login https://learn.microsoft.com/azure/active-directory/develop/msal-client-application-configuration#authority
+            redirectUri: string; // Points to window.location.origin. You must register this URI on Azure Portal/App Registration.
+            postLogoutRedirectUri: string; // Indicates the page to navigate after logout.
+            navigateToLoginRequestUrl: boolean; // If "true", will navigate back to the original request location before processing the auth code response.
+        };
         cache: {
-            cacheLocation: string, // Configures cache location. "sessionStorage" is more secure, but "localStorage" gives you SSO between tabs.
-            storeAuthStateInCookie: boolean // Set this to "true" if you are having issues on IE11 or Edge
-        }
-    },
+            cacheLocation: string; // Configures cache location. "sessionStorage" is more secure, but "localStorage" gives you SSO between tabs.
+            storeAuthStateInCookie: boolean; // Set this to "true" if you are having issues on IE11 or Edge
+        };
+    };
     loginRequest: {
         /**
          * Scopes you add here will be prompted for user consent during sign-in.
@@ -30,16 +30,16 @@ interface AuthSetup {
          * For more information about OIDC scopes, visit:
          * https://docs.microsoft.com/azure/active-directory/develop/v2-permissions-and-consent#openid-connect-scopes
          */
-        scopes: Array<string>
-    },
+        scopes: Array<string>;
+    };
     tokenRequest: {
-        scopes: Array<string>
-    }
+        scopes: Array<string>;
+    };
 }
 
 // Fetch the auth setup JSON data from the API if not already cached
 async function fetchAuthSetup(): Promise<AuthSetup> {
-    const response = await fetch('/auth_setup');
+    const response = await fetch("/auth_setup");
     if (!response.ok) {
         throw new Error(`auth setup response was not ok: ${response.status}`);
     }
@@ -69,18 +69,19 @@ const tokenRequest = authSetup.tokenRequest;
 
 // Build an absolute redirect URI using the current window's location and the relative redirect URI from auth setup
 export const getRedirectUri = () => {
-    return window.location.origin + authSetup.msalConfig.auth.redirectUri
-}
+    return window.location.origin + authSetup.msalConfig.auth.redirectUri;
+};
 
 // Get an access token for use with the API server.
 // ID token received when logging in may not be used for this purpose because it has the incorrect audience
 export const getToken = (client: IPublicClientApplication): Promise<AuthenticationResult | undefined> => {
-    return client.acquireTokenSilent({
-        ...tokenRequest,
-        redirectUri: getRedirectUri()
-    })
-    .catch((error) => {
-        console.log(error);
-        return undefined;
-    })
-}
+    return client
+        .acquireTokenSilent({
+            ...tokenRequest,
+            redirectUri: getRedirectUri()
+        })
+        .catch(error => {
+            console.log(error);
+            return undefined;
+        });
+};
diff --git a/app/frontend/src/components/Answer/Answer.module.css b/app/frontend/src/components/Answer/Answer.module.css
index f2c3bf6..782f05d 100644
--- a/app/frontend/src/components/Answer/Answer.module.css
+++ b/app/frontend/src/components/Answer/Answer.module.css
@@ -2,7 +2,9 @@
     padding: 20px;
     background: rgb(249, 249, 249);
     border-radius: 8px;
-    box-shadow: 0px 2px 4px rgba(0, 0, 0, 0.14), 0px 0px 2px rgba(0, 0, 0, 0.12);
+    box-shadow:
+        0px 2px 4px rgba(0, 0, 0, 0.14),
+        0px 0px 2px rgba(0, 0, 0, 0.12);
     outline: transparent solid 1px;
 }
 
diff --git a/app/frontend/src/components/Answer/Answer.tsx b/app/frontend/src/components/Answer/Answer.tsx
index 5e796e8..b79e806 100644
--- a/app/frontend/src/components/Answer/Answer.tsx
+++ b/app/frontend/src/components/Answer/Answer.tsx
@@ -30,7 +30,7 @@ export const Answer = ({
     showFollowupQuestions
 }: Props) => {
     const messageContent = answer.choices[0].message.content;
-    const parsedAnswer = useMemo(() => parseAnswerToHtml(messageContent, isStreaming, onCitationClicked ), [answer]);
+    const parsedAnswer = useMemo(() => parseAnswerToHtml(messageContent, isStreaming, onCitationClicked), [answer]);
 
     const sanitizedAnswerHtml = DOMPurify.sanitize(parsedAnswer.answerHtml);
 
diff --git a/app/frontend/src/components/Answer/AnswerParser.tsx b/app/frontend/src/components/Answer/AnswerParser.tsx
index 984216d..bf09a61 100644
--- a/app/frontend/src/components/Answer/AnswerParser.tsx
+++ b/app/frontend/src/components/Answer/AnswerParser.tsx
@@ -21,19 +21,19 @@ export function parseAnswerToHtml(answer: string, isStreaming: boolean, onCitati
     parsedAnswer = parsedAnswer.trim();
 
     // Omit a citation that is still being typed during streaming
-    if (isStreaming){
+    if (isStreaming) {
         let lastIndex = parsedAnswer.length;
         for (let i = parsedAnswer.length - 1; i >= 0; i--) {
-            if (parsedAnswer[i] === ']') {
+            if (parsedAnswer[i] === "]") {
                 break;
-            } else if (parsedAnswer[i] === '[') {
+            } else if (parsedAnswer[i] === "[") {
                 lastIndex = i;
                 break;
             }
         }
         const truncatedAnswer = parsedAnswer.substring(0, lastIndex);
         parsedAnswer = truncatedAnswer;
-    } 
+    }
 
     const parts = parsedAnswer.split(/\[([^\]]+)\]/g);
 
diff --git a/app/frontend/src/components/Example/Example.module.css b/app/frontend/src/components/Example/Example.module.css
index 796f217..2911450 100644
--- a/app/frontend/src/components/Example/Example.module.css
+++ b/app/frontend/src/components/Example/Example.module.css
@@ -20,7 +20,9 @@
 }
 
 .example:hover {
-    box-shadow: 0px 8px 16px rgba(0, 0, 0, 0.14), 0px 0px 2px rgba(0, 0, 0, 0.12);
+    box-shadow:
+        0px 8px 16px rgba(0, 0, 0, 0.14),
+        0px 0px 2px rgba(0, 0, 0, 0.12);
     outline: 2px solid rgba(115, 118, 225, 1);
 }
 
diff --git a/app/frontend/src/components/LoginButton/LoginButton.module.css b/app/frontend/src/components/LoginButton/LoginButton.module.css
index 25bbaba..f808ac9 100644
--- a/app/frontend/src/components/LoginButton/LoginButton.module.css
+++ b/app/frontend/src/components/LoginButton/LoginButton.module.css
@@ -2,4 +2,4 @@
     border-radius: 5px;
     padding: 30px 30px;
     font-weight: 100;
-}
\ No newline at end of file
+}
diff --git a/app/frontend/src/components/LoginButton/LoginButton.tsx b/app/frontend/src/components/LoginButton/LoginButton.tsx
index c8657b3..7378ecf 100644
--- a/app/frontend/src/components/LoginButton/LoginButton.tsx
+++ b/app/frontend/src/components/LoginButton/LoginButton.tsx
@@ -1,40 +1,39 @@
-import { DefaultButton } from '@fluentui/react';
-import { useMsal } from '@azure/msal-react';
+import { DefaultButton } from "@fluentui/react";
+import { useMsal } from "@azure/msal-react";
 
 import styles from "./LoginButton.module.css";
-import { getRedirectUri, loginRequest } from '../../authConfig';
-
+import { getRedirectUri, loginRequest } from "../../authConfig";
 
 export const LoginButton = () => {
-  const { instance } = useMsal();
-  const activeAccount = instance.getActiveAccount();
-  const handleLoginPopup = () => {
-      /**
-       * When using popup and silent APIs, we recommend setting the redirectUri to a blank page or a page
-       * that does not implement MSAL. Keep in mind that all redirect routes must be registered with the application
-       * For more information, please follow this link: https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/dev/lib/msal-browser/docs/login-user.md#redirecturi-considerations
-       */
-      instance
-          .loginPopup({
-              ...loginRequest,
-              redirectUri: getRedirectUri(),
-          })
-          .catch((error) => console.log(error));
-  };
-  const handleLogoutPopup = () => {
-    instance
-        .logoutPopup({
-            mainWindowRedirectUri: '/', // redirects the top level app after logout
-            account: instance.getActiveAccount(),
-        })
-        .catch((error) => console.log(error));
-  };
-  const logoutText = `Logout\n${activeAccount?.username}`
-  return (
-      <DefaultButton
-        text={activeAccount ? logoutText : "Login"}
-        className={styles.loginButton}
-        onClick={activeAccount ? handleLogoutPopup : handleLoginPopup}>
-      </DefaultButton>
-  )
+    const { instance } = useMsal();
+    const activeAccount = instance.getActiveAccount();
+    const handleLoginPopup = () => {
+        /**
+         * When using popup and silent APIs, we recommend setting the redirectUri to a blank page or a page
+         * that does not implement MSAL. Keep in mind that all redirect routes must be registered with the application
+         * For more information, please follow this link: https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/dev/lib/msal-browser/docs/login-user.md#redirecturi-considerations
+         */
+        instance
+            .loginPopup({
+                ...loginRequest,
+                redirectUri: getRedirectUri()
+            })
+            .catch(error => console.log(error));
+    };
+    const handleLogoutPopup = () => {
+        instance
+            .logoutPopup({
+                mainWindowRedirectUri: "/", // redirects the top level app after logout
+                account: instance.getActiveAccount()
+            })
+            .catch(error => console.log(error));
+    };
+    const logoutText = `Logout\n${activeAccount?.username}`;
+    return (
+        <DefaultButton
+            text={activeAccount ? logoutText : "Login"}
+            className={styles.loginButton}
+            onClick={activeAccount ? handleLogoutPopup : handleLoginPopup}
+        ></DefaultButton>
+    );
 };
diff --git a/app/frontend/src/components/LoginButton/index.tsx b/app/frontend/src/components/LoginButton/index.tsx
index 06bd34e..06a844c 100644
--- a/app/frontend/src/components/LoginButton/index.tsx
+++ b/app/frontend/src/components/LoginButton/index.tsx
@@ -1,1 +1,1 @@
-export * from "./LoginButton"
+export * from "./LoginButton";
diff --git a/app/frontend/src/components/QuestionInput/QuestionInput.module.css b/app/frontend/src/components/QuestionInput/QuestionInput.module.css
index 8fa6d53..419523e 100644
--- a/app/frontend/src/components/QuestionInput/QuestionInput.module.css
+++ b/app/frontend/src/components/QuestionInput/QuestionInput.module.css
@@ -1,6 +1,8 @@
 .questionInputContainer {
     border-radius: 8px;
-    box-shadow: 0px 8px 16px rgba(0, 0, 0, 0.14), 0px 0px 2px rgba(0, 0, 0, 0.12);
+    box-shadow:
+        0px 8px 16px rgba(0, 0, 0, 0.14),
+        0px 0px 2px rgba(0, 0, 0, 0.12);
     height: 90px;
     width: 100%;
     padding: 15px;
diff --git a/app/frontend/src/components/SupportingContent/SupportingContent.module.css b/app/frontend/src/components/SupportingContent/SupportingContent.module.css
index 3ce11a6..b38d5d9 100644
--- a/app/frontend/src/components/SupportingContent/SupportingContent.module.css
+++ b/app/frontend/src/components/SupportingContent/SupportingContent.module.css
@@ -10,7 +10,9 @@
     word-break: break-word;
     background: rgb(249, 249, 249);
     border-radius: 8px;
-    box-shadow: rgb(0 0 0 / 5%) 0px 0px 0px 1px, rgb(0 0 0 / 10%) 0px 2px 3px 0px;
+    box-shadow:
+        rgb(0 0 0 / 5%) 0px 0px 0px 1px,
+        rgb(0 0 0 / 10%) 0px 2px 3px 0px;
     outline: transparent solid 1px;
 
     display: flex;
diff --git a/app/frontend/src/components/TokenClaimsDisplay/TokenClaimsDisplay.tsx b/app/frontend/src/components/TokenClaimsDisplay/TokenClaimsDisplay.tsx
index fc1c948..53035a1 100644
--- a/app/frontend/src/components/TokenClaimsDisplay/TokenClaimsDisplay.tsx
+++ b/app/frontend/src/components/TokenClaimsDisplay/TokenClaimsDisplay.tsx
@@ -1,5 +1,5 @@
-import { Label } from '@fluentui/react';
-import { useMsal } from '@azure/msal-react';
+import { Label } from "@fluentui/react";
+import { useMsal } from "@azure/msal-react";
 import {
     DataGridBody,
     DataGridRow,
@@ -9,80 +9,69 @@ import {
     DataGridCell,
     createTableColumn,
     TableColumnDefinition
-} from '@fluentui/react-table';
+} from "@fluentui/react-table";
 
 type Claim = {
     name: string;
     value: string;
-}
-
+};
 
 export const TokenClaimsDisplay = () => {
     const { instance } = useMsal();
     const activeAccount = instance.getActiveAccount();
 
     const ToString = (a: string | any) => {
-        if (typeof a === 'string') {
+        if (typeof a === "string") {
             return a;
         } else {
             return JSON.stringify(a);
         }
     };
 
-    const items: Claim[] = activeAccount?.idTokenClaims ?
-        Object.keys(activeAccount.idTokenClaims).map<Claim>(
-            (key: string) => { return { name: key, value: ToString((activeAccount.idTokenClaims ?? {})[key]) }; }) :
-        []
+    const items: Claim[] = activeAccount?.idTokenClaims
+        ? Object.keys(activeAccount.idTokenClaims).map<Claim>((key: string) => {
+              return { name: key, value: ToString((activeAccount.idTokenClaims ?? {})[key]) };
+          })
+        : [];
 
     const columns: TableColumnDefinition<Claim>[] = [
         createTableColumn<Claim>({
-            columnId: 'name',
+            columnId: "name",
             compare: (a: Claim, b: Claim) => {
                 return a.name.localeCompare(b.name);
             },
             renderHeaderCell: () => {
-                return 'Name';
+                return "Name";
             },
             renderCell: item => {
-                return item.name
-            },
+                return item.name;
+            }
         }),
         createTableColumn<Claim>({
-            columnId: 'value',
+            columnId: "value",
             compare: (a: Claim, b: Claim) => {
                 return a.value.localeCompare(b.value);
             },
             renderHeaderCell: () => {
-                return 'Value';
+                return "Value";
             },
             renderCell: item => {
-                return item.value
-            },
+                return item.value;
+            }
         })
-    ]
+    ];
 
     return (
         <div>
             <Label>ID Token Claims</Label>
-            <DataGrid
-                items={items}
-                columns={columns}
-                sortable
-                getRowId={item => item.name}
-            >
+            <DataGrid items={items} columns={columns} sortable getRowId={item => item.name}>
                 <DataGridHeader>
-                <DataGridRow>
-                    {({ renderHeaderCell }) => <DataGridHeaderCell>{renderHeaderCell()}</DataGridHeaderCell>}
-                </DataGridRow>
+                    <DataGridRow>{({ renderHeaderCell }) => <DataGridHeaderCell>{renderHeaderCell()}</DataGridHeaderCell>}</DataGridRow>
                 </DataGridHeader>
                 <DataGridBody<Claim>>
-                {({ item, rowId }) => (
-                    <DataGridRow<Claim> key={rowId}>
-                    {({ renderCell }) => <DataGridCell>{renderCell(item)}</DataGridCell>}
-                    </DataGridRow>
-                )}
+                    {({ item, rowId }) => <DataGridRow<Claim> key={rowId}>{({ renderCell }) => <DataGridCell>{renderCell(item)}</DataGridCell>}</DataGridRow>}
                 </DataGridBody>
             </DataGrid>
         </div>
-    )
+    );
 };
diff --git a/app/frontend/src/components/TokenClaimsDisplay/index.tsx b/app/frontend/src/components/TokenClaimsDisplay/index.tsx
index 52c4634..baec9b5 100644
--- a/app/frontend/src/components/TokenClaimsDisplay/index.tsx
+++ b/app/frontend/src/components/TokenClaimsDisplay/index.tsx
@@ -1,1 +1,1 @@
-export * from './TokenClaimsDisplay'
\ No newline at end of file
+export * from "./TokenClaimsDisplay";
diff --git a/app/frontend/src/components/UserChatMessage/UserChatMessage.module.css b/app/frontend/src/components/UserChatMessage/UserChatMessage.module.css
index c80bea6..591f314 100644
--- a/app/frontend/src/components/UserChatMessage/UserChatMessage.module.css
+++ b/app/frontend/src/components/UserChatMessage/UserChatMessage.module.css
@@ -10,6 +10,8 @@
     padding: 20px;
     background: #e8ebfa;
     border-radius: 8px;
-    box-shadow: 0px 2px 4px rgba(0, 0, 0, 0.14), 0px 0px 2px rgba(0, 0, 0, 0.12);
+    box-shadow:
+        0px 2px 4px rgba(0, 0, 0, 0.14),
+        0px 0px 2px rgba(0, 0, 0, 0.12);
     outline: transparent solid 1px;
 }
diff --git a/app/frontend/src/index.css b/app/frontend/src/index.css
index 2efb29b..44e9bf2 100644
--- a/app/frontend/src/index.css
+++ b/app/frontend/src/index.css
@@ -12,7 +12,17 @@ body {
 html {
     background: #f2f2f2;
 
-    font-family: "Segoe UI", -apple-system, BlinkMacSystemFont, "Roboto", "Oxygen", "Ubuntu", "Cantarell", "Fira Sans", "Droid Sans", "Helvetica Neue",
+    font-family:
+        "Segoe UI",
+        -apple-system,
+        BlinkMacSystemFont,
+        "Roboto",
+        "Oxygen",
+        "Ubuntu",
+        "Cantarell",
+        "Fira Sans",
+        "Droid Sans",
+        "Helvetica Neue",
         sans-serif;
     -webkit-font-smoothing: antialiased;
     -moz-osx-font-smoothing: grayscale;
diff --git a/app/frontend/src/index.tsx b/app/frontend/src/index.tsx
index 47b0689..d0c06c3 100644
--- a/app/frontend/src/index.tsx
+++ b/app/frontend/src/index.tsx
@@ -2,9 +2,9 @@ import React from "react";
 import ReactDOM from "react-dom/client";
 import { createHashRouter, RouterProvider } from "react-router-dom";
 import { initializeIcons } from "@fluentui/react";
-import { MsalProvider } from '@azure/msal-react';
-import { PublicClientApplication, EventType, AccountInfo } from '@azure/msal-browser';
-import { msalConfig, useLogin } from './authConfig';
+import { MsalProvider } from "@azure/msal-react";
+import { PublicClientApplication, EventType, AccountInfo } from "@azure/msal-browser";
+import { msalConfig, useLogin } from "./authConfig";
 
 import "./index.css";
 
@@ -22,7 +22,7 @@ if (useLogin) {
     }
 
     // Listen for sign-in event and set active account
-    msalInstance.addEventCallback((event) => {
+    msalInstance.addEventCallback(event => {
         if (event.eventType === EventType.LOGIN_SUCCESS && event.payload) {
             const account = event.payload as AccountInfo;
             msalInstance.setActiveAccount(account);
@@ -33,9 +33,9 @@ if (useLogin) {
         <MsalProvider instance={msalInstance}>
             <Layout />
         </MsalProvider>
-    )
+    );
 } else {
-    layout = <Layout />
+    layout = <Layout />;
 }
 
 initializeIcons();
diff --git a/app/frontend/src/pages/chat/Chat.tsx b/app/frontend/src/pages/chat/Chat.tsx
index 788b1f1..1f95462 100644
--- a/app/frontend/src/pages/chat/Chat.tsx
+++ b/app/frontend/src/pages/chat/Chat.tsx
@@ -52,12 +52,9 @@ const Chat = () => {
             return new Promise(resolve => {
                 setTimeout(() => {
                     answer += newContent;
-                    const latestResponse: ChatAppResponse = { ...askResponse,
-                        choices: [{ ...askResponse.choices[0],
-                                    message: { content: answer,
-                                        role: askResponse.choices[0].message.role
-                                    } }
-                                ]
+                    const latestResponse: ChatAppResponse = {
+                        ...askResponse,
+                        choices: [{ ...askResponse.choices[0], message: { content: answer, role: askResponse.choices[0].message.role } }]
                     };
                     setStreamedAnswers([...answers, [question, latestResponse]]);
                     resolve(null);
@@ -78,12 +75,9 @@ const Chat = () => {
         } finally {
             setIsStreaming(false);
         }
-        const fullResponse: ChatAppResponse = { ...askResponse,
-            choices: [{ ...askResponse.choices[0],
-                        message: { content: answer,
-                            role: askResponse.choices[0].message.role
-                        } }
-                    ]
+        const fullResponse: ChatAppResponse = {
+            ...askResponse,
+            choices: [{ ...askResponse.choices[0], message: { content: answer, role: askResponse.choices[0].message.role } }]
         };
         return fullResponse;
     };
@@ -101,10 +95,10 @@ const Chat = () => {
         const token = client ? await getToken(client) : undefined;
 
         try {
-            const messages: ResponseMessage[] = answers.flatMap(a => ([
+            const messages: ResponseMessage[] = answers.flatMap(a => [
                 { content: a[0], role: "user" },
-                { content: a[1].choices[0].message.content, role: "bot" }
-            ]));
+                { content: a[1].choices[0].message.content, role: "assistant" }
+            ]);
 
             const request: ChatAppRequest = {
                 messages: [...messages, { content: question, role: "user" }],
diff --git a/app/frontend/src/pages/layout/Layout.tsx b/app/frontend/src/pages/layout/Layout.tsx
index 4e85fb6..88f167c 100644
--- a/app/frontend/src/pages/layout/Layout.tsx
+++ b/app/frontend/src/pages/layout/Layout.tsx
@@ -4,9 +4,9 @@ import github from "../../assets/github.svg";
 
 import styles from "./Layout.module.css";
 
-import { useLogin } from "../../authConfig"
+import { useLogin } from "../../authConfig";
 
-import { LoginButton } from "../../components/LoginButton"
+import { LoginButton } from "../../components/LoginButton";
 
 const Layout = () => {
     return (
@@ -43,7 +43,7 @@ const Layout = () => {
                         </ul>
                     </nav>
                     <h4 className={styles.headerRightText}>Azure OpenAI + Cognitive Search</h4>
-                    {useLogin && <LoginButton/>}
+                    {useLogin && <LoginButton />}
                 </div>
             </header>
 
diff --git a/app/frontend/src/pages/oneshot/OneShot.tsx b/app/frontend/src/pages/oneshot/OneShot.tsx
index 8a4083a..13b623f 100644
--- a/app/frontend/src/pages/oneshot/OneShot.tsx
+++ b/app/frontend/src/pages/oneshot/OneShot.tsx
@@ -35,7 +35,7 @@ export function Component(): JSX.Element {
     const [activeCitation, setActiveCitation] = useState<string>();
     const [activeAnalysisPanelTab, setActiveAnalysisPanelTab] = useState<AnalysisPanelTabs | undefined>(undefined);
 
-    const client = useLogin ? useMsal().instance : undefined
+    const client = useLogin ? useMsal().instance : undefined;
 
     const makeApiRequest = async (question: string) => {
         lastQuestionRef.current = question;
@@ -45,7 +45,7 @@ export function Component(): JSX.Element {
         setActiveCitation(undefined);
         setActiveAnalysisPanelTab(undefined);
 
-        const token = client ? await getToken(client) : undefined
+        const token = client ? await getToken(client) : undefined;
 
         try {
             const request: ChatAppRequest = {
@@ -195,7 +195,6 @@ export function Component(): JSX.Element {
                 onRenderFooterContent={() => <DefaultButton onClick={() => setIsConfigPanelOpen(false)}>Close</DefaultButton>}
                 isFooterAtBottom={true}
             >
-
                 <TextField
                     className={styles.oneshotSettingsSeparator}
                     defaultValue={promptTemplate}
@@ -236,7 +235,7 @@ export function Component(): JSX.Element {
                         onChange={onUseOidSecurityFilterChange}
                     />
                 )}
-                {useLogin &&  (
+                {useLogin && (
                     <Checkbox
                         className={styles.oneshotSettingsSeparator}
                         checked={useGroupsSecurityFilter}
@@ -256,7 +255,7 @@ export function Component(): JSX.Element {
                     required
                     onChange={onRetrievalModeChange}
                 />
-                { useLogin && <TokenClaimsDisplay />}
+                {useLogin && <TokenClaimsDisplay />}
             </Panel>
         </div>
     );
diff --git a/locustfile.py b/locustfile.py
index 3bcf62d..0bc5f0c 100644
--- a/locustfile.py
+++ b/locustfile.py
@@ -16,15 +16,16 @@ class ChatUser(HttpUser):
             json={
                 "history": [
                     {
-                        "user": random.choice(
+                        "content": random.choice(
                             [
                                 "What is included in my Northwind Health Plus plan that is not in standard?",
                                 "What does a Product Manager do?",
                                 "What happens in a performance review?",
                                 "Whats your whistleblower policy?",
                             ]
-                        )
-                    }
+                        ),
+                        "role": "user",
+                    },
                 ],
                 "overrides": {
                     "retrieval_mode": "hybrid",
@@ -40,11 +41,12 @@ class ChatUser(HttpUser):
             "/chat",
             json={
                 "history": [
+                    {"content": "What happens in a performance review?", "role": "user"},
                     {
-                        "user": "What happens in a performance review?",
-                        "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+                        "content": "During a performance review, employees will receive feedback on their performance over the past year, including both successes and areas for improvement. The feedback will be provided by the employee's supervisor and is intended to help the employee develop and grow in their role [employee_handbook-3.pdf]. The review is a two-way dialogue between the employee and their manager, so employees are encouraged to be honest and open during the process [employee_handbook-3.pdf]. The employee will also have the opportunity to discuss their goals and objectives for the upcoming year [employee_handbook-3.pdf]. A written summary of the performance review will be provided to the employee, which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+                        "role": "assistant",
                     },
-                    {"user": "Does my plan cover eye exams?"},
+                    {"content": "Does my plan cover eye exams?", "role": "user"},
                 ],
                 "overrides": {
                     "retrieval_mode": "hybrid",
diff --git a/scripts/requirements.txt b/scripts/requirements.txt
index a8a5d54..cb2f349 100644
--- a/scripts/requirements.txt
+++ b/scripts/requirements.txt
@@ -2,7 +2,7 @@
 # This file is autogenerated by pip-compile with Python 3.11
 # by the following command:
 #
-#    pip-compile scripts/requirements.in
+#    pip-compile requirements.in
 #
 aiohttp==3.8.5
     # via openai
@@ -140,7 +140,7 @@ typing-extensions==4.8.0
     #   azure-storage-file-datalake
 tzdata==2023.3
     # via pandas
-urllib3==2.0.6
+urllib3==2.0.7
     # via requests
 yarl==1.9.2
     # via aiohttp


Test patch:
diff --git a/.github/workflows/python-test.yaml b/.github/workflows/python-test.yaml
index 3bdc4df..5a6ae9d 100644
--- a/.github/workflows/python-test.yaml
+++ b/.github/workflows/python-test.yaml
@@ -15,7 +15,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: ["ubuntu-20.04"]
+        os: ["ubuntu-20.04", "windows-latest"]
         python_version: ["3.9", "3.10", "3.11"]
     steps:
         - uses: actions/checkout@v4
@@ -44,8 +44,10 @@ jobs:
         - name: Check formatting with black
           run: black . --check --verbose
         - name: Run Python tests
-          run: python3 -m pytest --cov --cov-fail-under=78
+          if: runner.os != 'Windows'
+          run: python3 -m pytest -s -vv --cov --cov-fail-under=78
         - name: Run E2E tests with Playwright
+          if: runner.os != 'Windows'
           run: |
             playwright install --with-deps
             python3 -m pytest tests/e2e.py
diff --git a/tests/snapshots/test_app/test_chat_with_history/client0/result.json b/tests/snapshots/test_app/test_chat_with_history/client0/result.json
new file mode 100644
index 0000000..089c1e8
--- /dev/null
+++ b/tests/snapshots/test_app/test_chat_with_history/client0/result.json
@@ -0,0 +1,18 @@
+{
+    "choices": [
+        {
+            "context": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>The capital of France is Paris. [Benefit_Options-2.pdf].<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What happens in a performance review?'}<br><br>{'role': 'assistant', 'content': \"During a performance review, employees will receive feedback on their performance over the past year, including both successes and areas for improvement. The feedback will be provided by the employee's supervisor and is intended to help the employee develop and grow in their role [employee_handbook-3.pdf]. The review is a two-way dialogue between the employee and their manager, so employees are encouraged to be honest and open during the process [employee_handbook-3.pdf]. The employee will also have the opportunity to discuss their goals and objectives for the upcoming year [employee_handbook-3.pdf]. A written summary of the performance review will be provided to the employee, which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].\"}<br><br>{'role': 'user', 'content': 'Is dental covered?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
+                "role": "assistant"
+            },
+            "session_state": null
+        }
+    ],
+    "object": "chat.completion"
+}
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_with_history/client1/result.json b/tests/snapshots/test_app/test_chat_with_history/client1/result.json
new file mode 100644
index 0000000..089c1e8
--- /dev/null
+++ b/tests/snapshots/test_app/test_chat_with_history/client1/result.json
@@ -0,0 +1,18 @@
+{
+    "choices": [
+        {
+            "context": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>The capital of France is Paris. [Benefit_Options-2.pdf].<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What happens in a performance review?'}<br><br>{'role': 'assistant', 'content': \"During a performance review, employees will receive feedback on their performance over the past year, including both successes and areas for improvement. The feedback will be provided by the employee's supervisor and is intended to help the employee develop and grow in their role [employee_handbook-3.pdf]. The review is a two-way dialogue between the employee and their manager, so employees are encouraged to be honest and open during the process [employee_handbook-3.pdf]. The employee will also have the opportunity to discuss their goals and objectives for the upcoming year [employee_handbook-3.pdf]. A written summary of the performance review will be provided to the employee, which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].\"}<br><br>{'role': 'user', 'content': 'Is dental covered?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
+                "role": "assistant"
+            },
+            "session_state": null
+        }
+    ],
+    "object": "chat.completion"
+}
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_with_long_history/client0/result.json b/tests/snapshots/test_app/test_chat_with_long_history/client0/result.json
new file mode 100644
index 0000000..ec5a666
--- /dev/null
+++ b/tests/snapshots/test_app/test_chat_with_long_history/client0/result.json
@@ -0,0 +1,18 @@
+{
+    "choices": [
+        {
+            "context": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>The capital of France is Paris. [Benefit_Options-2.pdf].<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What does a product manager do?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
+                "role": "assistant"
+            },
+            "session_state": null
+        }
+    ],
+    "object": "chat.completion"
+}
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_with_long_history/client1/result.json b/tests/snapshots/test_app/test_chat_with_long_history/client1/result.json
new file mode 100644
index 0000000..ec5a666
--- /dev/null
+++ b/tests/snapshots/test_app/test_chat_with_long_history/client1/result.json
@@ -0,0 +1,18 @@
+{
+    "choices": [
+        {
+            "context": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>The capital of France is Paris. [Benefit_Options-2.pdf].<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What does a product manager do?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
+                "role": "assistant"
+            },
+            "session_state": null
+        }
+    ],
+    "object": "chat.completion"
+}
\ No newline at end of file
diff --git a/tests/test_app.py b/tests/test_app.py
index ae110dd..d14c53b 100644
--- a/tests/test_app.py
+++ b/tests/test_app.py
@@ -1,4 +1,5 @@
 import json
+import logging
 import os
 from unittest import mock
 
@@ -326,19 +327,55 @@ async def test_chat_stream_text_filter(auth_client, snapshot):
 
 
 @pytest.mark.asyncio
-async def test_ask_session_state_persists(client, snapshot):
+async def test_chat_with_history(client, snapshot):
     response = await client.post(
-        "/ask",
+        "/chat",
         json={
-            "messages": [{"content": "What is the capital of France?", "role": "user"}],
+            "messages": [
+                {"content": "What happens in a performance review?", "role": "user"},
+                {
+                    "content": "During a performance review, employees will receive feedback on their performance over the past year, including both successes and areas for improvement. The feedback will be provided by the employee's supervisor and is intended to help the employee develop and grow in their role [employee_handbook-3.pdf]. The review is a two-way dialogue between the employee and their manager, so employees are encouraged to be honest and open during the process [employee_handbook-3.pdf]. The employee will also have the opportunity to discuss their goals and objectives for the upcoming year [employee_handbook-3.pdf]. A written summary of the performance review will be provided to the employee, which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+                    "role": "assistant",
+                },
+                {"content": "Is dental covered?", "role": "user"},
+            ],
+            "context": {
+                "overrides": {"retrieval_mode": "text"},
+            },
+        },
+    )
+    assert response.status_code == 200
+    result = await response.get_json()
+    assert result["choices"][0]["context"]["thoughts"].find("performance review") != -1
+    snapshot.assert_match(json.dumps(result, indent=4), "result.json")
+
+
+@pytest.mark.asyncio
+async def test_chat_with_long_history(client, snapshot, caplog):
+    """This test makes sure that the history is truncated to max tokens minus 1024."""
+    caplog.set_level(logging.DEBUG)
+    response = await client.post(
+        "/chat",
+        json={
+            "messages": [
+                {"role": "user", "content": "Is there a dress code?"},  # 9 tokens
+                {
+                    "role": "assistant",
+                    "content": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]"
+                    * 150,
+                },  # 3900 tokens
+                {"role": "user", "content": "What does a product manager do?"},  # 10 tokens
+            ],
             "context": {
                 "overrides": {"retrieval_mode": "text"},
             },
-            "session_state": {"conversation_id": 1234},
         },
     )
     assert response.status_code == 200
     result = await response.get_json()
+    # Assert that it doesn't find the first message, since it wouldn't fit in the max tokens.
+    assert result["choices"][0]["context"]["thoughts"].find("Is there a dress code?") == -1
+    assert "Reached max tokens" in caplog.text
     snapshot.assert_match(json.dumps(result, indent=4), "result.json")
 
 
@@ -351,7 +388,6 @@ async def test_chat_session_state_persists(client, snapshot):
             "context": {
                 "overrides": {"retrieval_mode": "text"},
             },
-            "stream": False,
             "session_state": {"conversation_id": 1234},
         },
     )
diff --git a/tests/test_chatapproach.py b/tests/test_chatapproach.py
index 5bf779c..fa81ebf 100644
--- a/tests/test_chatapproach.py
+++ b/tests/test_chatapproach.py
@@ -36,13 +36,15 @@ def test_get_messages_from_history():
         system_prompt="You are a bot.",
         model_id="gpt-35-turbo",
         history=[
+            {"role": "user", "content": "What happens in a performance review?"},
             {
-                "user": "What happens in a performance review?",
-                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+                "role": "assistant",
+                "content": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
             },
-            {"user": "What does a Product Manager do?"},
+            {"role": "user", "content": "What does a Product Manager do?"},
         ],
         user_content="What does a Product Manager do?",
+        max_tokens=3000,
     )
     assert messages == [
         {"role": "system", "content": "You are a bot."},
@@ -64,11 +66,12 @@ def test_get_messages_from_history_truncated():
         system_prompt="You are a bot.",
         model_id="gpt-35-turbo",
         history=[
+            {"role": "user", "content": "What happens in a performance review?"},
             {
-                "user": "What happens in a performance review?",
-                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+                "role": "assistant",
+                "content": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
             },
-            {"user": "What does a Product Manager do?"},
+            {"role": "user", "content": "What does a Product Manager do?"},
         ],
         user_content="What does a Product Manager do?",
         max_tokens=10,
@@ -85,24 +88,66 @@ def test_get_messages_from_history_truncated_longer():
     )
 
     messages = chat_approach.get_messages_from_history(
-        system_prompt="You are a bot.",
+        system_prompt="You are a bot.",  # 8 tokens
         model_id="gpt-35-turbo",
         history=[
+            {"role": "user", "content": "What happens in a performance review?"},  # 10 tokens
             {
-                "user": "What happens in a performance review?",
-                "bot": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
-            },
+                "role": "assistant",
+                "content": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+            },  # 102 tokens
+            {"role": "user", "content": "Is there a dress code?"},  # 9 tokens
             {
-                "user": "Is there a dress code?",
-                "bot": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]",
-            },
-            {"user": "What does a Product Manager do?"},
+                "role": "assistant",
+                "content": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]",
+            },  # 26 tokens
+            {"role": "user", "content": "What does a Product Manager do?"},  # 10 tokens
+        ],
+        user_content="What does a Product Manager do?",
+        max_tokens=55,
+    )
+    assert messages == [
+        {"role": "system", "content": "You are a bot."},
+        {"role": "user", "content": "Is there a dress code?"},
+        {
+            "role": "assistant",
+            "content": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]",
+        },
+        {"role": "user", "content": "What does a Product Manager do?"},
+    ]
+
+
+def test_get_messages_from_history_truncated_break_pair():
+    """Tests that the truncation breaks the pair of messages."""
+    chat_approach = ChatReadRetrieveReadApproach(
+        None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "", "en-us", "lexicon"
+    )
+
+    messages = chat_approach.get_messages_from_history(
+        system_prompt="You are a bot.",  # 8 tokens
+        model_id="gpt-35-turbo",
+        history=[
+            {"role": "user", "content": "What happens in a performance review?"},  # 10 tokens
+            {
+                "role": "assistant",
+                "content": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+            },  # 102 tokens
+            {"role": "user", "content": "Is there a dress code?"},  # 9 tokens
+            {
+                "role": "assistant",
+                "content": "Yes, there is a dress code at Contoso Electronics. Look sharp! [employee_handbook-1.pdf]",
+            },  # 26 tokens
+            {"role": "user", "content": "What does a Product Manager do?"},  # 10 tokens
         ],
         user_content="What does a Product Manager do?",
-        max_tokens=30,
+        max_tokens=147,
     )
     assert messages == [
         {"role": "system", "content": "You are a bot."},
+        {
+            "role": "assistant",
+            "content": "During the performance review at Contoso Electronics, the supervisor will discuss the employee's performance over the past year and provide feedback on areas for improvement. They will also provide an opportunity for the employee to discuss their goals and objectives for the upcoming year. The review is a two-way dialogue between managers and employees, and employees will receive a written summary of their performance review which will include a rating of their performance, feedback, and goals and objectives for the upcoming year [employee_handbook-3.pdf].",
+        },
         {"role": "user", "content": "Is there a dress code?"},
         {
             "role": "assistant",
diff --git a/tests/test_messagebuilder.py b/tests/test_messagebuilder.py
index 85a5497..3b6cb68 100644
--- a/tests/test_messagebuilder.py
+++ b/tests/test_messagebuilder.py
@@ -8,7 +8,7 @@ def test_messagebuilder():
         {"role": "system", "content": "You are a bot."}
     ]
     assert builder.model == "gpt-35-turbo"
-    assert builder.token_length == 8
+    assert builder.count_tokens_for_message(builder.messages[0]) == 8
 
 
 def test_messagebuilder_append():
@@ -21,7 +21,8 @@ def test_messagebuilder_append():
         {"role": "user", "content": "Hello, how are you?"},
     ]
     assert builder.model == "gpt-35-turbo"
-    assert builder.token_length == 17
+    assert builder.count_tokens_for_message(builder.messages[0]) == 8
+    assert builder.count_tokens_for_message(builder.messages[1]) == 9
 
 
 def test_messagebuilder_unicode():
@@ -31,7 +32,7 @@ def test_messagebuilder_unicode():
         {"role": "system", "content": "á"}
     ]
     assert builder.model == "gpt-35-turbo"
-    assert builder.token_length == 4
+    assert builder.count_tokens_for_message(builder.messages[0]) == 4
 
 
 def test_messagebuilder_unicode_append():
@@ -44,4 +45,5 @@ def test_messagebuilder_unicode_append():
         {"role": "user", "content": "á"},
     ]
     assert builder.model == "gpt-35-turbo"
-    assert builder.token_length == 8
+    assert builder.count_tokens_for_message(builder.messages[0]) == 4
+    assert builder.count_tokens_for_message(builder.messages[1]) == 4