Repository: Azure-Samples/azure-search-openai-demo
Pull request #780 (2023-10-12 13:37:47)

support non English languages
## Purpose
<!-- Describe the intention of the changes being proposed. What problem does it solve or functionality does it add? -->
* First of all, thank you for providing this useful sample code :) For now, to support non English languages, several hardcoded codes should be overwritten. In fact, we Japanese fork the repo and overwrite them to support customers (ref. [nohanaga/azure-search-openai-demo](https://github.com/nohanaga/azure-search-openai-demo), [asashiho/azure-search-openai-demo](https://github.com/asashiho/azure-search-openai-demo), [ks6088ts-labs/_azure-search-openai-demo](https://github.com/ks6088ts-labs/_azure-search-openai-demo), etc). But this approach brings some pain to catch up with the updates on new commits. So, in this PR, configurable parameters are introduced to get rid of those hardcoded codes. This fix can enlarge customers to worldwide, I think.

## Does this introduce a breaking change?
<!-- Mark one with an "x". -->
```
[ ] Yes
[x] No
```

## Pull Request Type
What kind of change does this Pull Request introduce?

<!-- Please check the one that applies to this PR using "x". -->
```
[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:
```

## How to Test
*  Get the code

```
# To use Japanese languages
azd env set AZURE_SEARCH_QUERY_LANGUAGE ja-jp
azd env set AZURE_SEARCH_QUERY_SPELLER none
azd env set AZURE_SEARCH_ANALYZER_NAME ja.lucene

# deploy
azd up
```

## What to Check
Verify that the following are valid
* Check the App Service environment variables
* Check the analyzer settings of Cognitive Search index


## Other Information
<!-- Add any other helpful information that may be needed here. -->

pamelafox: Neat! So you don't also have to change the English prompts?

Patch:
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
index bf063db..1e1641f 100644
--- a/.devcontainer/devcontainer.json
+++ b/.devcontainer/devcontainer.json
@@ -15,7 +15,8 @@
             "extensions": [
                 "ms-azuretools.azure-dev",
                 "ms-azuretools.vscode-bicep",
-                "ms-python.python"
+                "ms-python.python",
+                "esbenp.prettier-vscode"
             ]
         }
     },
diff --git a/.vscode/extensions.json b/.vscode/extensions.json
index 12faf24..c38e473 100644
--- a/.vscode/extensions.json
+++ b/.vscode/extensions.json
@@ -1,6 +1,8 @@
 {
     "recommendations": [
-        "esbenp.prettier-vscode",
-        "ms-azuretools.azure-dev"
+        "ms-azuretools.azure-dev",
+        "ms-azuretools.vscode-bicep",
+        "ms-python.python",
+        "esbenp.prettier-vscode"
     ]
-}
\ No newline at end of file
+}
diff --git a/.vscode/launch.json b/.vscode/launch.json
index 52c350c..97633e0 100644
--- a/.vscode/launch.json
+++ b/.vscode/launch.json
@@ -26,27 +26,10 @@
         },
         {
             "name": "Frontend: watch",
-            "type": "node",
+            "type": "node-terminal",
             "request": "launch",
+            "command": "npm run dev",
             "cwd": "${workspaceFolder}/app/frontend",
-            "runtimeExecutable": "npm",
-            "runtimeArgs": [
-                "run-script",
-                "watch"
-            ],
-            "console": "integratedTerminal",
-        },
-        {
-            "name": "Frontend: build",
-            "type": "node",
-            "request": "launch",
-            "cwd": "${workspaceFolder}/app/frontend",
-            "runtimeExecutable": "npm",
-            "runtimeArgs": [
-                "run-script",
-                "build"
-            ],
-            "console": "integratedTerminal",
         }
     ],
     "inputs": [
diff --git a/README.md b/README.md
index 636e460..a09d5e8 100644
--- a/README.md
+++ b/README.md
@@ -170,6 +170,11 @@ When you run `azd up` after and are prompted to select a value for `openAiResour
 1. If the search service's SKU is not standard, then run `azd env set AZURE_SEARCH_SERVICE_SKU {Name of SKU}`. The free tier won't work as it doesn't support managed identity. ([See other possible values](https://learn.microsoft.com/azure/templates/microsoft.search/searchservices?pivots=deployment-language-bicep#sku))
 1. If you have an existing index that is set up with all the expected fields, then run `azd env set AZURE_SEARCH_INDEX {Name of existing index}`. Otherwise, the `azd up` command will create a new index.
 
+You can also customize the search service (new or existing) for non-English searches:
+
+1. To configure the language of the search query to a value other than "en-us", run `azd env set AZURE_SEARCH_QUERY_LANGUAGE {Name of query language}`. ([See other possible values](https://learn.microsoft.com/python/api/azure-search-documents/azure.search.documents.models.querylanguage?view=azure-python-preview))
+1. To turn off the spell checker, run `azd env set AZURE_SEARCH_QUERY_SPELLER none`. ([See other possible values](https://learn.microsoft.com/python/api/azure-search-documents/azure.search.documents.models.queryspellertype?view=azure-python-preview))
+1. To configure the name of the analyzer to use for a searchable text field to a value other than "en.microsoft", run `azd env set AZURE_SEARCH_ANALYZER_NAME {Name of analyzer name}`. ([See other possible values](https://learn.microsoft.com/dotnet/api/microsoft.azure.search.models.field.analyzer?view=azure-dotnet-legacy&viewFallbackFrom=azure-dotnet))
 
 #### Other existing Azure resources
 
diff --git a/app/backend/app.py b/app/backend/app.py
index f39e5b7..990c86b 100644
--- a/app/backend/app.py
+++ b/app/backend/app.py
@@ -96,7 +96,9 @@ async def ask():
         # Workaround for: https://github.com/openai/openai-python/issues/371
         async with aiohttp.ClientSession() as s:
             openai.aiosession.set(s)
-            r = await approach.run(request_json["messages"], context=context)
+            r = await approach.run(
+                request_json["messages"], context=context, session_state=request_json.get("session_state")
+            )
         return jsonify(r)
     except Exception as e:
         logging.exception("Exception in /ask")
@@ -118,7 +120,12 @@ async def chat():
     context["auth_claims"] = await auth_helper.get_auth_claims_if_enabled(request.headers)
     try:
         approach = current_app.config[CONFIG_CHAT_APPROACH]
-        result = await approach.run(request_json["messages"], stream=request_json.get("stream", False), context=context)
+        result = await approach.run(
+            request_json["messages"],
+            stream=request_json.get("stream", False),
+            context=context,
+            session_state=request_json.get("session_state"),
+        )
         if isinstance(result, dict):
             return jsonify(result)
         else:
@@ -178,6 +185,9 @@ async def setup_clients():
     KB_FIELDS_CONTENT = os.getenv("KB_FIELDS_CONTENT", "content")
     KB_FIELDS_SOURCEPAGE = os.getenv("KB_FIELDS_SOURCEPAGE", "sourcepage")
 
+    AZURE_SEARCH_QUERY_LANGUAGE = os.getenv("AZURE_SEARCH_QUERY_LANGUAGE", "en-us")
+    AZURE_SEARCH_QUERY_SPELLER = os.getenv("AZURE_SEARCH_QUERY_SPELLER", "lexicon")
+
     # Use the current user identity to authenticate with Azure OpenAI, Cognitive Search and Blob Storage (no secrets needed,
     # just use 'az login' locally, and managed identity when deployed on Azure). If you need to use keys, use separate AzureKeyCredential instances with the
     # keys for each service
@@ -235,6 +245,8 @@ async def setup_clients():
         OPENAI_EMB_MODEL,
         KB_FIELDS_SOURCEPAGE,
         KB_FIELDS_CONTENT,
+        AZURE_SEARCH_QUERY_LANGUAGE,
+        AZURE_SEARCH_QUERY_SPELLER,
     )
 
     current_app.config[CONFIG_CHAT_APPROACH] = ChatReadRetrieveReadApproach(
@@ -246,6 +258,8 @@ async def setup_clients():
         OPENAI_EMB_MODEL,
         KB_FIELDS_SOURCEPAGE,
         KB_FIELDS_CONTENT,
+        AZURE_SEARCH_QUERY_LANGUAGE,
+        AZURE_SEARCH_QUERY_SPELLER,
     )
 
 
diff --git a/app/backend/approaches/chatreadretrieveread.py b/app/backend/approaches/chatreadretrieveread.py
index 9561398..432f0ac 100644
--- a/app/backend/approaches/chatreadretrieveread.py
+++ b/app/backend/approaches/chatreadretrieveread.py
@@ -63,6 +63,8 @@ If you cannot generate a search query, return just the number 0.
         embedding_model: str,
         sourcepage_field: str,
         content_field: str,
+        query_language: str,
+        query_speller: str,
     ):
         self.search_client = search_client
         self.openai_host = openai_host
@@ -72,6 +74,8 @@ If you cannot generate a search query, return just the number 0.
         self.embedding_model = embedding_model
         self.sourcepage_field = sourcepage_field
         self.content_field = content_field
+        self.query_language = query_language
+        self.query_speller = query_speller
         self.chatgpt_token_limit = get_token_limit(chatgpt_model)
 
     async def run_until_final_call(
@@ -151,8 +155,8 @@ If you cannot generate a search query, return just the number 0.
                 query_text,
                 filter=filter,
                 query_type=QueryType.SEMANTIC,
-                query_language="en-us",
-                query_speller="lexicon",
+                query_language=self.query_language,
+                query_speller=self.query_speller,
                 semantic_configuration_name="default",
                 top=top,
                 query_caption="extractive|highlight-false" if use_semantic_captions else None,
@@ -224,23 +228,40 @@ If you cannot generate a search query, return just the number 0.
         return (extra_info, chat_coroutine)
 
     async def run_without_streaming(
-        self, history: list[dict[str, str]], overrides: dict[str, Any], auth_claims: dict[str, Any]
+        self,
+        history: list[dict[str, str]],
+        overrides: dict[str, Any],
+        auth_claims: dict[str, Any],
+        session_state: Any = None,
     ) -> dict[str, Any]:
         extra_info, chat_coroutine = await self.run_until_final_call(
             history, overrides, auth_claims, should_stream=False
         )
         chat_resp = dict(await chat_coroutine)
         chat_resp["choices"][0]["context"] = extra_info
+        chat_resp["choices"][0]["session_state"] = session_state
         return chat_resp
 
     async def run_with_streaming(
-        self, history: list[dict[str, str]], overrides: dict[str, Any], auth_claims: dict[str, Any]
+        self,
+        history: list[dict[str, str]],
+        overrides: dict[str, Any],
+        auth_claims: dict[str, Any],
+        session_state: Any = None,
     ) -> AsyncGenerator[dict, None]:
         extra_info, chat_coroutine = await self.run_until_final_call(
             history, overrides, auth_claims, should_stream=True
         )
         yield {
-            "choices": [{"delta": {"role": self.ASSISTANT}, "context": extra_info, "finish_reason": None, "index": 0}],
+            "choices": [
+                {
+                    "delta": {"role": self.ASSISTANT},
+                    "context": extra_info,
+                    "session_state": session_state,
+                    "finish_reason": None,
+                    "index": 0,
+                }
+            ],
             "object": "chat.completion.chunk",
         }
 
@@ -258,10 +279,10 @@ If you cannot generate a search query, return just the number 0.
             # Workaround for: https://github.com/openai/openai-python/issues/371
             async with aiohttp.ClientSession() as s:
                 openai.aiosession.set(s)
-                response = await self.run_without_streaming(messages, overrides, auth_claims)
+                response = await self.run_without_streaming(messages, overrides, auth_claims, session_state)
             return response
         else:
-            return self.run_with_streaming(messages, overrides, auth_claims)
+            return self.run_with_streaming(messages, overrides, auth_claims, session_state)
 
     def get_messages_from_history(
         self,
diff --git a/app/backend/approaches/retrievethenread.py b/app/backend/approaches/retrievethenread.py
index cb58e97..861023d 100644
--- a/app/backend/approaches/retrievethenread.py
+++ b/app/backend/approaches/retrievethenread.py
@@ -47,6 +47,8 @@ info4.pdf: In-network institutions include Overlake, Swedish and others in the r
         embedding_model: str,
         sourcepage_field: str,
         content_field: str,
+        query_language: str,
+        query_speller: str,
     ):
         self.search_client = search_client
         self.openai_host = openai_host
@@ -56,6 +58,8 @@ info4.pdf: In-network institutions include Overlake, Swedish and others in the r
         self.embedding_deployment = embedding_deployment
         self.sourcepage_field = sourcepage_field
         self.content_field = content_field
+        self.query_language = query_language
+        self.query_speller = query_speller
 
     async def run(
         self,
@@ -90,8 +94,8 @@ info4.pdf: In-network institutions include Overlake, Swedish and others in the r
                 query_text,
                 filter=filter,
                 query_type=QueryType.SEMANTIC,
-                query_language="en-us",
-                query_speller="lexicon",
+                query_language=self.query_language,
+                query_speller=self.query_speller,
                 semantic_configuration_name="default",
                 top=top,
                 query_caption="extractive|highlight-false" if use_semantic_captions else None,
@@ -146,4 +150,5 @@ info4.pdf: In-network institutions include Overlake, Swedish and others in the r
             + "\n\n".join([str(message) for message in messages]),
         }
         chat_completion.choices[0]["context"] = extra_info
+        chat_completion.choices[0]["session_state"] = session_state
         return chat_completion
diff --git a/app/frontend/package.json b/app/frontend/package.json
index aea31d5..d2cdde4 100644
--- a/app/frontend/package.json
+++ b/app/frontend/package.json
@@ -9,7 +9,7 @@
   "scripts": {
     "dev": "vite",
     "build": "tsc && vite build",
-    "watch": "tsc && vite build --watch"
+    "preview": "vite preview"
   },
   "dependencies": {
     "@azure/msal-react": "^2.0.4",
diff --git a/app/frontend/src/api/models.ts b/app/frontend/src/api/models.ts
index ccc1178..3cc3f65 100644
--- a/app/frontend/src/api/models.ts
+++ b/app/frontend/src/api/models.ts
@@ -33,6 +33,7 @@ export type ResponseChoice = {
     index: number;
     message: ResponseMessage;
     context: ResponseContext;
+    session_state: any;
 };
 
 export type ChatAppResponseOrError = {
@@ -52,4 +53,5 @@ export type ChatAppRequest = {
     messages: ResponseMessage[];
     context?: ChatAppRequestContext;
     stream?: boolean;
+    session_state: any;
 };
diff --git a/app/frontend/src/authConfig.ts b/app/frontend/src/authConfig.ts
index e3a96ee..37326c3 100644
--- a/app/frontend/src/authConfig.ts
+++ b/app/frontend/src/authConfig.ts
@@ -67,12 +67,17 @@ export const loginRequest = authSetup.loginRequest;
 
 const tokenRequest = authSetup.tokenRequest;
 
+// Build an absolute redirect URI using the current window's location and the relative redirect URI from auth setup
+export const getRedirectUri = () => {
+    return window.location.origin + authSetup.msalConfig.auth.redirectUri
+}
+
 // Get an access token for use with the API server.
 // ID token received when logging in may not be used for this purpose because it has the incorrect audience
 export const getToken = (client: IPublicClientApplication): Promise<AuthenticationResult | undefined> => {
     return client.acquireTokenSilent({
         ...tokenRequest,
-        redirectUri: authSetup.msalConfig.auth.redirectUri
+        redirectUri: getRedirectUri()
     })
     .catch((error) => {
         console.log(error);
diff --git a/app/frontend/src/components/LoginButton/LoginButton.tsx b/app/frontend/src/components/LoginButton/LoginButton.tsx
index f2a52d9..c8657b3 100644
--- a/app/frontend/src/components/LoginButton/LoginButton.tsx
+++ b/app/frontend/src/components/LoginButton/LoginButton.tsx
@@ -2,7 +2,7 @@ import { DefaultButton } from '@fluentui/react';
 import { useMsal } from '@azure/msal-react';
 
 import styles from "./LoginButton.module.css";
-import { loginRequest } from '../../authConfig';
+import { getRedirectUri, loginRequest } from '../../authConfig';
 
 
 export const LoginButton = () => {
@@ -17,7 +17,7 @@ export const LoginButton = () => {
       instance
           .loginPopup({
               ...loginRequest,
-              redirectUri: '/redirect',
+              redirectUri: getRedirectUri(),
           })
           .catch((error) => console.log(error));
   };
diff --git a/app/frontend/src/pages/chat/Chat.tsx b/app/frontend/src/pages/chat/Chat.tsx
index cba00b1..788b1f1 100644
--- a/app/frontend/src/pages/chat/Chat.tsx
+++ b/app/frontend/src/pages/chat/Chat.tsx
@@ -121,7 +121,9 @@ const Chat = () => {
                         use_oid_security_filter: useOidSecurityFilter,
                         use_groups_security_filter: useGroupsSecurityFilter
                     }
-                }
+                },
+                // ChatAppProtocol: Client must pass on any session state received from the server
+                session_state: answers.length ? answers[answers.length - 1][1].choices[0].session_state : null
             };
 
             const response = await chatApi(request, token?.accessToken);
diff --git a/app/frontend/src/pages/oneshot/OneShot.tsx b/app/frontend/src/pages/oneshot/OneShot.tsx
index c536697..8a4083a 100644
--- a/app/frontend/src/pages/oneshot/OneShot.tsx
+++ b/app/frontend/src/pages/oneshot/OneShot.tsx
@@ -68,7 +68,9 @@ export function Component(): JSX.Element {
                         use_oid_security_filter: useOidSecurityFilter,
                         use_groups_security_filter: useGroupsSecurityFilter
                     }
-                }
+                },
+                // ChatAppProtocol: Client must pass on any session state received from the server
+                session_state: answer ? answer.choices[0].session_state : null
             };
             const result = await askApi(request, token?.accessToken);
             setAnswer(result);
diff --git a/app/frontend/vite.config.ts b/app/frontend/vite.config.ts
index b5ff872..1e8c4ef 100644
--- a/app/frontend/vite.config.ts
+++ b/app/frontend/vite.config.ts
@@ -25,6 +25,8 @@ export default defineConfig({
     },
     server: {
         proxy: {
+            "/content/": "http://localhost:50505",
+            "/auth_setup": "http://localhost:50505",
             "/ask": "http://localhost:50505",
             "/chat": "http://localhost:50505"
         }
diff --git a/infra/main.bicep b/infra/main.bicep
index d0db43e..1e8af2b 100644
--- a/infra/main.bicep
+++ b/infra/main.bicep
@@ -22,6 +22,8 @@ param searchServiceLocation string = ''
 @allowed(['basic', 'standard', 'standard2', 'standard3', 'storage_optimized_l1', 'storage_optimized_l2'])
 param searchServiceSkuName string // Set in main.parameters.json
 param searchIndexName string // Set in main.parameters.json
+param searchQueryLanguage string // Set in main.parameters.json
+param searchQuerySpeller string // Set in main.parameters.json
 
 param storageAccountName string = ''
 param storageResourceGroupName string = ''
@@ -152,6 +154,8 @@ module backend 'core/host/appservice.bicep' = {
       AZURE_STORAGE_CONTAINER: storageContainerName
       AZURE_SEARCH_INDEX: searchIndexName
       AZURE_SEARCH_SERVICE: searchService.outputs.name
+      AZURE_SEARCH_QUERY_LANGUAGE: searchQueryLanguage
+      AZURE_SEARCH_QUERY_SPELLER: searchQuerySpeller
       APPLICATIONINSIGHTS_CONNECTION_STRING: useApplicationInsights ? monitoring.outputs.applicationInsightsConnectionString : ''
       // Shared by all OpenAI deployments
       OPENAI_HOST: openAiHost
diff --git a/infra/main.parameters.json b/infra/main.parameters.json
index d18a4b8..81311a7 100644
--- a/infra/main.parameters.json
+++ b/infra/main.parameters.json
@@ -47,6 +47,12 @@
     "searchServiceSkuName": {
       "value": "${AZURE_SEARCH_SERVICE_SKU=standard}"
     },
+    "searchQueryLanguage": {
+      "value": "${AZURE_SEARCH_QUERY_LANGUAGE=en-us}"
+    },
+    "searchQuerySpeller": {
+      "value": "${AZURE_SEARCH_QUERY_SPELLER=lexicon}"
+    },
     "storageAccountName": {
       "value": "${AZURE_STORAGE_ACCOUNT}"
     },
diff --git a/scripts/prepdocs.ps1 b/scripts/prepdocs.ps1
index e01f99b..8097b3d 100755
--- a/scripts/prepdocs.ps1
+++ b/scripts/prepdocs.ps1
@@ -27,7 +27,11 @@ if ($env:AZURE_ADLS_GEN2_STORAGE_ACCOUNT) {
   }
   $aclArg = "--useacls"
 }
-$argumentList = "./scripts/prepdocs.py `"$cwd/data/*`" $adlsGen2StorageAccountArg $adlsGen2FilesystemArg $adlsGen2FilesystemPathArg " + `
+# Optional Search Analyzer name if using a custom analyzer
+if ($env:AZURE_SEARCH_ANALYZER_NAME) {
+  $searchAnalyzerNameArg = "--searchanalyzername $env:AZURE_SEARCH_ANALYZER_NAME"
+}
+$argumentList = "./scripts/prepdocs.py `"$cwd/data/*`" $adlsGen2StorageAccountArg $adlsGen2FilesystemArg $adlsGen2FilesystemPathArg $searchAnalyzerNameArg " + `
 "$aclArg --storageaccount $env:AZURE_STORAGE_ACCOUNT --container $env:AZURE_STORAGE_CONTAINER " + `
 "--searchservice $env:AZURE_SEARCH_SERVICE --openaihost `"$env:OPENAI_HOST`" " + `
 "--openaiservice `"$env:AZURE_OPENAI_SERVICE`" --openaikey `"$env:OPENAI_API_KEY`" " + `
diff --git a/scripts/prepdocs.py b/scripts/prepdocs.py
index a718758..90c8001 100644
--- a/scripts/prepdocs.py
+++ b/scripts/prepdocs.py
@@ -345,7 +345,7 @@ def create_search_index():
     )
     fields = [
         SimpleField(name="id", type="Edm.String", key=True),
-        SearchableField(name="content", type="Edm.String", analyzer_name="en.microsoft"),
+        SearchableField(name="content", type="Edm.String", analyzer_name=args.searchanalyzername),
         SearchField(
             name="embedding",
             type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
@@ -654,6 +654,12 @@ if __name__ == "__main__":
         required=False,
         help="Optional. Use this Azure Cognitive Search account key instead of the current user identity to login (use az login to set current user for Azure)",
     )
+    parser.add_argument(
+        "--searchanalyzername",
+        required=False,
+        default="en.microsoft",
+        help="Optional. Name of the Azure Cognitive Search analyzer to use for the content field in the index",
+    )
     parser.add_argument("--openaihost", help="Host of the API used to compute embeddings ('azure' or 'openai')")
     parser.add_argument("--openaiservice", help="Name of the Azure OpenAI service used to compute embeddings")
     parser.add_argument(
diff --git a/scripts/prepdocs.sh b/scripts/prepdocs.sh
index b58e2b7..bd7e2ae 100755
--- a/scripts/prepdocs.sh
+++ b/scripts/prepdocs.sh
@@ -17,8 +17,12 @@ if [ -n "$AZURE_ADLS_GEN2_STORAGE_ACCOUNT" ]; then
   aclArg="--useacls"
 fi
 
+if [ -n "$AZURE_SEARCH_ANALYZER_NAME" ]; then
+  searchAnalyzerNameArg="--searchanalyzername $AZURE_SEARCH_ANALYZER_NAME"
+fi
+
 ./scripts/.venv/bin/python ./scripts/prepdocs.py \
-'./data/*' $adlsGen2StorageAccountArg $adlsGen2FilesystemArg $adlsGen2FilesystemPathArg \
+'./data/*' $adlsGen2StorageAccountArg $adlsGen2FilesystemArg $adlsGen2FilesystemPathArg $searchAnalyzerNameArg \
 $aclArg  --storageaccount "$AZURE_STORAGE_ACCOUNT" \
 --container "$AZURE_STORAGE_CONTAINER" --searchservice "$AZURE_SEARCH_SERVICE" \
 --openaiservice "$AZURE_OPENAI_SERVICE" --openaideployment "$AZURE_OPENAI_EMB_DEPLOYMENT" \


Test patch:
diff --git a/.github/workflows/python-test.yaml b/.github/workflows/python-test.yaml
index 4ed5652..3bdc4df 100644
--- a/.github/workflows/python-test.yaml
+++ b/.github/workflows/python-test.yaml
@@ -5,6 +5,8 @@ on:
     branches: [ main ]
   pull_request:
     branches: [ main ]
+  schedule:
+    - cron: '0 0 * * *' # Run at midnight every day
 
 jobs:
   test_package:
diff --git a/tests/e2e.py b/tests/e2e.py
index 86ccea9..234676d 100644
--- a/tests/e2e.py
+++ b/tests/e2e.py
@@ -55,6 +55,9 @@ def test_home(page: Page, live_server_url: str):
 def test_chat(page: Page, live_server_url: str):
     # Set up a mock route to the /chat endpoint with streaming results
     def handle(route: Route):
+        # Assert that session_state is specified in the request (None for now)
+        session_state = route.request.post_data_json["session_state"]
+        assert session_state is None
         # Read the JSONL from our snapshot results and return as the response
         f = open("tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines")
         jsonl = f.read()
@@ -191,6 +194,9 @@ def test_chat_nonstreaming(page: Page, live_server_url: str):
 def test_ask(page: Page, live_server_url: str):
     # Set up a mock route to the /ask endpoint
     def handle(route: Route):
+        # Assert that session_state is specified in the request (None for now)
+        session_state = route.request.post_data_json["session_state"]
+        assert session_state is None
         # Read the JSON from our snapshot results and return as the response
         f = open("tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json")
         json = f.read()
diff --git a/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
index 75b7bac..809a74e 100644
--- a/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_hybrid/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
index 75b7bac..809a74e 100644
--- a/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_hybrid/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
index 75b7bac..809a74e 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
index 75b7bac..809a74e 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
index 22d5169..635be8a 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_filter/auth_client0/result.json
@@ -8,7 +8,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
index 1f44116..0b2118e 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
index 1f44116..0b2118e 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticcaptions/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
index 75b7bac..809a74e 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
index 75b7bac..809a74e 100644
--- a/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
+++ b/tests/snapshots/test_app/test_ask_rtr_text_semanticranker/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_ask_session_state_persists/client0/result.json b/tests/snapshots/test_app/test_ask_session_state_persists/client0/result.json
new file mode 100644
index 0000000..f194c95
--- /dev/null
+++ b/tests/snapshots/test_app/test_ask_session_state_persists/client0/result.json
@@ -0,0 +1,20 @@
+{
+    "choices": [
+        {
+            "context": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
+                "role": "assistant"
+            },
+            "session_state": {
+                "conversation_id": 1234
+            }
+        }
+    ],
+    "object": "chat.completion"
+}
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_ask_session_state_persists/client1/result.json b/tests/snapshots/test_app/test_ask_session_state_persists/client1/result.json
new file mode 100644
index 0000000..f194c95
--- /dev/null
+++ b/tests/snapshots/test_app/test_ask_session_state_persists/client1/result.json
@@ -0,0 +1,20 @@
+{
+    "choices": [
+        {
+            "context": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Question:<br>What is the capital of France?<br><br>Prompt:<br>{'role': 'system', 'content': \"You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. Use 'you' to refer to the individual asking the questions even if they ask with 'I'. Answer the following question using only the data provided in the sources below. For tabular information return it as an html table. Do not return markdown format. Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. If you cannot answer using the sources below, say you don't know. Use below example to answer\"}\n\n{'role': 'user', 'content': \"\\n'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'\\n\\nSources:\\ninfo1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family.\\ninfo2.pdf: Overlake is in-network for the employee plan.\\ninfo3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.\\ninfo4.pdf: In-network institutions include Overlake, Swedish and others in the region\\n\"}\n\n{'role': 'assistant', 'content': 'In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].'}\n\n{'role': 'user', 'content': 'What is the capital of France?\\nSources:\\n Benefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
+                "role": "assistant"
+            },
+            "session_state": {
+                "conversation_id": 1234
+            }
+        }
+    ],
+    "object": "chat.completion"
+}
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_hybrid/client0/result.json b/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
index 01e955c..1bb8a6f 100644
--- a/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_hybrid/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_hybrid/client1/result.json b/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
index 01e955c..1bb8a6f 100644
--- a/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_hybrid/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json b/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
index 3e8c52e..5e1b172 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json b/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
index 3e8c52e..5e1b172 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json b/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
index 367f04b..c000a96 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template_concat/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json b/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
index 367f04b..c000a96 100644
--- a/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_prompt_template_concat/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_session_state_persists/client0/result.json b/tests/snapshots/test_app/test_chat_session_state_persists/client0/result.json
new file mode 100644
index 0000000..a9325a1
--- /dev/null
+++ b/tests/snapshots/test_app/test_chat_session_state_persists/client0/result.json
@@ -0,0 +1,20 @@
+{
+    "choices": [
+        {
+            "context": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
+                "role": "assistant"
+            },
+            "session_state": {
+                "conversation_id": 1234
+            }
+        }
+    ],
+    "object": "chat.completion"
+}
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_session_state_persists/client1/result.json b/tests/snapshots/test_app/test_chat_session_state_persists/client1/result.json
new file mode 100644
index 0000000..a9325a1
--- /dev/null
+++ b/tests/snapshots/test_app/test_chat_session_state_persists/client1/result.json
@@ -0,0 +1,20 @@
+{
+    "choices": [
+        {
+            "context": {
+                "data_points": [
+                    "Benefit_Options-2.pdf: There is a whistleblower policy."
+                ],
+                "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"
+            },
+            "message": {
+                "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
+                "role": "assistant"
+            },
+            "session_state": {
+                "conversation_id": 1234
+            }
+        }
+    ],
+    "object": "chat.completion"
+}
\ No newline at end of file
diff --git a/tests/snapshots/test_app/test_chat_stream_session_state_persists/client0/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_session_state_persists/client0/result.jsonlines
new file mode 100644
index 0000000..904ca7e
--- /dev/null
+++ b/tests/snapshots/test_app/test_chat_stream_session_state_persists/client0/result.jsonlines
@@ -0,0 +1,3 @@
+{"choices": [{"delta": {"role": "assistant"}, "context": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "session_state": {"conversation_id": 1234}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris. [Benefit_Options-2.pdf]."}}]}
diff --git a/tests/snapshots/test_app/test_chat_stream_session_state_persists/client1/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_session_state_persists/client1/result.jsonlines
new file mode 100644
index 0000000..904ca7e
--- /dev/null
+++ b/tests/snapshots/test_app/test_chat_stream_session_state_persists/client1/result.jsonlines
@@ -0,0 +1,3 @@
+{"choices": [{"delta": {"role": "assistant"}, "context": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "session_state": {"conversation_id": 1234}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
+{"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris. [Benefit_Options-2.pdf]."}}]}
diff --git a/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
index 1b6b452..a702fe7 100644
--- a/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text/client0/result.jsonlines
@@ -1,3 +1,3 @@
-{"choices": [{"delta": {"role": "assistant"}, "context": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"choices": [{"delta": {"role": "assistant"}, "context": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "session_state": null, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
 {"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
 {"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris. [Benefit_Options-2.pdf]."}}]}
diff --git a/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
index 1b6b452..a702fe7 100644
--- a/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text/client1/result.jsonlines
@@ -1,3 +1,3 @@
-{"choices": [{"delta": {"role": "assistant"}, "context": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"choices": [{"delta": {"role": "assistant"}, "context": {"data_points": ["Benefit_Options-2.pdf: There is a whistleblower policy."], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\nBenefit_Options-2.pdf: There is a whistleblower policy.'}"}, "session_state": null, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
 {"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
 {"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris. [Benefit_Options-2.pdf]."}}]}
diff --git a/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines b/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
index 232c896..9f0c584 100644
--- a/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
+++ b/tests/snapshots/test_app/test_chat_stream_text_filter/auth_client0/result.jsonlines
@@ -1,3 +1,3 @@
-{"choices": [{"delta": {"role": "assistant"}, "context": {"data_points": [], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"}, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
+{"choices": [{"delta": {"role": "assistant"}, "context": {"data_points": [], "thoughts": "Searched for:<br>capital of France<br><br>Conversations:<br>{'role': 'system', 'content': \"Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.\\nAnswer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.\\nFor tabular information return it as an html table. Do not return markdown format. If the question is not in English, answer in the language used in the question.\\nEach source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].\\n\\n\\n\"}<br><br>{'role': 'user', 'content': 'What is the capital of France?\\n\\nSources:\\n'}"}, "session_state": null, "finish_reason": null, "index": 0}], "object": "chat.completion.chunk"}
 {"object": "chat.completion.chunk", "choices": [{"delta": {"role": "assistant"}}]}
 {"object": "chat.completion.chunk", "choices": [{"delta": {"content": "The capital of France is Paris. [Benefit_Options-2.pdf]."}}]}
diff --git a/tests/snapshots/test_app/test_chat_text/client0/result.json b/tests/snapshots/test_app/test_chat_text/client0/result.json
index 01e955c..1bb8a6f 100644
--- a/tests/snapshots/test_app/test_chat_text/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_text/client1/result.json b/tests/snapshots/test_app/test_chat_text/client1/result.json
index 01e955c..1bb8a6f 100644
--- a/tests/snapshots/test_app/test_chat_text/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json b/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
index 5b905a7..6dfbd05 100644
--- a/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_filter/auth_client0/result.json
@@ -8,7 +8,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
index 95a7485..47b36ea 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
index 95a7485..47b36ea 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticcaptions/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json b/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
index 01e955c..1bb8a6f 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticranker/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json b/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
index 01e955c..1bb8a6f 100644
--- a/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_text_semanticranker/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_vector/client0/result.json b/tests/snapshots/test_app/test_chat_vector/client0/result.json
index 4cdb616..60234c9 100644
--- a/tests/snapshots/test_app/test_chat_vector/client0/result.json
+++ b/tests/snapshots/test_app/test_chat_vector/client0/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/snapshots/test_app/test_chat_vector/client1/result.json b/tests/snapshots/test_app/test_chat_vector/client1/result.json
index 4cdb616..60234c9 100644
--- a/tests/snapshots/test_app/test_chat_vector/client1/result.json
+++ b/tests/snapshots/test_app/test_chat_vector/client1/result.json
@@ -10,7 +10,8 @@
             "message": {
                 "content": "The capital of France is Paris. [Benefit_Options-2.pdf].",
                 "role": "assistant"
-            }
+            },
+            "session_state": null
         }
     ],
     "object": "chat.completion"
diff --git a/tests/test_app.py b/tests/test_app.py
index 2ecb7bb..ae110dd 100644
--- a/tests/test_app.py
+++ b/tests/test_app.py
@@ -325,6 +325,59 @@ async def test_chat_stream_text_filter(auth_client, snapshot):
     snapshot.assert_match(result, "result.jsonlines")
 
 
+@pytest.mark.asyncio
+async def test_ask_session_state_persists(client, snapshot):
+    response = await client.post(
+        "/ask",
+        json={
+            "messages": [{"content": "What is the capital of France?", "role": "user"}],
+            "context": {
+                "overrides": {"retrieval_mode": "text"},
+            },
+            "session_state": {"conversation_id": 1234},
+        },
+    )
+    assert response.status_code == 200
+    result = await response.get_json()
+    snapshot.assert_match(json.dumps(result, indent=4), "result.json")
+
+
+@pytest.mark.asyncio
+async def test_chat_session_state_persists(client, snapshot):
+    response = await client.post(
+        "/chat",
+        json={
+            "messages": [{"content": "What is the capital of France?", "role": "user"}],
+            "context": {
+                "overrides": {"retrieval_mode": "text"},
+            },
+            "stream": False,
+            "session_state": {"conversation_id": 1234},
+        },
+    )
+    assert response.status_code == 200
+    result = await response.get_json()
+    snapshot.assert_match(json.dumps(result, indent=4), "result.json")
+
+
+@pytest.mark.asyncio
+async def test_chat_stream_session_state_persists(client, snapshot):
+    response = await client.post(
+        "/chat",
+        json={
+            "messages": [{"content": "What is the capital of France?", "role": "user"}],
+            "context": {
+                "overrides": {"retrieval_mode": "text"},
+            },
+            "stream": True,
+            "session_state": {"conversation_id": 1234},
+        },
+    )
+    assert response.status_code == 200
+    result = await response.get_data()
+    snapshot.assert_match(result, "result.jsonlines")
+
+
 @pytest.mark.asyncio
 async def test_format_as_ndjson():
     async def gen():
diff --git a/tests/test_chatapproach.py b/tests/test_chatapproach.py
index 4dc40d2..5bf779c 100644
--- a/tests/test_chatapproach.py
+++ b/tests/test_chatapproach.py
@@ -4,7 +4,9 @@ from approaches.chatreadretrieveread import ChatReadRetrieveReadApproach
 
 
 def test_get_search_query():
-    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+    chat_approach = ChatReadRetrieveReadApproach(
+        None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "", "en-us", "lexicon"
+    )
 
     payload = '{"id":"chatcmpl-81JkxYqYppUkPtOAia40gki2vJ9QM","object":"chat.completion","created":1695324963,"model":"gpt-35-turbo","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[{"index":0,"finish_reason":"function_call","message":{"role":"assistant","function_call":{"name":"search_sources","arguments":"{\\n\\"search_query\\":\\"accesstelemedicineservices\\"\\n}"}},"content_filter_results":{}}],"usage":{"completion_tokens":19,"prompt_tokens":425,"total_tokens":444}}'
     default_query = "hello"
@@ -14,7 +16,9 @@ def test_get_search_query():
 
 
 def test_get_search_query_returns_default():
-    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+    chat_approach = ChatReadRetrieveReadApproach(
+        None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "", "en-us", "lexicon"
+    )
 
     payload = '{"id":"chatcmpl-81JkxYqYppUkPtOAia40gki2vJ9QM","object":"chat.completion","created":1695324963,"model":"gpt-35-turbo","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[{"index":0,"finish_reason":"function_call","message":{"role":"assistant"},"content_filter_results":{}}],"usage":{"completion_tokens":19,"prompt_tokens":425,"total_tokens":444}}'
     default_query = "hello"
@@ -24,7 +28,9 @@ def test_get_search_query_returns_default():
 
 
 def test_get_messages_from_history():
-    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+    chat_approach = ChatReadRetrieveReadApproach(
+        None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "", "en-us", "lexicon"
+    )
 
     messages = chat_approach.get_messages_from_history(
         system_prompt="You are a bot.",
@@ -50,7 +56,9 @@ def test_get_messages_from_history():
 
 
 def test_get_messages_from_history_truncated():
-    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+    chat_approach = ChatReadRetrieveReadApproach(
+        None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "", "en-us", "lexicon"
+    )
 
     messages = chat_approach.get_messages_from_history(
         system_prompt="You are a bot.",
@@ -72,7 +80,9 @@ def test_get_messages_from_history_truncated():
 
 
 def test_get_messages_from_history_truncated_longer():
-    chat_approach = ChatReadRetrieveReadApproach(None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "")
+    chat_approach = ChatReadRetrieveReadApproach(
+        None, "", "gpt-35-turbo", "gpt-35-turbo", "", "", "", "", "en-us", "lexicon"
+    )
 
     messages = chat_approach.get_messages_from_history(
         system_prompt="You are a bot.",