Repository: microsoft/PubSec-Info-Assistant

Issue:
Answers do not appear to be created from chunks returned
**Describe the bug**
You may ask the system a question at the start of a chat (or mid-chat) and notice that the answer provided by the system is not based upon data contained within the response chunks provided to OpenAI service. This may happen when the answer given is correct, or when it is incorrect (unrelated to accuracy of the answer). 

**To Reproduce**
Steps to reproduce the behavior:
1. Ask a question that the system does not have information in documents to answer, such as "I use my boat for personal purposes only. Can I deduct the costs of the boat from my taxes?"
2. Note that the response may be factual, and may cite a document. 
3. Review the cited document and see that the data required is nowhere in the document. 
    a. Specific document used: https://files.taxfoundation.org/20210823155834/TaxEDU-Primer-Common-Tax-Questions-Answered.pdf
5. Review the Thought Process to confirm that no chunk provided contained the answer

**Expected behavior**
The system should say that it is unable to answer the question, and not cite any document or document chunk. 

**Screenshots**
![b7f86019-dd29-466c-8d09-f970611cb059](https://github.com/microsoft/PubSec-Info-Assistant/assets/93607917/d854cd07-ff92-4db3-a75a-2993b79e9b54)


**Desktop (please complete the following information):**
 - OS: Windows 11
 - Browser: Edge

**Alpha version details**
 - GitHub branch: main
 - [Tag](https://github.com/microsoft/PubSec-Info-Assistant/releases/tag/v0.1-Alpha)

**Additional context**
Tracked in AzDO 5791



Discussion:

Resolving patch:
diff --git a/app/backend/approaches/chatreadretrieveread.py b/app/backend/approaches/chatreadretrieveread.py
--- a/app/backend/approaches/chatreadretrieveread.py
+++ b/app/backend/approaches/chatreadretrieveread.py
@@ -11,35 +11,75 @@
 import json
 import logging
 from azure.storage.blob import BlobServiceClient, generate_account_sas, ResourceTypes, AccountSasPermissions
+import re
 
 # Simple retrieve-then-read implementation, using the Cognitive Search and OpenAI APIs directly. It first retrieves
 # top documents from search, then constructs a prompt with them, and then uses OpenAI to generate an completion 
 # (answer) with that prompt.
 class ChatReadRetrieveReadApproach(Approach):
     prompt_prefix = """

-    You are an Azure OpenAI Completion system. Your persona is {systemPersona} who helps answer questions about an agencies data. {response_length_prompt}
-    Answer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. For tabular information return it as an html table. Do not return markdown format.
+    You are an Azure OpenAI Completion system. Your persona is {systemPersona} who helps answer questions about an agency's data. {response_length_prompt}
+   
+   
+    Text:
+    Flight to Denver at 9:00 am tomorrow.
+
+    Prompt:
+    Question: Is my flight on time?
+
+    Steps:
+    1. Look for relevant information in the provided source document to answer the question.
+    2. If there is specific flight information available in the source document, provide an answer along with the appropriate citation.
+    3. If there is no information about the specific flight in the source document, respond with "I'm not sure" without providing any citation.
+    
+    
+    Response:
+
+    1. Look for relevant information in the provided source document to answer the question.
+    - Search for flight details matching the given flight to determine its current status.
+
+    2. If there is specific flight information available in the source document, provide an answer along with the appropriate citation.
+    - If the source document contains information about the current status of the specified flight, provide a response citing the relevant section of source documents. 
+    
+    
+    3. If there is no relevant information about the specific flight in the source document, respond with "I'm not sure" without providing any citation.
+    - If the source document does not contain information about specified flight and status, respond with "I'm not sure" as there is no basis to determine its current status.
+
+    Example Response:
+
+    Question: Is my flight on time?
+
+    <Response>I'm not sure. The provided source document does not include information about the current status of your specific flight.[no citations provided]</Response>
+    
+        
     User persona: {userPersona}
-    Each source has a file name followed by a pipe character and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].
+    Emphasize the use of facts listed in the provided source documents.Instruct the model to use source name for each fact used in the response.  Avoid generating speculative or generalized information. Each source has a file name followed by a pipe character and 
+    the actual information. Each source document has a file name followed by a pipe character and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, e.g. [info1.txt]. Don't combine sources, list each source separately, e.g. [info1.txt][info2.pdf].
+    Treat each search term as an individual keyword. Do not combine terms in quotes or brackets.
+    
+
+
     {follow_up_questions_prompt}
     {injected_prompt}
     Sources:
     {sources}
+    
+    
     

     {chat_history}
     """
-
     follow_up_questions_prompt_content = """
     Generate three very brief follow-up questions that the user would likely ask next about their agencies data. Use triple angle brackets to reference the questions, e.g. <<<Are there exclusions for prescriptions?>>>. Try not to repeat questions that have already been asked.
     Only generate questions and do not generate any text before or after the questions, such as 'Next Questions'
     """
 
     query_prompt_template = """
-    Below is a history of the conversation so far, and a new question asked by the user that needs to be answered by searching in a knowledge base.
+    Below is a history of the conversation so far, and a new question asked by the user that needs to be answered by searching in source documents
     Generate a search query based on the conversation and the new question. 
     Do not include cited source filenames and document names e.g info.txt or doc.pdf in the search query terms.
     Do not include any text inside [] or <<<>>> in the search query terms.
     If the question is not in English, translate the question to English before generating the search query.
+    Treat each search term as an individual keyword. Do not combine terms in quotes or brackets.
 
     Chat History:
     {chat_history}
@@ -49,6 +89,7 @@ class ChatReadRetrieveReadApproach(Approach):
 
     Search query:
     """
+   
 
     def __init__(self, search_client: SearchClient, oai_service_name: str, oai_service_key: str, chatgpt_deployment: str, gpt_deployment: str, sourcepage_field: str, content_field: str, blob_client: BlobServiceClient):
         self.search_client = search_client
@@ -61,6 +102,8 @@ def __init__(self, search_client: SearchClient, oai_service_name: str, oai_servi
         openai.api_base = 'https://' + oai_service_name + '.openai.azure.com/'
         openai.api_type = 'azure'
         openai.api_key = oai_service_key
+        
+       
 
 
     def run(self, history: list[dict], overrides: dict) -> any:
@@ -87,10 +130,12 @@ def run(self, history: list[dict], overrides: dict) -> any:
             n=1,
             stop=["\n"])
         q = completion.choices[0].text
+        
+        generated_query = q.strip('\"')
 
         # STEP 2: Retrieve relevant documents from the search index with the GPT optimized query
         if overrides.get("semantic_ranker"):
-            r = self.search_client.search(q,
+            r = self.search_client.search(generated_query,
                                           filter=filter,
                                           query_type=QueryType.SEMANTIC,
                                           query_language="en-us",
@@ -99,12 +144,19 @@ def run(self, history: list[dict], overrides: dict) -> any:
                                           top=top,
                                           query_caption="extractive|highlight-false" if use_semantic_captions else None)
         else:
-            r = self.search_client.search(q, filter=filter, top=top)
+            r = self.search_client.search(generated_query, filter=filter, top=top)
 
+               
+    
+            
         citation_lookup = {}  # dict of "FileX" monikor to the actual file name
         results = []  # list of results to be used in the prompt
         data_points = []  # list of data points to be used in the response
+        
+       
+            
         for idx, doc in enumerate(r):  # for each document in the search results
+           
             if use_semantic_captions:
                 # if using semantic captions, use the captions instead of the content
                 # include the "FileX" monikor in the prompt, and the actual file name in the response
@@ -168,6 +220,9 @@ def run(self, history: list[dict], overrides: dict) -> any:
             n=1,
             stop=["
", "
"]
         )
+  
+
+
 
         return {
             "data_points": data_points,


Test patch: