Repository: microsoft/PubSec-Info-Assistant

Issue:
Problems with foreign characters
**Describe the bug**
Foreign characters (danish "æøå" in this case) are incorrectly displayed in "Citation -> Document section -> Content". For instance, "å"  is shown as "Ã¥". Looks like a mixup of utf-8 and 16.

**To Reproduce**
Steps to reproduce the behavior:
- Upload a html document stored in utf-8 w. bom. In that file foreign characters are not escaped (and shouldn't be, because utf-8).

**Expected behavior**
The user should be presented with the proper characters in the UI.

**Alpha version details**
 - GitHub branch: main (gamma)
 - Latest commit: 594c9658d03c58f77661a63b3015ec9bbd5acfe0

**Additional context**




Discussion:
It seems that unicode characters are "double encoded" somewhere in the proces. Using this piece of c# I can get the correct text:

```
var bt = File.ReadAllBytes("some-file-from-content-storage-container.json");
var s = System.Text.Encoding.UTF8.GetString(bt);

bt = System.Text.Encoding.Latin1.GetBytes(s);
s = System.Text.Encoding.UTF8.GetString(bt);
Console.WriteLine(s);
```
@AThomsen , thanks for the feedback. I have opened an internal ticket on this for the team to investigate. We will post updates here when we have them. 

Resolving patch:
diff --git a/functions/FileLayoutParsingOther/__init__.py b/functions/FileLayoutParsingOther/__init__.py
--- a/functions/FileLayoutParsingOther/__init__.py
+++ b/functions/FileLayoutParsingOther/__init__.py
@@ -65,7 +65,10 @@ def main(msg: func.QueueMessage) -> None:
             statusLog.upsert_document(blob_name, f'{function_name} - HTML generated from DocX by mammoth', StatusClassification.DEBUG)
             html = result.value # The generated HTML
         else:
-            html = response.text 
+            # Extract the content and detect the charset to decode
+            html_content = response.content
+            detected_charset = utilities.extract_charset(html_content)
+            html = html_content.decode(detected_charset)
                             
         # build the document map from HTML for all non-pdf file types
         statusLog.upsert_document(blob_name, f'{function_name} - Starting document map build', StatusClassification.DEBUG)
diff --git a/functions/shared_code/utilities.py b/functions/shared_code/utilities.py
--- a/functions/shared_code/utilities.py
+++ b/functions/shared_code/utilities.py
@@ -431,4 +431,22 @@ def build_chunks(self, document_map, myblob_name, myblob_uri, chunk_target_size)
 
         logging.info("Chunking is complete \n")
         return chunk_count
+    
+    # Function to detect and extract the charset from the HTML content
+    def extract_charset(self, html_content):
+        soup = BeautifulSoup(html_content, 'html.parser')
+        meta_tags = soup.find_all('meta', charset=True)
+    
+        if meta_tags:
+            return meta_tags[0]['charset'].strip().lower()
+    
+        # If no charset is specified in meta tags, try to detect from the Content-Type header
+        content_type = soup.find('meta', {'http-equiv': 'Content-Type'})
+        if content_type and 'content' in content_type.attrs:
+            content_type_value = content_type['content']
+            if 'charset' in content_type_value:
+                return content_type_value.split('charset=')[-1].strip().lower()
+    
+        # Default to utf-8 if charset is not specified
+        return 'utf-8'
     
\ No newline at end of file


Test patch: