Repository: Azure-Samples/azure-search-openai-demo (python)

Issue:
Error in ingestion of .docx file type due ocrHighResolution when using Content Understanding.
<!--
IF SUFFICIENT INFORMATION IS NOT PROVIDED VIA THE FOLLOWING TEMPLATE THE ISSUE MIGHT BE CLOSED WITHOUT FURTHER CONSIDERATION OR INVESTIGATION
-->
> Please provide us with the following information:
> ---------------------------------------------------------------

### This issue is for a: (mark with an `x`)
```
- [ x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
```

### Minimal steps to reproduce
>scripts/prepdocs.ps1

### Any log messages given by the failure
>
 INFO     Extracting text from 'C:\Users\USER\Desktop\mco-vision/data\Document\Sample.docx' using Azure Document         pdfparser.py:66
                    Intelligence
Traceback (most recent call last):
  File "C:\Users\USER\Desktop\mco-vision\app\backend\prepdocs.py", line 439, in <module>
    loop.run_until_complete(main(ingestion_strategy, setup_index=not args.remove and not args.removeall))
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 650, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\USER\Desktop\mco-vision\app\backend\prepdocs.py", line 244, in main
    await strategy.run()
  File "C:\Users\USER\Desktop\mco-vision\app\backend\prepdocslib\filestrategy.py", line 101, in run
    sections = await parse_file(file, self.file_processors, self.category, self.image_embeddings)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Desktop\mco-vision\app\backend\prepdocslib\filestrategy.py", line 29, in parse_file
    pages = [page async for page in processor.parser.parse(content=file.content)]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Desktop\mco-vision\app\backend\prepdocslib\filestrategy.py", line 29, in <listcomp>
    pages = [page async for page in processor.parser.parse(content=file.content)]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Desktop\mco-vision\app\backend\prepdocslib\pdfparser.py", line 80, in parse
    poller = await document_intelligence_client.begin_analyze_document(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Desktop\mco-vision\.venv\Lib\site-packages\azure\core\tracing\decorator_async.py", line 94, in wrapper_use_tracer
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Desktop\mco-vision\.venv\Lib\site-packages\azure\ai\documentintelligence\aio\_operations\_patch.py", line 529, in begin_analyze_document
    raw_result = await self._analyze_document_initial(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Desktop\mco-vision\.venv\Lib\site-packages\azure\ai\documentintelligence\aio\_operations\_operations.py", line 158, in _analyze_document_initial
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: (InvalidArgument) Invalid argument.
Code: InvalidArgument
Message: Invalid argument.
Inner error: {
    "code": "InvalidParameter",
    "message": "The parameter ocrHighResolution for file type Docx is invalid: The feature is invalid or not supported."
}

### Expected/desired behavior
>

### OS and Version?
> Windows 11

### azd version?
> azd version 1.11.0 (commit 5b92e0687e1fa96dfc8292f4b900c0c58610b6a5)

### Versions
>

### Mention any other details that might be useful
Please let me know what resolution for this should be, as in the previous releases the .docx easily got ingested.
But when used with Content Understanding it gives the ocrHighResolution error.
It is a critical issue, your help is appreciated.
> ---------------------------------------------------------------
> Thanks! We'll be in touch soon.


Pull request:
Handle lack of support for docx/pptx/xlsx for media description
## Purpose

Fixes #2242 

Unfortunately, the figures extraction doesn't yet work for office documents. This PR makes the flow default to standard ingestion for them. We could also have it error out entirely, like it does now, but with a more helpful message. Am not sure what developers would prefer.

Still working on the unit test

## Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.

```
[ ] Yes
[X] No
```

## Does this require changes to learn.microsoft.com docs?

This repository is referenced by [this tutorial](https://learn.microsoft.com/azure/developer/python/get-started-app-chat-template)
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

```
[ ] Yes
[X] No
```

## Type of change

```
[X] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:
```

## Code quality checklist

See [CONTRIBUTING.md](https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/CONTRIBUTING.md#submit-pr) for more details.

- [ ] The current tests all pass (`python -m pytest`).
- [ ] I added tests that prove my fix is effective or that my feature works
- [ ] I ran `python -m pytest --cov` to verify 100% coverage of added lines
- [ ] I ran `python -m mypy` to check for type errors
- [ ] I either used the pre-commit hooks or ran `ruff` and `black` manually on my code.


Patch:
diff --git a/app/backend/prepdocslib/pdfparser.py b/app/backend/prepdocslib/pdfparser.py
index 1fcbbc9..a40e69b 100644
--- a/app/backend/prepdocslib/pdfparser.py
+++ b/app/backend/prepdocslib/pdfparser.py
@@ -14,6 +14,7 @@ from azure.ai.documentintelligence.models import (
 )
 from azure.core.credentials import AzureKeyCredential
 from azure.core.credentials_async import AsyncTokenCredential
+from azure.core.exceptions import HttpResponseError
 from PIL import Image
 from pypdf import PdfReader
 
@@ -68,6 +69,7 @@ class DocumentAnalysisParser(Parser):
         async with DocumentIntelligenceClient(
             endpoint=self.endpoint, credential=self.credential
         ) as document_intelligence_client:
+            file_analyzed = False
             if self.use_content_understanding:
                 if self.content_understanding_endpoint is None:
                     raise ValueError("Content Understanding is enabled but no endpoint was provided")
@@ -77,15 +79,29 @@ class DocumentAnalysisParser(Parser):
                     )
                 cu_describer = ContentUnderstandingDescriber(self.content_understanding_endpoint, self.credential)
                 content_bytes = content.read()
-                poller = await document_intelligence_client.begin_analyze_document(
-                    model_id="prebuilt-layout",
-                    analyze_request=AnalyzeDocumentRequest(bytes_source=content_bytes),
-                    output=["figures"],
-                    features=["ocrHighResolution"],
-                    output_content_format="markdown",
-                )
-                doc_for_pymupdf = pymupdf.open(stream=io.BytesIO(content_bytes))
-            else:
+                try:
+                    poller = await document_intelligence_client.begin_analyze_document(
+                        model_id="prebuilt-layout",
+                        analyze_request=AnalyzeDocumentRequest(bytes_source=content_bytes),
+                        output=["figures"],
+                        features=["ocrHighResolution"],
+                        output_content_format="markdown",
+                    )
+                    doc_for_pymupdf = pymupdf.open(stream=io.BytesIO(content_bytes))
+                    file_analyzed = True
+                except HttpResponseError as e:
+                    content.seek(0)
+                    if e.error and e.error.code == "InvalidArgument":
+                        logger.error(
+                            "This document type does not support media description. Proceeding with standard analysis."
+                        )
+                    else:
+                        logger.error(
+                            "Unexpected error analyzing document for media description: %s. Proceeding with standard analysis.",
+                            e,
+                        )
+
+            if file_analyzed is False:
                 poller = await document_intelligence_client.begin_analyze_document(
                     model_id=self.model_id, analyze_request=content, content_type="application/octet-stream"
                 )
diff --git a/docs/deploy_features.md b/docs/deploy_features.md
index ea0c7e8..7c5f2a4 100644
--- a/docs/deploy_features.md
+++ b/docs/deploy_features.md
@@ -163,7 +163,6 @@ By default, if your documents contain image-like figures, the data ingestion pro
 so users will not be able to ask questions about them.
 
 You can optionably enable the description of media content using Azure Content Understanding. When enabled, the data ingestion process will send figures to Azure Content Understanding and replace the figure with the description in the indexed document.
-To learn more about this process and compare it to the gpt-4 vision integration, see [this guide](./data_ingestion.md#media-description).
 
 To enable media description with Azure Content Understanding, run:
 
@@ -175,6 +174,9 @@ If you have already run `azd up`, you will need to run `azd provision` to create
 If you have already indexed your documents and want to re-index them with the media descriptions,
 first [remove the existing documents](./data_ingestion.md#removing-documents) and then [re-ingest the data](./data_ingestion.md#indexing-additional-documents).
 
+⚠️ This feature does not yet support DOCX, PPTX, or XLSX formats. If you have figures in those formats, they will be ignored.
+Convert them first to PDF or image formats to enable media description.
+
 ## Enabling client-side chat history
 
 This feature allows users to view the chat history of their conversation, stored in the browser using [IndexedDB](https://developer.mozilla.org/docs/Web/API/IndexedDB_API). That means the chat history will be available only on the device where the chat was initiated. To enable browser-stored chat history, run:


Test patch:
diff --git a/tests/test_pdfparser.py b/tests/test_pdfparser.py
index 408aa2b..2f21fe6 100644
--- a/tests/test_pdfparser.py
+++ b/tests/test_pdfparser.py
@@ -1,8 +1,9 @@
 import io
+import json
 import logging
 import math
 import pathlib
-from unittest.mock import AsyncMock, MagicMock
+from unittest.mock import AsyncMock, MagicMock, Mock
 
 import pymupdf
 import pytest
@@ -17,6 +18,7 @@ from azure.ai.documentintelligence.models import (
     DocumentTable,
     DocumentTableCell,
 )
+from azure.core.exceptions import HttpResponseError
 from PIL import Image, ImageChops
 
 from prepdocslib.mediadescriber import ContentUnderstandingDescriber
@@ -308,3 +310,63 @@ async def test_parse_doc_with_figures(monkeypatch):
         pages[0].text
         == "# Simple Figure\n\nThis text is before the figure and NOT part of it.\n\n\n<figure><figcaption>Figure 1<br>Pie chart</figcaption></figure>\n\n\nThis is text after the figure that's not part of it."
     )
+
+
+@pytest.mark.asyncio
+async def test_parse_unsupportedformat(monkeypatch, caplog):
+    mock_poller = MagicMock()
+
+    async def mock_begin_analyze_document(self, model_id, analyze_request, **kwargs):
+
+        if kwargs.get("features") == ["ocrHighResolution"]:
+
+            class FakeErrorOne:
+                def __init__(self):
+                    self.error = Mock(message="A fake error", code="FakeErrorOne")
+
+            class FakeHttpResponse(HttpResponseError):
+                def __init__(self, response, error, *args, **kwargs):
+                    self.error = error
+                    super().__init__(self, response=response, *args, **kwargs)
+
+            message = {
+                "error": {
+                    "code": "InvalidArgument",
+                    "message": "A fake error",
+                }
+            }
+            response = Mock(status_code=500, headers={})
+            response.text = lambda encoding=None: json.dumps(message).encode("utf-8")
+            response.headers["content-type"] = "application/json"
+            response.content_type = "application/json"
+            raise FakeHttpResponse(response, FakeErrorOne())
+        else:
+            return mock_poller
+
+    async def mock_poller_result():
+        return AnalyzeResult(
+            content="Page content",
+            pages=[DocumentPage(page_number=1, spans=[DocumentSpan(offset=0, length=12)])],
+            tables=[],
+            figures=[],
+        )
+
+    monkeypatch.setattr(DocumentIntelligenceClient, "begin_analyze_document", mock_begin_analyze_document)
+    monkeypatch.setattr(mock_poller, "result", mock_poller_result)
+
+    parser = DocumentAnalysisParser(
+        endpoint="https://example.com",
+        credential=MockAzureCredential(),
+        use_content_understanding=True,
+        content_understanding_endpoint="https://example.com",
+    )
+    content = io.BytesIO(b"pdf content bytes")
+    content.name = "test.docx"
+    with caplog.at_level(logging.ERROR):
+        pages = [page async for page in parser.parse(content)]
+        assert "This document type does not support media description." in caplog.text
+
+    assert len(pages) == 1
+    assert pages[0].page_num == 0
+    assert pages[0].offset == 0
+    assert pages[0].text == "Page content"