Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] HTTP 500 in Tika for some OpenDocument Documents
### Description

When recreating my local paperless instance, I had to rerun the document archival and thumbnail jobs for all documents.

Those failed for all OpenDocument Documents from my originals. 

I noticed this being related to the latest changes in paperless_tika/parser.py. After testing the parser.py from 1.15 all worked.

I am already working on further investigation on my fork and will start the corresponding pull request.



### Steps to reproduce

upload ODT document -> parsing fails

### Webserver logs

```bash
---TIKA---
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Stream Closed
	at java.base/java.io.FileInputStream.available0(Native Method)
	at java.base/java.io.FileInputStream.available(FileInputStream.java:415)
	at org.apache.cxf.attachment.DelegatingInputStream.available(DelegatingInputStream.java:75)
	at org.apache.cxf.helpers.IOUtils.consume(IOUtils.java:382)
	at org.apache.cxf.attachment.DelegatingInputStream.close(DelegatingInputStream.java:46)
	at org.apache.tika.server.core.resource.TikaResource.parse(TikaResource.java:374)
	at org.apache.tika.server.core.resource.TikaResource.parseToMetadata(TikaResource.java:611)
	at org.apache.tika.server.core.resource.TikaResource.getJsonFromMultipart(TikaResource.java:564)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
	... 28 more

---PAPERLESS---
[2023-07-06 17:57:46,034] [INFO] [paperless.consumer] Consuming 2023-03-28 Johannes Gutenberg-Universität Mainz Datenbank Praktikum Gruppe 3.odt
[2023-07-06 17:57:46,112] [INFO] [paperless.parsing.tika] Sending /tmp/paperless/paperless-ngxzed2cbo7/2023-03-28 Johannes Gutenberg-Universität Mainz Datenbank Praktikum Gruppe 3.odt to Tika server
[2023-07-06 17:57:47,289] [ERROR] [paperless.consumer] Error while consuming document 2023-03-28 Johannes Gutenberg-Universität Mainz Datenbank Praktikum Gruppe 3.odt: Could not parse /tmp/paperless/paperless-ngxzed2cbo7/2023-03-28 Johannes Gutenberg-Universität Mainz Datenbank Praktikum Gruppe 3.odt with tika server at http://localhost:9998: Server error '500 Server Error' for url 'http://localhost:9998/tika/form/text'
For more information check: https://httpstatuses.com/500
Traceback (most recent call last):
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 54, in parse
    parsed = client.tika.as_text.from_file(document_path, mime_type)
  File "/usr/local/lib/python3.9/site-packages/tika_client/_resource_tika.py", line 36, in from_file
    return self._decoded_response(self._put_multipart(self.MULTI_PART_PLAIN_TEXT_CONTENT, filepath, mime_type))
  File "/usr/local/lib/python3.9/site-packages/tika_client/_utils.py", line 46, in _put_multipart
    resp.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Server Error' for url 'http://localhost:9998/tika/form/text'
For more information check: https://httpstatuses.com/500

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/paperless/src/documents/consumer.py", line 382, in try_consume_file
    document_parser.parse(self.path, mime_type, self.filename)
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 56, in parse
    raise ParseError(
documents.parsers.ParseError: Could not parse /tmp/paperless/paperless-ngxzed2cbo7/2023-03-28 Johannes Gutenberg-Universität Mainz Datenbank Praktikum Gruppe 3.odt with tika server at http://localhost:9998: Server error '500 Server Error' for url 'http://localhost:9998/tika/form/text'
For more information check: https://httpstatuses.com/500
[2023-07-06 17:57:47,309] [ERROR] [celery.app.trace] Task documents.tasks.consume_file[1a0c8479-65a4-4de7-a431-29ecb537a030] raised unexpected: ConsumerError("2023-03-28 Johannes Gutenberg-Universität Mainz Datenbank Praktikum Gruppe 3.odt: Error while consuming document 2023-03-28 Johannes Gutenberg-Universität Mainz Datenbank Praktikum Gruppe 3.odt: Could not parse /tmp/paperless/paperless-ngxzed2cbo7/2023-03-28 Johannes Gutenberg-Universität Mainz Datenbank Praktikum Gruppe 3.odt with tika server at http://localhost:9998: Server error '500 Server Error' for url 'http://localhost:9998/tika/form/text'\nFor more information check: https://httpstatuses.com/500")
Traceback (most recent call last):
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 54, in parse
    parsed = client.tika.as_text.from_file(document_path, mime_type)
  File "/usr/local/lib/python3.9/site-packages/tika_client/_resource_tika.py", line 36, in from_file
    return self._decoded_response(self._put_multipart(self.MULTI_PART_PLAIN_TEXT_CONTENT, filepath, mime_type))
  File "/usr/local/lib/python3.9/site-packages/tika_client/_utils.py", line 46, in _put_multipart
    resp.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Server Error' for url 'http://localhost:9998/tika/form/text'
For more information check: https://httpstatuses.com/500
```


### Browser logs

_No response_

### Paperless-ngx version

1.16.5

### Host OS

Truenas

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
The cause seems to be the switch from the tika client library to tika-client. The later does not work for all document types. Was there a specific reason for the switch? 
Yes i suspect this is related to that. The reasons for the change are explained here https://github.com/paperless-ngx/paperless-ngx/pull/3517
THanks for the info @shamoon , going a different route then, than my first PR... for me the issue is actually quite major, as no Open Document files can be parsed by paperless anymore..
The new library includes testing with odt files and works there without issue.  Is this some other document type?  Can you send a sample?

And just to note, a 500 is Tika crashing internally, so it might be something in there.  But with a sample, I'll look at it in a couple weeks or so
You can search/report Tika issues here: https://issues.apache.org/jira/projects/TIKA/issues/

I'll also get the paperless image updated to v2.8.0 at some point this week, in case it's something already fixed.
The paperless Tika image is updated to 2.8.0, you can give that a shot.

You can also try to `curl` the file to Tika and see what happens.  Something like:
`curl -v -F 'upload=@sample.odt' -H "Accept: application/json" "http://localhost:9998/tika/form" | jq`
I haven't been able to reproduce this with ODT files from LibreOffice or Google Docs nor DOCX from Google Docs or Microsoft Word.  I tried spreadsheets as well and against 1.16 and dev images.
That's a different error and already fixed.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Several ODT files work with version 1.17, but not all. The issue seems to be related to images in the files. Removing the images resolves the problem.

**text+image_broken.odt**
`Could not parse /tmp/paperless/paperless-ngxkgf_v9p2/text+image_broken.odt with tika server at http://tika:9998: Server error '500 Server Error' for url 'http://tika:9998/tika/form/text' For more information check: https://httpstatuses.com/500`


<details>
<summary>paperless.log</summary>

```
[2023-08-05 01:05:53,148] [INFO] [paperless.consumer] Consuming text+image_broken.odt
[2023-08-05 01:05:53,159] [DEBUG] [paperless.consumer] Detected mime type: application/vnd.oasis.opendocument.text
[2023-08-05 01:05:53,242] [DEBUG] [paperless.consumer] Parser: TikaDocumentParser
[2023-08-05 01:05:53,254] [DEBUG] [paperless.consumer] Parsing text+image_broken.odt...
[2023-08-05 01:05:53,262] [INFO] [paperless.parsing.tika] Sending /tmp/paperless/paperless-ngxkgf_v9p2/text+image_broken.odt to Tika server
[2023-08-05 01:05:53,345] [DEBUG] [paperless.parsing.tika] Deleting directory /tmp/paperless/paperless-qbx1zmmq
[2023-08-05 01:05:53,358] [ERROR] [paperless.consumer] Error while consuming document text+image_broken.odt: Could not parse /tmp/paperless/paperless-ngxkgf_v9p2/text+image_broken.odt with tika server at http://tika:9998: Server error '500 Server Error' for url 'http://tika:9998/tika/form/text'

For more information check: https://httpstatuses.com/500

Traceback (most recent call last):
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 55, in parse
    parsed = client.tika.as_text.from_file(document_path, mime_type)
  File "/usr/local/lib/python3.9/site-packages/tika_client/_resource_tika.py", line 36, in from_file
    return self._decoded_response(self._put_multipart(self.MULTI_PART_PLAIN_TEXT_CONTENT, filepath, mime_type))
  File "/usr/local/lib/python3.9/site-packages/tika_client/_utils.py", line 46, in _put_multipart
    resp.raise_for_status()
  File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Server Error' for url 'http://tika:9998/tika/form/text'

For more information check: https://httpstatuses.com/500

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/paperless/src/documents/consumer.py", line 424, in try_consume_file
    document_parser.parse(self.path, mime_type, self.filename)
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 57, in parse
    raise ParseError(
documents.parsers.ParseError: Could not parse /tmp/paperless/paperless-ngxkgf_v9p2/text+image_broken.odt with tika server at http://tika:9998: Server error '500 Server Error' for url 'http://tika:9998/tika/form/text'

For more information check: https://httpstatuses.com/500
```
</details>

[text only_works.odt](https://github.com/paperless-ngx/paperless-ngx/files/12265096/text.only_works.odt)
[text+image_broken.odt](https://github.com/paperless-ngx/paperless-ngx/files/12265097/text%2Bimage_broken.odt)


That's useful to know, though it's still Tika crashing in this case.  And using the broken file via basic `curl` also returns a 500 error, so it's not something `httpx` is doing wrong either.  Somehow, content vs multi-part breaks tika.  I'll jump through the hoops to report it upstream there.
Not particularly easy to get an account, but I've created an upstream for this: https://issues.apache.org/jira/browse/TIKA-4110
No fixes with Tika 2.9.0.  And no comments or updates on the created bug report
With the current GitHub checkout of the main branch of tika, I cannot reproduce this issue anymore. Even the second command, with the form, works:

Request: 

`curl -H "Accept: application/json" --form file=@sample.odt http://localhost:9998/tika/form`

Response:

`{"X-TIKA:Parsed-By":"org.apache.tika.parser.EmptyParser","X-TIKA:Parsed-By-Full-Set":"org.apache.tika.parser.EmptyParser","X-TIKA:content":"<html xmlns=\"http://www.w3.org/1999/xhtml\">\n<head>\n<meta name=\"X-TIKA:Parsed-By\" content=\"org.apache.tika.parser.EmptyParser\" />\n<meta name=\"Content-Length\" content=\"130012\" />\n<meta name=\"Content-Type\" content=\"application/vnd.oasis.opendocument.text\" />\n<title></title>\n</head>\n<body /></html>","Content-Length":"130012","Content-Type":"application/vnd.oasis.opendocument.text"}`
Ah, now I see, I don't seem to be having any parser configured ... probably that's why it works.

Resolving patch:
diff --git a/src/paperless_tika/parsers.py b/src/paperless_tika/parsers.py
--- a/src/paperless_tika/parsers.py
+++ b/src/paperless_tika/parsers.py
@@ -52,7 +52,18 @@ def parse(self, document_path: Path, mime_type: str, file_name=None):
 
         try:
             with TikaClient(tika_url=settings.TIKA_ENDPOINT) as client:
-                parsed = client.tika.as_text.from_file(document_path, mime_type)
+                try:
+                    parsed = client.tika.as_text.from_file(document_path, mime_type)
+                except httpx.HTTPStatusError as err:
+                    # Workaround https://issues.apache.org/jira/browse/TIKA-4110
+                    # Tika fails with some files as multi-part form data
+                    if err.response.status_code == httpx.codes.INTERNAL_SERVER_ERROR:
+                        parsed = client.tika.as_text.from_buffer(
+                            document_path.read_bytes(),
+                            mime_type,
+                        )
+                    else:  # pragma: nocover
+                        raise
         except Exception as err:
             raise ParseError(
                 f"Could not parse {document_path} with tika server at "


Test patch:
diff --git a/src/paperless_tika/tests/samples/multi-part-broken.odt b/src/paperless_tika/tests/samples/multi-part-broken.odt
new file mode 100644
Binary files /dev/null and b/src/paperless_tika/tests/samples/multi-part-broken.odt differ
diff --git a/src/paperless_tika/tests/test_live_tika.py b/src/paperless_tika/tests/test_live_tika.py
--- a/src/paperless_tika/tests/test_live_tika.py
+++ b/src/paperless_tika/tests/test_live_tika.py
@@ -111,3 +111,27 @@ def test_basic_parse_doc(self):
         self.assertIsNotNone(self.parser.archive_path)
         with open(self.parser.archive_path, "rb") as f:
             self.assertTrue(b"PDF-" in f.read()[:10])
+
+    def test_tika_fails_multi_part(self):
+        """
+        GIVEN:
+            - An input ODT format document
+            - The document is known to crash Tika when uploaded via multi-part form data
+        WHEN:
+            - The document is parsed
+        THEN:
+            - Document content is correct
+            - Document date is correct
+        See also:
+            - https://issues.apache.org/jira/browse/TIKA-4110
+        """
+        test_file = self.SAMPLE_DIR / "multi-part-broken.odt"
+
+        util_call_with_backoff(
+            self.parser.parse,
+            [test_file, "application/vnd.oasis.opendocument.text"],
+        )
+
+        self.assertIsNotNone(self.parser.archive_path)
+        with open(self.parser.archive_path, "rb") as f:
+            self.assertTrue(b"PDF-" in f.read()[:10])