Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] .pptx and .odp files are not being added to paperless
### Description

Currently paperless does not add the file when uploading or cunsuming a `.pptx` or `.odp` file. 

### Steps to reproduce


1. Create a new `.pptx` file, type anything inside. [test.pptx](https://github.com/paperless-ngx/paperless-ngx/files/11860142/test.pptx)
2. Upload/consume the `.pptx` file
3. See error


#### Compose
```
version: "3.4"
services:
  redis:
    image: docker.io/library/redis:7
    container_name: services.redis-paperless.example.com
    restart: unless-stopped
    volumes:
      - redisdata:/data
    networks:
      - paperless-network
  
  postgres:
    image: docker.io/library/postgres:13
    restart: unless-stopped
    container_name: services.postgres-paperless.example.com
    volumes:
      - ./db-data:/var/lib/postgresql/data #Exists only if you use PostgreSQL and contains the database.
    environment:
      POSTGRES_DB: paperless
      POSTGRES_USER: paperless
      POSTGRES_PASSWORD: paperless
    networks:
      - paperless-network

  paperless-ngx:
    image: ghcr.io/paperless-ngx/paperless-ngx:latest
    restart: unless-stopped
    container_name: services.paperless.example.com
    depends_on:
      - postgres
      - redis
      - gotenberg
      - tika
    #ports: 
      #- "8000:8000" # using nginx proxy manager
    volumes:
      - ./export:/usr/src/paperless/export 
      - ./data:/usr/src/paperless/data  # This is where auxillary data is stored. This folder also contains the SQLite database, if you use it.
      - /srv/compose/syncthing/syncthing/data/documents/paperless-media:/usr/src/paperless/media # This is where your documents are stored.
      - /srv/compose/syncthing/syncthing/data/documents/paperless-consume:/usr/src/paperless/consume # Paperless consumes documents placed here.
    env_file: docker-compose.env
    networks:
      - proxy-network
      - paperless-network
    environment:
      PAPERLESS_REDIS: redis://services.redis-paperless.example.com:6379
      PAPERLESS_DBHOST: services.postgres-paperless.example.com
      PAPERLESS_TIKA_ENABLED: 1
      PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://services.gotenberg-paperless.example.com:3000
      PAPERLESS_TIKA_ENDPOINT: http://services.tika-paperless.example.com:9998

  gotenberg:
    image: docker.io/gotenberg/gotenberg:7.8
    container_name: services.gotenberg-paperless.example.com
    networks:
      - paperless-network
    restart: unless-stopped
    # The gotenberg chromium route is used to convert .eml files. We do not
    # want to allow external content like tracking pixels or even javascript.
    command:
      - "gotenberg"
      - "--chromium-disable-javascript=true"
      - "--chromium-allow-list=file:///tmp/.*F"

  tika:
    image: ghcr.io/paperless-ngx/tika:latest
    container_name: services.tika-paperless.example.com
    restart: unless-stopped
    networks:
      - paperless-network

volumes:
  redisdata:

networks:
  paperless-network:
     name: paperless-network
  proxy-network:
    external: true
```
Relevant env variables
```
USERMAP_UID=1000
USERMAP_GID=1000
PAPERLESS_FILENAME_FORMAT_REMOVE_NONE=True 
#https://docs.paperless-ngx.com/advanced_usage/#file-name-handling
PAPERLESS_FILENAME_FORMAT={correspondent}/{created_year}/{document_type}/{created_year}-{created_month}-{title}
```
### Tika Logs
```
INFO  [qtp495051009-22] 15:35:34,770 org.apache.tika.server.core.resource.TikaResource /tika (application/vnd.openxmlformats-officedocument.presentationml.presentation)
```

### Webserver logs

```bash
[2023-06-25 18:35:34,661] [INFO] [paperless.consumer] Consuming test.pptx
[2023-06-25 18:35:34,737] [INFO] [paperless.parsing.tika] Sending /tmp/paperless/paperless-ngx6rsjscda/test.pptx to Tika server
[2023-06-25 18:35:34,832] [WARNING] [tika-client.utils] Under-specified content-type: application/vnd.openxmlformats-officedocument.presentationml.presentation
[2023-06-25 18:35:34,843] [ERROR] [celery.app.trace] Task documents.tasks.consume_file[793a90de-2b7d-4ab2-8616-8873db55a4fa] raised unexpected: AttributeError("'TikaResponse' object has no attribute 'created'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/src/paperless/src/documents/tasks.py", line 133, in consume_file
    document = Consumer().try_consume_file(
  File "/usr/src/paperless/src/documents/consumer.py", line 382, in try_consume_file
    document_parser.parse(self.path, mime_type, self.filename)
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 70, in parse
    self.date = parsed.created
AttributeError: 'TikaResponse' object has no attribute 'created'
```


### Paperless-ngx version

1.16.3

### Host OS

Debian LXC on Proxmox Host

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

docker-compose.yml, docker-compose.env

### Other

_No response_
[BUG] 1.16.3: Tika error ascii' codec can't encode character '\xfc' 
### Description

Consuming a specific MS Word 365 document fails with this error message:

`20230625 LBV Antrag Kostenerstattung für Meldebescheinigung Familienzuschlag.docx: Error while consuming document 20230625 LBV Antrag Kostenerstattung für Meldebescheinigung Familienzuschlag.docx: Could not parse /tmp/paperless/paperless-ngx6efyyzjg/20230625 LBV Antrag Kostenerstattung für Meldebescheinigung Familienzuschlag.docx with tika server at http://tika:9998: 'ascii' codec can't encode character '\xfc' in position 59: ordinal not in range(128)`

Unfortunately I cannot attach the document because it contains personal infos, sorry!

### Steps to reproduce

Upload MS Word docx into consume directory.

### Webserver logs

```bash
[2023-06-25 19:13:14,318] [ERROR] [paperless.consumer] Error while consuming document 20230625 LBV Antrag Kostenerstattung für Meldebescheinigung Familienzuschlag.docx: Could not parse /tmp/paperless/paperless-ngxk4ony929/20230625 LBV Antrag Kostenerstattung für Meldebescheinigung Familienzuschlag.docx with tika server at http://tika:9998: 'ascii' codec can't encode character '\xfc' in position 59: ordinal not in range(128)

Traceback (most recent call last):

  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 55, in parse

    parsed = client.tika.as_text.from_file(document_path, mime_type)

  File "/usr/local/lib/python3.9/site-packages/tika_client/_resource_tika.py", line 36, in from_file

    return self._decoded_response(self._put_multipart(self.MULTI_PART_PLAIN_TEXT_CONTENT, filepath, mime_type))

  File "/usr/local/lib/python3.9/site-packages/tika_client/_utils.py", line 34, in _put_multipart

    resp = self.client.post(

  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1132, in post

    return self.request(

  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 801, in request

    request = self.build_request(

  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 347, in build_request

    headers = self._merge_headers(headers)

  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 414, in _merge_headers

    merged_headers.update(headers)

  File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 200, in update

    headers = Headers(headers)

  File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 71, in __init__

    self._list = [

  File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 75, in <listcomp>

    normalize_header_value(v, encoding),

  File "/usr/local/lib/python3.9/site-packages/httpx/_utils.py", line 53, in normalize_header_value

    return value.encode(encoding or "ascii")

UnicodeEncodeError: 'ascii' codec can't encode character '\xfc' in position 59: ordinal not in range(128)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/usr/src/paperless/src/documents/consumer.py", line 382, in try_consume_file

    document_parser.parse(self.path, mime_type, self.filename)

  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 57, in parse

    raise ParseError(

documents.parsers.ParseError: Could not parse /tmp/paperless/paperless-ngxk4ony929/20230625 LBV Antrag Kostenerstattung für Meldebescheinigung Familienzuschlag.docx with tika server at http://tika:9998: 'ascii' codec can't encode character '\xfc' in position 59: ordinal not in range(128)
```


### Browser logs

_No response_

### Paperless-ngx version

1.6.3

### Host OS

Debian 11

### Installation method

Docker - official image

### Browser

Chrome

### Configuration changes

_No response_

### Other

_No response_


Discussion:
I think this is essentially the same as #3678 --> https://github.com/stumpylog/tika-client/issues/2
We recently updated to use a Tika library written by one of our maintainers and I think certain file formats are causing issues

I know he's AFK for a bit but planning to address these when he returns
Without a sample this will be tough to deal with. Moreover it may be an upstream issue with tika itself, which is not our software. If you cant upload the file, can you try to run it directly against tika and post the results of that?
The problem seems to be the umlaut "ü" in the filename ("für"). A document containing just the word "Test" with the filename "20230625 LBV Antrag Kostenerstattung für Meldebescheinigung Familienzuschlag.docx" brings up the error. After changing the filename in  "20230625 LBV Antrag Kostenerstattung fuer Meldebescheinigung Familienzuschlag.docx" everything works fine.


I might need to normalize or convert headers in my client.  I'll have to look at it more when possible
same issue here wir "ä", "ü", "ö" etc

Resolving patch:
diff --git a/src/paperless_mail/parsers.py b/src/paperless_mail/parsers.py
--- a/src/paperless_mail/parsers.py
+++ b/src/paperless_mail/parsers.py
@@ -13,7 +13,6 @@
 from imap_tools import MailAttachment
 from imap_tools import MailMessage
 from tika_client import TikaClient
-from tika_client.data_models import TikaKey
 
 from documents.parsers import DocumentParser
 from documents.parsers import ParseError
@@ -173,12 +172,8 @@ def tika_parse(self, html: str):
             with TikaClient(tika_url=self.tika_server) as client:
                 parsed = client.tika.as_text.from_buffer(html, "text/html")
 
-                if hasattr(parsed, "content") and parsed.content is not None:
+                if parsed.content is not None:
                     return parsed.content.strip()
-                elif TikaKey.Content in parsed.data:
-                    # May not be a completely handled type, but
-                    # the Tika response may still include content
-                    return parsed.data[TikaKey.Content].strip()
                 return ""
         except Exception as err:
             raise ParseError(
diff --git a/src/paperless_tika/parsers.py b/src/paperless_tika/parsers.py
--- a/src/paperless_tika/parsers.py
+++ b/src/paperless_tika/parsers.py
@@ -4,7 +4,6 @@
 import httpx
 from django.conf import settings
 from tika_client import TikaClient
-from tika_client.data_models import TikaKey
 
 from documents.parsers import DocumentParser
 from documents.parsers import ParseError
@@ -59,13 +58,9 @@ def parse(self, document_path: Path, mime_type: str, file_name=None):
                 f"{settings.TIKA_ENDPOINT}: {err}",
             ) from err
 
-        self.text = None
-        if hasattr(parsed, "content") and parsed.content is not None:
-            self.text = parsed.content.strip()
-        elif TikaKey.Content in parsed.data:
-            # May not be a completely handled type, but
-            # the Tika response may still include content
-            self.text = parsed.data[TikaKey.Content].strip()
+        self.text = parsed.content
+        if self.text is not None:
+            self.text = self.text.strip()
 
         self.date = parsed.created
         self.archive_path = self.convert_to_pdf(document_path, file_name)


Test patch: