Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Barcode splitting not working with v1.7.1 (file extension detection bug)
### Description

After updating to v1.7.1, the new splitting by barcodes feature doesn't work anymore for me.

From what I can tell, the log message `Unsupported file format for barcode reader:` indicates that the file extension calculated in `documents/tasks.py` - [`consume_file` here](https://github.com/paperless-ngx/paperless-ngx/blob/26eb7ecdb50ee8925fa402556a917c562a7b1e8b/src/documents/tasks.py#L237) is empty.

After switching back to v1.7.0, it works again.

### Steps to reproduce

1. Upload multi-page PDF with PATCHT barcode page inbetween (EDIT: via the web ui upload button)
2. After processing, file is not split and shown as one document

### Webserver logs

```bash
2022-05-15 00:00:37,691] [WARNING] [paperless.tasks] Unsupported file format for barcode reader:
[2022-05-15 00:00:37,741] [INFO] [paperless.consumer] Consuming 2022-05-14_02.pdf
[2022-05-15 00:00:37,745] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2022-05-15 00:00:37,751] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2022-05-15 00:00:37,758] [DEBUG] [paperless.consumer] Parsing 2022-05-14_02.pdf...
[2022-05-15 00:00:37,853] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /tmp/paperless/paperless-upload-mmaedjbi
[2022-05-15 00:00:37,915] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': '/tmp/paperless/paperless-upload-mmaedjbi', 'output_file': '/tmp/paperless/paperless-zycya93h/archive.pdf', 'use_threads': True, 'jobs': '2', 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'sidecar': '/tmp/paperless/paperless-zycya93h/sidecar.txt'}
```


### Paperless-ngx version

1.7.1

### Host OS

Ubuntu 20.04

### Installation method

Docker

### Browser

Firefox

### Configuration changes

PAPERLESS_CONSUMER_ENABLE_BARCODES=true  
PAPERLESS_CONSUMER_BARCODE_STRING=PATCHT

### Other

_No response_
[BUG] Barcode splitting not working with v1.7.1 (file extension detection bug)
### Description

After updating to v1.7.1, the new splitting by barcodes feature doesn't work anymore for me.

From what I can tell, the log message `Unsupported file format for barcode reader:` indicates that the file extension calculated in `documents/tasks.py` - [`consume_file` here](https://github.com/paperless-ngx/paperless-ngx/blob/26eb7ecdb50ee8925fa402556a917c562a7b1e8b/src/documents/tasks.py#L237) is empty.

After switching back to v1.7.0, it works again.

### Steps to reproduce

1. Upload multi-page PDF with PATCHT barcode page inbetween (EDIT: via the web ui upload button)
2. After processing, file is not split and shown as one document

### Webserver logs

```bash
2022-05-15 00:00:37,691] [WARNING] [paperless.tasks] Unsupported file format for barcode reader:
[2022-05-15 00:00:37,741] [INFO] [paperless.consumer] Consuming 2022-05-14_02.pdf
[2022-05-15 00:00:37,745] [DEBUG] [paperless.consumer] Detected mime type: application/pdf
[2022-05-15 00:00:37,751] [DEBUG] [paperless.consumer] Parser: RasterisedDocumentParser
[2022-05-15 00:00:37,758] [DEBUG] [paperless.consumer] Parsing 2022-05-14_02.pdf...
[2022-05-15 00:00:37,853] [DEBUG] [paperless.parsing.tesseract] Extracted text from PDF file /tmp/paperless/paperless-upload-mmaedjbi
[2022-05-15 00:00:37,915] [DEBUG] [paperless.parsing.tesseract] Calling OCRmyPDF with args: {'input_file': '/tmp/paperless/paperless-upload-mmaedjbi', 'output_file': '/tmp/paperless/paperless-zycya93h/archive.pdf', 'use_threads': True, 'jobs': '2', 'language': 'deu', 'output_type': 'pdfa', 'progress_bar': False, 'skip_text': True, 'clean': True, 'sidecar': '/tmp/paperless/paperless-zycya93h/sidecar.txt'}
```


### Paperless-ngx version

1.7.1

### Host OS

Ubuntu 20.04

### Installation method

Docker

### Browser

Firefox

### Configuration changes

PAPERLESS_CONSUMER_ENABLE_BARCODES=true  
PAPERLESS_CONSUMER_BARCODE_STRING=PATCHT

### Other

_No response_


Discussion:
Yeah, can confirm this, same issue with docker installation and 1.7.1
Could you post the filename?
The split feature checks for .pdf file extension and it appears that it doesn't find any.

Also, do you upload per web ui, folder or email? 
> Could you post the filename? The split feature checks for .pdf file extension and it appears that it doesn't find any.
> 
> Also, do you upload per web ui, folder or email?

Good point, I uploaded it via the web ui, I haven't tested the other ways yet.

The filename was `2022-05-14_02.pdf`
I don't have access to a pc right now, but I suspect the upload through the web ui somehow strips the extension.

I suspect the upload through the folder will work just fine
Well the frontend posts documents to `/api/documents/post_document/` as form-data which gives a random file name like `/tmp/paperless/paperless-upload-n86vou29`, but the file name is of course sent otherwise that data would be lost (this is just how web file uploads work btw, this isnt Paperless per se):
```
------WebKitFormBoundaryATAY199OoXiGIB9r
Content-Disposition: form-data; name="document"; filename="samplefile.pdf"
Content-Type: application/pdf


------WebKitFormBoundaryATAY199OoXiGIB9r--
```

But:
1. That has not changed in a long time, let alone between 1.7.0 and 1.7.1. Im surprised to know it worked in 1.7.0 and not in 1.7.1 since Im not aware of anything that would affect that.
2. I didnt review the original PR but I am surprised to learn it works with file name as opposed to mime-type or something like that, since file naming can be unreliable (not that mime-type doesnt have its problems!).
The thing that changed is #766

There it checks for file extension and splits if it detects a pdf or Tiff file.

Maybe we should MIME check, or explicitly pass the file name.. 
Ah yes. Yea I meant no changes on the frontend that would affect that.

I think should use mime IMHO, it’s not perfect but I think still the most reliable method. I suppose could use both mime and file name too
right, if i put the same file in the cosume folder, everything works just fine. WebUi doesnt work But good to know, i only tried it via web, to see if splitting works... :)
For me, i`m fine if it works right now, cosume Folder is for me the normal way to scan. :)
I'll look into it in the coming days. @aytec good to know it works with the folder :+1: 
@shamoon I looked at the code, and maybe we can use it with `override_filename` additionally to the `path` and maybe the MIME detection. That should make it more robust
Fyi I'll agree that this is working in 1.7.1 when processing through file consumption.  I processed many documents last night using this.
Yeah, can confirm this, same issue with docker installation and 1.7.1
Could you post the filename?
The split feature checks for .pdf file extension and it appears that it doesn't find any.

Also, do you upload per web ui, folder or email? 
> Could you post the filename? The split feature checks for .pdf file extension and it appears that it doesn't find any.
> 
> Also, do you upload per web ui, folder or email?

Good point, I uploaded it via the web ui, I haven't tested the other ways yet.

The filename was `2022-05-14_02.pdf`
I don't have access to a pc right now, but I suspect the upload through the web ui somehow strips the extension.

I suspect the upload through the folder will work just fine
Well the frontend posts documents to `/api/documents/post_document/` as form-data which gives a random file name like `/tmp/paperless/paperless-upload-n86vou29`, but the file name is of course sent otherwise that data would be lost (this is just how web file uploads work btw, this isnt Paperless per se):
```
------WebKitFormBoundaryATAY199OoXiGIB9r
Content-Disposition: form-data; name="document"; filename="samplefile.pdf"
Content-Type: application/pdf


------WebKitFormBoundaryATAY199OoXiGIB9r--
```

But:
1. That has not changed in a long time, let alone between 1.7.0 and 1.7.1. Im surprised to know it worked in 1.7.0 and not in 1.7.1 since Im not aware of anything that would affect that.
2. I didnt review the original PR but I am surprised to learn it works with file name as opposed to mime-type or something like that, since file naming can be unreliable (not that mime-type doesnt have its problems!).
The thing that changed is #766

There it checks for file extension and splits if it detects a pdf or Tiff file.

Maybe we should MIME check, or explicitly pass the file name.. 
Ah yes. Yea I meant no changes on the frontend that would affect that.

I think should use mime IMHO, it’s not perfect but I think still the most reliable method. I suppose could use both mime and file name too
right, if i put the same file in the cosume folder, everything works just fine. WebUi doesnt work But good to know, i only tried it via web, to see if splitting works... :)
For me, i`m fine if it works right now, cosume Folder is for me the normal way to scan. :)
I'll look into it in the coming days. @aytec good to know it works with the folder :+1: 
@shamoon I looked at the code, and maybe we can use it with `override_filename` additionally to the `path` and maybe the MIME detection. That should make it more robust
Fyi I'll agree that this is working in 1.7.1 when processing through file consumption.  I processed many documents last night using this.

Resolving patch:
diff --git a/src/documents/tasks.py b/src/documents/tasks.py
--- a/src/documents/tasks.py
+++ b/src/documents/tasks.py
@@ -4,6 +4,7 @@
 import tempfile
 from typing import List  # for type hinting. Can be removed, if only Python >3.8 is used
 
+import magic
 import tqdm
 from asgiref.sync import async_to_sync
 from channels.layers import get_channel_layer
@@ -95,19 +96,33 @@ def barcode_reader(image) -> List[str]:
     return barcodes
 
 
+def get_file_type(path: str) -> str:
+    """
+    Determines the file type, based on MIME type.
+
+    Returns the MIME type.
+    """
+    mime_type = magic.from_file(path, mime=True)
+    logger.debug(f"Detected mime type: {mime_type}")
+    return mime_type
+
+
 def convert_from_tiff_to_pdf(filepath: str) -> str:
     """
-    converts a given TIFF image file to pdf into a temp. directory.
+    converts a given TIFF image file to pdf into a temporary directory.
+
     Returns the new pdf file.
     """
     file_name = os.path.splitext(os.path.basename(filepath))[0]
-    file_extension = os.path.splitext(os.path.basename(filepath))[1].lower()
+    mime_type = get_file_type(filepath)
     tempdir = tempfile.mkdtemp(prefix="paperless-", dir=settings.SCRATCH_DIR)
     # use old file name with pdf extension
-    if file_extension == ".tif" or file_extension == ".tiff":
+    if mime_type == "image/tiff":
         newpath = os.path.join(tempdir, file_name + ".pdf")
     else:
-        logger.warning(f"Cannot convert from {str(file_extension)} to pdf.")
+        logger.warning(
+            f"Cannot convert mime type {str(mime_type)} from {str(filepath)} to pdf.",
+        )
         return None
     with Image.open(filepath) as image:
         images = []
@@ -231,17 +246,17 @@ def consume_file(
         document_list = []
         converted_tiff = None
         if settings.CONSUMER_BARCODE_TIFF_SUPPORT:
-            supported_extensions = [".pdf", ".tiff", ".tif"]
+            supported_mime = ["image/tiff", "application/pdf"]
         else:
-            supported_extensions = [".pdf"]
-        file_extension = os.path.splitext(os.path.basename(path))[1].lower()
-        if file_extension not in supported_extensions:
+            supported_mime = ["application/pdf"]
+        mime_type = get_file_type(path)
+        if mime_type not in supported_mime:
             # if not supported, skip this routine
             logger.warning(
-                f"Unsupported file format for barcode reader: {str(file_extension)}",
+                f"Unsupported file format for barcode reader: {str(mime_type)}",
             )
         else:
-            if file_extension in {".tif", ".tiff"}:
+            if mime_type == "image/tiff":
                 file_to_process = convert_from_tiff_to_pdf(path)
             else:
                 file_to_process = path


Test patch:
diff --git a/src/documents/tests/test_tasks.py b/src/documents/tests/test_tasks.py
--- a/src/documents/tests/test_tasks.py
+++ b/src/documents/tests/test_tasks.py
@@ -204,6 +204,34 @@ def test_barcode_reader_custom_128_separator(self):
         img = Image.open(test_file)
         self.assertEqual(tasks.barcode_reader(img), ["CUSTOM BARCODE"])
 
+    def test_get_mime_type(self):
+        tiff_file = os.path.join(
+            os.path.dirname(__file__),
+            "samples",
+            "simple.tiff",
+        )
+        pdf_file = os.path.join(
+            os.path.dirname(__file__),
+            "samples",
+            "simple.pdf",
+        )
+        png_file = os.path.join(
+            os.path.dirname(__file__),
+            "samples",
+            "barcodes",
+            "barcode-128-custom.png",
+        )
+        tiff_file_no_extension = os.path.join(settings.SCRATCH_DIR, "testfile1")
+        pdf_file_no_extension = os.path.join(settings.SCRATCH_DIR, "testfile2")
+        shutil.copy(tiff_file, tiff_file_no_extension)
+        shutil.copy(pdf_file, pdf_file_no_extension)
+
+        self.assertEqual(tasks.get_file_type(tiff_file), "image/tiff")
+        self.assertEqual(tasks.get_file_type(pdf_file), "application/pdf")
+        self.assertEqual(tasks.get_file_type(tiff_file_no_extension), "image/tiff")
+        self.assertEqual(tasks.get_file_type(pdf_file_no_extension), "application/pdf")
+        self.assertEqual(tasks.get_file_type(png_file), "image/png")
+
     def test_convert_from_tiff_to_pdf(self):
         test_file = os.path.join(
             os.path.dirname(__file__),
@@ -469,7 +497,7 @@ def test_consume_barcode_unsupported_jpg_file(self, m):
         self.assertEqual(
             cm.output,
             [
-                "WARNING:paperless.tasks:Unsupported file format for barcode reader: .jpg",
+                "WARNING:paperless.tasks:Unsupported file format for barcode reader: image/jpeg",
             ],
         )
         m.assert_called_once()
@@ -481,6 +509,26 @@ def test_consume_barcode_unsupported_jpg_file(self, m):
         self.assertIsNone(kwargs["override_document_type_id"])
         self.assertIsNone(kwargs["override_tag_ids"])
 
+    @override_settings(
+        CONSUMER_ENABLE_BARCODES=True,
+        CONSUMER_BARCODE_TIFF_SUPPORT=True,
+    )
+    def test_consume_barcode_supported_no_extension_file(self):
+        """
+        This test assumes barcode and TIFF support are enabled and
+        the user uploads a supported image file, but without extension
+        """
+        test_file = os.path.join(
+            os.path.dirname(__file__),
+            "samples",
+            "barcodes",
+            "patch-code-t-middle.tiff",
+        )
+        dst = os.path.join(settings.SCRATCH_DIR, "patch-code-t-middle")
+        shutil.copy(test_file, dst)
+
+        self.assertEqual(tasks.consume_file(dst), "File successfully split")
+
     @mock.patch("documents.tasks.sanity_checker.check_sanity")
     def test_sanity_check_success(self, m):
         m.return_value = SanityCheckMessages()