Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] `document_importer` errors out if mail account and/or mail rules are defined in `manifest.json`
### Description

From Matrix:

> Hi, I'm trying to use the document_importer to import v1.6.0 exports to a new v1.7.0 docker instance. Connected to the new v1.7.0 docker CLI and ran the importer but receive this error message. Has anyone run into this?
> 
> UPDATE: Same error occurred in v1.6.0 so I went ahead and I removed the part in `manifest.json` where mail rules are described (`model": "paperless_mail.mailaccount` and `model": "paperless_mail.mailrule`) and the document import ran successfully.

I believe it has something to do with #270, but can't say if it can/should be fixed.

### Expected behavior

n/a

### Steps to reproduce

n/a

### Webserver logs

```bash
root@1ab8a0c4e385:/usr/src/paperless/export# document\_importer /usr/src/paperless/export

root@1ab8a0c4e385:/usr/src/paperless/export# document_importer /usr/src/paperless/export
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/django/db/models/[options.py](http://options.py/)", line 672, in get_field
return self.fields_map[field_name]
KeyError: 'assign_tag'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/django/core/serializers/[json.py](http://json.py/)", line 70, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/usr/local/lib/python3.9/site-packages/django/core/serializers/[python.py](http://python.py/)", line 131, in Deserializer
field = Model._meta.get_field(field_name)
File "/usr/local/lib/python3.9/site-packages/django/db/models/[options.py](http://options.py/)", line 674, in get_field
raise FieldDoesNotExist(
django.core.exceptions.FieldDoesNotExist: MailRule has no field named 'assign_tag'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/paperless/src/[manage.py](http://manage.py/)", line 11, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 446, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.9/site-packages/django/core/management/[base.py](http://base.py/)", line 414, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/[base.py](http://base.py/)", line 460, in execute
output = self.handle(*args, **options)
File "/usr/src/paperless/src/documents/management/commands/document_[importer.py](http://importer.py/)", line 87, in handle
call_command("loaddata", manifest_path)
File "/usr/local/lib/python3.9/site-packages/django/core/management/init.py", line 198, in call_command
return command.execute(*args, **defaults)
File "/usr/local/lib/python3.9/site-packages/django/core/management/[base.py](http://base.py/)", line 460, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/[loaddata.py](http://loaddata.py/)", line 102, in handle
self.loaddata(fixture_labels)
File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/[loaddata.py](http://loaddata.py/)", line 163, in loaddata
self.load_label(fixture_label)
File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/[loaddata.py](http://loaddata.py/)", line 251, in load_label
for obj in objects:
File "/usr/local/lib/python3.9/site-packages/django/core/serializers/[json.py](http://json.py/)", line 74, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/usr/src/paperless/export/manifest.json':
```
```


### Screenshots

_No response_

### Paperless-ngx version

1.6.0 and 1.7.0

### Host OS

Linux

### Installation method

Docker

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
This exception makes sense, because the importer isn't able to run migrations on the serialized JSON data.

I don't think we can make django run the migrations on serialized data (someone familiar can correct me), so the solution may be:

1. Include the paperless version in the manifest
2. Don't allow importing if the versions don't match
3. Update the documentation to state exports and imports are only going to work if the version matches, but that version upgrading after import will work OK
4. (optional) Describe how an advanced user could edit the manifest to try anyway, just in case no migrations have be added since the exported version
Thank you for the explanation!

If I understand correctly, updating the docs may be the only thing required of us. 

* If migrations have **not** changed between versions, the import will likely succeed. 
* If migrations have changed between versions, the import will fail. We can explain in the [docs](https://paperless-ngx.readthedocs.io/en/latest/administration.html?highlight=exporter#document-importer) why that is.
 
I don't see the benefit of disallowing importing with mismatched versions. Do you agree? Perhaps instead a mismatched version would catch the error trace and share a friendlier error to the user?
Yes, that's the docs updates.  

I was thinking preventing the import with a version mismatch would make the error more clear than the jumble out stuff out of django.  We could catch the error instead and give a similar message though
But if there aren't any migrations between versions, the document import would still fail because of the version mismatch, right? I don't see the reason for adding a premature failure. Maybe I'm missing something though

> We could catch the error instead and give a similar message though

I think that's best, get rid of the jumble. A `KeyError` exception _and_ a manifest version mismatch means we know the most likely issue.
That's true and a new version wouldn't always have migrations, so catching and explaining it instead sounds better now.

Resolving patch:
diff --git a/src/documents/management/commands/document_archiver.py b/src/documents/management/commands/document_archiver.py
--- a/src/documents/management/commands/document_archiver.py
+++ b/src/documents/management/commands/document_archiver.py
@@ -152,4 +152,4 @@ def handle(self, *args, **options):
                     ),
                 )
         except KeyboardInterrupt:
-            print("Aborting...")
+            self.stdout.write(self.style.NOTICE(("Aborting...")))
diff --git a/src/documents/management/commands/document_exporter.py b/src/documents/management/commands/document_exporter.py
--- a/src/documents/management/commands/document_exporter.py
+++ b/src/documents/management/commands/document_exporter.py
@@ -22,6 +22,7 @@
 from documents.settings import EXPORTER_FILE_NAME
 from documents.settings import EXPORTER_THUMBNAIL_NAME
 from filelock import FileLock
+from paperless import version
 from paperless.db import GnuPG
 from paperless_mail.models import MailAccount
 from paperless_mail.models import MailRule
@@ -232,12 +233,18 @@ def dump(self, progress_bar_disable=False):
                         archive_target,
                     )
 
-        # 4. write manifest to target forlder
+        # 4.1 write manifest to target folder
         manifest_path = os.path.abspath(os.path.join(self.target, "manifest.json"))
 
         with open(manifest_path, "w") as f:
             json.dump(manifest, f, indent=2)
 
+        # 4.2 write version information to target folder
+        version_path = os.path.abspath(os.path.join(self.target, "version.json"))
+
+        with open(version_path, "w") as f:
+            json.dump({"version": version.__full_version_str__}, f, indent=2)
+
         if self.delete:
             # 5. Remove files which we did not explicitly export in this run
 
diff --git a/src/documents/management/commands/document_importer.py b/src/documents/management/commands/document_importer.py
--- a/src/documents/management/commands/document_importer.py
+++ b/src/documents/management/commands/document_importer.py
@@ -6,9 +6,11 @@
 
 import tqdm
 from django.conf import settings
+from django.core.exceptions import FieldDoesNotExist
 from django.core.management import call_command
 from django.core.management.base import BaseCommand
 from django.core.management.base import CommandError
+from django.core.serializers.base import DeserializationError
 from django.db.models.signals import m2m_changed
 from django.db.models.signals import post_save
 from documents.models import Document
@@ -16,6 +18,7 @@
 from documents.settings import EXPORTER_FILE_NAME
 from documents.settings import EXPORTER_THUMBNAIL_NAME
 from filelock import FileLock
+from paperless import version
 
 from ...file_handling import create_source_path_directory
 from ...signals.handlers import update_filename_and_move_files
@@ -53,6 +56,7 @@ def __init__(self, *args, **kwargs):
         BaseCommand.__init__(self, *args, **kwargs)
         self.source = None
         self.manifest = None
+        self.version = None
 
     def handle(self, *args, **options):
 
@@ -66,12 +70,30 @@ def handle(self, *args, **options):
         if not os.access(self.source, os.R_OK):
             raise CommandError("That path doesn't appear to be readable")
 
-        manifest_path = os.path.join(self.source, "manifest.json")
+        manifest_path = os.path.normpath(os.path.join(self.source, "manifest.json"))
         self._check_manifest_exists(manifest_path)
 
         with open(manifest_path) as f:
             self.manifest = json.load(f)
 
+        version_path = os.path.normpath(os.path.join(self.source, "version.json"))
+        if os.path.exists(version_path):
+            with open(version_path) as f:
+                self.version = json.load(f)["version"]
+                # Provide an initial warning if needed to the user
+                if self.version != version.__full_version_str__:
+                    self.stdout.write(
+                        self.style.WARNING(
+                            "Version mismatch: "
+                            f"Currently {version.__full_version_str__},"
+                            f" importing {self.version}."
+                            " Continuing, but import may fail.",
+                        ),
+                    )
+
+        else:
+            self.stdout.write(self.style.NOTICE("No version.json file located"))
+
         self._check_manifest()
         with disable_signal(
             post_save,
@@ -84,12 +106,36 @@ def handle(self, *args, **options):
                 sender=Document.tags.through,
             ):
                 # Fill up the database with whatever is in the manifest
-                call_command("loaddata", manifest_path)
+                try:
+                    call_command("loaddata", manifest_path)
+                except (FieldDoesNotExist, DeserializationError) as e:
+                    self.stdout.write(self.style.ERROR("Database import failed"))
+                    if (
+                        self.version is not None
+                        and self.version != version.__full_version_str__
+                    ):
+                        self.stdout.write(
+                            self.style.ERROR(
+                                "Version mismatch: "
+                                f"Currently {version.__full_version_str__},"
+                                f" importing {self.version}",
+                            ),
+                        )
+                        raise e
+                    else:
+                        self.stdout.write(
+                            self.style.ERROR("No version information present"),
+                        )
+                        raise e
 
                 self._import_files_from_manifest(options["no_progress_bar"])
 
-        print("Updating search index...")
-        call_command("document_index", "reindex")
+        self.stdout.write("Updating search index...")
+        call_command(
+            "document_index",
+            "reindex",
+            no_progress_bar=options["no_progress_bar"],
+        )
 
     @staticmethod
     def _check_manifest_exists(path):
@@ -132,7 +178,7 @@ def _import_files_from_manifest(self, progress_bar_disable):
         os.makedirs(settings.THUMBNAIL_DIR, exist_ok=True)
         os.makedirs(settings.ARCHIVE_DIR, exist_ok=True)
 
-        print("Copy files into paperless...")
+        self.stdout.write("Copy files into paperless...")
 
         manifest_documents = list(
             filter(lambda r: r["model"] == "documents.document", self.manifest),


Test patch: