Repository: projectcaluma/alexandria
Pull request #816: Fix file copying
Author: login: martinwe-adfinis
Created: 2025-04-17T14:21:43Z | Merged: 2025-04-24T13:36:04Z

This is a follow-up on both #757 and #798:

 * Pass the bucket *name* to Django storages (not the bucket *object*).
 * Copy the `FieldFile` object when copying a document's file, to prevent both files referencing the same object in the storage.


Diff:
diff --git a/alexandria/core/api.py b/alexandria/core/api.py
index d541e803..9cc328ae 100644
--- a/alexandria/core/api.py
+++ b/alexandria/core/api.py
@@ -41,20 +41,18 @@ def copy_document(
     document_files = models.File.objects.filter(
         document=document, variant=models.File.Variant.ORIGINAL.value
     ).order_by("created_at")
-    new_files = []
     for document_file in document_files:
-        new_files.append(
-            create_file(
-                name=document_file.name,
-                document=new_document,
-                content=document_file.content,
-                mime_type=document_file.mime_type,
-                size=document_file.size,
-                user=document_file.created_by_user,
-                group=document_file.created_by_group,
-                metainfo=document_file.metainfo,
-            )
+        new_file = create_file(
+            name=document_file.name,
+            document=new_document,
+            content=document_file.content,
+            mime_type=document_file.mime_type,
+            size=document_file.size,
+            user=document_file.created_by_user,
+            group=document_file.created_by_group,
+            metainfo=document_file.metainfo,
         )
+        new_file.content.copy(f"{new_file.pk}_{new_file.name}")
 
     return new_document
 
diff --git a/alexandria/core/management/commands/fix_copied_file_references.py b/alexandria/core/management/commands/fix_copied_file_references.py
new file mode 100644
index 00000000..24fb4e8d
--- /dev/null
+++ b/alexandria/core/management/commands/fix_copied_file_references.py
@@ -0,0 +1,108 @@
+from collections import defaultdict
+
+import botocore
+from django.core.management.base import BaseCommand
+from tqdm import tqdm
+
+from alexandria.core.models import File
+
+
+class Command(BaseCommand):
+    help = "Fixes copied file references by creating a new copy."
+
+    def add_arguments(self, parser):
+        parser.add_argument("--dry", dest="dry", action="store_true", default=False)
+        parser.add_argument("--since", dest="since", type=str, default=None)
+
+    def handle(self, *args, **options):
+        filename_counts = defaultdict(list)
+        file_list = File.objects.filter(variant="original")
+
+        for tmp_file in tqdm(file_list.order_by("created_at").iterator()):
+            filename = tmp_file.content.name
+            filename_counts[filename].append(tmp_file.pk)
+
+        fix_count = 0
+        for filename, pks in filename_counts.items():
+            # Skip files that only have one reference.
+            pks_len = len(pks)
+            if pks_len <= 1:
+                continue
+
+            duplicates_query = File.objects.filter(pk__in=pks).order_by("created_at")
+
+            # Filter copies by file creation timestamp if provided
+            if options["since"]:
+                duplicates_query = duplicates_query.filter(
+                    created_at__gt=options["since"]
+                )
+
+            original_duplicate, *duplicates = duplicates_query
+
+            ignored_output = ""
+            duplicates_len = len(duplicates)
+            if duplicates_len < (pks_len - 1):
+                ignored_len = pks_len - duplicates_len
+                ignored_output = f"/{pks_len} (ignored {ignored_len} files older than {options['since']})"
+
+            checksum = original_duplicate.checksum
+            self.stdout.write(
+                self.style.SUCCESS(
+                    f'Fixing filename "{filename}" with {duplicates_len}{ignored_output} duplicate(s)'
+                )
+            )
+
+            for duplicate in duplicates:
+                # Check if the duplicate has a checksum and skip if it does not match
+                # the original. If it does not have a checksum, we try to copy the
+                # original file.
+                if duplicate.checksum and duplicate.checksum != checksum:
+                    self.stderr.write(
+                        self.style.ERROR(
+                            f"Failed: File ID {duplicate.pk} with filename {filename} has a different checksum than the original."
+                        )
+                    )
+                    continue
+
+                new_filename = f"{duplicate.pk}_{duplicate.name}"
+
+                try:
+                    # perform the copy operation
+                    if not options["dry"]:
+                        duplicate.content.copy(f"{duplicate.pk}_{duplicate.name}")
+
+                    self.stdout.write(
+                        self.style.SUCCESS(
+                            f" > Fixed file ID {duplicate.pk} with new filename {new_filename} (Checksum: {checksum})"
+                        )
+                    )
+                    fix_count += 1
+                except botocore.exceptions.ClientError as e:
+                    if e.response["Error"]["Code"] == "NoSuchKey":
+                        self.stderr.write(
+                            self.style.ERROR(
+                                f" > Failed: File ID {duplicate.pk} with filename {duplicate.content.name} does not exist in S3. (Checksum: {checksum})"
+                            )
+                        )
+                    else:
+                        self.stderr.write(
+                            self.style.ERROR(
+                                f" > Failed: Error while fixing File ID {duplicate.pk} with filename {duplicate.content.name}, client error: {e} (Checksum: {checksum})"
+                            )
+                        )
+                except Exception as e:
+                    self.stderr.write(
+                        self.style.ERROR(
+                            f" > Failed: Error while fixing File ID {duplicate.pk} with filename {duplicate.content.name}, error: {e} (Checksum: {checksum})"
+                        )
+                    )
+
+        self.stdout.write("")
+        if fix_count > 0:
+            self.stdout.write(
+                self.style.SUCCESS(f"Fixed {fix_count} files with copied references.")
+            )
+        else:
+            self.stdout.write(
+                self.style.WARNING("No files with copied references where fixed.")
+            )
diff --git a/alexandria/core/tests/test_api.py b/alexandria/core/tests/test_api.py
index 85018f3e..2489ce39 100644
--- a/alexandria/core/tests/test_api.py
+++ b/alexandria/core/tests/test_api.py
@@ -87,6 +87,8 @@ def test_copy_document_api(db, category, category_factory, same_category):
     # files will retain the user/group of the original document
     assert files[0].created_by_user == "Foo"
     assert files[0].created_by_group == "Baz"
+    # copied file content should have a new name
+    assert str(files[0].content) != str(first_file.content)
 
     # new thumbnail for first file
     assert files[2].document.pk == copied_doc.pk
@@ -99,6 +101,8 @@ def test_copy_document_api(db, category, category_factory, same_category):
     assert files[1].name == "Mee2.pdf"
     assert files[1].mime_type == "image/jpeg"
     assert files[1].size == 2
+    # copied file content should have a new name
+    assert str(files[1].content) != str(extra_file.content)
     # files will retain the user/group of the original document
     assert files[1].created_by_user == "Foo2"
     assert files[1].created_by_group == "Baz2"
@@ -106,3 +110,67 @@ def test_copy_document_api(db, category, category_factory, same_category):
     # new thumbnail for extra file
     assert files[3].document.pk == copied_doc.pk
     assert files[3].variant == File.Variant.THUMBNAIL
+
+
+def test_copy_document_deleted_original(db, category_factory):
+    """
+    Test that copied files are still readable after the original document is deleted.
+    """
+    category = category_factory()
+    doc, _first_file = api.create_document_file(
+        user="Foo",
+        group="Baz",
+        category=category,
+        document_title="Bar.pdf",
+        file_name="Mee.pdf",
+        file_content=SimpleUploadedFile(
+            name="test.png",
+            content=FileData.png,
+            content_type="png",
+        ),
+        mime_type="image/png",
+        file_size=1,
+    )
+    api.create_file(
+        document=doc,
+        user="Foo2",
+        group="Baz2",
+        name="Mee2.pdf",
+        content=SimpleUploadedFile(
+            name="test2.png",
+            content=FileData.png800,
+            content_type="png",
+        ),
+        mime_type="image/jpeg",
+        size=2,
+    )
+
+    copied_doc = api.copy_document(doc, "CopyUser", "CopyGroup", category)
+    files = doc.files.filter(variant="original").order_by("variant", "created_at")
+    files_copied = copied_doc.files.filter(variant="original").order_by(
+        "variant", "created_at"
+    )
+
+    def _filecontent_is_readable(file):
+        """
+        Helper function to check if the file content is readable.
+        """
+        content = file.content
+        content.open()
+        try:
+            return content.readable()
+        finally:
+            content.close()
+
+    # check if original and copied files are readable.
+    assert _filecontent_is_readable(files[0])
+    assert _filecontent_is_readable(files[1])
+    assert _filecontent_is_readable(files_copied[0])
+    assert _filecontent_is_readable(files_copied[1])
+    assert files[0].content.name != files_copied[0].content.name
+    assert files[1].content.name != files_copied[1].content.name
+
+    # delete original doc and check if copied files are still readable.
+    doc.delete()
+    assert _filecontent_is_readable(files_copied[0])
+    assert _filecontent_is_readable(files_copied[1])
diff --git a/alexandria/storages/fields.py b/alexandria/storages/fields.py
index 01db87a8..d28a968d 100644
--- a/alexandria/storages/fields.py
+++ b/alexandria/storages/fields.py
@@ -35,11 +35,11 @@ def copy(self, target_name: str):
         if isinstance(self.storage, S3Storage):
             copy_args = {
                 "CopySource": {
-                    "Bucket": self.storage.bucket,
+                    "Bucket": self.storage.bucket.name,
                     "Key": self.name,
                 },
                 # Destination settings
-                "Bucket": self.storage.bucket,
+                "Bucket": self.storage.bucket.name,
                 "Key": target_name,
             }
             if isinstance(self.storage, SsecGlobalS3Storage):
diff --git a/alexandria/storages/tests/test_dynamic_field.py b/alexandria/storages/tests/test_dynamic_field.py
index d0b384e9..dd97dc18 100644
--- a/alexandria/storages/tests/test_dynamic_field.py
+++ b/alexandria/storages/tests/test_dynamic_field.py
@@ -29,9 +29,34 @@ def test_dynamic_storage_select_global_ssec(
         with pytest.raises(raises):
             file_factory(encryption_status=settings.ALEXANDRIA_ENCRYPTION_METHOD)
         return
-    file_factory(encryption_status=settings.ALEXANDRIA_ENCRYPTION_METHOD)
+    file = file_factory(encryption_status=settings.ALEXANDRIA_ENCRYPTION_METHOD)
     assert SsecGlobalS3Storage.save.called_once()
 
+    # test copy file parameters through S3 copy
+    mock_api_call = mocker.patch(
+        "botocore.client.BaseClient._make_api_call", return_value=None
+    )
+    current_filename = file.content.name
+    new_filename = "new-file-name"
+    file.content.copy(new_filename)
+
+    assert mock_api_call.called_once()
+    assert mock_api_call.call_args[0] == (
+        "CopyObject",
+        {
+            "CopySource": {
+                "Bucket": settings.ALEXANDRIA_S3_BUCKET_NAME,
+                "Key": current_filename,
+            },
+            "CopySourceSSECustomerAlgorithm": "AES256",
+            "CopySourceSSECustomerKey": settings.ALEXANDRIA_S3_STORAGE_SSEC_SECRET,
+            "Bucket": settings.ALEXANDRIA_S3_BUCKET_NAME,
+            "Key": new_filename,
+            "SSECustomerAlgorithm": "AES256",
+            "SSECustomerKey": settings.ALEXANDRIA_S3_STORAGE_SSEC_SECRET,
+        },
+    )
+
 
 @pytest.mark.parametrize(
     "method,storage_backend",


Reviews:
user: login: winged
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T08:28:26Z
user: login: nlzet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T09:55:18Z
user: login: nlzet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T09:55:26Z
user: login: nlzet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T09:55:35Z
user: login: nlzet
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T09:56:05Z
user: login: winged
body: LGTM now, nice work!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T13:33:10Z

Review comments:
diff_hunk: @@ -99,10 +101,76 @@ def test_copy_document_api(db, category, category_factory, same_category):
     assert files[1].name == "Mee2.pdf"
     assert files[1].mime_type == "image/jpeg"
     assert files[1].size == 2
+    # copied file content should have a new name
+    assert str(files[1].content) != str(extra_file.content)
     # files will retain the user/group of the original document
     assert files[1].created_by_user == "Foo2"
     assert files[1].created_by_group == "Baz2"
 
     # new thumbnail for extra file
     assert files[3].document.pk == copied_doc.pk
     assert files[3].variant == File.Variant.THUMBNAIL
+
+
+def test_copy_document_deleted_original(db, category_factory):
+    """
+    Test that copied files are still readable after the original document is deleted.
+    """
+    category = category_factory()
+    doc, _first_file = api.create_document_file(
+        "Foo",
+        "Baz",
+        category,
+        "Bar.pdf",
+        "Mee.pdf",
+        SimpleUploadedFile(
+            name="test.png",
+            content=FileData.png,
+            content_type="png",
+        ),
+        "image/png",
+        1,
+    )
+    api.create_file(
+        doc,
+        "Foo2",
+        "Baz2",
+        "Mee2.pdf",
+        SimpleUploadedFile(
+            name="test2.png",
+            content=FileData.png800,
+            content_type="png",
+        ),
+        "image/jpeg",
+        2,
+    )
path: alexandria/core/tests/test_api.py
user: login: winged
body: Just for readability, please use named arguments if it's not obvious what the parameters mean
created_at: 2025-04-24T08:16:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+import botocore
+from django.core.management.base import BaseCommand
+from tqdm import tqdm
+
+from alexandria.core.models import File
+
+
+class Command(BaseCommand):
+    help = "Fixes copied file references by creating a new copy."
+
+    def add_arguments(self, parser):
+        parser.add_argument("--dry", dest="dry", action="store_true", default=False)
+
+    def handle(self, *args, **options):
+        filename_counts = {}
+        for tmp_file in tqdm(
+            File.objects.filter(variant="original").order_by("created_at")
+        ):
path: alexandria/core/management/commands/fix_copied_file_references.py
user: login: winged
body: This could be a huge number of files, and in this way, we're loading them all into memory at the same time.

Querysets can be turned into iterators that load the data batch-by-batch from DB, improving memory usage:

```suggestion
        for tmp_file in tqdm(
            File.objects.filter(variant="original").order_by("created_at").iterator()
        ):
```
created_at: 2025-04-24T08:20:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+import botocore
+from django.core.management.base import BaseCommand
+from tqdm import tqdm
+
+from alexandria.core.models import File
+
+
+class Command(BaseCommand):
+    help = "Fixes copied file references by creating a new copy."
+
+    def add_arguments(self, parser):
+        parser.add_argument("--dry", dest="dry", action="store_true", default=False)
+
+    def handle(self, *args, **options):
+        filename_counts = {}
+        for tmp_file in tqdm(
+            File.objects.filter(variant="original").order_by("created_at")
+        ):
+            filename = tmp_file.content.name
+            if filename in filename_counts:
+                filename_counts[filename].append(tmp_file.pk)
+            else:
+                filename_counts[filename] = [tmp_file.pk]
path: alexandria/core/management/commands/fix_copied_file_references.py
user: login: winged
body: If you use a `defaultdict` for this, you can get rid of the `if` check here, and just call `.append()` directly.
For example:
```python
from collections import defaultdict
foo = defaultdict(list)
foo["test"].append(1)
...
```
created_at: 2025-04-24T08:22:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+import botocore
+from django.core.management.base import BaseCommand
+from tqdm import tqdm
+
+from alexandria.core.models import File
+
+
+class Command(BaseCommand):
+    help = "Fixes copied file references by creating a new copy."
+
+    def add_arguments(self, parser):
+        parser.add_argument("--dry", dest="dry", action="store_true", default=False)
+
+    def handle(self, *args, **options):
+        filename_counts = {}
+        for tmp_file in tqdm(
+            File.objects.filter(variant="original").order_by("created_at")
+        ):
+            filename = tmp_file.content.name
+            if filename in filename_counts:
+                filename_counts[filename].append(tmp_file.pk)
+            else:
+                filename_counts[filename] = [tmp_file.pk]
+
+        fix_count = 0
+        for filename, pks in filename_counts.items():
+            # Skip files that only have one reference.
+            if len(pks) <= 1:
+                continue
+
+            # Find duplicates, which are files with the same filename and no checksum.
+            duplicates = File.objects.filter(pk__in=pks).order_by("created_at")
+            checksum = duplicates.first().checksum
+
+            for duplicate in duplicates:
+                if duplicate.checksum != checksum:
+                    self.stderr.write(
+                        self.style.ERROR(
+                            f"Failed: File {duplicate.pk} with filename {filename} has a different checksum than the original."
+                        )
+                    )
+                    continue
+
+                current_filename = duplicate.content.name
+                new_filename = f"{duplicate.pk}_{duplicate.name}"
+
+                # Only fix copied files, skip the original
+                if current_filename == new_filename:
+                    continue
path: alexandria/core/management/commands/fix_copied_file_references.py
user: login: winged
body: This seems a bit fragile, no? You already know that the original is the first one in the duplicates QS, so why not make use of that fact instead?

Maybe something along the lines of `original, *duplicates = File.objects.filter(...)`
created_at: 2025-04-24T08:26:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -99,10 +101,76 @@ def test_copy_document_api(db, category, category_factory, same_category):
     assert files[1].name == "Mee2.pdf"
     assert files[1].mime_type == "image/jpeg"
     assert files[1].size == 2
+    # copied file content should have a new name
+    assert str(files[1].content) != str(extra_file.content)
     # files will retain the user/group of the original document
     assert files[1].created_by_user == "Foo2"
     assert files[1].created_by_group == "Baz2"
 
     # new thumbnail for extra file
     assert files[3].document.pk == copied_doc.pk
     assert files[3].variant == File.Variant.THUMBNAIL
+
+
+def test_copy_document_deleted_original(db, category_factory):
+    """
+    Test that copied files are still readable after the original document is deleted.
+    """
+    category = category_factory()
+    doc, _first_file = api.create_document_file(
+        "Foo",
+        "Baz",
+        category,
+        "Bar.pdf",
+        "Mee.pdf",
+        SimpleUploadedFile(
+            name="test.png",
+            content=FileData.png,
+            content_type="png",
+        ),
+        "image/png",
+        1,
+    )
+    api.create_file(
+        doc,
+        "Foo2",
+        "Baz2",
+        "Mee2.pdf",
+        SimpleUploadedFile(
+            name="test2.png",
+            content=FileData.png800,
+            content_type="png",
+        ),
+        "image/jpeg",
+        2,
+    )
path: alexandria/core/tests/test_api.py
user: login: nlzet
body: Changed
created_at: 2025-04-24T09:55:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+import botocore
+from django.core.management.base import BaseCommand
+from tqdm import tqdm
+
+from alexandria.core.models import File
+
+
+class Command(BaseCommand):
+    help = "Fixes copied file references by creating a new copy."
+
+    def add_arguments(self, parser):
+        parser.add_argument("--dry", dest="dry", action="store_true", default=False)
+
+    def handle(self, *args, **options):
+        filename_counts = {}
+        for tmp_file in tqdm(
+            File.objects.filter(variant="original").order_by("created_at")
+        ):
path: alexandria/core/management/commands/fix_copied_file_references.py
user: login: nlzet
body: Added iterator
created_at: 2025-04-24T09:55:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+import botocore
+from django.core.management.base import BaseCommand
+from tqdm import tqdm
+
+from alexandria.core.models import File
+
+
+class Command(BaseCommand):
+    help = "Fixes copied file references by creating a new copy."
+
+    def add_arguments(self, parser):
+        parser.add_argument("--dry", dest="dry", action="store_true", default=False)
+
+    def handle(self, *args, **options):
+        filename_counts = {}
+        for tmp_file in tqdm(
+            File.objects.filter(variant="original").order_by("created_at")
+        ):
+            filename = tmp_file.content.name
+            if filename in filename_counts:
+                filename_counts[filename].append(tmp_file.pk)
+            else:
+                filename_counts[filename] = [tmp_file.pk]
path: alexandria/core/management/commands/fix_copied_file_references.py
user: login: nlzet
body: Changed to defaultdict
created_at: 2025-04-24T09:55:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+import botocore
+from django.core.management.base import BaseCommand
+from tqdm import tqdm
+
+from alexandria.core.models import File
+
+
+class Command(BaseCommand):
+    help = "Fixes copied file references by creating a new copy."
+
+    def add_arguments(self, parser):
+        parser.add_argument("--dry", dest="dry", action="store_true", default=False)
+
+    def handle(self, *args, **options):
+        filename_counts = {}
+        for tmp_file in tqdm(
+            File.objects.filter(variant="original").order_by("created_at")
+        ):
+            filename = tmp_file.content.name
+            if filename in filename_counts:
+                filename_counts[filename].append(tmp_file.pk)
+            else:
+                filename_counts[filename] = [tmp_file.pk]
+
+        fix_count = 0
+        for filename, pks in filename_counts.items():
+            # Skip files that only have one reference.
+            if len(pks) <= 1:
+                continue
+
+            # Find duplicates, which are files with the same filename and no checksum.
+            duplicates = File.objects.filter(pk__in=pks).order_by("created_at")
+            checksum = duplicates.first().checksum
+
+            for duplicate in duplicates:
+                if duplicate.checksum != checksum:
+                    self.stderr.write(
+                        self.style.ERROR(
+                            f"Failed: File {duplicate.pk} with filename {filename} has a different checksum than the original."
+                        )
+                    )
+                    continue
+
+                current_filename = duplicate.content.name
+                new_filename = f"{duplicate.pk}_{duplicate.name}"
+
+                # Only fix copied files, skip the original
+                if current_filename == new_filename:
+                    continue
path: alexandria/core/management/commands/fix_copied_file_references.py
user: login: nlzet
body: Changed to exclude the original from the collection in the for loop
created_at: 2025-04-24T09:56:05Z
author_association: CONTRIBUTOR

PR comments:
user: login: martinwe-adfinis
created_at: 2025-04-17T14:25:13Z
body: This is more of a draft, this still needs:

 * Tests
 * Setting the correct name in the storage (just using `content.name` doesn't properly replace whitespaces etc.)

More importantly, however: The `Document` model provides a `clone()` method that appears to already do most of what we're doing in the API's `copy_document()` method. Is there anything that speaks against reusing that here?
author_association: CONTRIBUTOR
user: login: nlzet
created_at: 2025-04-22T10:51:12Z
body: @anehx added tests for the changes and a new proposed management command to fix existing copied file references
author_association: CONTRIBUTOR
user: login: winged
created_at: 2025-04-23T08:05:16Z
body: Please also add a new test that explicitly does the following procedure:

* Create a document & file
* Make a copy
* Delete original document & file
* Verify copy is still there and readable
author_association: CONTRIBUTOR
user: login: nlzet
created_at: 2025-04-23T10:50:59Z
body: @winged added a new test to explicitly test for content readability of copied files, after deleting the original document
author_association: CONTRIBUTOR