Repository: svthalia/concrexit

Issue:
Delete files when models deleted?
In GitLab by @joostrijneveld on Jul 14, 2016, 21:45

There seems to be an easy system-wide way to accomplish this by installing [django-cleanup](https://github.com/un1t/django-cleanup), but is this something we want for every FileField and ImageField? Is there a scenario where we might reference the same logical file from two models?


Discussion:
In GitLab by @thomwiggers on Jul 27, 2016, 23:29

django-cleanup is al wel stil sinds december...
In GitLab by @thomwiggers on Aug 5, 2016, 11:33

Ik denk dat dit vooral relevant is voor avatars en fotoalbums. Ik stem in die gevallen voor verwijderen. Documenten lijken me minder kritisch: ze nemen niet heel veel ruimte in, het is een beperkte groep die ze uploadt, en een zekere impliciete archief-functie is niet heel kwalijk.
In GitLab by @joostrijneveld on Aug 7, 2016, 17:03

Avatars kunnen we makkelijk oplossen door gewoon op te slaan als `username.{jpg, png, gif}` en in de `save()` eventuele bestaande avatars even opruimen voor de gegeven username.. En voor photos de album-map verwijderen in de `delete()`? Thabloids overschrijven nu de bestaande files als het jaartal + issuenummer overeen komt, maar misschien moet daar ook nog een `delete()` gemaakt worden voor wanneer 't niet wordt overschreven (bijv. je upload een Thabloid voor 1984 en verwijdert 'm weer, dan blijft er een lading files over).

Klinkt inderdaad alsof het niet nodig is om hier speciaal een dependency voor te gebruiken die dit algemeen doet.
In GitLab by @se-bastiaan on May 27, 2019, 22:40

https://www.algotech.solutions/blog/python/deleting-unused-django-media-files/
In GitLab by @thomwiggers on May 28, 2019, 09:27

Doesn't that management command do practically the same as `django-cleanup`, linked in the first post?

The signals seem slightly iffy still, as they don't guarantee that no files will be left behind (like, if the application gets shut down or crashes before a save gets committed).
In GitLab by @se-bastiaan on May 28, 2019, 09:29

There is also a lib that does the same as the management command. I really like how it works. If it works. Django-cleanup uses signals and some cache. 

Signals are working well for registrations and the payment registration but it's much more complicated to build ourselves. A management command allows periodic cleaning and we can decide when exactly to run it

Resolving patch:
diff --git a/website/utils/management/commands/remove_unused_media.py b/website/utils/management/commands/remove_unused_media.py
new file mode 100644
--- /dev/null
+++ b/website/utils/management/commands/remove_unused_media.py
@@ -0,0 +1,197 @@
+import os
+import re
+
+from django.apps import apps
+from django.conf import settings
+from django.core.files.storage import DefaultStorage
+from django.core.management.base import BaseCommand
+from django.core.validators import EMPTY_VALUES
+from django.db import models
+from django.utils import timezone
+
+from thaliawebsite.storage.backend import get_public_storage
+
+
+def get_file_fields():
+    all_models = apps.get_models()
+    fields = []
+    for model in all_models:
+        for field in model._meta.get_fields():
+            if isinstance(field, models.FileField):
+                fields.append(field)
+    return fields
+
+
+def remove_empty_dirs(path=None):
+    if not path:
+        path = settings.MEDIA_ROOT
+
+    if not os.path.isdir(path):
+        return False
+
+    listdir = [os.path.join(path, filename) for filename in os.listdir(path)]
+
+    if all(list(map(remove_empty_dirs, listdir))):
+        os.rmdir(path)
+        return True
+    else:
+        return False
+
+
+def _get_used_thabloid_pages(storage, path):
+    file_name = os.path.splitext(os.path.basename(path))[0]
+    folder_path = os.path.join(os.path.dirname(path), "pages", file_name)
+    return set(
+        map(lambda x: os.path.join(folder_path, x), storage.listdir(folder_path)[1])
+    )
+
+
+def get_used_media(storage):
+    media = set()
+
+    for field in get_file_fields():
+        is_null = {
+            "%s__isnull" % field.name: True,
+        }
+        is_empty = {
+            "%s" % field.name: "",
+        }
+
+        if type(field.storage) != type(storage):
+            continue
+
+        for value in (
+            field.model._base_manager.values_list(field.name, flat=True)
+            .exclude(**is_empty)
+            .exclude(**is_null)
+        ):
+            if value not in EMPTY_VALUES:
+                if "thabloid" in str(field.model):
+                    media.update(_get_used_thabloid_pages(field.storage, value))
+                media.add(value)
+
+    return media
+
+
+def get_all_media(storage, minimum_file_age=None, folder=""):
+    media = set()
+    initial_time = timezone.now()
+
+    if not storage.exists(folder):
+        return []
+
+    dirs, files = storage.listdir(folder)
+
+    # Remove file locations in case they are nested
+    # (public is in the private media dir at the time of writing this code)
+    if settings.PUBLIC_MEDIA_LOCATION in dirs:
+        dirs.remove(settings.PUBLIC_MEDIA_LOCATION)
+    if settings.PRIVATE_MEDIA_LOCATION in dirs:
+        dirs.remove(settings.PRIVATE_MEDIA_LOCATION)
+
+    for name in files:
+        name = os.path.join(folder, name)
+        if minimum_file_age:
+            file_age = initial_time - storage.get_modified_time(name)
+
+            if file_age.total_seconds() < minimum_file_age:
+                continue
+        else:
+            media.add(name)
+
+    for name in dirs:
+        media.update(
+            get_all_media(storage, minimum_file_age, os.path.join(folder, name))
+        )
+
+    return media
+
+
+def get_unused_media(storage, minimum_file_age=None):
+    all_media = get_all_media(storage, minimum_file_age)
+    used_media = get_used_media(storage)
+
+    return all_media - used_media
+
+
+class Command(BaseCommand):
+
+    help = "Clean unused media files which have no reference in models"
+
+    def add_arguments(self, parser):
+
+        parser.add_argument(
+            "--noinput",
+            "--no-input",
+            dest="interactive",
+            action="store_false",
+            default=True,
+            help="Do not ask confirmation",
+        )
+
+        parser.add_argument(
+            "--minimum-file-age",
+            dest="minimum_file_age",
+            default=60,
+            type=int,
+            help="Skip files younger this age (sec)",
+        )
+
+        parser.add_argument(
+            "-n",
+            "--dry-run",
+            dest="dry_run",
+            action="store_true",
+            default=False,
+            help="Dry run without any affect on your data",
+        )
+
+    def _show_files_to_delete(self, storage_type, unused_media):
+        self.stdout.write(
+            f"Total {storage_type} files will be removed: {len(unused_media)}"
+        )
+
+    def handle(self, *args, **options):
+        private_storage = DefaultStorage()
+        unused_private_media = get_unused_media(
+            private_storage,
+            minimum_file_age=options.get("minimum_file_age"),
+        )
+
+        public_storage = get_public_storage()
+        unused_public_media = get_unused_media(
+            public_storage,
+            minimum_file_age=options.get("minimum_file_age"),
+        )
+
+        if not unused_public_media and not unused_private_media:
+            self.stdout.write("Nothing to delete. Exit")
+            return
+
+        if options.get("dry_run"):
+            self._show_files_to_delete("private", unused_private_media)
+            self._show_files_to_delete("public", unused_public_media)
+            self.stdout.write("Dry run. Exit.")
+            return
+
+        if options.get("interactive"):
+            self._show_files_to_delete("private", unused_private_media)
+            self._show_files_to_delete("public", unused_public_media)
+
+            question = f"Are you sure you want to remove {len(unused_private_media) + len(unused_public_media)} unused files? (y/N)"
+
+            if input(question).upper() != "Y":
+                self.stdout.write("Interrupted by user. Exit.")
+                return
+
+        for f in unused_private_media:
+            private_storage.delete(f)
+
+        for f in unused_public_media:
+            public_storage.delete(f)
+
+        remove_empty_dirs()
+
+        self.stdout.write(
+            f"Done. Total files removed: {len(unused_private_media) + len(unused_public_media)}"
+        )


Test patch: