Repository: hydroshare/hydroshare

Issue:
If no downloads within a year, should we "offboard" a resource's zipped bagit archive
As we contemplate changing our storage backend and aim to turn on quota enforcement, it would be good to contemplate how much of our storage is being used by "cold" files that no one is accessing. Zipped bagit archives could be removed after some interval of inactivity (I propose a year)--in this case "inactivity" would mean no recent downloads.


Discussion:
What does offboarding mean in terms of the longevity of the content on HS?
I'm assuming that by "removing the bag" you just mean the zipped bagit bag containing the resource's files. The content files themselves would still remain in HydroShare, and the bag would be recreated at the next event that triggers bag creation, correct? I definitely don't think I would advocate at all for removing resources/content after a year that hasn't been downloaded (but I don't think that's what you mean). 

The Bagit packaging scheme includes checksums for files. Those checksums are meant to ensure the integrity of the data. If the bag is moved from one place to another, you can check the checksums and make sure you got all of the content with fidelity. I think that is mostly for data transfer, and I don't know of anyone or any applications that actually do anything with the checksums, but maybe a related question here would be - if you remove the bag, are there any mechanisms for ensuring the integrity of the stored files over time or across moves? What if HydroShare migrates everything to cloud storage? How will we ensure the integrity of all content when it is moved?
The storage systems themselves ensure data integrity/durability.  Amazon claims [99.999999999% durability](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html).  Checksums may also be used in data transfers for [object integrity](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html).

@devincowan - do you have stats on how much storage the bags use?  How much storage would we save if implemented today?
Sorry for the confusion @ClaraCogswell and @horsburgh. Jeff's interpretation was indeed my intent, I updated the issue name and description to make it more clear.

And Scott has a good point, we should crunch some stats. I will do that

Resolving patch:
diff --git a/hs_core/management/commands/compute_bag_size_depending_on_last_update.py b/hs_core/management/commands/compute_bag_size_depending_on_last_update.py
new file mode 100644
--- /dev/null
+++ b/hs_core/management/commands/compute_bag_size_depending_on_last_update.py
@@ -0,0 +1,54 @@
+import math
+from django.core.management.base import BaseCommand
+
+from hs_core.models import BaseResource
+from django.core.exceptions import ValidationError
+from django.utils import timezone
+from datetime import timedelta
+
+
+class Command(BaseCommand):
+    help = "Compute the size of the bagit archive size/storage for resources that haven't been updated recently"
+
+    def add_arguments(self, parser):
+
+        parser.add_argument('--weeks', type=int, dest='weeks', help='include res not updated in the last X weeks')
+
+    def convert_size(self, size_bytes):
+        if size_bytes == 0:
+            return "0B"
+        size_name = ("B", "KB", "MB", "GB", "TB")
+        i = int(math.floor(math.log(size_bytes, 1024)))
+        p = math.pow(1024, i)
+        s = round(size_bytes / p, 2)
+        return "%s %s" % (s, size_name[i])
+
+    def handle(self, *args, **options):
+        resources = []
+        add_message = ""
+        weeks = options['weeks']
+        resources = BaseResource.objects.all()
+
+        if weeks:
+            print(f"FILTERING TO INCLUDE RESOURCES NOT UPDATED IN UPDATED IN LAST {weeks} WEEKS")
+            add_message = f"for resources not updated in last {weeks} weeks"
+            cuttoff_time = timezone.now() - timedelta(weeks=weeks)
+            resources = resources.filter(updated__lte=cuttoff_time)
+        cumulative_size = 0
+        count = len(resources)
+        counter = 1
+        print(f"Iterating {count} resources...")
+        for res in resources:
+            converted_size = self.convert_size(cumulative_size)
+            print(f"{counter}/{count}: cumulative thus far = {converted_size}")
+            try:
+                src_file = res.bag_path
+                istorage = res.get_irods_storage()
+                fsize = istorage.size(src_file)
+                cumulative_size += fsize
+                print(f"Size for {src_file} = {fsize}")
+            except ValidationError as ve:
+                print(f"Size not computed for {res.short_id}: {ve}")
+            counter += 1
+        converted_size = self.convert_size(cumulative_size)
+        print(f"Total cumulative size {add_message}: {converted_size}")


Test patch: