Repository: hydroshare/hydroshare

Issue:
cleanup dangling files (existing in iRods, not in Django)
This resource: https://www.hydroshare.org/resource/f8199e05ce4f477abebfddc82eb5aa14/

I am suspicious that dangling files (existing in iRods, not in Django) might be caused by an issue in [data_store_remove_folder](https://github.com/hydroshare/hydroshare/blob/2.6.1/hs_core/views/resource_folder_hierarchy.py#L708)

I can see in the `access.log` that the final entry before we started seeing django `file_list` errors was the following:
> 152.54.2.89 - - [29/Jun/2023:20:51:57 +0000] "POST /hsapi/_internal/data-store-delete-folder/ HTTP/1.0" 499 0 "https://www.hydroshare.org/resource/f8199e05ce4f477abebfddc82eb5aa14/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"

Immediately preceding the first errors in the django log
```
$ grep f8199e05ce4f477abebfddc82eb5aa14 django.log
[29/Jun/2023 20:52:06] ERROR [django.request:224] Internal Server Error: /hsapi/resource/f8199e05ce4f477abebfddc82eb5aa14/file_list/
django.core.exceptions.ValidationError: ['f8199e05ce4f477abebfddc82eb5aa14/data/contents/Seismic/Line5/Line 5_08072021/5006.dat cannot be found in iRODS']
[29/Jun/2023 21:15:50] ERROR [django.request:224] Internal Server Error: /hsapi/resource/f8199e05ce4f477abebfddc82eb5aa14/file_list/
django.core.exceptions.ValidationError: ['f8199e05ce4f477abebfddc82eb5aa14/data/contents/Seismic/Line 5/Line 5_08072021/5524.dat cannot be found in iRODS']
```

It's perplexing though because it appears that the iRODS removal occurs _before_ the deletion in Django:
https://github.com/hydroshare/hydroshare/blob/2.6.1/hs_core/views/utils.py#L1458-L1460

But I'm nevertheless suspicious and we should take a gander at [data_store_remove_folder](https://github.com/hydroshare/hydroshare/blob/2.6.1/hs_core/views/resource_folder_hierarchy.py#L708)

_Originally posted by @devincowan in https://github.com/hydroshare/hydroshare/issues/5122#issuecomment-1633332749_
            


Discussion:
@devincowan Thanks for digging out this information. 

> It's perplexing though because it appears that the iRODS removal occurs before the deletion in Django:

Yes the folder delete deletes all files in a folder from iRODS first then tries to remove the same from the DB. In this case, it seems the removal of all files from DB probably failed in the midway. That is the reason file list can see files in DB but some of them are missing in iRODS. May be we have to make the folder delete operation an async operation. 
It is pretty easy to reproduce this behavior on HS v2.7.1

1. Make a folder, add a dozen files (they don't have to be large)
2. In the UI delete the folder and immediately refresh the page
3. run `repair_resource` and you will find dangling iRods files
Ran this [repair_resource command](https://github.com/hydroshare/hydroshare/blob/4432-resource-file-duplicates/hs_core/management/commands/repair_resource.py) with `--failfast` and `--dryrun` to get a feel for % resources that are in a "corrupt" state.

Made it through a third of our resources in 8hrs. Found 170 corrupt resources in that batch (3-4%).

This job was with `--failfast` so it was stopping on the first Django/iRods mismatch and it wasn't repairing anything. It stands to reason that any job that was actually repairing resources would take a VERY long time to complete (if at all). And that's assuming that we know which direction should be taken as "source of truth" (iRods or Django) when there is a discrepancy. To really be sure, we would have to check logs for each of the corrupt resources. Extrapolating 4% out to all of our resources, we are talking about checking logs for ~600 resources. Not possible manually.

I think our resources are better spent trying to prevent this issue moving forward, rather than bulk repairing previous resources.

We could run a script to detect all resources that have one or more files in iRods missing in Django, and email the owner of the resource with a list of said mismatch asking if they would like their files restored, with some followup process that would allow us efficiently handle those who respond with a "yes." But my guess is that most owners aren't even aware that their resource is in this state and an email along these lines could be baffling.
@devincowan 

> And that's assuming that we know which direction should be taken as "source of truth" (iRods or Django) when there is a discrepancy. 
If a file is missing in iRODS but present in Django DB then the file record from DB should be deleted. To fix any other possible discrepancies between iRODS and DB probably requires resource owner inputs. 
@pkdash good point that we can be confident repairing in the case that a file is missing in iRods but present in Django. I suspect that this is a very small percentage of the cases (because most cases I saw are due to incomplete delete jobs which remove from Django and leave in iRods).

EDIT: looks like 8 of the 170 corrupt resources from running the script fit this case (exist in Django, not in iRods).
Making a note here that I ran into this issue recently:

1. I did a bulk delete that was interrupted leaving dangling iRods files
2. I attempt to upload a file that I previously deleted (one that is not shown in Django)
3. The upload fails with a message that the "file already exists" (which is confusing because it is not shown in the Django UI)
After our call, I have 2 action items:

1. Revise our repair_resource management job so that we can run it periodically and it only targets recently edited resources
2. User messaging to notify when resources have been "repaired" (email generated at the time of the periodic task as well as UI notification on the first viewing of a repaired resource)
@martinseul (these numbers are for published resources only)
> Number of published resources that had at least one file issue: 466
> ****************************************************************************************************
> Total number of files missing in Django: 8132
> Number of published resources with at least one missing django file: 433
> ****************************************************************************************************
> Total number of files missing in iRods: 50470
> Number of published resources with at least one missing irods file: 53

Resolving patch:
diff --git a/hs_core/tasks.py b/hs_core/tasks.py
--- a/hs_core/tasks.py
+++ b/hs_core/tasks.py
@@ -196,7 +196,8 @@ def nightly_repair_resource_files():
         # followed by those previously checked, prioritizing the oldest checked date
         recently_updated_rids = [res.short_id for res in recently_updated_resources]
         not_recently_updated = BaseResource.objects \
-            .exclude(short_id__in=recently_updated_rids, raccess__published=True) \
+            .exclude(short_id__in=recently_updated_rids) \
+            .exclude(raccess__published=True) \
             .order_by(F('files_checked').asc(nulls_first=True))
         for res in not_recently_updated:
             check_time(start_time, settings.NIGHTLY_RESOURCE_REPAIR_DURATION)


Test patch: