Repository: hydroshare/hydroshare

Issue:
metrics should track resource_id in Resources index
**Description of the bug**
We have multiple ES documents for a single resource. Here is an example resource:
https://www.hydroshare.org/resource/d0d9c63a7a794129a247b2cfd617ad13/

But wait how's it possible that we have multiple docs for a single resource?
First I checked the logstash pipeline to see how we were uniquely identifying. We fingerprint based on res_created_date + usr_id
https://github.com/hydroshare/docker-elk/blob/main/logstash/pipeline/12-hs-statistics-resources.conf#L34-L39
Ideally this would be on res_id but we dont have that (I've made a note to add it). Okay that seems like it should work...

But uh oh, when I check in the actual log files:
```
$ grep -r "South Carolina Piedmont," resources-details*
...REDACTED FOR BREVITY
resources-details-08.29.2023.log:08/29/2023,06/29/2023 17:39:26.771094,South Carolina Piedmont,CompositeResource,85962041699,public,University Faculty,11438
resources-details-08.30.2023.log:08/30/2023,06/29/2023 17:39:26.771094,South Carolina Piedmont,CompositeResource,85962041699,public,University Faculty,11438
resources-details-08.31.2023.log:08/31/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,48575117,private,University Faculty,11438
resources-details-09.01.2023.log:09/01/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,48575117,private,University Faculty,11438
resources-details-09.02.2023.log:09/02/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,5327823407,private,University Faculty,11438
resources-details-09.03.2023.log:09/03/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,5327823407,private,University Faculty,11438
resources-details-09.04.2023.log:09/04/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,5327823407,private,University Faculty,11438
resources-details-09.05.2023.log:09/05/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,5327823407,private,University Faculty,11438
resources-details-09.06.2023.log:09/06/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,17288658016,private,University Faculty,11438
resources-details-09.07.2023.log:09/07/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,17288658016,private,University Faculty,11438
resources-details-09.08.2023.log:09/08/2023,08/30/2023 16:00:00.854193,South Carolina Piedmont,CompositeResource,22116214503,private,University Faculty,11438
resources-details-09.09.2023.log:09/09/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,22123484988,public,University Faculty,11438
resources-details-09.10.2023.log:09/10/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,22123484988,public,University Faculty,11438
resources-details-09.11.2023.log:09/11/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,22123484988,public,University Faculty,11438
resources-details-09.12.2023.log:09/12/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,22123484988,public,University Faculty,11438
resources-details-09.13.2023.log:09/13/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,22123484988,public,University Faculty,11438
resources-details-09.14.2023.log:09/14/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.15.2023.log:09/15/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.16.2023.log:09/16/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.17.2023.log:09/17/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.18.2023.log:09/18/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.19.2023.log:09/19/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.20.2023.log:09/20/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.21.2023.log:09/21/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.22.2023.log:09/22/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.23.2023.log:09/23/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.24.2023.log:09/24/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.25.2023.log:09/25/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
resources-details-09.26.2023.log:09/26/2023,09/08/2023 16:21:00.257259,South Carolina Piedmont,CompositeResource,19290808932,public,University Faculty,11438
```
^^^see how the second date (which we take as resource creation when ingesting) is changing! Hmmm. Resource creation date should stay constant. It looks to me like we are grabbing the updated date instead of the created date.

But when I check how we are dumping the res_created_date in the stats management command:
https://github.com/hydroshare/hydroshare/blob/2.9.2/hs_tracking/management/commands/stats.py#L159-L160
Hmm that seems like it should be grabbing the created date... so I'm not seeing where the issue is. Keeping looking.

I guess it's possible, since we don't track the resource_id, that every time the resource created date is changing, it's because this user actually did create a new resource in HS with the same name. But that doesn't really look to be the case when we compare with the "create" activity index for this user

As a start, we should track resource_id in our Resources index, so hope that will help us figure these things out




Discussion:

Resolving patch:
diff --git a/hs_tracking/management/commands/stats.py b/hs_tracking/management/commands/stats.py
--- a/hs_tracking/management/commands/stats.py
+++ b/hs_tracking/management/commands/stats.py
@@ -149,7 +149,8 @@ def resources_details(self):
             'size',
             'publication status',
             'user type',
-            'user id'
+            'user id',
+            'resource id'
         ]
         w.writerow(fields)
         failed_resource_ids = []
@@ -163,7 +164,8 @@ def resources_details(self):
                     r.size,
                     r.raccess.sharing_status,
                     r.user.userprofile.user_type,
-                    r.user_id
+                    r.user_id,
+                    r.short_id
                 ]
                 w.writerow([str(v) for v in values])
 


Test patch: