Repository: rucio/rucio

Issue:
Report deletion events when the destination does not exist
Motivation
----------

Currently, when the Reaper fails to delete a file because the destination does not exist, no event is reported. This can be potentially useful information.

Modification
------------

Report a `deletion-notfound` event.



Discussion:
~~resolved #4272~~
https://github.com/rucio/rucio/blob/69e41371ab367407347a5a9f1a8aa95a6b5a1f8b/lib/rucio/daemons/reaper/reaper2.py#L577
> resolved #4272 .

Sorry, that PR is not related. The relevant part of the code is here instead:

https://github.com/rucio/rucio/blob/69e41371ab367407347a5a9f1a8aa95a6b5a1f8b/lib/rucio/daemons/reaper/reaper2.py#L192-L195
Oops, sorry! But still isn't the desired message logged. Or is it that, I did not just point to the wrong code but also mixed up events with logs.  Kindly, direct me to an example/definition of an event in Rucio.
An event is reported with `add_message()`. The real question is whether additional changes are needed in order to process it, either in Rucio or in our monitoring infrastructure (which lives outside of this repository). To that, I don’t have an answer.
Making a distinction between "deletion success" and "deletion not found" is potentially helpful. Down the line, this gives us some statistics on files that were lost on storage. If there are data centres with particularly high numbers of deletion not found that can inform us about more systematic problems.

Resolving patch:
diff --git a/lib/rucio/daemons/reaper/reaper.py b/lib/rucio/daemons/reaper/reaper.py
--- a/lib/rucio/daemons/reaper/reaper.py
+++ b/lib/rucio/daemons/reaper/reaper.py
@@ -197,16 +197,22 @@ def delete_from_storage(replicas, prot, rse_info, staging_areas, auto_exclude_th
 
                 deletion_dict['duration'] = duration
                 add_message('deletion-done', deletion_dict)
-                logger(logging.INFO, 'Deletion SUCCESS of %s:%s as %s on %s in %s seconds', replica['scope'], replica['name'], replica['pfn'], rse_name, duration)
+                logger(logging.INFO, 'Deletion SUCCESS of %s:%s as %s on %s in %.2f seconds', replica['scope'], replica['name'], replica['pfn'], rse_name, duration)
 
             except SourceNotFound:
-                err_msg = 'Deletion NOTFOUND of %s:%s as %s on %s' % (replica['scope'], replica['name'], replica['pfn'], rse_name)
+                duration = time.time() - start
+                err_msg = 'Deletion NOTFOUND of %s:%s as %s on %s in %.2f seconds' % (replica['scope'], replica['name'], replica['pfn'], rse_name, duration)
                 logger(logging.WARNING, '%s', err_msg)
+                deletion_dict['reason'] = 'File Not Found'
+                deletion_dict['duration'] = duration
+                add_message('deletion-not-found', deletion_dict)
                 deleted_files.append({'scope': replica['scope'], 'name': replica['name']})
 
             except (ServiceUnavailable, RSEAccessDenied, ResourceTemporaryUnavailable) as error:
-                logger(logging.WARNING, 'Deletion NOACCESS of %s:%s as %s on %s: %s', replica['scope'], replica['name'], replica['pfn'], rse_name, str(error))
+                duration = time.time() - start
+                logger(logging.WARNING, 'Deletion NOACCESS of %s:%s as %s on %s: %s in %.2f', replica['scope'], replica['name'], replica['pfn'], rse_name, str(error), duration)
                 deletion_dict['reason'] = str(error)
+                deletion_dict['duration'] = duration
                 add_message('deletion-failed', deletion_dict)
                 noaccess_attempts += 1
                 if noaccess_attempts >= auto_exclude_threshold:
@@ -217,8 +223,10 @@ def delete_from_storage(replicas, prot, rse_info, staging_areas, auto_exclude_th
                     break
 
             except Exception as error:
-                logger(logging.CRITICAL, 'Deletion CRITICAL of %s:%s as %s on %s: %s', replica['scope'], replica['name'], replica['pfn'], rse_name, str(traceback.format_exc()))
+                duration = time.time() - start
+                logger(logging.CRITICAL, 'Deletion CRITICAL of %s:%s as %s on %s in %.2f seconds : %s', replica['scope'], replica['name'], replica['pfn'], rse_name, duration, str(traceback.format_exc()))
                 deletion_dict['reason'] = str(error)
+                deletion_dict['duration'] = duration
                 add_message('deletion-failed', deletion_dict)
 
         if pfns_to_bulk_delete and prot.attributes['scheme'] == 'globus':


Test patch: