Repository: rucio/rucio

Issue:
Rules in state REPLICATING with no requests
Motivation
----------
We observed some cases where there a replication rules in state `REPLICATING` with no requests. Per definition of the state machine this should not be possible. 

The error was tracked down to the workflow of the `conveyor-finisher` which handles `FAILED` requests differently then `DONE` requests. For `DONE` requests the workflow is correct, for failed ones, in the 3rd failure attempt, the procedure is to archive the request first and at the end of the workflow update all failed replicas (and their rules). This update is done in a different transactions though. The issue is that if the update fails, e.g. due to row locks which is a valid case, there is no proper fallback to this, since the request has already been archived.

Modification
------------
The workflow needs to be changed similar to the `DONE` workflow, thus the request should only be archived AFTER the update was successful. 



Discussion:

Resolving patch:
diff --git a/lib/rucio/api/replica.py b/lib/rucio/api/replica.py
--- a/lib/rucio/api/replica.py
+++ b/lib/rucio/api/replica.py
@@ -19,6 +19,7 @@
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2017-2019
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
 # - Andrew Lister <andrew.lister@stfc.ac.uk>, 2019
+# - Martin Barisits <martin.barisits@cern.ch>, 2019
 #
 # PY3K COMPATIBLE
 
@@ -308,7 +309,7 @@ def list_dataset_replicas_vp(scope, name, deep=False):
         yield api_update_return_dict(r)
 
 
-def list_datasets_per_rse(rse, filters=None, limit=None):
+def list_datasets_per_rse(rse, filters={}, limit=None):
     """
     :param scope: The scope of the dataset.
     :param name: The name of the dataset.
diff --git a/lib/rucio/core/message.py b/lib/rucio/core/message.py
--- a/lib/rucio/core/message.py
+++ b/lib/rucio/core/message.py
@@ -15,7 +15,7 @@
 # Authors:
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2014-2019
 # - Vincent Garonne <vgaronne@gmail.com>, 2014-2017
-# - Martin Barisits <martin.barisits@cern.ch>, 2014-2016
+# - Martin Barisits <martin.barisits@cern.ch>, 2014-2019
 # - Robert Illingworth <illingwo@fnal.gov>, 2018
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018
 #
@@ -29,6 +29,7 @@
 
 
 from rucio.common.exception import InvalidObject, RucioException
+from rucio.common.utils import APIEncoder
 from rucio.db.sqla.models import Message, MessageHistory
 from rucio.db.sqla.session import transactional_session
 
@@ -46,7 +47,7 @@ def add_message(event_type, payload, session=None):
     """
 
     try:
-        payload = json.dumps(payload)
+        payload = json.dumps(payload, cls=APIEncoder)
     except TypeError as e:
         raise InvalidObject('Invalid JSON for payload: %(e)s' % locals())
 
diff --git a/lib/rucio/daemons/conveyor/finisher.py b/lib/rucio/daemons/conveyor/finisher.py
--- a/lib/rucio/daemons/conveyor/finisher.py
+++ b/lib/rucio/daemons/conveyor/finisher.py
@@ -16,7 +16,7 @@
 # - Wen Guan <wguan.icedew@gmail.com>, 2015-2016
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2015
 # - Vincent Garonne <vgaronne@gmail.com>, 2015-2018
-# - Martin Barisits <martin.barisits@cern.ch>, 2015-2017
+# - Martin Barisits <martin.barisits@cern.ch>, 2015-2019
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2017-2018
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018
 # - Andrew Lister <andrew.lister@stfc.ac.uk>, 2019
@@ -251,8 +251,8 @@ def __handle_requests(reqs, suspicious_patterns, retry_protocol_mismatches, prep
                 __check_suspicious_files(req, suspicious_patterns)
                 tss = time.time()
                 try:
-                    new_req = request_core.requeue_and_archive(req, retry_protocol_mismatches)
-                    if new_req:
+                    if request_core.should_retry_request(req, retry_protocol_mismatches):
+                        new_req = request_core.requeue_and_archive(req, retry_protocol_mismatches)
                         # should_retry_request and requeue_and_archive are not in one session,
                         # another process can requeue_and_archive and this one will return None.
                         record_timer('daemons.conveyor.common.update_request_state.request-requeue_and_archive', (time.time() - tss) * 1000)
@@ -278,8 +278,8 @@ def __handle_requests(reqs, suspicious_patterns, retry_protocol_mismatches, prep
                     continue
                 try:
                     tss = time.time()
-                    new_req = request_core.requeue_and_archive(req, retry_protocol_mismatches)
-                    if new_req:
+                    if request_core.should_retry_request(req, retry_protocol_mismatches):
+                        new_req = request_core.requeue_and_archive(req, retry_protocol_mismatches)
                         record_timer('daemons.conveyor.common.update_request_state.request-requeue_and_archive', (time.time() - tss) * 1000)
                         logging.warn(prepend_str + 'REQUEUED SUBMITTING DID %s:%s REQUEST %s AS %s TRY %s' % (req['scope'],
                                                                                                               req['name'],


Test patch: