internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
Reaper should stop operating if there is a judge-evaluator backlog
Motivation
----------
There is potential data loss if the judge-evaluator gets stuck and the reaper continues working. One possibility to prevent this, would be to stop the reaper once it is detected that the judge-evaluator has a significant backlog and restart it once the backlog is processed.


Discussion:
Discussed in [Dev Meeting on 2018-10-11](https://indico.cern.ch/event/760955/): We should do it like planned.

Resolving patch:
diff --git a/lib/rucio/rse/protocols/gfal.py b/lib/rucio/rse/protocols/gfal.py
--- a/lib/rucio/rse/protocols/gfal.py
+++ b/lib/rucio/rse/protocols/gfal.py
@@ -356,7 +356,7 @@ def stat(self, path):
 
         message = "\n"
         try:
-            ret[PREFERRED_CHECKSUM] = ctx.checksum(path, str(PREFERRED_CHECKSUM.capitalize()))
+            ret[PREFERRED_CHECKSUM] = ctx.checksum(path, str(PREFERRED_CHECKSUM.upper()))
             return ret
         except Exception as error:
             message += 'Error while processing gfal checksum call (%s). Error: %s \n' % (PREFERRED_CHECKSUM, str(error))
@@ -365,7 +365,7 @@ def stat(self, path):
             if checksum_name == PREFERRED_CHECKSUM:
                 continue
             try:
-                ret[checksum_name] = ctx.checksum(path, str(checksum_name.capitalize()))
+                ret[checksum_name] = ctx.checksum(path, str(checksum_name.upper()))
                 return ret
             except Exception as error:
                 message += 'Error while processing gfal checksum call (%s). Error: %s \n' % (checksum_name, str(error))
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.21.6',
-    'branch_nick': 'patch-0-Release__1_21_6_preparation',
-    'revision_id': '5f9e6db16019ae94518f6d7e591d611ceb925c03',
-    'revno': 9097
+    'version': '1.21.7',
+    'branch_nick': 'patch-0-Release__1_21_7_preparation',
+    'revision_id': '8e7e22354ba4ca5cfbc8ea945dbe8c6707e42c02',
+    'revno': 9122
 }


Test patch:
Repository: rucio/rucio

Issue:
Declaring temporary unavailable replicas also declares the files lost
Motivation
----------
Temporary unavailable replicas : CLI bug that declare the files lost





Reaper should stop operating if there is a judge-evaluator backlog
Motivation
----------
There is potential data loss if the judge-evaluator gets stuck and the reaper continues working. One possibility to prevent this, would be to stop the reaper once it is detected that the judge-evaluator has a significant backlog and restart it once the backlog is processed.


Discussion:

Discussed in [Dev Meeting on 2018-10-11](https://indico.cern.ch/event/760955/): We should do it like planned.

Resolving patch:
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.21.6',
-    'branch_nick': 'patch-0-Release__1_21_6_preparation',
-    'revision_id': '5f9e6db16019ae94518f6d7e591d611ceb925c03',
-    'revno': 9097
+    'version': '1.21.7',
+    'branch_nick': 'patch-0-Release__1_21_7_preparation',
+    'revision_id': '8e7e22354ba4ca5cfbc8ea945dbe8c6707e42c02',
+    'revno': 9122
 }


Test patch:
Repository: rucio/rucio

Issue:
Bug in the selection of compatible checksum in the conveyor
Motivation
----------
Bug in the selection of compatible checksum in the conveyor : If no checksum type is specified as a RSE attribute, it assumes that all checksum are supported, and it override `verify_checksum`






Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -771,7 +771,8 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     overwrite = False
                     transfer_dst_type = "TAPE"
 
-                # IV - get external_host
+                # IV - get external_host + strict_copy
+                strict_copy = rse_attrs[dest_rse_id].get('strict_copy', False)
                 fts_hosts = rse_attrs[dest_rse_id].get('fts', None)
                 source_globus_endpoint_id = rse_attrs[source_rse_id].get('globus_endpoint_id', None)
                 dest_globus_endpoint_id = rse_attrs[dest_rse_id].get('globus_endpoint_id', None)
@@ -853,6 +854,8 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                 if multihop:
                     transfers[req_id]['multihop'] = True
                     transfers[req_id]['initial_request_id'] = req_id
+                if strict_copy:
+                    transfers[req_id]['strict_copy'] = strict_copy
 
             else:
                 current_schemes = transfers[req_id]['schemes']
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -281,8 +281,8 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, source_strateg
                     # source and dst support some bot none in common (dst priority)
                     verify_checksum = 'destination'
                 else:
-                    # source and dst have some in common
-                    verify_checksum = 'both'
+                    # Don't override the value in the file_metadata
+                    pass
 
         t_file = {'sources': transfer['sources'],
                   'destinations': transfer['dest_urls'],


Test patch:
Repository: rucio/rucio

Issue:
strict_copy mode not properly working
Motivation
----------
Conveyor : strict_copy mode not properly working





Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -35,6 +35,7 @@
 import imp
 import json
 import logging
+import re
 import time
 import traceback
 
@@ -771,6 +772,11 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     overwrite = False
                     transfer_dst_type = "TAPE"
 
+                sign_url = rse_attrs[dest_rse_id].get('sign_url', None)
+                if sign_url == 'gcs':
+                    dest_url = re.sub('davs', 'gclouds', dest_url)
+                    dest_url = re.sub('https', 'gclouds', dest_url)
+
                 # IV - get external_host + strict_copy
                 strict_copy = rse_attrs[dest_rse_id].get('strict_copy', False)
                 fts_hosts = rse_attrs[dest_rse_id].get('fts', None)
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -303,6 +303,7 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, source_strateg
                 set_checksum_value(t_file, dest_supported_checksums)
 
         multihop = transfer.get('multihop', False)
+        strict_copy = transfer.get('strict_copy', False)
 
         external_host = transfer['external_host']
         scope = t_file['metadata']['scope']
@@ -328,6 +329,8 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, source_strateg
                       's3alternate': True}
         if multihop:
             job_params['multihop'] = True
+        if strict_copy:
+            job_params['strict_copy'] = True
 
         # Don't put optional & missing keys in the parameters
         if transfer['dest_spacetoken']:


Test patch:
Repository: rucio/rucio

Issue:
rucio-admin script does not call the get_account_limits() with the correct local/global locality
The rucio-admin script does not call the get_account_limits() with the correct local/global locality. Need to implement a wrapper function in order to select the correct option.



Discussion:

Resolving patch:
diff --git a/lib/rucio/client/accountclient.py b/lib/rucio/client/accountclient.py
--- a/lib/rucio/client/accountclient.py
+++ b/lib/rucio/client/accountclient.py
@@ -242,6 +242,23 @@ def list_account_rules(self, account):
             exc_cls, exc_msg = self._get_exception(headers=res.headers, status_code=res.status_code, data=res.content)
             raise exc_cls(exc_msg)
 
+    def get_account_limits(self, account, rse_expression, locality):
+        """
+        Return the correct account limits for the given locality.
+
+        :param account:        The account name.
+        :param rse_expression: Valid RSE expression
+        :param locality:       The scope of the account limit. 'local' or 'global'.
+        """
+
+        if locality == 'local':
+            return self.get_local_account_limit(account, rse_expression)
+        elif locality == 'global':
+            return self.get_global_account_limit(account, rse_expression)
+        else:
+            from rucio.common.exception import UnsupportedOperation
+            raise UnsupportedOperation('The provided locality (%s) for the account limit was invalid' % locality)
+
     def get_global_account_limit(self, account, rse_expression):
         """
         List the account limit for the specific RSE expression.


Test patch:
Repository: rucio/rucio

Issue:
Modify RSEs regex for Belleii schema
Motivation
----------
The current regex doesn't fit to Belle2 convention




Discussion:
+ New naming convention

Resolving patch:
diff --git a/lib/rucio/common/schema/belleii.py b/lib/rucio/common/schema/belleii.py
--- a/lib/rucio/common/schema/belleii.py
+++ b/lib/rucio/common/schema/belleii.py
@@ -15,6 +15,7 @@
 # Authors:
 # - Martin Barisits <martin.barisits@cern.ch>, 2019
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2019
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2020
 #
 # PY3K COMPATIBLE
 
@@ -84,7 +85,7 @@
 
 RSE = {"description": "RSE name",
        "type": "string",
-       "pattern": "^([A-Z0-9]+([_-][A-Z0-9]+)*)$"}
+       "pattern": "^([A-Z])+((([A-Za-z0-9]+))*(-)?)*$"}
 
 RSE_ATTRIBUTE = {"description": "RSE attribute",
                  "type": "string",
diff --git a/lib/rucio/rse/protocols/protocol.py b/lib/rucio/rse/protocols/protocol.py
--- a/lib/rucio/rse/protocols/protocol.py
+++ b/lib/rucio/rse/protocols/protocol.py
@@ -16,7 +16,7 @@
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2012-2014
 # - Vincent Garonne <vgaronne@gmail.com>, 2012-2016
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2019
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2016
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2020
 # - Wen Guan <wguan.icedew@gmail.com>, 2014-2017
 # - Cheng-Hsi Chao <cheng-hsi.chao@cern.ch>, 2014
 # - Tobias Wegner <twegner@cern.ch>, 2017
@@ -153,6 +153,28 @@ def __identity(scope, name, rse, rse_attrs, protocol_attrs):
             scope = scope.replace('.', '/')
         return '%s/%s' % (scope, name)
 
+    @staticmethod
+    def __belleii(scope, name, rse, rse_attrs, protocol_attrs):
+        """
+        Given a LFN, convert it directly to a path using the mapping:
+
+            path -> path
+        This is valid only for the belleii convention where the scope can be determined
+        from the LFN using a determinitic function.
+
+        :param scope: Scope of the LFN.
+        :param name: File name of the LFN.
+        :param rse: RSE for PFN (ignored)
+        :param rse_attrs: RSE attributes for PFN (ignored)
+        :param protocol_attrs: RSE protocol attributes for PFN (ignored)
+        :returns: Path for use in the PFN generation.
+        """
+        del scope
+        del rse
+        del rse_attrs
+        del protocol_attrs
+        return name
+
     @staticmethod
     def __ligo(scope, name, rse, rse_attrs, protocol_attrs):
         """
@@ -182,6 +204,7 @@ def _module_init_(cls):
         cls.register(cls.__hash, "hash")
         cls.register(cls.__identity, "identity")
         cls.register(cls.__ligo, "ligo")
+        cls.register(cls.__belleii, "belleii")
         policy_module = None
         try:
             policy_module = config.config_get('policy', 'lfn2pfn_module')


Test patch:
Repository: rucio/rucio

Issue:
strict_copy mode not properly working
Motivation
----------
Conveyor : strict_copy mode not properly working



rucio-admin script does not call the get_account_limits() with the correct local/global locality
The rucio-admin script does not call the get_account_limits() with the correct local/global locality. Need to implement a wrapper function in order to select the correct option.



Discussion:

Resolving patch:
diff --git a/lib/rucio/client/accountclient.py b/lib/rucio/client/accountclient.py
--- a/lib/rucio/client/accountclient.py
+++ b/lib/rucio/client/accountclient.py
@@ -242,6 +242,23 @@ def list_account_rules(self, account):
             exc_cls, exc_msg = self._get_exception(headers=res.headers, status_code=res.status_code, data=res.content)
             raise exc_cls(exc_msg)
 
+    def get_account_limits(self, account, rse_expression, locality):
+        """
+        Return the correct account limits for the given locality.
+
+        :param account:        The account name.
+        :param rse_expression: Valid RSE expression
+        :param locality:       The scope of the account limit. 'local' or 'global'.
+        """
+
+        if locality == 'local':
+            return self.get_local_account_limit(account, rse_expression)
+        elif locality == 'global':
+            return self.get_global_account_limit(account, rse_expression)
+        else:
+            from rucio.common.exception import UnsupportedOperation
+            raise UnsupportedOperation('The provided locality (%s) for the account limit was invalid' % locality)
+
     def get_global_account_limit(self, account, rse_expression):
         """
         List the account limit for the specific RSE expression.


Test patch:
Repository: rucio/rucio

Issue:
strict_copy mode not properly working
Motivation
----------
Conveyor : strict_copy mode not properly working



rucio-admin script does not call the get_account_limits() with the correct local/global locality
The rucio-admin script does not call the get_account_limits() with the correct local/global locality. Need to implement a wrapper function in order to select the correct option.



Discussion:

Resolving patch:
diff --git a/lib/rucio/client/accountclient.py b/lib/rucio/client/accountclient.py
--- a/lib/rucio/client/accountclient.py
+++ b/lib/rucio/client/accountclient.py
@@ -242,6 +242,23 @@ def list_account_rules(self, account):
             exc_cls, exc_msg = self._get_exception(headers=res.headers, status_code=res.status_code, data=res.content)
             raise exc_cls(exc_msg)
 
+    def get_account_limits(self, account, rse_expression, locality):
+        """
+        Return the correct account limits for the given locality.
+
+        :param account:        The account name.
+        :param rse_expression: Valid RSE expression
+        :param locality:       The scope of the account limit. 'local' or 'global'.
+        """
+
+        if locality == 'local':
+            return self.get_local_account_limit(account, rse_expression)
+        elif locality == 'global':
+            return self.get_global_account_limit(account, rse_expression)
+        else:
+            from rucio.common.exception import UnsupportedOperation
+            raise UnsupportedOperation('The provided locality (%s) for the account limit was invalid' % locality)
+
     def get_global_account_limit(self, account, rse_expression):
         """
         List the account limit for the specific RSE expression.
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -791,10 +791,11 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     continue
                 if retry_count is None:
                     retry_count = 0
+                external_host = ''
                 if fts_hosts:
                     fts_list = fts_hosts.split(",")
+                    external_host = fts_list[0]
 
-                external_host = fts_list[0]
                 if retry_other_fts:
                     external_host = fts_list[retry_count % len(fts_list)]
 


Test patch:
Repository: rucio/rucio

Issue:
Partitioning algorithm wrong for thread_id=0
Motivation
----------
The partitioning algorithm is not working correctly for `thread_id=0`. It then wrongly goes to a mode assuming there are no partitions.



Discussion:

Resolving patch:
diff --git a/lib/rucio/db/sqla/__init__.py b/lib/rucio/db/sqla/__init__.py
--- a/lib/rucio/db/sqla/__init__.py
+++ b/lib/rucio/db/sqla/__init__.py
@@ -9,7 +9,7 @@
 
 def filter_thread_work(session, query, total_threads, thread_id, hash_variable=None):
     """ Filters a query to partition thread workloads based on the thread id and total number of threads """
-    if thread_id and total_threads and (total_threads - 1) > 0:
+    if thread_id is not None and total_threads is not None and (total_threads - 1) > 0:
         if session.bind.dialect.name == 'oracle':
             bindparams = [bindparam('thread_id', thread_id), bindparam('total_threads', total_threads - 1)]
             if not hash_variable:


Test patch:
Repository: rucio/rucio

Issue:
Update XRootD certs
Motivation
----------

For Kubernetes.


Modify RSEs regex for Belleii schema
Motivation
----------
The current regex doesn't fit to Belle2 convention




Discussion:

+ New naming convention

Resolving patch:
diff --git a/lib/rucio/common/schema/belleii.py b/lib/rucio/common/schema/belleii.py
--- a/lib/rucio/common/schema/belleii.py
+++ b/lib/rucio/common/schema/belleii.py
@@ -15,6 +15,7 @@
 # Authors:
 # - Martin Barisits <martin.barisits@cern.ch>, 2019
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2019
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2020
 #
 # PY3K COMPATIBLE
 
@@ -84,7 +85,7 @@
 
 RSE = {"description": "RSE name",
        "type": "string",
-       "pattern": "^([A-Z0-9]+([_-][A-Z0-9]+)*)$"}
+       "pattern": "^([A-Z])+((([A-Za-z0-9]+))*(-)?)*$"}
 
 RSE_ATTRIBUTE = {"description": "RSE attribute",
                  "type": "string",
diff --git a/lib/rucio/rse/protocols/protocol.py b/lib/rucio/rse/protocols/protocol.py
--- a/lib/rucio/rse/protocols/protocol.py
+++ b/lib/rucio/rse/protocols/protocol.py
@@ -16,7 +16,7 @@
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2012-2014
 # - Vincent Garonne <vgaronne@gmail.com>, 2012-2016
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2019
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2016
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2020
 # - Wen Guan <wguan.icedew@gmail.com>, 2014-2017
 # - Cheng-Hsi Chao <cheng-hsi.chao@cern.ch>, 2014
 # - Tobias Wegner <twegner@cern.ch>, 2017
@@ -153,6 +153,28 @@ def __identity(scope, name, rse, rse_attrs, protocol_attrs):
             scope = scope.replace('.', '/')
         return '%s/%s' % (scope, name)
 
+    @staticmethod
+    def __belleii(scope, name, rse, rse_attrs, protocol_attrs):
+        """
+        Given a LFN, convert it directly to a path using the mapping:
+
+            path -> path
+        This is valid only for the belleii convention where the scope can be determined
+        from the LFN using a determinitic function.
+
+        :param scope: Scope of the LFN.
+        :param name: File name of the LFN.
+        :param rse: RSE for PFN (ignored)
+        :param rse_attrs: RSE attributes for PFN (ignored)
+        :param protocol_attrs: RSE protocol attributes for PFN (ignored)
+        :returns: Path for use in the PFN generation.
+        """
+        del scope
+        del rse
+        del rse_attrs
+        del protocol_attrs
+        return name
+
     @staticmethod
     def __ligo(scope, name, rse, rse_attrs, protocol_attrs):
         """
@@ -182,6 +204,7 @@ def _module_init_(cls):
         cls.register(cls.__hash, "hash")
         cls.register(cls.__identity, "identity")
         cls.register(cls.__ligo, "ligo")
+        cls.register(cls.__belleii, "belleii")
         policy_module = None
         try:
             policy_module = config.config_get('policy', 'lfn2pfn_module')


Test patch:
Repository: rucio/rucio

Issue:
Bug in archive content history
Motivation
----------
Traceback
```
2020-03-05 13:38:04,692 11088   CRITICAL        Thread [33/220] :  Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/daemons/reaper/reaper2.py", line 518, in reaper
    delete_replicas(rse_id=rse_id, files=deleted_files)
  File "/usr/lib/python2.7/site-packages/rucio/db/sqla/session.py", line 374, in new_funct
    raise DatabaseException(str(error))
DatabaseException: Database exception.
Details: (cx_Oracle.IntegrityError) ORA-02292: integrity constraint (ATLAS_RUCIO.DATASET_LOCKS_DID_FK) violated - child record found
```






Discussion:

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -2005,6 +2005,7 @@ def insert_content_history(content_clause, did_created_at, session=None):
     :param did_created_at: Creation date of the did
     :param session: The database session in use.
     """
+    new_did_created_at = did_created_at
     query = session.query(models.DataIdentifierAssociation.scope,
                           models.DataIdentifierAssociation.name,
                           models.DataIdentifierAssociation.child_scope,
@@ -2022,6 +2023,8 @@ def insert_content_history(content_clause, did_created_at, session=None):
         filter(or_(*content_clause))
 
     for cont in query.all():
+        if not did_created_at:
+            new_did_created_at = cont.created_at
         models.DataIdentifierAssociationHistory(
             scope=cont.scope,
             name=cont.name,
@@ -2037,6 +2040,6 @@ def insert_content_history(content_clause, did_created_at, session=None):
             rule_evaluation=cont.rule_evaluation,
             updated_at=cont.updated_at,
             created_at=cont.created_at,
-            did_created_at=did_created_at,
+            did_created_at=new_did_created_at,
             deleted_at=datetime.utcnow()
         ).save(session=session, flush=False)


Test patch: