internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
do not skip CA check on retry
Motivation
----------
If the server can't be connected the retry will ignore the CA.

https://github.com/rucio/rucio/blob/master/lib/rucio/client/baseclient.py#L353


Discussion:

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -1,4 +1,4 @@
-# Copyright 2012-2018 CERN for the benefit of the ATLAS collaboration.
+# Copyright 2012-2020 CERN for the benefit of the ATLAS collaboration.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,7 +16,7 @@
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2012-2013
 # - Vincent Garonne <vgaronne@gmail.com>, 2012-2018
 # - Yun-Pin Sun <winter0128@gmail.com>, 2013
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2019
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2020
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2014-2015
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2015
 # - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2015
@@ -350,7 +350,6 @@ def _send_request(self, url, headers=None, type='GET', data=None, params=None, s
                     return
             except ConnectionError as error:
                 LOG.warning('ConnectionError: ' + str(error))
-                self.ca_cert = False
                 if retry > self.request_retries:
                     raise
                 continue


Test patch:
Repository: rucio/rucio

Issue:
Conveyor submitter creates partitions wrongly
Motivation
----------
There is an unnecessary `-1` when creating the partitions for the conveyor-submitter.

Modification
------------
https://github.com/rucio/rucio/blob/930bf853c1a3399592a974108c8805c24ae859a2/lib/rucio/daemons/conveyor/submitter.py#L160
Remove the `-1` there


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/submitter.py b/lib/rucio/daemons/conveyor/submitter.py
--- a/lib/rucio/daemons/conveyor/submitter.py
+++ b/lib/rucio/daemons/conveyor/submitter.py
@@ -17,7 +17,7 @@
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2019
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2013
 # - Vincent Garonne <vgaronne@gmail.com>, 2014-2018
-# - Martin Barisits <martin.barisits@cern.ch>, 2014-2017
+# - Martin Barisits <martin.barisits@cern.ch>, 2014-2020
 # - Wen Guan <wguan.icedew@gmail.com>, 2014-2016
 # - Tomas Kouba <tomas.kouba@cern.ch>, 2014
 # - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2016
@@ -157,7 +157,7 @@ def submitter(once=False, rses=None, mock=False,
 
                 logging.info('%s Starting to get transfer transfers for %s', prepend_str, activity)
                 start_time = time.time()
-                transfers = __get_transfers(total_workers=heart_beat['nr_threads'] - 1,
+                transfers = __get_transfers(total_workers=heart_beat['nr_threads'],
                                             worker_number=heart_beat['assign_thread'],
                                             failover_schemes=failover_scheme,
                                             limit=bulk,


Test patch:
Repository: rucio/rucio

Issue:
Fix extract_scope method for BelleII
Motivation
----------
The method doesn't work properly for users





Discussion:

Resolving patch:
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -768,12 +768,12 @@ def extract_scope_belleii(did, scopes):
     if did.find('/belle/Raw/') > -1:
         return 'raw', did
     if did.find('/belle/user/') > -1:
-        if len(split_did) > 5:
-            if len(split_did[4]) == 1 and 'user.%s' % (split_did[5]) in scopes:
-                return 'user.%s' % split_did[5], did
         if len(split_did) > 4:
-            if 'user.%s' % (split_did[4]) in scopes:
+            if len(split_did[3]) == 1 and 'user.%s' % (split_did[4]) in scopes:
                 return 'user.%s' % split_did[4], did
+        if len(split_did) > 3:
+            if 'user.%s' % (split_did[3]) in scopes:
+                return 'user.%s' % split_did[3], did
         return 'user', did
     if did.find('/belle/data/') > -1:
         if len(split_did) > 5:


Test patch:
Repository: rucio/rucio

Issue:
Documentation: Added 1.22.8.post1 release notes
Documentation: Added 1.22.8.post1 release notes


Discussion:

Resolving patch:
diff --git a/lib/rucio/web/rest/webpy/v1/credential.py b/lib/rucio/web/rest/webpy/v1/credential.py
--- a/lib/rucio/web/rest/webpy/v1/credential.py
+++ b/lib/rucio/web/rest/webpy/v1/credential.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-# Copyright 2012-2018 CERN for the benefit of the ATLAS collaboration.
+# Copyright 2012-2020 CERN for the benefit of the ATLAS collaboration.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,7 +14,7 @@
 # limitations under the License.
 #
 # Authors:
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2012-2018
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2012-2020
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018-2019
 #
 # PY3K COMPATIBLE
@@ -25,13 +25,12 @@
     from urlparse import parse_qs
 except ImportError:
     from urllib.parse import parse_qs
-from web import application, ctx, OK, header, InternalError
+from web import application, ctx, OK, header, InternalError, loadhook, unloadhook
 
-from rucio.api.authentication import validate_auth_token
 from rucio.api.credential import get_signed_url
 from rucio.common.exception import RucioException
 from rucio.common.utils import generate_http_error
-from rucio.web.rest.common import RucioController, check_accept_header_wrapper
+from rucio.web.rest.common import RucioController, check_accept_header_wrapper, rucio_loadhook, rucio_unloadhook
 
 URLS = (
     '/signurl?$', 'SignURL',
@@ -84,14 +83,6 @@ def GET(self):
         if ip is None:
             ip = ctx.ip
 
-        try:
-            validate_auth_token(ctx.env.get('HTTP_X_RUCIO_AUTH_TOKEN'))
-        except RucioException as e:
-            raise generate_http_error(500, e.__class__.__name__, e.args[0][0])
-        except Exception as e:
-            print(format_exc())
-            raise InternalError(e)
-
         rse, svc, operation, url = None, None, None, None
         try:
             params = parse_qs(ctx.query[1:])
@@ -134,4 +125,6 @@ def GET(self):
 ----------------------"""
 
 APP = application(URLS, globals())
+APP.add_processor(loadhook(rucio_loadhook))
+APP.add_processor(unloadhook(rucio_unloadhook))
 application = APP.wsgifunc()


Test patch:
Repository: rucio/rucio

Issue:
Introduce caching of a MultiHop path to optimize algorithm runtime
Motivation
----------
Currently the graph of a multihop resolve is cached, but not the result of a specific path from A to B. However, these calculations do not need to be repeated over and over again and can be cached.




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -488,6 +488,11 @@ def get_hops(source_rse_id, dest_rse_id, include_multihop=False, multihop_rses=N
     :raises:                   NoDistance
     """
 
+    # Check if there is a cached result
+    result = REGION_SHORT.get('get_hops_%s_%s' % (str(source_rse_id), str(dest_rse_id)))
+    if not isinstance(result, NoValue):
+        return result
+
     if multihop_rses is None:
         multihop_rses = []
 
@@ -506,12 +511,14 @@ def get_hops(source_rse_id, dest_rse_id, include_multihop=False, multihop_rses=N
                                                           operation_src='third_party_copy',
                                                           operation_dest='third_party_copy',
                                                           domain='wan')
-            return [{'source_rse_id': source_rse_id,
+            path = [{'source_rse_id': source_rse_id,
                      'dest_rse_id': dest_rse_id,
                      'source_scheme': matching_scheme[1],
                      'dest_scheme': matching_scheme[0],
                      'source_scheme_priority': matching_scheme[3],
                      'dest_scheme_priority': matching_scheme[2]}]
+            REGION_SHORT.set('get_hops_%s_%s' % (str(source_rse_id), str(dest_rse_id)), path)
+            return path
         except RSEProtocolNotSupported as error:
             if include_multihop:
                 # Delete the edge from the graph
@@ -577,6 +584,7 @@ def get_hops(source_rse_id, dest_rse_id, include_multihop=False, multihop_rses=N
                     except RSEProtocolNotSupported:
                         pass
     if dest_rse_id in visited_nodes:
+        REGION_SHORT.set('get_hops_%s_%s' % (str(source_rse_id), str(dest_rse_id)), visited_nodes[dest_rse_id]['path'])
         return visited_nodes[dest_rse_id]['path']
     else:
         raise NoDistance()


Test patch:
Repository: rucio/rucio

Issue:
BelleII : Change in the extract_scope algorithm
Motivation
----------
Change of the convention




Discussion:

Resolving patch:
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -761,14 +761,14 @@ def extract_scope_atlas(did, scopes):
 
 def extract_scope_belleii(did, scopes):
     split_did = did.split('/')
-    if did.find('/belle/MC/') > -1:
-        if len(split_did) > 5:
-            if split_did[4] in ['fab', 'merge1', 'skim']:
+    if did.startswith('/belle/MC/'):
+        if len(split_did) > 4:
+            if split_did[3] in ['fab', 'merge1', 'skim']:
                 return 'mc_tmp', did
         return 'mc', did
-    if did.find('/belle/Raw/') > -1:
+    if did.startswith('/belle/Raw/'):
         return 'raw', did
-    if did.find('/belle/user/') > -1:
+    if did.startswith('/belle/user/'):
         if len(split_did) > 4:
             if len(split_did[3]) == 1 and 'user.%s' % (split_did[4]) in scopes:
                 return 'user.%s' % split_did[4], did
@@ -776,12 +776,12 @@ def extract_scope_belleii(did, scopes):
             if 'user.%s' % (split_did[3]) in scopes:
                 return 'user.%s' % split_did[3], did
         return 'user', did
-    if did.find('/belle/data/') > -1:
-        if len(split_did) > 5:
-            if split_did[4] in ['fab', 'skim']:
+    if did.startswith('/belle/data/') or did.startswith('/belle/Data/'):
+        if len(split_did) > 4:
+            if split_did[3] in ['fab', 'skim']:
                 return 'data_tmp', did
         return 'data', did
-    if did.find('/belle/ddm/functional_tests/') > -1:
+    if did.startswith('/belle/ddm/functional_tests/') or did.startswith('/belle/ddm/tests/'):
         return 'test', did
     return 'other', did
 


Test patch:
Repository: rucio/rucio

Issue:
Return files size when listing scope DIDs
Motivation
----------
The API response when listing the DIDs belonging solely to a scope or to a container is very dissimilar.
While in the first case the bytes size is not shown:

```
{"scope": "O3", "type": "FILE", "name": "K-K1_llhoft-1266855936-4096.gwf", "parent": null, "level": 0}
```

in the second it is:

```
{"adler32": "3011a75c", "name": "K-K1_C10-1268027392-4096.gwf", "bytes": 34248761, "scope": "O3", "type": "FILE", "md5": "3a46a528ff1d618f5791e3f0eceb42f8"}
```

Having a similar pattern (or at least the bytes size) would greatly improve the operations of the FUSE-Posix module, since it would enable bulk file size retrieval even at the scope level (and not only within a dataset or container).

Modification
------------
It should suffice to edit the API implementation around https://github.com/rucio/rucio/blob/master/lib/rucio/core/did.py#L1098 in order to introduce the retrieval of other fields, as done here https://github.com/rucio/rucio/blob/master/lib/rucio/core/did.py#L1141 .




Discussion:
I'll handle this one.

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -30,6 +30,7 @@
 # - Luc Goossens <luc.goossens@cern.ch>, 2020
 # - Aristeidis Fkiaras <aristeidis.fkiaras@cern.ch>, 2019 - 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
+# - Vivek Nigam <viveknigam.nigam3@gmail.com>, 2020
 #
 # PY3K COMPATIBLE
 
@@ -1108,10 +1109,13 @@ def scope_list(scope, name=None, recursive=False, session=None):
 
     def __topdids(scope):
         c = session.query(models.DataIdentifierAssociation.child_name).filter_by(scope=scope, child_scope=scope)
-        q = session.query(models.DataIdentifier.name, models.DataIdentifier.did_type).filter_by(scope=scope)  # add type
+        q = session.query(models.DataIdentifier.name, models.DataIdentifier.did_type, models.DataIdentifier.bytes).filter_by(scope=scope)  # add type
         s = q.filter(not_(models.DataIdentifier.name.in_(c))).order_by(models.DataIdentifier.name)
         for row in s.yield_per(5):
-            yield {'scope': scope, 'name': row.name, 'type': row.did_type, 'parent': None, 'level': 0}
+            if row.did_type == DIDType.FILE:
+                yield {'scope': scope, 'name': row.name, 'type': row.did_type, 'parent': None, 'level': 0, 'bytes': row.bytes}
+            else:
+                yield {'scope': scope, 'name': row.name, 'type': row.did_type, 'parent': None, 'level': 0, 'bytes': None}
 
     def __diddriller(pdid):
         query_associ = session.query(models.DataIdentifierAssociation).filter_by(scope=pdid['scope'], name=pdid['name'])


Test patch:
Repository: rucio/rucio

Issue:
Conveyor submitter daemons are doing nothing if there are many requests with no sources. (Can reproduce)
For quite some time now in the ESCAPE Rucio instance there are no jobs submitted to fts unless I significantly raise the number of conveyor submitter threads every now and then.

After some digging up I found out that the conveyor submitter daemons are fed with jobs that have no sources. Then they go through them and of course do not submit them. 

The issue is that they do not mark them as unavailable or something and they stay in the requests queue. On the next loop they are picked up again and this continues as an endless cycle of doing nothing.

They continue of the loop happens here https://github.com/rucio/rucio/blob/3c840247c1c3decd37ade5b5d9b0c5adb4e0a960/lib/rucio/core/transfer.py#L656

### How to replicate
Instructions
1. Create 1001 files stuck in replicating state (by a failed upload)
2. Create replication rules for those 1001 files.
3. Do a successful upload
4. Add a replication rule for the file
5. Run the conveyor submitter as many times as you want, the valid transfer will never be picked up.

CLI Instructions
```bash
$ git clone git@github.com:rucio/rucio.git
$ cd rucio
$ docker-compose --file etc/docker/dev/docker-compose-storage-localhost.yml up -d
$ docker exec -it dev_rucio_1 /bin/bash
$ ./tools/run_tests_docker.sh -i
$ ./tools/docker_activate_rses.sh
$ rucio-admin rse add FAIL
$ rucio-admin rse add-protocol --hostname fail.com --scheme root --prefix /fail FAIL
$ seq 1010 | xargs -n 1 -I {} -P 10 touch failed_upload{}
$ seq 1010 | xargs -n 1 -I {} -P 10 rucio upload --scope test --rse FAIL failed_upload{}
$ seq 1010 | xargs -n 1 -I {} -P 10 rucio add-rule test:failed_upload{} 1 XRD1
$ rucio-conveyor-submitter --run-once
# Nothing should have happened here since the uploads to MOCK fail by default
$ touch working_upload
$ rucio upload working_upload --scope test --rse XRD1
# This upload will succeed
$ rucio add-rule test:working_upload 1 XRD2
$ rucio-conveyor-submitter --run-once
# Request about test:working_upload will not be picked even though it is valid
```




Discussion:
I have to track this a bit why that was changed like this. We do have the state `NO_SOURCES` which is the one these requests should be set to, but I can't say why this is not used here anymore. Needs investigation.

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -662,6 +662,10 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
 
         multihop = False
 
+        # Add req to req_no_source list (Will be removed later if needed)
+        if req_id not in reqs_no_source:
+            reqs_no_source.append(req_id)
+
         # source_rse_id will be None if no source replicas
         # rse will be None if rse is staging area
         if source_rse_id is None or rse is None:


Test patch:
Repository: rucio/rucio

Issue:
stateReason not filled in the traces after timeout
Motivation
----------
stateReason is not correctly populated to the trace, which causes wrong exception at the end of failed job.


Modification
------------
just place some protection here:
https://github.com/rucio/rucio/blob/master/lib/rucio/client/downloadclient.py#L574




Discussion:
it seems, that gfal is returning an exception which has no str interpretation.

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -453,6 +453,8 @@ def _download_item(self, item, trace, traces_copy_out, log_prefix=''):
         trace.setdefault('datasetScope', item.get('dataset_scope', ''))
         trace.setdefault('dataset', item.get('dataset_name', ''))
         trace.setdefault('filesize', item.get('bytes'))
+        trace.setdefault('clientState', 'PROCESSING')
+        trace.setdefault('stateReason', 'UNKNOWN')
 
         dest_file_paths = item['dest_file_paths']
 


Test patch:
Repository: rucio/rucio

Issue:
Off by one error in core/transfer.py
Motivation
----------

Reported by Patrick Austin

It looks like this commit https://github.com/rucio/rucio/commit/b519409488ad2af867c866e4e72cd96ecdbf86bc
introduced a bug in core/transfer.py where another field was selected out of the database, but the destination RSE in line 1254 was not updated to reflect this.


Modification
------------

Change 9 to 10 in line 1254



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -27,6 +27,7 @@
 # - Gabriele <sucre.91@hotmail.it>, 2019
 # - Jaroslav Guenther <jaroslav.guenther@cern.ch>, 2019-2020
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
+# - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 #
 # PY3K COMPATIBLE
 
@@ -1355,7 +1356,7 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
     if rses:
         result = []
         for item in query.all():
-            dest_rse_id = item[9]
+            dest_rse_id = item[10]
             if dest_rse_id in rses:
                 result.append(item)
         return result


Test patch:
Repository: rucio/rucio

Issue:
Use X509_CERT_DIR if defined
Motivation
----------
Some sites have not the CA cert directory defined at the standard place. The client should try to use first the `X509_CERT_DIR` variable if defined, then the `ca_cert` defined in `rucio.cfg`






Discussion:

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -17,7 +17,7 @@
 # - Vincent Garonne <vincent.garonne@cern.ch>, 2012-2018
 # - Yun-Pin Sun <winter0128@gmail.com>, 2013
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2020
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2014-2015
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2014-2020
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2015
 # - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2015-2018
 # - Martin Barisits <martin.barisits@cern.ch>, 2016-2019
@@ -44,6 +44,7 @@
 from __future__ import print_function
 
 import imp
+import os
 import random
 import sys
 import traceback
@@ -249,12 +250,15 @@ def __init__(self, rucio_host=None, auth_host=None, account=None, ca_cert=None,
             raise ClientProtocolNotSupported('\'%s\' not supported' % auth_scheme)
 
         if (rucio_scheme == 'https' or auth_scheme == 'https') and ca_cert is None:
-            LOG.debug('HTTPS is required, but no ca_cert was passed. Trying to get it from the config file.')
-            try:
-                self.ca_cert = path.expandvars(config_get('client', 'ca_cert'))
-            except (NoOptionError, NoSectionError):
-                LOG.debug('No ca_cert found in configuration. Falling back to Mozilla default CA bundle (certifi).')
-                self.ca_cert = True
+            LOG.debug('HTTPS is required, but no ca_cert was passed. Trying to get it from X509_CERT_DIR.')
+            self.ca_cert = os.environ.get('X509_CERT_DIR', None)
+            if self.ca_cert is None:
+                LOG.debug('HTTPS is required, but no ca_cert was passed and X509_CERT_DIR is not defined. Trying to get it from the config file.')
+                try:
+                    self.ca_cert = path.expandvars(config_get('client', 'ca_cert'))
+                except (NoOptionError, NoSectionError):
+                    LOG.debug('No ca_cert found in configuration. Falling back to Mozilla default CA bundle (certifi).')
+                    self.ca_cert = True
 
         self.list_hosts = [self.host]
 


Test patch:
Repository: rucio/rucio

Issue:
Duplicate prometheus metrics when testing/importing hermes, hermes2
Motivation
----------
Importing hermes and hermes2 at the same time now fails in testing, because the prometheus metrics are named the same and are initialized on module import.

The error is:
```
ValueError: Duplicated timeseries in CollectorRegistry: set(['rucio_daemons_hermes_reconnect_total', 'rucio_daemons_hermes_reconnect_created'])
```

which is caused on this line:
https://github.com/rucio/rucio/blob/a4ea2cf45961641e9f266f07fefc0a8e8be96498/lib/rucio/daemons/hermes/hermes2.py#L73

See the checks on PR https://github.com/rucio/rucio/pull/3947


Modification
------------
Move initialization to the `run` Method


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/hermes/hermes2.py b/lib/rucio/daemons/hermes/hermes2.py
--- a/lib/rucio/daemons/hermes/hermes2.py
+++ b/lib/rucio/daemons/hermes/hermes2.py
@@ -70,7 +70,7 @@
 
 GRACEFUL_STOP = threading.Event()
 
-RECONNECT_COUNTER = Counter('rucio_daemons_hermes_reconnect', 'Counts Hermes reconnects to different ActiveMQ brokers', labelnames=('host',))
+RECONNECT_COUNTER = Counter('rucio_daemons_hermes2_reconnect', 'Counts Hermes2 reconnects to different ActiveMQ brokers', labelnames=('host',))
 
 
 def default(datetype):


Test patch: