internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
Multihop problems with multiple sources
Motivation
----------
This issue comes up in cases when a destination will require multhop and there are mixed tape/disk sources. Depending on the order these sources are iterated in the loop, the `bring_online` attribute is set for the transfer due to one of the sources being tape.  Finally even if the disk source is selected for the multihop, the transfer is still submitted with `bring_online`

This however uncovered a somewhat bigger issue that in case of multhops we do not properly evaluate sources. Instead the algorithm will use the last source retrieved from the database, even if this one is worse (e.g. a tape source)




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -1,4 +1,5 @@
-# Copyright 2013-2020 CERN for the benefit of the ATLAS collaboration.
+# -*- coding: utf-8 -*-
+# Copyright 2013-2020 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +16,7 @@
 # Authors:
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2020
 # - Martin Barisits <martin.barisits@cern.ch>, 2017-2020
-# - Vincent Garonne <vgaronne@gmail.com>, 2017
+# - Vincent Garonne <vincent.garonne@cern.ch>, 2017
 # - Igor Mandrichenko <rucio@fermicloud055.fnal.gov>, 2018
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2018-2020
 # - dciangot <diego.ciangottini@cern.ch>, 2018
@@ -24,9 +25,12 @@
 # - Andrew Lister <andrew.lister@stfc.ac.uk>, 2019
 # - Brandon White <bjwhite@fnal.gov>, 2019
 # - maatthias <maatthias@gmail.com>, 2019
-# - Gabriele <sucre.91@hotmail.it>, 2019
+# - Gabriele Fronze' <gfronze@cern.ch>, 2019
 # - Jaroslav Guenther <jaroslav.guenther@cern.ch>, 2019-2020
+# - Matt Snyder <msnyder@bnl.gov>, 2020
+# - Eric Vaandering <ewv@fnal.gov>, 2020
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
+# - Nick Smith <nick.smith@cern.ch>, 2020
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 #
 # PY3K COMPATIBLE
@@ -1244,7 +1248,7 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                                              'src_spacetoken': None,
                                              'dest_spacetoken': dest_spacetoken,
                                              'overwrite': transfers[req_id]['overwrite'],
-                                             'bring_online': transfers[req_id]['bring_online'],
+                                             'bring_online': bring_online,
                                              'copy_pin_lifetime': transfers[req_id]['copy_pin_lifetime'],
                                              'external_host': transfers[req_id]['external_host'],
                                              'selection_strategy': 'auto',


Test patch:
Repository: rucio/rucio

Issue:
Protection for negative ranking needed in multihop resolve algorithm
Motivation
----------
Currently the algorithm might end up in an infinite loop if negative `ranking` values are stored to the `distances` table.

Modification
------------
Protection against negative values. Assume 0 instead.




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -1448,10 +1448,13 @@ def __load_distance_edges_node(rse_id, session=None):
         for distance in session.query(models.Distance).join(models.RSE, models.RSE.id == models.Distance.dest_rse_id)\
                                .filter(models.Distance.src_rse_id == rse_id)\
                                .filter(models.RSE.deleted == false()).all():
+            if distance.ranking is None:
+                continue
+            ranking = distance.ranking if distance.ranking >= 0 else 0
             if distance.src_rse_id in distance_graph:
-                distance_graph[distance.src_rse_id][distance.dest_rse_id] = distance.ranking
+                distance_graph[distance.src_rse_id][distance.dest_rse_id] = ranking
             else:
-                distance_graph[distance.src_rse_id] = {distance.dest_rse_id: distance.ranking}
+                distance_graph[distance.src_rse_id] = {distance.dest_rse_id: ranking}
         REGION_SHORT.set('distance_graph_%s' % str(rse_id), distance_graph)
         result = distance_graph
     return result


Test patch:
Repository: rucio/rucio

Issue:
xrootd: stat broken on py3
Motivation
----------
The python3 response from the stat is of type byte, which requires a modification of split() and decode(). This is backward compatible to py2.


Discussion:

Resolving patch:
diff --git a/lib/rucio/rse/protocols/xrootd.py b/lib/rucio/rse/protocols/xrootd.py
--- a/lib/rucio/rse/protocols/xrootd.py
+++ b/lib/rucio/rse/protocols/xrootd.py
@@ -1,4 +1,5 @@
-# Copyright 2013-2020 CERN for the benefit of the ATLAS collaboration.
+# -*- coding: utf-8 -*-
+# Copyright 2013-2020 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,14 +18,12 @@
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2013
 # - Cheng-Hsi Chao <cheng-hsi.chao@cern.ch>, 2014
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2016-2020
-# - Nicolo Magini <Nicolo.Magini@cern.ch>, 2018
+# - Nicolo Magini <nicolo.magini@cern.ch>, 2018
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018
 # - Andrew Lister <andrew.lister@stfc.ac.uk>, 2019
-# - Tomas Javurek <tomasjavurek09@gmail.com>, 2020
+# - Tomas Javurek <tomas.javurek@cern.ch>, 2020
 # - Martin Barisits <martin.barisits@cern.ch>, 2020
-#  - Eli Chadwick, <eli.chadwick@stfc.ac.uk>, 2020
-#
-# PY3K COMPATIBLE
+# - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
 
 import os
 import logging
@@ -115,14 +114,16 @@ def stat(self, path):
             self.logger.info('xrootd.stat: filesize cmd: {}'.format(cmd))
             status_stat, out, err = execute(cmd)
             if status_stat == 0:
-                ret['filesize'] = out.split('\n')[2].split()[-1]
+                ret['filesize'] = out.split(b'\n')[2].split()[-1].decode()
 
             # xrdfs query checksum for getting checksum
             cmd = 'XrdSecPROTOCOL=gsi xrdfs %s:%s query checksum %s' % (self.hostname, self.port, path)
             self.logger.info('xrootd.stat: checksum cmd: {}'.format(cmd))
             status_query, out, err = execute(cmd)
             if status_query == 0:
-                chsum, value = out.strip('\n').split()
+                chsum, value = out.strip(b'\n').split()
+                chsum = chsum.decode()
+                value = value.decode()
                 ret[chsum] = value
 
         except Exception as e:


Test patch:
Repository: rucio/rucio

Issue:
No proper source evaluation for multi-hop
Motivation
----------
Multi hops transfer do not properly evaluate sources. Instead the algorithm will use the last source retrieved from the database, even if this one is worse (e.g. a tape source)

Modification
------------
Proper evaluation is needed.




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -1265,6 +1265,7 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     # We make the assumption that the hop is never made through TAPE
                     transfers[req_id]['file_metadata']['src_type'] = 'DISK'
                     transfers[req_id]['sources'] = [(source_rse_name, source_url, source_rse_id, 0, 0)]
+                    transfers[req_id]['bring_online'] = bring_online
         if req_id in reqs_no_source:
             reqs_no_source.remove(req_id)
         if req_id in reqs_only_tape_source:


Test patch:
Repository: rucio/rucio

Issue:
source replica expression ignored for multi-hop transfers
Motivation
----------
The `source-replica-expression` of a rule is ignored in the case of multi-hop transfers.




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -686,13 +686,26 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
 
         dict_attributes = get_attributes(attributes)
 
-        # Check if the source and destination are blacklisted
+        # Check if the source and destination are blocked
         if source_rse_id in unavailable_read_rse_ids:
             continue
         if dest_rse_id in unavailable_write_rse_ids:
-            logging.warning('RSE %s is blacklisted for write. Will skip the submission of new jobs', dest_rse_name)
+            logging.warning('RSE %s is blocked for write. Will skip the submission of new jobs', dest_rse_name)
             continue
 
+        # parse source expression
+        source_replica_expression = dict_attributes.get('source_replica_expression', None)
+        if source_replica_expression:
+            try:
+                parsed_rses = parse_expression(source_replica_expression, session=session)
+            except InvalidRSEExpression as error:
+                logging.error("Invalid RSE exception %s: %s", source_replica_expression, str(error))
+                continue
+            else:
+                allowed_rses = [x['id'] for x in parsed_rses]
+                if source_rse_id not in allowed_rses:
+                    continue
+
         # Call the get_hops function to create a list of RSEs used for the transfer
         # In case the source_rse and the dest_rse are connected, the list contains only the destination RSE
         # In case of non-connected, the list contains all the intermediary RSEs
@@ -741,19 +754,6 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
             if rses and dest_rse_id not in rses:
                 continue
 
-            # parse source expression
-            source_replica_expression = dict_attributes.get('source_replica_expression', None)
-            if source_replica_expression:
-                try:
-                    parsed_rses = parse_expression(source_replica_expression, session=session)
-                except InvalidRSEExpression as error:
-                    logging.error("Invalid RSE exception %s: %s", source_replica_expression, str(error))
-                    continue
-                else:
-                    allowed_rses = [x['id'] for x in parsed_rses]
-                    if source_rse_id not in allowed_rses:
-                        continue
-
             # Get the source rse information
             if source_rse_id not in rses_info:
                 rses_info[source_rse_id] = rsemgr.get_rse_info(rse=source_rse_name,


Test patch:
Repository: rucio/rucio

Issue:
Possibility to overide the subscription comment
Motivation
----------
The comment of the subscription is used for the rules created by the transmogrifier. It should be possible to override it in the subscription parameter `rule` in particular for the chained subscriptions





Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/transmogrifier/transmogrifier.py b/lib/rucio/daemons/transmogrifier/transmogrifier.py
--- a/lib/rucio/daemons/transmogrifier/transmogrifier.py
+++ b/lib/rucio/daemons/transmogrifier/transmogrifier.py
@@ -301,6 +301,8 @@ def transmogrifier(bulk=5, once=False, sleep_time=60):
                                         purge_replicas = False
                                     rse_expression = str(rule_dict['rse_expression'])
                                     comment = str(subscription['comments'])
+                                    if 'comments' in rule_dict:
+                                        comment = str(rule_dict['comments'])
                                     subscription_id = str(subscription['id'])
                                     account = subscription['account']
                                     copies = int(rule_dict['copies'])


Test patch:
Repository: rucio/rucio

Issue:
Add documentation to dirac endpoint
Motivation
----------
The dirac REST endpoint is missing documentation.


Discussion:

Resolving patch:
diff --git a/lib/rucio/web/rest/flaskapi/v1/dirac.py b/lib/rucio/web/rest/flaskapi/v1/dirac.py
--- a/lib/rucio/web/rest/flaskapi/v1/dirac.py
+++ b/lib/rucio/web/rest/flaskapi/v1/dirac.py
@@ -21,7 +21,7 @@
 
 from traceback import format_exc
 
-from flask import Blueprint, request
+from flask import Flask, Blueprint, request
 from flask.views import MethodView
 
 from rucio.api.dirac import add_files
@@ -38,17 +38,25 @@ class AddFiles(MethodView):
 
     def post(self):
         """
-        Create file replicas at a given RSE.
+        Atomic method used by the RucioFileCatalog plugin in Dirac that :
+        - Creates files and their replicas
+        - Creates the dataset containing the files and attach the files to the dataset
+        - Creates a rule on the dataset with RSE expression ANY and grouping NONE
+        - Creates all the container hierarchy containing the dataset
 
-        HTTP Success:
-            201 Created
+        ..:quickref: AddFiles; Method used by the RucioFileCatalog plugin in Dirac.
 
-        HTTP Error:
-            401 Unauthorized
-            405 Method Not Allowed
-            409 Conflict
-            500 Internal Error
-            503 Service Unavailable
+        :<json list lfns: List of lfn (dictionary {'lfn': <lfn>, 'rse': <rse>, 'bytes': <bytes>, 'adler32': <adler32>, 'guid': <guid>, 'pfn': <pfn>}.
+        :<json bool ignore_availability: A boolean to choose if unavailable sites need to be ignored.
+
+        :status 201: Created.
+        :status 400: Cannot decode json parameter list.
+        :status 401: Invalid auth token.
+        :status 404: DID not found.
+        :status 405: Unsupported Operation.
+        :status 409: Duplicate.
+        :status 500: Internal Error.
+        :status 503: Temporary error.
         """
         try:
             parameters = parse_response(request.data)
@@ -82,7 +90,7 @@ def post(self):
         return 'Created', 201
 
 
-def blueprint():
+def blueprint(no_doc=True):
     bp = Blueprint('dirac', __name__, url_prefix='/dirac')
 
     add_file_view = AddFiles.as_view('addfiles')
@@ -92,3 +100,10 @@ def blueprint():
     bp.before_request(request_auth_env)
     bp.after_request(response_headers)
     return bp
+
+
+def make_doc():
+    """ Only used for sphinx documentation """
+    doc_app = Flask(__name__)
+    doc_app.register_blueprint(blueprint(no_doc=False))
+    return doc_app


Test patch:
Repository: rucio/rucio

Issue:
Unhelpful error message for certificates with incorrect permissions
Motivation
----------
Using auth_type=x509, authentication fails when the personal certificate and private key have not the permission 400. Mines had 600. And I got the following error message.

```
$ rucio whoami
Internal error: Request for authentication token returned no result!
2020-10-13 08:58:00,389    ERROR    Cannot authenticate.
Details: x509 authentication failed for account=pchanial with identity={'client_cert': '/opt/rucio/etc/client.crt', 'client_key': '/opt/rucio/etc/client.key'}

```

Modification
------------
Certificate permissions should be checked and the user should be told what to do to fix it if they are not 400, to prevent wasting time on checking the validity of his/her certificates.




Discussion:
Thanks. But it looks like the fix is still too liberal since it only checks the group and other permissions. In the issue I've reported, the fix wouldn't have helped since the certificate permission was 600 instead of 400. Also, in the error message, I guess 'maximum' was intended instead of 'minimum'.
Interesting, my openssl did not complain for 600. I will add the check for this as well.

Maximum/Minimum: hah, classic.

Thanks!

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -222,8 +222,8 @@ def __init__(self, rucio_host=None, auth_host=None, account=None, ca_cert=None,
                         raise MissingClientParameter('X.509 client key not found: %s' % self.creds['client_key'])
                     else:
                         perms = oct(os.stat(self.creds['client_key']).st_mode)[-3:]
-                        if int(perms[-2]) > 0 or int(perms[-1]) > 0:
-                            raise CannotAuthenticate('X.509 authentication selected, but private key (%s) permissions are liberal (minimum expected: 0400, found %s)' % (self.creds['client_key'], perms))
+                        if perms != '400':
+                            raise CannotAuthenticate('X.509 authentication selected, but private key (%s) permissions are liberal (required: 400, found: %s)' % (self.creds['client_key'], perms))
                 elif self.auth_type == 'x509_proxy':
                     try:
                         self.creds['client_proxy'] = path.abspath(path.expanduser(path.expandvars(config_get('client', 'client_x509_proxy'))))


Test patch:
Repository: rucio/rucio

Issue:
Only return the per-account usage for the 'rucio' source
Motivation
----------

When running `rucio list-rse-usage --show-accounts RSE`, the per accounting information is added to all usage counters when in reality it only makes sense to have it in the `rucio` source.

Modification
------------

Do not add the `per-account` information to the other sources.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/rse.py b/lib/rucio/core/rse.py
--- a/lib/rucio/core/rse.py
+++ b/lib/rucio/core/rse.py
@@ -701,7 +701,7 @@ def get_rse_usage(rse_id, source=None, session=None, per_account=False):
                      'total': total,
                      'files': row.files,
                      'updated_at': row.updated_at}
-        if per_account:
+        if per_account and row.source == 'rucio':
             query_account_usage = session.query(models.AccountUsage).filter_by(rse_id=rse_id)
             account_usages = []
             for row in query_account_usage:


Test patch:
Repository: rucio/rucio

Issue:
Possibility to force IPv4 transfers
Motivation
----------
By default FTS will try IPv6 transfers, but some sites prefer using IPv4. Need to have the possibility to force IPv4





Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -846,6 +846,8 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     source_url = re.sub('davs', 'gclouds', source_url)
                     source_url = re.sub('https', 'gclouds', source_url)
 
+                use_ipv4 = rse_attrs[source_rse_id].get('use_ipv4', False) or rse_attrs[dest_rse_id].get('use_ipv4', False)
+
                 # IV - get external_host + strict_copy + archive timeout
                 strict_copy = rse_attrs[dest_rse_id].get('strict_copy', False)
                 fts_hosts = rse_attrs[dest_rse_id].get('fts', None)
@@ -936,6 +938,8 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     transfers[req_id]['initial_request_id'] = req_id
                 if strict_copy:
                     transfers[req_id]['strict_copy'] = strict_copy
+                if use_ipv4:
+                    transfers[req_id]['use_ipv4'] = True
                 if archive_timeout and (rses_info[dest_rse_id]['rse_type'] == RSEType.TAPE
                                         or rses_info[dest_rse_id]['rse_type'] == 'TAPE'):
                     try:
@@ -1047,6 +1051,9 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
 
                 # transfers[id]['src_urls'].append((source_rse_id, source_url))
                 transfers[req_id]['sources'].append((rse, source_url, source_rse_id, ranking, link_ranking))
+                # if one source has force IPv4, force IPv4 for the whole job
+                if rse_attrs[source_rse_id].get('use_ipv4', True):
+                    transfers[req_id]['use_ipv4'] = True
 
         except Exception:
             logging.critical("Exception happened when trying to get transfer for request %s: %s" % (req_id, traceback.format_exc()))
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -311,6 +311,7 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, source_strateg
 
         multihop = transfer.get('multihop', False)
         strict_copy = transfer.get('strict_copy', False)
+        use_ipv4 = transfer.get('use_ipv4', False)
 
         external_host = transfer['external_host']
         scope = t_file['metadata']['scope']
@@ -342,6 +343,9 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, source_strateg
             job_params['multihop'] = True
         if strict_copy:
             job_params['strict_copy'] = True
+        if use_ipv4:
+            job_params['ipv4'] = True
+            job_params['ipv6'] = False
 
         # Don't put optional & missing keys in the parameters
         if transfer['dest_spacetoken']:


Test patch:
Repository: rucio/rucio

Issue:
RSE set-attribute fails if RSE does not exist
Motivation
----------


```
# rucio-admin rse set-attribute --key whatever --value true --rse I_DO_NOT_EXIST
An unknown exception occurred.
Details: no error information passed (http status code: 500 ('internal_server_error', 'server_error', '/o\\', '\xe2\x9c\x97'))
```


Discussion:

Resolving patch:
diff --git a/lib/rucio/web/rest/flaskapi/v1/rse.py b/lib/rucio/web/rest/flaskapi/v1/rse.py
--- a/lib/rucio/web/rest/flaskapi/v1/rse.py
+++ b/lib/rucio/web/rest/flaskapi/v1/rse.py
@@ -285,6 +285,8 @@ def post(self, rse, key):
             return generate_http_error_flask(401, 'AccessDenied', error.args[0])
         except Duplicate as error:
             return generate_http_error_flask(409, 'Duplicate', error.args[0])
+        except RSENotFound as error:
+            return generate_http_error_flask(404, 'RSENotFound', error.args[0])
         except Exception as error:
             print(format_exc())
             return str(error), 500
diff --git a/lib/rucio/web/rest/webpy/v1/rse.py b/lib/rucio/web/rest/webpy/v1/rse.py
--- a/lib/rucio/web/rest/webpy/v1/rse.py
+++ b/lib/rucio/web/rest/webpy/v1/rse.py
@@ -299,6 +299,8 @@ def POST(self, rse, key):
             raise generate_http_error(401, 'AccessDenied', error.args[0])
         except Duplicate as error:
             raise generate_http_error(409, 'Duplicate', error.args[0])
+        except RSENotFound as error:
+            raise generate_http_error(404, 'RSENotFound', error.args[0])
         except Exception as error:
             raise InternalError(error)
 


Test patch: