internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
SRM to HTTPS optional compatibility
Motivation
----------
https://github.com/rucio/rucio/blob/882b662a24c75d515080606be635959566b17f14/lib/rucio/common/constants.py#L42
Defines a scheme_map between protocol compatibility in Rucio. We need an additional compatibility between `srm` and `https` (and vice-versa) which is only optionally (based on a config parameter) activated.

Modification
------------
Check if an option
```
[transfers]
srm_https_compatibility = True
```
is set and add it to the scheme_map.


Discussion:
```diff
-srm_http_compatibility = True
+srm_https_compatibility = True
```
?

Resolving patch:
diff --git a/lib/rucio/common/constants.py b/lib/rucio/common/constants.py
--- a/lib/rucio/common/constants.py
+++ b/lib/rucio/common/constants.py
@@ -21,11 +21,15 @@
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018
 # - Sahan Dilshan <32576163+sahandilshan@users.noreply.github.com>, 2021
 # - David Población Criado <david.poblacion.criado@cern.ch>, 2021
+# - Radu Carpa <radu.carpa@cern.ch>, 2021
 # - Rakshita Varadarajan <rakshitajps@gmail.com>, 2021
+# - Joel Dierkes <joel.dierkes@cern.ch>, 2021
 
 from collections import namedtuple
 from enum import Enum
 
+from rucio.common.config import config_get
+
 """
 Constants.
 
@@ -50,6 +54,9 @@
               'scp': ['scp'],
               'rsync': ['rsync'],
               'rclone': ['rclone']}
+if config_get('transfers', 'srm_https_compatibility', raise_exception=False, default=False):
+    SCHEME_MAP['srm'].append('https')
+    SCHEME_MAP['https'].append('srm')
 
 SUPPORTED_PROTOCOLS = ['gsiftp', 'srm', 'root', 'davs', 'http', 'https', 'file', 's3', 's3+rucio', 's3+https', 'storm', 'srm+https', 'scp', 'rsync', 'rclone']
 


Test patch:
Repository: rucio/rucio

Issue:
bring_online incorrectly set in multihops with tape source
Motivation
----------
The job parameters in fts are constructed from last hop (which is the initial transfer that triggered the multihop). This behavior was changed in https://github.com/rucio/rucio/pull/4837 . Until then, the job parameters where built from first hop because it was the first one inserted into the "transfers" dictionary and so the first one analyzed.

This results in multihop jobs being submitted for transfers from tape without bring_online set





Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -342,8 +342,13 @@ def bulk_group_transfers_for_fts(transfers, policy='rule', group_bulk=200, sourc
             # for multihop transfers, all the path is submitted as a separate job
             job_params = _build_job_params(transfer_path[-1])
             for transfer in transfer_path[:-1]:
+                hop_params = _build_job_params(transfer)
                 # Only allow overwrite if all transfers in multihop allow it
-                job_params['overwrite'] = _build_job_params(transfer)['overwrite'] and job_params['overwrite']
+                job_params['overwrite'] = hop_params['overwrite'] and job_params['overwrite']
+                # Activate bring_online if it was requested by first hop (it is a multihop starting at a tape)
+                # We don't allow multihop via a tape, so bring_online should not be set on any other hop
+                if transfer is transfer_path[0] and hop_params['bring_online']:
+                    job_params['bring_online'] = hop_params['bring_online']
 
             group_key = 'multihop_%s' % transfer_path[-1].rws.request_id
             grouped_transfers[group_key] = {'transfers': transfer_path, 'job_params': job_params}


Test patch:
Repository: rucio/rucio

Issue:
Error message 'NoSuchBucket' not matched via stringification
The following test uses `str(e)` to find out whether the string contains 'NoSuchBucket', however (at least in boto 2.49.0) that message is contained in `e.message` and the default error stringification (inherited form BotoServerError):
https://github.com/boto/boto/blob/91ba037e54ef521c379263b0ac769c66182527d7/boto/exception.py#L154-L156
does not include that instance attribute. 

https://github.com/rucio/rucio/blob/59af37a14e343f938131863969d130f5336e4a28/lib/rucio/rse/protocols/s3boto.py#L247


Discussion:
According with https://github.com/boto/boto/blob/develop/boto/s3/connection.py#L557-L567 and https://github.com/boto/boto/blob/develop/tests/unit/s3/test_connection.py#L393 it seems that the 'NoSuchBucket' string is stored in `e.error_code` instead of in `e.message`

Resolving patch:
diff --git a/lib/rucio/rse/protocols/s3boto.py b/lib/rucio/rse/protocols/s3boto.py
--- a/lib/rucio/rse/protocols/s3boto.py
+++ b/lib/rucio/rse/protocols/s3boto.py
@@ -1,17 +1,26 @@
-# Copyright European Organization for Nuclear Research (CERN)
+# -*- coding: utf-8 -*-
+# Copyright 2014-2021 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
-# You may not use this file except in compliance with the License.
+# you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
-# http://www.apache.org/licenses/LICENSE-2.0
 #
-# Authors:
-# - Wen Guan, <wen.guan@cern.ch>, 2014-2017
-# - Vincent Garonne, <vincent.garonne@cern.ch>, 2016-2017
-# - Nicolo Magini, <nicolo.magini@cern.ch>, 2018
-# - Hannes Hansen, <hannes.jakob.hansen@cern.ch>, 2019
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 #
-# PY3K COMPATIBLE
+# Authors:
+# - Wen Guan <wen.guan@cern.ch>, 2014-2017
+# - Vincent Garonne <vincent.garonne@cern.ch>, 2016-2017
+# - Martin Barisits <martin.barisits@cern.ch>, 2016-2017
+# - Nicolo Magini <nicolo.magini@cern.ch>, 2018
+# - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
+# - Tomas Javurek <tomas.javurek@cern.ch>, 2020
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2021
 
 import os
 try:
@@ -244,7 +253,7 @@ def put(self, source, target, source_dir=None, transfer_timeout=None):
                 raise exception.DestinationNotAccessible('Cannot get the destionation key from S3')
             key.set_contents_from_filename(full_name)
         except boto.exception.S3ResponseError as e:
-            if e.status == 404 and 'NoSuchBucket' in str(e):
+            if e.status == 404 and 'NoSuchBucket' in e.error_code:
                 try:
                     bucket, key = self.get_bucket_key(target, create=True)
                     key.set_contents_from_filename(full_name)


Test patch:
Repository: rucio/rucio

Issue:
Overly restrictive proxy permission check causes compatibility problems
Motivation
----------
This issue from ~6 months ago seems to have resulted in a change to the required permissions on the proxy file for X.509 auth in the client code:
https://github.com/rucio/rucio/issues/4071

If I run voms-proxy-init or dirac-proxy-init, these tools write a proxy with 0600 permissions by default, resulting in this error message from the rucio client:
`Details: X.509 authentication selected, but private key (/tmp/x509up_u12345) permissions are liberal (required: 400, found: 600)`

If I change the permissions to 0400, then rucio client works, but the other tools break:

```
$ voms-proxy-init 
Enter GRID pass phrase for this identity:
Error creating proxy certificate: /tmp/x509up_u12345 (Permission denied)
$ dirac-proxy-init 
Generating proxy...
Enter Certificate password:
Couldn't generate proxy: Cannot write to file ( 1013 : /tmp/x509up_u12345 :IOError(13, 'Permission denied'))
```

This is also likely to cause us problems within jobs on a WN as the proxy permissions there are also normally 0600 by default (and changing it to 0400 in the job may cause renewal issues).

Modification
------------

Could this check please be reverted back to just checking that the group + other permissions bits are empty? I don't think there are any security issues with the user being able to write to their own proxy.




Discussion:

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -222,8 +222,8 @@ def __init__(self, rucio_host=None, auth_host=None, account=None, ca_cert=None,
                         raise MissingClientParameter('X.509 client key not found: %s' % self.creds['client_key'])
                     else:
                         perms = oct(os.stat(self.creds['client_key']).st_mode)[-3:]
-                        if perms != '400':
-                            raise CannotAuthenticate('X.509 authentication selected, but private key (%s) permissions are liberal (required: 400, found: %s)' % (self.creds['client_key'], perms))
+                        if perms not in ['400', '600']:
+                            raise CannotAuthenticate('X.509 authentication selected, but private key (%s) permissions are liberal (required: 400 or 600, found: %s)' % (self.creds['client_key'], perms))
                 elif self.auth_type == 'x509_proxy':
                     try:
                         self.creds['client_proxy'] = path.abspath(path.expanduser(path.expandvars(config_get('client', 'client_x509_proxy'))))


Test patch:
Repository: rucio/rucio

Issue:
Deletion timeout does not work on GridFTP
Motivation
----------

The newly-introduced deletion timeout does not appear to work on GridFTP.

https://github.com/rucio/rucio/blob/6a853eb36b25cbbd2b49159def9d6db93dd16422/lib/rucio/rse/protocols/gfal.py#L241

Modification
------------

Replace `GSIFTP PLUGIN` with `GRIDFTP PLUGIN`.


Discussion:

Resolving patch:
diff --git a/lib/rucio/rse/protocols/gfal.py b/lib/rucio/rse/protocols/gfal.py
--- a/lib/rucio/rse/protocols/gfal.py
+++ b/lib/rucio/rse/protocols/gfal.py
@@ -238,7 +238,7 @@ def connect(self):
                 timeout = int(TIMEOUT)
                 self.__ctx.set_opt_integer("HTTP PLUGIN", "OPERATION_TIMEOUT", timeout)
                 self.__ctx.set_opt_integer("SRM PLUGIN", "OPERATION_TIMEOUT", timeout)
-                self.__ctx.set_opt_integer("GSIFTP PLUGIN", "OPERATION_TIMEOUT", timeout)
+                self.__ctx.set_opt_integer("GRIDFTP PLUGIN", "OPERATION_TIMEOUT", timeout)
             except ValueError:
                 self.logger(logging.ERROR, 'wrong timeout value %s', TIMEOUT)
 


Test patch:
Repository: rucio/rucio

Issue:
increase verbosity of logger when running pytests
Motivation
----------
Failed tests are difficult to debug, because by default only the WARNING (and up) level is captured by patest. 


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -315,7 +315,7 @@ def add_rule(dids, account, copies, rse_expression, grouping, weight, lifetime,
                 logger(logging.DEBUG, "Created rule %s for injection", str(new_rule.id))
                 if delay_injection:
                     new_rule.created_at = datetime.utcnow() + timedelta(seconds=delay_injection)
-                    logger(logging.DEBUG, "Scheduled rule %s for injection on %s", (str(new_rule.id), new_rule.created_at))
+                    logger(logging.DEBUG, "Scheduled rule %s for injection on %s", str(new_rule.id), new_rule.created_at)
                 continue
 
             # If Split Container is chosen, the rule will be processed ASYNC


Test patch:
Repository: rucio/rucio

Issue:
Incorrect variable type passed to request.core.get_next counter
Motivation
----------

https://github.com/rucio/rucio/blob/63823eceebe68294c98364884a33172d14591da9/lib/rucio/core/request.py#L313

results in this error on the statsd side: 
```
19 Nov 16:32:19 - DEBUG: Bad line:  'T'>, <RequestType.STAGEIN in msg "rucio_int_okd.core.request.get_next.[<RequestType.TRANSFER: 'T'>, <RequestType.STAGEIN: 'I'>, <RequestType.STAGEOUT: 'O'>].[<RequestState.DONE: 'D'>, <RequestState.FAILED: 'F'>, <RequestState.LOST: 'L'>, <RequestState.SUBMITTING: 'G'>, <RequestState.SUBMISSION_FAILED: 'A'>, <RequestState.NO_SOURCES: 'N'>, <RequestState.ONLY_TAPE_SOURCES: 'O'>, <RequestState.MISMATCH_SCHEME: 'M'>]:1|c"
```

The issue is probably there since the "enum" refactoring. 

Modification
------------

Request_type must be explicitly converted to a string when passing to record_counter.



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/request.py b/lib/rucio/core/request.py
--- a/lib/rucio/core/request.py
+++ b/lib/rucio/core/request.py
@@ -309,8 +309,10 @@ def get_next(request_type, state, limit=100, older_than=None, rse_id=None, activ
     :param session:           Database session to use.
     :returns:                 Request as a dictionary.
     """
-
-    record_counter('core.request.get_next.{request_type}.{state}', labels={'request_type': request_type, 'state': state})
+    request_type_metric_label = '.'.join(a.name for a in request_type) if isinstance(request_type, list) else request_type.name
+    state_metric_label = '.'.join(s.name for s in state) if isinstance(state, list) else state.name
+    record_counter('core.request.get_next.{request_type}.{state}', labels={'request_type': request_type_metric_label,
+                                                                           'state': state_metric_label})
 
     # lists of one element are not allowed by SQLA, so just duplicate the item
     if type(request_type) is not list:


Test patch:
diff --git a/lib/rucio/tests/test_conveyor.py b/lib/rucio/tests/test_conveyor.py
--- a/lib/rucio/tests/test_conveyor.py
+++ b/lib/rucio/tests/test_conveyor.py
@@ -384,6 +384,12 @@ def __source_exists(src_rse_id, scope, name, session=None):
     # Only one request was handled; doesn't matter that it's multisource
     assert metrics_mock.get_sample_value('rucio_daemons_conveyor_finisher_handle_requests_total') >= 1
     assert metrics_mock.get_sample_value('rucio_daemons_conveyor_poller_update_request_state_total', labels={'updated': 'True'}) >= 1
+    assert metrics_mock.get_sample_value(
+        'rucio_core_request_get_next_total',
+        labels={
+            'request_type': 'TRANSFER.STAGEIN.STAGEOUT',
+            'state': 'DONE.FAILED.LOST.SUBMITTING.SUBMISSION_FAILED.NO_SOURCES.ONLY_TAPE_SOURCES.MISMATCH_SCHEME'}
+    )
 
 
 @skip_rse_tests_with_accounts
Repository: rucio/rucio

Issue:
PolicyPackageNotFound: Module rucio.common.permission.generic_multi_vo not found
Hi,

While setting up an instance with multi_vo mode enabled using the default permissions package, I got the following error on the client (just from running rucio whoami):
```
2021-04-26 10:40:51,035 ERROR   Cannot authenticate.
Details: PolicyPackageNotFound: Module rucio.common.permission.generic_multi_vo not found 
```

I think this may just be a simple typo in the fallback package name here:
https://github.com/rucio/rucio/blob/b7102af49b39ed79e3fafa43d82e91620d721be6/lib/rucio/core/permission/__init__.py#L88

Is this meant to be "rucio.core.permission." instead and could it please be fixed?

Regards,
Simon



Discussion:
Yes that should be fixed. Thanks for reporting.

Resolving patch:
diff --git a/lib/rucio/core/permission/__init__.py b/lib/rucio/core/permission/__init__.py
--- a/lib/rucio/core/permission/__init__.py
+++ b/lib/rucio/core/permission/__init__.py
@@ -85,7 +85,7 @@ def load_permission_for_vo(vo):
                 POLICY = GENERIC_FALLBACK
             POLICY = 'rucio.core.permission.' + POLICY.lower()
     else:
-        POLICY = 'rucio.common.permission.' + GENERIC_FALLBACK.lower()
+        POLICY = 'rucio.core.permission.' + GENERIC_FALLBACK.lower()
 
     try:
         module = importlib.import_module(POLICY)


Test patch:
Repository: rucio/rucio

Issue:
Rucio breaks DIRAC run
Motivation
----------
After the 1.27.0rc1 release, the DIRAC runs are throwing an error produced by Rucio. The cause is a missing `rucio.cfg` file. The `is_client` function thus throws the following error:
```
2021-11-25T05:03:46.4056355Z �[1m�[31mE               RuntimeError: Could not load Rucio configuration file. Rucio looked in the following paths for a configuration file, in order:�[0m
2021-11-25T05:03:46.4057329Z �[1m�[31mE               	/opt/rucio/etc/rucio.cfg�[0m
```

https://github.com/rucio/rucio/blob/b820877f58c975740d9ea1fd448c3dcf2eb013cf/lib/rucio/common/utils.py#L1560-L1579


Issue: DIRACGrid/DIRAC#5637

Modification
------------
@bari12 suggested that the `is_client` function should return `True` in case of a missing configuration file.


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -1564,12 +1564,16 @@ def is_client():
     :returns client_mode: True if is called from a client, False if it is called from a server/daemon
     """
     if 'RUCIO_CLIENT_MODE' not in os.environ:
-        if config_has_section('database'):
-            client_mode = False
-        elif config_has_section('client'):
+        try:
+            if config_has_section('database'):
+                client_mode = False
+            elif config_has_section('client'):
+                client_mode = True
+            else:
+                client_mode = False
+        except RuntimeError:
+            # If no configuration file is found the default value should be True
             client_mode = True
-        else:
-            client_mode = False
     else:
         if os.environ['RUCIO_CLIENT_MODE']:
             client_mode = True


Test patch:
Repository: rucio/rucio

Issue:
regression in conveyor database partition variable name
Motivation
----------
The database table is "requests". By error, we try to access "request" (singular). 





Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -1565,7 +1565,7 @@ def __list_transfer_requests_and_source_replicas(
     """
 
     if partition_hash_var is None:
-        partition_hash_var = 'request.id'
+        partition_hash_var = 'requests.id'
 
     if request_state is None:
         request_state = RequestState.QUEUED


Test patch:
Repository: rucio/rucio

Issue:
unexpected keyword argument 'id' in receiver
Motivation
----------
Receiver fails to start:
```
TypeError: receiver() got an unexpected keyword argument 'id'
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/receiver.py b/lib/rucio/daemons/conveyor/receiver.py
--- a/lib/rucio/daemons/conveyor/receiver.py
+++ b/lib/rucio/daemons/conveyor/receiver.py
@@ -259,7 +259,7 @@ def run(once=False, total_threads=1, full_mode=False):
         raise exception.DatabaseException('Database was not updated, daemon won\'t start')
 
     logging.info('starting receiver thread')
-    threads = [threading.Thread(target=receiver, kwargs={'id': i,
+    threads = [threading.Thread(target=receiver, kwargs={'id_': i,
                                                          'full_mode': full_mode,
                                                          'total_threads': total_threads}) for i in range(0, total_threads)]
 


Test patch:
Repository: rucio/rucio

Issue:
DAVS to SRM optional compatibility
Motivation
----------
Same as #4944.


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/constants.py b/lib/rucio/common/constants.py
--- a/lib/rucio/common/constants.py
+++ b/lib/rucio/common/constants.py
@@ -57,6 +57,8 @@
 if config_get('transfers', 'srm_https_compatibility', raise_exception=False, default=False):
     SCHEME_MAP['srm'].append('https')
     SCHEME_MAP['https'].append('srm')
+    SCHEME_MAP['srm'].append('davs')
+    SCHEME_MAP['davs'].append('srm')
 
 SUPPORTED_PROTOCOLS = ['gsiftp', 'srm', 'root', 'davs', 'http', 'https', 'file', 's3', 's3+rucio', 's3+https', 'storm', 'srm+https', 'scp', 'rsync', 'rclone']
 


Test patch:
Repository: rucio/rucio

Issue:
Duplicated timeseries in CollectorRegistry error in submitter
Motivation
----------

```
Could not submit transfer to https://fts.usatlas.bnl.gov:8446 - Duplicated timeseries in CollectorRegistry: {'rucio_transfertool_fts3_submit_transfer_sum', 'rucio_transfertool_fts3_submit_transfer', 'rucio_transfertool_fts3_submit_transfer_created', 'rucio_transfertool_fts3_submit_transfer_count'}
```





Discussion:

Resolving patch:
diff --git a/lib/rucio/transfertool/fts3.py b/lib/rucio/transfertool/fts3.py
--- a/lib/rucio/transfertool/fts3.py
+++ b/lib/rucio/transfertool/fts3.py
@@ -58,7 +58,6 @@
 
 from dogpile.cache import make_region
 from dogpile.cache.api import NoValue
-from prometheus_client import Summary
 
 from rucio.common.config import config_get, config_get_bool
 from rucio.common.constants import FTS_JOB_TYPE, FTS_STATE
@@ -91,7 +90,6 @@
                                   documentation='Number of bulk queries', labelnames=('state', 'host'))
 QUERY_DETAILS_COUNTER = MultiCounter(prom='rucio_transfertool_fts3_query_details', statsd='transfertool.fts3.{host}.query_details.{state}',
                                      documentation='Number of detailed status queries', labelnames=('state', 'host'))
-SUBMISSION_TIMER = Summary('rucio_transfertool_fts3_submit_transfer', 'Timer for transfer submission', labelnames=('host',))
 
 
 ALLOW_USER_OIDC_TOKENS = config_get('conveyor', 'allow_user_oidc_tokens', False, False)
@@ -546,7 +544,6 @@ def submit(self, transfers, job_params, timeout=None):
                                         timeout=timeout)
             labels = {'host': self.__extract_host(self.external_host)}
             record_timer('transfertool.fts3.submit_transfer.{host}', (time.time() - start_time) * 1000 / len(files), labels=labels)
-            SUBMISSION_TIMER.labels(**labels).observe((time.time() - start_time) * 1000 / len(files))
         except ReadTimeout as error:
             raise TransferToolTimeout(error)
         except JSONDecodeError as error:


Test patch:
Repository: rucio/rucio

Issue:
error in nongrid-trace section name on rucio-cfg generation 
Motivation
----------
nogrid-trace should be nongrid-trace at: 
https://github.com/rucio/rucio/blob/d4b4f92a0155875c8d1e1ced0fa644ff09997dff/tools/merge_rucio_configs.py#L36



Discussion:

Resolving patch:
diff --git a/tools/merge_rucio_configs.py b/tools/merge_rucio_configs.py
--- a/tools/merge_rucio_configs.py
+++ b/tools/merge_rucio_configs.py
@@ -33,7 +33,7 @@
     'messaging_cache': 'messaging-cache',
     'messaging_hermes': 'messaging-hermes',
     'messaging_hermes2': 'hermes',
-    'nongrid_trace': 'nogrid-trace',
+    'nongrid_trace': 'nongrid-trace',
     'tracer_kronos': 'tracer-kronos',
 }
 


Test patch: