internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
Transfer : Fix compatible transfers mapping
Motivation
----------
The table mapping compatible mapping is erroneous 

Modification
------------
Fix the mapping table


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/constants.py b/lib/rucio/common/constants.py
--- a/lib/rucio/common/constants.py
+++ b/lib/rucio/common/constants.py
@@ -6,7 +6,7 @@
 #
 # Authors:
 # - Vincent Garonne, <vincent.garonne@cern.ch>, 2013
-# - Cedric Serfon, <cedric.serfon@cern.ch>, 2015
+# - Cedric Serfon, <cedric.serfon@cern.ch>, 2015-2018
 # - Mario Lassnig, <mario.lassnig@cern.ch>, 2018
 
 """
@@ -23,8 +23,9 @@
 KEY_TYPES = ['ALL', 'COLLECTION', 'FILE', 'DERIVED']
 # all(container, dataset, file), collection(dataset or container), file, derived(compute from file for collection)
 
-SCHEME_MAP = {'srm': ['srm', 'gsiftp', 's3'],
-              'gsiftp': ['srm', 'gsiftp', 's3'],
-              'https': ['https', 'davs'],
-              'davs': ['https', 'davs'],
-              's3': ['https', 'davs', 'gsiftp', 'srm']}
+SCHEME_MAP = {'srm': ['srm', 'gsiftp'],
+              'gsiftp': ['srm', 'gsiftp'],
+              'https': ['https', 'davs', 's3'],
+              'davs': ['https', 'davs', 's3'],
+              'root': ['root'],
+              's3': ['https', 'davs', 's3']}


Test patch:
Repository: rucio/rucio

Issue:
remove unused file from common
Motivation
----------
lib/rucio/common/client.py is probably unused and can be removed

Modification
------------



Discussion:

Resolving patch:
diff --git a/lib/rucio/common/client.py b/lib/rucio/common/client.py
deleted file mode 100644
--- a/lib/rucio/common/client.py
+++ /dev/null
@@ -1,41 +0,0 @@
-# Copyright European Organization for Nuclear Research (CERN)
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# You may not use this file except in compliance with the License.
-# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
-#
-# Authors:
-# - Vincent Garonne, <vincent.garonne@cern.ch>, 2012
-
-import httplib
-
-
-class BaseClient(object):
-
-    """A base client class"""
-
-    DEFAULT_PORT = 80
-
-    OK_RESPONSE_CODES = (
-        httplib.OK,
-        httplib.CREATED,
-        httplib.ACCEPTED,
-        httplib.NO_CONTENT,
-    )
-
-    def __init__(self, host, port=None, use_ssl=False, auth_tok=None, creds=None):
-        """
-        Creates a new client to some service.
-
-        :param host: The host where service resides
-        :param port: The port where service resides
-        :param use_ssl: Should we use HTTPS?
-        :param auth_tok: The auth token to pass to the server
-        :param creds: The credentials to pass to the auth plugin
-        """
-        self.host = host
-        self.port = port or self.DEFAULT_PORT
-        self.use_ssl = use_ssl
-        self.auth_tok = auth_tok
-        self.creds = creds or {}
-        self.connection = None


Test patch:
Repository: rucio/rucio

Issue:
Regression in bulk_group_transfer due to missing key
Motivation
----------
Due to the new verify_checksum mapping.
```
2018-04-24 14:36:42,467 2991    INFO    0:0 Got 1 transfers for Functional Test XrootD
2018-04-24 14:36:42,468 2991    INFO    0:0 Starting to group transfers for Functional Test XrootD
2018-04-24 14:36:42,468 2991    CRITICAL        0:0 Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/submitter.py”, line 157, in submitter
  grouped_jobs = bulk_group_transfer(transfers, group_policy, group_bulk, fts_source_strategy, max_time_in_queue)
File “/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/common.py”, line 156, in bulk_group_transfer
  ‘verify_checksum’: transfer[‘verify_checksum’],
KeyError: ‘verify_checksum’
```


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -148,12 +148,13 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, fts_source_str
             grouped_transfers[external_host] = {}
             grouped_jobs[external_host] = []
 
+        verify_checksum = transfer.get('verify_checksum', 'both')
         file = {'sources': transfer['sources'],
                 'destinations': transfer['dest_urls'],
                 'metadata': transfer['file_metadata'],
                 'filesize': int(transfer['file_metadata']['filesize']),
                 'checksum': None,
-                'verify_checksum': transfer['verify_checksum'],
+                'verify_checksum': verify_checksum,
                 'selection_strategy': fts_source_strategy,
                 'request_type': transfer['file_metadata'].get('request_type', None),
                 'activity': str(transfer['file_metadata']['activity'])}
@@ -163,7 +164,7 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, fts_source_str
             if 'adler32' in list(file['metadata'].keys()) and file['metadata']['adler32']:
                 file['checksum'] = 'ADLER32:%s' % str(file['metadata']['adler32'])
 
-        job_params = {'verify_checksum': transfer['verify_checksum'],
+        job_params = {'verify_checksum': verify_checksum,
                       'copy_pin_lifetime': transfer['copy_pin_lifetime'] if transfer['copy_pin_lifetime'] else -1,
                       'bring_online': transfer['bring_online'] if transfer['bring_online'] else None,
                       'job_metadata': {'issuer': 'rucio'},  # finaly job_meta will like this. currently job_meta will equal file_meta to include request_id and etc.


Test patch:
Repository: rucio/rucio

Issue:
Regression in bulk_group_transfer due to missing key
Motivation
----------
Due to the new verify_checksum mapping.
```
2018-04-24 14:36:42,467 2991    INFO    0:0 Got 1 transfers for Functional Test XrootD
2018-04-24 14:36:42,468 2991    INFO    0:0 Starting to group transfers for Functional Test XrootD
2018-04-24 14:36:42,468 2991    CRITICAL        0:0 Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/submitter.py”, line 157, in submitter
  grouped_jobs = bulk_group_transfer(transfers, group_policy, group_bulk, fts_source_strategy, max_time_in_queue)
File “/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/common.py”, line 156, in bulk_group_transfer
  ‘verify_checksum’: transfer[‘verify_checksum’],
KeyError: ‘verify_checksum’
```
Regression in fts3 transfertool due to private variables
Motivation
----------
There are two private variables in the module which are not accessible to the class.


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -148,12 +148,13 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, fts_source_str
             grouped_transfers[external_host] = {}
             grouped_jobs[external_host] = []
 
+        verify_checksum = transfer.get('verify_checksum', 'both')
         file = {'sources': transfer['sources'],
                 'destinations': transfer['dest_urls'],
                 'metadata': transfer['file_metadata'],
                 'filesize': int(transfer['file_metadata']['filesize']),
                 'checksum': None,
-                'verify_checksum': transfer['verify_checksum'],
+                'verify_checksum': verify_checksum,
                 'selection_strategy': fts_source_strategy,
                 'request_type': transfer['file_metadata'].get('request_type', None),
                 'activity': str(transfer['file_metadata']['activity'])}
@@ -163,7 +164,7 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, fts_source_str
             if 'adler32' in list(file['metadata'].keys()) and file['metadata']['adler32']:
                 file['checksum'] = 'ADLER32:%s' % str(file['metadata']['adler32'])
 
-        job_params = {'verify_checksum': transfer['verify_checksum'],
+        job_params = {'verify_checksum': verify_checksum,
                       'copy_pin_lifetime': transfer['copy_pin_lifetime'] if transfer['copy_pin_lifetime'] else -1,
                       'bring_online': transfer['bring_online'] if transfer['bring_online'] else None,
                       'job_metadata': {'issuer': 'rucio'},  # finaly job_meta will like this. currently job_meta will equal file_meta to include request_id and etc.


Test patch:
Repository: rucio/rucio

Issue:
core.account_limit.get_rse_account_usage should liste accounts who have usage, but no quota
Motivation
----------
Due to the join of the two tables, only accounts with usage and quota are listed, however, accounts without quota should be listed as well.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/account_limit.py b/lib/rucio/core/account_limit.py
--- a/lib/rucio/core/account_limit.py
+++ b/lib/rucio/core/account_limit.py
@@ -1,16 +1,21 @@
-'''
-  Copyright European Organization for Nuclear Research (CERN)
-
-  Licensed under the Apache License, Version 2.0 (the "License");
-  You may not use this file except in compliance with the License.
-  You may obtain a copy of the License at
-  http://www.apache.org/licenses/LICENSE-2.0
-
-  Authors:
-  - Martin Barisits, <martin.barisits@cern.ch>, 2013-2015
-  - Cedric Serfon, <cedric.serfon@cern.ch>, 2015
-  - Vincent Garonne, <vincent.garonne@cern.ch>, 2015
-'''
+# Copyright 2013-2018 CERN for the benefit of the ATLAS collaboration.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Authors:
+#  - Martin Barisits, <martin.barisits@cern.ch>, 2013-2018
+#  - Cedric Serfon, <cedric.serfon@cern.ch>, 2015
+#  - Vincent Garonne, <vincent.garonne@cern.ch>, 2015
 
 from sqlalchemy.orm.exc import NoResultFound
 from sqlalchemy.sql.expression import and_, or_
@@ -32,7 +37,7 @@ def get_rse_account_usage(rse, session=None):
     result = []
     rse_id = get_rse_id(rse=rse, session=session)
     query = session.query(models.AccountUsage.account, models.AccountUsage.files, models.AccountUsage.bytes, models.AccountLimit.bytes)
-    query = query.join(models.AccountLimit, and_(models.AccountUsage.account == models.AccountLimit.account, models.AccountUsage.rse_id == models.AccountLimit.rse_id)).filter(models.AccountUsage.rse_id == rse_id)
+    query = query.outerjoin(models.AccountLimit, and_(models.AccountUsage.account == models.AccountLimit.account, models.AccountUsage.rse_id == models.AccountLimit.rse_id)).filter(models.AccountUsage.rse_id == rse_id)
     account_limits_tmp = query.all()
     for row in account_limits_tmp:
         result.append({'rse': rse, 'account': row[0], 'used_files': row[1], 'used_bytes': row[2], 'quota_bytes': row[3]})


Test patch:
Repository: rucio/rucio

Issue:
Transfers : Bug with verify checksum
Motivation
----------
The conveyor crash if verify_checksum is a bool instead of a string





Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -662,13 +662,13 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                 fts_list = fts_hosts.split(",")
 
                 verify_checksum = 'both'
-                if rse_attrs[dest_rse_id].get('verify_checksum', 'True').lower() == 'false':
-                    if rse_attrs[source_rse_id].get('verify_checksum', 'True').lower() == 'false':
+                if not rse_attrs[dest_rse_id].get('verify_checksum', True):
+                    if not rse_attrs[source_rse_id].get('verify_checksum', True):
                         verify_checksum = 'none'
                     else:
                         verify_checksum = 'source'
                 else:
-                    if rse_attrs[source_rse_id].get('verify_checksum', 'True').lower() == 'false':
+                    if not rse_attrs[source_rse_id].get('verify_checksum', True):
                         verify_checksum = 'destination'
                     else:
                         verify_checksum = 'both'


Test patch:
Repository: rucio/rucio

Issue:
schema: add root and webdav functional test for atlas
Motivation
----------
To clearly separate functional tests for 3rd-party protocol commissioning.


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/schema/atlas.py b/lib/rucio/common/schema/atlas.py
--- a/lib/rucio/common/schema/atlas.py
+++ b/lib/rucio/common/schema/atlas.py
@@ -1,18 +1,20 @@
-'''
-  Copyright European Organization for Nuclear Research (CERN)
-
-  Licensed under the Apache License, Version 2.0 (the "License");
-  You may not use this file except in compliance with the License.
-  You may obtain a copy of the License at
-  http://www.apache.org/licenses/LICENSE-2.0
-
-  Authors:
-  - Vincent Garonne, <vincent.garonne@cern.ch>, 2013-2017
-  - Cedric Serfon, <cedric.serfon@cern.ch>, 2014-2015, 2017
-  - Joaquin Bogado, <joaquin.bogado@cern.ch>, 2015
-  - Mario Lassnig, <mario.lassnig@cern.ch>, 2015
-  - Martin Barisits, <martin.barisits@cern.ch>, 2016
-'''
+# Copyright 2017-2018 CERN for the benefit of the ATLAS collaboration.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Authors:
+# - Vincent Garonne <vgaronne@gmail.com>, 2017
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2018
 
 from jsonschema import validate, ValidationError
 
@@ -37,7 +39,8 @@
 ACTIVITY = {"description": "Activity name",
             "type": "string",
             "enum": ["Data Brokering", "Data Consolidation", "Data rebalancing",
-                     "Debug", "Express", "Functional Test", "Group Subscriptions",
+                     "Debug", "Express", "Functional Test", "Functional Test XrootD",
+                     "Functional Test WebDAV", "Group Subscriptions",
                      "Production Input", "Production Output",
                      "Analysis Input", "Analysis Output", "Staging",
                      "T0 Export", "T0 Tape", "Upload/Download (Job)",


Test patch:
Repository: rucio/rucio

Issue:
Permission : Fix update_replicas_states permission for ATLAS
Motivation
----------
The permission for update_replicas_states is too restrictive and should be extended to TEST endpoints




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/permission/atlas.py b/lib/rucio/core/permission/atlas.py
--- a/lib/rucio/core/permission/atlas.py
+++ b/lib/rucio/core/permission/atlas.py
@@ -15,7 +15,7 @@
 # Authors:
 # - Vincent Garonne <vgaronne@gmail.com>, 2016
 # - Martin Barisits <martin.barisits@cern.ch>, 2016-2018
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2016-2017
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2016-2018
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2018
 
 import rucio.core.authentication
@@ -767,18 +767,15 @@ def perm_update_replicas_states(issuer, kwargs):
     for kv in list_account_attributes(account=issuer):
         if kv['key'].startswith('group-') and kv['value'] in ['admin', 'user']:
             phys_group.append(kv['key'].partition('-')[2])
+    rse_attr = list_rse_attributes(rse=rse)
     if phys_group:
-        rse_attr = list_rse_attributes(rse=rse)
         if rse_attr.get('type', '') == 'GROUPDISK':
             if rse_attr.get('physgroup', '') in phys_group:
                 return True
-
-    return rse.endswith('SCRATCHDISK')\
-        or rse.endswith('USERDISK')\
-        or rse.endswith('MOCK')\
-        or rse.endswith('LOCALGROUPDISK')\
-        or issuer == 'root'\
-        or has_account_attribute(account=issuer, key='admin')
+    else:
+        return rse_attr.get('type', '') in ['SCRATCHDISK', 'MOCK', 'LOCALGROUPDISK', 'TEST']\
+            or issuer == 'root'\
+            or has_account_attribute(account=issuer, key='admin')
 
 
 def perm_queue_requests(issuer, kwargs):


Test patch:
Repository: rucio/rucio

Issue:
Permission : Fix update_replicas_states permission for ATLAS
Motivation
----------
The permission for update_replicas_states is too restrictive and should be extended to TEST endpoints


Travis build fails because of ipaddress
Motivation
----------
Travis fails to build containers because pip refuses to install ipaddress which is already installed by yum.

Modification
------------
- [ ] In travis Dockerfile: remove ipaddress site-package before pip calls



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/permission/atlas.py b/lib/rucio/core/permission/atlas.py
--- a/lib/rucio/core/permission/atlas.py
+++ b/lib/rucio/core/permission/atlas.py
@@ -15,7 +15,7 @@
 # Authors:
 # - Vincent Garonne <vgaronne@gmail.com>, 2016
 # - Martin Barisits <martin.barisits@cern.ch>, 2016-2018
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2016-2017
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2016-2018
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2018
 
 import rucio.core.authentication
@@ -767,18 +767,15 @@ def perm_update_replicas_states(issuer, kwargs):
     for kv in list_account_attributes(account=issuer):
         if kv['key'].startswith('group-') and kv['value'] in ['admin', 'user']:
             phys_group.append(kv['key'].partition('-')[2])
+    rse_attr = list_rse_attributes(rse=rse)
     if phys_group:
-        rse_attr = list_rse_attributes(rse=rse)
         if rse_attr.get('type', '') == 'GROUPDISK':
             if rse_attr.get('physgroup', '') in phys_group:
                 return True
-
-    return rse.endswith('SCRATCHDISK')\
-        or rse.endswith('USERDISK')\
-        or rse.endswith('MOCK')\
-        or rse.endswith('LOCALGROUPDISK')\
-        or issuer == 'root'\
-        or has_account_attribute(account=issuer, key='admin')
+    else:
+        return rse_attr.get('type', '') in ['SCRATCHDISK', 'MOCK', 'LOCALGROUPDISK', 'TEST']\
+            or issuer == 'root'\
+            or has_account_attribute(account=issuer, key='admin')
 
 
 def perm_queue_requests(issuer, kwargs):


Test patch:
Repository: rucio/rucio

Issue:
RSE manager: remove .part from filename if adler32 is set and ignore_checksum is true
Motivation
----------
rsemanager.download, downloads the file as <filename>.part if an adler32 checksum is given.
If ignore_checksum is set to True the file will not be renamed thus the .part will not be removed from the file name.

Modification
------------
Make the  os.rename call independent of ignore_checksum


Discussion:
@berghaus You recently did some changes in this code I think. Can you maybe have a look on this issue? 😃 

Resolving patch:
diff --git a/lib/rucio/rse/rsemanager.py b/lib/rucio/rse/rsemanager.py
--- a/lib/rucio/rse/rsemanager.py
+++ b/lib/rucio/rse/rsemanager.py
@@ -267,16 +267,19 @@ def download(rse_settings, files, dest_dir=None, force_scheme=None, ignore_check
                     if printstatements:
                         print('File downloaded. Will be validated')
 
-                    if not ignore_checksum:
+                    if ignore_checksum:
+                        if printstatements:
+                            print('Skipping checksum validation')
+                    else:
                         ruciochecksum = f['adler32'] if f['adler32'] else f['md5']
                         localchecksum = utils.adler32(tempfile) if f['adler32'] else utils.md5(tempfile)
                         if localchecksum == ruciochecksum:
                             if printstatements:
                                 print('File validated')
-                            os.rename(tempfile, finalfile)
                         else:
                             os.unlink(tempfile)
                             raise exception.FileConsistencyMismatch('Checksum mismatch : local %s vs recorded %s' % (str(localchecksum), str(ruciochecksum)))
+                    os.rename(tempfile, finalfile)
                 else:
                     protocol.get(pfn, '%s/%s' % (target_dir, f['name']), transfer_timeout=transfer_timeout)
                 ret['%s:%s' % (f['scope'], f['name'])] = True


Test patch:
Repository: rucio/rucio

Issue:
Fixes for replica selection in conveyor
Motivation
----------
The replica selection in the conveyor is based on two things: ranking in the sources table and ranking in the distances table. Two things need to be changed:
- [x] In certain branches of the code sources with distance `ranking` of 0 are excluded. This is wrong. Only ranking of `None` should be excluded.
- [x] For the tape selection the distance ranking with the highest value is used. This is wrong. The lowest value should be used.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -718,7 +718,7 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                 if source_rse_id is None or rse is None:
                     continue
 
-                if link_ranking is None or link_ranking == 0:
+                if link_ranking is None:
                     logging.debug("Request %s: no link from %s to %s" % (id, source_rse_id, dest_rse_id))
                     continue
 


Test patch: