internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
RUCIO_ACCOUNT should override the Rucio configuration file
Motivation
----------
Use of the RUCIO_ACCOUNT environment variable should override the account specified in the Rucio configuration file.





Discussion:

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -215,12 +215,12 @@ def __init__(self, rucio_host=None, auth_host=None, account=None, ca_cert=None,
         if account is None:
             LOG.debug('no account passed. Trying to get it from the config file.')
             try:
-                self.account = config_get('client', 'account')
-            except (NoOptionError, NoSectionError):
+                self.account = environ['RUCIO_ACCOUNT']
+            except KeyError:
                 try:
-                    self.account = environ['RUCIO_ACCOUNT']
-                except KeyError:
-                    raise MissingClientParameter('Option \'account\' cannot be found in config file and RUCIO_ACCOUNT is not set.')
+                    self.account = config_get('client', 'account')
+                except (NoOptionError, NoSectionError):
+                        raise MissingClientParameter('Option \'account\' cannot be found in config file and RUCIO_ACCOUNT is not set.')
 
         token_path = self.TOKEN_PATH_PREFIX + self.account
         self.token_file = token_path + '/' + self.TOKEN_PREFIX + self.account


Test patch:
Repository: rucio/rucio

Issue:
Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas
Motivation
----------

Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas







Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -808,7 +808,7 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
         .filter(models.Request.request_type == RequestType.TRANSFER)\
         .join(models.RSE, models.RSE.id == models.Request.dest_rse_id)\
         .filter(models.RSE.deleted == false())\
-        .filter(models.RSE.availability.in_((0, 1, 4, 5)))
+        .filter(models.RSE.availability.in_((2, 3, 6, 7)))
 
     if isinstance(older_than, datetime.datetime):
         sub_requests = sub_requests.filter(models.Request.requested_at < older_than)


Test patch:
Repository: rucio/rucio

Issue:
Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas
Motivation
----------

Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas





M2Crpyto installation blocks all tests
Motivation
----------
The `fts3-rest-api` package has workarounds in their setup.py which invalidates the dependency selection of M2Crypto package. 
Since there is now a new version of M2Crpypto this corrupts the installation of the dependency. Only possible workaround is to update the dependency version to have a consistent installation. In the medium term we should get rid of `fts3-rest-api` and `M2Crypto` dependencies.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -808,7 +808,7 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
         .filter(models.Request.request_type == RequestType.TRANSFER)\
         .join(models.RSE, models.RSE.id == models.Request.dest_rse_id)\
         .filter(models.RSE.deleted == false())\
-        .filter(models.RSE.availability.in_((0, 1, 4, 5)))
+        .filter(models.RSE.availability.in_((2, 3, 6, 7)))
 
     if isinstance(older_than, datetime.datetime):
         sub_requests = sub_requests.filter(models.Request.requested_at < older_than)


Test patch:
Repository: rucio/rucio

Issue:
delete_messages crashes when a message longer than 4000 is provided
Motivation
----------
delete_messages crashes when a message longer than 4000 is provided. The problem is in `session.bulk_insert_mappings(MessageHistory, messages)`. The payload that is provided by Hermes comes from `retrieve_messages`, but in case the payload registered in the DB is `nolimit`, `retrieve_messages` replaces `payload with `payload_nolimit`, generating a Database exception :
```
DatabaseException: Database exception.
Details: (cx_Oracle.DatabaseError) ORA-01461: can bind a LONG value only for insert into a LONG column
[SQL: INSERT INTO atlas_rucio.messages_history (id, event_type, payload, updated_at, created_at) VALUES (:id, :event_type, :payload, :updated_at, :created_at)]
[parameters: xxxx'}"}]
(Background on this error at: http://sqlalche.me/e/4xp6)
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/message.py b/lib/rucio/core/message.py
--- a/lib/rucio/core/message.py
+++ b/lib/rucio/core/message.py
@@ -143,6 +143,8 @@ def delete_messages(messages, session=None):
     message_condition = []
     for message in messages:
         message_condition.append(Message.id == message['id'])
+        if len(message['payload']) > 4000:
+            message['payload_nolimit'] = message.pop('payload')
 
     try:
         if message_condition:


Test patch:
Repository: rucio/rucio

Issue:
Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas
Motivation
----------

Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas





delete_messages crashes when a message longer than 4000 is provided
Motivation
----------
delete_messages crashes when a message longer than 4000 is provided. The problem is in `session.bulk_insert_mappings(MessageHistory, messages)`. The payload that is provided by Hermes comes from `retrieve_messages`, but in case the payload registered in the DB is `nolimit`, `retrieve_messages` replaces `payload with `payload_nolimit`, generating a Database exception :
```
DatabaseException: Database exception.
Details: (cx_Oracle.DatabaseError) ORA-01461: can bind a LONG value only for insert into a LONG column
[SQL: INSERT INTO atlas_rucio.messages_history (id, event_type, payload, updated_at, created_at) VALUES (:id, :event_type, :payload, :updated_at, :created_at)]
[parameters: xxxx'}"}]
(Background on this error at: http://sqlalche.me/e/4xp6)
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -808,7 +808,7 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
         .filter(models.Request.request_type == RequestType.TRANSFER)\
         .join(models.RSE, models.RSE.id == models.Request.dest_rse_id)\
         .filter(models.RSE.deleted == false())\
-        .filter(models.RSE.availability.in_((0, 1, 4, 5)))
+        .filter(models.RSE.availability.in_((2, 3, 6, 7)))
 
     if isinstance(older_than, datetime.datetime):
         sub_requests = sub_requests.filter(models.Request.requested_at < older_than)
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.20.0',
-    'branch_nick': 'master',
-    'revision_id': '0226a0dbfb3899568dcefd930bf27a7816521b21',
-    'revno': 8293
+    'version': '1.20.1',
+    'branch_nick': 'patch-0-Release__Rucio_1_20_1_release_preparation',
+    'revision_id': '377f7ffc67f3bf43796cec5f5d1f8c444e2528c6',
+    'revno': 8310
 }


Test patch:
Repository: rucio/rucio

Issue:
Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas
Motivation
----------

Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas





delete_messages crashes when a message longer than 4000 is provided
Motivation
----------
delete_messages crashes when a message longer than 4000 is provided. The problem is in `session.bulk_insert_mappings(MessageHistory, messages)`. The payload that is provided by Hermes comes from `retrieve_messages`, but in case the payload registered in the DB is `nolimit`, `retrieve_messages` replaces `payload with `payload_nolimit`, generating a Database exception :
```
DatabaseException: Database exception.
Details: (cx_Oracle.DatabaseError) ORA-01461: can bind a LONG value only for insert into a LONG column
[SQL: INSERT INTO atlas_rucio.messages_history (id, event_type, payload, updated_at, created_at) VALUES (:id, :event_type, :payload, :updated_at, :created_at)]
[parameters: xxxx'}"}]
(Background on this error at: http://sqlalche.me/e/4xp6)
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -808,7 +808,7 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
         .filter(models.Request.request_type == RequestType.TRANSFER)\
         .join(models.RSE, models.RSE.id == models.Request.dest_rse_id)\
         .filter(models.RSE.deleted == false())\
-        .filter(models.RSE.availability.in_((0, 1, 4, 5)))
+        .filter(models.RSE.availability.in_((2, 3, 6, 7)))
 
     if isinstance(older_than, datetime.datetime):
         sub_requests = sub_requests.filter(models.Request.requested_at < older_than)


Test patch:
Repository: rucio/rucio

Issue:
Crash on __list_transfer_requests_and_source_replicas
Motivation
----------

```
2019-06-19 14:17:43,558 32683   CRITICAL        Thread [1/31] : Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/submitter.py", line 162, in submitter
    retry_other_fts=retry_other_fts)
  File "/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/submitter.py", line 286, in __get_transfers
    failover_schemes=failover_schemes)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python2.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python2.7/site-packages/rucio/db/sqla/session.py", line 281, in new_funct
    raise DatabaseException(str(error))
DatabaseException: Database exception.
Details: (cx_Oracle.DatabaseError) ORA-00918: column ambiguously defined
```





Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -820,11 +820,11 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
         if session.bind.dialect.name == 'oracle':
             bindparams = [bindparam('worker_number', worker_number),
                           bindparam('total_workers', total_workers)]
-            sub_requests = sub_requests.filter(text('ORA_HASH(id, :total_workers) = :worker_number', bindparams=bindparams))
+            sub_requests = sub_requests.filter(text('ORA_HASH(requests.id, :total_workers) = :worker_number', bindparams=bindparams))
         elif session.bind.dialect.name == 'mysql':
-            sub_requests = sub_requests.filter(text('mod(md5(id), %s) = %s' % (total_workers + 1, worker_number)))
+            sub_requests = sub_requests.filter(text('mod(md5(requests.id), %s) = %s' % (total_workers + 1, worker_number)))
         elif session.bind.dialect.name == 'postgresql':
-            sub_requests = sub_requests.filter(text('mod(abs((\'x\'||md5(id::text))::bit(32)::int), %s) = %s' % (total_workers + 1, worker_number)))
+            sub_requests = sub_requests.filter(text('mod(abs((\'x\'||md5(requests.id::text))::bit(32)::int), %s) = %s' % (total_workers + 1, worker_number)))
 
     if limit:
         sub_requests = sub_requests.limit(limit)


Test patch:
Repository: rucio/rucio

Issue:
Crash on __list_transfer_requests_and_source_replicas
Motivation
----------

```
2019-06-19 14:17:43,558 32683   CRITICAL        Thread [1/31] : Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/submitter.py", line 162, in submitter
    retry_other_fts=retry_other_fts)
  File "/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/submitter.py", line 286, in __get_transfers
    failover_schemes=failover_schemes)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python2.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python2.7/site-packages/rucio/db/sqla/session.py", line 281, in new_funct
    raise DatabaseException(str(error))
DatabaseException: Database exception.
Details: (cx_Oracle.DatabaseError) ORA-00918: column ambiguously defined
```





Discussion:

Resolving patch:
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -156,7 +156,7 @@ def upload(self, items, summary_file_path=None):
             if not is_deterministic and not pfn:
                 logger.error('PFN has to be defined for NON-DETERMINISTIC RSE.')
                 continue
-            if pfn and not is_deterministic:
+            if pfn and is_deterministic:
                 logger.warning('Upload with given pfn implies that no_register is True, except non-deterministic RSEs')
                 no_register = True
 
@@ -166,7 +166,7 @@ def upload(self, items, summary_file_path=None):
             # if register_after_upload, file should be overwritten if it is not registered
             # otherwise if file already exists on RSE we're done
             if register_after_upload:
-                if rsemgr.exists(rse_settings, file_did):
+                if rsemgr.exists(rse_settings, pfn if pfn else file_did):
                     try:
                         self.client.get_did(file['did_scope'], file['did_name'])
                         logger.info('File already registered. Skipping upload.')


Test patch:
Repository: rucio/rucio

Issue:
fts3 transfer tool reads wrong config
Motivation
----------
The fts3 transfer tools in line 500 calls:
https://github.com/rucio/rucio/blob/9ee492985fb4fcc04a5956b69f87f3e29c88b4a7/lib/rucio/transfertool/fts3.py#L500

whereas the correct one would be:
`policy = config_get('policy', 'permission')`




Discussion:

Resolving patch:
diff --git a/lib/rucio/transfertool/fts3.py b/lib/rucio/transfertool/fts3.py
--- a/lib/rucio/transfertool/fts3.py
+++ b/lib/rucio/transfertool/fts3.py
@@ -497,7 +497,7 @@ def set_se_config(self, storage_element, inbound_max_active=None, outbound_max_a
         params_dict = {storage_element: {'operations': {}, 'se_info': {}}}
         if staging is not None:
             try:
-                policy = config_get('permission', 'policy')
+                policy = config_get('policy', 'permission')
             except Exception:
                 logging.warn('Could not get policy from config')
             params_dict[storage_element]['operations'] = {policy: {'staging': staging}}


Test patch:
Repository: rucio/rucio

Issue:
Reaper crashes when it tries to delete a DIDs that is a constituent of an archive
Motivation
----------
Reaper crashes when it tries to delete a DIDs that is a constituent of an archive








Discussion:

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -1395,7 +1395,9 @@ def delete_replicas(rse, files, ignore_availability=True, session=None):
                                      ~exists(select([1]).prefix_with("/*+ INDEX(REPLICAS REPLICAS_PK) */", dialect='oracle')).where(and_(models.RSEFileAssociation.scope == file['scope'], models.RSEFileAssociation.name == file['name']))))
 
         did_condition.append(and_(models.DataIdentifier.scope == file['scope'], models.DataIdentifier.name == file['name'], models.DataIdentifier.availability != DIDAvailability.LOST,
-                                  ~exists(select([1]).prefix_with("/*+ INDEX(REPLICAS REPLICAS_PK) */", dialect='oracle')).where(and_(models.RSEFileAssociation.scope == file['scope'], models.RSEFileAssociation.name == file['name']))))
+                                  ~exists(select([1]).prefix_with("/*+ INDEX(REPLICAS REPLICAS_PK) */", dialect='oracle')).where(and_(models.RSEFileAssociation.scope == file['scope'], models.RSEFileAssociation.name == file['name'])),
+                                  ~exists(select([1]).prefix_with("/*+ INDEX(ARCHIVE_CONTENTS ARCH_CONTENTS_PK) */", dialect='oracle')).where(and_(models.ConstituentAssociation.child_scope == file['scope'],
+                                                                                                                                                   models.ConstituentAssociation.child_name == file['name']))))
 
         archive_contents_condition.append(and_(models.ConstituentAssociation.scope == file['scope'],
                                                models.ConstituentAssociation.name == file['name'],


Test patch:
Repository: rucio/rucio

Issue:
Temporary unavailable files injection fails if duplucates
Motivation
----------
Traceback :
```
Details: [u'(cx_Oracle.IntegrityError) ORA-00001: unique constraint (ATLAS_RUCIO.BAD_PFNS_PK) violated']
Rucio exited with an unexpected/unknown error, please provide the traceback below to the developers.
Traceback (most recent call last):
  File "/usr/bin/rucio-admin", line 129, in new_funct
    return function(*args, **kwargs)
  File "/usr/bin/rucio-admin", line 1034, in declare_temporary_unavailable_replicas
    client.add_bad_pfns(pfns=chunk, reason=args.reason, state='TEMPORARY_UNAVAILABLE', expires_at=expiration_date)
  File "/usr/lib/python2.7/site-packages/rucio/client/replicaclient.py", line 336, in add_bad_pfns
    raise exc_cls(exc_msg)
RucioException: An unknown exception occurred.
Details: [u'(cx_Oracle.IntegrityError) ORA-00001: unique constraint (ATLAS_RUCIO.BAD_PFNS_PK) violated']
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -2695,7 +2695,9 @@ def add_bad_pfns(pfns, account, state, reason=None, expires_at=None, session=Non
     pfns = clean_surls(pfns)
     for pfn in pfns:
         new_pfn = models.BadPFNs(path=str(pfn), account=account, state=rep_state, reason=reason, expires_at=expires_at)
+        new_pfn = session.merge(new_pfn)
         new_pfn.save(session=session, flush=False)
+
     try:
         session.flush()
     except IntegrityError as error:


Test patch:
Repository: rucio/rucio

Issue:
small enhancements for data import
Motivation
----------
- catch exceptions (for example if distance gets created where the RSE got deleted before), rollback and return error message -> already done because session handling already rollbacks if there is an exception
- sort protocols in order to compare the dictionaries in the tests
- fix tests/test_import_export.py where destination RSE of distances are deleted before distance creation


Modification
------------





Discussion:

Resolving patch:
diff --git a/lib/rucio/core/rse.py b/lib/rucio/core/rse.py
--- a/lib/rucio/core/rse.py
+++ b/lib/rucio/core/rse.py
@@ -926,6 +926,7 @@ def get_rse_protocols(rse, schemes=None, session=None):
             pass  # If value is not a JSON string
 
         info['protocols'].append(p)
+    info['protocols'] = sorted(info['protocols'], key=lambda p: (p['hostname'], p['scheme'], p['port']))
     return info
 
 


Test patch:
diff --git a/lib/rucio/tests/test_import_export.py b/lib/rucio/tests/test_import_export.py
--- a/lib/rucio/tests/test_import_export.py
+++ b/lib/rucio/tests/test_import_export.py
@@ -105,6 +105,11 @@ def setup(self):
         add_rse(self.old_rse_3)
         self.old_rse_id_3 = get_rse_id(self.old_rse_3)
 
+        # RSE 4 that already exists
+        self.old_rse_4 = rse_name_generator()
+        add_rse(self.old_rse_4)
+        self.old_rse_id_4 = get_rse_id(self.old_rse_4)
+
         # Distance that already exists
         add_distance(self.old_rse_id_1, self.old_rse_id_2)
 
@@ -173,7 +178,9 @@ def setup(self):
                     'availability_delete': False,
                     'availability_read': False,
                     'availability_write': True
-                }
+                },
+                self.old_rse_2: {},
+                self.old_rse_3: {}
             },
             'distances': {
                 self.old_rse_1: {
@@ -227,9 +234,9 @@ def test_importer_core(self):
         distance = get_distances(self.old_rse_id_1, self.old_rse_id_3)[0]
         assert_equal(distance['ranking'], 4)
 
-        # RSE 3 should be flagged as deleted as it is missing in the import data
+        # RSE 4 should be flagged as deleted as it is missing in the import data
         with assert_raises(RSENotFound):
-            get_rse(self.old_rse_3)
+            get_rse(self.old_rse_4)
 
         import_data(data=self.data2)
         import_data(data=self.data3)
@@ -271,7 +278,7 @@ def test_importer_client(self):
         assert_equal(distance['ranking'], 4)
 
         with assert_raises(RSENotFound):
-            get_rse(self.old_rse_3)
+            get_rse(self.old_rse_4)
 
         import_client.import_data(data=self.data2)
         import_client.import_data(data=self.data3)
@@ -319,7 +326,7 @@ def test_importer_rest(self):
         assert_equal(distance['ranking'], 4)
 
         with assert_raises(RSENotFound):
-            get_rse(self.old_rse_3)
+            get_rse(self.old_rse_4)
 
         r2 = TestApp(import_app.wsgifunc(*mw)).post('/', headers=headers2, expect_errors=True, params=render_json(**self.data2))
         assert_equal(r2.status, 201)
Repository: rucio/rucio

Issue:
Kronos needs rse_id update
Motivation
----------
#2551 missed Kronos calls to touch_replica, touch_collection_replicas, and touch_dataset_locks


Modification
------------
Fix Kronos so that it converts to rse_id before making these calls.




Discussion:

Resolving patch:
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.20.2',
-    'branch_nick': 'patch-0-Release__1_20_2_preparation',
-    'revision_id': 'e34b4d7d73f9c358664e42ceb427d2dbe209e9ca',
-    'revno': 8378
+    'version': '1.20.3rc1',
+    'branch_nick': 'patch-0-Release__1_20_3rc1_preparation',
+    'revision_id': '374e51a371a9c6ae8f7dcc7e2a90e5665bc3a65f',
+    'revno': 8404
 }


Test patch: