anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: rucio/rucio
Issue: Temporary unavailable files in bad_pfns not removed if processed after expiration date 

Motivation
----------
Minos : Temporary unavailable files in bad_pfns not removed if processed after expiration date 






Changed files:
lib/rucio/daemons/badreplicas/minos.py

diff --git a/lib/rucio/daemons/badreplicas/minos.py b/lib/rucio/daemons/badreplicas/minos.py
index 9dda82898..d1426418c 100644
--- a/lib/rucio/daemons/badreplicas/minos.py
+++ b/lib/rucio/daemons/badreplicas/minos.py
@@ -28,6 +28,7 @@ import traceback
 import threading
 import time
 
+from datetime import datetime
 from sys import stdout, argv
 
 from rucio.db.sqla.constants import BadFilesStatus, BadPFNStatus, ReplicaState
@@ -227,6 +228,9 @@ def minos(bulk=1000, once=False, sleep_time=60):
                                     if rep['rse_id'] in unavailable_states:
                                         logging.info(prepend_str + '%s is in unavailable state. Will be removed from the list of bad PFNs' % str(rep['pfn']))
                                         bulk_delete_bad_pfns(pfns=[rep['pfn']], session=None)
+                                    elif expires_at < datetime.now():
+                                        logging.info('%s PFN %s expiration time (%s) is older than now and is not in unavailable state. Removing the PFNs from bad_pfns', prepend_str, str(rep['pfn']), expires_at)
+                                        bulk_delete_bad_pfns(pfns=[rep['pfn']], session=None)
                                 except (DataIdentifierNotFound, ReplicaNotFound) as error:
                                     logging.error(prepend_str + 'Will remove %s from the list of bad PFNs' % str(rep['pfn']))
                                     bulk_delete_bad_pfns(pfns=[rep['pfn']], session=None)
Repository: rucio/rucio
Issue: Conveyor Finisher fails to retry requests in the "protocol mismatch" state

Hi,

Even with conveyor.retry_protocol_mismatches set to True, the finisher seems to fail to retry protocol mismatch errors at all (it just ignores them). This seems to be handled by these lines:
https://github.com/rucio/rucio/blob/916987976a1f994a64e4ef3dce837b49141a8f1b/lib/rucio/daemons/conveyor/finisher.py#L215
https://github.com/rucio/rucio/blob/916987976a1f994a64e4ef3dce837b49141a8f1b/lib/rucio/daemons/conveyor/finisher.py#L278

But it never gets any requests in the MISMATCH_SCHEME state, seemingly because this constant isn't included in the main database query filter here:
https://github.com/rucio/rucio/blob/916987976a1f994a64e4ef3dce837b49141a8f1b/lib/rucio/daemons/conveyor/finisher.py#L121-L129

Would it be possible for this to be fixed?

Regards,
Simon


Changed files:
lib/rucio/daemons/conveyor/finisher.py

diff --git a/lib/rucio/daemons/conveyor/finisher.py b/lib/rucio/daemons/conveyor/finisher.py
index 0a69bcde7..acbcaf234 100644
--- a/lib/rucio/daemons/conveyor/finisher.py
+++ b/lib/rucio/daemons/conveyor/finisher.py
@@ -17,7 +17,7 @@
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2015-2019
 # - Vincent Garonne <vgaronne@gmail.com>, 2015-2018
 # - Martin Barisits <martin.barisits@cern.ch>, 2015-2019
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2017-2019
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2017-2020
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018
 # - Robert Illingworth <illingwo@fnal.gov>, 2019
 # - Andrew Lister <andrew.lister@stfc.ac.uk>, 2019
@@ -119,8 +119,10 @@ def finisher(once=False, sleep_time=60, activities=None, bulk=100, db_bulk=1000)
                 logging.debug('%s Working on activity %s', prepend_str, activity)
                 time1 = time.time()
                 reqs = request_core.get_next(request_type=[RequestType.TRANSFER, RequestType.STAGEIN, RequestType.STAGEOUT],
-                                             state=[RequestState.DONE, RequestState.FAILED, RequestState.LOST, RequestState.SUBMITTING,
-                                                    RequestState.SUBMISSION_FAILED, RequestState.NO_SOURCES, RequestState.ONLY_TAPE_SOURCES],
+                                             state=[RequestState.DONE, RequestState.FAILED,
+                                                    RequestState.LOST, RequestState.SUBMITTING,
+                                                    RequestState.SUBMISSION_FAILED, RequestState.NO_SOURCES,
+                                                    RequestState.ONLY_TAPE_SOURCES, RequestState.MISMATCH_SCHEME],
                                              limit=db_bulk,
                                              older_than=datetime.datetime.utcnow(),
                                              total_workers=heart_beat['nr_threads'] - 1,
Repository: rucio/rucio
Issue: Reaper 2.0 is creating error traces for SourceNotFound

Motivation
----------

For example:
```Thread [17/20] : Deletion NOTFOUND of HOST.1 as IP|HOST:PORT/PATH on MWT2_DATADISK```
This is different from the previous Reaper implementation. Such traces reduce the deletion efficiency as we are currently calculating it in our monitoring. Is it intentional?

Modification
------------

Potentially do not submit traces for `SourceNotFound`.

Changed files:
lib/rucio/daemons/reaper/reaper2.py

diff --git a/lib/rucio/daemons/reaper/reaper2.py b/lib/rucio/daemons/reaper/reaper2.py
index e74cbbbb2..031474ace 100644
--- a/lib/rucio/daemons/reaper/reaper2.py
+++ b/lib/rucio/daemons/reaper/reaper2.py
@@ -19,7 +19,7 @@
 # - Wen Guan <wguan.icedew@gmail.com>, 2016
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018-2019
 # - Dimitrios Christidis <dimitrios.christidis@cern.ch>, 2019
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2019
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2019-2020
 # - Andrew Lister <andrew.lister@stfc.ac.uk>, 2019
 # - Brandon White <bjwhite@fnal.gov>, 2019
 #
@@ -48,7 +48,6 @@ from dogpile.cache import make_region
 from dogpile.cache.api import NO_VALUE
 from sqlalchemy.exc import DatabaseError, IntegrityError
 
-from rucio.db.sqla.constants import ReplicaState
 from rucio.common.config import config_get
 from rucio.common.exception import (DatabaseException, RSENotFound, ConfigNotFound, ReplicaUnAvailable, ReplicaNotFound, ServiceUnavailable, RSEAccessDenied, ResourceTemporaryUnavailable, SourceNotFound)
 from rucio.common.utils import chunks
@@ -164,9 +163,6 @@ def delete_from_storage(replicas, prot, rse_info, staging_areas, prepend_str):
                 err_msg = 'Deletion NOTFOUND of %s:%s as %s on %s' % (replica['scope'], replica['name'], replica['pfn'], rse_name)
                 logging.warning('%s %s', prepend_str, err_msg)
                 deleted_files.append({'scope': replica['scope'], 'name': replica['name']})
-                if replica['state'] == ReplicaState.AVAILABLE:
-                    deletion_dict['reason'] = str(err_msg)
-                    add_message('deletion-failed', deletion_dict)
 
             except (ServiceUnavailable, RSEAccessDenied, ResourceTemporaryUnavailable) as error:
                 logging.warning('%s Deletion NOACCESS of %s:%s as %s on %s: %s', prepend_str, replica['scope'], replica['name'], replica['pfn'], rse_name, str(error))
Repository: rucio/rucio
Issue: Crash when enabling multihop mode

Motivation
----------
The submitter crash when enabling multihop mode







Changed files:
lib/rucio/core/request.py, lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/request.py b/lib/rucio/core/request.py
index c8d5f92b6..02ef5f52d 100644
--- a/lib/rucio/core/request.py
+++ b/lib/rucio/core/request.py
@@ -12,7 +12,7 @@
 # - Wen Guan, <wen.guan@cern.ch>, 2014-2016
 # - Joaquin Bogado, <jbogadog@cern.ch>, 2016
 # - Thomas Beermann, <thomas.beermann@cern.ch>, 2016
-# - Cedric Serfon, <cedric.serfon@cern.ch>, 2017-2019
+# - Cedric Serfon, <cedric.serfon@cern.ch>, 2017-2020
 # - Hannes Hansen, <hannes.jakob.hansen@cern.ch>, 2018-2019
 # - Andrew Lister, <andrew.lister@stfc.ac.uk>, 2019
 #
@@ -485,6 +485,7 @@ def set_request_state(request_id, new_state, transfer_id=None, transferred_at=No
 
     record_counter('core.request.set_request_state')
 
+    rowcount = 0
     try:
         update_items = {'state': new_state, 'updated_at': datetime.datetime.utcnow()}
         if transferred_at:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 187df6c2e..57ef2df68 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -8,7 +8,7 @@
 # Authors:
 # - Martin Barisits, <martin.barisits@cern.ch>, 2017-2018
 # - Mario Lassnig, <mario.lassnig@cern.ch>, 2017-2018
-# - Cedric Serfon, <cedric.serfon@cern.ch>, 2018-2019
+# - Cedric Serfon, <cedric.serfon@cern.ch>, 2018-2020
 # - Hannes Hansen, <hannes.jakob.hansen@cern.ch>, 2018
 # - Robert Illingworth, <illingwo@fnal.gov>, 2019
 # - Andrew Lister, <andrew.lister@stfc.ac.uk>, 2019
@@ -1038,7 +1038,11 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                         # Need to fail all the intermediate requests + the initial one and exit the multihop loop
                         logging.warning('Multihop : A request already exists for the transfer between %s and %s. Will cancel all the parent requests', source_rse_name, dest_rse_name)
                         parent_requests.append(req_id)
-                        set_requests_state(request_ids=parent_requests, new_state=RequestState.FAILED, session=session)
+                        try:
+                            set_requests_state(request_ids=parent_requests, new_state=RequestState.FAILED, session=session)
+                        except UnsupportedOperation:
+                            logging.error('Multihop : Cannot cancel all the parent requests : %s', str(parent_requests))
+
                         # Remove from the transfer dictionary all the requests
                         for cur_req_id in parent_requests:
                             transfers.pop(cur_req_id, None)
Repository: rucio/rucio
Issue: Fix variable initialization in delete_replica

Motivation
----------
The rowcount variable is not properly reinitialized in `delete_replicas`




Changed files:
lib/rucio/core/replica.py

diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
index b0c858179..1af52f0e9 100644
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -14,7 +14,7 @@
 #
 # Authors:
 # - Vincent Garonne <vgaronne@gmail.com>, 2013-2018
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2019
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2020
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2013-2014
 # - Martin Barisits <martin.barisits@cern.ch>, 2013-2019
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2014-2019
@@ -1407,6 +1407,7 @@ def delete_replicas(rse_id, files, ignore_availability=True, session=None):
             filter(or_(*chunk)).\\
             delete(synchronize_session=False)
 
+    rowcount = 0
     for chunk in chunks(replica_condition, 10):
         for (scope, name, rid, replica_bytes) in session.query(models.RSEFileAssociation.scope, models.RSEFileAssociation.name, models.RSEFileAssociation.rse_id, models.RSEFileAssociation.bytes).\\
                 with_hint(models.RSEFileAssociation, "INDEX(REPLICAS REPLICAS_PK)", 'oracle').filter(models.RSEFileAssociation.rse_id == rse_id).filter(or_(*chunk)):
Repository: rucio/rucio
Issue: Bug when a file with no available replicas is declared

Motivation
----------
When a replica has no replicas `AVAILABLE` and one declare one of these replicas as lost, all the requests for the other replicas are canceled but not the one of the file that was declared.







Changed files:
lib/rucio/core/rule.py

diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
index 03590e973..995d17f24 100644
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -16,7 +16,7 @@
 # - Vincent Garonne <vgaronne@gmail.com>, 2012-2018
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2018
 # - Martin Barisits <martin.barisits@cern.ch>, 2013-2020
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2014-2019
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2014-2020
 # - David Cameron <d.g.cameron@gmail.com>, 2014
 # - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2014-2018
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2014-2015
@@ -41,9 +41,9 @@ except ImportError:
 from copy import deepcopy
 from datetime import datetime, timedelta
 from re import match
-from six import string_types
 from string import Template
 
+from six import string_types
 from sqlalchemy.exc import IntegrityError, StatementError
 from sqlalchemy.orm.exc import NoResultFound
 from sqlalchemy.sql import func
@@ -1700,6 +1700,8 @@ def update_rules_for_lost_replica(scope, name, rse_id, nowait=False, session=Non
 
     locks = session.query(models.ReplicaLock).filter(models.ReplicaLock.scope == scope, models.ReplicaLock.name == name, models.ReplicaLock.rse_id == rse_id).with_for_update(nowait=nowait).all()
     replica = session.query(models.RSEFileAssociation).filter(models.RSEFileAssociation.scope == scope, models.RSEFileAssociation.name == name, models.RSEFileAssociation.rse_id == rse_id).with_for_update(nowait=nowait).one()
+    requests = session.query(models.Request).filter(models.Request.scope == scope, models.Request.name == name, models.Request.dest_rse_id == rse_id).with_for_update(nowait=nowait).all()
+
     rse = get_rse_name(rse_id, session=session)
 
     datasets = []
@@ -1708,6 +1710,9 @@ def update_rules_for_lost_replica(scope, name, rse_id, nowait=False, session=Non
         if {'name': parent['name'], 'scope': parent['scope']} not in datasets:
             datasets.append({'name': parent['name'], 'scope': parent['scope']})
 
+    for request in requests:
+        session.delete(request)
+
     for lock in locks:
         rule = session.query(models.ReplicationRule).filter(models.ReplicationRule.id == lock.rule_id).with_for_update(nowait=nowait).one()
         rule_state_before = rule.state
Repository: rucio/rucio
Issue: Atropos can’t fetch the list of rules beyond EOL

Motivation
----------

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/atropos/atropos.py", line 110, in atropos
    rses = parse_expression(rule.rse_expression, filter={'vo': rule.account.vo})
AttributeError: 'result' object has no attribute 'account'
```

This bug was introduced with commit ec57f5731f.

Modification
------------

Simply add `account` to the returned columns of `get_rules_beyond_eol()`.


Changed files:
lib/rucio/core/rule.py

diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
index 42f8c3ff1..701c074ee 100644
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -1628,7 +1628,8 @@ def get_rules_beyond_eol(date_check, worker_number, total_workers, session):
                           models.ReplicationRule.locked,
                           models.ReplicationRule.id,
                           models.ReplicationRule.eol_at,
-                          models.ReplicationRule.expires_at).\\
+                          models.ReplicationRule.expires_at,
+                          models.ReplicationRule.account).\\
         filter(models.ReplicationRule.eol_at < date_check)
 
     query = filter_thread_work(session=session, query=query, total_threads=total_workers, thread_id=worker_number, hash_variable='name')
Repository: rucio/rucio
Issue: `rucio --help` fails on misconfigured sites

Motivation
----------
Even on a bad installation or typos in the configuration system, `rucio --help` ought to work.

The current version of `rucio` imports the `DownloadClient` at the module level.  If there is an error in the `rucio.cfg` that causes the `DownloadClient` constructor to throw an exception, you can't get to `--help`.

Example:
```
$ rucio --help 
Traceback (most recent call last):
  File "/home/cse496/bbockelm/software/rucio-doma/bin/rucio", line 63, in <module>
    from rucio.client.downloadclient import DownloadClient
  File "/home/cse496/bbockelm/projects/rucio/lib/rucio/client/downloadclient.py", line 34, in <module>
    client = Client()
... stack trace goes on for miles ...
  File "/home/cse496/bbockelm/software/rucio-doma/lib/python2.7/site-packages/requests/adapters.py", line 412, in send
    self.cert_verify(conn, request.url, verify, cert)
  File "/home/cse496/bbockelm/software/rucio-doma/lib/python2.7/site-packages/requests/adapters.py", line 227, in cert_verify
    "invalid path: {0}".format(cert_loc))
IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ca.crt
```

In my case, the problem was caused by an invalid path in `rucio.cfg` for the CA bundle.

Modification
------------
Failures to create the clients should be swallowed until they are actually used.  Potentially we shouldn't even import the clients until needed.

Changed files:
lib/rucio/common/config.py

diff --git a/lib/rucio/common/config.py b/lib/rucio/common/config.py
index 19cadfc5a..d3afacd04 100644
--- a/lib/rucio/common/config.py
+++ b/lib/rucio/common/config.py
@@ -54,7 +54,7 @@ def config_get(section, option, raise_exception=True, default=None, clean_cached
     global __CONFIG
     try:
         return get_config().get(section, option)
-    except (ConfigParser.NoOptionError, ConfigParser.NoSectionError) as err:
+    except (ConfigParser.NoOptionError, ConfigParser.NoSectionError, RuntimeError) as err:
         if raise_exception and default is None:
             raise err
         if clean_cached:
Repository: rucio/rucio
Issue: srm+https not correctly stripped (related #4513)

It is necessary to changes `srm+https://...` scheme to `srm://...` before passing this URL to gfal, but current code remove too many characters and as a result we get `srm//...` without `:`

https://github.com/rucio/rucio/blob/f1159fbf1e6002715a77b613d35d037c55082803/lib/rucio/core/transfer.py#L685
https://github.com/rucio/rucio/blob/f1159fbf1e6002715a77b613d35d037c55082803/lib/rucio/core/transfer.py#L701


Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 520edd4e5..027195be1 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -682,7 +682,7 @@ def __rewrite_source_url(source_url, source_sign_url, dest_sign_url, source_sche
             source_url = source_sign_url_map[source_sign_url] + source_url[5:]
 
     if source_url[:12] == 'srm+https://':
-        source_url = 'srm' + source_url[10:]
+        source_url = 'srm' + source_url[9:]
     return source_url
 
 
@@ -698,7 +698,7 @@ def __rewrite_dest_url(dest_url, dest_sign_url, dest_scheme):
         dest_url = re.sub('https', 's3s', dest_url)
 
     if dest_url[:12] == 'srm+https://':
-        dest_url = 'srm' + dest_url[10:]
+        dest_url = 'srm' + dest_url[9:]
     return dest_url
Repository: rucio/rucio
Issue: Exporter: Broken distance option in Flask API

Motivation
----------
The distance option for the in Flask API exporter is broken (but worked fine with webpy)






Changed files:
lib/rucio/tests/test_import_export.py, lib/rucio/web/rest/flaskapi/v1/export.py

diff --git a/lib/rucio/tests/test_import_export.py b/lib/rucio/tests/test_import_export.py
index 7626db3de..06f9bc63b 100644
--- a/lib/rucio/tests/test_import_export.py
+++ b/lib/rucio/tests/test_import_export.py
@@ -1252,6 +1252,8 @@ def test_export_client(vo, distances_data):
         }
     }
     assert parse_response(render_json(**distances_cmp)) == data['distances']
+    data = export_client.export_data(distance=False)
+    assert 'distances' not in data
 
 
 @pytest.mark.noparallel(reason='modifies distance on pre-defined RSE')
diff --git a/lib/rucio/web/rest/flaskapi/v1/export.py b/lib/rucio/web/rest/flaskapi/v1/export.py
index 2ec52e1bd..ead96499c 100644
--- a/lib/rucio/web/rest/flaskapi/v1/export.py
+++ b/lib/rucio/web/rest/flaskapi/v1/export.py
@@ -56,7 +56,7 @@ class Export(ErrorHandlingMethodView):
         :status 406: Not Acceptable
         :returns: dictionary with rucio data
         """
-        distance = request.args.get('distance', default=True)
+        distance = request.args.get('distance', default='True') == 'True'
         return Response(render_json(**export_data(issuer=request.environ.get('issuer'), distance=distance, vo=request.environ.get('vo'))), content_type='application/json')
Repository: rucio/rucio
Issue: Incorrect documentation on passing the dynamic parameter to /dids/(scope)/(name)

Motivation
----------

Following the documentation, adding `?dynamic` to the URL has no effect. Instead, one has to use something like `?dynamic=anything`. Possibly also applies to `/dids/(scope)/(name)/dids`.

Modification
------------

Either address the documentation or investigate why the parameter isn’t properly propagated.

Changed files:
lib/rucio/web/rest/flaskapi/v1/dids.py

diff --git a/lib/rucio/web/rest/flaskapi/v1/dids.py b/lib/rucio/web/rest/flaskapi/v1/dids.py
index 5274a6fdc..aa36b4196 100644
--- a/lib/rucio/web/rest/flaskapi/v1/dids.py
+++ b/lib/rucio/web/rest/flaskapi/v1/dids.py
@@ -498,7 +498,7 @@ class Attachment(ErrorHandlingMethodView):
 
         .. sourcecode:: http
 
-            GET /dids/scope1/dataset1?dynamic HTTP/1.1
+            GET /dids/scope1/dataset1 HTTP/1.1
             Host: rucio.cern.ch
 
         **Example response**:
@@ -513,7 +513,6 @@ class Attachment(ErrorHandlingMethodView):
              "bytes": 234, "length": 3, "account": "jdoe", "open": True,
              "monotonic": False, "expired_at": null}
 
-        :query dynamic: Flag to dynamically calculate size for open DIDs
         :resheader Content-Type: application/x-json-stream
         :status 200: DID found
         :status 401: Invalid Auth Token
Repository: rucio/rucio
Issue: Fix default rse attribute reg-exp

Motivation
----------
The `default_rse_attribute` regular expression was made hard-coded (instead of pulled in from the schema) based on concerns that the expression parser could be impacted by faulty attributes. However, the hard-coded regexp was defined to strict, causing issues for Belle II.

Modification
------------
Change regexp to `([A-Za-z0-9]+([_-][A-Za-z0-9]+)*)`


Changed files:
lib/rucio/core/rse_expression_parser.py

diff --git a/lib/rucio/core/rse_expression_parser.py b/lib/rucio/core/rse_expression_parser.py
index de3a97d86..6eb86345f 100644
--- a/lib/rucio/core/rse_expression_parser.py
+++ b/lib/rucio/core/rse_expression_parser.py
@@ -39,7 +39,7 @@ from rucio.core.rse import list_rses, get_rses_with_attribute, get_rse_attribute
 from rucio.db.sqla.session import transactional_session
 
 
-DEFAULT_RSE_ATTRIBUTE = r'([A-Z0-9]+([_-][A-Za-z0-9]+)*)'
+DEFAULT_RSE_ATTRIBUTE = r'([A-Za-z0-9]+([_-][A-Za-z0-9]+)*)'
 RSE_ATTRIBUTE = r'([A-Za-z0-9\\._-]+[=<>][A-Za-z0-9_-]+)'
 PRIMITIVE = r'(\\(*(%s|%s|%s)\\)*)' % (RSE_ATTRIBUTE, DEFAULT_RSE_ATTRIBUTE, r'\\*')
 UNION = r'(\\|%s)' % (PRIMITIVE)
Repository: rucio/rucio
Issue: Python 2 compatibility broke with 1.26 release

Motivation
----------
For some reason (probably due to a testing dependency) this did not show up in our tests:

```
$ rucio --version
Traceback (most recent call last):
  File "/opt/rucio/bin/rucio", line 71, in <module>
    from rucio.client import Client
  File "/opt/rucio/lib/rucio/client/__init__.py", line 20, in <module>
    from .client import Client  # noqa: F401
  File "/opt/rucio/lib/rucio/client/client.py", line 30, in <module>
    from rucio.client.accountclient import AccountClient
  File "/opt/rucio/lib/rucio/client/accountclient.py", line 38, in <module>
    from rucio.client.baseclient import BaseClient
  File "/opt/rucio/lib/rucio/client/baseclient.py", line 69, in <module>
    from rucio.common.extra import import_extras
  File "/opt/rucio/lib/rucio/common/extra.py", line 20, in <module>
    from typing import TYPE_CHECKING
ImportError: No module named typing
```


Modification
------------
Remove any typing imports for 1.26, add dependency to `typing` for 1.27 for python < 3.5



Changed files:
lib/rucio/common/extra.py

diff --git a/lib/rucio/common/extra.py b/lib/rucio/common/extra.py
index 9fb11d88f..45dc344fc 100644
--- a/lib/rucio/common/extra.py
+++ b/lib/rucio/common/extra.py
@@ -17,14 +17,9 @@
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2021
 
 import importlib
-from typing import TYPE_CHECKING
-
-if TYPE_CHECKING:
-    from typing import List, Dict, Any
 
 
 def import_extras(module_list):
-    # type: (List[str]) -> Dict[str, Any]
     out = dict()
     for mod in module_list:
         out[mod] = None
Repository: rucio/rucio
Issue: Fix setup_webui script

Motivation
----------
Script has a wrong import, needs to be fixed.



Changed files:
setup_webui.py

diff --git a/setup_webui.py b/setup_webui.py
index ef91603cb..65afdd410 100644
--- a/setup_webui.py
+++ b/setup_webui.py
@@ -35,7 +35,7 @@ except ImportError:
     from setuputil import get_rucio_version
 
 name = 'rucio-webui'
-packages = ['rucio', 'rucio.web', 'rucio.web.ui', 'rucio.web.ui.flask', 'rucio.web.flask.common']
+packages = ['rucio', 'rucio.web', 'rucio.web.ui', 'rucio.web.ui.flask', 'rucio.web.ui.flask.common']
 data_files = []
 description = "Rucio WebUI Package"
Repository: rucio/rucio
Issue: Transfers without sources stall the conveyor submitter

Motivation
----------
Transfers without sources are never removed from the transfer queue and stall the submitter.

Modification
------------
```
if req_id not in reqs_no_source:
    reqs_no_source.append(req_id)
```
needs to be added before https://github.com/rucio/rucio/blob/510adbd9be99c7bc3111cc88a68fb889b222dd40/lib/rucio/core/transfer.py#L425

Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index ee77d3f95..dd88dc533 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -6,7 +6,7 @@
 # http://www.apache.org/licenses/LICENSE-2.0
 #
 # Authors:
-# - Martin Barisits, <martin.barisits@cern.ch>, 2017-2018
+# - Martin Barisits, <martin.barisits@cern.ch>, 2017-2021
 # - Mario Lassnig, <mario.lassnig@cern.ch>, 2017-2018
 # - Cedric Serfon, <cedric.serfon@cern.ch>, 2018-2019
 # - Hannes Hansen, <hannes.jakob.hansen@cern.ch>, 2018
@@ -422,6 +422,8 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
 
         # source_rse_id will be None if no source replicas
         # rse will be None if rse is staging area
+        if req_id not in reqs_no_source:
+            reqs_no_source.append(req_id)
         if source_rse_id is None or rse is None:
             continue
Repository: rucio/rucio
Issue: Typo in oracle version check

Motivation
----------
The check for the oracle version needed for the generic meta data has a typo in calling the connection function [here](https://github.com/rucio/rucio/blob/master/lib/rucio/core/did.py#L1291)

Modification
------------
```
session.connection().connection
```
instead of 
```
session.connection.connection
```


Changed files:
lib/rucio/core/did.py

diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
index 5e46a7887..94ef52408 100644
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -1288,7 +1288,7 @@ def get_did_meta(scope, name, session=None):
     :param name: the name of the did
     """
     if session.bind.dialect.name == 'oracle':
-        oracle_version = int(session.connection.connection.version.split('.')[0])
+        oracle_version = int(session.connection().connection.version.split('.')[0])
         if oracle_version < 12:
             raise NotImplementedError
Repository: rucio/rucio
Issue: Cannot declare bad PFNs

Motivation
----------

We have identified another regression from the recent SQLAlchemy changes. In [this](https://github.com/rucio/rucio/blob/master/lib/rucio/core/permission/atlas.py#L1142) part of the code, `state` can be `BAD`, but `str(BadPFNStatus.BAD)` evaluates to `'BadPFNStatus.BAD'`.

Modification
------------

Use `BadPFNStatus.*.name` instead.


Changed files:
lib/rucio/core/permission/atlas.py

diff --git a/lib/rucio/core/permission/atlas.py b/lib/rucio/core/permission/atlas.py
index dc08496da..003ee7a83 100644
--- a/lib/rucio/core/permission/atlas.py
+++ b/lib/rucio/core/permission/atlas.py
@@ -1139,10 +1139,10 @@ def perm_add_bad_pfns(issuer, kwargs):
     :param kwargs: List of arguments for the action.
     :returns: True if account is allowed, otherwise False
     """
-    if kwargs['state'] in [str(BadPFNStatus.BAD), str(BadPFNStatus.TEMPORARY_UNAVAILABLE)]:
+    if kwargs['state'] in [BadPFNStatus.BAD.name, BadPFNStatus.TEMPORARY_UNAVAILABLE.name]:
         is_cloud_admin = bool([acc_attr for acc_attr in list_account_attributes(account=issuer) if (acc_attr['key'].startswith('cloud-')) and (acc_attr['value'] == 'admin')])
         return _is_root(issuer) or has_account_attribute(account=issuer, key='admin') or is_cloud_admin
-    elif kwargs['state'] == str(BadPFNStatus.SUSPICIOUS):
+    elif kwargs['state'] == BadPFNStatus.SUSPICIOUS.name:
         return True
     return _is_root(issuer)
Repository: rucio/rucio
Issue: rucio download wrongly resolves wildcarded DIDs

Motivation
----------
In 1.17.* the wildcard download stopped working. (Related to re-implementation of downloader)

Changed files:
lib/rucio/client/downloadclient.py

diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
index 9528d1e2c..e0278f979 100644
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -871,7 +871,7 @@ class DownloadClient:
                     new_item['type'] = dids['did_type'].upper()
                     new_item['name'] = dids['name']
                     new_item['did'] = '%s:%s' % (did_scope, dids['name'])
-                    resolved_items.append(new_item)
+                    resolved_items.append(copy.deepcopy(new_item))
             else:
                 new_item['type'] = self.client.get_did(did_scope, did_name)['type'].upper()
                 new_item['name'] = did_name
Repository: rucio/rucio
Issue: Client fails if auth type is x509_proxy and X509_USER_PROXY is not set

The client fails if the authentication type is x509_proxy and the X509_USER_PROXY environment variable is not set with a very confusing error message:
```
$ rucio whoami
2018-09-27 09:22:57,749	ERROR	coercing to Unicode: need string or buffer, module found
```


Changed files:
lib/rucio/client/baseclient.py, lib/rucio/core/lifetime_exception.py

diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
index 1174c41b4..1562a75c6 100644
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -182,7 +182,7 @@ class BaseClient(object):
                         else:
                             fname = '/tmp/x509up_u%d' % geteuid()
                             if path.exists(fname):
-                                self.creds['client_proxy'] = path
+                                self.creds['client_proxy'] = fname
                             else:
                                 raise MissingClientParameter('Cannot find a valid X509 proxy; not in %s, $X509_USER_PROXY not set, and '
                                                              '\\'x509_proxy\\' not set in the configuration file.' % fname)
diff --git a/lib/rucio/core/lifetime_exception.py b/lib/rucio/core/lifetime_exception.py
index 87f27a91e..1c97da138 100644
--- a/lib/rucio/core/lifetime_exception.py
+++ b/lib/rucio/core/lifetime_exception.py
@@ -42,19 +42,11 @@ def list_exceptions(exception_id, states, session=None):
     if states:
         state_clause = [models.LifetimeExceptions.state == state for state in states]
 
-    query = session.query(models.LifetimeExceptions.id,
-                          models.LifetimeExceptions.scope, models.LifetimeExceptions.name,
-                          models.LifetimeExceptions.did_type,
-                          models.LifetimeExceptions.account,
-                          models.LifetimeExceptions.pattern,
-                          models.LifetimeExceptions.comments,
-                          models.LifetimeExceptions.state,
-                          models.LifetimeExceptions.expires_at,
-                          models.LifetimeExceptions.created_at)
+    query = session.query(models.LifetimeExceptions)
     if state_clause != []:
         query = query.filter(or_(*state_clause))
     if exception_id:
-        query = query.filter(id=exception_id)
+        query = query.filter_by(id=exception_id)
 
     for exception in query.yield_per(5):
         yield {'id': exception.id, 'scope': exception.scope, 'name': exception.name,
Repository: rucio/rucio
Issue: Client fails if auth type is x509_proxy and X509_USER_PROXY is not set

The client fails if the authentication type is x509_proxy and the X509_USER_PROXY environment variable is not set with a very confusing error message:
```
$ rucio whoami
2018-09-27 09:22:57,749	ERROR	coercing to Unicode: need string or buffer, module found
```


Changed files:
lib/rucio/client/baseclient.py

diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
index 1174c41b4..1562a75c6 100644
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -182,7 +182,7 @@ class BaseClient(object):
                         else:
                             fname = '/tmp/x509up_u%d' % geteuid()
                             if path.exists(fname):
-                                self.creds['client_proxy'] = path
+                                self.creds['client_proxy'] = fname
                             else:
                                 raise MissingClientParameter('Cannot find a valid X509 proxy; not in %s, $X509_USER_PROXY not set, and '
                                                              '\\'x509_proxy\\' not set in the configuration file.' % fname)
Repository: rucio/rucio
Issue: Hermes2: Submission of deletion events in local time

Motivation
----------
Hermes2 submits the deletion events in local time. It needs to be changed to UTC.



Changed files:
lib/rucio/daemons/hermes/hermes2.py

diff --git a/lib/rucio/daemons/hermes/hermes2.py b/lib/rucio/daemons/hermes/hermes2.py
index 11dd1e64f..9199848a7 100644
--- a/lib/rucio/daemons/hermes/hermes2.py
+++ b/lib/rucio/daemons/hermes/hermes2.py
@@ -14,7 +14,7 @@
 # limitations under the License.
 #
 # Authors:
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2020
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2020-2021
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2020
 
@@ -354,8 +354,8 @@ def aggregate_to_influx(messages, bin_size, endpoint, prepend_str):
         elif event_type in ['deletion-failed', 'deletion-done']:
             created_at = message['created_at']
             if bin_size == '1m':
-                created_at = created_at.replace(second=0, microsecond=0)
-            created_at = int(created_at.strftime('%s')) * 1000000000
+                created_at = created_at.replace(second=0, microsecond=0, tzinfo=datetime.timezone.utc).timestamp()
+            created_at = int(created_at) * 1000000000
             created_at += microsecond
             if created_at not in bins:
                 bins[created_at] = {}
Repository: rucio/rucio
Issue: core/rule.py reencodes meta over and over again

Motivation
----------

core/replica.py reencodes meta to JSON over and over again inside a loop on DIDs. Eventually this makes a string so big it causes an exception in oracle. It's not clear if this also corrupts the intended metadata

Modification
------------

Use a dedicated variable for the results of the JSON encoding



Changed files:
lib/rucio/core/rule.py

diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
index d3100511c..5fdb7609d 100644
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -30,7 +30,7 @@
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
-# - Eric Vaandering <ewv@fnal.gov>, 2020
+# - Eric Vaandering <ewv@fnal.gov>, 2020-2021
 
 from __future__ import division
 
@@ -231,11 +231,12 @@ def add_rule(dids, account, copies, rse_expression, grouping, weight, lifetime,
 
             # 4. Create the replication rule
             with record_timer_block('rule.add_rule.create_rule'):
+                meta_json = None
                 if meta is not None:
                     try:
-                        meta = json.dumps(meta)
+                        meta_json = json.dumps(meta)
                     except Exception:
-                        meta = None
+                        meta_json = None
 
                 new_rule = models.ReplicationRule(account=account,
                                                   name=elem['name'],
@@ -257,7 +258,7 @@ def add_rule(dids, account, copies, rse_expression, grouping, weight, lifetime,
                                                   ignore_account_limit=ignore_account_limit,
                                                   priority=priority,
                                                   split_container=split_container,
-                                                  meta=meta,
+                                                  meta=meta_json,
                                                   eol_at=eol_at)
                 try:
                     new_rule.save(session=session)
Repository: rucio/rucio
Issue: imp / importlib incompatibility

Motivation
----------

The fallback load of `imp` is not good enough, this would only start working on Python 3.9.

Changed files:
lib/rucio/client/baseclient.py, lib/rucio/common/utils.py

diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
index 80cfa576b..f90d67c20 100644
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2012-2020 CERN
+# Copyright 2012-2021 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,7 +17,7 @@
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2012-2020
 # - Vincent Garonne <vincent.garonne@cern.ch>, 2012-2018
 # - Yun-Pin Sun <winter0128@gmail.com>, 2013
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2020
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2021
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2014-2020
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2015
 # - Joaquín Bogado <jbogado@linti.unlp.edu.ar>, 2015-2018
@@ -46,7 +46,14 @@ try:
     import importlib
     importlib.util.find_spec('')
 except AttributeError:
+    pass
+
+# Fallback until Python <3.9
+# Must be removed with Python 3.9
+try:
     import imp
+except AttributeError:
+    pass
 
 import os
 import random
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
index 7047046ed..b03bb2f45 100644
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -39,7 +39,14 @@ try:
     import importlib
     importlib.util.find_spec('')
 except AttributeError:
+    pass
+
+# Fallback until Python <3.9
+# Must be removed with Python 3.9
+try:
     import imp
+except AttributeError:
+    pass
 
 import base64
 import copy
Repository: rucio/rucio
Issue: Bug in the filter_thread_work algorithm for Postgres backend

Motivation
----------
The partitioning algorithm sometimes fails with `Details: (psycopg2.errors.NumericValueOutOfRange) integer out of range` for Postgres backend, e.g. : 
```
rucio=> select abs(('x'||md5(requests.id::text))::bit(64)::int) from requests limit 1;
ERROR:  integer out of range
rucio=> select abs(('x'||md5(requests.id::text))::bit(64)::bigint) from requests limit 1;
         abs         
---------------------
 9096650785057471382
(1 row)
```





Changed files:
lib/rucio/db/sqla/__init__.py

diff --git a/lib/rucio/db/sqla/__init__.py b/lib/rucio/db/sqla/__init__.py
index 0bb6b89e2..d4b8aebec 100644
--- a/lib/rucio/db/sqla/__init__.py
+++ b/lib/rucio/db/sqla/__init__.py
@@ -1,8 +1,24 @@
-# Copyright European Organization for Nuclear Research (CERN)
+# -*- coding: utf-8 -*-
+# Copyright 2013-2022 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
-# You may not use this file except in compliance with the License.
-# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Authors:
+# - Vincent Garonne <vincent.garonne@cern.ch>, 2013-2015
+# - Brandon White <bjwhite@fnal.gov>, 2019
+# - Martin Barisits <martin.barisits@cern.ch>, 2020
+# - Radu Carpa <radu.carpa@cern.ch>, 2021
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2022
 
 from sqlalchemy.sql.expression import bindparam, text
 
@@ -23,7 +39,7 @@ def filter_thread_work(session, query, total_threads, thread_id, hash_variable=N
                 query = query.filter(text('mod(md5(%s), %s) = %s' % (hash_variable, total_threads, thread_id)))
         elif session.bind.dialect.name == 'postgresql':
             if not hash_variable:
-                query = query.filter(text('mod(abs((\\'x\\'||md5(id::text))::bit(32)::int), %s) = %s' % (total_threads, thread_id)))
+                query = query.filter(text('mod(abs((\\'x\\'||md5(id::text))::bit(32)::bigint), %s) = %s' % (total_threads, thread_id)))
             else:
-                query = query.filter(text('mod(abs((\\'x\\'||md5(%s::text))::bit(32)::int), %s) = %s' % (hash_variable, total_threads, thread_id)))
+                query = query.filter(text('mod(abs((\\'x\\'||md5(%s::text))::bit(32)::bigint), %s) = %s' % (hash_variable, total_threads, thread_id)))
     return query
Repository: rucio/rucio
Issue: Upload broken for cloud resources

Motivation
----------
There was a regression in the uploadclient in 1.27.6 that prevent upload to cloud resources




Changed files:
lib/rucio/client/uploadclient.py

diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
index 0e4c73ee4..542bae94f 100644
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -17,7 +17,7 @@
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2012-2015
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2012-2021
 # - Vincent Garonne <vincent.garonne@cern.ch>, 2012-2018
-# - Martin Barisits <martin.barisits@cern.ch>, 2017-2021
+# - Martin Barisits <martin.barisits@cern.ch>, 2017-2022
 # - Tobias Wegner <twegner@cern.ch>, 2018
 # - Joaquín Bogado <jbogado@linti.unlp.edu.ar>, 2018
 # - Nicolo Magini <nicolo.magini@cern.ch>, 2018
@@ -41,6 +41,7 @@
 # - David Población Criado <david.poblacion.criado@cern.ch>, 2021
 # - Joel Dierkes <joel.dierkes@cern.ch>, 2021
 # - Nicholas Smith <nick.smith@cern.ch>, 2021
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2022
 
 import copy
 import json
@@ -594,7 +595,7 @@ class UploadClient:
         logger = self.logger
 
         # Construct protocol for write operation.
-        # IMPORTANT: All upload stat() checks are always done with the write_protocol
+        # IMPORTANT: All upload stat() checks are always done with the write_protocol EXCEPT for cloud resources (signed URL for write cannot be used for read)
         protocol_write = self._create_protocol(rse_settings, 'write', force_scheme=force_scheme, domain=domain, impl=impl)
 
         base_name = lfn.get('filename', lfn['name'])
@@ -625,8 +626,15 @@ class UploadClient:
         pfn_tmp = '%s.rucio.upload' % pfn if protocol_write.renaming else pfn
 
         # Either DID eixsts or not register_after_upload
-        if protocol_write.overwrite is False and delete_existing is False and protocol_write.exists(pfn):
-            raise FileReplicaAlreadyExists('File %s in scope %s already exists on storage as PFN %s' % (name, scope, pfn))  # wrong exception ?
+        if protocol_write.overwrite is False and delete_existing is False:
+            if sign_service:
+                # Construct protocol for read ONLY for cloud resources and get signed URL for GET
+                protocol_read = self._create_protocol(rse_settings, 'read', domain=domain, impl=impl)
+                readpfn = self.client.get_signed_url(rse_settings['rse'], sign_service, 'read', pfn)    # NOQA pylint: disable=undefined-variable
+                if protocol_read.exists(readpfn):
+                    raise FileReplicaAlreadyExists('File %s in scope %s already exists on storage as PFN %s' % (name, scope, pfn))  # wrong exception ?
+            if not sign_service and protocol_write.exists(pfn):
+                raise FileReplicaAlreadyExists('File %s in scope %s already exists on storage as PFN %s' % (name, scope, pfn))  # wrong exception ?
 
         # Removing tmp from earlier attempts
         if protocol_write.exists(pfn_tmp):
Repository: rucio/rucio
Issue: Enforce timeout on VP calls

Motivation
----------
Not all queries to the external VP service have a timeout enforced which can lead to several session issues.

Modification
------------
Include a timeout.



Changed files:
lib/rucio/core/replica.py

diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
index 24e749426..02a609f2e 100644
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -1028,7 +1028,7 @@ def get_multi_cache_prefix(cache_site, filename, logger=logging.log):
     x_caches = REGION.get('CacheSites')
     if x_caches is NO_VALUE:
         try:
-            response = requests.get('{}/serverRanges'.format(vp_endpoint), verify=False)
+            response = requests.get('{}/serverRanges'.format(vp_endpoint), timeout=1, verify=False)
             if response.ok:
                 x_caches = response.json()
                 REGION.set('CacheSites', x_caches)
Repository: rucio/rucio
Issue: Preparer bulk size argument not used

Motivation
----------
One function expects the "limit" argument, while we set the "bulk" argument (collected into the kwargs and ignored)

Changed files:
lib/rucio/daemons/conveyor/preparer.py

diff --git a/lib/rucio/daemons/conveyor/preparer.py b/lib/rucio/daemons/conveyor/preparer.py
index 6822e4720..f95e75640 100644
--- a/lib/rucio/daemons/conveyor/preparer.py
+++ b/lib/rucio/daemons/conveyor/preparer.py
@@ -106,7 +106,7 @@ def preparer(once, sleep_time, bulk, partition_wait_time=10):
     )
 
 
-def run_once(bulk: int = 100, heartbeat_handler: "Optional[HeartbeatHandler]" = None, limit: "Optional[int]" = None, session: "Optional[Session]" = None, **kwargs) -> bool:
+def run_once(bulk: int = 100, heartbeat_handler: "Optional[HeartbeatHandler]" = None, session: "Optional[Session]" = None, **kwargs) -> bool:
     if heartbeat_handler:
         worker_number, total_workers, logger = heartbeat_handler.live()
     else:
@@ -118,7 +118,7 @@ def run_once(bulk: int = 100, heartbeat_handler: "Optional[HeartbeatHandler]" =
         req_sources = list_transfer_requests_and_source_replicas(
             total_workers=total_workers,
             worker_number=worker_number,
-            limit=limit,
+            limit=bulk,
             request_state=RequestState.PREPARING,
             session=session
         )
@@ -129,7 +129,7 @@ def run_once(bulk: int = 100, heartbeat_handler: "Optional[HeartbeatHandler]" =
             transfertool_filter = get_transfertool_filter(lambda rse_id: get_supported_transfertools(rse_id=rse_id, session=session))
             requests = reduce_requests(req_sources, [rse_lookup_filter, sort_requests_minimum_distance, transfertool_filter], logger=logger)
             count = preparer_update_requests(requests, session=session)
-            updated_msg = f'updated {count}/{limit} requests'
+            updated_msg = f'updated {count}/{bulk} requests'
     except RucioException:
         logger(logging.ERROR, 'errored with a RucioException, retrying later', exc_info=True)
         count = 0