internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
poller fails for the globus transfertool 
Motivation
----------

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/poller.py", line 265, in _poll_transfers
    logger(logging.INFO, 'Polling %i transfers against %s with timeout %s' % (len(transfers_by_eid), transfertool_obj, timeout))
TypeError: __str__ returned non-string (type NoneType)

```

also : 
```
2022-03-28 08:43:49,293	root	1	ERROR	Request ccb7e7ac00c04ce395bc6c91a055ee70 should not be updated to 'Failed' or 'Done' without external transfer_id

```


Discussion:

Resolving patch:
diff --git a/lib/rucio/transfertool/globus.py b/lib/rucio/transfertool/globus.py
--- a/lib/rucio/transfertool/globus.py
+++ b/lib/rucio/transfertool/globus.py
@@ -57,10 +57,11 @@ def bulk_group_transfers(transfer_paths, policy='single', group_bulk=200):
 class GlobusTransferStatusReport(TransferStatusReport):
 
     supported_db_fields = [
-        'state'
+        'state',
+        'external_id',
     ]
 
-    def __init__(self, request_id, globus_response):
+    def __init__(self, request_id, external_id, globus_response):
         super().__init__(request_id)
 
         if globus_response == 'FAILED':
@@ -71,6 +72,9 @@ def __init__(self, request_id, globus_response):
             new_state = RequestState.SUBMITTED
 
         self.state = new_state
+        self.external_id = None
+        if new_state in [RequestState.FAILED, RequestState.DONE]:
+            self.external_id = external_id
 
     def initialize(self, session, logger=logging.log):
         pass
@@ -89,6 +93,8 @@ def __init__(self, external_host, logger=logging.log, group_bulk=200, group_poli
 
         :param external_host:   The external host where the transfertool API is running
         """
+        if not external_host:
+            external_host = 'Globus Online Transfertool'
         super().__init__(external_host, logger)
         self.group_bulk = group_bulk
         self.group_policy = group_policy
@@ -191,7 +197,7 @@ def bulk_query(self, requests_by_eid, timeout=None):
         response = {}
         for transfer_id, requests in requests_by_eid.items():
             for request_id in requests:
-                response.setdefault(transfer_id, {})[request_id] = GlobusTransferStatusReport(request_id, job_responses[transfer_id])
+                response.setdefault(transfer_id, {})[request_id] = GlobusTransferStatusReport(request_id, transfer_id, job_responses[transfer_id])
         return response
 
     def bulk_update(self, resps, request_ids):


Test patch:
Repository: rucio/rucio

Issue:
Incorrect number of format arguments in one submitter log
Motivation
----------

```
conveyor-submitter[37/112]: Failed to register transfer state with error
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/common.py", line 249, in _submit_transfers
    external_id=eid, submitted_at=datetime.datetime.utcnow(), logger=logger)
  File "/usr/local/lib/python3.6/site-packages/rucio/db/sqla/session.py", line 374, in new_funct
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rucio/core/transfer.py", line 565, in set_transfers_state
    raise RucioException("Failed to set requests %s tansfer %s: request doesn't exist or is not in SUBMITTING state" % rws)
TypeError: not enough arguments for format string
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -459,7 +459,7 @@ def set_transfers_state(transfers, state, submitted_at, external_host, external_
                                       synchronize_session=False)
 
             if rowcount == 0:
-                raise RucioException("Failed to set requests %s tansfer %s: request doesn't exist or is not in SUBMITTING state" % rws)
+                raise RucioException("%s: failed to set transfer state: request doesn't exist or is not in SUBMITTING state" % rws)
 
             msg = {'request-id': rws.request_id,
                    'request-type': rws.request_type,


Test patch:
Repository: rucio/rucio

Issue:
Incorrect number of format arguments in one submitter log
Motivation
----------

```
conveyor-submitter[37/112]: Failed to register transfer state with error
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/common.py", line 249, in _submit_transfers
    external_id=eid, submitted_at=datetime.datetime.utcnow(), logger=logger)
  File "/usr/local/lib/python3.6/site-packages/rucio/db/sqla/session.py", line 374, in new_funct
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rucio/core/transfer.py", line 565, in set_transfers_state
    raise RucioException("Failed to set requests %s tansfer %s: request doesn't exist or is not in SUBMITTING state" % rws)
TypeError: not enough arguments for format string
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -562,7 +562,7 @@ def set_transfers_state(transfers, state, submitted_at, external_host, external_
                                       synchronize_session=False)
 
             if rowcount == 0:
-                raise RucioException("Failed to set requests %s tansfer %s: request doesn't exist or is not in SUBMITTING state" % rws)
+                raise RucioException("%s: failed to set transfer state: request doesn't exist or is not in SUBMITTING state" % rws)
 
             msg = {'request-id': rws.request_id,
                    'request-type': rws.request_type,


Test patch:
Repository: rucio/rucio

Issue:
Preparer bulk size argument not used
Motivation
----------
One function expects the "limit" argument, while we set the "bulk" argument (collected into the kwargs and ignored)


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/preparer.py b/lib/rucio/daemons/conveyor/preparer.py
--- a/lib/rucio/daemons/conveyor/preparer.py
+++ b/lib/rucio/daemons/conveyor/preparer.py
@@ -106,7 +106,7 @@ def preparer(once, sleep_time, bulk, partition_wait_time=10):
     )
 
 
-def run_once(bulk: int = 100, heartbeat_handler: "Optional[HeartbeatHandler]" = None, limit: "Optional[int]" = None, session: "Optional[Session]" = None, **kwargs) -> bool:
+def run_once(bulk: int = 100, heartbeat_handler: "Optional[HeartbeatHandler]" = None, session: "Optional[Session]" = None, **kwargs) -> bool:
     if heartbeat_handler:
         worker_number, total_workers, logger = heartbeat_handler.live()
     else:
@@ -118,7 +118,7 @@ def run_once(bulk: int = 100, heartbeat_handler: "Optional[HeartbeatHandler]" =
         req_sources = list_transfer_requests_and_source_replicas(
             total_workers=total_workers,
             worker_number=worker_number,
-            limit=limit,
+            limit=bulk,
             request_state=RequestState.PREPARING,
             session=session
         )
@@ -129,7 +129,7 @@ def run_once(bulk: int = 100, heartbeat_handler: "Optional[HeartbeatHandler]" =
             transfertool_filter = get_transfertool_filter(lambda rse_id: get_supported_transfertools(rse_id=rse_id, session=session))
             requests = reduce_requests(req_sources, [rse_lookup_filter, sort_requests_minimum_distance, transfertool_filter], logger=logger)
             count = preparer_update_requests(requests, session=session)
-            updated_msg = f'updated {count}/{limit} requests'
+            updated_msg = f'updated {count}/{bulk} requests'
     except RucioException:
         logger(logging.ERROR, 'errored with a RucioException, retrying later', exc_info=True)
         count = 0


Test patch:
Repository: rucio/rucio

Issue:
Globus transfers not being handled properly by MONIT
Motivation
----------

The protocol is not properly extracted, see:
https://monit-grafana.cern.ch/dashboard/snapshot/en9K4Tc2vSpD54ToTLr5pydfei1GSAil

Modification
------------

Not sure whether something to address on our side or MONIT’s.


Discussion:
This looks to me more on the monit side in the transformation job. I think we need to check this with them.
Submitted: RQF2010563
https://github.com/rucio/rucio/blob/abe3c77ed31b89661a7a13209d58ff3b62e2a266/lib/rucio/core/request.py#L1575
Maybe?

Resolving patch:
diff --git a/lib/rucio/core/request.py b/lib/rucio/core/request.py
--- a/lib/rucio/core/request.py
+++ b/lib/rucio/core/request.py
@@ -1571,8 +1571,12 @@ def add_monitor_message(new_state, request, additional_fields, session=None):
 
     if message['started_at'] and message['transferred_at']:
         message['duration'] = (message['transferred_at'] - message['started_at']).seconds
-    if message['dst-url']:
-        message['protocol'] = message['dst-url'].split(':')[0]
+    if not message.get('protocol'):
+        dst_url = message['dst-url']
+        if dst_url and ':' in dst_url:
+            message['protocol'] = dst_url.split(':')[0]
+        elif request.get('transfertool'):
+            message['protocol'] = request['transfertool']
     if not message.get('src-rse'):
         src_rse_id = request.get('source_rse_id', None)
         if src_rse_id:
diff --git a/lib/rucio/transfertool/globus.py b/lib/rucio/transfertool/globus.py
--- a/lib/rucio/transfertool/globus.py
+++ b/lib/rucio/transfertool/globus.py
@@ -80,7 +80,7 @@ def initialize(self, session, logger=logging.log):
         pass
 
     def get_monitor_msg_fields(self, session, logger=logging.log):
-        return {}
+        return {'protocol': 'globus'}
 
 
 class GlobusTransferTool(Transfertool):


Test patch:
Repository: rucio/rucio

Issue:
impossibility to assign a transfertool crashes the submitter
Motivation
----------

```
2022-04-06 14:23:48,863 root    1       CRITICAL        conveyor-submitter[17/61]: Exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/common.py", line 171, in run_conveyor_daemon
    must_sleep = run_once_fnc(activity=activity, heartbeat_handler=heartbeat_handler)
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/submitter.py", line 98, in run_once
    logger=logger,
  File "/usr/local/lib/python3.6/site-packages/rucio/db/sqla/session.py", line 376, in new_funct
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/common.py", line 231, in next_transfers_to_submit
    session=session,
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/common.py", line 332, in __assign_paths_to_transfertool_and_create_hops
    hops_to_submit, builder_to_use = tt_assignment[0]
IndexError: list index out of range
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -294,6 +294,10 @@ def __assign_paths_to_transfertool_and_create_hops(
         # Prioritize the paths which need less transfertool transitions.
         # Ideally, the entire path should be submitted to a single transfertool
         for transfer_path, tt_assignment in sorted(tt_assignments, key=lambda t: len(t[1])):
+            if not tt_assignment:
+                logger(logging.INFO, '%s: None of the transfertools can submit the request: %s', request_id, [c.__name__ for c in transfertool_classes])
+                continue
+
             # Set the 'transfertool' field on the intermediate hops which should be created in the database
             for sub_path, tt_builder in tt_assignment:
                 if tt_builder:


Test patch:
Repository: rucio/rucio

Issue:
Deprecation message in Paramiko
Motivation
----------
Paramiko outputs a deprecation message due to an outdated security algorithm. We do not depend on that algorithm.
https://github.com/paramiko/paramiko/pull/2039

Modification
------------
The paramiko team is aware of the problem. They opened a PR to fix it.




Discussion:
Proposed solution: Set the `cryptography` version fixed to the last working one. The only relevant change in the new version 37.0.0 is the deprecation message, which fails some of our test cases. As soon as `paramiko` does not use the deprecated feature anymore we can dump up the version.

Resolving patch:
diff --git a/lib/rucio/common/extra.py b/lib/rucio/common/extra.py
--- a/lib/rucio/common/extra.py
+++ b/lib/rucio/common/extra.py
@@ -17,6 +17,7 @@
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2021
 
 import importlib
+import warnings
 
 
 def import_extras(module_list):
@@ -24,7 +25,12 @@ def import_extras(module_list):
     for mod in module_list:
         out[mod] = None
         try:
-            out[mod] = importlib.import_module(mod)
+            with warnings.catch_warnings():
+                # TODO: remove when https://github.com/paramiko/paramiko/issues/2038 is fixed
+                warnings.filterwarnings('ignore', 'Blowfish has been deprecated', module='paramiko')
+                # TODO: deprecated python 2 and 3.6 too ...
+                warnings.filterwarnings('ignore', 'Python .* is no longer supported', module='paramiko')
+                out[mod] = importlib.import_module(mod)
         except ImportError:
             pass
     return out


Test patch:
Repository: rucio/rucio

Issue:
Update permission module for ESCAPE
Motivation
----------
The `perm_get_signed_url` permission wrongly requests `key=admin`.


Modification
------------
Change to `key=sign-gcs`




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/permission/escape.py b/lib/rucio/core/permission/escape.py
--- a/lib/rucio/core/permission/escape.py
+++ b/lib/rucio/core/permission/escape.py
@@ -1011,7 +1011,7 @@ def perm_get_signed_url(issuer, kwargs, session=None):
     :param session: The DB session to use
     :returns: True if account is allowed to call the API call, otherwise False
     """
-    return _is_root(issuer) or has_account_attribute(account=issuer, key='admin', session=session)
+    return _is_root(issuer) or has_account_attribute(account=issuer, key='sign-gcs', session=session)
 
 
 def perm_add_bad_pfns(issuer, kwargs, session=None):


Test patch:
Repository: rucio/rucio

Issue:
reaper busy-looping over rses with greedyDeletion even if nothing to delete
Motivation
----------
A recent change in reaper reworked the greedy_deletion workflow. This resulted in a regression due to this line:
https://github.com/rucio/rucio/blob/c7636d249397c4ad6ab7019d4ed1ca585d7cfaa1/lib/rucio/daemons/reaper/reaper.py#L585

Reaper is never sleeeping for greedyDeletion rses.





Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/reaper/reaper.py b/lib/rucio/daemons/reaper/reaper.py
--- a/lib/rucio/daemons/reaper/reaper.py
+++ b/lib/rucio/daemons/reaper/reaper.py
@@ -582,7 +582,7 @@ def _run_once(rses_to_process, chunk_size, greedy, scheme,
                                                                              only_delete_obsolete=only_delete_obsolete,
                                                                              session=None)
             logger(logging.DEBUG, 'list_and_mark_unlocked_replicas on %s for %s bytes in %s seconds: %s replicas', rse.name, needed_free_space, time.time() - del_start_time, len(replicas))
-            if not enable_greedy and len(replicas) < chunk_size:
+            if (len(replicas) == 0 and enable_greedy) or (len(replicas) < chunk_size and not enable_greedy):
                 logger(logging.DEBUG, 'Not enough replicas to delete on %s (%s requested vs %s returned). Will skip any new attempts on this RSE until next cycle', rse.name, chunk_size, len(replicas))
                 REGION.set('pause_deletion_%s' % rse.id, True)
                 work_remaining_by_rse[rse] = False


Test patch:
Repository: rucio/rucio

Issue:
tracer-kronos too verbose at INFO level
Motivation
----------

It outputs tens of millions of messages per day. Most of them with the following useless content: 
```
2022-05-16 13:59:17,294	root	1	INFO	kronos-file[1/4] updated 1 replica(s)
```

Modification
------------
Substitute this message with a counter.




Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/tracer/kronos.py b/lib/rucio/daemons/tracer/kronos.py
--- a/lib/rucio/daemons/tracer/kronos.py
+++ b/lib/rucio/daemons/tracer/kronos.py
@@ -294,7 +294,8 @@ def __update_atime(self):
             self.__logger(logging.ERROR, "Cannot update replicas.", exc_info=True)
             record_counter('daemons.tracer.kronos.update_error')
 
-        self.__logger(logging.INFO, 'updated %d replica(s)' % len(replicas))
+        record_counter('daemons.tracer.kronos.updated_replicas')
+        self.__logger(logging.DEBUG, 'updated %d replica(s)' % len(replicas))
 
 
 def kronos_file(thread=0, dataset_queue=None, sleep_time=60):


Test patch:
Repository: rucio/rucio

Issue:
receiver incorrectly handles multi-source transfers
Motivation
----------

In case of multi-source transfers, receiver analyzes the message even if the job didn't yet finish:
https://github.com/rucio/rucio/blob/8acc9c9b873eef1d7e3b081821665ee03b95f4a0/lib/rucio/daemons/conveyor/receiver.py#L93

If the the message concerns one failed source, this results in a transition of the transfer to a failed state :  
https://github.com/rucio/rucio/blob/8acc9c9b873eef1d7e3b081821665ee03b95f4a0/lib/rucio/transfertool/fts3.py#L523

One may be miss-leaded, me included until now, that the `t_final_transfer_state` field is related to the job_state, but it  doesn't seem to be the case on FTS side. It depends on the file_state and the job state may not yet be final: 
https://gitlab.cern.ch/fts/fts3/-/blob/a34b78420e7f06aebca4f468e39312189e9e1454/src/url-copy/LegacyReporter.cpp#L214

This means that the receiver may transition the file to a failed state even if the job is not yet completed. 





Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/receiver.py b/lib/rucio/daemons/conveyor/receiver.py
--- a/lib/rucio/daemons/conveyor/receiver.py
+++ b/lib/rucio/daemons/conveyor/receiver.py
@@ -73,9 +73,7 @@ def on_message(self, frame):
            and 'issuer' in msg['job_metadata'].keys() \
            and str(msg['job_metadata']['issuer']) == str('rucio'):
 
-            if 'job_state' in msg.keys() and (
-                    str(msg['job_state']) != str('ACTIVE')
-                    or str(msg['job_state']) == str('ACTIVE') and 'job_m_replica' in msg.keys() and (str(msg['job_m_replica']).lower() == str('true'))):
+            if 'job_state' in msg.keys() and str(msg['job_state']) != str('ACTIVE'):
                 record_counter('daemons.conveyor.receiver.message_rucio')
 
                 self._perform_request_update(msg)


Test patch:
Repository: rucio/rucio

Issue:
Check before deleting with GFAL
Motivation
----------
This is a patch CMS has been running with for years. I'd like to see it merged if possible. Basically just check that a file exists before trying to delete it.


Modification
------------

See PR



Discussion:

Resolving patch:
diff --git a/lib/rucio/rse/protocols/gfal.py b/lib/rucio/rse/protocols/gfal.py
--- a/lib/rucio/rse/protocols/gfal.py
+++ b/lib/rucio/rse/protocols/gfal.py
@@ -491,9 +491,12 @@ def __gfal2_rm(self, paths):
 
         try:
             for path in paths:
-                ret = ctx.unlink(str(path))
-                if ret:
-                    return ret
+                if self.__gfal2_exist(path) == 0:
+                    ret = ctx.unlink(str(path))
+                    if ret:
+                        return ret
+                else:
+                    raise exception.SourceNotFound
             return ret
         except gfal2.GError as error:  # pylint: disable=no-member
             if error.code == errno.ENOENT or 'No such file' in str(error):


Test patch:
Repository: rucio/rucio

Issue:
Query not properly executed
Motivation
----------

While reviewing #5340, I noticed that there’s a query execution that doesn’t look right (the query is not stored in `stmt`).

https://github.com/rucio/rucio/blob/35d7ce449cfcfb895d19b0ea8d4b77065ce8eb19/lib/rucio/core/replica.py#L2136-L2146

Side note: there’s room for improvement in the coding style.

Modification
------------





Discussion:

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -2116,7 +2116,7 @@ def __delete_replicas_without_temp_tables(rse_id, files, ignore_availability=Tru
         rowcount += result.rowcount
 
     for chunk in chunks(bad_replica_condition, 10):
-        update(models.BadReplicas).\
+        stmt = update(models.BadReplicas).\
             where(models.BadReplicas.rse_id == rse_id). \
             where(or_(*chunk)). \
             where(models.BadReplicas.state == BadFilesStatus.BAD) .\
@@ -2126,7 +2126,7 @@ def __delete_replicas_without_temp_tables(rse_id, files, ignore_availability=Tru
                 updated_at=datetime.utcnow()
         )
 
-        result = session.execute(stmt)
+        session.execute(stmt)
 
     if rowcount != len(files):
         raise exception.ReplicaNotFound("One or several replicas don't exist.")


Test patch:
Repository: rucio/rucio

Issue:
to speed up polling of FTS transfer status at file level, not to wait for the FTS job to end
Motivation
----------
Right now Rucio polls from FTS on transfer status of individual files, only when the FTS job reaches a final state. 
Since one FTS job can have O(100) files in it, this way the status update of individual files to upstream clients is 
slowed down. One such use case is in Data Carousel, ProdSys2 relies on Rucio to tell how many files are in "active"
state, the delay causes the number of real staging requests at the site level is sometimes much less than the number
shown at ProdSys2, making it hard to throttle the site staging profile. 


Modification
------------

The desired behavior is for Rucio to poll status of individual files regardless of the FTS job state, so to speed 
up the status updates of ready files for upstream processing. 



Discussion:
Looked into this one today a bit. Feels like working in a mine field. A file in a `failed` state doesn't necessarily mean that the transfer must be transitioned to a failed state (warm hello from multi-source transfers); and there is the `unused` state, which has a different meaning for multi-source; and multi-hop transfers; I think I'll manage to find a reasonable way to make the state transition, but for now it only gave me a headache. 
We want to also handle files one by one for failures. So re-open the issue. 

Resolving patch:
diff --git a/lib/rucio/transfertool/fts3.py b/lib/rucio/transfertool/fts3.py
--- a/lib/rucio/transfertool/fts3.py
+++ b/lib/rucio/transfertool/fts3.py
@@ -573,10 +573,12 @@ def initialize(self, session, logger=logging.log):
         if file_state_is_final:
             if file_state == FTS_STATE.FINISHED:
                 new_state = RequestState.DONE
-            elif job_state_is_final and file_state == FTS_STATE.FAILED \
-                    and self._is_recoverable_fts_overwrite_error(self.request(session), reason, self._file_metadata):
-                new_state = RequestState.DONE
-            elif job_state_is_final and file_state in (FTS_STATE.FAILED, FTS_STATE.CANCELED):
+            elif file_state == FTS_STATE.FAILED and not self._multi_sources:  # for multi-source transfers we must wait for the job to be in a final state
+                if self._is_recoverable_fts_overwrite_error(self.request(session), reason, self._file_metadata):
+                    new_state = RequestState.DONE
+                else:
+                    new_state = RequestState.FAILED
+            elif job_state_is_final and file_state == FTS_STATE.CANCELED:
                 new_state = RequestState.FAILED
             elif job_state_is_final and file_state == FTS_STATE.NOT_USED:
                 if job_state == FTS_STATE.FINISHED:


Test patch: