internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
Tracer Kronos extremely verbose
Motivation
----------
In atlas we collected 80 million logs from kronos in last 24 hours. This is more than all other daemons combined. 
Many of those logs are completely useless as they state 
```
kronos-file[3/4] got a resubmitted report
```
or
```
kronos-file[3/4] hit locked row, resubmitted to queue
```

Having a prometheus counter would be enough. 

Modification
------------
Reduce the verbosity in logs. 




Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/tracer/kronos.py b/lib/rucio/daemons/tracer/kronos.py
--- a/lib/rucio/daemons/tracer/kronos.py
+++ b/lib/rucio/daemons/tracer/kronos.py
@@ -88,7 +88,6 @@ def on_message(self, frame):
 
         if 'resubmitted' in frame.headers:
             record_counter('daemons.tracer.kronos.received_resubmitted')
-            self.__logger(logging.WARNING, 'got a resubmitted report')
 
         try:
             if appversion == 'dq2':
@@ -288,14 +287,12 @@ def __update_atime(self):
                         resubmit['vo'] = replica['scope'].vo
                     self.__conn.send(body=jdumps(resubmit), destination=self.__queue, headers={'appversion': 'rucio', 'resubmitted': '1'})
                     record_counter('daemons.tracer.kronos.sent_resubmitted')
-                    self.__logger(logging.WARNING, 'hit locked row, resubmitted to queue')
             record_timer('daemons.tracer.kronos.update_atime', (time() - start_time) * 1000)
         except Exception:
             self.__logger(logging.ERROR, "Cannot update replicas.", exc_info=True)
             record_counter('daemons.tracer.kronos.update_error')
 
         record_counter('daemons.tracer.kronos.updated_replicas')
-        self.__logger(logging.DEBUG, 'updated %d replica(s)' % len(replicas))
 
 
 def kronos_file(thread=0, dataset_queue=None, sleep_time=60):


Test patch:
Repository: rucio/rucio

Issue:
Cache detection code results in wrong logging level 
Motivation
----------
The cache detection code prints some logging messages.
https://github.com/rucio/rucio/blob/6af4b0115dad3eb8d84be5ce0b394742d4c4aa0c/lib/rucio/common/cache.py#L35

This initializes automatically the logger with the wrong configuration, because [basicConfig](https://docs.python.org/3.9/library/logging.html#logging.basicConfig) doesn't do anything if the root handler is already setup.

Modification
------------

That cache detection code seemed hacky in the first place, but is now resulting in multiple annoyances : https://github.com/rucio/rucio/issues/5656 for example. 

The simplest fix is to remove the warning messages. This means cache will be silently ignored. This is not different from what would have happened with the old memcached library. 



Discussion:

Resolving patch:
diff --git a/lib/rucio/common/cache.py b/lib/rucio/common/cache.py
--- a/lib/rucio/common/cache.py
+++ b/lib/rucio/common/cache.py
@@ -14,7 +14,6 @@
 # limitations under the License.
 from __future__ import absolute_import
 
-import logging
 from dogpile.cache import make_region
 
 from rucio.common.config import config_get
@@ -32,10 +31,8 @@
         _mc_client = pymemcache.Client(CACHE_URL, connect_timeout=1, timeout=1)
         _mc_client.version()
 except IOError:
-    logging.warning("Cannot connect to memcached at {}. Caching will be disabled".format(CACHE_URL))
     ENABLE_CACHING = False
 except ImportError:
-    logging.warning("Cannot import pymemcache. Caching will be disabled")
     ENABLE_CACHING = False
 finally:
     if _mc_client:


Test patch:
Repository: rucio/rucio

Issue:
config_get_list incorrectly handles empty strings
Motivation
----------
it returns `['']` rather than an empty list




Discussion:

Resolving patch:
diff --git a/lib/rucio/common/config.py b/lib/rucio/common/config.py
--- a/lib/rucio/common/config.py
+++ b/lib/rucio/common/config.py
@@ -289,6 +289,8 @@ def __convert_string_to_list(string):
 
     :returns: A list extracted from the string.
     """
+    if not string or not string.strip():
+        return []
     string = string.split(',')
     return [item.strip(' ') for item in string]
 


Test patch:
Repository: rucio/rucio

Issue:
allow to follow one thread in logs
Motivation
----------
We use a decorated logger to prepend the worker number to the log messages, but this number is not actually stable through the lifetime of the thread. 

When trying to debug an issue, it may be useful to easily filter all messages from the same thread in kibana. 

Modification
------------
Prepend some hash to the decorated logger for easy filtering 




Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/common.py b/lib/rucio/daemons/common.py
--- a/lib/rucio/daemons/common.py
+++ b/lib/rucio/daemons/common.py
@@ -14,6 +14,7 @@
 # limitations under the License.
 
 import datetime
+import hashlib
 import logging
 import os
 import socket
@@ -45,6 +46,7 @@ def __init__(self, executable, renewal_interval, logger_prefix=None):
         self.hostname = socket.getfqdn()
         self.pid = os.getpid()
         self.hb_thread = threading.current_thread()
+        self.logger_id = hashlib.sha1(f'{self.hostname}:{self.pid}:{self.hb_thread}'.encode('utf-8')).hexdigest()[:7]
 
         self.logger = None
         self.last_heart_beat = None
@@ -75,7 +77,7 @@ def live(self, force_renew=False, payload=None):
             else:
                 self.last_heart_beat = heartbeat.live(self.executable, self.hostname, self.pid, self.hb_thread, payload=payload)
 
-            prefix = '%s[%i/%i]: ' % (self.logger_prefix, self.last_heart_beat['assign_thread'], self.last_heart_beat['nr_threads'])
+            prefix = '%s[%s:%i/%i]: ' % (self.logger_prefix, self.logger_id, self.last_heart_beat['assign_thread'], self.last_heart_beat['nr_threads'])
             self.logger = formatted_logger(logging.log, prefix + '%s')
 
             if not self.last_time:


Test patch:
Repository: rucio/rucio

Issue:
exception in temporary table undertaker
Motivation
----------
```
 undertaker[8f12bab:3/23]: Got database error Database exception. Details: (cx_Oracle.IntegrityError) ORA-00001: unique constraint (ATLAS_RUCIO.TEMPORARY_SCOPE_NAME_1_PK) violated [SQL: INSERT INTO atlas_rucio."TEMPORARY_SCOPE_NAME_1" (scope, name) SELECT atlas_rucio.contents.child_scope, atlas_rucio.contents.child_name  FROM atlas_rucio."TEMPORARY_SCOPE_NAME_0" JOIN atlas_rucio.contents ON atlas_rucio.contents.scope = atlas_rucio."TEMPORARY_SCOPE_NAME_0".scope AND atlas_rucio.contents.name = atlas_rucio."TEMPORARY_SCOPE_NAME_0".name  WHERE atlas_rucio.contents.child_type = :child_type_1] [parameters: {'child_type_1': 'F'}] (Background on this error at: https://sqlalche.me/e/14/gkpj).
```




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -1394,6 +1394,7 @@ def _delete_dids(
                      models.DataIdentifierAssociation.name == temp_table.name)
             ).where(
                 models.DataIdentifierAssociation.child_type == DIDType.FILE
+            ).distinct(
             )
         )
         session.execute(stmt)


Test patch:
Repository: rucio/rucio

Issue:
Failed multi-source transfers not transitioned to failed state
Motivation
----------

this condition is incomplete https://github.com/rucio/rucio/blob/5804712c7eb187ffb1fcfc3bbf49d89c02c0aeff/lib/rucio/transfertool/fts3.py#L576





Discussion:

Resolving patch:
diff --git a/lib/rucio/transfertool/fts3.py b/lib/rucio/transfertool/fts3.py
--- a/lib/rucio/transfertool/fts3.py
+++ b/lib/rucio/transfertool/fts3.py
@@ -573,7 +573,8 @@ def initialize(self, session, logger=logging.log):
         if file_state_is_final:
             if file_state == FTS_STATE.FINISHED:
                 new_state = RequestState.DONE
-            elif file_state == FTS_STATE.FAILED and not self._multi_sources:  # for multi-source transfers we must wait for the job to be in a final state
+            elif file_state == FTS_STATE.FAILED and job_state == FTS_STATE.FAILED or \
+                    file_state == FTS_STATE.FAILED and not self._multi_sources:  # for multi-source transfers we must wait for the job to be in a final state
                 if self._is_recoverable_fts_overwrite_error(self.request(session), reason, self._file_metadata):
                     new_state = RequestState.DONE
                 else:


Test patch:
Repository: rucio/rucio

Issue:
flask imported in clients
Motivation
----------
```
Traceback (most recent call last):
  File "/var/home/dchristidis/.local/bin/rucio", line 52, in <module>
    from rucio.rse.protocols.protocol import RSEProtocol
  File "/var/home/dchristidis/.local/lib/python3.10/site-packages/rucio/rse/__init__.py", line 20, in <module>
    from rucio.rse import rsemanager
  File "/var/home/dchristidis/.local/lib/python3.10/site-packages/rucio/rse/rsemanager.py", line 26, in <module>
    from rucio.common.logging import formatted_logger
  File "/var/home/dchristidis/.local/lib/python3.10/site-packages/rucio/common/logging.py", line 25, in <module>
    from flask import has_request_context, request
ModuleNotFoundError: No module named 'flask'
```





Discussion:

Resolving patch:
diff --git a/lib/rucio/common/logging.py b/lib/rucio/common/logging.py
--- a/lib/rucio/common/logging.py
+++ b/lib/rucio/common/logging.py
@@ -22,8 +22,6 @@
 from traceback import format_tb
 from typing import TYPE_CHECKING
 
-from flask import has_request_context, request
-
 from rucio.common.config import config_get, config_get_bool
 
 if TYPE_CHECKING:
@@ -127,6 +125,10 @@ def _get_request_data(request_path: "Sequence[str]") -> "Callable[[LogDataSource
     in the flask request object at request_path
     """
 
+    # The import fails if imported inside a client due to rsemanager.
+    # TODO: move to top of file once we got rid of/refactored rsemanager
+    from flask import has_request_context, request
+
     def _request_data_formatter(record_formatter: "LogDataSource", record: "LogRecord") -> "Generator[Tuple[str, Optional[Any]]]":
         value = None
         if has_request_context() and request_path:


Test patch:
Repository: rucio/rucio

Issue:
receiver doesn't correctly handle intermediate hops on fts 3.12
Motivation
----------
The last major FTS release started to correctly fill "job_state" for those messages as "ACTIVE". This is a behavioral change which makes rucio skip those messages. 

https://github.com/rucio/rucio/blob/5e911923990cc9c3caa0afbfec919c3ea4121edc/lib/rucio/daemons/conveyor/receiver.py#L74




Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/receiver.py b/lib/rucio/daemons/conveyor/receiver.py
--- a/lib/rucio/daemons/conveyor/receiver.py
+++ b/lib/rucio/daemons/conveyor/receiver.py
@@ -71,7 +71,7 @@ def on_message(self, frame):
            and 'issuer' in msg['job_metadata'].keys() \
            and str(msg['job_metadata']['issuer']) == str('rucio'):
 
-            if 'job_state' in msg.keys() and str(msg['job_state']) != str('ACTIVE'):
+            if 'job_state' in msg.keys() and (str(msg['job_state']) != str('ACTIVE') or msg.get('job_multihop', False) is True):
                 record_counter('daemons.conveyor.receiver.message_rucio')
 
                 self._perform_request_update(msg)


Test patch:
Repository: rucio/rucio

Issue:
webdav module: TLS version
Motivation
----------
In module lib/rucio/rse/protocols/webdav.py, class TLSv1HttpAdapter,
ssl version is explicitly set like this
    ssl_version=ssl.PROTOCOL_TLSv1

This seems to mean TLS version 1.0 which is deprecated and considered insecure. 

Modification
------------
Version 1.2 or higher if available should be preferred, such as
    ssl_version=ssl.PROTOCOL_TLSv1_2




Discussion:
Hi, yes, you seem correct, however when looking at the documentation it says:
```
Deprecated since version 3.6: OpenSSL has deprecated all version specific protocols.
```
Is the correct way now simply not to set a ssl_version at all?
Hi, yes it is probably better not to set it explicitly!
I will test.
Functionally fixed, code comments/method name however still say TLSv1 - needs to be fixed

Resolving patch:
diff --git a/lib/rucio/rse/protocols/webdav.py b/lib/rucio/rse/protocols/webdav.py
--- a/lib/rucio/rse/protocols/webdav.py
+++ b/lib/rucio/rse/protocols/webdav.py
@@ -29,9 +29,9 @@
 from rucio.rse.protocols import protocol
 
 
-class TLSv1HttpAdapter(HTTPAdapter):
+class TLSHTTPAdapter(HTTPAdapter):
     '''
-    Class to force the SSL protocol to TLSv1
+    Class to force the SSL protocol to latest TLS
     '''
     def init_poolmanager(self, connections, maxsize, block=False):
         self.poolmanager = PoolManager(num_pools=connections,
@@ -198,7 +198,7 @@ def connect(self, credentials={}):
         except KeyError:
             self.timeout = 300
         self.session = requests.Session()
-        self.session.mount('https://', TLSv1HttpAdapter())
+        self.session.mount('https://', TLSHTTPAdapter())
         if self.auth_token:
             self.session.headers.update({'Authorization': 'Bearer ' + self.auth_token})
         # "ping" to see if the server is available


Test patch:
Repository: rucio/rucio

Issue:
Bad replicas declaration silently fails with server on 1.29.2 and client <1.29.2
Motivation
----------
This commit modified the REST API for declaring bad replicas :  
https://github.com/rucio/rucio/commit/335bc1deb7623f363da95ca6709c9a8de83f2d21
It replaces the `pfns` parameter by `replicas`. The client API is properly modified, so if one uses `1.29.2` on the server and client everything is fine. But using a client prior `1.29.2` with a server on `1.29.2` will silently fail the declaration of bad files (which did happen to ATLAS). Would it be possible to still use `pfns` if `replicas` is not send ?

Modification
------------





Discussion:

Resolving patch:
diff --git a/lib/rucio/web/rest/flaskapi/v1/replicas.py b/lib/rucio/web/rest/flaskapi/v1/replicas.py
--- a/lib/rucio/web/rest/flaskapi/v1/replicas.py
+++ b/lib/rucio/web/rest/flaskapi/v1/replicas.py
@@ -818,6 +818,12 @@ def post(self):
                     type: array
                     items:
                       type: string
+                  pfns:
+                    deprecated: true
+                    description: The list of pfns, for backward compatibility with older versions of the ReplicaClient
+                    type: array
+                    items:
+                      type: string
                   reason:
                     description: The reason for the declaration.
                     type: string
@@ -837,7 +843,7 @@ def post(self):
             description: Not acceptable
         """
         parameters = json_parameters()
-        replicas = param_get(parameters, 'replicas', default=[])
+        replicas = param_get(parameters, 'replicas', default=[]) or param_get(parameters, 'pfns', default=[])
         reason = param_get(parameters, 'reason', default=None)
 
         try:


Test patch:
Repository: rucio/rucio

Issue:
Clients try to access memcached 
Motivation
----------
Some places in clients indirectly (via imports) try to access cache values. The old library used for memcache silently ignored those accesses, but pymemcache doesn't like it. For now, the code which disables the caching does the trick, but prints a warning message, which isn't very user-friendly. 

Modification
------------
Ensure that clients never try to access the cache. 




Discussion:
The warning was removed by a recent commit, but the issue is still relevant. The problem is that clients try to access memcached, not the warning message.

Resolving patch:
diff --git a/lib/rucio/rse/__init__.py b/lib/rucio/rse/__init__.py
--- a/lib/rucio/rse/__init__.py
+++ b/lib/rucio/rse/__init__.py
@@ -15,7 +15,6 @@
 
 from dogpile.cache import make_region
 
-from rucio.common.cache import make_region_memcached
 from rucio.common.utils import is_client
 from rucio.rse import rsemanager
 
@@ -81,6 +80,7 @@ def generate_key(rse, vo='def', session=None):
 if rsemanager.SERVER_MODE:   # pylint:disable=no-member
     from rucio.core.rse import get_rse_protocols, get_rse_id
     from rucio.core.vo import map_vo
+    from rucio.common.cache import make_region_memcached
 
     def tmp_rse_info(rse=None, vo='def', rse_id=None, session=None):
         if rse_id is None:


Test patch:
Repository: rucio/rucio

Issue:
Access information from traces are not propagated to datasets
Motivation
----------

It has come to our attention that, while files have proper access times, datasets do not.

https://github.com/rucio/rucio/blob/58a5ff3bff3bdff2b7251167522dbfe4c0b444f8/lib/rucio/daemons/tracer/kronos.py#L410-L416

Effectively, `__update_datasets()` is not run during the daemon loop; it’s only run when the daemon exits. Most likely, there isn’t enough time to process all the datasets. This is a regression from #5687.

Modification
------------

Replace this obsolete and non-functional wait mechanism with the `run_daemon()`’s `sleep_time`.


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/tracer/kronos.py b/lib/rucio/daemons/tracer/kronos.py
--- a/lib/rucio/daemons/tracer/kronos.py
+++ b/lib/rucio/daemons/tracer/kronos.py
@@ -389,7 +389,8 @@ def run_once_kronos_file(heartbeat_handler: HeartbeatHandler, return_values: Dic
             conn.subscribe(destination=config_get('tracer-kronos', 'queue'), ack='client-individual', id=subscription_id, headers={'activemq.prefetchSize': prefetch_size})
 
 
-def kronos_dataset(once: bool = False, dataset_queue: Queue = None, sleep_time: int = 60):
+def kronos_dataset(dataset_queue: Queue, once: bool = False, sleep_time: int = 60):
+    return_values = {'heartbeat_handler': HeartbeatHandler("kronos-dataset", 10)}
     run_daemon(
         once=once,
         graceful_stop=graceful_stop,
@@ -400,23 +401,21 @@ def kronos_dataset(once: bool = False, dataset_queue: Queue = None, sleep_time:
         run_once_fnc=functools.partial(
             run_once_kronos_dataset,
             dataset_queue=dataset_queue,
+            return_values=return_values,
         )
     )
 
-    # once again for the backlog
-    __update_datasets(dataset_queue)
-
+    # once again for potential backlog
+    run_once_kronos_dataset(dataset_queue=dataset_queue, return_values=return_values, heartbeat_handler=return_values['heartbeat_handler'], sleep_time=sleep_time)
 
-def run_once_kronos_dataset(heartbeat_handler: HeartbeatHandler, dataset_queue: Queue, **kwargs):
-    _, _, logger = heartbeat_handler.live()
-    dataset_wait = config_get_int('tracer-kronos', 'dataset_wait')
-    start = datetime.now()
-    if (datetime.now() - start).seconds > dataset_wait:
-        __update_datasets(dataset_queue=dataset_queue, logger=logger)
-        start = datetime.now()
 
+def run_once_kronos_dataset(dataset_queue: Queue, return_values: dict, heartbeat_handler: HeartbeatHandler, **kwargs):
+    if heartbeat_handler is None:
+        if "heartbeat_handler" not in return_values.keys():
+            return_values["heartbeat_handler"] = HeartbeatHandler("kronos-dataset", 10)
+        heartbeat_handler = return_values["heartbeat_handler"]
 
-def __update_datasets(dataset_queue, logger=logging.log):
+    _, _, logger = heartbeat_handler.live()
     len_ds = dataset_queue.qsize()
     datasets = {}
     dslocks = {}


Test patch:
Repository: rucio/rucio

Issue:
Traces: Fix default value of tracedir  
Motivation
----------
The GitHub workflows in the documentation repo have been failing.
https://github.com/maany/documentation/actions/runs/3313567847/jobs/5471731568#step:5:169

The associated error message is:
```
IsADirectoryError: [Errno 21] Is a directory: '/var/log/rucio/trace
```
This arises from the fact that the default value of CONFIG_TRACE_TRACEDIR is `/var/log/rucio` 
https://github.com/rucio/rucio/blob/9080cd679d8e7049e10bcc3f7e7580551af59f24/lib/rucio/core/trace.py#L39
As a result, the 
```
ROTATING_HANDLER= logging.handlers.RotatingFileHandler(filename='%s/trace' % CONFIG_TRACE_TRACEDIR, maxBytes=CONFIG_TRACE_MAXBYTES, backupCount=CONFIG_TRACE_BACKUPCOUNT)
```
tries to create a `RotatingFileHandler` with the filename `/var/log/rucio/trace`, which is a directory

Modification
------------
Change `CONFIG_TRACE_TRACEDIR ` to `/var/log/rucio/trace`



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/trace.py b/lib/rucio/core/trace.py
--- a/lib/rucio/core/trace.py
+++ b/lib/rucio/core/trace.py
@@ -36,7 +36,7 @@
 
 CONFIG_TRACE_LOGLEVEL = getattr(logging, config_get('trace', 'loglevel', raise_exception=False, default='DEBUG').upper())
 CONFIG_TRACE_LOGFORMAT = config_get('trace', 'logformat', raise_exception=False, default='%(message)s')
-CONFIG_TRACE_TRACEDIR = config_get('trace', 'tracedir', raise_exception=False, default='/var/log/rucio')
+CONFIG_TRACE_TRACEDIR = config_get('trace', 'tracedir', raise_exception=False, default='/var/log/rucio/trace')
 CONFIG_TRACE_MAXBYTES = config_get_int('trace', 'maxbytes', raise_exception=False, default=1000000000)
 CONFIG_TRACE_BACKUPCOUNT = config_get_int('trace', 'backupCount', raise_exception=False, default=10)
 


Test patch:
Repository: rucio/rucio

Issue:
Fix UnicodeEncodeError while using PDB in baseclient.py _send_request
Motivation
----------
While using `remote_pdb` in a telnet client on `automatix` daemon pod in OKD4, when I try to use the `l` command to print the line numbers, `remote_pdb` throws `UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 112: ordinal not in range(128)`. This is because the character on this comment line next to the word `server` in `baseclient.py` is not a ascii single quotation (27), but a unicode right single quotation (u2019): https://github.com/rucio/rucio/blob/9dc4238217a37b938f120b3bac2007487f6a063e/lib/rucio/client/baseclient.py#L367

The other single quotes in the code are standard ascii quotations, such as on this line: https://github.com/rucio/rucio/blob/9dc4238217a37b938f120b3bac2007487f6a063e/lib/rucio/client/baseclient.py#L92

Modification
------------
Change the right quotation to an ascii quotation.




Discussion:

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -364,7 +364,7 @@ def _send_request(self, url, headers=None, type_='GET', data=None, params=None,
         :param get_token: (optional) if it is called from a _get_token function.
         :param cert: (optional) if String, path to the SSL client cert file (.pem). If Tuple, (cert, key) pair.
         :param auth: (optional) auth tuple to enable Basic/Digest/Custom HTTP Auth.
-        :param verify: (optional) either a boolean, in which case it controls whether we verify the server’s TLS
+        :param verify: (optional) either a boolean, in which case it controls whether we verify the server's TLS
                        certificate, or a string, in which case it must be a path to a CA bundle to use.
         :return: the HTTP return body.
         """


Test patch: