internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
Scope name validation regex allows single quotes
The scope validation regex for atlas/generic/icecube (but not cms) allows single quotes, which seems like a bad idea.

`"^[a-zA-Z'_'-.0-9]{1,%s}$" % SCOPE_LENGTH` looks wrong, and I'm guessing it ought to be `"^[a-zA-Z_\\-.0-9]{1,%s}$" % SCOPE_LENGTH`. Unless someone really does have this requirement?


Discussion:
@briedel is this an icecube requirement? Because if not then it's almost certainly a mistake.
Per @briedel in Slack, there's no need to allow single quotes.  Probably an error.

@illingwo - any chance you want to put in a PR?

Resolving patch:
diff --git a/lib/rucio/common/schema/atlas.py b/lib/rucio/common/schema/atlas.py
--- a/lib/rucio/common/schema/atlas.py
+++ b/lib/rucio/common/schema/atlas.py
@@ -52,7 +52,7 @@
 
 SCOPE = {"description": "Scope name",
          "type": "string",
-         "pattern": "^[a-zA-Z'_'-.0-9]{1,%s}$" % SCOPE_LENGTH}
+         "pattern": "^[a-zA-Z_\\-.0-9]{1,%s}$" % SCOPE_LENGTH}
 
 R_SCOPE = {"description": "Scope name",
            "type": "string",
diff --git a/lib/rucio/common/schema/generic.py b/lib/rucio/common/schema/generic.py
--- a/lib/rucio/common/schema/generic.py
+++ b/lib/rucio/common/schema/generic.py
@@ -49,7 +49,7 @@
 
 SCOPE = {"description": "Scope name",
          "type": "string",
-         "pattern": "^[a-zA-Z'_'-.0-9]{1,%s}$" % SCOPE_LENGTH}
+         "pattern": "^[a-zA-Z_\\-.0-9]{1,%s}$" % SCOPE_LENGTH}
 
 R_SCOPE = {"description": "Scope name",
            "type": "string",
diff --git a/lib/rucio/common/schema/icecube.py b/lib/rucio/common/schema/icecube.py
--- a/lib/rucio/common/schema/icecube.py
+++ b/lib/rucio/common/schema/icecube.py
@@ -50,7 +50,7 @@
 
 SCOPE = {"description": "Scope name",
          "type": "string",
-         "pattern": "^[a-zA-Z'_'-.0-9]{1,%s}$" % SCOPE_LENGTH}
+         "pattern": "^[a-zA-Z_\\-.0-9]{1,%s}$" % SCOPE_LENGTH}
 
 R_SCOPE = {"description": "Scope name",
            "type": "string",


Test patch:
Repository: rucio/rucio

Issue:
metalink: make glfn optional
Motivation
----------

Right now the glfn is hardcoded with the /atlas path in the metalink. Make it optional for non-ATLAS VOs.


Discussion:

Resolving patch:
diff --git a/lib/rucio/web/rest/webpy/v1/replica.py b/lib/rucio/web/rest/webpy/v1/replica.py
--- a/lib/rucio/web/rest/webpy/v1/replica.py
+++ b/lib/rucio/web/rest/webpy/v1/replica.py
@@ -371,8 +371,11 @@ def POST(self):
                         yield '  <hash type="md5">' + rfile['md5'] + '</hash>\n'
                     yield '  <size>' + str(rfile['bytes']) + '</size>\n'
 
-                    yield '  <glfn name="/atlas/rucio/%s:%s">' % (rfile['scope'], rfile['name'])
-                    yield '</glfn>\n'
+                    yield '  <glfn name="/%s/rucio/%s:%s"></glfn>\n' % (config_get('policy', 'schema',
+                                                                                   raise_exception=False,
+                                                                                   default='generic'),
+                                                                        rfile['scope'],
+                                                                        rfile['name'])
 
                     # TODO: deprecate this
                     if select == 'geoip':


Test patch:
Repository: rucio/rucio

Issue:
Unhandled exception in list_replicas
Motivation
----------
Traceback :

```
[Tue Feb 20 23:32:25.701947 2018] [:error] [pid 5431:tid 139990096934656] Traceback (most recent call last):
[Tue Feb 20 23:32:25.701980 2018] [:error] [pid 5431:tid 139990096934656]   File "/usr/lib/python2.7/site-packages/rucio/core/replica.py", line 752, in _list_replicas
[Tue Feb 20 23:32:25.701984 2018] [:error] [pid 5431:tid 139990096934656]     domain=domain)['scheme'])
[Tue Feb 20 23:32:25.701986 2018] [:error] [pid 5431:tid 139990096934656]   File "/usr/lib/python2.7/site-packages/rucio/rse/rsemanager.py", line 126, in select_protocol
[Tue Feb 20 23:32:25.701989 2018] [:error] [pid 5431:tid 139990096934656]     candidates = _get_possible_protocols(rse_settings, operation, scheme, domain)
[Tue Feb 20 23:32:25.701991 2018] [:error] [pid 5431:tid 139990096934656]   File "/usr/lib/python2.7/site-packages/rucio/rse/rsemanager.py", line 102, in _get_possible_protocols
[Tue Feb 20 23:32:25.701994 2018] [:error] [pid 5431:tid 139990096934656]     ' found : %s.' % str(rse_settings))
[Tue Feb 20 23:32:25.701996 2018] [:error] [pid 5431:tid 139990096934656] RSEProtocolNotSupported: RSE does not support requested protocol.
```

Modification
------------
Need to catch RSEProtocolNotSupported exception



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -868,6 +868,8 @@ def _list_replicas(dataset_clause, file_clause, state_clause, show_pfns,
                                 rse_schemes.append(rsemgr.select_protocol(rse_settings=rse_info[rse],
                                                                           operation='read',
                                                                           domain=domain)['scheme'])
+                        except exception.RSEProtocolNotSupported:
+                            pass  # no need to be verbose
                         except Exception:
                             print(format_exc())
 


Test patch:
Repository: rucio/rucio

Issue:
parse_json not python3 compatible
Motivation
----------

String responses cannot be decoded into UTF-8 in Python 3, since they're already UTF-8 by default. The function has been removed from the language.

Modification
------------

provided by @XeBoris

```
def parse_response(data):
   """ JSON render function
   """
   ret_obj = None
   try:
       ret_obj = data.decode('utf-8')
   except AttributeError:
       ret_obj = data
   
   return json.loads(ret_obj)
```


Discussion:
We should check if there is a common implementation to do this. 

Resolving patch:
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -272,9 +272,16 @@ def datetime_parser(dct):
 
 
 def parse_response(data):
-    """ JSON render function
     """
-    return json.loads(data.decode('utf-8'), object_hook=datetime_parser)
+    JSON render function
+    """
+    ret_obj = None
+    try:
+        ret_obj = data.decode('utf-8')
+    except AttributeError:
+        ret_obj = data
+
+    return json.loads(ret_obj, object_hook=datetime_parser)
 
 
 def generate_http_error(status_code, exc_cls, exc_msg):


Test patch:
Repository: rucio/rucio

Issue:
Add top-level attribute EVENT_TYPE to messages
Motivation
----------
Idea is to be able to make filters on the broker based on the EVENT_TYPE.


Discussion:
+ scope ?
`Scope` might make sense, I'll have a look.
Up
Can we increase the priority of this ticket ? It's mainly for AMI

Resolving patch:
diff --git a/lib/rucio/daemons/hermes/hermes.py b/lib/rucio/daemons/hermes/hermes.py
--- a/lib/rucio/daemons/hermes/hermes.py
+++ b/lib/rucio/daemons/hermes/hermes.py
@@ -13,11 +13,13 @@
 # limitations under the License.
 #
 # Authors:
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2014-2017
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2014-2018
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2014
 # - Wen Guan <wguan.icedew@gmail.com>, 2014-2015
 # - Vincent Garonne <vgaronne@gmail.com>, 2015-2018
 # - Martin Barisits <martin.barisits@cern.ch>, 2016-2017
+#
+# PY3K COMPATIBLE
 
 '''
    Hermes is a daemon to deliver messages: to a messagebroker via STOMP, or emails via SMTP.
@@ -257,11 +259,14 @@ def deliver_messages(once=False, brokers_resolved=None, thread=0, bulk=1000, del
                                              heartbeat['nr_threads'],
                                              host_and_ports)
                                 conn.connect(wait=True)
+
                         conn.send(body=json.dumps({'event_type': str(message['event_type']).lower(),
                                                    'payload': message['payload'],
                                                    'created_at': str(message['created_at'])}),
                                   destination=destination,
-                                  headers={'persistent': 'true'})
+                                  headers={'persistent': 'true',
+                                           'event_type': str(message['event_type']).lower()})
+
                         to_delete.append({'id': message['id'],
                                           'created_at': message['created_at'],
                                           'updated_at': message['created_at'],
@@ -411,9 +416,9 @@ def run(once=False, send_email=True, threads=1, bulk=1000, delay=10, broker_time
                                                                          'bulk': bulk,
                                                                          'delay': delay,
                                                                          'broker_timeout': broker_timeout,
-                                                                         'broker_retry': broker_retry}) for i in xrange(0, threads)]
+                                                                         'broker_retry': broker_retry}) for i in range(0, threads)]
 
-        for thrd in xrange(0, 1):
+        for thrd in range(0, 1):
             thread_list.append(threading.Thread(target=deliver_emails, kwargs={'thread': thrd,
                                                                                'bulk': bulk,
                                                                                'delay': delay}))


Test patch:
Repository: rucio/rucio

Issue:
transmogrifier misbehaving with zero subscriptions
Motivation
----------
If transmogrifier is set up to run without specifying any subscriptions, the CPU load becomes excessive.

Kronos shows similar heavy CPU loads in the absence of subscriptions.

Modification
------------



Discussion:
I guess the second sentence should read "in the absence of messages in the queue" ;-)

Resolving patch:
diff --git a/lib/rucio/daemons/transmogrifier.py b/lib/rucio/daemons/transmogrifier.py
--- a/lib/rucio/daemons/transmogrifier.py
+++ b/lib/rucio/daemons/transmogrifier.py
@@ -141,7 +141,7 @@ def is_matching_subscription(subscription, did, metadata):
     return True
 
 
-def transmogrifier(bulk=5, once=False):
+def transmogrifier(bulk=5, once=False, sleep_time=60):
     """
     Creates a Transmogrifier Worker that gets a list of new DIDs for a given hash,
     identifies the subscriptions matching the DIDs and
@@ -150,6 +150,7 @@ def transmogrifier(bulk=5, once=False):
     :param thread: Thread number at startup.
     :param bulk: The number of requests to process.
     :param once: Run only once.
+    :param sleep_time: Time between two cycles.
     """
 
     executable = ' '.join(argv)
@@ -378,14 +379,15 @@ def transmogrifier(bulk=5, once=False):
             monitor.record_counter(counters='transmogrifier.addnewrule.error', delta=1)
         if once is True:
             break
-        if tottime < 10:
-            time.sleep(10 - tottime)
+        if tottime < sleep_time:
+            logging.info(prepend_str + 'Will sleep for %s seconds' % (sleep_time - tottime))
+            time.sleep(sleep_time - tottime)
     heartbeat.die(executable, hostname, pid, hb_thread)
     logging.info(prepend_str + 'Graceful stop requested')
     logging.info(prepend_str + 'Graceful stop done')
 
 
-def run(threads=1, bulk=100, once=False):
+def run(threads=1, bulk=100, once=False, sleep_time=60):
     """
     Starts up the transmogrifier threads.
     """
@@ -396,6 +398,7 @@ def run(threads=1, bulk=100, once=False):
     else:
         logging.info('starting transmogrifier threads')
         thread_list = [threading.Thread(target=transmogrifier, kwargs={'once': once,
+                                                                       'sleep_time': sleep_time,
                                                                        'bulk': bulk}) for _ in range(0, threads)]
         [thread.start() for thread in thread_list]
         logging.info('waiting for interrupts')


Test patch:
Repository: rucio/rucio

Issue:
Improve error message for PFN mismatch
Motivation
----------
Sometimes we get the error in panda for our jobs:

```
ddm, 200: Could not add files to DDM: Details: [u"One of the PFNs provided does not match the Rucio expected PFN : [u'gsiftp://dtn01.nersc.gov/project/projectdirs/m2015/atlasdatadisk/rucio/mc16_13TeV/3b/cd/log.11350637._033145.job.log.tgz.1'] vs [u'gsiftp://dtn01.nersc.gov:2811/project/projectdirs/m2015/atlasdatadisk/rucio/mc16_13TeV/3b/cd/log.11350637._033145.job.log.tgz.1'] ([

{'scope': u'mc16_13TeV', 'name': u'log.11350637._033145.job.log.tgz.1'}
])"]
```

In this message it does not make it clear, which path we should be editing vs which one Rucio is expected. I found the message sitting in 
rucio/src/lib/rucio/core/replica.py
around line 1023:
raise exception.InvalidPath('One of the PFNs provided does not match the Rucio expected PFN : %s vs %s (%s)' % (str(pfns), str(expected_pfns), str(lfns)))

So I see the first is probably what we set in AGIS and the second is what you are expecting. Can you make this clear in the error message?

Thanks,
Taylor

_Migrated from [JIRA](https://its.cern.ch/jira/browse/RUCIO-2692)_


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -1254,8 +1254,8 @@ def add_replicas(rse, files, account, rse_id=None, ignore_availability=True,
             expected_pfns = clean_surls(expected_pfns.values())
             pfns = clean_surls(pfns)
             if pfns != expected_pfns:
-                print('ALERT: One of the PFNs provided does not match the Rucio expected PFN : %s vs %s (%s)' % (str(pfns), str(expected_pfns), str(lfns)))
-                raise exception.InvalidPath('One of the PFNs provided does not match the Rucio expected PFN : %s vs %s (%s)' % (str(pfns), str(expected_pfns), str(lfns)))
+                print('ALERT: One of the PFNs provided does not match the Rucio expected PFN : got %s, expected %s (%s)' % (str(pfns), str(expected_pfns), str(lfns)))
+                raise exception.InvalidPath('One of the PFNs provided does not match the Rucio expected PFN : got %s, expected %s (%s)' % (str(pfns), str(expected_pfns), str(lfns)))
 
     nbfiles, bytes = __bulk_add_replicas(rse_id=replica_rse.id, files=files, account=account, session=session)
     increase(rse_id=replica_rse.id, files=nbfiles, bytes=bytes, session=session)


Test patch:
Repository: rucio/rucio

Issue:
urlparse.urlparse call in transfertool interface is wrong
Motivation
----------
This was changed in a recent PR, however, the call on `urlparse.urlparse` is wrong, `urlparse` method should be called directly.


Discussion:

Resolving patch:
diff --git a/lib/rucio/transfertool/fts3.py b/lib/rucio/transfertool/fts3.py
--- a/lib/rucio/transfertool/fts3.py
+++ b/lib/rucio/transfertool/fts3.py
@@ -508,7 +508,7 @@ def set_se_status(self, storage_element, message, ban=True, timeout=None):
     @staticmethod
     def __extract_host(external_host):
         # graphite does not like the dots in the FQDN
-        return urlparse.urlparse(external_host).hostname.replace('.', '_')
+        return urlparse(external_host).hostname.replace('.', '_')
 
     def __get_transfer_baseid_voname(self):
         """


Test patch:
Repository: rucio/rucio

Issue:
REST call for requests is broken by '/' in DIDs
Motivation
----------
As discussed in #1786 , the REST call of https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/webpy/v1/request.py get broken for DIDs containing `/`

Modification
------------
I think that rather than doing this (*) we can pass all the needed information (scope, name, rse) as params instead of including them in the url path. 

(*) https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/webpy/v1/request.py#L38


Discussion:
In https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/webpy/v1/did.py we did this with the SCOPE_NAME_REGEXP 
There is an issue with this way though (see #1746) but in this case it should be fine.
I think just submitting these as params is not a bad idea, but due to backward compatibility I would go the regexp way.
Ok, thanks, I missed the SCOPE_NAME_REGEXP thing. I'll go with that then.

Resolving patch:
diff --git a/lib/rucio/web/rest/webpy/v1/request.py b/lib/rucio/web/rest/webpy/v1/request.py
--- a/lib/rucio/web/rest/webpy/v1/request.py
+++ b/lib/rucio/web/rest/webpy/v1/request.py
@@ -26,6 +26,7 @@
 from web import application, ctx, loadhook, header
 
 from rucio.api import request
+from rucio.common.schema import SCOPE_NAME_REGEXP
 from rucio.common.utils import generate_http_error, APIEncoder
 from rucio.web.rest.common import rucio_loadhook, RucioController, exception_wrapper
 
@@ -35,7 +36,7 @@
 SH.setLevel(DEBUG)
 LOGGER.addHandler(SH)
 
-URLS = ('/(.+)/(.+)/(.+)', 'RequestGet',)
+URLS = ('%s/(.+)' % SCOPE_NAME_REGEXP, 'RequestGet',)
 
 
 class RequestGet(RucioController):


Test patch:
Repository: rucio/rucio

Issue:
Wrongly handled exception makes the poller crash
Motivation
----------
Traceback :
```
2018-11-15 00:13:40,539 23840   ERROR   Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/poller.py", line 248, in poll_transfers
    for transfer_id in resps:
UnboundLocalError: local variable 'resps' referenced before assignment
```



Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/poller.py b/lib/rucio/daemons/conveyor/poller.py
--- a/lib/rucio/daemons/conveyor/poller.py
+++ b/lib/rucio/daemons/conveyor/poller.py
@@ -232,8 +232,23 @@ def poll_transfers(external_host, xfers, prepend_str='', request_ids=None, timeo
             logging.info(prepend_str + 'Polling %i transfers against %s with timeout %s' % (len(xfers), external_host, timeout))
             resps = transfer_core.bulk_query_transfers(external_host, xfers, 'fts3', timeout)
             record_timer('daemons.conveyor.poller.bulk_query_transfers', (time.time() - tss) * 1000 / len(xfers))
-        except (TransferToolTimeout, TransferToolWrongAnswer) as error:
+        except TransferToolTimeout as error:
             logging.error(prepend_str + str(error))
+            return
+        except TransferToolWrongAnswer as error:
+            logging.error(prepend_str + str(error))
+            logging.error(prepend_str + 'Problem querying %s on %s. All jobs are being checked individually' % (str(xfers), external_host))
+            for xfer in xfers:
+                try:
+                    logging.debug(prepend_str + 'Checking %s on %s' % (xfer, external_host))
+                    status = transfer_core.bulk_query_transfers(external_host, [xfer, ], 'fts3', timeout)
+                    logging.debug(prepend_str + str(status))
+                    if xfer in status and isinstance(status[xfer], Exception):
+                        logging.error(prepend_str + 'Problem querying %s on %s . Error returned : %s' % (xfer, external_host, str(status[xfer])))
+                except Exception as err:
+                    logging.error(prepend_str + 'Problem querying %s on %s . Error returned : %s' % (xfer, external_host, str(err)))
+                    break
+            return
         except RequestException as error:
             logging.error(prepend_str + "Failed to contact FTS server: %s" % (str(error)))
             return


Test patch:
Repository: rucio/rucio

Issue:
Kronos crashes when updating eol_at
Motivation
----------
Kronos crashes when updating eol_at


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/lock.py b/lib/rucio/core/lock.py
--- a/lib/rucio/core/lock.py
+++ b/lib/rucio/core/lock.py
@@ -431,8 +431,8 @@ def touch_dataset_locks(dataset_locks, session=None):
         try:
             session.query(models.DatasetLock).filter_by(scope=dataset_lock['scope'], name=dataset_lock['name'], rse_id=dataset_lock['rse_id']).\
                 update({'accessed_at': dataset_lock.get('accessed_at') or now}, synchronize_session=False)
-            for rule_id in session.query(models.DatasetLock.rule_id).filter_by(scope=dataset_lock['scope'], name=dataset_lock['name'], rse_id=dataset_lock['rse_id']):
-                session.query(models.ReplicationRule).filter_by(id=rule_id).update({'eol_at': eol_at}, synchronize_session=False)
+            for res in session.query(models.DatasetLock.rule_id).filter_by(scope=dataset_lock['scope'], name=dataset_lock['name'], rse_id=dataset_lock['rse_id']):
+                session.query(models.ReplicationRule).filter_by(id=res[0]).update({'eol_at': eol_at}, synchronize_session=False)
         except DatabaseError:
             return False
 


Test patch:
Repository: rucio/rucio

Issue:
Typos and other small fixes
Motivation
----------

Stumbled upon them while using Rucio. This issue collects them, though they aren’t related.

Modification
------------

Small patches of little consequence.


Discussion:

Resolving patch:
diff --git a/lib/rucio/client/rseclient.py b/lib/rucio/client/rseclient.py
--- a/lib/rucio/client/rseclient.py
+++ b/lib/rucio/client/rseclient.py
@@ -196,7 +196,7 @@ def list_rse_attributes(self, rse):
 
         :param rse: The RSE name.
 
-        :return: True if RSE attribute was created successfully else False.
+        :return: A ``dict`` with the RSE attribute name/value pairs.
         """
         path = '/'.join([self.RSE_BASEURL, rse, 'attr/'])
         url = build_url(choice(self.list_hosts), path=path)
diff --git a/lib/rucio/common/dumper/__init__.py b/lib/rucio/common/dumper/__init__.py
--- a/lib/rucio/common/dumper/__init__.py
+++ b/lib/rucio/common/dumper/__init__.py
@@ -247,7 +247,7 @@ def to_datetime(str_or_datetime):
 def agis_endpoints_data(cache=True):
     '''
     Returns the result of querying the ddmendpoints list from AGIS
-    and parsing the results with json.lads().
+    and parsing the results with json.loads().
 
     :param cache: If True only makes one query to AGIS and returns
     the same result for subsequent invocations. Consider the effects
diff --git a/lib/rucio/daemons/auditor/__init__.py b/lib/rucio/daemons/auditor/__init__.py
--- a/lib/rucio/daemons/auditor/__init__.py
+++ b/lib/rucio/daemons/auditor/__init__.py
@@ -69,7 +69,7 @@ def consistency(rse, delta, configuration, cache_dir, results_dir):
 
 
 def guess_replica_info(path):
-    """Try to extract the scope and name from a path
+    """Try to extract the scope and name from a path.
 
     ``path`` should be an ``str`` with the relative path to the file on
     the RSE.
@@ -87,7 +87,7 @@ def guess_replica_info(path):
 
 
 def bz2_compress_file(source, chunk_size=65000):
-    """Compress a file with bzip2
+    """Compress a file with bzip2.
 
     The destination is the path passed through ``source`` extended with
     '.bz2'.  The original file is deleted.
@@ -115,7 +115,7 @@ def bz2_compress_file(source, chunk_size=65000):
 
 
 def process_output(output, sanity_check=True, compress=True):
-    """Perform post-consistency-check actions
+    """Perform post-consistency-check actions.
 
     DARK files are put in the quarantined-replica table so that they
     may be deleted by the Dark Reaper.  LOST files are currently


Test patch: