anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: rucio/rucio
Issue: kronos crashes for touch events

Motivation
----------
The handling of touch events in kronos does not convert the scope/datasetScope to a InternalScope object, which crashes the update the accessed_at timestamp for that file/dataset.


Modification
------------

Convert scopes for touch events to InternalScope


Changed files:
lib/rucio/daemons/tracer/kronos.py

diff --git a/lib/rucio/daemons/tracer/kronos.py b/lib/rucio/daemons/tracer/kronos.py
index c33e13b13..a24c0e916 100644
--- a/lib/rucio/daemons/tracer/kronos.py
+++ b/lib/rucio/daemons/tracer/kronos.py
@@ -239,7 +239,7 @@ class AMQConsumer(object):
                             logging.error(format_exc())
                             record_counter('daemons.tracer.kronos.rse_not_found')
                     if 'datasetScope' in report:
-                        self.__dataset_queue.put({'scope': report['datasetScope'], 'name': report['dataset'], 'rse_id': rse_id, 'accessed_at': datetime.utcfromtimestamp(report['traceTimeentryUnix'])})
+                        self.__dataset_queue.put({'scope': InternalScope(report['datasetScope']), 'name': report['dataset'], 'rse_id': rse_id, 'accessed_at': datetime.utcfromtimestamp(report['traceTimeentryUnix'])})
                         continue
                     else:
                         if 'remoteSite' not in report:
Repository: rucio/rucio
Issue: Conveyor submitter creates partitions wrongly

Motivation
----------
There is an unnecessary `-1` when creating the partitions for the conveyor-submitter.

Modification
------------
https://github.com/rucio/rucio/blob/930bf853c1a3399592a974108c8805c24ae859a2/lib/rucio/daemons/conveyor/submitter.py#L160
Remove the `-1` there

Changed files:
lib/rucio/daemons/conveyor/submitter.py

diff --git a/lib/rucio/daemons/conveyor/submitter.py b/lib/rucio/daemons/conveyor/submitter.py
index 3c3ed4f5d..4ef2ffc10 100644
--- a/lib/rucio/daemons/conveyor/submitter.py
+++ b/lib/rucio/daemons/conveyor/submitter.py
@@ -17,7 +17,7 @@
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2019
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2013
 # - Vincent Garonne <vgaronne@gmail.com>, 2014-2018
-# - Martin Barisits <martin.barisits@cern.ch>, 2014-2017
+# - Martin Barisits <martin.barisits@cern.ch>, 2014-2020
 # - Wen Guan <wguan.icedew@gmail.com>, 2014-2016
 # - Tomas Kouba <tomas.kouba@cern.ch>, 2014
 # - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2016
@@ -157,7 +157,7 @@ def submitter(once=False, rses=None, mock=False,
 
                 logging.info('%s Starting to get transfer transfers for %s', prepend_str, activity)
                 start_time = time.time()
-                transfers = __get_transfers(total_workers=heart_beat['nr_threads'] - 1,
+                transfers = __get_transfers(total_workers=heart_beat['nr_threads'],
                                             worker_number=heart_beat['assign_thread'],
                                             failover_schemes=failover_scheme,
                                             limit=bulk,
Repository: rucio/rucio
Issue: REST call for requests is broken by '/' in DIDs

Motivation
----------
As discussed in #1786 , the REST call of https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/webpy/v1/request.py get broken for DIDs containing `/`

Modification
------------
I think that rather than doing this (*) we can pass all the needed information (scope, name, rse) as params instead of including them in the url path. 

(*) https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/webpy/v1/request.py#L38

Changed files:
lib/rucio/web/rest/webpy/v1/request.py

diff --git a/lib/rucio/web/rest/webpy/v1/request.py b/lib/rucio/web/rest/webpy/v1/request.py
index 5a456d3b5..dd49b127a 100755
--- a/lib/rucio/web/rest/webpy/v1/request.py
+++ b/lib/rucio/web/rest/webpy/v1/request.py
@@ -26,6 +26,7 @@ from logging import getLogger, StreamHandler, DEBUG
 from web import application, ctx, loadhook, header
 
 from rucio.api import request
+from rucio.common.schema import SCOPE_NAME_REGEXP
 from rucio.common.utils import generate_http_error, APIEncoder
 from rucio.web.rest.common import rucio_loadhook, RucioController, exception_wrapper
 
@@ -35,7 +36,7 @@ SH = StreamHandler()
 SH.setLevel(DEBUG)
 LOGGER.addHandler(SH)
 
-URLS = ('/(.+)/(.+)/(.+)', 'RequestGet',)
+URLS = ('%s/(.+)' % SCOPE_NAME_REGEXP, 'RequestGet',)
 
 
 class RequestGet(RucioController):
Repository: rucio/rucio
Issue: Fix extract_scope method for BelleII

Motivation
----------
The method doesn't work properly for users




Changed files:
lib/rucio/common/utils.py

diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
index bd7646d73..7cbcba2ca 100644
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -768,12 +768,12 @@ def extract_scope_belleii(did, scopes):
     if did.find('/belle/Raw/') > -1:
         return 'raw', did
     if did.find('/belle/user/') > -1:
-        if len(split_did) > 5:
-            if len(split_did[4]) == 1 and 'user.%s' % (split_did[5]) in scopes:
-                return 'user.%s' % split_did[5], did
         if len(split_did) > 4:
-            if 'user.%s' % (split_did[4]) in scopes:
+            if len(split_did[3]) == 1 and 'user.%s' % (split_did[4]) in scopes:
                 return 'user.%s' % split_did[4], did
+        if len(split_did) > 3:
+            if 'user.%s' % (split_did[3]) in scopes:
+                return 'user.%s' % split_did[3], did
         return 'user', did
     if did.find('/belle/data/') > -1:
         if len(split_did) > 5:
Repository: rucio/rucio
Issue: parse_json not python3 compatible

Motivation
----------

String responses cannot be decoded into UTF-8 in Python 3, since they're already UTF-8 by default. The function has been removed from the language.

Modification
------------

provided by @XeBoris

```
def parse_response(data):
   """ JSON render function
   """
   ret_obj = None
   try:
       ret_obj = data.decode('utf-8')
   except AttributeError:
       ret_obj = data
   
   return json.loads(ret_obj)
```

Changed files:
lib/rucio/common/utils.py

diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
index fe7695560..0b1e9f03a 100644
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -272,9 +272,16 @@ def datetime_parser(dct):
 
 
 def parse_response(data):
-    """ JSON render function
     """
-    return json.loads(data.decode('utf-8'), object_hook=datetime_parser)
+    JSON render function
+    """
+    ret_obj = None
+    try:
+        ret_obj = data.decode('utf-8')
+    except AttributeError:
+        ret_obj = data
+
+    return json.loads(ret_obj, object_hook=datetime_parser)
 
 
 def generate_http_error(status_code, exc_cls, exc_msg):
Repository: rucio/rucio
Issue: Multihop problems with multiple sources

Motivation
----------
This issue comes up in cases when a destination will require multhop and there are mixed tape/disk sources. Depending on the order these sources are iterated in the loop, the `bring_online` attribute is set for the transfer due to one of the sources being tape.  Finally even if the disk source is selected for the multihop, the transfer is still submitted with `bring_online`

This however uncovered a somewhat bigger issue that in case of multhops we do not properly evaluate sources. Instead the algorithm will use the last source retrieved from the database, even if this one is worse (e.g. a tape source)



Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 2aa364698..4388fb25c 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -1,4 +1,5 @@
-# Copyright 2013-2020 CERN for the benefit of the ATLAS collaboration.
+# -*- coding: utf-8 -*-
+# Copyright 2013-2020 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +16,7 @@
 # Authors:
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2020
 # - Martin Barisits <martin.barisits@cern.ch>, 2017-2020
-# - Vincent Garonne <vgaronne@gmail.com>, 2017
+# - Vincent Garonne <vincent.garonne@cern.ch>, 2017
 # - Igor Mandrichenko <rucio@fermicloud055.fnal.gov>, 2018
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2018-2020
 # - dciangot <diego.ciangottini@cern.ch>, 2018
@@ -24,9 +25,12 @@
 # - Andrew Lister <andrew.lister@stfc.ac.uk>, 2019
 # - Brandon White <bjwhite@fnal.gov>, 2019
 # - maatthias <maatthias@gmail.com>, 2019
-# - Gabriele <sucre.91@hotmail.it>, 2019
+# - Gabriele Fronze' <gfronze@cern.ch>, 2019
 # - Jaroslav Guenther <jaroslav.guenther@cern.ch>, 2019-2020
+# - Matt Snyder <msnyder@bnl.gov>, 2020
+# - Eric Vaandering <ewv@fnal.gov>, 2020
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
+# - Nick Smith <nick.smith@cern.ch>, 2020
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 #
 # PY3K COMPATIBLE
@@ -1244,7 +1248,7 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                                              'src_spacetoken': None,
                                              'dest_spacetoken': dest_spacetoken,
                                              'overwrite': transfers[req_id]['overwrite'],
-                                             'bring_online': transfers[req_id]['bring_online'],
+                                             'bring_online': bring_online,
                                              'copy_pin_lifetime': transfers[req_id]['copy_pin_lifetime'],
                                              'external_host': transfers[req_id]['external_host'],
                                              'selection_strategy': 'auto',
Repository: rucio/rucio
Issue: source replica expression ignored for multi-hop transfers

Motivation
----------
The `source-replica-expression` of a rule is ignored in the case of multi-hop transfers.



Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index d72034f5a..9b67df71b 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -686,13 +686,26 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
 
         dict_attributes = get_attributes(attributes)
 
-        # Check if the source and destination are blacklisted
+        # Check if the source and destination are blocked
         if source_rse_id in unavailable_read_rse_ids:
             continue
         if dest_rse_id in unavailable_write_rse_ids:
-            logging.warning('RSE %s is blacklisted for write. Will skip the submission of new jobs', dest_rse_name)
+            logging.warning('RSE %s is blocked for write. Will skip the submission of new jobs', dest_rse_name)
             continue
 
+        # parse source expression
+        source_replica_expression = dict_attributes.get('source_replica_expression', None)
+        if source_replica_expression:
+            try:
+                parsed_rses = parse_expression(source_replica_expression, session=session)
+            except InvalidRSEExpression as error:
+                logging.error("Invalid RSE exception %s: %s", source_replica_expression, str(error))
+                continue
+            else:
+                allowed_rses = [x['id'] for x in parsed_rses]
+                if source_rse_id not in allowed_rses:
+                    continue
+
         # Call the get_hops function to create a list of RSEs used for the transfer
         # In case the source_rse and the dest_rse are connected, the list contains only the destination RSE
         # In case of non-connected, the list contains all the intermediary RSEs
@@ -741,19 +754,6 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
             if rses and dest_rse_id not in rses:
                 continue
 
-            # parse source expression
-            source_replica_expression = dict_attributes.get('source_replica_expression', None)
-            if source_replica_expression:
-                try:
-                    parsed_rses = parse_expression(source_replica_expression, session=session)
-                except InvalidRSEExpression as error:
-                    logging.error("Invalid RSE exception %s: %s", source_replica_expression, str(error))
-                    continue
-                else:
-                    allowed_rses = [x['id'] for x in parsed_rses]
-                    if source_rse_id not in allowed_rses:
-                        continue
-
             # Get the source rse information
             if source_rse_id not in rses_info:
                 rses_info[source_rse_id] = rsemgr.get_rse_info(rse=source_rse_name,
Repository: rucio/rucio
Issue: Bug in Belle II permission for dirac

Motivation
----------

There's a bug in the belleii permission that prevent normal users to register files.



Changed files:
lib/rucio/core/permission/belleii.py

diff --git a/lib/rucio/core/permission/belleii.py b/lib/rucio/core/permission/belleii.py
index 38d3182f7..0ba175239 100644
--- a/lib/rucio/core/permission/belleii.py
+++ b/lib/rucio/core/permission/belleii.py
@@ -24,6 +24,7 @@
 #
 # PY3K COMPATIBLE
 
+from rucio.common.types import InternalScope
 from rucio.core.lifetime_exception import list_exceptions
 import rucio.core.scope
 from rucio.core.account import list_account_attributes, has_account_attribute
@@ -382,9 +383,12 @@ def perm_add_dids(issuer, kwargs):
     # Check the accounts of the issued rules
     if issuer != 'root' and not has_account_attribute(account=issuer, key='admin'):
         for did in kwargs['dids']:
+            if not rucio.core.scope.is_scope_owner(scope=InternalScope(did['scope']), account=issuer):
+                return False
             for rule in did.get('rules', []):
                 if rule['account'] != issuer:
                     return False
+        return True
 
     return _is_root(issuer) or has_account_attribute(account=issuer, key='admin')
Repository: rucio/rucio
Issue: RSEType not migrated to new enum in list_replicas

Motivation
----------

list_replicas() response comes back with `type:RSEType.DISK` instead of `type:DISK`.

This breaks the pilot which makes a strict check.


Changed files:
lib/rucio/core/replica.py

diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
index 9f5c1d400..b84ce2f3a 100644
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2013-2020 CERN
+# Copyright 2013-2021 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,7 +18,7 @@
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2020
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2013-2014
 # - Martin Barisits <martin.barisits@cern.ch>, 2013-2020
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2014-2020
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2014-2021
 # - David Cameron <david.cameron@cern.ch>, 2014
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2014-2020
 # - Wen Guan <wen.guan@cern.ch>, 2014-2015
@@ -1111,7 +1111,7 @@ def _list_replicas(dataset_clause, file_clause, state_clause, show_pfns,
                     for tmp_pfn in pfns:
                         file['pfns'][tmp_pfn[0]] = {'rse_id': tmp_pfn[4]['rse_id'] if tmp_pfn[1] == 'zip' else rse_id,
                                                     'rse': tmp_pfn[4]['rse'] if tmp_pfn[1] == 'zip' else rse,
-                                                    'type': tmp_pfn[4]['type'] if tmp_pfn[1] == 'zip' else str(rse_type),
+                                                    'type': tmp_pfn[4]['type'] if tmp_pfn[1] == 'zip' else str(rse_type.name),
                                                     'volatile': tmp_pfn[4]['volatile'] if tmp_pfn[1] == 'zip' else volatile,
                                                     'domain': tmp_pfn[1],
                                                     'priority': tmp_pfn[2],
Repository: rucio/rucio
Issue: Incorrect use of arguments in Dark Reaper

Motivation
----------

I am further confused by commit f5578416bdfc071ebac14ca3cd2f64b669993597. `list_rses()` returns a list of strings, each containing an RSE ID (previously the RSE name instead). Yet the `reaper()` function seems to have been modified to expect a list of dictionaries.

Modification
------------




Changed files:
lib/rucio/daemons/reaper/dark_reaper.py

diff --git a/lib/rucio/daemons/reaper/dark_reaper.py b/lib/rucio/daemons/reaper/dark_reaper.py
index 1fb123799..34ea698bc 100644
--- a/lib/rucio/daemons/reaper/dark_reaper.py
+++ b/lib/rucio/daemons/reaper/dark_reaper.py
@@ -23,6 +23,7 @@
 # - Brandon White <bjwhite@fnal.gov>, 2019
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
+# - Dimitrios Christidis <dimitrios.christidis@cern.ch>, 2020
 
 '''
 Dark Reaper is a daemon to manage quarantined file deletion.
@@ -76,13 +77,13 @@ def reaper(rses=[], worker_number=0, total_workers=1, chunk_size=100, once=False
     :param once: If True, only runs one iteration of the main loop.
     :param scheme: Force the reaper to use a particular protocol, e.g., mock.
     """
-    logging.info('Starting Dark Reaper %s-%s: Will work on RSEs: %s', worker_number, total_workers, ', '.join([rse['rse'] for rse in rses]))
+    logging.info('Starting Dark Reaper %s-%s: Will work on RSEs: %s', worker_number, total_workers, ', '.join(rses))
 
     pid = os.getpid()
     thread = threading.current_thread()
     hostname = socket.gethostname()
     executable = ' '.join(sys.argv)
-    hash_executable = hashlib.sha256(sys.argv[0] + ''.join([rse['rse'] for rse in rses])).hexdigest()
+    hash_executable = hashlib.sha256(sys.argv[0] + ''.join(rses)).hexdigest()
     sanity_check(executable=None, hostname=hostname)
 
     while not GRACEFUL_STOP.is_set():
@@ -93,14 +94,13 @@ def reaper(rses=[], worker_number=0, total_workers=1, chunk_size=100, once=False
             nothing_to_do = True
 
             random.shuffle(rses)
-            for rse in rses:
-                rse_id = rse['id']
-                rse = rse['rse']
+            for rse_id in rses:
                 replicas = list_quarantined_replicas(rse_id=rse_id,
                                                      limit=chunk_size, worker_number=worker_number,
                                                      total_workers=total_workers)
 
                 rse_info = rsemgr.get_rse_info(rse_id=rse_id)
+                rse = rse_info['rse']
                 prot = rsemgr.create_protocol(rse_info, 'delete', scheme=scheme)
                 deleted_replicas = []
                 try:
Repository: rucio/rucio
Issue: PFNs that can’t be matched to an RSE remain in the BadPFNs table forever

Motivation
----------

For example:
```
$ rucio-admin replicas declare-bad --reason test srm://example.com/foo/bar/baz
Chunk 1/1 : 1 replicas successfully declared
--------------------------------
Summary
1 replicas successfully declared
```
The return values from [`get_pfn_to_rse()`](https://github.com/rucio/rucio/blob/420d5ee55f525c620bdc5decfef2f4a269dbf2e5/lib/rucio/daemons/badreplicas/minos.py#L138) are:
* `_ = 'srm'`
* `tmp_dict_rse = {}`
* `tmp_unknown_replicas = {'unknown': ['srm://example.com/foo/bar/baz']}`

However, the `for` loop that follows goes over the items in `tmp_dict_rse`. Since it’s empty, the PFNs are not processed.

Modification
------------

As far as I can tell, [line 143](https://github.com/rucio/rucio/blob/420d5ee55f525c620bdc5decfef2f4a269dbf2e5/lib/rucio/daemons/badreplicas/minos.py#L143) need to be moved out of that inner `for` loop.


Changed files:
lib/rucio/daemons/badreplicas/minos.py

diff --git a/lib/rucio/daemons/badreplicas/minos.py b/lib/rucio/daemons/badreplicas/minos.py
index a4ff1555e..1a57a9a95 100644
--- a/lib/rucio/daemons/badreplicas/minos.py
+++ b/lib/rucio/daemons/badreplicas/minos.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2018-2020 CERN
+# Copyright 2018-2021 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -22,6 +22,7 @@
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
+# - Dimitrios Christidis <dimitrios.christidis@cern.ch>, 2021
 
 from __future__ import division
 
@@ -140,7 +141,7 @@ def minos(bulk=1000, once=False, sleep_time=60):
                             if rse_id not in dict_rse:
                                 dict_rse[rse_id] = []
                             dict_rse[rse_id].extend(tmp_dict_rse[rse_id])
-                            unknown_replicas.extend(tmp_unknown_replicas.get('unknown', []))
+                        unknown_replicas.extend(tmp_unknown_replicas.get('unknown', []))
                     # The replicas in unknown_replicas do not exist, so we flush them from bad_pfns
                     if unknown_replicas:
                         logging.info(prepend_str + 'The following replicas are unknown and will be removed : %s' % str(unknown_replicas))
Repository: rucio/rucio
Issue: Protocol mismatch for multisource transfers

Motivation
----------
There seems to be a bug with protocol mismatch which somehow skips the existing Rucio protocol mapping checks and submits mismatched protocols. 


Modification
------------
Possibly an issue in the core.transfer loop with caching.


Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 587b4a070..d59c2abf6 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -660,7 +660,6 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
     multi_hop_dict = {}
 
     rse_mapping = {}
-    current_schemes = SUPPORTED_PROTOCOLS
 
     multihop_rses = []
     try:
@@ -917,7 +916,7 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     file_metadata['previous_attempt_id'] = previous_attempt_id
 
                 transfers[req_id] = {'request_id': req_id,
-                                     'schemes': __add_compatible_schemes(schemes=[destination_protocol], allowed_schemes=current_schemes),
+                                     'schemes': __add_compatible_schemes(schemes=[destination_protocol], allowed_schemes=SUPPORTED_PROTOCOLS),
                                      'account': account,
                                      # 'src_urls': [source_url],
                                      'sources': [(rse, source_url, source_rse_id, ranking if ranking is not None else 0, link_ranking)],
Repository: rucio/rucio
Issue: Conveyor wrongly force tranfers to use IPv4

Motivation
----------
Conveyor wrongly force tranfers to use IPv4




Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 011eea28f..f5f0493ff 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -1069,7 +1069,8 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                 # transfers[id]['src_urls'].append((source_rse_id, source_url))
                 transfers[req_id]['sources'].append((rse, source_url, source_rse_id, ranking, link_ranking))
                 # if one source has force IPv4, force IPv4 for the whole job
-                if rse_attrs[source_rse_id].get('use_ipv4', True):
+                use_ipv4 = rse_attrs[source_rse_id].get('use_ipv4', False)
+                if use_ipv4:
                     transfers[req_id]['use_ipv4'] = True
 
         except Exception:
Repository: rucio/rucio
Issue: conveyor submitter crashes on python3

Motivation
----------
The submitter crashed when run with python3 with the following message. This doesn't happen in python2.


```
CRITICAL        Thread [0/73] :  Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/submitter.py", line 170, in submitter
    retry_other_fts=retry_other_fts)
  File "/usr/local/lib/python3.6/site-packages/rucio/daemons/conveyor/submitter.py", line 337, in __get_transfers
    failover_schemes=failover_schemes)
  File "/usr/local/lib/python3.6/site-packages/rucio/db/sqla/session.py", line 377, in new_funct
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rucio/core/transfer.py", line 723, in get_transfer_requests_and_source_replicas
    session=session)
  File "/usr/local/lib/python3.6/site-packages/rucio/db/sqla/session.py", line 393, in new_funct
    result = function(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rucio/core/transfer.py", line 582, in get_hops
    scheme=limit_dest_schemes if out_v == dest_rse_id and limit_dest_schemes else None)
  File "/usr/local/lib/python3.6/site-packages/rucio/rse/rsemanager.py", line 712, in find_matching_scheme
    dest_candidates = sorted(dest_candidates, key=lambda k: k['domains'][domain][operation_dest])
TypeError: '<' not supported between instances of 'NoneType' and 'int'
```




Changed files:
lib/rucio/rse/rsemanager.py

diff --git a/lib/rucio/rse/rsemanager.py b/lib/rucio/rse/rsemanager.py
index e6f3c804c..61cc82f30 100644
--- a/lib/rucio/rse/rsemanager.py
+++ b/lib/rucio/rse/rsemanager.py
@@ -17,7 +17,7 @@
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2012-2015
 # - Vincent Garonne <vincent.garonne@cern.ch>, 2012-2018
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2012-2020
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2012-2019
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2012-2020
 # - Yun-Pin Sun <winter0128@gmail.com>, 2013
 # - Wen Guan <wen.guan@cern.ch>, 2014-2017
 # - Martin Barisits <martin.barisits@cern.ch>, 2017-2020
@@ -681,7 +681,8 @@ def find_matching_scheme(rse_settings_dest, rse_settings_src, operation_src, ope
             if protocol['scheme'] not in scheme:
                 tbr.append(protocol)
                 continue
-        if protocol['domains'].get(domain, {}).get(operation_src, 1) == 0:
+        prot = protocol['domains'].get(domain, {}).get(operation_src, 1)
+        if prot is None or prot == 0:
             tbr.append(protocol)
     for r in tbr:
         src_candidates.remove(r)
@@ -696,7 +697,8 @@ def find_matching_scheme(rse_settings_dest, rse_settings_src, operation_src, ope
             if protocol['scheme'] not in scheme:
                 tbr.append(protocol)
                 continue
-        if protocol['domains'].get(domain, {}).get(operation_dest, 1) == 0:
+        prot = protocol['domains'].get(domain, {}).get(operation_dest, 1)
+        if prot is None or prot == 0:
             tbr.append(protocol)
     for r in tbr:
         dest_candidates.remove(r)
Repository: rucio/rucio
Issue: Fix wrong grouping for the rules in dirac add_files

Motivation
----------
Rule is has parameter `GROUPING` instead of `grouping`. Need to be fixed

Changed files:
lib/rucio/core/dirac.py

diff --git a/lib/rucio/core/dirac.py b/lib/rucio/core/dirac.py
index 8b00161d4..daf604ef8 100644
--- a/lib/rucio/core/dirac.py
+++ b/lib/rucio/core/dirac.py
@@ -97,7 +97,7 @@ def add_files(lfns, account, ignore_availability, session=None):
                     account=InternalAccount(account),
                     statuses=None,
                     meta=None,
-                    rules=[{'copies': 1, 'rse_expression': 'ANY=true', 'weight': None, 'account': InternalAccount(account), 'lifetime': None, 'GROUPING': 'NONE'}],
+                    rules=[{'copies': 1, 'rse_expression': 'ANY=true', 'weight': None, 'account': InternalAccount(account), 'lifetime': None, 'grouping': 'NONE'}],
                     lifetime=None,
                     dids=None,
                     rse_id=None,
Repository: rucio/rucio
Issue: Clean URLs coming from AWS

Motivation
----------

Have to remove signature for successful registration replica matching:

https://github.com/rucio/rucio/blob/master/lib/rucio/common/utils.py#L668

Changed files:
lib/rucio/common/utils.py

diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
index 277871e78..5b7ed6779 100644
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -670,6 +670,8 @@ def clean_surls(surls):
             surl = re.sub('/srm/managerv2\\?SFN=', '', surl)  # NOQA: W605
         if surl.startswith('https://storage.googleapis.com'):
             surl = surl.split('?GoogleAccessId')[0]
+        if '?X-Amz' in surl:
+            surl = surl.split('?X-Amz')[0]
         res.append(surl)
     res.sort()
     return res
Repository: rucio/rucio
Issue: No proper source evaluation for multi-hop

Motivation
----------
Multi hops transfer do not properly evaluate sources. Instead the algorithm will use the last source retrieved from the database, even if this one is worse (e.g. a tape source)

Modification
------------
Proper evaluation is needed.



Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 320a32851..d72034f5a 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -1265,6 +1265,7 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     # We make the assumption that the hop is never made through TAPE
                     transfers[req_id]['file_metadata']['src_type'] = 'DISK'
                     transfers[req_id]['sources'] = [(source_rse_name, source_url, source_rse_id, 0, 0)]
+                    transfers[req_id]['bring_online'] = bring_online
         if req_id in reqs_no_source:
             reqs_no_source.remove(req_id)
         if req_id in reqs_only_tape_source:
Repository: rucio/rucio
Issue: xrootd: stat broken on py3

Motivation
----------
The python3 response from the stat is of type byte, which requires a modification of split() and decode(). This is backward compatible to py2.

Changed files:
lib/rucio/rse/protocols/xrootd.py

diff --git a/lib/rucio/rse/protocols/xrootd.py b/lib/rucio/rse/protocols/xrootd.py
index 60e192234..a020ec863 100644
--- a/lib/rucio/rse/protocols/xrootd.py
+++ b/lib/rucio/rse/protocols/xrootd.py
@@ -1,4 +1,5 @@
-# Copyright 2013-2020 CERN for the benefit of the ATLAS collaboration.
+# -*- coding: utf-8 -*-
+# Copyright 2013-2020 CERN
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,14 +18,12 @@
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2013
 # - Cheng-Hsi Chao <cheng-hsi.chao@cern.ch>, 2014
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2016-2020
-# - Nicolo Magini <Nicolo.Magini@cern.ch>, 2018
+# - Nicolo Magini <nicolo.magini@cern.ch>, 2018
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018
 # - Andrew Lister <andrew.lister@stfc.ac.uk>, 2019
-# - Tomas Javurek <tomasjavurek09@gmail.com>, 2020
+# - Tomas Javurek <tomas.javurek@cern.ch>, 2020
 # - Martin Barisits <martin.barisits@cern.ch>, 2020
-#  - Eli Chadwick, <eli.chadwick@stfc.ac.uk>, 2020
-#
-# PY3K COMPATIBLE
+# - Eli Chadwick <eli.chadwick@stfc.ac.uk>, 2020
 
 import os
 import logging
@@ -115,14 +114,16 @@ class Default(protocol.RSEProtocol):
             self.logger.info('xrootd.stat: filesize cmd: {}'.format(cmd))
             status_stat, out, err = execute(cmd)
             if status_stat == 0:
-                ret['filesize'] = out.split('\\n')[2].split()[-1]
+                ret['filesize'] = out.split(b'\\n')[2].split()[-1].decode()
 
             # xrdfs query checksum for getting checksum
             cmd = 'XrdSecPROTOCOL=gsi xrdfs %s:%s query checksum %s' % (self.hostname, self.port, path)
             self.logger.info('xrootd.stat: checksum cmd: {}'.format(cmd))
             status_query, out, err = execute(cmd)
             if status_query == 0:
-                chsum, value = out.strip('\\n').split()
+                chsum, value = out.strip(b'\\n').split()
+                chsum = chsum.decode()
+                value = value.decode()
                 ret[chsum] = value
 
         except Exception as e:
Repository: united-manufacturing-hub/united-manufacturing-hub
Issue: [Barcodereader] Add support for numpad

### Expected Behavior

When i connect a barcodereader i want that the keyboard input is displayed on mqtt.

### Current Behavior

When connecting a barcodereader that is emulating a keyboard with numpar the output is "null"

### Relevant log output

_No response_

### Steps to Reproduce

1. Install a mitutoyo barcodesanner
2. He will emulate numbers with num 1 etc.
3. The Output of barcodereader is null

### Context (Environment)

I am using the UMH in Version 0.8.5 with a mitutoyo barcoderader

### Possible Solution

ad scancodes for numpad

### Version

v0.8.x

### Anything else?

_No response_

Changed files:
barcodereader/src/main.py

diff --git a/barcodereader/src/main.py b/barcodereader/src/main.py
index b3ab63b1..1865321a 100644
--- a/barcodereader/src/main.py
+++ b/barcodereader/src/main.py
@@ -66,7 +66,9 @@ scancodes = {
     20: u't', 21: u'y', 22: u'u', 23: u'i', 24: u'o', 25: u'p', 26: u'[', 27: u']', 28: u'CRLF', 29: u'LCTRL',
     30: u'a', 31: u's', 32: u'd', 33: u'f', 34: u'g', 35: u'h', 36: u'j', 37: u'k', 38: u'l', 39: u';',
     40: u'"', 41: u'`', 42: u'LSHFT', 43: u'\\\\', 44: u'z', 45: u'x', 46: u'c', 47: u'v', 48: u'b', 49: u'n',
-    50: u'm', 51: u',', 52: u'.', 53: u'/', 54: u'RSHFT', 56: u'LALT', 57: u' ', 100: u'RALT'
+    50: u'm', 51: u',', 52: u'.', 53: u'/', 54: u'RSHFT', 56: u'LALT', 57: u' ', 71: u'num 7',
+    72: u'num 8', 73: u'num 9', 75: u'num 4',  76: u'num 5',  77: u'num 6', 79: u'num 1', 80: u'num 2', 81: u'num 3', 82: u'num 0',
+    86: u'num .',100: u'RALT'
 }
 
 capscodes = {
Repository: tencent-quantum-lab/tensorcircuit
Issue: Unexpected warning when calculating expectation with readout error

## Issue Description

Unexpected warning when calculating expectation with readout error.
The problem is at
https://github.com/tencent-quantum-lab/tensorcircuit/blob/b413089bd514c90b7998b9d2ca9125d5debd9849/tensorcircuit/noisemodel.py#L282

When counting gates, ``standardize_gate`` is called.

## How to Reproduce

```python
c = DMCircuit(2)

nc = NoiseConf()
re = 0.95
nc.add_noise("readout", [[re, re],[re, re]])
c.sample_expectation_ps([0], shots=1024, noise_conf=nc)
```

### Error Output

```python
WARNING:tensorcircuit.abstractcircuit:gate name not in the common gate set that tc supported
```

## Environment Context

commit df6679561e71cbc3eec1d3f5f2c731b008591a8f


Changed files:
tensorcircuit/noisemodel.py, tensorcircuit/abstractcircuit.py

diff --git a/tensorcircuit/abstractcircuit.py b/tensorcircuit/abstractcircuit.py
index ce08da1..f17680a 100644
--- a/tensorcircuit/abstractcircuit.py
+++ b/tensorcircuit/abstractcircuit.py
@@ -453,7 +453,9 @@ class AbstractCircuit:
                 name = g1
                 break
         if name not in sgates + vgates + mpogates:
-            logger.warning("gate name not in the common gate set that tc supported")
+            logger.warning(
+                "gate name %s not in the common gate set that tc supported" % name
+            )
         return name
 
     def gate_count(self, gate_list: Optional[Sequence[str]] = None) -> int:
diff --git a/tensorcircuit/noisemodel.py b/tensorcircuit/noisemodel.py
index 6f50beb..a222a23 100644
--- a/tensorcircuit/noisemodel.py
+++ b/tensorcircuit/noisemodel.py
@@ -279,7 +279,10 @@ def sample_expectation_ps_noisfy(
     if noise_conf is None:
         noise_conf = NoiseConf()
 
-    num_quantum = c.gate_count(list(noise_conf.nc.keys()))
+    lgate = list(noise_conf.nc.keys())
+    if "readout" in lgate:
+        lgate.remove("readout")
+    num_quantum = c.gate_count(lgate)
 
     if noise_conf.has_readout is True:
         readout_error = noise_conf.nc["readout"]["Default"]