anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: rucio/rucio
Issue: importlib not python 2.6 compatible

Motivation
----------
For the dynamic lfn2pfn function the module `importlib` is used. However, importlib is not part of python 2.6, thus it breaks the clients used on the ATLAS local root base. One option would be to use the backport of importlib for python 2.6 (on pypi) - needs to be investigated.

https://pypi.python.org/pypi/importlib

* Adding importlib to the dependencies works for python 2.6 and 2.7, but a install on python 3.x fails like this. 

For now we move the import to a conditional part of the code, as no module gets dynamically loaded at the moment anyway; This needs to be addressed later on with the dynamic loading of permissions and policies, as this should be handled in a similar way.

Changed files:
lib/rucio/rse/protocols/protocol.py

diff --git a/lib/rucio/rse/protocols/protocol.py b/lib/rucio/rse/protocols/protocol.py
index fdcedefd6..a054ad292 100644
--- a/lib/rucio/rse/protocols/protocol.py
+++ b/lib/rucio/rse/protocols/protocol.py
@@ -13,13 +13,13 @@ Authors:
 - Cheng-Hsi Chao, <cheng-hsi.chao@cern.ch>, 2014
 - Mario Lassnig, <mario.lassnig@cern.ch>, 2017
 - Brian Bockelman, <bbockelm@cse.unl.edu>, 2018
+- Martin Barisits, <martin.barisits@cern.ch>, 2018
 
 This module defines the base class for implementing a transfer protocol,
 along with some of the default methods for LFN2PFN translations.
 """
 
 import hashlib
-import importlib
 
 from urlparse import urlparse
 from ConfigParser import NoOptionError, NoSectionError
@@ -143,7 +143,11 @@ class RSEDeterministicTranslation(object):
         except (NoOptionError, NoSectionError):
             pass
         if policy_module:
+            # TODO: The import of importlib is done like this due to a dependency issue with python 2.6 and incompatibility of the module with py3.x
+            # More information https://github.com/rucio/rucio/issues/875
+            import importlib
             importlib.import_module(policy_module)
+
         cls._DEFAULT_LFN2PFN = config.get_lfn2pfn_algorithm_default()
 
     def path(self, scope, name):
Repository: rucio/rucio
Issue: Upload fails to delete left-overs

Motivation
----------
deletion left-overs of previous attempts of upload is silently failing.
https://github.com/rucio/rucio/blob/master/lib/rucio/rse/rsemanager.py#L409
consequently, upload of current attempt fails. 

Modification
------------
Under investigation. 

Changed files:
lib/rucio/rse/rsemanager.py

diff --git a/lib/rucio/rse/rsemanager.py b/lib/rucio/rse/rsemanager.py
index 5d21fb864..4b3350b9c 100755
--- a/lib/rucio/rse/rsemanager.py
+++ b/lib/rucio/rse/rsemanager.py
@@ -406,9 +406,11 @@ def upload(rse_settings, lfns, source_dir=None, force_pfn=None, force_scheme=Non
             else:
                 if protocol.exists('%s.rucio.upload' % pfn):  # Check for left over of previous unsuccessful attempts
                     try:
-                        protocol_delete.delete('%s.rucio.upload', list(protocol_delete.lfns2pfns(make_valid_did(lfn)).values())[0])
+                        protocol_delete.delete('%s.rucio.upload' % list(protocol_delete.lfns2pfns(make_valid_did(lfn)).values())[0])
                     except Exception as e:
                         ret['%s:%s' % (scope, name)] = exception.RSEOperationNotSupported('Unable to remove temporary file %s.rucio.upload: %s' % (pfn, str(e)))
+                        gs = False
+                        continue
                 try:  # Try uploading file
                     protocol.put(base_name, '%s.rucio.upload' % pfn, source_dir, transfer_timeout=transfer_timeout)
                 except Exception as e:
Repository: rucio/rucio
Issue: common/utils fails when loading dysfunctional paramiko version

Motivation
----------
Paramiko needs to be imported in a try/except block.


Changed files:
lib/rucio/common/utils.py

diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
index 4f05f0ff1..b7677b17f 100644
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -97,7 +97,10 @@ if EXTRA_MODULES['web']:
     from web import HTTPError
 
 if EXTRA_MODULES['paramiko']:
-    from paramiko import RSAKey
+    try:
+        from paramiko import RSAKey
+    except Exception:
+        EXTRA_MODULES['paramiko'] = False
 
 if EXTRA_MODULES['flask']:
     from flask import Response
@@ -660,7 +663,7 @@ def ssh_sign(private_key, message):
     :return: Base64 encoded signature as a string.
     """
     if not EXTRA_MODULES['paramiko']:
-        raise MissingModuleException('The paramiko module is not installed.')
+        raise MissingModuleException('The paramiko module is not installed or faulty.')
     sio_private_key = StringIO(private_key)
     priv_k = RSAKey.from_private_key(sio_private_key)
     sio_private_key.close()
Repository: rucio/rucio
Issue: rest: metalink in replica frontend badly constructed

Motivation
----------

The metalink frontend is still using the old RSE scheme instead of the new PFN scheme.

Changed files:
lib/rucio/web/rest/webpy/v1/replica.py

diff --git a/lib/rucio/web/rest/webpy/v1/replica.py b/lib/rucio/web/rest/webpy/v1/replica.py
index 85404269a..158aeacce 100755
--- a/lib/rucio/web/rest/webpy/v1/replica.py
+++ b/lib/rucio/web/rest/webpy/v1/replica.py
@@ -357,10 +357,9 @@ class ListReplicas(RucioController):
 
                     replicas = []
                     dictreplica = {}
-                    for rse in rfile['rses']:
-                        for replica in rfile['rses'][rse]:
-                            replicas.append(replica)
-                            dictreplica[replica] = (rfile['pfns'][replica]['domain'], rfile['pfns'][replica]['priority'], rse)
+                    for replica in rfile['pfns'].keys():
+                        replicas.append(replica)
+                        dictreplica[replica] = (rfile['pfns'][replica]['domain'], rfile['pfns'][replica]['priority'], rfile['pfns'][replica]['rse'])
 
                     yield ' <file name="' + rfile['name'] + '">\\n'
                     yield '  <identity>' + rfile['scope'] + ':' + rfile['name'] + '</identity>\\n'
Repository: rucio/rucio
Issue: Fix activity in BB8

Motivation
----------

BB8 uses activity `Data Rebalancing` but the activity defined in ATLAS schema is `Data rebalancing`. We should use the same activity everywhere, and it should be consistent with the share defined in FTS

Changed files:
lib/rucio/daemons/bb8/common.py

diff --git a/lib/rucio/daemons/bb8/common.py b/lib/rucio/daemons/bb8/common.py
index 6a2185f62..1fe2a33c1 100644
--- a/lib/rucio/daemons/bb8/common.py
+++ b/lib/rucio/daemons/bb8/common.py
@@ -243,7 +243,7 @@ def rebalance_rse(rse, max_bytes=1E9, max_files=None, dry_run=False, exclude_exp
                 # Rebalance this rule
                 if not dry_run:
                     child_rule_id = rebalance_rule(parent_rule_id=rule_id,
-                                                   activity='Data Rebalancing',
+                                                   activity='Data rebalancing',
                                                    rse_expression=target_rse_exp,
                                                    priority=priority,
                                                    source_replica_expression=source_replica_expression,
Repository: rucio/rucio
Issue: Link_ranking based sorting in case of multipe tape replicas is done wrong

Motivation
----------
Sorting in case of multiple tape replicas should be done in reverse order, 1 being best, 11 being worst. Also cases where the ranking is 0 should be skipped.


Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index b76bff3f5..c3ae3e0f9 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -708,7 +708,7 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                 if source_rse_id is None or rse is None:
                     continue
 
-                if link_ranking is None:
+                if link_ranking is None or link_ranking == 0:
                     logging.debug("Request %s: no link from %s to %s" % (id, source_rse_id, dest_rse_id))
                     continue
 
@@ -785,7 +785,7 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                     else:
                         # the sources already founded is Tape too.
                         # multiple Tape source replicas are not allowed in FTS3.
-                        if transfers[id]['sources'][0][3] > ranking or (transfers[id]['sources'][0][3] == ranking and transfers[id]['sources'][0][4] >= link_ranking):
+                        if transfers[id]['sources'][0][3] > ranking or (transfers[id]['sources'][0][3] == ranking and transfers[id]['sources'][0][4] <= link_ranking):
                             continue
                         else:
                             transfers[id]['sources'] = []
Repository: rucio/rucio
Issue: `import rucio.core.rule` fails with missing section

The work in #339 caused a regression if the `permission` section does not exist.  From the logs:

```
mod_wsgi (pid=18803): Exception occurred processing WSGI script '/usr/lib/python2.7/site-packages/rucio/web/rest/did.py'.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/web/rest/did.py", line 22, in <module>
    from rucio.api.did import (add_did, add_dids, list_content, list_content_history,
  File "/usr/lib/python2.7/site-packages/rucio/api/did.py", line 19, in <module>
    from rucio.core import did, naming_convention, meta as meta_core
  File "/usr/lib/python2.7/site-packages/rucio/core/did.py", line 35, in <module>
    import rucio.core.rule
  File "/usr/lib/python2.7/site-packages/rucio/core/rule.py", line 2888, in <module>
    def archive_localgroupdisk_datasets(scope, name, session=None):
  File "/usr/lib/python2.7/site-packages/rucio/common/policy.py", line 78, in policy_filter
    policy = get_policy()
  File "/usr/lib/python2.7/site-packages/rucio/common/policy.py", line 40, in get_policy
    policy = config_get('permission', 'policy')
  File "/usr/lib/python2.7/site-packages/rucio/common/config.py", line 30, in config_get
    return __CONFIG.get(section, option)
  File "/usr/lib64/python2.7/ConfigParser.py", line 607, in get
    raise NoSectionError(section)
NoSectionError: No section: 'permission'
```

Seems that this exception type is simply not caught.

Changed files:
lib/rucio/common/policy.py

diff --git a/lib/rucio/common/policy.py b/lib/rucio/common/policy.py
index b0c3efc7e..62403709c 100644
--- a/lib/rucio/common/policy.py
+++ b/lib/rucio/common/policy.py
@@ -38,7 +38,7 @@ def get_policy():
     if isinstance(policy, NoValue):
         try:
             policy = config_get('permission', 'policy')
-        except NoOptionError:
+        except (NoOptionError, NoSectionError):
             policy = 'atlas'
         REGION.set('policy', policy)
     return policy
Repository: rucio/rucio
Issue: Crash when trying to find matching scheme

From the CMS instance, I noticed this exception thrown in the `conveyor-transfer-submitter` after creating some minimal RSEs and trying to trigger a transfer
```
2018-01-11 20:59:36,983	4217	CRITICAL	Exception happened when trying to get transfer for request 8a818df9ca4047e3a6b36f95b98a5c7e: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/core/transfer.py", line 567, in get_transfer_requests_and_source_replicas
    scheme=current_schemes)
  File "/usr/lib/python2.7/site-packages/rucio/rse/rsemanager.py", line 640, in find_matching_scheme
    dest_candidates = sorted(dest_candidates, key=lambda k: k['domains'][domain][operation_dest])
  File "/usr/lib/python2.7/site-packages/rucio/rse/rsemanager.py", line 640, in <lambda>
    dest_candidates = sorted(dest_candidates, key=lambda k: k['domains'][domain][operation_dest])
KeyError: None
```


Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index c3ae3e0f9..35b6cd1e6 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -7,7 +7,7 @@
 #
 # Authors:
 # - Martin Barisits, <martin.barisits@cern.ch>, 2017
-# - Mario Lassnig, <mario.lassnig@cern.ch>, 2017
+# - Mario Lassnig, <mario.lassnig@cern.ch>, 2017-2018
 
 import datetime
 import json
@@ -558,12 +558,13 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                 allow_tape_source = True
 
                 # Find matching scheme between destination and source
+                # TODO: In the future, move to domain "third_party_copy"
                 try:
                     matching_scheme = rsemgr.find_matching_scheme(rse_settings_dest=rses_info[dest_rse_id],
                                                                   rse_settings_src=rses_info[source_rse_id],
                                                                   operation_src='read',
                                                                   operation_dest='write',
-                                                                  domain=None,
+                                                                  domain='wan',
                                                                   scheme=current_schemes)
                 except RSEProtocolNotSupported:
                     logging.error('Operation "write" not supported by %s with schemes %s' % (rses_info[dest_rse_id]['rse'], current_schemes))
Repository: rucio/rucio
Issue: Move rule permission check is not working

Motivation
----------
For rucio move_rule the permission check is not working, thus the default permission is used.


Changed files:
lib/rucio/core/permission/atlas.py

diff --git a/lib/rucio/core/permission/atlas.py b/lib/rucio/core/permission/atlas.py
index 1092445b5..7792b2b28 100644
--- a/lib/rucio/core/permission/atlas.py
+++ b/lib/rucio/core/permission/atlas.py
@@ -11,7 +11,7 @@
 # - Yun-Pin Sun, <yun-pin.sun@cern.ch>, 2012-2013
 # - Ralph Vigne, <ralph.vigne@cern.ch>, 2013
 # - Mario Lassnig, <mario.lassnig@cern.ch>, 2013-2015, 2017
-# - Martin Barisits, <martin.barisits@cern.ch>, 2013-2016
+# - Martin Barisits, <martin.barisits@cern.ch>, 2013-2018
 # - Cedric Serfon, <cedric.serfon@cern.ch>, 2013-2017
 # - Joaquin Bogado, <joaquin.bogado@cern.ch>, 2015
 
@@ -60,6 +60,7 @@ def has_permission(issuer, action, kwargs):
             'approve_rule': perm_approve_rule,
             'update_subscription': perm_update_subscription,
             'reduce_rule': perm_reduce_rule,
+            'move_rule': perm_move_rule,
             'get_auth_token_user_pass': perm_get_auth_token_user_pass,
             'get_auth_token_gss': perm_get_auth_token_gss,
             'get_auth_token_x509': perm_get_auth_token_x509,
Repository: rucio/rucio
Issue: New psycopg2-binary release incompatible with current sqlalchemy version

Motivation
----------
Yesterday a new release (2.8) of `psycopg2-binary` was published. With this release, the tests do not run through, as exceptions are handled differently with our current version of sqlalchemy.
https://pypi.org/project/psycopg2-binary/2.8/

Modification
------------
For now we will fix `setup_rucio.py` to not install the 2.8 version thus:
`psycopg2-binary>=2.4.2<2.8'

Once we update sqlalchemy to a newer version, this can be relaxed.

Changed files:
setup_rucio.py

diff --git a/setup_rucio.py b/setup_rucio.py
index d7e4d3a6a..7729b7c82 100644
--- a/setup_rucio.py
+++ b/setup_rucio.py
@@ -15,7 +15,7 @@
 # Authors:
 # - Vincent Garonne <vgaronne@gmail.com>, 2014-2018
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2014-2018
-# - Martin Barisits <martin.barisits@cern.ch>, 2016-2018
+# - Martin Barisits <martin.barisits@cern.ch>, 2016-2019
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2019
 
 '''
@@ -164,7 +164,7 @@ def write_requirements():
 
 
 oracle_extras = ['cx_oracle==6.3.1']
-postgresql_extras = ['psycopg2-binary>=2.4.2']
+postgresql_extras = ['psycopg2-binary>=2.4.2,<2.8']
 mysql_extras = ['PyMySQL']
 kerberos_extras = ['kerberos>=1.3.0', 'pykerberos>=1.2.1', 'requests-kerberos>=0.12.0']
 dev_extras = parse_requirements(requirements_files=['tools/pip-requires-test', ])
Repository: rucio/rucio
Issue: New check_accept_header_wrapper block auto-documentation of REST endpoints

Motivation
----------
The addition of the `check_accept_header_wrapper` decorator method in the flask rest api blocks the auto-documentation of the GET endpoints. These endpoints (mostly GET) are now removed from readthedocs. Needs to be fixed.


Changed files:
lib/rucio/web/rest/flaskapi/v1/common.py

diff --git a/lib/rucio/web/rest/flaskapi/v1/common.py b/lib/rucio/web/rest/flaskapi/v1/common.py
index 719076d4e..359c23180 100755
--- a/lib/rucio/web/rest/flaskapi/v1/common.py
+++ b/lib/rucio/web/rest/flaskapi/v1/common.py
@@ -20,6 +20,7 @@
 # PY3K COMPATIBLE
 
 from __future__ import print_function
+from functools import wraps
 from flask import request
 from time import time
 from traceback import format_exc
@@ -69,6 +70,7 @@ def after_request(response):
 def check_accept_header_wrapper_flask(supported_content_types):
     """ Decorator to check if an endpoint supports the requested content type. """
     def wrapper(f):
+        @wraps(f)
         def decorated(*args, **kwargs):
             requested_content_type = request.environ.get('HTTP_ACCEPT')
             request_type_allowed = True
Repository: rucio/rucio
Issue: Rucio client package imports rucio.common.schema which is not part of distribution

Motivation
----------
From Isabel:
```
cmd: rucio upload --scope user.itrigger --rse TRIUMF-LCG2_SCRATCHDISK /tmp/itrigger/itrigger/.alrb/testing/tester.NKHUnK/rucio/bash/rucioUpload
2020-04-06 18:30:38,129
INFO Preparing upload for file myFile.d314ec0d-6684-4e3f-8501-47467586e335.log
2020-04-06 18:30:38,293
INFO Successfully added replica in Rucio catalogue at TRIUMF-LCG2_SCRATCHDISK
2020-04-06 18:30:38,842
INFO Successfully added replication rule at TRIUMF-LCG2_SCRATCHDISK
2020-04-06 18:30:39,962
ERROR No module named schema
2020-04-06 18:30:39,963
ERROR 
Rucio exited with an unexpected/unknown error.
Please rerun the last command with the "-v" option to gather more information.
If you're sure there is a problem with Rucio itself, please followup at: https://github.com/rucio/rucio/issues
rucio upload dataset (bash)                                         [FAILED]
```
For some reason the schema module seems to be imported, which is not part of the client distribution.

Modification
------------
We can add it to the distribution, but for me it is not clear why this is even needed, so we should understand this first.


Changed files:
setup_rucio_client.py

diff --git a/setup_rucio_client.py b/setup_rucio_client.py
index 9e8b99437..a5e5d507a 100644
--- a/setup_rucio_client.py
+++ b/setup_rucio_client.py
@@ -14,7 +14,7 @@
 #
 # Authors:
 # - Vincent Garonne <vgaronne@gmail.com>, 2014-2018
-# - Martin Barisits <martin.barisits@cern.ch>, 2017-2019
+# - Martin Barisits <martin.barisits@cern.ch>, 2017-2020
 
 import os
 import re
@@ -39,7 +39,7 @@ sys.path.insert(0, os.path.abspath('lib/'))
 COPY_ARGS = sys.argv[1:]
 NAME = 'rucio-clients'
 IS_RELEASE = False
-PACKAGES = ['rucio', 'rucio.client', 'rucio.common',
+PACKAGES = ['rucio', 'rucio.client', 'rucio.common', 'rucio.common.schema', 
             'rucio.rse.protocols', 'rucio.rse', 'rucio.tests']
 REQUIREMENTS_FILES = ['etc/pip-requires-client']
 DESCRIPTION = "Rucio Client Lite Package"
Repository: rucio/rucio
Issue: Fix source scheme in case of transfers from GCS

Motivation
----------
Fix source scheme in case of transfers from GCS




Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 184b0426e..6b354923c 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -800,9 +800,16 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     if source_protocol in ['davs', 'https']:
                         source_url += '?copy_mode=%s' % WEBDAV_TRANSFER_MODE
 
+                source_sign_url = rse_attrs[source_rse_id].get('sign_url', None)
+                if source_sign_url == 'gcs':
+                    source_url = re.sub('davs', 'gclouds', source_url)
+                    source_url = re.sub('https', 'gclouds', source_url)
+
                 # IV - get external_host + strict_copy
                 strict_copy = rse_attrs[dest_rse_id].get('strict_copy', False)
                 fts_hosts = rse_attrs[dest_rse_id].get('fts', None)
+                if source_sign_url == 'gcs':
+                    fts_hosts = rse_attrs[source_rse_id].get('fts', None)
                 source_globus_endpoint_id = rse_attrs[source_rse_id].get('globus_endpoint_id', None)
                 dest_globus_endpoint_id = rse_attrs[dest_rse_id].get('globus_endpoint_id', None)
 
@@ -946,6 +953,11 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                     if source_protocol in ['davs', 'https']:
                         source_url += '?copy_mode=%s' % WEBDAV_TRANSFER_MODE
 
+                source_sign_url = rse_attrs[source_rse_id].get('sign_url', None)
+                if source_sign_url == 'gcs':
+                    source_url = re.sub('davs', 'gclouds', source_url)
+                    source_url = re.sub('https', 'gclouds', source_url)
+
                 # III - The transfer queued previously is a multihop, but this one is direct.
                 # Reset the sources, remove the multihop flag
                 if transfers[req_id].get('multihop', False):
Repository: rucio/rucio
Issue: Poller is too chatty

Motivation
----------
Dumps internal dictionaries into logfile, blowing up the logfile size on the nodes.


Changed files:
lib/rucio/daemons/conveyor/poller.py

diff --git a/lib/rucio/daemons/conveyor/poller.py b/lib/rucio/daemons/conveyor/poller.py
index 05c921830..8d7f79d67 100644
--- a/lib/rucio/daemons/conveyor/poller.py
+++ b/lib/rucio/daemons/conveyor/poller.py
@@ -1,4 +1,4 @@
-# Copyright 2013-2018 CERN for the benefit of the ATLAS collaboration.
+# Copyright 2013-2020 CERN for the benefit of the ATLAS collaboration.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,7 +13,7 @@
 # limitations under the License.
 #
 # Authors:
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2015
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2013-2020
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2018
 # - Vincent Garonne <vgaronne@gmail.com>, 2014-2018
 # - Wen Guan <wguan.icedew@gmail.com>, 2014-2016
@@ -248,7 +248,6 @@ def poll_transfers(external_host, xfers, prepend_str='', request_ids=None, timeo
                 try:
                     logging.debug(prepend_str + 'Checking %s on %s' % (xfer, external_host))
                     status = transfer_core.bulk_query_transfers(external_host, [xfer, ], TRANSFER_TOOL, timeout)
-                    logging.debug(prepend_str + str(status))
                     if xfer in status and isinstance(status[xfer], Exception):
                         logging.error(prepend_str + 'Problem querying %s on %s . Error returned : %s' % (xfer, external_host, str(status[xfer])))
                 except Exception as err:
Repository: rucio/rucio
Issue: config cache prevents updating a value before expiration

Motivation
----------
```python
config.set('throttler', 'release_strategy', 'fifo')
config.set('throttler', 'release_strategy', 'grouped_fifo')
```
will result in a database duplicate exception because `config.set()` checks if there is already a value which will be `False` if you try to set a new value under the expiration time because it got set to `False` with the first command.


Modification
------------
change the `has_option()` check in `config.set()` to `has_option(use_cache=False)`



Changed files:
lib/rucio/core/config.py

diff --git a/lib/rucio/core/config.py b/lib/rucio/core/config.py
index 7904e6459..a61f32079 100644
--- a/lib/rucio/core/config.py
+++ b/lib/rucio/core/config.py
@@ -200,7 +200,7 @@ def set(section, option, value, session=None):
     :param session: The database session in use.
     """
 
-    if not has_option(section=section, option=option, session=session):
+    if not has_option(section=section, option=option, use_cache=False, session=session):
         new_option = models.Config(section=section, opt=option, value=value)
         new_option.save(session=session)
     else:
Repository: rucio/rucio
Issue: Sorting of replicas in legacy RSE field corrupted

Motivation
----------

The 'RSE' field in the list_replicas reply is still used by the pilot. In the case of multiple different DIDs requested in a single list_replicas call, the sorting per RSE will only be correct for the last RSE if there are multiple potential RSEs.

Changed files:
lib/rucio/core/replica.py

diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
index acb0a1fba..d6b505e9a 100644
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -1017,11 +1017,18 @@ def _list_replicas(dataset_clause, file_clause, state_clause, show_pfns,
                     for i in range(0, len(tmp)):
                         file['pfns'][tmp[i][2]]['priority'] = i + 1
                         file['rses'] = {}
-                        for t_rse, t_pfn in [(file['pfns'][t_pfn]['rse'], t_pfn) for t_pfn in file['pfns']]:
+
+                        rse_pfns = []
+                        for t_rse, t_priority, t_pfn in [(file['pfns'][t_pfn]['rse'], file['pfns'][t_pfn]['priority'], t_pfn) for t_pfn in file['pfns']]:
+                            rse_pfns.append((t_rse, t_priority, t_pfn))
+                        rse_pfns = sorted(rse_pfns)
+
+                        for t_rse, t_priority, t_pfn in rse_pfns:
                             if t_rse in file['rses']:
                                 file['rses'][t_rse].append(t_pfn)
                             else:
                                 file['rses'][t_rse] = [t_pfn]
+
                     yield file
                     file = {}
 
@@ -1071,7 +1078,7 @@ def _list_replicas(dataset_clause, file_clause, state_clause, show_pfns,
         rse_pfns = []
         for t_rse, t_priority, t_pfn in [(file['pfns'][t_pfn]['rse'], file['pfns'][t_pfn]['priority'], t_pfn) for t_pfn in file['pfns']]:
             rse_pfns.append((t_rse, t_priority, t_pfn))
-            rse_pfns = sorted(rse_pfns)
+        rse_pfns = sorted(rse_pfns)
 
         for t_rse, t_priority, t_pfn in rse_pfns:
             if t_rse in file['rses']:
Repository: rucio/rucio
Issue: Calculation error of progress class for Rule PROGRESS notifications

Motivation
----------
The progress class is calculated like `int(float(replicating_locks) / float(total_locks) * 10) * 10`

Modification
------------
It should be calculated like `int(float(total_locks - replicating_locks) / float(total_locks) * 10) * 10`


Changed files:
lib/rucio/core/rule.py

diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
index 1025460ab..8573482aa 100644
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -3047,7 +3047,7 @@ def __progress_class(replicating_locks, total_locks):
     """
 
     try:
-        return int(float(replicating_locks) / float(total_locks) * 10) * 10
+        return int(float(total_locks - replicating_locks) / float(total_locks) * 10) * 10
     except Exception:
         return 0
Repository: rucio/rucio
Issue: If policy section not available in client config, no exception should be raised

Motivation
----------
There are two `config_get('policy', ...)` calls in the bin/rucio code. However, if the policy section is not available in the config an error is raised. This should not happen. Default values should be used. ("")

Changed files:
lib/rucio/common/config.py

diff --git a/lib/rucio/common/config.py b/lib/rucio/common/config.py
index 7348589fa..e6cddc76d 100644
--- a/lib/rucio/common/config.py
+++ b/lib/rucio/common/config.py
@@ -55,7 +55,7 @@ def config_get(section, option, raise_exception=True, default=None):
     try:
         return __CONFIG.get(section, option)
     except (ConfigParser.NoOptionError, ConfigParser.NoSectionError) as err:
-        if raise_exception:
+        if raise_exception and default is None:
             raise err
         return default
Repository: rucio/rucio
Issue: If policy section not available in client config, no exception should be raised

Motivation
----------
There are two `config_get('policy', ...)` calls in the bin/rucio code. However, if the policy section is not available in the config an error is raised. This should not happen. Default values should be used. ("")

Changed files:
lib/rucio/common/config.py, lib/rucio/core/rule.py

diff --git a/lib/rucio/common/config.py b/lib/rucio/common/config.py
index 7348589fa..e6cddc76d 100644
--- a/lib/rucio/common/config.py
+++ b/lib/rucio/common/config.py
@@ -55,7 +55,7 @@ def config_get(section, option, raise_exception=True, default=None):
     try:
         return __CONFIG.get(section, option)
     except (ConfigParser.NoOptionError, ConfigParser.NoSectionError) as err:
-        if raise_exception:
+        if raise_exception and default is None:
             raise err
         return default
 
diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
index 1025460ab..8573482aa 100644
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -3047,7 +3047,7 @@ def __progress_class(replicating_locks, total_locks):
     """
 
     try:
-        return int(float(replicating_locks) / float(total_locks) * 10) * 10
+        return int(float(total_locks - replicating_locks) / float(total_locks) * 10) * 10
     except Exception:
         return 0
Repository: rucio/rucio
Issue: Temporary unavailable files injection fails if duplucates

Motivation
----------
Traceback :
```
Details: [u'(cx_Oracle.IntegrityError) ORA-00001: unique constraint (ATLAS_RUCIO.BAD_PFNS_PK) violated']
Rucio exited with an unexpected/unknown error, please provide the traceback below to the developers.
Traceback (most recent call last):
  File "/usr/bin/rucio-admin", line 129, in new_funct
    return function(*args, **kwargs)
  File "/usr/bin/rucio-admin", line 1034, in declare_temporary_unavailable_replicas
    client.add_bad_pfns(pfns=chunk, reason=args.reason, state='TEMPORARY_UNAVAILABLE', expires_at=expiration_date)
  File "/usr/lib/python2.7/site-packages/rucio/client/replicaclient.py", line 336, in add_bad_pfns
    raise exc_cls(exc_msg)
RucioException: An unknown exception occurred.
Details: [u'(cx_Oracle.IntegrityError) ORA-00001: unique constraint (ATLAS_RUCIO.BAD_PFNS_PK) violated']
```



Changed files:
lib/rucio/core/replica.py

diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
index d109033c1..e5b006452 100644
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -2695,7 +2695,9 @@ def add_bad_pfns(pfns, account, state, reason=None, expires_at=None, session=Non
     pfns = clean_surls(pfns)
     for pfn in pfns:
         new_pfn = models.BadPFNs(path=str(pfn), account=account, state=rep_state, reason=reason, expires_at=expires_at)
+        new_pfn = session.merge(new_pfn)
         new_pfn.save(session=session, flush=False)
+
     try:
         session.flush()
     except IntegrityError as error:
Repository: rucio/rucio
Issue: Reaper crashes when it tries to delete a DIDs that is a constituent of an archive

Motivation
----------
Reaper crashes when it tries to delete a DIDs that is a constituent of an archive







Changed files:
lib/rucio/core/replica.py

diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
index d109033c1..7aa88a723 100644
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -1395,7 +1395,9 @@ def delete_replicas(rse, files, ignore_availability=True, session=None):
                                      ~exists(select([1]).prefix_with("/*+ INDEX(REPLICAS REPLICAS_PK) */", dialect='oracle')).where(and_(models.RSEFileAssociation.scope == file['scope'], models.RSEFileAssociation.name == file['name']))))
 
         did_condition.append(and_(models.DataIdentifier.scope == file['scope'], models.DataIdentifier.name == file['name'], models.DataIdentifier.availability != DIDAvailability.LOST,
-                                  ~exists(select([1]).prefix_with("/*+ INDEX(REPLICAS REPLICAS_PK) */", dialect='oracle')).where(and_(models.RSEFileAssociation.scope == file['scope'], models.RSEFileAssociation.name == file['name']))))
+                                  ~exists(select([1]).prefix_with("/*+ INDEX(REPLICAS REPLICAS_PK) */", dialect='oracle')).where(and_(models.RSEFileAssociation.scope == file['scope'], models.RSEFileAssociation.name == file['name'])),
+                                  ~exists(select([1]).prefix_with("/*+ INDEX(ARCHIVE_CONTENTS ARCH_CONTENTS_PK) */", dialect='oracle')).where(and_(models.ConstituentAssociation.child_scope == file['scope'],
+                                                                                                                                                   models.ConstituentAssociation.child_name == file['name']))))
 
         archive_contents_condition.append(and_(models.ConstituentAssociation.scope == file['scope'],
                                                models.ConstituentAssociation.name == file['name'],
Repository: rucio/rucio
Issue: delete_messages crashes when a message longer than 4000 is provided

Motivation
----------
delete_messages crashes when a message longer than 4000 is provided. The problem is in `session.bulk_insert_mappings(MessageHistory, messages)`. The payload that is provided by Hermes comes from `retrieve_messages`, but in case the payload registered in the DB is `nolimit`, `retrieve_messages` replaces `payload with `payload_nolimit`, generating a Database exception :
```
DatabaseException: Database exception.
Details: (cx_Oracle.DatabaseError) ORA-01461: can bind a LONG value only for insert into a LONG column
[SQL: INSERT INTO atlas_rucio.messages_history (id, event_type, payload, updated_at, created_at) VALUES (:id, :event_type, :payload, :updated_at, :created_at)]
[parameters: xxxx'}"}]
(Background on this error at: http://sqlalche.me/e/4xp6)
```



Changed files:
lib/rucio/core/message.py

diff --git a/lib/rucio/core/message.py b/lib/rucio/core/message.py
index 52b25007f..3e85862e3 100644
--- a/lib/rucio/core/message.py
+++ b/lib/rucio/core/message.py
@@ -143,6 +143,8 @@ def delete_messages(messages, session=None):
     message_condition = []
     for message in messages:
         message_condition.append(Message.id == message['id'])
+        if len(message['payload']) > 4000:
+            message['payload_nolimit'] = message.pop('payload')
 
     try:
         if message_condition:
Repository: rucio/rucio
Issue: Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas

Motivation
----------

Wrong states on RSE avaiability state in __list_transfer_requests_and_source_replicas






Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 9bc1c0143..8b5179897 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -808,7 +808,7 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
         .filter(models.Request.request_type == RequestType.TRANSFER)\\
         .join(models.RSE, models.RSE.id == models.Request.dest_rse_id)\\
         .filter(models.RSE.deleted == false())\\
-        .filter(models.RSE.availability.in_((0, 1, 4, 5)))
+        .filter(models.RSE.availability.in_((2, 3, 6, 7)))
 
     if isinstance(older_than, datetime.datetime):
         sub_requests = sub_requests.filter(models.Request.requested_at < older_than)
Repository: rucio/rucio
Issue: Wrongly handled exception makes the poller crash

Motivation
----------
Traceback :
```
2018-11-15 00:13:40,539 23840   ERROR   Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/daemons/conveyor/poller.py", line 248, in poll_transfers
    for transfer_id in resps:
UnboundLocalError: local variable 'resps' referenced before assignment
```


Changed files:
lib/rucio/daemons/conveyor/poller.py

diff --git a/lib/rucio/daemons/conveyor/poller.py b/lib/rucio/daemons/conveyor/poller.py
index cc30bed34..673a3233a 100644
--- a/lib/rucio/daemons/conveyor/poller.py
+++ b/lib/rucio/daemons/conveyor/poller.py
@@ -232,8 +232,23 @@ def poll_transfers(external_host, xfers, prepend_str='', request_ids=None, timeo
             logging.info(prepend_str + 'Polling %i transfers against %s with timeout %s' % (len(xfers), external_host, timeout))
             resps = transfer_core.bulk_query_transfers(external_host, xfers, 'fts3', timeout)
             record_timer('daemons.conveyor.poller.bulk_query_transfers', (time.time() - tss) * 1000 / len(xfers))
-        except (TransferToolTimeout, TransferToolWrongAnswer) as error:
+        except TransferToolTimeout as error:
             logging.error(prepend_str + str(error))
+            return
+        except TransferToolWrongAnswer as error:
+            logging.error(prepend_str + str(error))
+            logging.error(prepend_str + 'Problem querying %s on %s. All jobs are being checked individually' % (str(xfers), external_host))
+            for xfer in xfers:
+                try:
+                    logging.debug(prepend_str + 'Checking %s on %s' % (xfer, external_host))
+                    status = transfer_core.bulk_query_transfers(external_host, [xfer, ], 'fts3', timeout)
+                    logging.debug(prepend_str + str(status))
+                    if xfer in status and isinstance(status[xfer], Exception):
+                        logging.error(prepend_str + 'Problem querying %s on %s . Error returned : %s' % (xfer, external_host, str(status[xfer])))
+                except Exception as err:
+                    logging.error(prepend_str + 'Problem querying %s on %s . Error returned : %s' % (xfer, external_host, str(err)))
+                    break
+            return
         except RequestException as error:
             logging.error(prepend_str + "Failed to contact FTS server: %s" % (str(error)))
             return
Repository: rucio/rucio
Issue: Auth endpoint in webui broken with reversed DN

Motivation
----------
Sometimes the DN is returned in reversed order to the webui code. This is problematic within the auth endpoint of the webui which is used for the account switcher.


Modification
------------
Fix the DN in case it is reversed.



Changed files:
lib/rucio/web/ui/common/utils.py

diff --git a/lib/rucio/web/ui/common/utils.py b/lib/rucio/web/ui/common/utils.py
index 907f373a0..bee212d90 100644
--- a/lib/rucio/web/ui/common/utils.py
+++ b/lib/rucio/web/ui/common/utils.py
@@ -6,7 +6,7 @@
 # You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 #
 # Authors:
-# - Thomas Beermann, <thomas.beermann@cern.ch>, 2014-2019
+# - Thomas Beermann, <thomas.beermann@cern.ch>, 2014-2020
 # - Ruturaj Gujar, <ruturaj.gujar23@gmail.com>, 2019
 # - Jaroslav Guenther, <jaroslav.guenther@cern.ch>, 2019-2020
 #
@@ -157,6 +157,8 @@ def get_token(token_method, acc=None, idt=None, pwd=None):
         acc = ctx.env.get('HTTP_X_RUCIO_ACCOUNT')
     if not idt:
         idt = ctx.env.get('SSL_CLIENT_S_DN')
+        if not idt.startswith('/'):
+            idt = '/%s' % '/'.join(idt.split(',')[::-1])
     if not (acc and idt):
         return None
     try:
Repository: rucio/rucio
Issue: Injector can crash when split_container option is enabled

Motivation
----------
The `split_container` option is enabled when requests on containers are done via R2D2. It creates rules on the constituent datasets of the container instead of a single rule on the container. 
- It can trigger `IntegrityError` in the case a rule for scope, name, account, rse_expression already exists. 
- Similarly the same error can be raised if 2 constituents contain the same datasets.

Modification
------------
To handle the first issue, the injector must check if a rule on scope, name, account, rse_expression already exists. To handle the second one, it should make a set from the list returned by `list_child_datasets`.



Changed files:
lib/rucio/core/rule.py

diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
index 478f01166..871201db5 100644
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -605,6 +605,9 @@ def inject_rule(rule_id, session=None):
         logging.debug("Creating dataset rules for Split Container rule %s" % (str(rule.id)))
         # Get all child datasets and put rules on them
         dids = [{'scope': dataset['scope'], 'name': dataset['name']} for dataset in rucio.core.did.list_child_datasets(scope=rule.scope, name=rule.name, session=session)]
+        # Remove duplicates from the list of dictionaries
+        dids = [dict(t) for t in {tuple(d.items()) for d in dids}]
+        # Remove dids which already have a similar rule
         dids = [did for did in dids if session.query(models.ReplicationRule).filter_by(scope=did['scope'], name=did['name'], account=rule.account, rse_expression=rule.rse_expression).count() == 0]
         if rule.expires_at:
             lifetime = (rule.expires_at - datetime.utcnow()).days * 24 * 3600 + (rule.expires_at - datetime.utcnow()).seconds
Repository: rucio/rucio
Issue: stager_area RSEs don't work with a non-deterministic source RSE

Motivation
----------

If the source RSE is non-deterministic then even if the deterministic flag is set to false on an associated staging_area the staged replicas don't set the path, making them unusable.

Modification
------------

Set the path when the RSE is non-deterministic.



Changed files:
lib/rucio/daemons/conveyor/finisher.py

diff --git a/lib/rucio/daemons/conveyor/finisher.py b/lib/rucio/daemons/conveyor/finisher.py
index 5549b4d52..89f04e0f3 100644
--- a/lib/rucio/daemons/conveyor/finisher.py
+++ b/lib/rucio/daemons/conveyor/finisher.py
@@ -219,7 +219,7 @@ def __handle_requests(reqs, suspicious_patterns, prepend_str=''):
                 replica['archived'] = False
 
                 # for TAPE, replica path is needed
-                if req['request_type'] == RequestType.TRANSFER and req['dest_rse_id'] in undeterministic_rses:
+                if req['request_type'] in (RequestType.TRANSFER, RequestType.STAGEIN) and req['dest_rse_id'] in undeterministic_rses:
                     if req['dest_rse_id'] not in rses_info:
                         dest_rse = get_rse_name(rse_id=req['dest_rse_id'])
                         rses_info[req['dest_rse_id']] = rsemanager.get_rse_info(dest_rse)
Repository: rucio/rucio
Issue: Conveyor submitter fts error message wrong

Motivation
----------
In https://github.com/rucio/rucio/blob/b8a77623801da54c88b249fbc643657a0a04453d/lib/rucio/core/transfer.py#L576 the error message says the fts host is missing for the source, while it should say it is missing for the destination.

Changed files:
lib/rucio/core/transfer.py

diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
index 118f0fa24..2f72d8ac9 100644
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -573,7 +573,7 @@ def get_transfer_requests_and_source_replicas(total_workers=0, worker_number=0,
                 # get external_host
                 fts_hosts = rse_attrs[dest_rse_id].get('fts', None)
                 if not fts_hosts:
-                    logging.error('Source RSE %s FTS attribute not defined - SKIP REQUEST %s' % (rse, req_id))
+                    logging.error('Destination RSE %s FTS attribute not defined - SKIP REQUEST %s' % (dest_rse, req_id))
                     continue
                 if retry_count is None:
                     retry_count = 0
Repository: rucio/rucio
Issue: Fix for unicode enforced strings brakes Python 2.7 deployment

Motivation
----------
The fix deployed in #3480 apparently brakes Python 2.7 deployments.

Modification
------------

I will put the change back and make it conditional to Python 2.7:
```python
if sys.version_info[0] < 3:
    kw['convert_unicode'] = 'force'
```
This should at least fix it for Python 2.7, possibly for 3.* as well.

Changed files:
lib/rucio/db/sqla/models.py

diff --git a/lib/rucio/db/sqla/models.py b/lib/rucio/db/sqla/models.py
index 3439427c3..675827b3e 100644
--- a/lib/rucio/db/sqla/models.py
+++ b/lib/rucio/db/sqla/models.py
@@ -18,7 +18,7 @@
 # - Angelos Molfetas, <angelos.molfetas@cern.ch>, 2012
 # - Ralph Vigne, <ralph.vigne@cern.ch>, 2013
 # - Cedric Serfon, <cedric.serfon@cern.ch>, 2013-2018
-# - Martin Barisits, <martin.barisits@cern.ch>, 2013-2019
+# - Martin Barisits, <martin.barisits@cern.ch>, 2013-2020
 # - Wen Guan, <wen.guan@cern.ch>, 2015
 # - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2015-2019
 # - Hannes Hansen, <hannes.jakob.hansen@cern.ch>, 2019
@@ -29,6 +29,7 @@
 # PY3K COMPATIBLE
 
 import datetime
+import sys
 import uuid
 
 from six import iteritems
@@ -58,17 +59,20 @@ from rucio.db.sqla.types import InternalScopeString as _InternalScopeString
 # Recipe to force str instead if unicode
 # https://groups.google.com/forum/#!msg/sqlalchemy/8Xn31vBfGKU/bAGLNKapvSMJ
 def String(*arg, **kw):
-    kw['convert_unicode'] = True
+    if sys.version_info[0] < 3:
+        kw['convert_unicode'] = 'force'
     return _String(*arg, **kw)
 
 
 def InternalScopeString(*arg, **kw):
-    kw['convert_unicode'] = True
+    if sys.version_info[0] < 3:
+        kw['convert_unicode'] = 'force'
     return _InternalScopeString(*arg, **kw)
 
 
 def InternalAccountString(*arg, **kw):
-    kw['convert_unicode'] = True
+    if sys.version_info[0] < 3:
+        kw['convert_unicode'] = 'force'
     return _InternalAccountString(*arg, **kw)
Repository: rucio/rucio
Issue: __sort_link_ranking in conveyor is done in wrong order

Motivation
----------
https://github.com/rucio/rucio/blob/defa6efc011bb5c30ddf1177f7d6ea26bc8556b4/lib/rucio/daemons/conveyor/submitter.py#L361
This should not be done in reverse but in a normal sort.

Modification
------------
Remove `reverse=True`



Changed files:
lib/rucio/daemons/conveyor/submitter.py

diff --git a/lib/rucio/daemons/conveyor/submitter.py b/lib/rucio/daemons/conveyor/submitter.py
index 9b9bb3ab0..1aef4f854 100644
--- a/lib/rucio/daemons/conveyor/submitter.py
+++ b/lib/rucio/daemons/conveyor/submitter.py
@@ -363,7 +363,7 @@ def __sort_link_ranking(sources):
             rank_sources[link_ranking] = []
         rank_sources[link_ranking].append(source)
     rank_keys = list(rank_sources.keys())
-    rank_keys.sort(reverse=True)
+    rank_keys.sort()
     for rank_key in rank_keys:
         sources_list = rank_sources[rank_key]
         random.shuffle(sources_list)