internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
If policy section not available in client config, no exception should be raised
Motivation
----------
There are two `config_get('policy', ...)` calls in the bin/rucio code. However, if the policy section is not available in the config an error is raised. This should not happen. Default values should be used. ("")


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/config.py b/lib/rucio/common/config.py
--- a/lib/rucio/common/config.py
+++ b/lib/rucio/common/config.py
@@ -55,7 +55,7 @@ def config_get(section, option, raise_exception=True, default=None):
     try:
         return __CONFIG.get(section, option)
     except (ConfigParser.NoOptionError, ConfigParser.NoSectionError) as err:
-        if raise_exception:
+        if raise_exception and default is None:
             raise err
         return default
 


Test patch:
Repository: rucio/rucio

Issue:
If policy section not available in client config, no exception should be raised
Motivation
----------
There are two `config_get('policy', ...)` calls in the bin/rucio code. However, if the policy section is not available in the config an error is raised. This should not happen. Default values should be used. ("")


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/config.py b/lib/rucio/common/config.py
--- a/lib/rucio/common/config.py
+++ b/lib/rucio/common/config.py
@@ -55,7 +55,7 @@ def config_get(section, option, raise_exception=True, default=None):
     try:
         return __CONFIG.get(section, option)
     except (ConfigParser.NoOptionError, ConfigParser.NoSectionError) as err:
-        if raise_exception:
+        if raise_exception and default is None:
             raise err
         return default
 
diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -3047,7 +3047,7 @@ def __progress_class(replicating_locks, total_locks):
     """
 
     try:
-        return int(float(replicating_locks) / float(total_locks) * 10) * 10
+        return int(float(total_locks - replicating_locks) / float(total_locks) * 10) * 10
     except Exception:
         return 0
 


Test patch:
Repository: rucio/rucio

Issue:
list_replicas should raise DID not found
Motivation
----------
```
>>> from rucio.client.client import Client
>>> c = Client()
>>> print(c.list_replicas([{'scope':'user.mlassnig', 'name':'afasf'}], metalink=True))
</metalink>
```


Modification
------------





Discussion:
@TWAtGH this will still need an extension in the downloadclient. list_replicas replies with `['']` in the json case (already now), and (after this patch) with `'<?xml version="1.0" encoding="UTF-8"?><metalink xmlns="urn:ietf:params:xml:ns:metalink"></metalink>'` in the metalink case.

Resolving patch:
diff --git a/lib/rucio/web/rest/webpy/v1/replica.py b/lib/rucio/web/rest/webpy/v1/replica.py
--- a/lib/rucio/web/rest/webpy/v1/replica.py
+++ b/lib/rucio/web/rest/webpy/v1/replica.py
@@ -170,7 +170,9 @@ def GET(self, scope, name):
                             break
                     yield ' </file>\n'
 
-            # don't forget to send the metalink footer
+            # ensure complete metalink
+            if __first and metalink:
+                yield '<?xml version="1.0" encoding="UTF-8"?>\n<metalink xmlns="urn:ietf:params:xml:ns:metalink">\n'
             if metalink:
                 yield '</metalink>\n'
 
@@ -454,7 +456,9 @@ def POST(self):
                             break
                     yield ' </file>\n'
 
-            # don't forget to send the metalink footer
+            # ensure complete metalink
+            if __first and metalink:
+                yield '<?xml version="1.0" encoding="UTF-8"?>\n<metalink xmlns="urn:ietf:params:xml:ns:metalink">\n'
             if metalink:
                 yield '</metalink>\n'
 


Test patch:
Repository: rucio/rucio

Issue:
list_replicas should raise DID not found
Motivation
----------
```
>>> from rucio.client.client import Client
>>> c = Client()
>>> print(c.list_replicas([{'scope':'user.mlassnig', 'name':'afasf'}], metalink=True))
</metalink>
```


Modification
------------





Discussion:

Resolving patch:
diff --git a/lib/rucio/web/rest/webpy/v1/replica.py b/lib/rucio/web/rest/webpy/v1/replica.py
--- a/lib/rucio/web/rest/webpy/v1/replica.py
+++ b/lib/rucio/web/rest/webpy/v1/replica.py
@@ -170,7 +170,9 @@ def GET(self, scope, name):
                             break
                     yield ' </file>\n'
 
-            # don't forget to send the metalink footer
+            # ensure complete metalink
+            if __first and metalink:
+                yield '<?xml version="1.0" encoding="UTF-8"?>\n<metalink xmlns="urn:ietf:params:xml:ns:metalink">\n'
             if metalink:
                 yield '</metalink>\n'
 
@@ -454,7 +456,9 @@ def POST(self):
                             break
                     yield ' </file>\n'
 
-            # don't forget to send the metalink footer
+            # ensure complete metalink
+            if __first and metalink:
+                yield '<?xml version="1.0" encoding="UTF-8"?>\n<metalink xmlns="urn:ietf:params:xml:ns:metalink">\n'
             if metalink:
                 yield '</metalink>\n'
 


Test patch:
Repository: rucio/rucio

Issue:
revert xrdcp workaround in downloadclient as soon as gfal is fixed
Motivation
----------
We implemented a workaround (#1596) since gfal doesnt correctly call xroot copy when an archive PFN is given.
This can be removed again as soon as gfal is fixed.

Modification
------------



Discussion:

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -574,8 +574,9 @@ def _download_item(self, item, trace, log_prefix=''):
             self._send_trace(trace)
             return item
 
-        success = False
         # try different PFNs until one succeeded
+        temp_file_path = item['temp_file_path']
+        success = False
         i = 0
         while not success and i < len(sources):
             source = sources[i]
@@ -584,43 +585,6 @@ def _download_item(self, item, trace, log_prefix=''):
             rse_name = source['rse']
             scheme = pfn.split(':')[0]
 
-            # this is a workaround to fix that gfal doesnt use root's -z option for archives
-            # this will be removed as soon as gfal has fixed this
-            temp_file_path = item['temp_file_path']
-            dest_file_path = item['dest_file_paths'][0]
-            unzip_arg_name = '?xrdcl.unzip='
-            if scheme == 'root' and unzip_arg_name in pfn:
-                logger.info('%sFound xrdcl.unzip in PFN. Using xrdcp overwrite.' % log_prefix)
-                filename_in_archive = ''
-                pfn_filename_start = pfn.find(unzip_arg_name) + len(unzip_arg_name)
-                for c in pfn[pfn_filename_start:]:
-                    if c == '&' or c == '?':
-                        break
-                    filename_in_archive += c
-
-                dest_file_path = os.path.join(os.path.dirname(dest_file_path), filename_in_archive)
-                temp_file_path = '%s.part' % dest_file_path
-                cmd = 'xrdcp -vf %s -z %s %s' % (pfn, filename_in_archive, temp_file_path)
-                start_time = time.time()
-                try:
-                    logger.debug('Executing: %s' % cmd)
-                    status, out, err = execute(cmd)
-                except Exception as error:
-                    logger.debug('xrdcp execution failed')
-                    logger.debug(error)
-                    continue
-                end_time = time.time()
-                success = (status == 0)
-                if not success:
-                    logger.debug('xrdcp status: %s' % status)
-                    logger.debug('xrdcp stdout: %s' % out)
-                    logger.debug('xrdcp stderr: %s' % err)
-                    trace['clientState'] = ('%s' % err)
-                    self._send_trace(trace)
-                    continue
-                else:
-                    break
-
             try:
                 rse = rsemgr.get_rse_info(rse_name)
             except RSENotFound:


Test patch:
Repository: rucio/rucio

Issue:
Issue in client_extract download 
Motivation
----------



Modification
------------





Discussion:

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -1314,20 +1314,22 @@ def _prepare_items_for_download(self, did_to_options, merged_items_with_sources,
                 # dont move from mixed list to pure list
                 continue
 
-            # remove mixed cea id from mixed map and add it to pure map
-            cea_id_mixed_to_fiids.pop(cea_id_mixed)
+            # first add cea_id to pure map so it can be removed from mixed map later
             cea_id_pure_to_fiids.setdefault(cea_id_mixed, set()).update(fiids_mixed)
 
-            # move file items from mixed list to pure list
-            for fiid_mixed in fiids_mixed:
+            # now update all file_item mixed/pure maps
+            for fiid_mixed in list(fiids_mixed):
                 file_item = fiid_to_file_item[fiid_mixed]
+                # add cea id to file_item pure map
                 file_item.setdefault('cea_ids_pure', set()).add(cea_id_mixed)
 
-                # remove references from file to mixed archive and from mixed archive to file
-                for cea_id_mixed in file_item.pop('cea_ids_mixed'):
-                    cea_id_mixed_to_fiids[cea_id_mixed].discard(fiid_mixed)
-                    if not len(cea_id_mixed_to_fiids[cea_id_mixed]):
-                        cea_id_mixed_to_fiids.pop(cea_id_mixed)
+                # remove file item mixed map and
+                # remove references from all other mixed archives to file_item
+                for cea_id_mixed2 in file_item.pop('cea_ids_mixed'):
+                    cea_id_mixed_to_fiids[cea_id_mixed2].remove(fiid_mixed)
+
+            # finally remove cea_id from mixed map
+            cea_id_mixed_to_fiids.pop(cea_id_mixed)
 
         for file_item in all_file_items:
             cea_ids_pure = file_item.get('cea_ids_pure', set())


Test patch:
Repository: rucio/rucio

Issue:
Issue in client_extract download 
Motivation
----------



Modification
------------





Discussion:

Resolving patch:
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.19.5',
-    'branch_nick': 'patch-0-1_19_5_preparation',
-    'revision_id': '9e14d56c9d958e5348b19ddc7e5fa45d4a778807',
-    'revno': 7951
+    'version': '1.19.6',
+    'branch_nick': 'patch-0-Release__Rucio_1_19_6_preparation',
+    'revision_id': 'a8c639a7a70a9e605ad90535d28d2eab04d89cce',
+    'revno': 7992
 }


Test patch:
Repository: rucio/rucio

Issue:
docker demo fails to build image due to errors(?) in Dockerfile
Motivation
----------
I tried the docker demo on Ubuntu 18.04 and it fails during the build process due to errors in the docker file, namely this line:
https://github.com/rucio/rucio/blob/master/etc/docker/demo/Dockerfile#L16
I think neither sudo nor the plus sign should be in that line.
It worked after removing those

Modification
------------
remove of `sudo` and the `+` sign from 
https://github.com/rucio/rucio/blob/master/etc/docker/demo/Dockerfile#L16



Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/bb8/common.py b/lib/rucio/daemons/bb8/common.py
--- a/lib/rucio/daemons/bb8/common.py
+++ b/lib/rucio/daemons/bb8/common.py
@@ -54,7 +54,7 @@
                     format='%(asctime)s\t%(process)d\t%(levelname)s\t%(message)s')
 
 
-def rebalance_rule(parent_rule, activity, rse_expression, priority, source_replica_expression=None, comment=None):
+def rebalance_rule(parent_rule, activity, rse_expression, priority, source_replica_expression='*\\bb8-enabled=false', comment=None):
     """
     Rebalance a replication rule to a new RSE
 
@@ -332,7 +332,7 @@ def select_target_rse(parent_rule, current_rse, rse_expression, subscription_id,
 
 
 @transactional_session
-def rebalance_rse(rse, max_bytes=1E9, max_files=None, dry_run=False, exclude_expression=None, comment=None, force_expression=None, mode=None, priority=3, source_replica_expression=None, session=None):
+def rebalance_rse(rse, max_bytes=1E9, max_files=None, dry_run=False, exclude_expression=None, comment=None, force_expression=None, mode=None, priority=3, source_replica_expression='*\\bb8-enabled=false', session=None):
     """
     Rebalance data from an RSE
 


Test patch:
Repository: rucio/rucio

Issue:
wrong python version in python3 travis test
Motivation
----------
The travis tests for python3.5 uses python3 in the container but inside in the rucio container where python2 is still used.


Modification
------------
Use build arguments in the dockerfile to install rucio with python2 or python3 depending on the python version in the travis container.




Discussion:

Resolving patch:
diff --git a/setup_rucio.py b/setup_rucio.py
--- a/setup_rucio.py
+++ b/setup_rucio.py
@@ -17,6 +17,7 @@
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2014-2018
 # - Martin Barisits <martin.barisits@cern.ch>, 2016-2019
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2019
+# - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
 
 '''
 Setup.py which moves kerberos modules in extra dependencies to be buildthedocs compliant.
@@ -81,13 +82,13 @@ def run_git_command(cmd):
 
 if using_pip and using_git:
     git_version_cmd = '''git describe --dirty=-dev`date +%s`'''
-    pkg_version = run_git_command(git_version_cmd)
+    pkg_version = run_git_command(git_version_cmd).decode()
     branch_nick_cmd = 'git branch | grep -Ei "\* (.*)" | cut -f2 -d" "'
-    branch_nick = run_git_command(branch_nick_cmd)
+    branch_nick = run_git_command(branch_nick_cmd).decode()
     revid_cmd = "git rev-parse HEAD"
-    revid = run_git_command(revid_cmd)
+    revid = run_git_command(revid_cmd).decode()
     revno_cmd = "git --no-pager log --oneline | wc -l"
-    revno = run_git_command(revno_cmd)
+    revno = run_git_command(revno_cmd).decode()
     version_file = open("lib/rucio/vcsversion.py", 'w')
     version_file.write("""'''\n"""
                        """This file is automatically generated by setup.py, So don't edit it. :)\n"""
@@ -101,7 +102,7 @@ def run_git_command(cmd):
                        """}""" % (pkg_version, branch_nick, revid, revno))
     version_file.close()
 else:
-    pkg_version = version.version_string()
+    pkg_version = version.version_string().decode()
 
 cmdclass = {}
 


Test patch:
Repository: rucio/rucio

Issue:
When rules with COPYING replicas are cancelled, they should be tombstoned
Motivation
----------
State transition `COPYING` to `UNAVAILABLE` - Replicas should get **epoche** tombstone.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -2903,6 +2903,7 @@ def __delete_lock_and_update_replica(lock, purge_replicas=False, nowait=False, s
                 replica.tombstone = replica.created_at
             if lock.state == LockState.REPLICATING:
                 replica.state = ReplicaState.UNAVAILABLE
+                replica.tombstone = OBSOLETE
                 return True
     except NoResultFound:
         logging.error("Replica for lock %s:%s for rule %s on rse %s could not be found", lock.scope, lock.name, str(lock.rule_id), get_rse_name(lock.rse_id, session=session))


Test patch:
Repository: rucio/rucio

Issue:
Optimize the update of the final states in the necromancer
Motivation
----------
Currently the necromancer update up-to 10000000 bad replicas states in one transaction


Modification
------------
Update by chunks




Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/badreplicas/necromancer.py b/lib/rucio/daemons/badreplicas/necromancer.py
--- a/lib/rucio/daemons/badreplicas/necromancer.py
+++ b/lib/rucio/daemons/badreplicas/necromancer.py
@@ -13,7 +13,7 @@
 # limitations under the License.
 #
 # Authors:
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2014-2017
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2014-2019
 # - Vincent Garonne <vgaronne@gmail.com>, 2015-2018
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2015
 # - Wen Guan <wguan.icedew@gmail.com>, 2015
@@ -21,17 +21,21 @@
 #
 # PY3K COMPATIBLE
 
+from __future__ import division
+
 import logging
 import os
 import socket
 import threading
 import time
 
+from math import ceil
 from sys import exc_info, stdout, argv
 from traceback import format_exception
 
 from rucio.db.sqla.constants import ReplicaState
 from rucio.common.config import config_get
+from rucio.common.utils import chunks
 from rucio.common.exception import DatabaseException
 from rucio.core import monitor, heartbeat
 from rucio.core.replica import list_bad_replicas, get_replicas_state, list_bad_replicas_history, update_bad_replicas_history
@@ -112,11 +116,18 @@ def necromancer(thread=0, bulk=5, once=False):
             now = time.time()
             if (now - update_history_time) > update_history_threshold:
                 logging.info(prepend_str + 'Last update of history table %s seconds ago. Running update.' % (now - update_history_time))
-                bad_replicas = list_bad_replicas_history(limit=10000000,
+                bad_replicas = list_bad_replicas_history(limit=1000000,
                                                          thread=heart_beat['assign_thread'],
                                                          total_threads=heart_beat['nr_threads'])
                 for rse_id in bad_replicas:
-                    update_bad_replicas_history(bad_replicas[rse_id], rse_id)
+                    chunk_size = 1000
+                    nchunk = int(ceil(len(bad_replicas[rse_id]) / chunk_size))
+                    logging.debug(prepend_str + 'Update history for rse_id %s' % (rse_id))
+                    cnt = 0
+                    for chunk in chunks(bad_replicas[rse_id], chunk_size):
+                        logging.debug(prepend_str + ' History for rse_id %s : chunk %i/%i' % (rse_id, cnt, nchunk))
+                        cnt += 1
+                        update_bad_replicas_history(chunk, rse_id)
                 logging.info(prepend_str + 'History table updated in %s seconds' % (time.time() - now))
                 update_history_time = time.time()
 


Test patch:
Repository: rucio/rucio

Issue:
Filter out requests on blacklisted RSE
Motivation
----------
If a site is blacklisted but has a huge number of requests in state `Q` it can lead to starvation on other RSEs.


Modification
------------
Exclude RSEs blacklisted for write from the conveyor loop




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -791,7 +791,6 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
     :param session:          Database session to use.
     :returns:                List.
     """
-
     sub_requests = session.query(models.Request.id,
                                  models.Request.rule_id,
                                  models.Request.scope,
@@ -806,7 +805,10 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
                                  models.Request.retry_count)\
         .with_hint(models.Request, "INDEX(REQUESTS REQUESTS_TYP_STA_UPD_IDX)", 'oracle')\
         .filter(models.Request.state == RequestState.QUEUED)\
-        .filter(models.Request.request_type == RequestType.TRANSFER)
+        .filter(models.Request.request_type == RequestType.TRANSFER)\
+        .join(models.RSE, models.RSE.id == models.Request.dest_rse_id)\
+        .filter(models.RSE.deleted == false())\
+        .filter(models.RSE.availability.in_((0, 1, 4, 5)))
 
     if isinstance(older_than, datetime.datetime):
         sub_requests = sub_requests.filter(models.Request.requested_at < older_than)


Test patch:
Repository: rucio/rucio

Issue:
Filter out requests on blacklisted RSE
Motivation
----------
If a site is blacklisted but has a huge number of requests in state `Q` it can lead to starvation on other RSEs.


Modification
------------
Exclude RSEs blacklisted for write from the conveyor loop


double requirement definition
Motivation
----------
`pip install -r tools/pip-requires-client -r tools/pip-requires -r tools/pip-requires-test` in the dev Dockerfile breaks on next branch with:
`ERROR: Double requirement given: pysftp==0.2.9 (from -r tools/pip-requires (line 11)) (already in pysftp<0.3,>=0.2.9 (from -r tools/pip-requires-client (line 16)), name='pysftp')`


Modification
------------





Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -791,7 +791,6 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
     :param session:          Database session to use.
     :returns:                List.
     """
-
     sub_requests = session.query(models.Request.id,
                                  models.Request.rule_id,
                                  models.Request.scope,
@@ -806,7 +805,10 @@ def __list_transfer_requests_and_source_replicas(total_workers=0, worker_number=
                                  models.Request.retry_count)\
         .with_hint(models.Request, "INDEX(REQUESTS REQUESTS_TYP_STA_UPD_IDX)", 'oracle')\
         .filter(models.Request.state == RequestState.QUEUED)\
-        .filter(models.Request.request_type == RequestType.TRANSFER)
+        .filter(models.Request.request_type == RequestType.TRANSFER)\
+        .join(models.RSE, models.RSE.id == models.Request.dest_rse_id)\
+        .filter(models.RSE.deleted == false())\
+        .filter(models.RSE.availability.in_((0, 1, 4, 5)))
 
     if isinstance(older_than, datetime.datetime):
         sub_requests = sub_requests.filter(models.Request.requested_at < older_than)


Test patch:
Repository: rucio/rucio

Issue:
double requirement definition
Motivation
----------
`pip install -r tools/pip-requires-client -r tools/pip-requires -r tools/pip-requires-test` in the dev Dockerfile breaks on next branch with:
`ERROR: Double requirement given: pysftp==0.2.9 (from -r tools/pip-requires (line 11)) (already in pysftp<0.3,>=0.2.9 (from -r tools/pip-requires-client (line 16)), name='pysftp')`


Modification
------------



Optimize the update of the final states in the necromancer
Motivation
----------
Currently the necromancer update up-to 10000000 bad replicas states in one transaction


Modification
------------
Update by chunks




Discussion:

Resolving patch:
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.20.0rc1',
-    'branch_nick': 'feature-0-Release__Rucio_1_20_0rc1_preparation',
-    'revision_id': '194eed686dee2ce76ad4a60283f62a4e33f9a38f',
-    'revno': 8288
+    'version': '1.20.0',
+    'branch_nick': 'master',
+    'revision_id': '0226a0dbfb3899568dcefd930bf27a7816521b21',
+    'revno': 8293
 }


Test patch: