internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
Fix for update_replicas_states function description in replicaclient.py
Motivation
----------
Function description seems to have been copy-pasted from somewhere without changes. 


Discussion:

Resolving patch:
diff --git a/lib/rucio/client/replicaclient.py b/lib/rucio/client/replicaclient.py
--- a/lib/rucio/client/replicaclient.py
+++ b/lib/rucio/client/replicaclient.py
@@ -225,9 +225,16 @@ def update_replicas_states(self, rse, files):
 
         :param rse: the RSE name.
         :param files: The list of files. This is a list of DIDs like :
-            [{'scope': <scope1>, 'name': <name1>}, {'scope': <scope2>, 'name': <name2>}, ...]
-
-        :return: True if files have been deleted successfully.
+            [{'scope': <scope1>, 'name': <name1>, 'state': <state1>}, {'scope': <scope2>, 'name': <name2>, 'state': <state2>}, ...],
+            where a state value can be either of:
+            'A' (available)
+            'S' (suspicious)
+            'U' (unavailable)
+            'R' (recovered)
+            'B' (bad)
+            'L' (lost)
+            'D' (deleted)
+        :return: True if replica states have been updated successfully, otherwise an exception is raised.
 
         """
         url = build_url(choice(self.list_hosts), path=self.REPLICAS_BASEURL)


Test patch:
Repository: rucio/rucio

Issue:
Detection of client location fails on IPv6-only node
Motivation
----------

```
Traceback (most recent call last):
  File "./t", line 5, in <module>
    s.connect(("8.8.8.8", 80))
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 101] Network is unreachable

Traceback (most recent call last):
  File "./t", line 5, in <module>
    s.connect(("8.8.8.8", 80))
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 101] Network is unreachable
```


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -1,4 +1,4 @@
-# Copyright 2012-2018 CERN for the benefit of the ATLAS collaboration.
+# Copyright 2012-2019 CERN for the benefit of the ATLAS collaboration.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,7 @@
 # Authors:
 # - Vincent Garonne <vgaronne@gmail.com>, 2012-2018
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2012-2018
-# - Mario Lassnig <mario.lassnig@cern.ch>, 2012-2018
+# - Mario Lassnig <mario.lassnig@cern.ch>, 2012-2019
 # - Cedric Serfon <cedric.serfon@cern.ch>, 2013-2017
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2013
 # - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2015-2018
@@ -656,8 +656,8 @@ class Color:
 
 def detect_client_location():
     """
-    Open a UDP socket to a machine on the internet, to get the local IP address
-    of the requesting client.
+    Open a UDP socket to a machine on the internet, to get the local IPv4 and IPv6
+    addresses of the requesting client.
 
     Try to determine the sitename automatically from common environment variables,
     in this order: SITE_NAME, ATLAS_SITE_NAME, OSG_SITE_NAME. If none of these exist
@@ -672,12 +672,21 @@ def detect_client_location():
     except Exception:
         pass
 
+    ip6 = '::'
+    try:
+        s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
+        s.connect(("2001:4860:4860:0:0:0:0:8888", 80))
+        ip6 = s.getsockname()[0]
+    except Exception:
+        pass
+
     site = os.environ.get('SITE_NAME',
                           os.environ.get('ATLAS_SITE_NAME',
                                          os.environ.get('OSG_SITE_NAME',
                                                         'ROAMING')))
 
     return {'ip': ip,
+            'ip6': ip6,
             'fqdn': socket.getfqdn(),
             'site': site}
 


Test patch:
Repository: rucio/rucio

Issue:
missing default value for filter options in downloadclient
Motivation
----------
There should be a default value if no filter option is passed to the downloadclient.


Modification
------------
Change https://github.com/rucio/rucio/blob/master/lib/rucio/client/downloadclient.py#L1071 to `item.get('filters', {})`




Discussion:

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -1068,7 +1068,7 @@ def _resolve_and_merge_input_items(self, items):
         while len(items) > 0:
             item = items.pop()
 
-            filters = item.get('filters')
+            filters = item.get('filters', {})
             item_dids = item.pop('did')
             if item_dids[0] is None:
                 logger.debug('Resolving DIDs by using filter options')


Test patch:
diff --git a/lib/rucio/tests/test_download.py b/lib/rucio/tests/test_download.py
new file mode 100644
--- /dev/null
+++ b/lib/rucio/tests/test_download.py
@@ -0,0 +1,53 @@
+# Copyright 2012-2018 CERN for the benefit of the ATLAS collaboration.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Authors:
+# - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
+#
+# PY3K COMPATIBLE
+
+from nose.tools import assert_true
+
+from rucio.client.downloadclient import DownloadClient
+from rucio.common.utils import generate_uuid
+from rucio.tests.common import execute, file_generator
+
+
+class TestDownloadClient(object):
+
+    def test_download_item(self):
+        """ DOWNLOAD (CLIENT): download DIDs. """
+        download_client = DownloadClient()
+        tmp_file1 = file_generator()
+        scope = 'mock'
+        name = tmp_file1[5:]
+        uuid = generate_uuid()
+        cmd = 'rucio upload --rse {0} --scope {1} --guid {2} {3}'.format('MOCK4', scope, uuid, tmp_file1)
+        exitcode, out, err = execute(cmd)
+
+        # Download specific DID
+        result = download_client.download_dids([{'did': '%s:%s' % (scope, name)}])
+        assert_true(result)
+
+        # Download with wildcard
+        result = download_client.download_dids([{'did': '%s:%s' % (scope, name[:10] + '*')}])
+        assert_true(result)
+
+        # Download with filter
+        result = download_client.download_dids([{'filters': {'guid': uuid, 'scope': scope}}])
+        assert_true(result)
+
+        # Download with wildcard and name
+        result = download_client.download_dids([{'did': '%s:%s' % (scope, '*'), 'filters': {'guid': uuid}}])
+        assert_true(result)
Repository: rucio/rucio

Issue:
generic metadata problems
Motivation
----------

Adding generic metadata with add-did-meta does not work, and requesting with get-did-meta should not fail with "No DID found", when instead it should be "No generic metadata found."


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -23,6 +23,7 @@
 # - Joaquin Bogado <jbogado@linti.unlp.edu.ar>, 2014-2015
 # - Wen Guan <wguan.icedew@gmail.com>, 2015
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018-2019
+# - Tobias Wegner <twegner@cern.ch>, 2019
 #
 # PY3K COMPATIBLE
 
@@ -1326,7 +1327,7 @@ def get_did_meta(scope, name, session=None):
         meta = getattr(row, 'meta')
         return json.loads(meta) if session.bind.dialect.name in ['oracle', 'sqlite'] else meta
     except NoResultFound:
-        raise exception.DataIdentifierNotFound("Data identifier '%(scope)s:%(name)s' not found" % locals())
+        raise exception.DataIdentifierNotFound("No generic metadata found for '%(scope)s:%(name)s'" % locals())
 
 
 @transactional_session
@@ -1339,7 +1340,7 @@ def add_did_meta(scope, name, meta, session=None):
     :param meta: the metadata to be added or updated
     """
     if session.bind.dialect.name == 'oracle':
-        oracle_version = int(session.connection.connection.version.split('.')[0])
+        oracle_version = int(session.connection().connection.version.split('.')[0])
         if oracle_version < 12:
             raise NotImplementedError
 
@@ -1385,7 +1386,7 @@ def delete_did_meta(scope, name, key, session=None):
     :param key: the key to be deleted
     """
     if session.bind.dialect.name == 'oracle':
-        oracle_version = int(session.connection.connection.version.split('.')[0])
+        oracle_version = int(session.connection().connection.version.split('.')[0])
         if oracle_version < 12:
             raise NotImplementedError
 
@@ -1410,7 +1411,7 @@ def delete_did_meta(scope, name, key, session=None):
 
         row.meta = existing_meta
     except NoResultFound:
-        raise exception.DataIdentifierNotFound("Data identifier '%(scope)s:%(name)s' not found" % locals())
+        raise exception.DataIdentifierNotFound("Key not found for data identifier '%(scope)s:%(name)s'" % locals())
 
 
 @read_session
@@ -1426,7 +1427,7 @@ def list_dids_by_meta(scope, select, session=None):
     if session.bind.dialect.name == 'sqlite':
         raise NotImplementedError
     if session.bind.dialect.name == 'oracle':
-        oracle_version = int(session.connection.connection.version.split('.')[0])
+        oracle_version = int(session.connection().connection.version.split('.')[0])
         if oracle_version < 12:
             raise NotImplementedError
 


Test patch:
Repository: rucio/rucio

Issue:
Historical view for used space by account and RSE
Motivation
----------

One needs a view to monitor the evolution of used space by account for a given or a set of RSEs.





Discussion:

Resolving patch:
diff --git a/lib/rucio/web/ui/main.py b/lib/rucio/web/ui/main.py
--- a/lib/rucio/web/ui/main.py
+++ b/lib/rucio/web/ui/main.py
@@ -19,7 +19,7 @@
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2014-2018
 # - Martin Barisits <martin.barisits@cern.ch>, 2014-2015
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2015
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2015-2018
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2015-2019
 # - Stefan Prenner <stefan.prenner@cern.ch>, 2018
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
 #
@@ -83,6 +83,7 @@
     ATLAS_URLS = (
         '/', 'AtlasIndex',
         '/account_usage', 'AccountUsage',
+        '/account_usage_history', 'AccountUsageHistory',
         '/dumps', 'Dumps',
         '/accounting', 'Accounting',
         '/conditions_summary', 'Cond',
@@ -112,6 +113,14 @@ def GET(self):  # pylint:disable=no-self-use,invalid-name
         return check_token(render.account_usage())
 
 
+class AccountUsageHistory(object):
+    """ Group Account Usage overview """
+    def GET(self):  # pylint:disable=no-self-use,invalid-name
+        """ GET """
+        render = template.render(join(dirname(__file__), 'templates/'))
+        return check_token(render.account_usage_history())
+
+
 class AccountRSEUsage(object):
     """ RSE usage per account  """
     def GET(self):  # pylint:disable=no-self-use,invalid-name


Test patch:
Repository: rucio/rucio

Issue:
Historical view for used space by account and RSE
Motivation
----------

One needs a view to monitor the evolution of used space by account for a given or a set of RSEs.



Demo docs pointing to wrong location
Motivation
----------

`https://rucio.readthedocs.io/en/latest/rucio_demo.html`

The first link "setting up demo..." is 404, should point to `https://hub.docker.com/r/rucio/demo`


Discussion:

Resolving patch:
diff --git a/lib/rucio/web/ui/main.py b/lib/rucio/web/ui/main.py
--- a/lib/rucio/web/ui/main.py
+++ b/lib/rucio/web/ui/main.py
@@ -19,7 +19,7 @@
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2014-2018
 # - Martin Barisits <martin.barisits@cern.ch>, 2014-2015
 # - Ralph Vigne <ralph.vigne@cern.ch>, 2015
-# - Cedric Serfon <cedric.serfon@cern.ch>, 2015-2018
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2015-2019
 # - Stefan Prenner <stefan.prenner@cern.ch>, 2018
 # - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2019
 #
@@ -83,6 +83,7 @@
     ATLAS_URLS = (
         '/', 'AtlasIndex',
         '/account_usage', 'AccountUsage',
+        '/account_usage_history', 'AccountUsageHistory',
         '/dumps', 'Dumps',
         '/accounting', 'Accounting',
         '/conditions_summary', 'Cond',
@@ -112,6 +113,14 @@ def GET(self):  # pylint:disable=no-self-use,invalid-name
         return check_token(render.account_usage())
 
 
+class AccountUsageHistory(object):
+    """ Group Account Usage overview """
+    def GET(self):  # pylint:disable=no-self-use,invalid-name
+        """ GET """
+        render = template.render(join(dirname(__file__), 'templates/'))
+        return check_token(render.account_usage_history())
+
+
 class AccountRSEUsage(object):
     """ RSE usage per account  """
     def GET(self):  # pylint:disable=no-self-use,invalid-name


Test patch:
Repository: rucio/rucio

Issue:
Historical view for used space by account and RSE
Motivation
----------

One needs a view to monitor the evolution of used space by account for a given or a set of RSEs.



Fix grammatical errors and typos 
Motivation
----------
My first contribution towards rucio


Modification
------------
Fixed grammatical errors and typos in `database.rst` and `installing_daemons.rst`




Discussion:

Resolving patch:
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -128,6 +128,7 @@ def upload(self, items, summary_file_path=None):
         # clear this set again to ensure that we only try to register datasets once
         registered_dataset_dids = set()
         num_succeeded = 0
+        summary = []
         for file in files:
             basename = file['basename']
             logger.info('Preparing upload for file %s' % basename)
@@ -185,7 +186,6 @@ def upload(self, items, summary_file_path=None):
             protocols = rsemgr.get_protocols_ordered(rse_settings=rse_settings, operation='write', scheme=force_scheme)
             protocols.reverse()
             success = False
-            summary = []
             state_reason = ''
             while not success and len(protocols):
                 protocol = protocols.pop()


Test patch:
Repository: rucio/rucio

Issue:
New check_accept_header_wrapper block auto-documentation of REST endpoints
Motivation
----------
The addition of the `check_accept_header_wrapper` decorator method in the flask rest api blocks the auto-documentation of the GET endpoints. These endpoints (mostly GET) are now removed from readthedocs. Needs to be fixed.



Discussion:

Resolving patch:
diff --git a/lib/rucio/web/rest/flaskapi/v1/common.py b/lib/rucio/web/rest/flaskapi/v1/common.py
--- a/lib/rucio/web/rest/flaskapi/v1/common.py
+++ b/lib/rucio/web/rest/flaskapi/v1/common.py
@@ -20,6 +20,7 @@
 # PY3K COMPATIBLE
 
 from __future__ import print_function
+from functools import wraps
 from flask import request
 from time import time
 from traceback import format_exc
@@ -69,6 +70,7 @@ def after_request(response):
 def check_accept_header_wrapper_flask(supported_content_types):
     """ Decorator to check if an endpoint supports the requested content type. """
     def wrapper(f):
+        @wraps(f)
         def decorated(*args, **kwargs):
             requested_content_type = request.environ.get('HTTP_ACCEPT')
             request_type_allowed = True


Test patch:
Repository: rucio/rucio

Issue:
Sorting of replicas in legacy RSE field corrupted
Motivation
----------

The 'RSE' field in the list_replicas reply is still used by the pilot. In the case of multiple different DIDs requested in a single list_replicas call, the sorting per RSE will only be correct for the last RSE if there are multiple potential RSEs.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -1017,11 +1017,18 @@ def _list_replicas(dataset_clause, file_clause, state_clause, show_pfns,
                     for i in range(0, len(tmp)):
                         file['pfns'][tmp[i][2]]['priority'] = i + 1
                         file['rses'] = {}
-                        for t_rse, t_pfn in [(file['pfns'][t_pfn]['rse'], t_pfn) for t_pfn in file['pfns']]:
+
+                        rse_pfns = []
+                        for t_rse, t_priority, t_pfn in [(file['pfns'][t_pfn]['rse'], file['pfns'][t_pfn]['priority'], t_pfn) for t_pfn in file['pfns']]:
+                            rse_pfns.append((t_rse, t_priority, t_pfn))
+                        rse_pfns = sorted(rse_pfns)
+
+                        for t_rse, t_priority, t_pfn in rse_pfns:
                             if t_rse in file['rses']:
                                 file['rses'][t_rse].append(t_pfn)
                             else:
                                 file['rses'][t_rse] = [t_pfn]
+
                     yield file
                     file = {}
 
@@ -1071,7 +1078,7 @@ def _list_replicas(dataset_clause, file_clause, state_clause, show_pfns,
         rse_pfns = []
         for t_rse, t_priority, t_pfn in [(file['pfns'][t_pfn]['rse'], file['pfns'][t_pfn]['priority'], t_pfn) for t_pfn in file['pfns']]:
             rse_pfns.append((t_rse, t_priority, t_pfn))
-            rse_pfns = sorted(rse_pfns)
+        rse_pfns = sorted(rse_pfns)
 
         for t_rse, t_priority, t_pfn in rse_pfns:
             if t_rse in file['rses']:


Test patch:
Repository: rucio/rucio

Issue:
config cache prevents updating a value before expiration
Motivation
----------
```python
config.set('throttler', 'release_strategy', 'fifo')
config.set('throttler', 'release_strategy', 'grouped_fifo')
```
will result in a database duplicate exception because `config.set()` checks if there is already a value which will be `False` if you try to set a new value under the expiration time because it got set to `False` with the first command.


Modification
------------
change the `has_option()` check in `config.set()` to `has_option(use_cache=False)`




Discussion:

Resolving patch:
diff --git a/lib/rucio/core/config.py b/lib/rucio/core/config.py
--- a/lib/rucio/core/config.py
+++ b/lib/rucio/core/config.py
@@ -200,7 +200,7 @@ def set(section, option, value, session=None):
     :param session: The database session in use.
     """
 
-    if not has_option(section=section, option=option, session=session):
+    if not has_option(section=section, option=option, use_cache=False, session=session):
         new_option = models.Config(section=section, opt=option, value=value)
         new_option.save(session=session)
     else:


Test patch:
Repository: rucio/rucio

Issue:
propagate correct ignore_checksum for archive download
Motivation
----------



Modification
------------





Discussion:

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -666,9 +666,12 @@ def _download_item(self, item, trace, log_prefix=''):
                 if success and not item.get('merged_options', {}).get('ignore_checksum', False):
                     rucio_checksum = item.get('adler32')
                     local_checksum = None
-                    if not rucio_checksum:
+                    if rucio_checksum is None:
                         rucio_checksum = item.get('md5')
-                        local_checksum = md5(temp_file_path)
+                        if rucio_checksum is None:
+                            logger.warning('%sNo remote checksum available. Skipping validation.' % log_prefix)
+                        else:
+                            local_checksum = md5(temp_file_path)
                     else:
                         local_checksum = adler32(temp_file_path)
 
@@ -677,10 +680,6 @@ def _download_item(self, item, trace, log_prefix=''):
                         os.unlink(temp_file_path)
                         logger.warning('%sChecksum validation failed for file: %s' % (log_prefix, did_str))
                         logger.debug('Local checksum: %s, Rucio checksum: %s' % (local_checksum, rucio_checksum))
-                        try:
-                            self.client.declare_suspicious_file_replicas([pfn], reason='Corrupted')
-                        except Exception:
-                            pass
                         trace['clientState'] = 'FAIL_VALIDATE'
                 if not success:
                     logger.warning('%sDownload attempt failed. Try %s/%s' % (log_prefix, attempt, retries))
@@ -1412,7 +1411,7 @@ def _prepare_items_for_download(self, did_to_options, merged_items_with_sources,
                             'dest_file_paths': [dest_path],
                             'temp_file_path': '%s.part' % dest_path,
                             'sources': file_item['sources'],
-                            'ignore_checksum': True,  # we currently dont have checksums for the archive
+                            'merged_options': {'ignore_checksum': True},  # we currently dont have checksums for the archive
                             'archive_items': []
                             }
                     cea_id_to_pack[cea_id] = pack


Test patch:
Repository: rucio/rucio

Issue:
New psycopg2-binary release incompatible with current sqlalchemy version
Motivation
----------
Yesterday a new release (2.8) of `psycopg2-binary` was published. With this release, the tests do not run through, as exceptions are handled differently with our current version of sqlalchemy.
https://pypi.org/project/psycopg2-binary/2.8/

Modification
------------
For now we will fix `setup_rucio.py` to not install the 2.8 version thus:
`psycopg2-binary>=2.4.2<2.8'

Once we update sqlalchemy to a newer version, this can be relaxed.


Discussion:

Resolving patch:
diff --git a/setup_rucio.py b/setup_rucio.py
--- a/setup_rucio.py
+++ b/setup_rucio.py
@@ -15,7 +15,7 @@
 # Authors:
 # - Vincent Garonne <vgaronne@gmail.com>, 2014-2018
 # - Thomas Beermann <thomas.beermann@cern.ch>, 2014-2018
-# - Martin Barisits <martin.barisits@cern.ch>, 2016-2018
+# - Martin Barisits <martin.barisits@cern.ch>, 2016-2019
 # - Mario Lassnig <mario.lassnig@cern.ch>, 2019
 
 '''
@@ -164,7 +164,7 @@ def write_requirements():
 
 
 oracle_extras = ['cx_oracle==6.3.1']
-postgresql_extras = ['psycopg2-binary>=2.4.2']
+postgresql_extras = ['psycopg2-binary>=2.4.2,<2.8']
 mysql_extras = ['PyMySQL']
 kerberos_extras = ['kerberos>=1.3.0', 'pykerberos>=1.2.1', 'requests-kerberos>=0.12.0']
 dev_extras = parse_requirements(requirements_files=['tools/pip-requires-test', ])


Test patch:
Repository: rucio/rucio

Issue:
Calculation error of progress class for Rule PROGRESS notifications
Motivation
----------
The progress class is calculated like `int(float(replicating_locks) / float(total_locks) * 10) * 10`

Modification
------------
It should be calculated like `int(float(total_locks - replicating_locks) / float(total_locks) * 10) * 10`



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -3047,7 +3047,7 @@ def __progress_class(replicating_locks, total_locks):
     """
 
     try:
-        return int(float(replicating_locks) / float(total_locks) * 10) * 10
+        return int(float(total_locks - replicating_locks) / float(total_locks) * 10) * 10
     except Exception:
         return 0
 


Test patch: