internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
Human friendly limit setting
Motivation
----------
Right now, setting limits via the command line (`rucio-admin account set-limits`) is a bit of black magic.  You have to specify in bytes (notoriously human-unfriendly) and there's an undocumented magic value of `-1` which is shorthand for "infinity".

Proposed Modification
------------
* Allow the CLI interface to accept units (`10GB` instead of `10000000000`)
* Accept "inf" and "infinity" (case-insensitive).


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/utils.py b/lib/rucio/common/utils.py
--- a/lib/rucio/common/utils.py
+++ b/lib/rucio/common/utils.py
@@ -23,6 +23,7 @@
 # - Frank Berghaus, <frank.berghaus@cern.ch>, 2017
 # - Brian Bockelman <bbockelm@cse.unl.edu>, 2018
 # - Tobias Wegner <twegner@cern.ch>, 2018
+# - Hannes Hansen <hannes.jakob.hansen@cern.ch>, 2018
 
 from __future__ import print_function
 
@@ -723,3 +724,33 @@ def add_url_query(url, query):
     mod_query.update(query)
     url_parts[4] = urllib.urlencode(mod_query)
     return urlparse.urlunparse(url_parts)
+
+
+def get_bytes_value_from_string(input_string):
+    """
+    Get bytes from a string that represents a storage value and unit
+
+    :param input_string: String containing a value and an unit
+    :return: Integer value representing the value in bytes
+    """
+    result = re.findall('^([0-9]+)([A-Za-z]+)$', input_string)
+    if result:
+        value = int(result[0][0])
+        unit = result[0][1].lower()
+        if unit == 'b':
+            value = value
+        elif unit == 'kb':
+            value = value * 1000
+        elif unit == 'mb':
+            value = value * 1000000
+        elif unit == 'gb':
+            value = value * 1000000000
+        elif unit == 'tb':
+            value = value * 1000000000000
+        elif unit == 'pb':
+            value = value * 1000000000000000
+        else:
+            return False
+        return value
+    else:
+        return False


Test patch:
Repository: rucio/rucio

Issue:
Using summary file in upload API will crash
Motivation
----------

Modification
------------



Discussion:

Resolving patch:
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -226,11 +226,11 @@ def upload(self, items, summary_file_path=None):
                 file_scope = file['did_scope']
                 file_name = file['did_name']
                 file_did_str = '%s:%s' % (file_scope, file_name)
-                final_summary[file_did_str] = {'scope': file['scope'],
-                                               'name': file['name'],
+                final_summary[file_did_str] = {'scope': file_scope,
+                                               'name': file_name,
                                                'bytes': file['bytes'],
                                                'rse': file['rse'],
-                                               'pfn': file['upload_result']['pfn'],
+                                               'pfn': file['upload_result'].get('pfn', ''),
                                                'guid': file['meta']['guid'],
                                                'adler32': file['adler32'],
                                                'md5': file['md5']}


Test patch:
Repository: rucio/rucio

Issue:
Wrong handling of RSE restriction which crashes judge-repairer
Motivation
----------
Wrong handling of the replica restriction causes crashes in Judge repairer.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/rule.py b/lib/rucio/core/rule.py
--- a/lib/rucio/core/rule.py
+++ b/lib/rucio/core/rule.py
@@ -2438,6 +2438,7 @@ def __evaluate_did_attach(eval_did, session=None):
                             #     possible_rses.extend(parse_expression(rule.rse_expression, filter={'availability_write': True}, session=session))
                         except (InvalidRSEExpression, RSEBlacklisted):
                             possible_rses = []
+                            break
 
                     source_rses = list(set([rse['id'] for rse in source_rses]))
                     possible_rses = list(set([rse['id'] for rse in possible_rses]))


Test patch:
Repository: rucio/rucio

Issue:
Don't send checksum to FTS if verify_checksum is disabled
Motivation
----------
Currently, when submitting a job to FTS where both RSEs have `"verify_checksum": False` it will be set to `"none"` in the metadata send to FTS but the checksum are still sent. This leads to certain problems with xrootd 3rd copy on DPM where checksum validation is not working. If a checksum is set in the metadata FTS will currently ignore the `"verify_checksum"` and will try to validate anyway.

Modification
------------
Don't send the checksum in the metadata when `"verify_checksum": "none"`



Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -178,7 +178,7 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, fts_source_str
                 'selection_strategy': fts_source_strategy,
                 'request_type': transfer['file_metadata'].get('request_type', None),
                 'activity': str(transfer['file_metadata']['activity'])}
-        if file['metadata'].get('verify_checksum', True):
+        if verify_checksum != 'none':
             if 'md5' in list(file['metadata'].keys()) and file['metadata']['md5']:
                 file['checksum'] = 'MD5:%s' % str(file['metadata']['md5'])
             if 'adler32' in list(file['metadata'].keys()) and file['metadata']['adler32']:


Test patch:
Repository: rucio/rucio

Issue:
Problems with accessing the REST API and the WebUI in the development docker container
Motivation
----------
It is not possible to access the REST API under 'https://localhost/proxy' and to access the Web UI under 'https://localhost/ui/' within the docker container build with the Dockerfile under /etc/docker/dev.

Modification
------------
A few modifications can be copied from the demo environment.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/lifetime_exception.py b/lib/rucio/core/lifetime_exception.py
--- a/lib/rucio/core/lifetime_exception.py
+++ b/lib/rucio/core/lifetime_exception.py
@@ -41,19 +41,11 @@ def list_exceptions(exception_id, states, session=None):
     if states:
         state_clause = [models.LifetimeExceptions.state == state for state in states]
 
-    query = session.query(models.LifetimeExceptions.id,
-                          models.LifetimeExceptions.scope, models.LifetimeExceptions.name,
-                          models.LifetimeExceptions.did_type,
-                          models.LifetimeExceptions.account,
-                          models.LifetimeExceptions.pattern,
-                          models.LifetimeExceptions.comments,
-                          models.LifetimeExceptions.state,
-                          models.LifetimeExceptions.expires_at,
-                          models.LifetimeExceptions.created_at)
+    query = session.query(models.LifetimeExceptions)
     if state_clause != []:
         query = query.filter(or_(*state_clause))
     if exception_id:
-        query = query.filter(id=exception_id)
+        query = query.filter_by(id=exception_id)
 
     for exception in query.yield_per(5):
         yield {'id': exception.id, 'scope': exception.scope, 'name': exception.name,


Test patch:
Repository: rucio/rucio

Issue:
Problems with accessing the REST API and the WebUI in the development docker container
Motivation
----------
It is not possible to access the REST API under 'https://localhost/proxy' and to access the Web UI under 'https://localhost/ui/' within the docker container build with the Dockerfile under /etc/docker/dev.

Modification
------------
A few modifications can be copied from the demo environment.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/lifetime_exception.py b/lib/rucio/core/lifetime_exception.py
--- a/lib/rucio/core/lifetime_exception.py
+++ b/lib/rucio/core/lifetime_exception.py
@@ -7,6 +7,7 @@
 #
 #  Authors:
 #  - Cedric Serfon, <cedric.serfon@cern.ch>, 2016-2018
+#  - Dimitrios Christidis, <dimitrios.christidis@cern.ch> 2018
 
 from re import match
 from datetime import datetime, timedelta
@@ -152,7 +153,7 @@ def update_exception(exception_id, state, session=None):
     """
     query = session.query(models.LifetimeExceptions).filter_by(id=exception_id)
     try:
-        query.one()
+        query.first()
     except NoResultFound:
         raise LifetimeExceptionNotFound
 


Test patch:
Repository: rucio/rucio

Issue:
Client fails if auth type is x509_proxy and X509_USER_PROXY is not set
The client fails if the authentication type is x509_proxy and the X509_USER_PROXY environment variable is not set with a very confusing error message:
```
$ rucio whoami
2018-09-27 09:22:57,749	ERROR	coercing to Unicode: need string or buffer, module found
```



Discussion:
@hahahannes can you please have a look on this. I expect it should be an easy fix.

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -182,7 +182,7 @@ def __init__(self, rucio_host=None, auth_host=None, account=None, ca_cert=None,
                         else:
                             fname = '/tmp/x509up_u%d' % geteuid()
                             if path.exists(fname):
-                                self.creds['client_proxy'] = path
+                                self.creds['client_proxy'] = fname
                             else:
                                 raise MissingClientParameter('Cannot find a valid X509 proxy; not in %s, $X509_USER_PROXY not set, and '
                                                              '\'x509_proxy\' not set in the configuration file.' % fname)


Test patch:
Repository: rucio/rucio

Issue:
Client fails if auth type is x509_proxy and X509_USER_PROXY is not set
The client fails if the authentication type is x509_proxy and the X509_USER_PROXY environment variable is not set with a very confusing error message:
```
$ rucio whoami
2018-09-27 09:22:57,749	ERROR	coercing to Unicode: need string or buffer, module found
```



Discussion:

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -182,7 +182,7 @@ def __init__(self, rucio_host=None, auth_host=None, account=None, ca_cert=None,
                         else:
                             fname = '/tmp/x509up_u%d' % geteuid()
                             if path.exists(fname):
-                                self.creds['client_proxy'] = path
+                                self.creds['client_proxy'] = fname
                             else:
                                 raise MissingClientParameter('Cannot find a valid X509 proxy; not in %s, $X509_USER_PROXY not set, and '
                                                              '\'x509_proxy\' not set in the configuration file.' % fname)
diff --git a/lib/rucio/core/lifetime_exception.py b/lib/rucio/core/lifetime_exception.py
--- a/lib/rucio/core/lifetime_exception.py
+++ b/lib/rucio/core/lifetime_exception.py
@@ -42,19 +42,11 @@ def list_exceptions(exception_id, states, session=None):
     if states:
         state_clause = [models.LifetimeExceptions.state == state for state in states]
 
-    query = session.query(models.LifetimeExceptions.id,
-                          models.LifetimeExceptions.scope, models.LifetimeExceptions.name,
-                          models.LifetimeExceptions.did_type,
-                          models.LifetimeExceptions.account,
-                          models.LifetimeExceptions.pattern,
-                          models.LifetimeExceptions.comments,
-                          models.LifetimeExceptions.state,
-                          models.LifetimeExceptions.expires_at,
-                          models.LifetimeExceptions.created_at)
+    query = session.query(models.LifetimeExceptions)
     if state_clause != []:
         query = query.filter(or_(*state_clause))
     if exception_id:
-        query = query.filter(id=exception_id)
+        query = query.filter_by(id=exception_id)
 
     for exception in query.yield_per(5):
         yield {'id': exception.id, 'scope': exception.scope, 'name': exception.name,


Test patch:
Repository: rucio/rucio

Issue:
Summary function crashes do to wrong dictionary entry
Motivation
----------
```
2018-09-28 06:02:51,797 DEBUG [Traceback (most recent call last):

  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.17.8/bin/rucio", line 154, in new_funct

    return function(*args, **kwargs)

  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.17.8/bin/rucio", line 868, in upload

    upload_client.upload(items, summary_file_path)

  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.17.8/lib/python2.7/site-packages/rucio/client/uploadclient.py", line 229, in upload

    final_summary[file_did_str] = {'scope': file['scope'],

KeyError: 'scope'
```



Discussion:

Resolving patch:
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -226,11 +226,11 @@ def upload(self, items, summary_file_path=None):
                 file_scope = file['did_scope']
                 file_name = file['did_name']
                 file_did_str = '%s:%s' % (file_scope, file_name)
-                final_summary[file_did_str] = {'scope': file['scope'],
-                                               'name': file['name'],
+                final_summary[file_did_str] = {'scope': file_scope,
+                                               'name': file_name,
                                                'bytes': file['bytes'],
                                                'rse': file['rse'],
-                                               'pfn': file['upload_result']['pfn'],
+                                               'pfn': file['upload_result'].get('pfn', ''),
                                                'guid': file['meta']['guid'],
                                                'adler32': file['adler32'],
                                                'md5': file['md5']}


Test patch:
Repository: rucio/rucio

Issue:
rucio download wrongly resolves wildcarded DIDs
Motivation
----------
In 1.17.* the wildcard download stopped working. (Related to re-implementation of downloader)


Discussion:

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -871,7 +871,7 @@ def _prepare_items_for_download(self, items):
                     new_item['type'] = dids['did_type'].upper()
                     new_item['name'] = dids['name']
                     new_item['did'] = '%s:%s' % (did_scope, dids['name'])
-                    resolved_items.append(new_item)
+                    resolved_items.append(copy.deepcopy(new_item))
             else:
                 new_item['type'] = self.client.get_did(did_scope, did_name)['type'].upper()
                 new_item['name'] = did_name


Test patch:
Repository: rucio/rucio

Issue:
group-policy option is not propagated properly to the conveyor
Motivation
----------
The `--group-policy` option is not propagated properly to the conveyor.


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -258,6 +258,9 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, fts_source_str
             if policy == 'activity_dest':
                 policy_key = '%s %s' % (activity, file['metadata']['dst_rse'])
                 policy_key = "_".join(policy_key.split(' '))
+            if policy == 'activity_src_dest':
+                policy_key = '%s %s %s' % (activity, file['metadata']['src_rse'], file['metadata']['dst_rse'])
+                policy_key = "_".join(policy_key.split(' '))
             # maybe here we need to hash the key if it's too long
 
             if USER_TRANSFERS not in ['cms'] or activity not in USER_ACTIVITY:


Test patch:
Repository: rucio/rucio

Issue:
Typo in oracle version check
Motivation
----------
The check for the oracle version needed for the generic meta data has a typo in calling the connection function [here](https://github.com/rucio/rucio/blob/master/lib/rucio/core/did.py#L1291)

Modification
------------
```
session.connection().connection
```
instead of 
```
session.connection.connection
```



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -1288,7 +1288,7 @@ def get_did_meta(scope, name, session=None):
     :param name: the name of the did
     """
     if session.bind.dialect.name == 'oracle':
-        oracle_version = int(session.connection.connection.version.split('.')[0])
+        oracle_version = int(session.connection().connection.version.split('.')[0])
         if oracle_version < 12:
             raise NotImplementedError
 


Test patch:
Repository: rucio/rucio

Issue:
Typo in oracle version check
Motivation
----------
The check for the oracle version needed for the generic meta data has a typo in calling the connection function [here](https://github.com/rucio/rucio/blob/master/lib/rucio/core/did.py#L1291)

Modification
------------
```
session.connection().connection
```
instead of 
```
session.connection.connection
```

Add comments to transmogrifier code
Motivation
----------

there is only few comments right now. Descriptiv comments would make the code much more understandable.



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -1288,7 +1288,7 @@ def get_did_meta(scope, name, session=None):
     :param name: the name of the did
     """
     if session.bind.dialect.name == 'oracle':
-        oracle_version = int(session.connection.connection.version.split('.')[0])
+        oracle_version = int(session.connection().connection.version.split('.')[0])
         if oracle_version < 12:
             raise NotImplementedError
 


Test patch:
Repository: rucio/rucio

Issue:
Typo in oracle version check
Motivation
----------
The check for the oracle version needed for the generic meta data has a typo in calling the connection function [here](https://github.com/rucio/rucio/blob/master/lib/rucio/core/did.py#L1291)

Modification
------------
```
session.connection().connection
```
instead of 
```
session.connection.connection
```



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -1288,7 +1288,7 @@ def get_did_meta(scope, name, session=None):
     :param name: the name of the did
     """
     if session.bind.dialect.name == 'oracle':
-        oracle_version = int(session.connection.connection.version.split('.')[0])
+        oracle_version = int(session.connection().connection.version.split('.')[0])
         if oracle_version < 12:
             raise NotImplementedError
 
diff --git a/lib/rucio/core/rse.py b/lib/rucio/core/rse.py
--- a/lib/rucio/core/rse.py
+++ b/lib/rucio/core/rse.py
@@ -22,6 +22,7 @@
 # - Wen Guan <wguan.icedew@gmail.com>, 2015-2016
 # - Brian Bockelman <bbockelm@cse.unl.edu>, 2018
 # - Frank Berghaus <frank.berghaus@cern.ch>, 2018
+# - Dimitrios Christidis <dimitrios.christidis@cern.ch>, 2018
 
 from re import match
 try:
@@ -519,6 +520,7 @@ def get_rse_usage(rse, source=None, rse_id=None, session=None, per_account=False
         rse_usage = {'rse': rse, 'source': row.source,
                      'used': row.used, 'free': row.free,
                      'total': total,
+                     'files': row.files,
                      'updated_at': row.updated_at}
         if per_account:
             query_account_usage = session.query(models.AccountUsage).filter_by(rse_id=rse_id)


Test patch:
Repository: rucio/rucio

Issue:
Outdated flask api doc string
Motivation
----------
The doc string for this [API](https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/flaskapi/v1/did.py#L106) seems to be outdated.

Modification
------------
I would add the query parameters 'created_before', 'created_after', 'name', 'length', 'length.gt', 'length.lt', 'length.lte' and 'length.gte'.


Discussion:

Resolving patch:
diff --git a/lib/rucio/web/rest/flaskapi/v1/did.py b/lib/rucio/web/rest/flaskapi/v1/did.py
--- a/lib/rucio/web/rest/flaskapi/v1/did.py
+++ b/lib/rucio/web/rest/flaskapi/v1/did.py
@@ -113,7 +113,7 @@ def get(self, scope):
 
         .. sourcecode:: http
 
-            GET /dids/scope1/dids/search?type=collection1&long=True HTTP/1.1
+            GET /dids/scope1/dids/search?type=collection&long=True&length.lt=10 HTTP/1.1
             Host: rucio.cern.ch
 
         **Example response**:
@@ -131,7 +131,15 @@ def get(self, scope):
 
         :query type: specify a DID type to search for
         :query long: set to True for long output, otherwise only name
-        :query recursive: recursively list DIDs content
+        :query recursive: set to True to recursively list DIDs content
+        :query created_before: Date string in RFC-1123 format where the creation date was earlier
+        :query created_after: Date string in RFC-1123 format where the creation date was later
+        :query length: Exact number of attached DIDs
+        :query length.gt: Number of attached DIDs greater than
+        :query length.lt: Number of attached DIDs less than
+        :query length.gte: Number of attached DIDs greater than or equal to
+        :query length.lte: Number of attached DIDs less than or equal to
+        :query name: Name or pattern of a DID name
         :resheader Content-Type: application/x-json-stream
         :status 200: DIDs found
         :status 401: Invalid Auth Token


Test patch: