internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: rucio/rucio

Issue:
Change the set_new_dids method to set NULL value instead of 0
Motivation
----------
The change done in https://github.com/rucio/rucio/pull/270 has changed the behaviour of the set_new_dids method. Before when new_flag was set to None, the value set in the DB was NULL, now it's 0. It is a problem because this column is indexed.


Modification
------------
Change the set_new_dids method to set NULL value instead of 0


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -7,7 +7,7 @@
   http://www.apache.org/licenses/LICENSE-2.0
 
   Authors:
-  - Vincent Garonne, <vincent.garonne@cern.ch>, 2012-2017
+  - Vincent Garonne, <vincent.garonne@cern.ch>, 2012-2018
   - Mario Lassnig, <mario.lassnig@cern.ch>, 2012-2015, 2017
   - Yun-Pin Sun, <yun-pin.sun@cern.ch>, 2013
   - Cedric Serfon, <cedric.serfon@cern.ch>, 2013-2015, 2017
@@ -788,11 +788,14 @@ def set_new_dids(dids, new_flag, session=None):
     :param new_flag: A boolean to flag new DIDs.
     :param session: The database session in use.
     """
+    if session.bind.dialect.name == 'postgresql':
+        new_flag = bool(new_flag)
     for did in dids:
         try:
+
             rowcount = session.query(models.DataIdentifier).\
                 filter_by(scope=did['scope'], name=did['name']).\
-                update({'is_new': bool(new_flag)}, synchronize_session=False)
+                update({'is_new': new_flag}, synchronize_session=False)
             if not rowcount:
                 raise exception.DataIdentifierNotFound("Data identifier '%s:%s' not found" % (did['scope'], did['name']))
         except DatabaseError as error:


Test patch:
Repository: rucio/rucio

Issue:
Creating incomplete message for datasets does not work
Motivation
----------
The creation of INCOMPLETE messages for datasets whose first file was deleted does not seem to work. This is related to the development done in #69 
There are no INCOMPLETE messages in the message_history to be found.

Expected behavior
-----------------
Generation of INCOMPLETE messages once the first file of a dataset is permanently deleted.


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -27,7 +27,7 @@
 from sqlalchemy import func, and_, or_, exists, not_
 from sqlalchemy.exc import DatabaseError, IntegrityError
 from sqlalchemy.orm.exc import FlushError, NoResultFound
-from sqlalchemy.sql.expression import case, bindparam, select, text, false
+from sqlalchemy.sql.expression import case, bindparam, select, text, false, true
 
 import rucio.core.lock
 
@@ -1180,7 +1180,8 @@ def delete_replicas(rse, files, ignore_availability=True, session=None):
                                               dialect='oracle')).
                            where(and_(models.DataIdentifierAssociation.scope == models.DataIdentifier.scope,
                                       models.DataIdentifierAssociation.name == models.DataIdentifier.name,
-                                      models.DataIdentifier.complete != false())))
+                                      or_(models.DataIdentifier.complete == true(),
+                                          models.DataIdentifier.complete is None))))
                 for parent_scope, parent_name, parent_did_type in modifieds:
                     message = {'scope': parent_scope,
                                'name': parent_name,


Test patch:
Repository: rucio/rucio

Issue:
Conveyor should only use schemes compatible to the destination scheme
_Migrated from JIRA [RUCIO-2817](https://its.cern.ch/jira/browse/RUCIO-2817)_


Discussion:
Right now also schemes compatible to the source schemes are used, which at the moment works out fine, but this should not be done as it could result in conflicts.

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -688,7 +688,7 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                     file_metadata['previous_attempt_id'] = previous_attempt_id
 
                 transfers[id] = {'request_id': id,
-                                 'schemes': __add_compatible_schemes(schemes=[matching_scheme[0], matching_scheme[1]], allowed_schemes=current_schemes),
+                                 'schemes': __add_compatible_schemes(schemes=[matching_scheme[0]], allowed_schemes=current_schemes),
                                  # 'src_urls': [source_url],
                                  'sources': [(rse, source_url, source_rse_id, ranking if ranking is not None else 0, link_ranking)],
                                  'dest_urls': [dest_url],


Test patch:
Repository: rucio/rucio

Issue:
Conveyor should only use schemes compatible to the destination scheme
_Migrated from JIRA [RUCIO-2817](https://its.cern.ch/jira/browse/RUCIO-2817)_


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -688,7 +688,7 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                     file_metadata['previous_attempt_id'] = previous_attempt_id
 
                 transfers[id] = {'request_id': id,
-                                 'schemes': __add_compatible_schemes(schemes=[matching_scheme[0], matching_scheme[1]], allowed_schemes=current_schemes),
+                                 'schemes': __add_compatible_schemes(schemes=[matching_scheme[0]], allowed_schemes=current_schemes),
                                  # 'src_urls': [source_url],
                                  'sources': [(rse, source_url, source_rse_id, ranking if ranking is not None else 0, link_ranking)],
                                  'dest_urls': [dest_url],


Test patch:
Repository: rucio/rucio

Issue:
Contributor guide should only be one line in the main documentation TOC 



Discussion:

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -208,7 +208,10 @@ def _get_exception(self, headers, status_code=None, data=None):
 
         :return: A rucio exception class and an error string.
         """
-        data = parse_response(data)
+        try:
+            data = parse_response(data)
+        except ValueError:
+            data = {}
         if 'ExceptionClass' not in data:
             if 'ExceptionMessage' not in data:
                 human_http_code = _codes.get(status_code, None)  # NOQA, pylint: disable-msg=W0612
diff --git a/lib/rucio/common/schema/cms.py b/lib/rucio/common/schema/cms.py
--- a/lib/rucio/common/schema/cms.py
+++ b/lib/rucio/common/schema/cms.py
@@ -12,6 +12,7 @@
   - Joaquin Bogado, <joaquin.bogado@cern.ch>, 2015
   - Mario Lassnig, <mario.lassnig@cern.ch>, 2015
   - Martin Barisits, <martin.barisits@cern.ch>, 2016
+  - Eric Vaandering, <ewv@fnal.gov>, 2018
 '''
 
 from jsonschema import validate, ValidationError
@@ -80,7 +81,7 @@
 
 RSE = {"description": "RSE name",
        "type": "string",
-       "pattern": "(T[0-3]_[A-Z]{2}_([_-][A-Z0-9]+)*)"}
+       "pattern": "(T[0-3]_[A-Z]{2}_([_-][A-Za-z0-9]+)*)"}
 
 RSE_ATTRIBUTE = {"description": "RSE attribute",
                  "type": "string",
@@ -88,7 +89,7 @@
 
 DEFAULT_RSE_ATTRIBUTE = {"description": "Default RSE attribute",
                          "type": "string",
-                         "pattern": r'([A-Z0-9]+([_-][A-Z0-9]+)*)'}
+                         "pattern": r'([A-Z0-9]+([_-][A-Za-z0-9]+)*)'}
 
 REPLICA_STATE = {"description": "Replica state",
                  "type": "string",


Test patch:
Repository: rucio/rucio

Issue:
Do not require space tokens for SRM
At many CMS sites, space tokens are not used even when SRM is utilized.  However, it's not currently possible to skip this setting.

It may be as simple as setting the space token to the empty string (``), which could be done automatically if not specified.

However, there might be hidden dependencies that want a non-empty-string space token.  Needs investigation.


Discussion:
Currently Rucio unconditionally passes  spacetoken json parameter to FTS job.  No matter what value is passed, this causes fts error on CMS sites which do not support spacetokens.
The desired behavior is: if space_token is not in the 'extended_attributes' , do not add spacetoken to fts json.  
Lines that need fixing:

* https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L171
* https://github.com/rucio/rucio/blob/master/lib/rucio/daemons/conveyor/common.py#L165

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -162,7 +162,7 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, fts_source_str
                 file['checksum'] = 'ADLER32:%s' % str(file['metadata']['adler32'])
 
         job_params = {'verify_checksum': True if file['checksum'] and file['metadata'].get('verify_checksum', True) else False,
-                      'spacetoken': transfer['dest_spacetoken'] if transfer['dest_spacetoken'] else 'null',
+                      'spacetoken': transfer['dest_spacetoken'] if transfer['dest_spacetoken'] else None,
                       'copy_pin_lifetime': transfer['copy_pin_lifetime'] if transfer['copy_pin_lifetime'] else -1,
                       'bring_online': transfer['bring_online'] if transfer['bring_online'] else None,
                       'job_metadata': {'issuer': 'rucio'},  # finaly job_meta will like this. currently job_meta will equal file_meta to include request_id and etc.
diff --git a/lib/rucio/transfertool/fts3.py b/lib/rucio/transfertool/fts3.py
--- a/lib/rucio/transfertool/fts3.py
+++ b/lib/rucio/transfertool/fts3.py
@@ -168,7 +168,7 @@ def submit_transfers(transfers, job_metadata):
                                   'activity': str(transfer['activity']),
                                   'selection_strategy': transfer.get('selection_strategy', 'auto')}],
                        'params': {'verify_checksum': True if transfer['checksum'] else False,
-                                  'spacetoken': transfer['dest_spacetoken'] if transfer['dest_spacetoken'] else 'null',
+                                  'spacetoken': transfer['dest_spacetoken'] if transfer['dest_spacetoken'] else None,
                                   'copy_pin_lifetime': transfer['copy_pin_lifetime'] if transfer['copy_pin_lifetime'] else -1,
                                   'bring_online': transfer['bring_online'] if transfer['bring_online'] else None,
                                   'job_metadata': job_metadata,


Test patch:
Repository: rucio/rucio

Issue:
Do not require space tokens for SRM
At many CMS sites, space tokens are not used even when SRM is utilized.  However, it's not currently possible to skip this setting.

It may be as simple as setting the space token to the empty string (``), which could be done automatically if not specified.

However, there might be hidden dependencies that want a non-empty-string space token.  Needs investigation.


Discussion:
Currently Rucio unconditionally passes  spacetoken json parameter to FTS job.  No matter what value is passed, this causes fts error on CMS sites which do not support spacetokens.
The desired behavior is: if space_token is not in the 'extended_attributes' , do not add spacetoken to fts json.  
Lines that need fixing:

* https://github.com/rucio/rucio/blob/master/lib/rucio/transfertool/fts3.py#L171
* https://github.com/rucio/rucio/blob/master/lib/rucio/daemons/conveyor/common.py#L165

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -208,7 +208,10 @@ def _get_exception(self, headers, status_code=None, data=None):
 
         :return: A rucio exception class and an error string.
         """
-        data = parse_response(data)
+        try:
+            data = parse_response(data)
+        except ValueError:
+            data = {}
         if 'ExceptionClass' not in data:
             if 'ExceptionMessage' not in data:
                 human_http_code = _codes.get(status_code, None)  # NOQA, pylint: disable-msg=W0612


Test patch:
Repository: rucio/rucio

Issue:
add requirements.readthedocs.txt
Motivation
----------
Express all dependencies needed to build the documentation on readthedocs



Discussion:

Resolving patch:
diff --git a/lib/rucio/common/config.py b/lib/rucio/common/config.py
--- a/lib/rucio/common/config.py
+++ b/lib/rucio/common/config.py
@@ -7,7 +7,7 @@
 #
 # Authors:
 # - Mario Lassnig, <mario.lassnig@cern.ch>, 2012
-# - Vincent Garonne, <vincent.garonne@cern.ch>, 2013-2016
+# - Vincent Garonne, <vincent.garonne@cern.ch>, 2013-2018
 # - Cedric Serfon, <cedric.serfon@cern.ch>, 2013
 
 """
@@ -19,6 +19,7 @@
 
 import os
 import json
+import sys
 
 import ConfigParser
 
@@ -123,6 +124,10 @@ def get_rse_credentials(path_to_credentials_file=None):
         break
 
 if not __HAS_CONFIG:
-    raise Exception('Could not load rucio configuration file rucio.cfg. \
-Rucio looks in the following directories for a configuration file, in order:\
-\n\t${RUCIO_HOME}/etc/rucio.cfg\n\t/opt/rucio/etc/rucio.cfg\n\t${VIRTUAL_ENV}/etc/rucio.cfg')
+    if not any("sphinx-build" in argc for argc in sys.argv):
+        # test to not fail when build the API doc
+        raise Exception('Could not load rucio configuration file rucio.cfg.'
+                        'Rucio looks in the following directories for a configuration file, in order:'
+                        '\n\t${RUCIO_HOME}/etc/rucio.cfg'
+                        '\n\t/opt/rucio/etc/rucio.cfg'
+                        '\n\t${VIRTUAL_ENV}/etc/rucio.cfg')


Test patch:
Repository: rucio/rucio

Issue:
Modify sphinx template to get the default in on rtd
Motivation
----------

Update of con.fy to get the default readthedocs template



Discussion:

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -48,7 +48,7 @@
 
 # General information about the project.
 project = u'Rucio'
-copyright = u'2017 CERN for the benefit of the ATLAS collaboration'
+copyright = u'Copyright (c) 2012-2018 CERN for the benefit of the ATLAS collaboration.'
 author = u'rucio-dev@cern.ch'
 
 # The version info for the project you're documenting, acts as replacement for
@@ -84,7 +84,7 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+# html_theme = 'alabaster'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -95,22 +95,22 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+# html_static_path = ['_static']
 
 # Custom sidebar templates, must be a dictionary that maps document names
 # to template names.
 #
 # This is required for the alabaster theme
 # refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars
-html_sidebars = {
-    '**': [
-        'about.html',
-        'navigation.html',
-        'relations.html',  # needs 'show_related': True theme option to display
-        'searchbox.html',
-        'donate.html',
-    ]
-}
+# html_sidebars = {
+#     '**': [
+#         'about.html',
+#         'navigation.html',
+#         'relations.html',  # needs 'show_related': True theme option to display
+#         'searchbox.html',
+#         'donate.html',
+#     ]
+# }
 
 
 # -- Options for HTMLHelp output ------------------------------------------


Test patch:
Repository: rucio/rucio

Issue:
S3 third party copy scheme mismatchs
Motivation
----------
FTS is currently supporting destination:S3 transfers only from source:gsiftp and source:srm endpoints. The natural compatibility of source:davs is not yet implement.

Modification
------------
Change SCHEME_MAPPING. Once FTS supports source:davs, the scheme can be extended again.


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/constants.py b/lib/rucio/common/constants.py
--- a/lib/rucio/common/constants.py
+++ b/lib/rucio/common/constants.py
@@ -7,6 +7,7 @@
 # Authors:
 # - Vincent Garonne, <vincent.garonne@cern.ch>, 2013
 # - Cedric Serfon, <cedric.serfon@cern.ch>, 2015
+# - Mario Lassnig, <mario.lassnig@cern.ch>, 2018
 
 """
 Constants.
@@ -22,8 +23,8 @@
 KEY_TYPES = ['ALL', 'COLLECTION', 'FILE', 'DERIVED']
 # all(container, dataset, file), collection(dataset or container), file, derived(compute from file for collection)
 
-SCHEME_MAP = {'srm': ['srm', 'gsiftp'],
-              'gsiftp': ['srm', 'gsiftp'],
-              'https': ['https', 'davs', 's3'],
-              'davs': ['https', 'davs', 's3'],
-              's3': ['https', 's3', 'davs']}
+SCHEME_MAP = {'srm': ['srm', 'gsiftp', 's3'],
+              'gsiftp': ['srm', 'gsiftp', 's3'],
+              'https': ['https', 'davs'],
+              'davs': ['https', 'davs'],
+              's3': ['https', 'davs', 'gsiftp', 'srm']}


Test patch:
Repository: rucio/rucio

Issue:
Documentation overhaul for the workshop
This issue is used to track the changes and todos necessary for the general documentation overhaul for the workshop.

# General tasks
## General information
- [x] Concepts and Terminology (#566, @bari12)
- [x] Architecture (@bari12) - Depending on how much needs to be done, some help might be needed here

## Developer documentation
- [x] Contributer guide (from github) - should be only one line in the TOC (@vingar)
- [x] Installing dev environment (@vingar) - Docker example written by @berghaus can be used here.
- [x] REST API description (@tbeerman) - Some general information, some examples, rest auto generated
- [x] API description auto-generated (@vingar)
- [x] CLI description auto-generated (@vingar @cserf) for rucio, rucio-admin utilities and daemons

## User documentation
- [x] Installing Rucio clients (@cserf) - Mostly already there, only docker part needs to be added
- [x] Rucio Clients howto (#551, @cserf, @TomasJavurek) - Should be a generalized version of the ATLAS RuucioHowTo
- [x] Demo section based on docker and different steps (#696 @tbeerman, @cserf) - Basically installing a server, daemons etc in docker and then doing some things in this env

## Operator documentation
- [x] General information (@mlassnig) - Different ways of deploying rucio. Dependencies (DB, Transfertool, Messaging, Monitoring)
- [ ] Docker deployment
- [x] How to configure Rucio: rses, accounts, etc (#736,  @cserf)
- [x] How to version the DB with alembic, upgrade and downgrade instructions (@mlassnig)
- [ ] (Puppet deployment)
- [ ] (Kubernetes deployment)

# Bits & Pieces
- [x] Concepts needs to be expanded with information how a replica actually works on storage, with a dataset as example; how deterministic/non deterministic works; how scopes come into play etc. (@bari12)
- [x] How to connect and configure a stroage element to Rucio (@cserf)
- [ ] We also need something about dumps and ACLs in the op documentation
- [ ] How consistency and recovery can be done
- [x] List of return codes of bin/rucio (@mlassnig)


Discussion:

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -50,7 +50,7 @@
 
 # General information about the project.
 project = u'Rucio'
-copyright = u'Copyright (c) 2012-2018 CERN for the benefit of the ATLAS collaboration.'
+copyright = u'2012-2018 CERN for the benefit of the ATLAS collaboration.'
 author = u'rucio-dev@cern.ch'
 
 # The version info for the project you're documenting, acts as replacement for


Test patch:
Repository: rucio/rucio

Issue:
transfers: force S3 alternative authentication method
Motivation
----------
Currently, FTS3 requires a compatibility option to authenticate with S3 storage. This can be removed once Davix is modified to do this automatically.

Modification
------------
Set "s3alternate":True in the job_param


Discussion:

Resolving patch:
diff --git a/lib/rucio/daemons/conveyor/common.py b/lib/rucio/daemons/conveyor/common.py
--- a/lib/rucio/daemons/conveyor/common.py
+++ b/lib/rucio/daemons/conveyor/common.py
@@ -6,7 +6,7 @@
 #
 # Authors:
 # - Vincent Garonne, <vincent.garonne@cern.ch>, 2012-2014
-# - Mario Lassnig, <mario.lassnig@cern.ch>, 2013-2015
+# - Mario Lassnig, <mario.lassnig@cern.ch>, 2013-2018
 # - Cedric Serfon, <cedric.serfon@cern.ch>, 2013-2016
 # - Wen Guan, <wen.guan@cern.ch>, 2014-2016
 # - Joaquin Bogado, <jbogadog@cern.ch>, 2016
@@ -168,7 +168,8 @@ def bulk_group_transfer(transfers, policy='rule', group_bulk=200, fts_source_str
                       'job_metadata': {'issuer': 'rucio'},  # finaly job_meta will like this. currently job_meta will equal file_meta to include request_id and etc.
                       'source_spacetoken': transfer['src_spacetoken'] if transfer['src_spacetoken'] else None,
                       'overwrite': transfer['overwrite'],
-                      'priority': 3}
+                      'priority': 3,
+                      's3alternate': True}
 
         if max_time_in_queue:
             if transfer['file_metadata']['activity'] in max_time_in_queue:


Test patch:
Repository: rucio/rucio

Issue:
Misleading warning in srm scheme 
Motivation
----------
The srm scheme currently requires space_token as  extended_attribute, however when space_token is not set, rucio complains about  extended_attributes and web_service_path not specfied.

Modification
------------
Happens in these lines: 
https://github.com/rucio/rucio/blob/5fa4bb25b6f5b75913773b65217a5fcde6e94f1c/lib/rucio/core/rse.py#L691-L692

Expected behavior
-----------------
Ideally , space_token would be an optional attribute,  and if not used, no spacetoken should be passed to FTS transfer job submission command.    See also  #574


Discussion:
Note - this is fixed in `next` but not in `master`.  So, I think this becomes a backport exercise.

Resolving patch:
diff --git a/lib/rucio/core/rse.py b/lib/rucio/core/rse.py
--- a/lib/rucio/core/rse.py
+++ b/lib/rucio/core/rse.py
@@ -688,7 +688,7 @@ def add_protocol(rse, parameter, session=None):
             pass  # String is not JSON
 
     if parameter['scheme'] == 'srm':
-        if ('space_token' not in parameter['extended_attributes']) or ('web_service_path' not in parameter['extended_attributes']):
+        if ('extended_attributes' not in parameter) or ('web_service_path' not in parameter['extended_attributes']):
             raise exception.InvalidObject('Missing values! For SRM, extended_attributes and web_service_path must be specified')
 
     try:


Test patch: