internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: castorini/pyserini

Issue:
Error for SPLADE on-the-fly encoding with pytorch  
command used:
```bash
python -m pyserini.search.lucene --threads 12 --batch-size 128 \
  --index msmarco-v1-passage-splade-pp-ed \
  --topics msmarco-passage-dev-subset \
  --encoder naver/splade-cocondenser-ensembledistil \
  --output run.msmarco-v1-passage.splade-pp-ed-pytorch.dev.txt \
  --hits 1000 --impact
```

error message:
> ...
>   File "/home/arthur/workplace/pyserini/pyserini/encode/_splade.py", line 28, in encode
>     raw_weights = self._output_to_weight_dicts(batch_token_ids, batch_weights)
> NameError: name 'batch_token_ids' is not defined



Discussion:
Confirmed that I can reproduce the bug.

Interestingly, from here - https://castorini.github.io/pyserini/2cr/msmarco-v1-passage.html

We have "SPLADE++ SelfDistil: query inference with ONNX" as a condition, but *not* "SPLADE++ SelfDistil: query inference with PyTorch."



> Confirmed that I can reproduce the bug.
> 
> Interestingly, from here - https://castorini.github.io/pyserini/2cr/msmarco-v1-passage.html
> 
> We have "SPLADE++ SelfDistil: query inference with ONNX" as a condition, but _not_ "SPLADE++ SelfDistil: query inference with PyTorch."

I will add all entries mentioned in the paper once all results are reproduced. 

Resolving patch:
diff --git a/pyserini/encode/_splade.py b/pyserini/encode/_splade.py
--- a/pyserini/encode/_splade.py
+++ b/pyserini/encode/_splade.py
@@ -25,7 +25,7 @@ def encode(self, text, max_length=256, **kwargs):
         batch_aggregated_logits, _ = torch.max(torch.log(1 + torch.relu(batch_logits))
                                                * input_attention.unsqueeze(-1), dim=1)
         batch_aggregated_logits = batch_aggregated_logits.cpu().detach().numpy()
-        raw_weights = self._output_to_weight_dicts(batch_token_ids, batch_weights)
+        raw_weights = self._output_to_weight_dicts(batch_aggregated_logits)
         return self._get_encoded_query_token_wight_dicts(raw_weights)[0]
 
     def _output_to_weight_dicts(self, batch_aggregated_logits):


Test patch:
diff --git a/tests/test_encoder.py b/tests/test_encoder.py
--- a/tests/test_encoder.py
+++ b/tests/test_encoder.py
@@ -233,6 +233,17 @@ def test_onnx_encode_unicoil(self):
 
         temp_object.close()
         del temp_object
+
+        temp_object1 = LuceneImpactSearcher(f'{self.index_dir}lucene9-index.cacm', 'naver/splade-cocondenser-ensembledistil')
+
+        # this function will never be called in _impact_searcher, here to check quantization correctness
+        results = temp_object1.encode("here is a test")
+        self.assertEqual(results.get("here"), 156)
+        self.assertEqual(results.get("a"), 31)
+        self.assertEqual(results.get("test"), 149)
+
+        temp_object1.close()
+        del temp_object1
         
     def tearDown(self):
         os.remove(self.tarball_name)
Repository: rucio/rucio

Issue:
db: fix outer join read lock for PostgreSQL
Motivation
----------
PostgreSQL is interpreting the SQL standard correctly, it's actually Oracle and MySQL which are doing non-standard things here. The problem you see is that for an outer join the nullable side cannot be locked for reading, so PostgreSQL stops. This is the last outstanding issue for PostgreSQL compatibility.

Modification
------------
Changing the query for add-rule from "select outer join for update" into something like "select for update union select".

Expected behavior
-----------------
add-rule will work on PostgreSQL.


Discussion:
I suggest raising the priority on this.

Resolving patch:
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.14.0.post1',
-    'branch_nick': 'patch-0-Release_procedure_fix',
-    'revision_id': 'e8d709d6a275dd81c673edec4e1229bac0de1a99',
-    'revno': 5642
+    'version': '1.14.1',
+    'branch_nick': 'patch-0-Preparation_for_1_14_1_release',
+    'revision_id': '94fc77cad3154b5807647c138674d959e6a029bb',
+    'revno': 5674
 }


Test patch:
Repository: rucio/rucio

Issue:
Add Extras dependency on the possible rucio backend (oracle, mysql, postgresql)
Motivation
----------
Express the "recommended" dependencies for the db backend

Modification
------------
Change of the main setup.py script to include extra dependencies

Expected behavior
-----------------
pip install rucio[oracle]
pip install rucio[postgresql]
pip install rucio[mysql]

or maybe define later some meta packages like rucio-oracle with install_requires = ["rucio[oracle]"]


Discussion:

Resolving patch:
diff --git a/setup_rucio.py b/setup_rucio.py
--- a/setup_rucio.py
+++ b/setup_rucio.py
@@ -113,6 +113,14 @@ def write_requirements():
         req_file.close()
 
 
+oracle_extras = ['cx_oracle>=5.1']
+postgresql_extras = ['psycopg2>=2.4.2']
+mysql_extras = ['PyMySQL']
+
+requires = parse_requirements(requirements_files=requirements_files)
+extras_require = dict(oracle=oracle_extras,
+                      postgresql=postgresql_extras,
+                      mysql=mysql_extras)
 requires = parse_requirements(requirements_files=requirements_files)
 depend_links = parse_dependency_links(requirements_files=requirements_files)
 
@@ -163,5 +171,6 @@ def get_file_list(self):
         'Programming Language :: Python :: 2.7',
         'Environment :: No Input/Output (Daemon)', ],
     install_requires=requires,
+    extras_require=extras_require,
     dependency_links=depend_links,
 )


Test patch:
Repository: rucio/rucio

Issue:
parse_pfns hotfix Fix #298



Discussion:

Resolving patch:
diff --git a/lib/rucio/rse/protocols/gfal.py b/lib/rucio/rse/protocols/gfal.py
--- a/lib/rucio/rse/protocols/gfal.py
+++ b/lib/rucio/rse/protocols/gfal.py
@@ -119,6 +119,8 @@ def parse_pfns(self, pfns):
             path = '/'.join(path.split('/')[:-1])
             if not path.startswith('/'):
                 path = '/' + path
+            if path != '/' and not path.endswith('/'):
+                path = path + '/'
             ret[pfn] = {'scheme': scheme, 'port': port, 'hostname': hostname, 'path': path, 'name': name, 'prefix': prefix, 'web_service_path': service_path}
 
         return ret


Test patch:
Repository: rucio/rucio

Issue:
Fixed badly merged conf.py file 




Discussion:

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -31,8 +31,8 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = ['sphinx.ext.autodoc',
-    'sphinx.ext.doctest',
-    'sphinx.ext.todo']
+              'sphinx.ext.doctest',
+              'sphinx.ext.todo']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -48,13 +48,8 @@
 
 # General information about the project.
 project = u'Rucio'
-<<<<<<< HEAD
-copyright = u'2017, rucio-dev@cern.ch'
+copyright = u'2017 CERN for the benefit of the ATLAS collaboration'
 author = u'rucio-dev@cern.ch'
-=======
-copyright = u'2017, rucio-dev@googlegroups.com'
-author = u'rucio-dev@googlegroups.com'
->>>>>>> a1ce07f41bd8f674e0a46f4f91c3edd100ee0861
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -149,11 +144,7 @@
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
     (master_doc, 'Rucio.tex', u'Rucio Documentation',
-<<<<<<< HEAD
      u'rucio-dev@cern.ch', 'manual'),
-=======
-     u'rucio-dev@googlegroups.com', 'manual'),
->>>>>>> a1ce07f41bd8f674e0a46f4f91c3edd100ee0861
 ]
 
 
@@ -177,6 +168,3 @@
      author, 'Rucio', 'One line description of project.',
      'Miscellaneous'),
 ]
-
-
-


Test patch:
Repository: rucio/rucio

Issue:
Fixed badly merged conf.py file 




Discussion:
Motivation
-----------

Badly merged conf.py file makes sphinx failing:

Running Sphinx v1.6.5
making output directory...

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/rucio-blueprint/envs/latest/local/lib/python2.7/site-packages/sphinx/cmdline.py", line 305, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/rucio-blueprint/envs/latest/local/lib/python2.7/site-packages/sphinx/application.py", line 168, in __init__
    confoverrides or {}, self.tags)
  File "/home/docs/checkouts/readthedocs.org/user_builds/rucio-blueprint/envs/latest/local/lib/python2.7/site-packages/sphinx/config.py", line 152, in __init__
    raise ConfigError(CONFIG_SYNTAX_ERROR % err)
ConfigError: There is a syntax error in your configuration file: invalid syntax (conf.py, line 51)

Configuration error:
There is a syntax error in your configuration file: invalid syntax (conf.py, line 51)
This is a duplicate of #255 
closed

Resolving patch:
diff --git a/lib/rucio/vcsversion.py b/lib/rucio/vcsversion.py
--- a/lib/rucio/vcsversion.py
+++ b/lib/rucio/vcsversion.py
@@ -4,8 +4,8 @@
 '''
 VERSION_INFO = {
     'final': True,
-    'version': '1.14.1',
-    'branch_nick': 'patch-0-Preparation_for_1_14_1_release',
-    'revision_id': '94fc77cad3154b5807647c138674d959e6a029bb',
-    'revno': 5674
+    'version': '1.14.2',
+    'branch_nick': 'patch-0-1_14_2_preparation',
+    'revision_id': '4e6e3c347088ff2d2959c077bf9dcf91a06a5bed',
+    'revno': 5697
 }
diff --git a/setup_rucio.py b/setup_rucio.py
--- a/setup_rucio.py
+++ b/setup_rucio.py
@@ -113,6 +113,14 @@ def write_requirements():
         req_file.close()
 
 
+oracle_extras = ['cx_oracle>=5.1']
+postgresql_extras = ['psycopg2>=2.4.2']
+mysql_extras = ['PyMySQL']
+
+requires = parse_requirements(requirements_files=requirements_files)
+extras_require = dict(oracle=oracle_extras,
+                      postgresql=postgresql_extras,
+                      mysql=mysql_extras)
 requires = parse_requirements(requirements_files=requirements_files)
 depend_links = parse_dependency_links(requirements_files=requirements_files)
 
@@ -163,5 +171,6 @@ def get_file_list(self):
         'Programming Language :: Python :: 2.7',
         'Environment :: No Input/Output (Daemon)', ],
     install_requires=requires,
+    extras_require=extras_require,
     dependency_links=depend_links,
 )


Test patch:
Repository: rucio/rucio

Issue:
Link_ranking based sorting in case of multipe tape replicas is done wrong
Motivation
----------
Sorting in case of multiple tape replicas should be done in reverse order, 1 being best, 11 being worst. Also cases where the ranking is 0 should be skipped.



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -708,7 +708,7 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                 if source_rse_id is None or rse is None:
                     continue
 
-                if link_ranking is None:
+                if link_ranking is None or link_ranking == 0:
                     logging.debug("Request %s: no link from %s to %s" % (id, source_rse_id, dest_rse_id))
                     continue
 
@@ -785,7 +785,7 @@ def get_transfer_requests_and_source_replicas(process=None, total_processes=None
                     else:
                         # the sources already founded is Tape too.
                         # multiple Tape source replicas are not allowed in FTS3.
-                        if transfers[id]['sources'][0][3] > ranking or (transfers[id]['sources'][0][3] == ranking and transfers[id]['sources'][0][4] >= link_ranking):
+                        if transfers[id]['sources'][0][3] > ranking or (transfers[id]['sources'][0][3] == ranking and transfers[id]['sources'][0][4] <= link_ranking):
                             continue
                         else:
                             transfers[id]['sources'] = []


Test patch:
Repository: rucio/rucio

Issue:
add # in cms regexp name
Modification
------------
add # in cms regexp name


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/schema/cms.py b/lib/rucio/common/schema/cms.py
--- a/lib/rucio/common/schema/cms.py
+++ b/lib/rucio/common/schema/cms.py
@@ -53,7 +53,7 @@
 
 NAME = {"description": "Data Identifier name",
         "type": "string",
-        "pattern": "^\/[A-Za-z0-9][A-Za-z0-9\\.\\-\\_\/]{1,255}$"}
+        "pattern": "^\/[A-Za-z0-9][A-Za-z0-9\\.\\-\\_\/\#]{1,255}$"}
 
 R_NAME = {"description": "Data Identifier name",
           "type": "string",


Test patch:
Repository: rucio/rucio

Issue:
PostgreSQL not working, 1 test failing
Motivation
----------
One PostgreSQL test is failing.

```
======================================================================
FAIL: DATA IDENTIFIERS (CLIENT): List dids by pattern.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/opt/rucio/lib/rucio/tests/test_did.py", line 187, in test_list_dids
    assert_equal(len(results), 2)
AssertionError: 3 != 2
-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): localhost
urllib3.connectionpool: DEBUG: https://localhost:443 "POST /accounts/jdoe/scopes/mock_ab5a437f644947e5b456 HTTP/1.1" 201 None
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): localhost
urllib3.connectionpool: DEBUG: https://localhost:443 "POST /replicas HTTP/1.1" 201 None
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (2): localhost
urllib3.connectionpool: DEBUG: https://localhost:443 "POST /replicas HTTP/1.1" 201 None
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (3): localhost
urllib3.connectionpool: DEBUG: https://localhost:443 "POST /replicas HTTP/1.1" 201 None
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): localhost
urllib3.connectionpool: DEBUG: https://localhost:443 "GET /dids/mock_ab5a437f644947e5b456/dids/search?type=file&name=file_a_%2A HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://localhost:443 "GET /dids/mock_ab5a437f644947e5b456/dids/search?type=file&name=file_a_1%2A HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://localhost:443 "GET /dids/mock_ab5a437f644947e5b456/dids/search?type=file&name=file_%2A_1%2A HTTP/1.1" 200 None
--------------------- >> end captured logging << ---------------------
```


Discussion:

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -1346,9 +1346,10 @@ def list_dids(scope, filters, type='collection', ignore_case=False, limit=None,
         if (isinstance(v, unicode) or isinstance(v, str)) and ('*' in v or '%' in v):
             if v in ('*', '%', u'*', u'%'):
                 continue
-            if session.bind.dialect.name == 'postgresql':  # PostgreSQL escapes automatically
+            if session.bind.dialect.name == 'postgresql':
                 query = query.filter(getattr(models.DataIdentifier, k).
-                                     like(v.replace('*', '%')))
+                                     like(v.replace('*', '%').replace('_', '\_'),
+                                          escape='\\'))
             else:
                 query = query.filter(getattr(models.DataIdentifier, k).
                                      like(v.replace('*', '%').replace('_', '\_'), escape='\\'))


Test patch:
Repository: rucio/rucio

Issue:
`import rucio.core.rule` fails with missing section
The work in #339 caused a regression if the `permission` section does not exist.  From the logs:

```
mod_wsgi (pid=18803): Exception occurred processing WSGI script '/usr/lib/python2.7/site-packages/rucio/web/rest/did.py'.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rucio/web/rest/did.py", line 22, in <module>
    from rucio.api.did import (add_did, add_dids, list_content, list_content_history,
  File "/usr/lib/python2.7/site-packages/rucio/api/did.py", line 19, in <module>
    from rucio.core import did, naming_convention, meta as meta_core
  File "/usr/lib/python2.7/site-packages/rucio/core/did.py", line 35, in <module>
    import rucio.core.rule
  File "/usr/lib/python2.7/site-packages/rucio/core/rule.py", line 2888, in <module>
    def archive_localgroupdisk_datasets(scope, name, session=None):
  File "/usr/lib/python2.7/site-packages/rucio/common/policy.py", line 78, in policy_filter
    policy = get_policy()
  File "/usr/lib/python2.7/site-packages/rucio/common/policy.py", line 40, in get_policy
    policy = config_get('permission', 'policy')
  File "/usr/lib/python2.7/site-packages/rucio/common/config.py", line 30, in config_get
    return __CONFIG.get(section, option)
  File "/usr/lib64/python2.7/ConfigParser.py", line 607, in get
    raise NoSectionError(section)
NoSectionError: No section: 'permission'
```

Seems that this exception type is simply not caught.


Discussion:

Resolving patch:
diff --git a/lib/rucio/common/policy.py b/lib/rucio/common/policy.py
--- a/lib/rucio/common/policy.py
+++ b/lib/rucio/common/policy.py
@@ -38,7 +38,7 @@ def get_policy():
     if isinstance(policy, NoValue):
         try:
             policy = config_get('permission', 'policy')
-        except NoOptionError:
+        except (NoOptionError, NoSectionError):
             policy = 'atlas'
         REGION.set('policy', policy)
     return policy


Test patch:
Repository: rucio/rucio

Issue:
Move rule permission check is not working
Motivation
----------
For rucio move_rule the permission check is not working, thus the default permission is used.



Discussion:

Resolving patch:
diff --git a/lib/rucio/core/permission/atlas.py b/lib/rucio/core/permission/atlas.py
--- a/lib/rucio/core/permission/atlas.py
+++ b/lib/rucio/core/permission/atlas.py
@@ -11,7 +11,7 @@
 # - Yun-Pin Sun, <yun-pin.sun@cern.ch>, 2012-2013
 # - Ralph Vigne, <ralph.vigne@cern.ch>, 2013
 # - Mario Lassnig, <mario.lassnig@cern.ch>, 2013-2015, 2017
-# - Martin Barisits, <martin.barisits@cern.ch>, 2013-2016
+# - Martin Barisits, <martin.barisits@cern.ch>, 2013-2018
 # - Cedric Serfon, <cedric.serfon@cern.ch>, 2013-2017
 # - Joaquin Bogado, <joaquin.bogado@cern.ch>, 2015
 
@@ -60,6 +60,7 @@ def has_permission(issuer, action, kwargs):
             'approve_rule': perm_approve_rule,
             'update_subscription': perm_update_subscription,
             'reduce_rule': perm_reduce_rule,
+            'move_rule': perm_move_rule,
             'get_auth_token_user_pass': perm_get_auth_token_user_pass,
             'get_auth_token_gss': perm_get_auth_token_gss,
             'get_auth_token_x509': perm_get_auth_token_x509,


Test patch:
Repository: rucio/rucio

Issue:
Avoid session reuse when new token is needed
Motivation
----------
In testing of `rucio upload`, I've sporadically hit cases where:
1. In a sequence of REST API calls, one call requires the client to retrieve a new Rucio token.
2. The client will contact the token issuer endpoint to retrieve a new token.
3. Since the client reuses the session object, an existing SSL session may be utilized.
4. If the existing SSL session wasn't created with an X509 user proxy (or established via TLS session reuse), then `mod_gridsite` isn't able to pass along the appropriate headers about the client DN.
5. Without the client DN, the issuer (correctly) refuses to issue a new token.

Modification
------------
Whenever the client receives an authorization failure, it should create a new session object prior to retrying the token retrieval.


Discussion:

Resolving patch:
diff --git a/lib/rucio/client/baseclient.py b/lib/rucio/client/baseclient.py
--- a/lib/rucio/client/baseclient.py
+++ b/lib/rucio/client/baseclient.py
@@ -277,6 +277,7 @@ def _send_request(self, url, headers=None, type='GET', data=None, params=None):
                 continue
 
             if result is not None and result.status_code == codes.unauthorized:  # pylint: disable-msg=E1101
+                self.session = session()
                 self.__get_token()
                 hds['X-Rucio-Auth-Token'] = self.auth_token
             else:
@@ -363,8 +364,10 @@ def __get_token_x509(self):
                 if retry > self.request_retries:
                     raise
 
-        if not result:
-            LOG.error('cannot get auth_token')
+        # Note a response object for a failed request evaluates to false, so we cannot
+        # use "not result" here
+        if result is None:
+            LOG.error('Internal error: Request for authentication token returned no result!')
             return False
 
         if result.status_code != codes.ok:   # pylint: disable-msg=E1101


Test patch:
Repository: rucio/rucio

Issue:
Lifetime in add_container ignore
Motivation
----------
Lifetime is a valid parameter of add_container(), but is ignored

Modification
------------

Expected behavior
-----------------
Should be passed through to add_did()


Discussion:

Resolving patch:
diff --git a/lib/rucio/client/didclient.py b/lib/rucio/client/didclient.py
--- a/lib/rucio/client/didclient.py
+++ b/lib/rucio/client/didclient.py
@@ -14,6 +14,7 @@
   - Ralph Vigne, <ralph.vigne@cern.ch>, 2015
   - Martin Barisits, <martin.barisits@cern.ch>, 2014-2015
   - Brian Bockelman, <bbockelm@cse.unl.edu>, 2018
+  - Eric Vaandering, <ewv@fnal.gov>, 2018
 '''
 
 from urllib import quote_plus
@@ -150,11 +151,11 @@ def add_container(self, scope, name, statuses=None, meta=None, rules=None, lifet
         :param scope: The scope name.
         :param name: The data identifier name.
         :param statuses: Dictionary with statuses, e.g.g {'monotonic':True}.
-        :meta: Meta-data associated with the data identifier is represented using key/value pairs in a dictionary.
-        :rules: Replication rules associated with the data identifier. A list of dictionaries, e.g., [{'copies': 2, 'rse_expression': 'TIERS1'}, ].
+        :param meta: Meta-data associated with the data identifier is represented using key/value pairs in a dictionary.
+        :param rules: Replication rules associated with the data identifier. A list of dictionaries, e.g., [{'copies': 2, 'rse_expression': 'TIERS1'}, ].
         :param lifetime: DID's lifetime (in seconds).
         """
-        return self.add_did(scope=scope, name=name, type='CONTAINER', statuses=statuses, meta=meta, rules=rules)
+        return self.add_did(scope=scope, name=name, type='CONTAINER', statuses=statuses, meta=meta, rules=rules, lifetime=lifetime)
 
     def add_containers(self, cnts):
         """


Test patch: