internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: bids-standard/pybids

Issue:
Update prep_zenodo.py to only count commits in grabbit up to 0.2.6
With #369, we're dropping the grabbit dependency, so changes there will no longer contribute to pybids.


Discussion:
Okay to close this? `prep_zenodo.py` hasn't been updated, but I'm not sure if we're still using it to generate the citation...
It's manually run.

Resolving patch:
diff --git a/tools/prep_zenodo.py b/tools/prep_zenodo.py
--- a/tools/prep_zenodo.py
+++ b/tools/prep_zenodo.py
@@ -11,7 +11,7 @@ def decommify(name):
 
 
 # List of repositories whose commits should be counted as contributions
-codependents = ['https://github.com/grabbles/grabbit.git']
+codependents = [('https://github.com/grabbles/grabbit.git', '0.2.6')]
 
 # Last shablona commit
 origin_commit = 'd72caaf5933907ed699d57faddaec7bfc836ce6f'
@@ -33,15 +33,23 @@ def decommify(name):
 with TemporaryDirectory() as tmpdir:
     tmppath = Path(tmpdir)
     for repo in codependents:
+        try:
+            repo, ref = repo
+        except (TypeError, ValueError):
+            ref = None
         repo_dir = str(tmppath / repo.rsplit('/', 1)[1].split('.', 1)[0])
         try:
-            clone = run(['git', 'clone', repo, repo_dir], check=True)
+            clone = run(['git', 'clone', '-q', repo, repo_dir], check=True)
         except CalledProcessError as err:
             raise RuntimeError("Could not clone {}".format(repo)) from err
-        tag = run(['git', '-C', repo_dir, 'tag'], stdout=PIPE)
-        latest_tag = tag.stdout.decode().strip().rsplit('\n', 1)[1]
+
+        if ref is None:
+            tag = run(['git', '-C', repo_dir, 'tag'], stdout=PIPE)
+            # latest tag
+            ref = tag.stdout.decode().strip().rsplit('\n', 1)[1]
+
         dep_shortlog = run(
-            ['git', '-C', repo_dir, 'shortlog', '-ns', latest_tag],
+            ['git', '-C', repo_dir, 'shortlog', '-ns', ref],
             stdout=PIPE)
         counts.extend(line.split('\t', 1)[::-1]
                       for line in dep_shortlog.stdout.decode().split('\n')


Test patch:
Repository: bids-standard/pybids

Issue:
0.9.4 Release
@tyarkoni We've built up a [changelog](https://github.com/bids-standard/pybids/releases/tag/untagged-aa777a4ea34ce82990cd), so it'd be good to release soon. Any blocking issues?


Discussion:
Nope, fine to release. There are some moderately high priority issues outstanding, but there's no need to include them in the next release. Thanks!

Resolving patch:
diff --git a/tools/prep_zenodo.py b/tools/prep_zenodo.py
--- a/tools/prep_zenodo.py
+++ b/tools/prep_zenodo.py
@@ -9,6 +9,11 @@
 def decommify(name):
     return ' '.join(name.split(', ')[::-1])
 
+# Users who have asked not to be cited at this time
+# XXX We should add a shortlog since the most recent tag and explicitly note
+# that a blacklisted user has contributed again recently, and verify they still
+# do not want to be cited.
+blacklist = {'Cecile Madjar'}
 
 # List of repositories whose commits should be counted as contributions
 codependents = [('https://github.com/grabbles/grabbit.git', '0.2.6')]
@@ -75,6 +80,7 @@ def decommify(name):
 creators = [
     creator_map.get(committer, {'name': committer})
     for committer in committers
+    if committer not in blacklist
     ]
 
 zenodo['creators'] = creators


Test patch:
Repository: bids-standard/pybids

Issue:
replace_entities() modifies entities
I guess it is by design, but replace_entities() modifies the input entities as it goes. I find any function that modifies the input values surprising, but it also means that previous path_patterns can affect the entities as they are iterated.

I think the function should return a new entities with the correct entities if this is useful. However, this failing, the function definitely shouldn't modify entities unless it actually returns something other than None.


Discussion:
Possibly related to #515 
I agree. I think we should make a copy of the entities dictionary at the start. Would you care to open a PR?
Not by design, definitely a bug. Thanks for catching this!

Resolving patch:
diff --git a/bids/layout/writing.py b/bids/layout/writing.py
--- a/bids/layout/writing.py
+++ b/bids/layout/writing.py
@@ -31,6 +31,7 @@ def replace_entities(entities, pattern):
         A new string with the entity values inserted where entity names
         were denoted in the provided pattern.
     """
+    entities = entities.copy()  # make a local copy, since dicts are mutable
     ents = re.findall(r'\{(.*?)\}', pattern)
     new_path = pattern
     for ent in ents:


Test patch:
Repository: bids-standard/pybids

Issue:
BIDSLayout.write_contents_to_file help for "strict" is incomplete
The doco for the `strict` parameter to `write_contents_to_file` in `BIDSLayout` looks incomplete. It's this in `master`:

```
        strict : bool
            If True, all entities must be matched inside a
            pattern in order to be a valid match. If False, extra entities
        validate : bool
```

That looks like it got truncated after "If False, extra entities"?


Discussion:

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -1537,6 +1537,7 @@ def write_contents_to_file(self, entities, path_patterns=None,
         strict : bool
             If True, all entities must be matched inside a
             pattern in order to be a valid match. If False, extra entities
+            will be ignored so long as all mandatory entities are found.
         validate : bool
             If True, built path must pass BIDS validator. If
             False, no validation is attempted, and an invalid path may be


Test patch:
Repository: bids-standard/pybids

Issue:
model: JSON to dict modified key values for transformation
In ` Replace` transformation, you specify as a dict which variables to transform.

e.g.:

```
{'LIKELY': "5"}
```

However, the parser from JSON to dict to convert BIDS Stats Models modifies keys to lower case, which in the case of specific case sensitive values modifies the transformation itself.


Discussion:

Resolving patch:
diff --git a/bids/utils.py b/bids/utils.py
--- a/bids/utils.py
+++ b/bids/utils.py
@@ -44,9 +44,10 @@ def alphanum_key(key):
 
 def convert_JSON(j):
     """ Recursively convert CamelCase keys to snake_case.
-    From: https://stackoverflow.com/questions/17156078/converting-identifier-naming-between-camelcase-and-underscores-during-json-seria
+    From: https://stackoverflow.com/questions/17156078/
+    converting-identifier-naming-between-camelcase-and-
+    underscores-during-json-seria
     """
-
     def camel_to_snake(s):
         a = re.compile('((?<=[a-z0-9])[A-Z]|(?!^)[A-Z](?=[a-z]))')
         return a.sub(r'_\1', s).lower()
@@ -54,7 +55,7 @@ def camel_to_snake(s):
     def convertArray(a):
         newArr = []
         for i in a:
-            if isinstance(i,list):
+            if isinstance(i, list):
                 newArr.append(convertArray(i))
             elif isinstance(i, dict):
                 newArr.append(convert_JSON(i))
@@ -66,7 +67,8 @@ def convertArray(a):
     for k, value in j.items():
         newK = camel_to_snake(k)
 
-        if isinstance(value, dict):
+        # Replace transformation uses a dict, so skip lower-casing
+        if isinstance(value, dict) and k != 'Replace':
             out[newK] = convert_JSON(value)
         elif isinstance(value, list):
             out[newK] = convertArray(value)


Test patch:
Repository: bids-standard/pybids

Issue:
Scale transformation shouldn't return NAs in event of constant input
If constant input is passed to the `Scale` transformation, a column of N/A values is returned. This should probably just fail with an exception (or at minimum, return a column of `0`).
Scale transformation shouldn't return NAs in event of constant input
If constant input is passed to the `Scale` transformation, a column of N/A values is returned. This should probably just fail with an exception (or at minimum, return a column of `0`).


Discussion:

Resolving patch:
diff --git a/bids/analysis/transformations/compute.py b/bids/analysis/transformations/compute.py
--- a/bids/analysis/transformations/compute.py
+++ b/bids/analysis/transformations/compute.py
@@ -138,9 +138,19 @@ class Scale(Transformation):
         None, no replacement is performed. If 'before', missing values are
         replaced with 0's before scaling. If 'after', missing values are
         replaced with 0 after scaling.
+
+    Notes
+    -----
+    If a constant column is passed in, and replace_na is None or 'before', an
+    exception will be raised.
     """
 
     def _transform(self, data, demean=True, rescale=True, replace_na=None):
+        if data.nunique() == 1 and replace_na in {None, 'before'}:
+            val = data.unique()[0]
+            raise ValueError("Cannot scale a column with constant value ({})! "
+                             "If you want a constant column of 0's returned, "
+                             "set replace_na to 'after'.".format(val))
         if replace_na == 'before':
             data = data.fillna(0.)
         if demean:


Test patch:
diff --git a/bids/analysis/tests/test_transformations.py b/bids/analysis/tests/test_transformations.py
--- a/bids/analysis/tests/test_transformations.py
+++ b/bids/analysis/tests/test_transformations.py
@@ -128,7 +128,7 @@ def test_sum(collection):
                       output='sum', weights=[1, 1, 1])
 
 
-def test_scale(collection):
+def test_scale(collection, sparse_run_variable_with_missing_values):
     transform.Scale(collection, variables=['RT', 'parametric gain'],
                     output=['RT_Z', 'gain_Z'], groupby=['run', 'subject'])
     groupby = collection['RT'].get_grouper(['run', 'subject'])
@@ -137,6 +137,16 @@ def test_scale(collection):
         groupby).apply(lambda x: (x - x.mean()) / x.std())
     assert np.allclose(z1, z2)
 
+    # Test constant input
+    coll = sparse_run_variable_with_missing_values
+    coll['var'].values.fillna(1)
+    with pytest.raises(ValueError, match="Cannot scale.*1.0"):
+        transform.Scale(coll, 'var').values
+    with pytest.raises(ValueError, match="Cannot scale.*1.0"):
+        transform.Scale(coll, 'var', replace_na='before').values
+    transform.Scale(coll, 'var', replace_na='after', output='zero')
+    assert coll['zero'].values.unique() == 0
+
 
 def test_demean(collection):
     transform.Demean(collection, variables=['RT'], output=['RT_dm'])
Repository: bids-standard/pybids

Issue:
Problem when using ``get`` with non-default BIDSLayout config
When using a custom `config` on a `BIDSLayout`, there will not be a `"bids"` config. This leads to a `KeyError` whenever a get query uses the `extension` filter.

https://github.com/bids-standard/pybids/blob/dbf79ea5ef2a8949a8ccb278ce168e3bd880af90/bids/layout/layout.py#L958-L961

For the fix, do we want to propagate this drop_dot check to all configs, or just skip if the default config is not present?

Related issue: https://github.com/templateflow/templateflow/issues/75


Discussion:
I would say skip if the default config is not present.

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -955,7 +955,7 @@ def get(self, return_type='object', target=None, scope='all',
                                '`layout.get(**filters)`.')
 
         # Strip leading periods if extensions were passed
-        if 'extension' in filters:
+        if 'extension' in filters and 'bids' in self.config:
             # XXX 0.14: Disable drop_dot option
             drop_dot = (self.config['bids'].entities['extension'].pattern ==
                         '[._]*[a-zA-Z0-9]*?\\.([^/\\\\]+)$')


Test patch:
Repository: bids-standard/pybids

Issue:
UnicodeDecodeError in index.py
I am working on a python package that uses templateflow which uses pybids. One of our users is running into a UnicodeDecodeError:
https://github.com/yeatmanlab/pyAFQ/issues/267
It looks like it may have a similar to this issue:
https://github.com/bids-standard/pybids/issues/69
I.e., change this line:
https://github.com/bids-standard/pybids/blob/750e189abfdb193569e4002b12217d2c154667ef/bids/layout/index.py#L278
to:
`with open(bf.path, 'r', encoding='utf-8') as handle: `


Discussion:
Seems reasonable. Care to open a PR?

Resolving patch:
diff --git a/bids/layout/index.py b/bids/layout/index.py
--- a/bids/layout/index.py
+++ b/bids/layout/index.py
@@ -275,7 +275,7 @@ def _index_metadata(self):
                     file_data[key] = defaultdict(list)
 
                 if ext == dot + 'json':
-                    with open(bf.path, 'r') as handle:
+                    with open(bf.path, 'r', encoding='utf-8') as handle:
                         try:
                             payload = json.load(handle)
                         except json.JSONDecodeError as e:


Test patch:
Repository: bids-standard/pybids

Issue:
Re-enable CI - Circle or GitHub Actions
Travis CI has abandoned OSS, so we must switch


Discussion:
If anybody's interested and would like a model, you can check out https://github.com/nipy/nibabel/pull/972.
+1 for GitHub Actions

Resolving patch:
diff --git a/bids/_version.py b/bids/_version.py
--- a/bids/_version.py
+++ b/bids/_version.py
@@ -40,7 +40,7 @@ def get_config():
     # _version.py
     cfg = VersioneerConfig()
     cfg.VCS = "git"
-    cfg.style = "pep440"
+    cfg.style = "pep440-pre"
     cfg.tag_prefix = ""
     cfg.parentdir_prefix = ""
     cfg.versionfile_source = "bids/_version.py"
diff --git a/bids/cli.py b/bids/cli.py
--- a/bids/cli.py
+++ b/bids/cli.py
@@ -30,7 +30,7 @@ def cli():
 @cli.command(context_settings=CONTEXT_SETTINGS)
 @click.argument('root', type=click.Path(file_okay=False, exists=True))
 @click.argument('db-path', type=click.Path(file_okay=False, resolve_path=True, exists=True))
-@click.option('--derivatives', multiple=True, default=False, show_default=True, flag_value=True,
+@click.option('--derivatives', multiple=True, default=[False], show_default=True, flag_value=True,
               help="Specifies whether and/or which derivatives to index.")
 @click.option('--reset-db', default=False, show_default=True, is_flag=True,
               help="Remove existing database index if present.")


Test patch:
Repository: bids-standard/pybids

Issue:
unclear validation error with dataset_description.json
A couple people have posted asking why their datasets could not be read (through fmriprep or mriqc), since the error message did not indicate which file was not formatted correctly.

example error message
```
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
```

- example 1: https://neurostars.org/t/fmriprep1-2-3-jsondecode-error-expecting-value/3352
- example 2: https://neurostars.org/t/week-8-quiz-question-10/18410


Discussion:

Resolving patch:
diff --git a/bids/layout/validation.py b/bids/layout/validation.py
--- a/bids/layout/validation.py
+++ b/bids/layout/validation.py
@@ -79,9 +79,21 @@ def validate_root(root, validate):
         else:
             description = None
     else:
-        with open(target, 'r', encoding='utf-8') as desc_fd:
-            description = json.load(desc_fd)
+        try:
+            with open(target, 'r', encoding='utf-8') as desc_fd:
+                description = json.load(desc_fd)
+        except json.JSONDecodeError:
+            description = None
         if validate:
+
+            if description is None:
+                raise BIDSValidationError(
+                    "'dataset_description.json' is not a valid json file."
+                    " There is likely a typo in your 'dataset_description.json'."
+                    "\nExample contents of 'dataset_description.json': \n%s" %
+                    json.dumps(EXAMPLE_BIDS_DESCRIPTION)
+                )
+            
             for k in MANDATORY_BIDS_FIELDS:
                 if k not in description:
                     raise BIDSValidationError(


Test patch:
diff --git a/bids/layout/tests/test_layout.py b/bids/layout/tests/test_layout.py
--- a/bids/layout/tests/test_layout.py
+++ b/bids/layout/tests/test_layout.py
@@ -53,6 +53,17 @@ def test_layout_repr(layout_7t_trt):
     assert "Subjects: 10 | Sessions: 20 | Runs: 20" in str(layout_7t_trt)
 
 
+def test_invalid_dataset_description(tmp_path):
+    shutil.copytree(join(get_test_data_path(), '7t_trt'), tmp_path / "7t_dset")
+    (tmp_path / "7t_dset" / "dataset_description.json").write_text(
+        "I am not a valid json file"
+    )
+    with pytest.raises(BIDSValidationError) as exc:
+        BIDSLayout(tmp_path / "7t_dset")
+
+    assert "is not a valid json file" in str(exc.value)
+
+
 def test_layout_repr_overshadow_run(tmp_path):
     """A test creating a layout to replicate #681."""
     shutil.copytree(join(get_test_data_path(), '7t_trt'), tmp_path / "7t_trt")
Repository: bids-standard/pybids

Issue:
Add Meta model type
FEMA contrast type is now becoming "meta" model type, as opposed to "glm".


Discussion:

Resolving patch:
diff --git a/bids/modeling/model_spec.py b/bids/modeling/model_spec.py
--- a/bids/modeling/model_spec.py
+++ b/bids/modeling/model_spec.py
@@ -22,7 +22,8 @@ def create_model_spec(df, model, *args, **kwargs):
     """
     kind = model.get('type', 'glm').lower()
     SpecCls = {
-        'glm': GLMMSpec
+        'glm': GLMMSpec,
+        'meta': MetaAnalysisSpec,
     }[kind]
     return SpecCls.from_df(df, model, *args, **kwargs)
 
@@ -273,6 +274,10 @@ def from_df(cls, df, model, metadata=None, formula=None):
         return GLMMSpec(**kwargs)
 
 
+class MetaAnalysisSpec(GLMMSpec):
+    pass
+
+
 class Term(object):
     """Represents a model term.
 


Test patch:
Repository: bids-standard/pybids

Issue:
Most grouping variables are not available to analysis nodes
If we try to group a node by a "group" or "diagnosis" criterion, PyBIDS will silently drop it.

```YAML
  "GroupBy": ["group", "contrast"],
```


Discussion:

Resolving patch:
diff --git a/bids/modeling/statsmodels.py b/bids/modeling/statsmodels.py
--- a/bids/modeling/statsmodels.py
+++ b/bids/modeling/statsmodels.py
@@ -18,10 +18,6 @@
 import warnings
 
 
-# Only entities in this list can be used in grouping
-VALID_GROUPING_ENTITIES = {'run', 'session', 'subject', 'task', 'contrast'}
-
-
 def validate_model(model):
     """Validate a BIDS-StatsModel structure.
 
@@ -343,9 +339,7 @@ def _build_groups(objects, group_by):
 
         # Single-run tasks and single-session subjects may not have entities
         dummy_groups = {"run", "session"} - set(df.columns)
-
-        # sanitize grouping entities, otherwise weird things can happen
-        group_by = set(group_by) & VALID_GROUPING_ENTITIES - dummy_groups
+        group_by = set(group_by) - dummy_groups
 
         # Verify all columns in group_by exist and raise sensible error if not
         missing_vars = list(group_by - set(df.columns))
@@ -583,11 +577,7 @@ def __init__(self, node, entities={}, collections=None, inputs=None,
         if inputs:
             dfs.append(self._inputs_to_df(inputs))
 
-        # merge all the DataFrames into one DF to rule them all
-        def merge_dfs(a, b):
-            on = list(set(a.columns) & set(b.columns) & VALID_GROUPING_ENTITIES)
-            return a.merge(b, on=on)
-        df = reduce(merge_dfs, dfs)
+        df = reduce(pd.DataFrame.merge, dfs)
 
         var_names = list(self.node.model['x'])
 


Test patch:
Repository: bids-standard/pybids

Issue:
SQLAlchemy 1.4 migration
Apparently SQLAlchemy is undergoing a massive refactor for version 2.0, and is using 1.4 as a stepping stone. The idea seems to be break some things, let people fix them with a bunch of docs about upcoming best practices, and hopefully the next breakage won't be as painful.

SQLAlchemy has a 1.4.0b1 release that broke our `--pre` tests, which was a good heads up. For the time being, we've set our dependency to `sqlalchemy < 1.4.0.dev0`, but we should try to migrate.


Discussion:
Makes sense, and I'll try to get to it soon, but I am *not* looking forward to it 😬

Resolving patch:
diff --git a/bids/layout/models.py b/bids/layout/models.py
--- a/bids/layout/models.py
+++ b/bids/layout/models.py
@@ -10,12 +10,16 @@
 from functools import lru_cache
 from collections import UserDict
 
-from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.ext.associationproxy import association_proxy
 from sqlalchemy.orm.collections import attribute_mapped_collection
-from sqlalchemy import (Column, String, Boolean, ForeignKey, Table)
+from sqlalchemy import Column, String, Boolean, ForeignKey, Table
 from sqlalchemy.orm import reconstructor, relationship, backref, object_session
 
+try:
+    from sqlalchemy.orm import declarative_base
+except ImportError:  # sqlalchemy < 1.4
+    from sqlalchemy.ext.declarative import declarative_base
+
 from ..utils import listify
 from .writing import build_path, write_to_file
 from ..config import get_option


Test patch:
diff --git a/bids/layout/tests/test_models.py b/bids/layout/tests/test_models.py
--- a/bids/layout/tests/test_models.py
+++ b/bids/layout/tests/test_models.py
@@ -178,7 +178,8 @@ def test_load_existing_config():
     session.commit()
 
     from sqlalchemy.orm.exc import FlushError
-    with pytest.raises(FlushError):
+    from sqlalchemy.exc import DBAPIError
+    with pytest.raises((FlushError, DBAPIError)):
         second = Config.load({"name": "dummy"})
         session.add(second)
         session.commit()
diff --git a/bids/layout/tests/test_writing.py b/bids/layout/tests/test_writing.py
--- a/bids/layout/tests/test_writing.py
+++ b/bids/layout/tests/test_writing.py
@@ -37,7 +37,7 @@ def writable_file(tmpdir):
 
     session.add_all(list(ents.values()) + tags + [bf])
     session.commit()
-    return bf
+    yield bf
 
 
 @pytest.fixture(scope='module')