internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: bids-standard/pybids

Issue:
get_design_matrix force dense mode, index must be integer
If the duration of a Run (in `RunInfo`) is not an integer (e.g. 1560.0), an error will be thrown when forcing the design matrix to be dense, because a float is used as a numpy index. 


Discussion:

Resolving patch:
diff --git a/bids/analysis/transformations/munge.py b/bids/analysis/transformations/munge.py
--- a/bids/analysis/transformations/munge.py
+++ b/bids/analysis/transformations/munge.py
@@ -108,8 +108,8 @@ class to_dense(Transformation):
     _input_type = 'variable'
     _return_type = 'variable'
 
-    def _transform(self, var):
-        return var.to_dense()
+    def _transform(self, var, sampling_rate=10):
+        return var.to_dense(sampling_rate=sampling_rate)
 
 
 class assign(Transformation):
diff --git a/bids/variables/variables.py b/bids/variables/variables.py
--- a/bids/variables/variables.py
+++ b/bids/variables/variables.py
@@ -320,8 +320,8 @@ def to_dense(self, sampling_rate):
             if onsets[i] < last_ind:
                 start += self.run_info[run_i].duration * sampling_rate
                 run_i += 1
-            _onset = start + onsets[i]
-            _offset = _onset + durations[i]
+            _onset = int(start + onsets[i])
+            _offset = int(_onset + durations[i])
             ts[_onset:_offset] = val
             last_ind = onsets[i]
 


Test patch:
Repository: bids-standard/pybids

Issue:
variables: BIDSVariableCollection.to_df loses n/as
This relates to: https://github.com/poldracklab/fitlins/issues/56

If there are `n/a`s in a `Sparse` variable, and `get_design_collection` is called with `force=True`, missing values are ommited (even if `Analysis.setup` was called with `drop_na=False`, and the variables originally had the `n/a`s)

`BIDSVariableCollection.to_df` does have a `fillna` argument (which by default is `nan`), but the following line removes `n/a`s prior to this: 

https://github.com/INCF/pybids/blob/87015d897ab881d0db47994595d63bba06daa6ab/bids/variables/kollekshuns.py#L116



Discussion:
`pivot_table` has a `dropna` argument that's `True` by default; I think we just need to set it to `False` if the passed `fillna` argument is anything other than the default (i.e., `np.nan`). Can you submit a PR?
Err, sorry--I think we want to set it to `False`, regardless.
Yep, tested that would fix the problem. Will do

Resolving patch:
diff --git a/bids/variables/kollekshuns.py b/bids/variables/kollekshuns.py
--- a/bids/variables/kollekshuns.py
+++ b/bids/variables/kollekshuns.py
@@ -113,9 +113,11 @@ def to_df(self, variables=None, format='wide', fillna=np.nan, **kwargs):
             return df.reset_index(drop=True).fillna(fillna)
 
         ind_cols = list(set(df.columns) - {'condition', 'amplitude'})
+
+        df['amplitude'] = df['amplitude'].fillna('n/a')
         df = df.pivot_table(index=ind_cols, columns='condition',
                             values='amplitude', aggfunc='first')
-        df = df.reset_index().fillna(fillna)
+        df = df.reset_index().replace('n/a', fillna)
         df.columns.name = None
         return df
 


Test patch:
Repository: bids-standard/pybids

Issue:
BIDS Validator data not installed
When installing pybids through pip using an archive (e.g. tarball, zip, instead of from github with `-e`), the data for the `BIDSValidator` is not installed, causing `BIDSLayout` to fail initating
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/bids/layout/layout.py", line 143, in __init__
    **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/grabbit/core.py", line 419, in __init__
    self.index()
  File "/usr/local/lib/python3.6/dist-packages/grabbit/core.py", line 610, in index
    _index_dir(path, list(domains))
  File "/usr/local/lib/python3.6/dist-packages/grabbit/core.py", line 603, in _index_dir
    elif self._validate_file(full_path):
  File "/usr/local/lib/python3.6/dist-packages/bids/layout/layout.py", line 204, in _validate_file
    return self.validator.is_bids(to_check)
  File "/usr/local/lib/python3.6/dist-packages/bids/layout/validation.py", line 64, in is_bids
    conditions.append(self.is_top_level(path))
  File "/usr/local/lib/python3.6/dist-packages/bids/layout/validation.py", line 75, in is_top_level
    with open(join(self.rule_dir, 'fixed_top_level_names.json'), 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.6/dist-packages/bids/layout/config/validator/fixed_top_level_names.json'
```


Discussion:

Resolving patch:
diff --git a/bids/layout/validation.py b/bids/layout/validation.py
--- a/bids/layout/validation.py
+++ b/bids/layout/validation.py
@@ -35,8 +35,7 @@ class BIDSValidator():
     """
 
     def __init__(self, index_associated=True):
-        self.rule_dir = join(dirname(abspath(__file__)), 'config',
-                              'validator')
+        self.rule_dir = join(dirname(abspath(__file__)),'config', 'validator')
         self.index_associated = index_associated
 
     def is_bids(self, path):
diff --git a/bids/version.py b/bids/version.py
--- a/bids/version.py
+++ b/bids/version.py
@@ -65,7 +65,7 @@ def package_files(directory):
 
 extra_files = package_files('path_to/extra_files_dir')
 PACKAGE_DATA = {
-    'bids.layout': ['config/*.json'],
+    'bids.layout': ['config/*.json', 'config/validator/*.json'],
     'bids.reports': ['config/*.json'],
     'bids': package_files('bids/tests/data')
 }


Test patch:
Repository: bids-standard/pybids

Issue:
bids_layout.py:121: ResourceWarning: unclosed file
File object of ``dataset_description.json`` is not closed.


Discussion:

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -107,7 +107,8 @@ def __init__(self, root, validate=True, index_associated=True,
                           "project root. Every valid BIDS dataset must have "
                           "this file.")
         else:
-            self.description = json.load(open(target, 'r'))
+            with open(target, 'r', encoding='utf-8') as desc_fd:
+                self.description = json.load(desc_fd)
             for k in ['Name', 'BIDSVersion']:
                 if k not in self.description:
                     raise ValueError("Mandatory '%s' field missing from "
@@ -183,7 +184,8 @@ def check_for_description(dir):
 
         for deriv in deriv_dirs:
             dd = os.path.join(deriv, 'dataset_description.json')
-            description = json.load(open(dd, 'r'))
+            with open(dd, 'r', encoding='utf-8') as ddfd:
+                description = json.load(ddfd)
             pipeline_name = description.get('PipelineDescription.Name', None)
             if pipeline_name is None:
                 raise ValueError("Every valid BIDS-derivatives dataset must "
@@ -587,8 +589,8 @@ def _get_metadata(self, path, **kwargs):
 
         for json_file_path in reversed(potential_jsons):
             if os.path.exists(json_file_path):
-                param_dict = json.load(open(json_file_path, "r",
-                                            encoding='utf-8'))
+                with open(json_file_path, 'r', encoding='utf-8') as fd:
+                    param_dict = json.load(fd)
                 results.update(param_dict)
 
         return results


Test patch:
Repository: bids-standard/pybids

Issue:
More useful _repr_ for BIDSFile
Since we switched to returning `BIDSFile` by default, the default results of `.get` is a bit ugly and not information. Let's add a `_repr_` to when printing these classes.

Ideally, you'd see all the entities associated with each file.


Discussion:
please!  FWIW, I thought at some point to suggest our "dirty" fix for such issues: we have a decorator `@auto_repr`  https://github.com/datalad/datalad/blob/master/datalad/utils.py#L165
and many classes use it;
```shell
$> git grep -A1 @auto_repr
datalad/downloaders/base.py:@auto_repr
datalad/downloaders/base.py-class DownloaderSession(object):
--
datalad/downloaders/base.py:@auto_repr
datalad/downloaders/base.py-@add_metaclass(ABCMeta)
--
datalad/downloaders/credentials.py:@auto_repr
datalad/downloaders/credentials.py-class Credential(object):
--
datalad/downloaders/credentials.py:@auto_repr
datalad/downloaders/credentials.py-class CompositeCredential(Credential):
```

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -44,6 +44,13 @@ def __getattr__(self, attr):
         raise AttributeError("%s object has no attribute named %r" %
                              (self.__class__.__name__, attr))
 
+    def __repr__(self):
+        source = ''
+        if self.layout.sources:
+            source = ", root='{}'".format(os.path.basename(self.layout.root))
+        return "<BIDSFile filename='{}'{}>".format(
+            os.path.relpath(self.path, start=self.layout.root), source)
+
     @property
     def image(self):
         """ Return the associated image file (if it exists) as a NiBabel object.
@@ -184,15 +191,16 @@ def check_for_description(dir):
 
         for p in paths:
             p = os.path.abspath(p)
-            if check_for_description(p):
-                deriv_dirs.append(p)
-            else:
-                subdirs = [d for d in os.listdir(p)
-                           if os.path.isdir(os.path.join(p, d))]
-                for sd in subdirs:
-                    sd = os.path.join(p, sd)
-                    if check_for_description(sd):
-                        deriv_dirs.append(sd)
+            if os.path.exists(p):
+                if check_for_description(p):
+                    deriv_dirs.append(p)
+                else:
+                    subdirs = [d for d in os.listdir(p)
+                               if os.path.isdir(os.path.join(p, d))]
+                    for sd in subdirs:
+                        sd = os.path.join(p, sd)
+                        if check_for_description(sd):
+                            deriv_dirs.append(sd)
 
         for deriv in deriv_dirs:
             dd = os.path.join(deriv, 'dataset_description.json')
@@ -335,6 +343,8 @@ def get(self, return_type='object', target=None, extensions=None,
 
         if derivatives == True:
             derivatives = list(self.derivatives.keys())
+        elif derivatives:
+            derivatives = listify(derivatives)
 
         # Separate entity kwargs from metadata kwargs
         ent_kwargs, md_kwargs = {}, {}


Test patch:
Repository: bids-standard/pybids

Issue:
validator.get_path_values raises KeyError: 'regexp'
What a great tutorial - keeps bearing fruits...  Continuing to #267 I was exploring new features of the validator, and found `get_path_values` which seems want ta path, so echoing `is_file` I pointed it to /sub-02/ses-01/anat/sub-02_ses-01_T2w.json to just get 
```
--------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-71-00eb4b2b179c> in <module>()
----> 1 validator.get_path_values('/sub-02/ses-01/anat/sub-02_ses-01_T2w.json')

/home/yoh/proj/bids/pybids/bids/layout/validation.pyc in get_path_values(self, path)
    164         values = {}
    165 
--> 166         regexps = self.get_regular_expressions('path.json')
    167 
    168         # capture subject

/home/yoh/proj/bids/pybids/bids/layout/validation.pyc in get_regular_expressions(self, filename)
    144             rule = rules[key]
    145 
--> 146             regexp = rule["regexp"]
    147 
    148             if "tokens" in rule:

KeyError: 'regexp'
```


Discussion:
We should probably just remove that whole section of the tutorial; I don't anticipate calling the validator directly being a common use case.
that function is not a part of the tutorial but just a part of the public api of the validator.  I was just trying things out.  

Resolving patch:
diff --git a/bids/layout/validation.py b/bids/layout/validation.py
--- a/bids/layout/validation.py
+++ b/bids/layout/validation.py
@@ -3,6 +3,7 @@
 import re
 import json
 from os.path import join, abspath, dirname
+from collections import namedtuple
 
 
 __all__ = ['BIDSValidator']
@@ -156,23 +157,6 @@ def get_regular_expressions(self, filename):
 
         return regexps
 
-    def get_path_values(self, path):
-        """Takes a file path and returns values found for the following path
-        keys:
-            sub-
-            ses-
-        """
-        values = {}
-
-        regexps = self.get_regular_expressions('path.json')
-
-        # capture subject
-        for paths in ['sub', 'ses']:
-            match = re.compile(regexps[paths]).findall(path)
-            values[paths] = match[1] if match & match[1] else None
-
-        return values
-
     def conditional_match(self, expression, path):
         match = re.compile(expression).findall(path)
         match = match[0] if len(match) >= 1 else False
@@ -249,6 +233,7 @@ def check_duplicate_files(layout):
     """
     
     import pandas as pd
+    import hashlib
     def md5(fname):
         hash_md5 = hashlib.md5()
         with open(fname, "rb") as f:


Test patch:
Repository: bids-standard/pybids

Issue:
get() breaks when return_type='dir'
When `return_type='dir'`, the result is always an empty list.


Discussion:

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -377,6 +377,11 @@ def get(self, return_type='object', target=None, extensions=None,
             (see return_type for details).
         """
 
+        # Warn users still expecting 0.6 behavior
+        if 'type' in kwargs:
+            raise ValueError("As of pybids 0.7.0, the 'type' argument has been"
+                             " replaced with 'suffix'.")
+
         if derivatives is True:
             derivatives = list(self.derivatives.keys())
         elif derivatives:
@@ -414,8 +419,9 @@ def get(self, return_type='object', target=None, extensions=None,
         # Get entity-based search results using the superclass's get()
         result = []
         result = super(
-            BIDSLayout, self).get(return_type, target, extensions, None,
-                                  regex_search, **ent_kwargs)
+            BIDSLayout, self).get(return_type, target=target,
+                                  extensions=extensions, domains=None,
+                                  regex_search=regex_search, **ent_kwargs)
 
         # Search the metadata if needed
         if return_type not in {'dir', 'id'}:


Test patch:
Repository: bids-standard/pybids

Issue:
BIDS validation fails because of missing leading /
The BIDS validator regular expressions expect absolute paths, but we've been feeding relative ones. Most files have been passing due to a separate bug in the validator regular expressions, but that was purely luck (or lack thereof). The `BIDSValidator` class should use absolute paths when checking against all rules.

See #339 for context.


Discussion:

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -305,17 +305,14 @@ def _validate_file(self, f):
         if not self.validate:
             return True
 
-        # For derivatives, we need to cheat a bit and construct a fake
-        # derivatives path--prepend 'derivatives' and the pipeline name
-        to_check = os.path.relpath(f, self.root)
+        # Derivatives are currently not validated.
         if 'derivatives' in self.domains:
-            to_check = os.path.join(
-                'derivatives', self.description['PipelineDescription']['Name'],
-                to_check)
+            return True
 
-        sep = os.path.sep
-        if to_check[:len(sep)] != sep:
-            to_check = sep + to_check
+        # BIDS validator expects absolute paths, but really these are relative
+        # to the BIDS project root.
+        to_check = os.path.relpath(f, self.root)
+        to_check = os.path.join(os.path.sep, to_check)
 
         return self.validator.is_bids(to_check)
 


Test patch:
Repository: bids-standard/pybids

Issue:
Design matrix has nvols + 1 rows after convolution
For a dataset with 451 volumes, running `Split()` and then `Convolve(model='spm')` creates a variable collection that, when we run `get_design_matrix()` the output is 452 rows long.

We have a bizarre theory that odd numbers of TRs is hitting a weird rounding issue, but in trying to create a minimal example, we've brought my computer to its knees trying to run `Convolve`.

@tyarkoni I think this is probably your fault.

cc @yarikoptic @AdinaWagner 


Discussion:

Resolving patch:
diff --git a/bids/variables/io.py b/bids/variables/io.py
--- a/bids/variables/io.py
+++ b/bids/variables/io.py
@@ -141,13 +141,16 @@ def _load_time_variables(layout, dataset=None, columns=None, scan_length=None,
         if 'run' in entities:
             entities['run'] = int(entities['run'])
 
+        tr = layout.get_metadata(img_f, suffix='bold', domains=domains,
+                                 full_search=True)['RepetitionTime']
+
         # Get duration of run: first try to get it directly from the image
         # header; if that fails, try to get NumberOfVolumes from the
         # run metadata; if that fails, look for a scan_length argument.
         try:
             import nibabel as nb
             img = nb.load(img_f)
-            duration = img.shape[3] * img.header.get_zooms()[-1]
+            duration = img.shape[3] * tr
         except Exception as e:
             if scan_length is not None:
                 duration = scan_length
@@ -158,9 +161,6 @@ def _load_time_variables(layout, dataset=None, columns=None, scan_length=None,
                        "available, or manually specify the scan duration.")
                 raise ValueError(msg)
 
-        tr = layout.get_metadata(img_f, suffix='bold', domains=domains,
-                                 full_search=True)['RepetitionTime']
-
         run = dataset.get_or_create_node('run', entities, image_file=img_f,
                                          duration=duration, repetition_time=tr)
         run_info = run.get_info()


Test patch:
Repository: bids-standard/pybids

Issue:
Mention old behavior from 0.6 in code or in documentation for layout.get()?
This could be covered by issue #266.

Would it be a good idea to make notes of old behavior from 0.6 in code or in documentation for layout.get()?

I inherited some code which used the old 'modality' keyword from 0.6 and if there were warnings I have missed them.
I should have looked in the changelog first, but I wonder if a note in the documentation mentioning the changes would be appropriate. Perhaps in:
https://bids-standard.github.io/pybids/generated/bids.layout.BIDSLayout.html#bids.layout.BIDSLayout.get

That there is a warning/mention of some changed behavior, but not all, caused some confusion as well:
https://github.com/bids-standard/pybids/blob/13c857fcd966078d7603ba71ed73bc39a6c42d37/bids/layout/layout.py#L383-L386

Sorry if I messed up the format, or misunderstood something.




Discussion:
I agree it would be good to have a note about this in the `get` docstring. Feel free to submit a PR (the docs are autogenerated, so updating the docstring in `bids.layout.layout` is sufficient), otherwise I'll get to it next time I make changes.

As far as the code itself goes, the change in `modality` keyword is a bit difficult to handle, because there still *is* a `modality` keyword—it just now describes something else and much more specific (to be more in line with the spec). So we can't raise an error for it the way we do for `type`; at best we could produce a warning, but my concern was that that might be more confusing than *not* having a warning, since anyone who's using the updated definition of `modality` will then see an unnecessary message they don't have to do anything about.
Thank you for your answer!
I would really like to make an attempt to add the information and will make a PR as soon as possible.

I completely understand that it would be disruptive to have an error or a warning when one uses the `modality` keyword. A note in the documentations combined with the changelog is probably the best solution.

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -465,6 +465,13 @@ def get(self, return_type='object', target=None, extensions=None,
 
         Returns:
             A list of BIDSFiles (default) or strings (see return_type).
+
+        Notes:
+            As of pybids 0.7.0 some keywords have been changed. Namely: 'type'
+            becomes 'suffix', 'modality' becomes 'datatype', 'acq' becomes 
+            'acquisition' and 'mod' becomes 'modality'. Using the wrong version 
+            could result in get() silently returning wrong or no results. See 
+            the changelog for more details.
         """
 
         # Warn users still expecting 0.6 behavior


Test patch:
Repository: bids-standard/pybids

Issue:
minor demo/documentation typos
Cell 9 of the demo IPython notebook states
```Python
# Ask get() to return the ids of subjects that have T1w files
layout.get(return_type='id', target='subject')
```
but I think this should be specifying the suffic argument to do what is described in the comment?
```Python
# Ask get() to return the ids of subjects that have T1w files
layout.get(suffix='T1w', return_type='id', target='subject')
```


Also in section 3.2 of the docs, the text refers to `model.json` but the example code uses `model1.json` instead.



Discussion:

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -868,7 +868,17 @@ def build_path(self, source, path_patterns=None, strict=False, scope='all'):
                   values
             path_patterns (list): Optional path patterns to use to construct
                 the new file path. If None, the Layout-defined patterns will
-                be used.
+                be used. Entities should be represented by the name
+                surrounded by curly braces. Optional portions of the patterns
+                should be denoted by square brackets. Entities that require a
+                specific value for the pattern to match can pass them inside
+                carets. Default values can be assigned by specifying a string
+                after the pipe operator. E.g., (e.g., {type<image>|bold} would
+                only match the pattern if the entity 'type' was passed and its
+                value is "image", otherwise the default value "bold" will be
+                used).
+                    Example: 'sub-{subject}/[var-{name}/]{id}.csv'
+                    Result: 'sub-01/var-SES/1045.csv'
             strict (bool): If True, all entities must be matched inside a
                 pattern in order to be a valid match. If False, extra entities
                 will be ignored so long as all mandatory entities are found.
diff --git a/bids/reports/report.py b/bids/reports/report.py
--- a/bids/reports/report.py
+++ b/bids/reports/report.py
@@ -63,7 +63,11 @@ def generate(self, **kwargs):
         -------
         counter : :obj:`collections.Counter`
             A dictionary of unique descriptions across subjects in the dataset,
-            along with the number of times each pattern occurred.
+            along with the number of times each pattern occurred. In cases
+            where all subjects underwent the same protocol, the most common
+            pattern is most likely the most complete. In cases where the
+            dataset contains multiple protocols, each pattern will need to be
+            inspected manually.
 
         Examples
         --------


Test patch: