Repository: bids-standard/pybids

Issue:
partial metadata indexing?
I'm curious if anyone has thoughts on partial metadata indexing if metadata indexing takes so long (I do not know if partial metadata indexing would even be helpful, depending on what's taking up the most time).

I patched [BIDSLayoutIndexer's index_metadata function](https://github.com/jdkent/NiBetaSeries/blob/e92f952cdd6136fb85dc0707f6b683efb9cd674c/src/nibetaseries/workflows/utils.py#L19) to allow filters on the get data query.

## Use Case
I'm indexing a large(ish) dataset and only use the metadata associated with the bold files, so I only want to index the metadata for the bold files.

## Question
Would it beneficial (i.e., quicker indexing) to have the option to index metadata for a specific subset of files for a tool that only cares about that subset of files?


Discussion:
Partial indexing sounds like a good idea. Your implementation makes a lot of sense, and I'd be happy to review a PR. My only reservation is that unless we want to add yet another argument to the `BIDSLayout` initializer (which I don't think we do), users would have to first disable `index_metadata`, and then call `index_metadata()` explicitly on the `BIDSLayoutIndexer`. But that seems like a reasonable price to pay for this functionality.
I'll get a pull request up soon(ish). We could treat`index_metadata` similar to derivatives, where the input can either be a boolean or something else (in this case a dictionary that represents parameters we can pass to .`get`). We probably want to ensure return_type and make sure the json extension is included, and perhaps some other things I'm missing.

Resolving patch:
diff --git a/bids/layout/index.py b/bids/layout/index.py
--- a/bids/layout/index.py
+++ b/bids/layout/index.py
@@ -164,10 +164,32 @@ def index_files(self):
         """Index all files in the BIDS dataset. """
         self._index_dir(self.root, self.config)
 
-    def index_metadata(self):
-        """Index metadata for all files in the BIDS dataset. """
+    def index_metadata(self, **filters):
+        """Index metadata for all files in the BIDS dataset.
+
+        Parameters
+        ----------
+
+        **filters
+            keyword arguments passed to the .get() method of a
+            :obj:`bids.layout.BIDSLayout` object.
+            These keyword arguments define what files get selected
+            for metadata indexing.
+        """
+        
+        if filters:
+            # ensure we are returning objects
+            filters['return_type'] = 'object'
+            # until 0.11.0, user can specify extension or extensions
+            ext_key = 'extensions' if 'extensions' in filters else 'extension'
+            if filters.get(ext_key):
+                filters[ext_key] = listify(filters[ext_key])
+                # ensure json files are being indexed
+                if 'json' not in filters[ext_key]:
+                    filters[ext_key].append('json')
+
         # Process JSON files first if we're indexing metadata
-        all_files = self.layout.get(absolute_paths=True)
+        all_files = self.layout.get(absolute_paths=True, **filters)
 
         # Track ALL entities we've seen in file names or metadatas
         all_entities = {}


Test patch:
diff --git a/bids/layout/tests/test_layout.py b/bids/layout/tests/test_layout.py
--- a/bids/layout/tests/test_layout.py
+++ b/bids/layout/tests/test_layout.py
@@ -12,6 +12,7 @@
 import bids
 from bids.layout import (BIDSLayout, parse_file_entities, add_config_paths,
                          Query)
+from bids.layout.index import BIDSLayoutIndexer
 from bids.layout.models import Entity, Config
 from bids.tests import get_test_data_path
 from bids.utils import natural_sort
@@ -21,6 +22,28 @@ def test_layout_init(layout_7t_trt):
     assert isinstance(layout_7t_trt.files, dict)
 
 
+@pytest.mark.parametrize(
+    'index_metadata,query,result',
+    [
+        (True, None, 3.0),
+        (False, None, None),
+        (False, {}, 3.0),
+        (False, {'task': 'rest'}, 3.0),
+        (False, {'task': 'rest', 'extension': ['nii.gz']}, 3.0),
+        (False, {'task': 'rest', 'extension': 'nii.gz'}, 3.0),
+        (False, {'task': 'rest', 'extension': ['nii.gz', 'json'], 'return_type': 'file'}, 3.0),
+    ])
+def test_index_metadata(index_metadata, query, result):
+    data_dir = join(get_test_data_path(), '7t_trt')
+    layout = BIDSLayout(data_dir, index_metadata=index_metadata)
+    if not index_metadata and query is not None:
+        indexer = BIDSLayoutIndexer(layout)
+        indexer.index_metadata(**query)
+    sample_file = layout.get(task='rest', extension='nii.gz', acq='fullbrain')[0]
+    metadata = sample_file.get_metadata()
+    assert metadata.get('RepetitionTime') == result
+
+
 def test_layout_repr(layout_7t_trt):
     assert "Subjects: 10 | Sessions: 20 | Runs: 20" in str(layout_7t_trt)