Repository: bids-standard/pybids

Issue:
Poor performance of get_subjects() even when using indexing without metadata
I am puzzled by how slow get_subjects() is for me. It takes about 10 seconds to retrieve the subject ids for a 67 subject BIDS directory. This is after some improvement achieved by using a saved index file and not indexing metadata - two of the most common performance enhancement suggestions I see here.

I don't understand why gathering the subject IDs in such a small database wouldn't be almost instantaneous. Meanwhile, when I query for a single subject, the result is as quick as you'd expect.

Here is how I fetch my layout, using version 15.5:

```
bids_dir = "path/to/my/bids_dir"
database_path = "path/to/my/index"

indexer = BIDSLayoutIndexer(index_metadata=False)

BIDSLayout(
  bids_dir,
  indexer=indexer, database_path=database_path,
  reset_database=refresh
)
```

Is there some way I can configure the indexer to improve my performance here, or some other solution?

Thanks!



Discussion:
We just merged a bug-fix to the indexer (#936). Would you be willing to install master and see if the issue remains? The bug was related to traversing ignored directories despite them being ignored.

`pip install git+https://github.com/bids-standard/pybids.git`
I just tried and unfortunately get_subjects() is still taking the same amount of time.
Can you elaborate on what you mean by "querying for a single subject"?

I do recall this being rather slow at times, so I'll take a look. My guess is that it has to do with taking the "unique" set of subjects, and it's doing too many pre-querying to do so. 
So on a random dataset I had, I couldn't quite replicate your issue, but I do see that it takes 2 orders of magnitudes more to do `get_subjects` vs `get`:

```
In [7]: %timeit layout.get(subject='1')
5.02 ms ± 439 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [8]: %timeit layout.get_subjects()
660 ms ± 2.73 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

The reason for this is that `get_subjects` is not a special function, but rather a shortcut for the more general form: `layout.get(return_type='id', target='subject')`

I'll see if there's any quick fixes for this, but if not there is a major refactor coming up which will speed up pybids significantly: https://github.com/bids-standard/pybids/pull/863
Yup, so by querying a single subject, I mean the same thing as in your timing example - using `layout.get(subject='123')`

As far as replicating my issue, I believe your test showing a two order of magnitude difference in `get_subjects()` vs `(get(subject='123'))` captures exactly what I'm dealing with.

I appreciate you looking into this for me!


Maybe there is some way for me to index on 'id'?
Looks like it's slow because of this line:

https://github.com/bids-standard/pybids/blob/1ba4f66f590045bd2476bb63dbb735f2e26bee23/bids/layout/layout.py#L679

Essentially its asking for the meta-data of every indexed file, to later filter by only those that have the target (in this case `subject`) in their metadata. 

`.get_entities` is > 2 orders of magnitude slower than just checking `.entities`. I'm trying to figure out if just using the attribute would be reliable enough here. 

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -533,7 +533,6 @@ def to_df(self, metadata=False, **filters):
             data.loc[o] = pd.Series(dtype=float)
 
         return data.reset_index()
-
     def get(self, return_type='object', target=None, scope='all',
             regex_search=False, absolute_paths=None, invalid_filters='error',
             **filters):
@@ -673,10 +672,8 @@ def get(self, return_type='object', target=None, scope='all',
                 raise TargetError('If return_type is "id" or "dir", a valid '
                                  'target entity must also be specified.')
 
-            metadata = target not in self.get_entities(metadata=False)
-
             if return_type == 'id':
-                ent_iter = (x.get_entities(metadata=metadata) for x in results)
+                ent_iter = (x.entities for x in results)
                 results = list({
                     ents[target] for ents in ent_iter if target in ents
                 })
diff --git a/bids/layout/models.py b/bids/layout/models.py
--- a/bids/layout/models.py
+++ b/bids/layout/models.py
@@ -312,6 +312,9 @@ def get_entities(self, metadata=False, values='tags'):
             A dict, where keys are entity names and values are Entity
             instances.
         """
+        if metadata is None and values == 'tags':
+            return self.entities
+
         session = object_session(self)
         query = (session.query(Tag)
                  .filter_by(file_path=self.path)


Test patch:
diff --git a/bids/layout/tests/test_layout.py b/bids/layout/tests/test_layout.py
--- a/bids/layout/tests/test_layout.py
+++ b/bids/layout/tests/test_layout.py
@@ -683,6 +683,9 @@ def test_get_tr(layout_7t_trt):
     tr = layout_7t_trt.get_tr(subject=['01', '02'], acquisition="prefrontal")
     assert tr == 4.0
 
+    tr = layout_7t_trt.get_RepetitionTime()
+    assert sum([t in tr for t in [3.0, 4.0]]) == 2
+
 
 def test_to_df(layout_ds117):
     # Only filename entities