Repository: bids-standard/pybids

Issue:
.get returns double results when validate=False
Let's say I set up a derivative as a root directory, as a derivative (because I want it to use derivative entities, such as `desc`): Let's say I also said `validate=False`:

    layout = BIDSLayout('/data/fmriprep', derivatives=['/data/fmriprep'], validate=False)

Then if I ask for the following:

    layout.get(task='sherlockPart1', subject='01', datatype='func', suffix='bold')

I will get two results:

```
[<BIDSImageFile filename='/work/Sherlock/preproc/fmriprep/sub-01/func/sub-01_task-sherlockPart1_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz'>,
 <BIDSImageFile filename='/work/Sherlock/preproc/fmriprep/sub-01/func/sub-01_task-sherlockPart1_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz'>]
```

What is happening is that both the root layout, and the derivative layout are returning results.

This happens even if I ask for results only from derivatives:

    layout.get(task='sherlockPart1', subject='01', datatype='func', suffix='bold', scope='derivatives')

This happens because `_in_scope` determines if a layout is in the scope of `derivatives`  based on the `PipelineDescription`, which the root derivative has, even though derivative results were not applied when indexing.

If I `validate` the layout, then I only get one result, because the preproc file is not indexed at all by the root layout.

____

The reason this is relevant for fitlins, is that as @effigies pointed out, BIDS Model are not valid BIDS yet so we have to run with `validate=False`. 

I think the simple solution for `fitlins` is to add `desc=preproc` as a entity to filter that `get` call on, so that we only get 1 preproc file returned.

But I think it's a bug to say the root derivative is in the `derivatives` scope, even when that set of entities were not applied during indexing. 
Replace each nested list comprehension with a single DB query in BIDSLayout.__repr__
I think there is currently some serious performance issues with BIDSLayout. Using a somewhat average database of 132 subjects (1 session and 1 run per subject), it needs about 1:15 minute to get a layout object. Using the following code:

```
from bids import BIDSLayout
%lprun -f BIDSLayout.__init__ BIDSLayout("/media/christian/ElementsSE/MPI-Leipzig_Mind-Brain-Body-LEMON/BIDS_LEMON/")
```

I get the following profiling report :

```
Total time: 76.4714 s
File: /home/christian/pybids/bids/layout/layout.py
Function: __init__ at line 196

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   196                                               def __init__(self, root, validate=True, absolute_paths=True,
   197                                                            derivatives=False, config=None, sources=None, ignore=None,
   198                                                            force_index=None, config_filename='layout_config.json',
   199                                                            regex_search=False, database_path=None, database_file=None,
   200                                                            reset_database=False, index_metadata=True):
   201                                                   """Initialize BIDSLayout."""
   202         1          4.0      4.0      0.0          self.root = str(root)
   203         1          2.0      2.0      0.0          self.validate = validate
   204         1          1.0      1.0      0.0          self.absolute_paths = absolute_paths
   205         1          2.0      2.0      0.0          self.derivatives = {}
   206         1          2.0      2.0      0.0          self.sources = sources
   207         1          3.0      3.0      0.0          self.regex_search = regex_search
   208         1          2.0      2.0      0.0          self.config_filename = config_filename
   209                                                   # Store original init arguments as dictionary
   210         1          3.0      3.0      0.0          self._init_args = self._sanitize_init_args(
   211         1          2.0      2.0      0.0              root=root, validate=validate, absolute_paths=absolute_paths,
   212         1          2.0      2.0      0.0              derivatives=derivatives, ignore=ignore, force_index=force_index,
   213         1         91.0     91.0      0.0              index_metadata=index_metadata, config=config)
   214                                           
   215         1          4.0      4.0      0.0          if database_path is None and database_file is not None:
   216                                                       database_path = database_file
   217                                                       warnings.warn(
   218                                                           'In pybids 0.10 database_file argument was deprecated in favor'
   219                                                           ' of database_path, and will be removed in 0.12. '
   220                                                           'For now, treating database_file as a directory.',
   221                                                           DeprecationWarning)
   222         1          4.0      4.0      0.0          if database_path:
   223                                                       database_path = str(Path(database_path).absolute())
   224                                           
   225         1         47.0     47.0      0.0          self.session = None
   226                                           
   227         1      25891.0  25891.0      0.0          index_dataset = self._init_db(database_path, reset_database)
   228                                           
   229                                                   # Do basic BIDS validation on root directory
   230         1        488.0    488.0      0.0          self._validate_root()
   231                                           
   232         1          4.0      4.0      0.0          if ignore is None:
   233         1          3.0      3.0      0.0              ignore = self._default_ignore
   234                                           
   235                                                   # Instantiate after root validation to ensure os.path.join works
   236         1          3.0      3.0      0.0          self.ignore = [os.path.abspath(os.path.join(self.root, patt))
   237                                                                  if isinstance(patt, str) else patt
   238         1        102.0    102.0      0.0                         for patt in listify(ignore or [])]
   239         1          3.0      3.0      0.0          self.force_index = [os.path.abspath(os.path.join(self.root, patt))
   240                                                                       if isinstance(patt, str) else patt
   241         1          4.0      4.0      0.0                              for patt in listify(force_index or [])]
   242                                           
   243                                                   # Initialize the BIDS validator and examine ignore/force_index args
   244         1          4.0      4.0      0.0          self._validate_force_index()
   245                                           
   246         1          1.0      1.0      0.0          if index_dataset:
   247                                                       # Create Config objects
   248         1          2.0      2.0      0.0              if config is None:
   249         1          2.0      2.0      0.0                  config = 'bids'
   250         1          1.0      1.0      0.0              config = [Config.load(c, session=self.session)
   251         1      62271.0  62271.0      0.1                        for c in listify(config)]
   252         1         15.0     15.0      0.0              self.config = {c.name: c for c in config}
   253                                                       # Missing persistence of configs to the database
   254         2          6.0      3.0      0.0              for config_obj in self.config.values():
   255         1        308.0    308.0      0.0                  self.session.add(config_obj)
   256         1      27372.0  27372.0      0.0                  self.session.commit()
   257                                           
   258                                                       # Index files and (optionally) metadata
   259         1         35.0     35.0      0.0              indexer = BIDSLayoutIndexer(self)
   260         1   28127988.0 28127988.0     36.8              indexer.index_files()
   261         1          3.0      3.0      0.0              if index_metadata:
   262         1   48226769.0 48226769.0     63.1                  indexer.index_metadata()
   263                                                   else:
   264                                                       # Load Configs from DB
   265                                                       self.config = {c.name: c for c in self.session.query(Config).all()}
   266                                           
   267                                                   # Add derivatives if any are found
   268         1          3.0      3.0      0.0          if derivatives:
   269                                                       if derivatives is True:
   270                                                           derivatives = os.path.join(root, 'derivatives')
   271                                                       self.add_derivatives(
   272                                                           derivatives, parent_database_path=database_path,
   273                                                           validate=validate, absolute_paths=absolute_paths,
   274                                                           derivatives=None, sources=self, ignore=ignore,  config=None,
   275                                                           force_index=force_index, config_filename=config_filename,
   276                                                           regex_search=regex_search, index_metadata=index_metadata,
   277                                                           reset_database=index_dataset or reset_database
   278                                                           )
```

For day-to-day interaction with a dataset, development tests, etc.,  this kind of delay seems prohibitive to me...


Discussion:
Agreed, definitely a bug. Thanks!
It's not really avoidable (without a lot of optimization work) if you want the benefits of metadata indexing. If you don't need to search metadata, you can turn off `index_metadata` and that should speed things up by an order of magnitude. Alternatively, you can make sure to pass a `database_path` argument, in which case the indexed layout will be written out to disk the first time indexing happens, and subsequent initializations will be instantaneous.
Thanks for the tip about database_path. I should have seen it by myself in the doc; mea culpa. It would solve the issue (at least for my use-case)... but there is still something fishy with the (ORM management involved in) BIDSLayout. For example, if I time the execution of the code, it runs blazingly fast the second time I initialize a BIDSLayout object with the saved database:

```
from bids import BIDSLayout
path = "/media/christian/ElementsSE/MPI-Leipzig_Mind-Brain-Body-LEMON/BIDS_LEMON/"

import time
start_time = time.time()
BIDSLayout(path, database_path=path)
print("--- %s seconds ---" % (time.time() - start_time))
```

produces 

```
--- 0.0100860595703125 seconds ---
```

But the same code without the timing and printing part

```
from bids import BIDSLayout
path = "/media/christian/ElementsSE/MPI-Leipzig_Mind-Brain-Body-LEMON/BIDS_LEMON/"

BIDSLayout(path, database_path=path)
```

timed with a good old fashion chronometer took about 60s to run. When interrupted in the middle of this execution, it is always running within SQLAlchemy ORM methods. After some tests, I realize it is not related to the timing function specifically but to almost any operation called after BIDSLayout().

It is harder to demonstrate with PyBIDS test data because these datasets are much more lightweight which makes the timing much shorter, but still, it is reproducible. For example, running the following code (as a cell in jupyter cell or ipython block, or as a script...):

```
from pathlib import Path
import bids
import time

start_time = time.time()
BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt")
print("--- %s seconds ---" % (time.time() - start_time))

start_time = time.time()
BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(Path.home()))
print("--- %s seconds ---" % (time.time() - start_time))

start_time = time.time()
BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(Path.home()))
print("--- %s seconds ---" % (time.time() - start_time))
```

produces the following timing:

```
--- 2.427520513534546 seconds ---
--- 3.9162495136260986 seconds ---
--- 0.0035071372985839844 seconds ---
```

Then, after this code running once more time the last call without the timing, i.e.

```
BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(Path.home()))
```

execute in around 2s instead of the 0.003s obtained when using timing... Want to run it faster? Just wrap it a function. For example, this runs almost instantaneously:

```
def f1():
    BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(Path.home()))
f1()
```
same for any other instructions... For example, 
```
BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(Path.home()))
x = 1
```

also runs very fast. Can you reproduce this weird behavior on your end? As mentioned earlier, for this test sample, the differences seem not important (2s versus 0s) but on other datasets, we are speaking of minutes... I think there is probably something buggy somewhere in the specification of the ORM.... but it is just a guess.

@adelavega, any thoughts? I think you wrote the logic around `database_path`, and this seems to be a new issue (though I haven't attempted to reproduce it myself).
Ok... I got something better for you to investigate this. Here is what I think a reproducible test that could be used to avoid regression related to that:

```
import time
start_time = time.time()
print(BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(Path.home())))
time1 = time.time() - start_time

start_time = time.time()
BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(Path.home()))
time2 = time.time() - start_time

assert(time1/time2 < 10)
```

This currently fails for me.
Interesting... Will try to reproduce when I have a moment. I haven't seen this before, and it does look pretty funky.
That's pretty odd, but at least from what I gather, the database loading seems to be working. Rather it seems like an odd issue related to SQLAlchemy overhead? 

I'll see if I can reproduce.
@adelavega Yes, it loads fine aside from loading times due to this overhead.
The issue is not the `BIDSLayout` creation. And I think your test doesn't capture the issue, because the timing looks fine for me:

    In [8]: time1                                                                   
    Out[8]: 5.847425699234009

    In [9]: time2                                                                   
    Out[9]: 0.0030126571655273438


The tests still fails, because `time1/time2` is more than `10`. That is, the speed is up actually 1940x. That's fine.

However, I know why when you run the line again "without the timing" functions its longer.

In this case you are not only asking to create the layout, but also asking it to return a the `__repr__` summary of the `BIDSLayout` content, which takes some time to populate. 

If you were to do the following:

    layout = BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(p))  

then it would be instantaneous, because that doesn't trigger printing a summary.
    


I've actually had this same confusion before, so this tells me two things:
1) The `__repr__` should be simplified as to not choke up accidental prints
2) Performance with large datasets can be quite slow, and that is a more general problem. Looking like doing `index_metadata=False` has no effect on this, and I don't think there are any easy answers here.
Actually, looking at this more closely, looks `__repr__` is not using the sql queries, and is doing some slow nested list comprehensions, so this is probably the issue. There is a TODO comment to implement this. 

Especially calculating the number of sessions, looking over subjects, is pretty slow. 

https://github.com/bids-standard/pybids/blob/35e1296202959d375e570d08078282c26ad02bc0/bids/layout/layout.py#L302
@adelavega I think you are right that the different issues I observed (aside from the long initial indexing which is unavoidable) were dure to the call to `__repr__`. It gets even messier when working in a context like notebooks because `__repr__` is called automatically on the output of the last instruction. It gives apparently silly situations like
```
BIDSLayout(...)
```
being very slow, whereas
```
BIDSLayout(...)
print(" ")
```
is very fast. A bit confusing at first, but it makes sense now. It will definitely be nice to have a faster `__repr__` so that the code does not seem to hang for a long time on larger dataset because we mistakenly trigger a `__repr__` somewhere. For now, I'll just comment out the whole `__repr__` method on my local copy. Thanks for the input.

On second thought, is there a reason why these list comprehensions could not just be replaced by 
```
self = layout

root = self.root[-30:]
n_subjects = self.entities["subject"].count()
n_sessions = self.entities["session"].count()
n_runs = self.entities["run"].count()

s = ("BIDS Layout: ...{} | Subjects: {} | Sessions: {} | "
             "Runs: {}".format(root, n_subjects, n_sessions, n_runs))
print(s)
```
? This code runs very fast for me... whereas `repr(layout)` takes long.
Works for me...
```
from pathlib import Path
from bids import BIDSLayout
import bids
import time

layout = BIDSLayout(Path(bids.__file__).parent / "tests" / "data" / "7t_trt", database_path=(Path.home()),
                        reset_database=True)

start_time = time.time()
print(repr(layout))
print(time.time() - start_time)

def __repr2__(self):
    """Provide a tidy summary of key properties."""
    # TODO: Replace each nested list comprehension with a single DB query
    n_subjects = self.entities["subject"].count()
    n_sessions = self.entities["session"].count()
    n_runs = self.entities["run"].count()
    root = self.root[-30:]
    s = ("BIDS Layout: ...{} | Subjects: {} | Sessions: {} | "
         "Runs: {}".format(root, n_subjects, n_sessions, n_runs))
    return s

BIDSLayout.__repr__ = __repr2__

start_time = time.time()
print(repr(layout))
print(time.time() - start_time)
```
gives
```
BIDS Layout: .../pybids/bids/tests/data/7t_trt | Subjects: 10 | Sessions: 20 | Runs: 20
1.5868823528289795
BIDS Layout: .../pybids/bids/tests/data/7t_trt | Subjects: 10 | Sessions: 2 | Runs: 2
0.015100479125976562
```
Well, they don't give the same information... I think it's useful to know at a glance that, e.g., some sessions/runs are missing for some subjects. But I don't feel strongly about it one way or the other (and it wouldn't be more than a few lines to build the queries that return the total number of sessions and runs).
Yeah I sort of agree with getting the total number being nice. I looked briefly yesterday at building the queries and got stumped. Is it as easy as getting the total sessions, or do you have to nest with subjects?
I imagine the sql query is faster but it'd be good to test on a large dataset anyways. 
You could build a giant query of the kind that `_build_file_query()` constructs, but it might be more trouble than it's worth. Easier (and probably only slightly slower) is probably to do some filtering in Python. For sessions it's much easier, because `BIDSFile` instances have an `is_dir` property, so you can retrieve all the `BIDSFile` instances that have `is_dir=True` and at least one tag with `entity="session"`. Runs are a more difficult proposition. It might not be crazy to start by querying for all files with `entity="run"` and then just looping over them in Python. If that's passably fast (seems like it should be), that would be fine in my book.

It would probably be a good idea to add some internal utilities for this kind of thing (e.g., the pattern of wanting to filter the `BIDSFile` table on multiple joined tags is pretty common).
From digging through `get()`:

```Python
def __repr__(self):
    query = self._build_file_query()
    all_files = query.options(joinedload(bids.layout.BIDSFile.tags)
                              .joinedload(bids.layout.Tag.entity)).all()
    filtered = [f for f in all_files if 'subject' in f.entities] 
    sub = [f.entities['subject'] for f in filtered] 
    sub_sess = [(f.entities['subject'], f.entities['session']) for f in filtered if 'session' in f.entities]
    sub_run = [(f.entities['subject'], f.entities['run']) for f in filtered if 'run' in f.entities]
    n_subjects = len(set(sub))
    n_sessions = len(set(sub_sess))
    n_runs = len(set(sub_run))
    root = self.root[-30:]
    s = ("BIDS Layout: ...{} | Subjects: {} | Sessions: {} | "
         "Runs: {}".format(root, n_subjects, n_sessions, n_runs))
    return s
```

```Python
In [45]: %timeit __repr__(layout)                                                                   
73.1 ms ± 617 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [46]: %timeit repr(layout)                                                                       
1.48 s ± 95.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
That looks good performance-wise, but I worry it'll choke for large datasets, as it requires retrieval of basically the entire DB. 
Doesn't that mean all `get()` queries will also choke?
No, because `get()` passes the filters to `_build_file_query()`, so the filtering is done during query construction.

EDIT: Oh, you mean if you call it without any arguments? Yeah, but that's on the user. I think you should expect to get back a million records in that case, whereas I don't think the expectation is that you're loading the entire DB into memory just to print some summary info.
Well, right now `get_subjects()` and the like don't pass any filters.
...and that's probably why @christian-oreilly's `__repr__` calls are taking so long. ;)

I think the sensible solution here would be to introduce an intermediate layer between SQLAlchemy and `get()` that provides a more sensible API for common queries. In the interim, I'm not sure how many people actually use the `get_*` methods, so I don't see optimizing them as a higher priority, but probably we should do that too at some point.
In the short term, I think an inelegant but probably economical solution to the `__repr__` problem would be to retrieve all the `Tag` records (without any joins) and filter on those. They contain both `entity_name` and `file_path` fields, so they're sufficient for counting distinct records (though one would still need to join with `BIDSFile` to use this solution for the `get_*` methods).
How about using `distinct` in SQL? I just was confused a bit by the models to implement, but you should be able to do something like `db.session.query(Tag).filter_by(entitiy_name='session').distinct('file_path').count()`

Right?
That'll give you the count of all unique files that contain a session entity, which is not what we want. The `Tag` model is just a many-to-many join table from `BIDSFile` to `Entity`.
Ah yeah, you're right. I'm still not getting how to use just the Tag model to count the distinct combinations of subject and session though. 

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -299,12 +299,30 @@ def __getattr__(self, key):
 
     def __repr__(self):
         """Provide a tidy summary of key properties."""
-        # TODO: Replace each nested list comprehension with a single DB query
-        n_sessions = len([session for isub in self.get_subjects()
-                          for session in self.get_sessions(subject=isub)])
-        n_runs = len([run for isub in self.get_subjects()
-                      for run in self.get_runs(subject=isub)])
-        n_subjects = len(self.get_subjects())
+        n_subjects = len(
+            [s.value
+             for s in self.session.query(Tag).filter_by(
+                 entity_name='subject').group_by(Tag._value)]
+            )
+
+        n_sessions = len(
+            set(
+                (t.value, t.file.entities.get('subject'))
+                 for t in
+                 self.session.query(Tag).filter_by(entity_name='session')
+                 if t.file.entities.get('subject')
+                 )
+            )
+
+        n_runs = len(
+            set(
+                (t.value, t.file.entities.get('subject'))
+                 for t in
+                 self.session.query(Tag).filter_by(entity_name='run')
+                 if t.file.entities.get('subject')
+                 )
+            )
+
         root = self.root[-30:]
         s = ("BIDS Layout: ...{} | Subjects: {} | Sessions: {} | "
              "Runs: {}".format(root, n_subjects, n_sessions, n_runs))
@@ -487,7 +505,7 @@ def _in_scope(self, scope):
         # We assume something is a BIDS-derivatives dataset if it either has a
         # defined pipeline name, or is applying the 'derivatives' rules.
         pl_name = self.description.get("PipelineDescription", {}).get("Name")
-        is_deriv = bool(pl_name or ('derivatives' in self.config))
+        is_deriv = bool('derivatives' in self.config)
 
         return ((not is_deriv and 'raw' in scope) or
                 (is_deriv and ('derivatives' in scope or pl_name in scope)))


Test patch: