Repository: bids-standard/pybids

Issue:
memory issues when getting tsv in derivatives
Can't see this issue anywhere else. 

I can use `BIDSLaout.get()` to get the nifti files, and it does it in seconds. But if I try and get the confound tsv files instead, the python session will get killed cause it uses over 16GB of RAM to do it. 

> layout.get(scope='fMRIPrep', desc='confounds', suffix='regressors', extension='tsv', return_type='file')

Getting one subject will work use around 10GB of RAM (and getting per subject does not feel ideal). 

The following will work: 

> layout.get(scope='derivatives', desc='preproc', extension='nii.gz')

So the problem is only when getting tsv files. It doesn't matter if I set return_type to something else (I put return_type='file' in the example because its not just a problem when returning BIDSDataFile). There are not more confounds tsv files than preproc nifti files either. 

Is this a known/common problem or could there be a specific reason why I am hitting it? It feels counter-intuitive that getting the tsvs take up so much RAM. Is there a way around it which isn't just loop over subjects? 

## Information about dataset

Not sure if any of these details are relevant. But might be helpful.

I have a BIDS directory with ca 250 runs in total. There are about 27 subjects, 2 sessions and ca 5-9 runs for one session and 1-4 runs for another session. len(layout.files) is 8240. The derivatives are in fMRIPrep. There are some other directories in the derivatives, but nothing that is added. 

## Full code of what I am doing below: 

```
import bids
bids_dir = './' # pwd is in BIDS dataset
fmriprep_dir = 'derivatives/fmriprep-1.5.1/fmriprep/'
layout = bids.BIDSLayout(bids_dir)
layout.add_derivatives(bids_dir + fmriprep_dir)
# This will work
layout.get(scope='fMRIPrep', desc='preproc', extension='nii.gz')
# This will crash the session and use 100% memory on laptop with 16GB RAM
layout.get(scope='fMRIPrep', desc='confounds', suffix='regressors', extension='tsv', return_type='file')
```

Bids version: 0.9.4
memory issues when getting tsv in derivatives
Can't see this issue anywhere else. 

I can use `BIDSLaout.get()` to get the nifti files, and it does it in seconds. But if I try and get the confound tsv files instead, the python session will get killed cause it uses over 16GB of RAM to do it. 

> layout.get(scope='fMRIPrep', desc='confounds', suffix='regressors', extension='tsv', return_type='file')

Getting one subject will work use around 10GB of RAM (and getting per subject does not feel ideal). 

The following will work: 

> layout.get(scope='derivatives', desc='preproc', extension='nii.gz')

So the problem is only when getting tsv files. It doesn't matter if I set return_type to something else (I put return_type='file' in the example because its not just a problem when returning BIDSDataFile). There are not more confounds tsv files than preproc nifti files either. 

Is this a known/common problem or could there be a specific reason why I am hitting it? It feels counter-intuitive that getting the tsvs take up so much RAM. Is there a way around it which isn't just loop over subjects? 

## Information about dataset

Not sure if any of these details are relevant. But might be helpful.

I have a BIDS directory with ca 250 runs in total. There are about 27 subjects, 2 sessions and ca 5-9 runs for one session and 1-4 runs for another session. len(layout.files) is 8240. The derivatives are in fMRIPrep. There are some other directories in the derivatives, but nothing that is added. 

## Full code of what I am doing below: 

```
import bids
bids_dir = './' # pwd is in BIDS dataset
fmriprep_dir = 'derivatives/fmriprep-1.5.1/fmriprep/'
layout = bids.BIDSLayout(bids_dir)
layout.add_derivatives(bids_dir + fmriprep_dir)
# This will work
layout.get(scope='fMRIPrep', desc='preproc', extension='nii.gz')
# This will crash the session and use 100% memory on laptop with 16GB RAM
layout.get(scope='fMRIPrep', desc='confounds', suffix='regressors', extension='tsv', return_type='file')
```

Bids version: 0.9.4


Discussion:
I can semi-reproduce this with [///labs/poldrack/ds003_fmriprep](https://datasets.datalad.org/?dir=/labs/poldrack/ds003_fmriprep):

```Python
import bids
layout = bids.BIDSLayout('ds003_fmriprep/sourcedata', derivatives='ds003_fmriprep')
layout.get(desc='confounds', suffix='regressors', extension='tsv', return_type='file')
```

It consumes a fair bit of time and memory, although it does complete. A little poking around reveals the line that takes all the time to be:

https://github.com/bids-standard/pybids/blob/74cdc330cc9a79663f8ecdb52c75e52a0d3f07bb/bids/layout/layout.py#L831

Adding a debug break, `print(str(query))` shows:

```SQL
SELECT files.path AS files_path, files.filename AS files_filename, files.dirname AS files_dirname, files.is_dir AS files_is_dir, files.class_ AS files_class_, entities_1.name AS entities_1_name, entities_1.mandatory AS entities_1_mandatory, entities_1.pattern AS entities_1_pattern, entities_1.directory AS entities_1_directory, entities_1.is_metadata AS entities_1_is_metadata, entities_1._dtype AS entities_1__dtype, tags_1.file_path AS tags_1_file_path, tags_1.entity_name AS tags_1_entity_name, tags_1._value AS tags_1__value, tags_1._dtype AS tags_1__dtype 
FROM files JOIN tags ON files.path = tags.file_path LEFT OUTER JOIN tags AS tags_1 ON files.path = tags_1.file_path LEFT OUTER JOIN entities AS entities_1 ON entities_1.name = tags_1.entity_name 
WHERE files.is_dir = 0 AND (EXISTS (SELECT 1 
FROM tags 
WHERE files.path = tags.file_path AND tags.entity_name = ? AND tags._value = ?)) AND (EXISTS (SELECT 1 
FROM tags 
WHERE files.path = tags.file_path AND tags.entity_name = ? AND tags._value = ?)) AND (EXISTS (SELECT 1 
FROM tags 
WHERE files.path = tags.file_path AND tags.entity_name = ? AND tags._value = ?))
```

This is pretty opaque. It's hard to understand why this would take so long unless the regressors file is being read, but I can't find any code that would indicate that is being done.
Out of curiosity, I tried @effigies example, and it also took forever for me. 

I don't think any file reading is going on here, and maybe there is some optimization o that query that needs to be done. The fact that the last three `FROM... WHERE` clauses are seemingly repeated is suspicious...
The repetition isn't suspicious—there are three separate constraints that need to be merged, so you're going to see something like that no matter what. But that constructed query is probably just extremely inefficient. I don't have good SQL intuitions, but I'm sure there are ways to dramatically improve this. For one thing, the `Entity` model is being implicitly joined even though it doesn't seem necessary for anything (this may reflect forced eager loading at model level, which is useful elsewhere but should perhaps be disabled in `_build_file_query()`).

It would be good if someone with more SQL experience were willing to jump in here—otherwise it will likely require a bunch of reading and experimentation.

EDIT: to clarify, I suspect the increase in query time seen for the original problem has more to do with the addition of another constraint (`suffix`) than with the tsv extension.
Maybe I'm reading it wrong, but aren't the first two WHERE clauses the same? And the last one is a subset of the first two? 

I don't have a good intuition as to if that's the problem either though.
The `?` are placeholders for tuples of field names and values, so those get filled in. I think they're `('desc', 'suffix', 'extension')` in this case.
I wonder if the WHERE conditions can be put as a single line with `OR` separators, and if that would be faster. It may be having to search/filter three separate times.

https://stackoverflow.com/questions/2162621/why-do-multiple-where-conditions-slow-query-rather-than-speed-it-up

Maybe we can check the execution plan on such a query to get an idea of what the problem is. 
Not possible—at least not while maintaining this general query structure. The operation here is over rows of `Tag`, and there are different rows for the `desc`, `suffix`, etc. of each `BIDSFile`. So what it's doing is identifying all the `BIDSFile` rows for which there's a least one (joined) `Tag` row that meets each of the conditions individually. You can't `OR` this, because the file has to meet *all* the constraints to be returned.

Ultimately we might need to re-normalize the DB or use a bit field (which probably would only work for queries with keys from the closed vocabulary, and not arbitrary metadata keys) to do this efficiently. We could also revert to doing a lot of the logic in native Python, but that kind of defeats the point of having a DB. But I'd prefer to experiment with different SQLAlchemy constructions first.
I also note that we don't currently have indexes (!). That might be a sensible place to start...
Ah, that certainly seems like low hanging fruit. I'm sure in theory this query could be fast, so I definitely don't want to drop SQLAlchemy, that would be quite the admission of defeat.
I'm happy to report that I have no idea how indexes work, and have made no progress on improving performance.
I can semi-reproduce this with [///labs/poldrack/ds003_fmriprep](https://datasets.datalad.org/?dir=/labs/poldrack/ds003_fmriprep):

```Python
import bids
layout = bids.BIDSLayout('ds003_fmriprep/sourcedata', derivatives='ds003_fmriprep')
layout.get(desc='confounds', suffix='regressors', extension='tsv', return_type='file')
```

It consumes a fair bit of time and memory, although it does complete. A little poking around reveals the line that takes all the time to be:

https://github.com/bids-standard/pybids/blob/74cdc330cc9a79663f8ecdb52c75e52a0d3f07bb/bids/layout/layout.py#L831

Adding a debug break, `print(str(query))` shows:

```SQL
SELECT files.path AS files_path, files.filename AS files_filename, files.dirname AS files_dirname, files.is_dir AS files_is_dir, files.class_ AS files_class_, entities_1.name AS entities_1_name, entities_1.mandatory AS entities_1_mandatory, entities_1.pattern AS entities_1_pattern, entities_1.directory AS entities_1_directory, entities_1.is_metadata AS entities_1_is_metadata, entities_1._dtype AS entities_1__dtype, tags_1.file_path AS tags_1_file_path, tags_1.entity_name AS tags_1_entity_name, tags_1._value AS tags_1__value, tags_1._dtype AS tags_1__dtype 
FROM files JOIN tags ON files.path = tags.file_path LEFT OUTER JOIN tags AS tags_1 ON files.path = tags_1.file_path LEFT OUTER JOIN entities AS entities_1 ON entities_1.name = tags_1.entity_name 
WHERE files.is_dir = 0 AND (EXISTS (SELECT 1 
FROM tags 
WHERE files.path = tags.file_path AND tags.entity_name = ? AND tags._value = ?)) AND (EXISTS (SELECT 1 
FROM tags 
WHERE files.path = tags.file_path AND tags.entity_name = ? AND tags._value = ?)) AND (EXISTS (SELECT 1 
FROM tags 
WHERE files.path = tags.file_path AND tags.entity_name = ? AND tags._value = ?))
```

This is pretty opaque. It's hard to understand why this would take so long unless the regressors file is being read, but I can't find any code that would indicate that is being done.
Out of curiosity, I tried @effigies example, and it also took forever for me. 

I don't think any file reading is going on here, and maybe there is some optimization o that query that needs to be done. The fact that the last three `FROM... WHERE` clauses are seemingly repeated is suspicious...
The repetition isn't suspicious—there are three separate constraints that need to be merged, so you're going to see something like that no matter what. But that constructed query is probably just extremely inefficient. I don't have good SQL intuitions, but I'm sure there are ways to dramatically improve this. For one thing, the `Entity` model is being implicitly joined even though it doesn't seem necessary for anything (this may reflect forced eager loading at model level, which is useful elsewhere but should perhaps be disabled in `_build_file_query()`).

It would be good if someone with more SQL experience were willing to jump in here—otherwise it will likely require a bunch of reading and experimentation.

EDIT: to clarify, I suspect the increase in query time seen for the original problem has more to do with the addition of another constraint (`suffix`) than with the tsv extension.
Maybe I'm reading it wrong, but aren't the first two WHERE clauses the same? And the last one is a subset of the first two? 

I don't have a good intuition as to if that's the problem either though.
The `?` are placeholders for tuples of field names and values, so those get filled in. I think they're `('desc', 'suffix', 'extension')` in this case.
I wonder if the WHERE conditions can be put as a single line with `OR` separators, and if that would be faster. It may be having to search/filter three separate times.

https://stackoverflow.com/questions/2162621/why-do-multiple-where-conditions-slow-query-rather-than-speed-it-up

Maybe we can check the execution plan on such a query to get an idea of what the problem is. 
Not possible—at least not while maintaining this general query structure. The operation here is over rows of `Tag`, and there are different rows for the `desc`, `suffix`, etc. of each `BIDSFile`. So what it's doing is identifying all the `BIDSFile` rows for which there's a least one (joined) `Tag` row that meets each of the conditions individually. You can't `OR` this, because the file has to meet *all* the constraints to be returned.

Ultimately we might need to re-normalize the DB or use a bit field (which probably would only work for queries with keys from the closed vocabulary, and not arbitrary metadata keys) to do this efficiently. We could also revert to doing a lot of the logic in native Python, but that kind of defeats the point of having a DB. But I'd prefer to experiment with different SQLAlchemy constructions first.
I also note that we don't currently have indexes (!). That might be a sensible place to start...
Ah, that certainly seems like low hanging fruit. I'm sure in theory this query could be fast, so I definitely don't want to drop SQLAlchemy, that would be quite the admission of defeat.
I'm happy to report that I have no idea how indexes work, and have made no progress on improving performance.

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -931,10 +931,6 @@ def get(self, return_type='object', target=None, scope='all',
         for l in layouts:
             query = l._build_file_query(filters=filters,
                                         regex_search=regex_search)
-            # Eager load associations, because mixing queries from different
-            # DB sessions causes objects to detach
-            query = query.options(joinedload(BIDSFile.tags)
-                                  .joinedload(Tag.entity))
             results.extend(query.all())
 
         # Convert to relative paths if needed


Test patch: