Repository: nipreps/fmriprep

Issue:
`collect_derivatives` uses incorrect "from" entity while searching for pre-calculated boldref -> fmap coregistrations (orig vs boldref)
### What happened?

fMRIPrep generates rigid xfms between the boldref and the fieldmap that have names like `sub-{subject}_ses-{session}_task-{task}_run-{run}_from-boldref_to-{fieldmap_id}_mode-image_xfm.txt`. 

I am trying to supply precalculated xfms with the `--derivatives` flag, but it doesn't seem like they are being picked up. 

### What command did you use?

```shell
docker run \
  --init \
  --rm \
  -it \
  -v $PWD:$PWD \
  nipreps/fmriprep:24.1.0 \
  --dummy-scans 15 \
  -w $PWD/work2 \
  --fs-license-file $PWD/license \
  --derivatives boldref2fmap=$PWD/sourcedata/boldref2fmap  \
  --notrack  \
  $PWD/rawdata $PWD/derivatives2/fmriprep participant


WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2024-09-27 22:38:47,401 [ WARNING] WARNING: SOCKS support in urllib3 requires the installation of optional dependencies: specifically, PySocks.  For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#socks-proxies
bids-validator@1.14.10
(node:10) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
	1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
		./README

	Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.

        Summary:                  Available Tasks:                     Available Modalities: 
        24 Files, 771.34MB        cuff                                 MRI                   
        1 - Subject               rest                                                       
        1 - Session               TODO: full task name for cuff                              
                                  TODO: full task name for rest                              


	If you have any questions, please post on https://neurostars.org/tags/bids.
2024-09-27 22:38:57,903 [ WARNING] WARNING: SOCKS support in urllib3 requires the installation of optional dependencies: specifically, PySocks.  For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#socks-proxies
2024-09-27 22:38:58,126 [ WARNING] WARNING: Niworkflows will be deprecating reporting in favor of a standalone library "nireports".
2024-09-27 22:38:58,251 [ WARNING] WARNING: Niworkflows will be deprecating visualizations in favor of a standalone library "nireports".
240927-22:38:59,171 nipype.workflow IMPORTANT:
	 Running fMRIPrep version 24.1.0
[...]
```


### What version of fMRIPrep are you running?

24.1.0

### How are you running fMRIPrep?

Docker

### Is your data BIDS valid?

Yes

### Are you reusing any previously computed results?

FreeSurfer

### Please copy and paste any relevant log output.

```shell
240927-22:38:59,200 nipype.workflow IMPORTANT:
	 Building fMRIPrep's workflow:
           * BIDS dataset path: /Users/psadil/Desktop/nstravel2/rawdata.
           * Participant list: ['travel2'].
           * Run identifier: 20240927-223847_deb204bf-a439-41a5-a8ed-4dda4f38d9b8.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Searching for derivatives: [PosixPath('/Users/psadil/Desktop/nstravel2/sourcedata/boldref2fmap')].
           * Pre-run FreeSurfer's SUBJECTS_DIR: /Users/psadil/Desktop/nstravel2/derivatives2/fmriprep/sourcedata/freesurfer.
2024-09-27 22:38:59,200 [IMPORTANT] Building fMRIPrep's workflow:
           * BIDS dataset path: /Users/psadil/Desktop/nstravel2/rawdata.
           * Participant list: ['travel2'].
           * Run identifier: 20240927-223847_deb204bf-a439-41a5-a8ed-4dda4f38d9b8.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Searching for derivatives: [PosixPath('/Users/psadil/Desktop/nstravel2/sourcedata/boldref2fmap')].
           * Pre-run FreeSurfer's SUBJECTS_DIR: /Users/psadil/Desktop/nstravel2/derivatives2/fmriprep/sourcedata/freesurfer.
```


### Additional information / screenshots

I guess the issue is that the searched for pattern uses [`"from": "orig",`](https://github.com/nipreps/fmriprep/blob/73189de5ee576ffc73ab432b7419304d44ce5776/fmriprep/data/io_spec.json#L42C8-L47)?

As in, below, I would have expected "boldref" to work but not "orig" 

```python
from pathlib import Path
from fmriprep.utils import bids
import tempfile
import json

entities = {"subject": "0", "task": "rest"}

for from_id in ["boldref", "orig"]:
    with tempfile.TemporaryDirectory() as tmpd_:
        tmpd = Path(tmpd_) / "boldref2fmap"
        funcd = tmpd / f"sub-{entities['subject']}" / "func"
        funcd.mkdir(parents=True)
        stem = f"sub-{entities['subject']}_task-{entities['task']}_from-{from_id}_to-auto00000_mode-image_xfm"
        (funcd / f"{stem}").with_suffix(".txt").write_text("placeholder")
        (funcd / f"{stem}").with_suffix(".json").write_text(
            json.dumps(entities)
        )
        derivs = bids.collect_derivatives(
            derivatives_dir=tmpd,
            entities=entities,
            fieldmap_id=None,
        )
        print(f"looking for from-{from_id}")
        print(derivs)

```

```shell
looking for from-boldref
defaultdict(<class 'list'>, {})
looking for from-orig
defaultdict(<class 'list'>, {'boldref2fmap': '/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/tmp58nmly9z/boldref2fmap/sub-0/func/sub-0_task-rest_from-orig_to-auto00000_mode-image_xfm.txt'})

```


FWIW, I'm trying to supply these xfms as a way to troubleshoot the issue described here: https://neurostars.org/t/poor-fieldmap-functional-registration-in-fmriprep-highly-variable-step/30397, which I mention shamelessly to get more eyes on it ;)

Discussion:

Patch:
diff --git a/fmriprep/data/io_spec.json b/fmriprep/data/io_spec.json
index 5d90f184..364cff7e 100644
--- a/fmriprep/data/io_spec.json
+++ b/fmriprep/data/io_spec.json
@@ -33,7 +33,7 @@
       },
       "boldref2anat": {
         "datatype": "func",
-        "from": "orig",
+        "from": "boldref",
         "to": "anat",
         "mode": "image",
         "suffix": "xfm",
@@ -41,7 +41,7 @@
       },
       "boldref2fmap": {
         "datatype": "func",
-        "from": "orig",
+        "from": "boldref",
         "mode": "image",
         "suffix": "xfm",
         "extension": ".txt"
diff --git a/fmriprep/utils/bids.py b/fmriprep/utils/bids.py
index 0bdc03ff..b2927547 100644
--- a/fmriprep/utils/bids.py
+++ b/fmriprep/utils/bids.py
@@ -68,14 +68,21 @@ def collect_derivatives(
             continue
         derivs_cache[f'{k}_boldref'] = item[0] if len(item) == 1 else item
 
+    transforms_cache = {}
     for xfm, q in spec['transforms'].items():
-        query = {**q, **entities}
-        if xfm == 'boldref2fmap':
-            query['to'] = fieldmap_id
-        item = layout.get(return_type='filename', **q)
+        # Transform extension will often not match provided entities
+        #   (e.g., ".nii.gz" vs ".txt").
+        # And transform suffixes will be "xfm",
+        #   whereas relevant src file will be "bold".
+        query = {**entities, **q}
+        if xfm == 'boldref2fmap' and fieldmap_id:
+            # fieldmaps have ids like auto_00000
+            query['to'] = fieldmap_id.replace('_', '')
+        item = layout.get(return_type='filename', **query)
         if not item:
             continue
-        derivs_cache[xfm] = item[0] if len(item) == 1 else item
+        transforms_cache[xfm] = item[0] if len(item) == 1 else item
+    derivs_cache['transforms'] = transforms_cache
     return derivs_cache
 
 


Test patch:
diff --git a/fmriprep/utils/tests/test_derivative_cache.py b/fmriprep/utils/tests/test_derivative_cache.py
new file mode 100644
index 00000000..1a2c59ac
--- /dev/null
+++ b/fmriprep/utils/tests/test_derivative_cache.py
@@ -0,0 +1,36 @@
+from pathlib import Path
+
+import pytest
+
+from fmriprep.utils import bids
+
+
+@pytest.mark.parametrize('xfm', ['boldref2fmap', 'boldref2anat', 'hmc'])
+def test_transforms_found_as_str(tmp_path: Path, xfm: str):
+    subject = '0'
+    task = 'rest'
+    fromto = {
+        'hmc': 'from-orig_to-boldref',
+        'boldref2fmap': 'from-boldref_to-auto00000',
+        'boldref2anat': 'from-boldref_to-anat',
+    }[xfm]
+
+    to_find = tmp_path.joinpath(
+        f'sub-{subject}', 'func', f'sub-{subject}_task-{task}_{fromto}_mode-image_xfm.txt'
+    )
+    to_find.parent.mkdir(parents=True)
+    to_find.touch()
+
+    entities = {
+        'subject': subject,
+        'task': task,
+        'suffix': 'bold',
+        'extension': '.nii.gz',
+    }
+
+    derivs = bids.collect_derivatives(
+        derivatives_dir=tmp_path,
+        entities=entities,
+        fieldmap_id='auto_00000',
+    )
+    assert derivs == {'transforms': {xfm: str(to_find)}}