Repository: nipy/heudiconv

Issue:
ERROR: Embedding failed: 'odict_values' object does not support indexing & BIDS validator
Dear Heudiconv's experts,

I have run heudiconv docker image on FDG PET data with the following command line et execution message:
```
docker run --rm -it -v /Users/mattvan83/fMRIprep/COMAJ:/data:ro -v /Users/mattvan83/fMRIprep/BIDS:/output nipy/heudiconv:latest -d /data/{subject}/*/* -s s001 -f /data/IMAP_AGEWELL_bids.py -b -o /output
/src/heudiconv/heudiconv/dicoms.py:7: UserWarning: The DICOM readers are highly experimental, unstable, and only work for Siemens time-series at the moment
Please use with caution.  We would be grateful for your help in improving them
  from nibabel.nicom import csareader
INFO: Running heudiconv version 0.5.2-dev
INFO: Need to process 1 study sessions
INFO: PROCESSING STARTS: {'subject': 's001', 'outdir': '/output/', 'session': None}
INFO: Processing 47 dicoms
INFO: Analyzing 47 dicoms
INFO: Generated sequence info with 1 entries
INFO: Doing conversion using dcm2niix
INFO: Converting /output/sub-s001/pet/sub-s001_task-rest_acq-FDG_rec-AC_run-01_pet (47 DICOMs) -> /output/sub-s001/pet . Converter: dcm2niix . Output types: ('nii.gz',)
181230-10:03:23,191 nipype.workflow INFO:
	 [Node] Setting-up "convert" in "/tmp/dcm2niixadifah8v/convert".
INFO: [Node] Setting-up "convert" in "/tmp/dcm2niixadifah8v/convert".
181230-10:03:23,603 nipype.workflow INFO:
	 [Node] Running "convert" ("nipype.interfaces.dcm2nii.Dcm2niix"), a CommandLine Interface with command:
dcm2niix -b y -z y -x n -t n -m n -f pet -o . -s n -v n /tmp/dcm2niixadifah8v/convert
INFO: [Node] Running "convert" ("nipype.interfaces.dcm2nii.Dcm2niix"), a CommandLine Interface with command:
dcm2niix -b y -z y -x n -t n -m n -f pet -o . -s n -v n /tmp/dcm2niixadifah8v/convert
181230-10:03:24,486 nipype.interface INFO:
	 stdout 2018-12-30T10:03:24.485856:Chris Rorden's dcm2niiX version v1.0.20180622 GCC6.3.0 (64-bit Linux)
INFO: stdout 2018-12-30T10:03:24.485856:Chris Rorden's dcm2niiX version v1.0.20180622 GCC6.3.0 (64-bit Linux)
181230-10:03:24,486 nipype.interface INFO:
	 stdout 2018-12-30T10:03:24.485856:Found 47 DICOM file(s)
INFO: stdout 2018-12-30T10:03:24.485856:Found 47 DICOM file(s)
181230-10:03:24,486 nipype.interface INFO:
	 stdout 2018-12-30T10:03:24.485856:Convert 47 DICOM as ./pet (256x256x47x1)
INFO: stdout 2018-12-30T10:03:24.485856:Convert 47 DICOM as ./pet (256x256x47x1)
181230-10:03:24,832 nipype.interface INFO:
	 stdout 2018-12-30T10:03:24.832191:compress: "/usr/bin/pigz" -n -f -6 "./pet.nii"
INFO: stdout 2018-12-30T10:03:24.832191:compress: "/usr/bin/pigz" -n -f -6 "./pet.nii"
181230-10:03:24,832 nipype.interface INFO:
	 stdout 2018-12-30T10:03:24.832191:Conversion required 1.112542 seconds (0.126743 for core code).
INFO: stdout 2018-12-30T10:03:24.832191:Conversion required 1.112542 seconds (0.126743 for core code).
181230-10:03:24,898 nipype.workflow INFO:
	 [Node] Finished "convert".
INFO: [Node] Finished "convert".
181230-10:03:25,70 nipype.workflow INFO:
	 [Node] Setting-up "embedder" in "/tmp/embedmetaf18vnj85/embedder".
INFO: [Node] Setting-up "embedder" in "/tmp/embedmetaf18vnj85/embedder".
181230-10:03:25,227 nipype.workflow INFO:
	 [Node] Running "embedder" ("nipype.interfaces.utility.wrappers.Function")
INFO: [Node] Running "embedder" ("nipype.interfaces.utility.wrappers.Function")
181230-10:03:26,887 nipype.workflow WARNING:
	 [Node] Error on "embedder" (/tmp/embedmetaf18vnj85/embedder)
WARNING: [Node] Error on "embedder" (/tmp/embedmetaf18vnj85/embedder)
ERROR: Embedding failed: 'odict_values' object does not support indexing
INFO: Post-treating /output/sub-s001/pet/sub-s001_task-rest_acq-FDG_rec-AC_run-01_pet.json file
INFO: Populating template files under /output/
INFO: PROCESSING DONE: {'subject': 's001', 'outdir': '/output/', 'session': None}
```

It seems that there is an error of embedder following the message above:
```
181230-10:03:26,887 nipype.workflow WARNING:
	 [Node] Error on "embedder" (/tmp/embedmetaf18vnj85/embedder)
WARNING: [Node] Error on "embedder" (/tmp/embedmetaf18vnj85/embedder)
ERROR: Embedding failed: 'odict_values' object does not support indexing
```
What is the impact of this error ?

Moreover, it seems that my BIDS PET data are in wrong format according BIDS validator:
```
Error 1: [Code 61] QUICK_VALIDATION_FAILED
Quick validation failed - the general folder structure does not resemble a BIDS dataset. Have you chosen the right folder (with "sub-*/" subfolders)? Check for structural/naming issues and presence of at least one subject.
```
I have used the following heuristics:
```
import os

def create_key(template, outtype=('nii.gz'), annotation_classes=None):
    if template is None or not template:
        raise ValueError('Template must be a valid format string')
    return (template, outtype, annotation_classes)


def infotodict(seqinfo):
    """Heuristic evaluator for determining which runs belong where

    allowed template fields - follow python string module:

    item: index within category
    subject: participant id
    seqitem: run number during scanning
    subindex: sub index within group
    """
    pet = create_key('sub-{subject}/pet/sub-{subject}_task-rest_acq-{acq}_rec-{rec}_run-{item:02d}_pet')

    info = {pet:[]}

    for idx, s in enumerate(seqinfo):
        if (s.dim3 == 47) and (s.TR == -1) and ('CERVEAU 3D SALENGRO' in s.protocol_name):
            rectype = s.series_description.split('_')[-1]
            info[pet].append({'item': s.series_id, 'acq': 'FDG', 'rec': rectype}) # append if multiple series meet criteria


    # You can even put checks in place for your protocol
    msg = []
    if len(info[pet]) != 1: msg.append('Missing correct number of pet runs')
    if msg:
        raise ValueError('\n'.join(msg))
    return info

```

Thanks in advance for any help.

Best,
Matt

Comments:

Patch:
diff --git a/heudiconv/dicoms.py b/heudiconv/dicoms.py
index b94013f..13a200b 100644
--- a/heudiconv/dicoms.py
+++ b/heudiconv/dicoms.py
@@ -353,7 +353,7 @@ def compress_dicoms(dicom_list, out_prefix, tempdirs, overwrite):
     return outtar
 
 
-def embed_nifti(dcmfiles, niftifile, infofile, bids_info, force, min_meta):
+def embed_nifti(dcmfiles, niftifile, infofile, bids_info, min_meta):
     """
 
     If `niftifile` doesn't exist, it gets created out of the `dcmfiles` stack,
@@ -370,7 +370,6 @@ def embed_nifti(dcmfiles, niftifile, infofile, bids_info, force, min_meta):
     niftifile
     infofile
     bids_info
-    force
     min_meta
 
     Returns
@@ -387,10 +386,11 @@ def embed_nifti(dcmfiles, niftifile, infofile, bids_info, force, min_meta):
 
     if not min_meta:
         import dcmstack as ds
-        stack = ds.parse_and_stack(dcmfiles, force=force).values()
+        stack = ds.parse_and_stack(dcmfiles, force=True).values()
         if len(stack) > 1:
             raise ValueError('Found multiple series')
-        stack = stack[0]
+        # may be odict now - iter to be safe
+        stack = next(iter(stack))
 
         #Create the nifti image using the data array
         if not op.exists(niftifile):
@@ -458,7 +458,7 @@ def embed_metadata_from_dicoms(bids, item_dicoms, outname, outname_bids,
     item_dicoms = list(map(op.abspath, item_dicoms))
 
     embedfunc = Node(Function(input_names=['dcmfiles', 'niftifile', 'infofile',
-                                           'bids_info', 'force', 'min_meta'],
+                                           'bids_info', 'min_meta'],
                               output_names=['outfile', 'meta'],
                               function=embed_nifti),
                      name='embedder')
@@ -466,13 +466,10 @@ def embed_metadata_from_dicoms(bids, item_dicoms, outname, outname_bids,
     embedfunc.inputs.niftifile = op.abspath(outname)
     embedfunc.inputs.infofile = op.abspath(scaninfo)
     embedfunc.inputs.min_meta = min_meta
-    if bids:
-        embedfunc.inputs.bids_info = load_json(op.abspath(outname_bids))
-    else:
-        embedfunc.inputs.bids_info = None
-    embedfunc.inputs.force = True
+    embedfunc.inputs.bids_info = load_json(op.abspath(outname_bids)) if bids else None
     embedfunc.base_dir = tmpdir
     cwd = os.getcwd()
+
     lgr.debug("Embedding into %s based on dicoms[0]=%s for nifti %s",
               scaninfo, item_dicoms[0], outname)
     try:


Test patch:
diff --git a/heudiconv/tests/test_dicoms.py b/heudiconv/tests/test_dicoms.py
index 536d9d4..de8e9a1 100644
--- a/heudiconv/tests/test_dicoms.py
+++ b/heudiconv/tests/test_dicoms.py
@@ -1,10 +1,11 @@
 import os.path as op
+import json
 
 import pytest
 
 from heudiconv.external.pydicom import dcm
 from heudiconv.cli.run import main as runner
-from heudiconv.dicoms import parse_private_csa_header
+from heudiconv.dicoms import parse_private_csa_header, embed_nifti
 from .utils import TESTS_DATA_PATH
 
 # Public: Private DICOM tags
@@ -23,3 +24,37 @@ def test_private_csa_header(tmpdir):
         assert parse_private_csa_header(dcm_data, pub, priv) != ''
         # and quickly run heudiconv with no conversion
         runner(['--files', dcm_file, '-c' 'none', '-f', 'reproin'])
+
+
+def test_nifti_embed(tmpdir):
+    """Test dcmstack's additional fields"""
+    tmpdir.chdir()
+    # set up testing files
+    dcmfiles = [op.join(TESTS_DATA_PATH, 'axasc35.dcm')]
+    infofile = 'infofile.json'
+
+    # 1) nifti does not exist
+    out = embed_nifti(dcmfiles, 'nifti.nii', 'infofile.json', None, False)
+    # string -> json
+    out = json.loads(out)
+    # should have created nifti file
+    assert op.exists('nifti.nii')
+
+    # 2) nifti exists
+    nifti, info = embed_nifti(dcmfiles, 'nifti.nii', 'infofile.json', None, False)
+    assert op.exists(nifti)
+    assert op.exists(info)
+    with open(info) as fp:
+        out2 = json.load(fp)
+
+    assert out == out2
+
+    # 3) with existing metadata
+    bids = {"existing": "data"}
+    nifti, info = embed_nifti(dcmfiles, 'nifti.nii', 'infofile.json', bids, False)
+    with open(info) as fp:
+        out3 = json.load(fp)
+
+    assert out3["existing"]
+    del out3["existing"]
+    assert out3 == out2 == out