Repository: nilearn/nilearn

Issue:
Unexpected image resampling when nifti doesn't have an sform
<!--Provide a brief description of the bug.-->

The bug pertains the transformation of niftis that do not have an sform (all 0s). This happens when, for example, using antsApplyTransforms ([from ANTs versions older than Feb. 2021](https://github.com/ANTsX/ANTs/wiki/How-does-ANTs-handle-qform-and-sform-in-NIFTI-1-images%3F#itk-nifti-io-for-ants-230-4)) to register images, where output files do not contain an sform. 
Using  `masker.fit_transform(img)` on such a Nifti will trigger a silent resampling if `mask_img` and `img` sforms differ, even if the affines from `mask_img` and `img` are identical (according to mask_img.affine and  img.affine).

I take this is not a nilearn bug per se, since it is ultimately responsability of the user to ensure their niftis make sense. However, given the vast array of tools people use to generate niftis, ANTs being a common example, I take the issue could lead to many silent bugs in people's analysis.

So what is there to do since nothing is really broken?

Should a resampling always be verbose? Currently int->float casting is verbose, and that is arguably a more benign issue than an unexpected image resampling.
Also, all 0s sforms are always suspect. Maybe a check that qform and sform are the same would be an extra safety check? If not fall back to qform or warn user with a link to documentation and proposed workarounds?

Current workarounds:

outside of python: `mri_convert data.nii.gz data.nii.gz`
inside python: `fixed = image.new_img_like(mask_img, nib.load(img).get_fdata(), mask_img.affine, copy_header=True)`


<!--Please fill in the following information, to the best of your ability.-->
Nilearn version:
latest from git

### Expected behavior
not sure

### Actual behavior
unexpected resample

### Steps and code to reproduce bug

```python
from nilearn import datasets, image, input_data

yeo = datasets.fetch_atlas_yeo_2011()
mask_img = image.binarize_img(yeo["thin_7"])
img = image.load_img(yeo["thin_17"])

masker = input_data.NiftiMasker(mask_img=mask_img)
correct = masker.fit_transform(img)

# if mask_img is missing sform -> silent resampling
mask_img.set_sform(None)
masker = input_data.NiftiMasker(mask_img=mask_img)
wrong = masker.fit_transform(img)

# if img is missing sform -> silent resampling
mask_img = image.binarize_img(yeo["thin_7"])
masker = input_data.NiftiMasker(mask_img=mask_img)
img.set_sform(None)
also_wrong = masker.fit_transform(img)

```



Discussion:
Thx for reporting.
In your example, `wrong` is completely off (correlation=0.036 with correct),  while `also_wrong` is simply approximate (correlation=0.999), due to useless interpolation.
The case of `wrong` is thus really fishy.
Indeed. The useless interpolation breaks integers though, which is bad for ROIs. But that's because the interp is not nearest neighbors.
Sorry, just following up on this. I think the issue is more serious than it appears. It just bit me again today.
Any image touched by any of ANTs tools compiled before Feb 2021 will not work properly when loaded either as a mask or dataset using NiftiMasker. It will be silently resampled and lead to subtly wrong results.

In your view, what would be the most appropriate place to fix this? 
And should resampling be turned off by default? The documentation doesn't mention that this is happening behind the scenes.



* I agree that  documentation should be explicit about resampling (!)
* I also think that it should be the verbose, but could be more verbose to let the user know.
* My main impression is that the (arguably  Nilearn-independent) issue could be dealt with explicitly, showing the fix you propose somewhere in the doc.
In summary I would handle this mostly as a documentation thing. 
What do others think ?
I think the issue is that 
1. niftis without sform are [valid according to the NIFTI spec ](https://nifti.nimh.nih.gov/nifti-1/documentation/nifti1fields/nifti1fields_pages/qsform.html)
2. many old software use niftis without sform, the widely used [ANTs did so until feb 2021](https://github.com/ANTsX/ANTs/wiki/How-does-ANTs-handle-qform-and-sform-in-NIFTI-1-images%3F#itk-nifti-io-for-ants-230-4)
3. nilearn maskers cannot handle niftis without sform
4. they produce a wrong result
5. the wrong result is produced **silently**
6. the feature that causes issues is not documented and cannot be deactivated

So my view is either we raise an error if confronted with sform_code = 0 and tell user what to do, and maybe point to this thread or docs, or we stop doing the silent thing that users did not ask for and that cannot be deactivated.

Are you suggesting that the missing sform could be imputed from the remainder of the header information ?
If yes, one should do so (and send a warning).
mri_convert imputes it, up to some numerical precision (seems to be copying qform into sform):

<img width="413" alt="image" src="https://user-images.githubusercontent.com/7881842/159541582-fb537f15-66d3-4ece-8924-1fe2696e686d.png">

That's a good idea. One could do so, provided the numerical precision issues would not lead to unwanted resampling/interp.
Hello ! 
Here is feedback following a quick investigation with @emdupre and discussion with @effigies.

We think that you shouldn't systematically impute the sform using the qform. If the sform is not there and the qform code is unknown, it is considered problematic, since it means the image and the data in its header are not coherent. Using the qform instead would only further obscure this.

For now, we will make changes to raise an warning if the sform is not found. Providing a "correct" image is considered a responsibility of the user.
To clarify my point, I would say you shouldn't impute the qform from the sform (or vice versa) when the corresponding codes are 0.

<details>
<summary>Following the NIfTI standard (click to expand)</summary>

[Link](https://github.com/NIFTI-Imaging/nifti_clib/blob/fb3bb5f47d93d49628f5a6d7d33afc016f6c05f8/niftilib/nifti1.h#L1033-L1131)

>```
>    METHOD 1 (the "old" way, used only when qform_code = 0):
>    -------------------------------------------------------
>    The coordinate mapping from (i,j,k) to (x,y,z) is the ANALYZE
>    7.5 way.  This is a simple scaling relationship:
> 
>      x = pixdim[1] * i
>      y = pixdim[2] * j
>      z = pixdim[3] * k
> 
>    No particular spatial orientation is attached to these (x,y,z)
>    coordinates.  (NIFTI-1 does not have the ANALYZE 7.5 orient field,
>    which is not general and is often not set properly.)  This method
>    is not recommended, and is present mainly for compatibility with
>    ANALYZE 7.5 files.
> 
>    METHOD 2 (used when qform_code > 0, which should be the "normal" case):
>    ---------------------------------------------------------------------
>    The (x,y,z) coordinates are given by the pixdim[] scales, a rotation
>    matrix, and a shift.  This method is intended to represent
>    "scanner-anatomical" coordinates, which are often embedded in the
>    image header (e.g., DICOM fields (0020,0032), (0020,0037), (0028,0030),
>    and (0018,0050)), and represent the nominal orientation and location of
>    the data.  This method can also be used to represent "aligned"
>    coordinates, which would typically result from some post-acquisition
>    alignment of the volume to a standard orientation (e.g., the same
>    subject on another day, or a rigid rotation to true anatomical
>    orientation from the tilted position of the subject in the scanner).
>    The formula for (x,y,z) in terms of header parameters and (i,j,k) is:
> 
>      [ x ]   [ R11 R12 R13 ] [        pixdim[1] * i ]   [ qoffset_x ]
>      [ y ] = [ R21 R22 R23 ] [        pixdim[2] * j ] + [ qoffset_y ]
>      [ z ]   [ R31 R32 R33 ] [ qfac * pixdim[3] * k ]   [ qoffset_z ]
> 
>    The qoffset_* shifts are in the NIFTI-1 header.  Note that the center
>    of the (i,j,k)=(0,0,0) voxel (first value in the dataset array) is
>    just (x,y,z)=(qoffset_x,qoffset_y,qoffset_z).
> 
>    The rotation matrix R is calculated from the quatern_* parameters.
>    This calculation is described below.
> 
>    The scaling factor qfac is either 1 or -1.  The rotation matrix R
>    defined by the quaternion parameters is "proper" (has determinant 1).
>    This may not fit the needs of the data; for example, if the image
>    grid is
>      i increases from Left-to-Right
>      j increases from Anterior-to-Posterior
>      k increases from Inferior-to-Superior
>    Then (i,j,k) is a left-handed triple.  In this example, if qfac=1,
>    the R matrix would have to be
> 
>      [  1   0   0 ]
>      [  0  -1   0 ]  which is "improper" (determinant = -1).
>      [  0   0   1 ]
> 
>    If we set qfac=-1, then the R matrix would be
> 
>      [  1   0   0 ]
>      [  0  -1   0 ]  which is proper.
>      [  0   0  -1 ]
> 
>    This R matrix is represented by quaternion [a,b,c,d] = [0,1,0,0]
>    (which encodes a 180 degree rotation about the x-axis).
> 
>    METHOD 3 (used when sform_code > 0):
>    -----------------------------------
>    The (x,y,z) coordinates are given by a general affine transformation
>    of the (i,j,k) indexes:
> 
>      x = srow_x[0] * i + srow_x[1] * j + srow_x[2] * k + srow_x[3]
>      y = srow_y[0] * i + srow_y[1] * j + srow_y[2] * k + srow_y[3]
>      z = srow_z[0] * i + srow_z[1] * j + srow_z[2] * k + srow_z[3]
> 
>    The srow_* vectors are in the NIFTI_1 header.  Note that no use is
>    made of pixdim[] in this method.
> 
>    WHY 3 METHODS?
>    --------------
>    Method 1 is provided only for backwards compatibility.  The intention
>    is that Method 2 (qform_code > 0) represents the nominal voxel locations
>    as reported by the scanner, or as rotated to some fiducial orientation and
>    location.  Method 3, if present (sform_code > 0), is to be used to give
>    the location of the voxels in some standard space.  The sform_code
>    indicates which standard space is present.  Both methods 2 and 3 can be
>    present, and be useful in different contexts (method 2 for displaying the
>    data on its original grid; method 3 for displaying it on a standard grid).
> 
>    In this scheme, a dataset would originally be set up so that the
>    Method 2 coordinates represent what the scanner reported.  Later,
>    a registration to some standard space can be computed and inserted
>    in the header.  Image display software can use either transform,
>    depending on its purposes and needs.
> 
>    In Method 2, the origin of coordinates would generally be whatever
>    the scanner origin is; for example, in MRI, (0,0,0) is the center
>    of the gradient coil.
> 
>    In Method 3, the origin of coordinates would depend on the value
>    of sform_code; for example, for the Talairach coordinate system,
>    (0,0,0) corresponds to the Anterior Commissure.
> ```

</details>

If `sform_code` is non-zero, you are done. You *may* want to copy this into the `qform` so that ANTs will treat the file the same as conformant implementations. If the `sform_code` is zero but the `qform_code` is non-zero, then again you *may* want to copy this into `sform` to avoid mismatch. If both are zero, then you are stuck guessing whether the image is an Analyze image, produced by ANTs, or just corrupted. There is *no* way for a tool to safely use either the qform or the sform and you should not try. You can suggest that the user inspect the image and fix the qform/sform and set appropriate codes.

My suggestion, IIRC, was:

```Python
sform, sform_code = img.get_sform(coded=True)
qform, qform_code = img.get_qform(coded=True)
if not sform_code and not qform_code:
    warn("qform and sform codes are zero; this file may be corrupted. Using fallback LAS affine.")
```

Resolving patch:
diff --git a/nilearn/image/resampling.py b/nilearn/image/resampling.py
--- a/nilearn/image/resampling.py
+++ b/nilearn/image/resampling.py
@@ -445,6 +445,14 @@ def resample_img(img, target_affine=None, target_shape=None,
     shape = img.shape
     affine = img.affine
 
+    # If later on we want to impute sform using qform add this condition
+    # see : https://github.com/nilearn/nilearn/issues/3168#issuecomment-1159447771 # noqa:E501
+    sform, sform_code = img.get_sform(coded=True)
+    if not sform_code:
+        warnings.warn("The provided image has no sform in its header. "
+                      "Please check the provided file. "
+                      "Results may not be as expected.")
+
     # noop cases
     if target_affine is None and target_shape is None:
         if copy and not input_img_is_string:


Test patch:
diff --git a/nilearn/image/tests/test_resampling.py b/nilearn/image/tests/test_resampling.py
--- a/nilearn/image/tests/test_resampling.py
+++ b/nilearn/image/tests/test_resampling.py
@@ -269,7 +269,11 @@ def test_resampling_error_checks():
         rot_img = resample_img(img_binary,
                                target_affine=rot,
                                interpolation='linear')
-
+    img_no_sform = Nifti1Image(data, affine)
+    img_no_sform.set_sform(None)
+    with pytest.warns(Warning, match="The provided image has no sform"):
+        resample_img(img_no_sform,
+                     target_affine=affine)
 
     # Noop
     target_shape = shape[:3]