Repository: DKISTDC/dkist

Issue:
FileManager doesn't update for certain slices
### Description

I have a ViSP dataset (ALMMM) which has five total pixel dimensions (and each FITS file has shape `(1, 915, 2555)`):
```
This VISP Dataset ALMMM consists of 8800 frames.
Files are stored in /home/svankooten/globus/pid_2_86/ALMMM

This Dataset has 5 pixel and 5 world dimensions.

The data are represented by a <class 'dask.array.core.Array'> object:
dask.array<reshape, shape=(4, 2, 1100, 915, 2555), dtype=float32, chunksize=(1, 1, 1, 915, 2555), chunktype=numpy.ndarray>

Array Dim  Axis Name                 Data size  Bounds
        0  polarization state                4  None
        1  raster map repeat number          2  None
        2  raster scan step number        1100  None
        3  dispersion axis                 915  None
        4  spatial along slit             2555  None

World Dim  Axis Name                  Physical Type                   Units
        4  stokes                     phys.polarization.stokes        unknown
        3  time                       time                            s
        2  helioprojective latitude   custom:pos.helioprojective.lat  arcsec
        1  wavelength                 em.wl                           nm
        0  helioprojective longitude  custom:pos.helioprojective.lon  arcsec
```

When I index the dataset on the first three dimensions, I get the dask graph down to one file:
```
In [8]: ds.data.npartitions
Out[8]: 8800

In [9]: ds[0].data.npartitions
Out[9]: 2200

In [10]: ds[0, 0].data.npartitions
Out[10]: 1100

In [11]: ds[0, 0, 0].data.npartitions
Out[11]: 1
```
The FileManager follows this for the first two slices, but not for the third:
```
In [13]: ds.files
Out[13]: FileManager containing 8800 files with each array having shape (1, 915, 2555)

In [14]: ds[0].files
Out[14]: FileManager containing 2200 files with each array having shape (1, 915, 2555)

In [15]: ds[0, 0].files
Out[15]: FileManager containing 1100 files with each array having shape (1, 915, 2555)

In [16]: ds[0, 0, 0].files
Out[16]: FileManager containing 1100 files with each array having shape (1, 915, 2555)
```

This happens also if the third axis is the only one sliced:
```
In [17]: ds[:, :, 0].data.npartitions
Out[17]: 8

In [18]: ds[:, :, 0].files
Out[18]: FileManager containing 8800 files with each array having shape (1, 915, 2555)
```
The headers table shows the same behavior as the `FileManager`, so I think the problem is in `FileManager._array_slice_to_loader_slice`. When I watch it in the debugger, it adds an extra slice in the `if fits_array_shape[0] == 1` part, and then the line `aslice = aslice[:-1*len(fits_array_shape)]` removes the intended slice for that third dimension, but I don't understand well enough what's supposed to be happening in there.

There doesn't seem to be the same problem with dataset BXXDZ, another one from later in the same day, but that one doesn't have the "raster map repeat number" pixel dimension that ALMMM does, so maybe that's why. With BXXDZ, `ds[0, 0].data` gives a dask array with 1 chunk, and `ds[0, 0].files` has one file like it should. That dataset has FITS files with shape `(1, 936, 2549)`, so still with the leading dimension of 1.

My datasets are still in their proprietary period, but I'm happy to grant access if I can or send over sample files.

### Expected vs Actual behavior

Expected: `ds[0, 0, 0].files` shows one file, to match the single chunk in `ds[0, 0, 0].data`.

Actual: `ds[0, 0, 0].files` shows the same 1100 files as `ds[0, 0].files`

### Steps to Reproduce

```python
import dkist
ds = dkist.load_dataset("/home/svankooten/globus/pid_2_86/ALMMM/")
ds[0, 0, 0].files
```

### System Details
```
==============================
dkist Installation Information
==============================

General
#######
OS: Ubuntu (22.04, Linux 6.5.0-45-generic)
Arch: 64bit, (x86_64)
dkist: 1.8.0
Installation path: /home/svankooten/.anaconda/lib/python3.12/site-packages/dkist-1.8.0.dist-info

Required Dependencies
#####################
aiohttp: 3.9.5
asdf: 3.3.0
asdf-astropy: 0.6.1
asdf-coordinates-schemas: 0.3.0
asdf-standard: 1.1.1
asdf-transform-schemas: 0.5.0
asdf-wcs-schemas: 0.4.0
astropy: 6.1.2
dask: 2024.7.1
globus-sdk: 3.45.0
gwcs: 0.21.0
matplotlib: 3.9.2
ndcube: 2.2.2
numpy: 1.26.4
parfive: 2.1.0
platformdirs: 4.2.2
sunpy: 6.0.0
tqdm: 4.66.4
```

Discussion:
codspeed-hq[bot]: ## [CodSpeed Performance Report](https://codspeed.io/DKISTDC/dkist/branches/SolarDrew:filemanager)

### Merging #453 will **not alter performance**

<sub>Comparing <code>SolarDrew:filemanager</code> (aa50744) with <code>main</code> (4e93042)</sub>



### Summary

`✅ 9` untouched benchmarks





Cadair: I think this needs a regression test and maybe tests with other data as well to make sure we haven't broken anything else.

Patch:
diff --git a/changelog/453.bugfix.rst b/changelog/453.bugfix.rst
new file mode 100644
index 0000000..fca4170
--- /dev/null
+++ b/changelog/453.bugfix.rst
@@ -0,0 +1,1 @@
+Minor tweak to correct indexing of >4D datasets.
diff --git a/dkist/io/file_manager.py b/dkist/io/file_manager.py
index a2f3e2f..bc9a995 100644
--- a/dkist/io/file_manager.py
+++ b/dkist/io/file_manager.py
@@ -242,7 +242,7 @@ class BaseFileManager:
         aslice = list(sanitize_slices(aslice, len(self.output_shape)))
         if fits_array_shape[0] == 1:
             # Insert a blank slice for the dummy dimension
-            aslice.insert(len(fits_array_shape) - 1, slice(None))
+            aslice.insert(-(len(fits_array_shape)-1), slice(None))
         # Now only use the dimensions of the slice not covered by the array axes
         aslice = aslice[:-1*len(fits_array_shape)]
         return tuple(aslice)


Test patch:
diff --git a/dkist/conftest.py b/dkist/conftest.py
index 9d6f648..b8bfee1 100644
--- a/dkist/conftest.py
+++ b/dkist/conftest.py
@@ -267,6 +267,31 @@ def dataset_4d(identity_gwcs_4d, empty_meta):
     return Dataset(array, wcs=identity_gwcs_4d, meta=empty_meta, unit=u.count)
 
 
+@pytest.fixture
+def dataset_5d(identity_gwcs_5d_stokes, empty_meta):
+    shape = (4, 40, 30, 20, 10)
+    x = np.ones(shape)
+    array = da.from_array(x, tuple(shape))
+
+    identity_gwcs_4d.pixel_shape = array.shape[::-1]
+    identity_gwcs_4d.array_shape = array.shape
+
+    ds = Dataset(array, wcs=identity_gwcs_5d_stokes, meta={"inventory": {}, "headers": Table()}, unit=u.count)
+    fileuris = np.array([f"dummyfile_{i}" for i in range(np.prod(shape[:-2]))]).reshape(shape[:-2])
+    ds._file_manager = FileManager.from_parts(fileuris, 0, float, shape[-2:], loader=AstropyFITSLoader, basepath="./")
+
+    return ds
+
+
+@pytest.fixture
+def dataset_5d_dummy_filemanager_axis(dataset_5d):
+    shape = dataset_5d.data.shape
+    fileuris = np.array([f"dummyfile_{i}" for i in range(np.prod(shape[:-2]))]).reshape(shape[:-2])
+    dataset_5d._file_manager = FileManager.from_parts(fileuris, 0, float, (1, *shape[-2:]), loader=AstropyFITSLoader, basepath="./")
+
+    return dataset_5d
+
+
 @pytest.fixture
 def eit_dataset():
     eitdir = Path(rootdir) / "EIT"
@@ -350,6 +375,16 @@ def visp_dataset_no_headers(tmp_path_factory):
     return load_dataset(vispdir / "test_visp_no_headers.asdf")
 
 
+@pytest.fixture
+def large_visp_no_dummy_axis(large_visp_dataset):
+    # Slightly tweaked dataset to remove the dummy axis in the file manager array shape.
+    shape = large_visp_dataset.data.shape[:2]
+    fileuris = np.array([f"dummyfile_{i}" for i in range(np.prod(shape))]).reshape(shape)
+    large_visp_dataset._file_manager = FileManager.from_parts(fileuris, 0, float, (50, 128), loader=AstropyFITSLoader, basepath="./")
+
+    return large_visp_dataset
+
+
 @pytest.hookimpl(hookwrapper=True)
 def pytest_runtest_call(item):
     try:
diff --git a/dkist/dataset/tests/test_dataset.py b/dkist/dataset/tests/test_dataset.py
index 0dc0653..90b8c57 100644
--- a/dkist/dataset/tests/test_dataset.py
+++ b/dkist/dataset/tests/test_dataset.py
@@ -176,3 +176,25 @@ def test_header_slicing_3D_slice(large_visp_dataset):
 
     assert len(sliced.files.filenames) == len(sliced_headers["FILENAME"]) == len(sliced.headers)
     assert (sliced.headers["DINDEX3", "DINDEX4"] == sliced_headers["DINDEX3", "DINDEX4"]).all()
+
+
+@pytest.mark.accept_cli_dataset
+def test_file_slicing_with_dummy_axis(dataset_5d_dummy_filemanager_axis):
+    ds = dataset_5d_dummy_filemanager_axis
+    shape = ds.data.shape
+    assert len(ds.files) == np.prod(shape[:3])
+    assert len(ds[0].files) == np.prod(shape[1:3])
+    assert len(ds[0, 0].files) == np.prod(shape[2])
+    assert len(ds[0, 0, 0].files) == 1
+    assert len(ds[0, 0, 0, 0].files) == 1
+
+
+@pytest.mark.accept_cli_dataset
+def test_file_slicing_without_dummy_axis(dataset_5d):
+    ds = dataset_5d
+    shape = ds.data.shape
+    assert len(ds.files) == np.prod(shape[:3])
+    assert len(ds[0].files) == np.prod(shape[1:3])
+    assert len(ds[0, 0].files) == np.prod(shape[2])
+    assert len(ds[0, 0, 0].files) == 1
+    assert len(ds[0, 0, 0, 0].files) == 1