Repository: pauldmccarthy/indexed_gzip (language: python)

Issue:
CrcError with fMRIPrep
With version 1.6.1, I am getting the following error reading a NIfTI file generated by nibabel. Specifically, the file is created by `FixHeaderApplyTransforms (niworkflows v1.3.4)`, which is part of fMRIprep.

```
>>> indexed_gzip.IndexedGzipFile("vol0245_xform-00245.nii.gz").read()
---------------------------------------------------------------------------
CrcError                                  Traceback (most recent call last)
<ipython-input-3-41ce3f08c5f5> in <module>
----> 1 indexed_gzip.IndexedGzipFile("vol0245_xform-00245.nii.gz").read()

indexed_gzip/indexed_gzip.pyx in indexed_gzip.indexed_gzip._IndexedGzipFile.read()

indexed_gzip/indexed_gzip.pyx in indexed_gzip.indexed_gzip._IndexedGzipFile.read()

CrcError: CRC/size validation failed - the GZIP data might be corrupt
```

With version 1.5.3, the file loads fine. Please let me know if I can e-mail you the file in question. 


Discussion / hints:
Hi @HippocampusGirl, is the CRC of the file valid? You can check by running `gzip -t vol0245_xform-00245.nii.gz`.

If it is not valid, then you may want to check with the fMRIprep people to see how the file has been created.

If it is valid, then this suggests a problem with `indexed_gzip` - in this case it would be great if you could share the file with me.

In any case, as an interim workaround, you can tell `indexed_gzip` to skip the CRC check by passing `skip_crc_check=True` when creating an `IndexedGzipFile` object.
Yes, the CRC is valid
```
; gzip -t -v vol0245_xform-00245.nii.gz
vol0245_xform-00245.nii.gz:      OK
```

I am sadly unable to pass `skip_crc_check=True`, as I am using `nibabel`, which does not allow the user to configure this option.
I am unable to share the file publicly. Could you provide me your e-mail address or any other way to send it? My e-mail address is <lea.waller@charite.de>
Sure, no problem - I'll send you an email now..
Seeing this in an independent reproduction here: https://github.com/nipreps/fmriprep/issues/2549

These files are being [generated with nibabel](https://github.com/nipreps/niworkflows/blob/32a33d61ab64e262a4ebcf8afcf232d70c2f92b3/niworkflows/utils/images.py#L95-L96).

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4164d6ec..e8b33402 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,14 @@
 # `indexed_gzip` changelog
 
 
+## 1.6.4 (October 18th 2021)
+
+
+* Fixed a bug related to buffering input data, which was causing a spurious
+  `CrcError` (#80, #87).
+
+
+
 ## 1.6.3 (September 14th 2021)
 
 
diff --git a/indexed_gzip/__init__.py b/indexed_gzip/__init__.py
index ad097221..8011a0e3 100644
--- a/indexed_gzip/__init__.py
+++ b/indexed_gzip/__init__.py
@@ -19,4 +19,4 @@
 """
 
 
-__version__ = '1.6.3'
+__version__ = '1.6.4'
diff --git a/indexed_gzip/zran.c b/indexed_gzip/zran.c
index 3ad1f90d..7804011a 100644
--- a/indexed_gzip/zran.c
+++ b/indexed_gzip/zran.c
@@ -1394,6 +1394,14 @@ static int _zran_read_data_from_file(zran_index_t *index,
 
             zran_log("End of file, stopping inflation\n");
 
+            /*
+             * Reset next_in pointer to beginning of
+             * read buffer, as we rotated it above,
+             * and the area that next_in was pointing
+             * to may have been overwritten by memmove.
+             */
+            stream->next_in = index->readbuf;
+
             /*
              * we have uncompressed everything,
              * so we now know its size.


Test patch:
diff --git a/indexed_gzip/tests/ctest_zran.pyx b/indexed_gzip/tests/ctest_zran.pyx
index ce20993a..913846f1 100644
--- a/indexed_gzip/tests/ctest_zran.pyx
+++ b/indexed_gzip/tests/ctest_zran.pyx
@@ -1445,3 +1445,52 @@ def test_inflateInit_leak_on_error():
     # it should remain stable
     mem = mem[5:]
     assert np.all(mem == mem[0])
+
+
+# pauldmccarthy/indexed_gzip#80
+def test_read_eof_memmove_rotate_bug(seed):
+
+    # This bug was triggered by the read buffer rotation
+    # that takes place in zran.c::_zran_read_data_from_file,
+    # and occurs when the file is at EOF, and the
+    # stream->next_in pointer is ahead of index->readbuf by
+    # less than stream->avail_in bytes. In this case, the
+    # source and dest pointers passed to memmove are
+    # overlapping, so the area pointed to by next_in is
+    # modified. The bug was that, when at EOF, the
+    # stream->next_in pointer was not being reset to point
+    # to the beginning of readbuf, so the subsequent read
+    # of the gzip footer in _zran_validate_stream was
+    # reading from the wrong location.
+    #
+    # We can trigger this situation by generating a file
+    # which has compressed file size (X * readbuf_size) + Y,
+    # for any integer x, and for 9 <= Y < 16
+
+    cdef zran.zran_index_t index
+    cdef FILE             *cfid
+
+    with tempdir():
+        nelems = np.random.randint(524288, 525000, 1)[0]
+        data   = np.random.random(nelems)
+        with gzip.open('test.gz', 'wb') as f:
+            f.write(data.tobytes())
+
+        fsize        = os.stat('test.gz').st_size
+        readbuf_size = fsize - 10
+
+        with open('test.gz', 'rb') as pyfid:
+            cfid = fdopen(pyfid.fileno(), 'rb')
+            assert not zran.zran_init(&index,
+                                      cfid,
+                                      NULL,
+                                      4194304,
+                                      32768,
+                                      readbuf_size,
+                                      zran.ZRAN_AUTO_BUILD)
+
+            eof = nelems * 8 - 1
+            got = zran.zran_seek(&index, eof, SEEK_SET, NULL)
+
+            assert got                    == zran.ZRAN_SEEK_OK, got
+            assert zran.zran_tell(&index) == eof
diff --git a/indexed_gzip/tests/test_zran.py b/indexed_gzip/tests/test_zran.py
index 2cb597d9..ca7aae2f 100644
--- a/indexed_gzip/tests/test_zran.py
+++ b/indexed_gzip/tests/test_zran.py
@@ -130,3 +130,6 @@ def test_standard_usage_with_null_padding(concat):
 
     def test_inflateInit_leak_on_error():
         ctest_zran.test_inflateInit_leak_on_error()
+
+    def test_read_eof_memmove_rotate_bug(seed):
+        ctest_zran.test_read_eof_memmove_rotate_bug(seed)