Repository: pauldmccarthy/indexed_gzip (language: python)

Issue:
Trailing null bytes cause errors when reading
I'm running into an interesting error in which trailing null bytes in a gzip file cause errors when reading a gzip file using `indexed_gzip`. Here's the closest reproduction (the trailing null bytes occur when I'm using bsdtar to tar and gzip a directory and print it to stdout, because in that case, bsdtar pads the gz file with null bytes):

```python
import tempfile
import subprocess
import shutil
from indexed_gzip import IndexedGzipFile

with tempfile.TemporaryDirectory() as tmpdir:
    with open(tmpdir + "/" + "tempfile", "wb") as f:
        f.write(b"hello world")
    
    # Make sure you're using bsdtar here, since bsdtar pads the gz file with null bytes
    # if the .tar.gz file is written to stdout.
    fileobj = subprocess.Popen(['tar', 'czf', '-', '-C', tmpdir, '.'], stdout=subprocess.PIPE).stdout
    with open('/tmp/out1.tar.gz', 'w+') as f:
        f.truncate(0)

    subprocess.Popen(['tar', 'czf', '/tmp/out2.tar.gz', '-C', tmpdir, '.'], stdout=subprocess.PIPE).stdout.read()
    with open('/tmp/out1.tar.gz', 'wb') as f:
        shutil.copyfileobj(fileobj, f)
    
    with open('/tmp/out1.tar.gz', 'rb') as f1, open('/tmp/out2.tar.gz', 'rb') as f2:
        igz1 = IndexedGzipFile(fileobj=f1)
        igz1.read()
        # the above line fails (nondeterministically, but most of the time) with the following stack trace:
        # Traceback (most recent call last):
        # File "test.py", line 22, in <module>
        #     igz1.read()  # this fails
        # File "indexed_gzip/indexed_gzip.pyx", line 647, in indexed_gzip.indexed_gzip._IndexedGzipFile.read
        #     with self.__file_handle():
        # File "indexed_gzip/indexed_gzip.pyx", line 662, in indexed_gzip.indexed_gzip._IndexedGzipFile.read
        #     raise ZranError('zran_read returned error '
        # indexed_gzip.indexed_gzip.ZranError: zran_read returned error (-3)
        # zran_free

        igz2 = IndexedGzipFile(fileobj=f2)
        igz2.read()  # this never fails
```

You can see the contents of /tmp/out1.tar.gz (padded with null bytes, nondeterministically fails) versus /tmp/out2.tar.gz (not padded, always works) here:

```bash
(venv) epicfaace@ashwins-mbp indexed_gzip % od -t x1 /tmp/out1.tar.gz
0000000    1f  8b  08  00  19  83  a0  60  00  03  ed  d1  41  0a  c2  30
0000020    10  05  d0  1c  65  4e  d0  fe  49  26  f5  3c  a1  26  58  88
0000040    b4  d4  8a  d7  37  b4  45  41  68  57  46  11  f3  36  7f  33
0000060    84  3f  99  aa  56  d9  01  38  00  34  67  63  e7  84  96  25
0000100    57  c4  02  0b  06  8b  61  02  6b  6d  58  91  cd  5f  4d  a9
0000120    eb  65  72  63  aa  e2  87  ae  0d  ce  b5  7e  63  2e  8d  85
0000140    b0  f3  ce  ba  c7  23  7f  44  55  4f  fe  3c  84  2e  6e  ed
0000160    fd  06  e9  3f  1a  91  9d  fb  b3  79  b9  bf  49  73  8a  90
0000200    af  d2  d3  9f  df  ff  e4  63  ec  e9  d6  8f  f1  f8  ed  2a
0000220    45  51  14  c5  07  dd  01  c1  00  32  68  00  0a  00  00  00
0000240    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
*
0024000
(venv) epicfaace@ashwins-mbp indexed_gzip % od -t x1 /tmp/out2.tar.gz
0000000    1f  8b  08  00  19  83  a0  60  00  03  ed  d1  41  0a  c2  30
0000020    10  05  d0  1c  65  4e  d0  fe  49  26  f5  3c  a1  26  58  88
0000040    b4  d4  8a  d7  37  b4  45  41  68  57  46  11  f3  36  7f  33
0000060    84  3f  99  aa  56  d9  01  38  00  34  67  63  e7  84  96  25
0000100    57  c4  02  0b  06  8b  61  02  6b  6d  58  91  cd  5f  4d  a9
0000120    eb  65  72  63  aa  e2  87  ae  0d  ce  b5  7e  63  2e  8d  85
0000140    b0  f3  ce  ba  c7  23  7f  44  55  4f  fe  3c  84  2e  6e  ed
0000160    fd  06  e9  3f  1a  91  9d  fb  b3  79  b9  bf  49  73  8a  90
0000200    af  d2  d3  9f  df  ff  e4  63  ec  e9  d6  8f  f1  f8  ed  2a
0000220    45  51  14  c5  07  dd  01  c1  00  32  68  00  0a  00  00    
0000237
```
Fix processing of gz files with null padding
Fix processing of gz files with null padding. Fixes https://github.com/pauldmccarthy/indexed_gzip/issues/69 and fixes https://github.com/codalab/codalab-worksheets/issues/3514.


Discussion / hints:
Note that [bsdtar's documentation does mention](https://opensource.apple.com/source/libarchive/libarchive-33.20.2/libarchive/doc/text/bsdtar.1.txt.auto.html) this behavior of padding gz files:

```
     All archive output is written in correctly-sized blocks, even if the out-
     put is being compressed.  Whether or not the last output block is padded
     to a full block size varies depending on the format and the output
     device.  For tar and cpio formats, the last block of output is padded to
     a full block size if the output is being written to standard output or to
     a character or block device such as a tape drive.	If the output is being
     written to a regular file, the last block will not be padded.  Many com-
     pressors, including gzip(1) and bzip2(1), complain about the null padding
     when decompressing an archive created by tar, although they still extract
     it correctly.
```

I'm a bit at a loss as to why padding a gz file with null bytes would cause issues when using it with IndexedGzipFile, though. Any thoughts @pauldmccarthy ?
Okay, I think I found a much simpler reproduction of this issue:

```python
import gzip
from io import BytesIO

igz1 = IndexedGzipFile(fileobj=BytesIO(gzip.compress(b"hello world") + b"\0" * 10))
igz1.read()
```

This consistently gives the error: "zran_read returned error (-3)"
This is how cpython handles it: https://github.com/python/cpython/blob/main/Lib/gzip.py#L534-L541
Hi @epicfaace, thanks for doing the research. I didn't know about padding bytes, as I've always gone to the [gzip RFC](https://datatracker.ietf.org/doc/html/rfc1952), which doesn't mention them at all. The ref in the cpython code (http://www.gzip.org/#faq8) doesn't seem to exist anymore, but I did find this: https://www.gnu.org/software/gzip/manual/gzip.html#Tapes

This has actually reminded me that I intended to add a CRC check into `indexed_gzip` - I'll have a play with your #70, and see if the code can be cleaned up a bit, and CRC checking implemented


The Python 2.7-Windows builds are failing because of https://github.com/pauldmccarthy/indexed_gzip/issues/71.

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0eca518b..e5cdc68e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,7 +1,7 @@
 # `indexed_gzip` changelog
 
 
-## 1.6.0 (..)
+## 1.6.0 (May 25th 2021)
 
 
 * Python 2.7 wheels for Windows are no longer being built (#71, 73).
@@ -10,6 +10,15 @@
   `indexed_gzip` can still be loaded, but index files created with
   `indexed_gzip` 1.6.0 cannot be loaded by older versions of `indexed_gzip`
   (#75).
+* CRC and size validation of uncompressed data is now performed by default,
+  on the first pass through a GZIP file. This can be disabled by setting the
+  new `skip_crc_check` argument to `False` when creating an
+  `IndexedGzipFile`. Validation is not performed when an existing index is
+  imported from file (#72).
+* Null padding bytes at the end of a GZIP file, or in between GZIP streams,
+  are now skipped over (#69, #70, #72).
+* Seek points are now created at the beginning of every GZIP stream, in files
+  containing concatenated streams (#72).
 
 
 ## 1.5.3 (March 23rd 2021)
diff --git a/indexed_gzip/__init__.py b/indexed_gzip/__init__.py
index 890baef4..42c72f07 100644
--- a/indexed_gzip/__init__.py
+++ b/indexed_gzip/__init__.py
@@ -19,4 +19,4 @@
 """
 
 
-__version__ = '1.5.3'
+__version__ = '1.6.0'
diff --git a/indexed_gzip/indexed_gzip.pyx b/indexed_gzip/indexed_gzip.pyx
index ca35449f..20f67ccf 100644
--- a/indexed_gzip/indexed_gzip.pyx
+++ b/indexed_gzip/indexed_gzip.pyx
@@ -100,6 +100,10 @@ class IndexedGzipFile(io.BufferedReader):
         :arg auto_build:       If ``True`` (the default), the index is
                                automatically built on calls to :meth:`seek`.
 
+        :arg skip_crc_check:   Defaults to ``False``. If ``True``, CRC/size
+                               validation of the uncompressed data is not
+                               performed.
+
         :arg spacing:          Number of bytes between index seek points.
 
         :arg window_size:      Number of bytes of uncompressed data stored with
@@ -237,6 +241,12 @@ cdef class _IndexedGzipFile:
     """
 
 
+    cdef readonly bint skip_crc_check
+    """Flag which is set to ``True`` if CRC/size validation of uncompressed
+    data is disabled.
+    """
+
+
     cdef readonly object filename
     """String containing path of file being indexed. Used to release and
     reopen file handles between seeks and reads.
@@ -274,7 +284,8 @@ cdef class _IndexedGzipFile:
                  readbuf_size=1048576,
                  readall_buf_size=16777216,
                  drop_handles=True,
-                 index_file=None):
+                 index_file=None,
+                 skip_crc_check=False):
         """Create an ``_IndexedGzipFile``. The file may be specified either
         with an open file handle (``fileobj``), or with a ``filename``. If the
         former, the file must have been opened in ``'rb'`` mode.
@@ -291,6 +302,12 @@ cdef class _IndexedGzipFile:
         :arg auto_build:       If ``True`` (the default), the index is
                                automatically built on calls to :meth:`seek`.
 
+        :arg skip_crc_check:   Defaults to ``False``. If ``True``, CRC/size
+                               validation of the uncompressed data is not
+                               performed. Automatically enabled if an
+                               ``index_file`` is provided, or if
+                               :meth:`import_index` is called.
+
         :arg spacing:          Number of bytes between index seek points.
 
         :arg window_size:      Number of bytes of uncompressed data stored with
@@ -366,13 +383,16 @@ cdef class _IndexedGzipFile:
         self.readbuf_size     = readbuf_size
         self.readall_buf_size = readall_buf_size
         self.auto_build       = auto_build
+        self.skip_crc_check   = skip_crc_check
         self.drop_handles     = drop_handles
         self.filename         = filename
         self.own_file         = own_file
         self.pyfid            = fileobj
 
-        if self.auto_build: flags = zran.ZRAN_AUTO_BUILD
-        else:               flags = 0
+        flags = 0
+
+        if auto_build:     flags |= zran.ZRAN_AUTO_BUILD
+        if skip_crc_check: flags |= zran.ZRAN_SKIP_CRC_CHECK
 
         # Set index.fd here just for the initial
         # call, as __file_handle may otherwise
@@ -567,8 +587,9 @@ cdef class _IndexedGzipFile:
         with self.__file_handle():
             ret = zran.zran_build_index(&self.index, 0, 0)
 
-        if ret != 0:
-            raise ZranError('zran_build_index returned error')
+        if ret != zran.ZRAN_BUILD_INDEX_OK:
+            raise ZranError('zran_build_index returned '
+                            'error: {}'.format(ret))
 
         log.debug('%s.build_full_index()', type(self).__name__)
 
@@ -602,10 +623,7 @@ cdef class _IndexedGzipFile:
         with self.__file_handle(), nogil:
             ret = zran.zran_seek(index, off, c_whence, NULL)
 
-        if ret < 0:
-            raise ZranError('zran_seek returned error: {}'.format(ret))
-
-        elif ret == zran.ZRAN_SEEK_NOT_COVERED:
+        if ret == zran.ZRAN_SEEK_NOT_COVERED:
             raise NotCoveredError('Index does not cover '
                                   'offset {}'.format(offset))
 
@@ -613,8 +631,12 @@ cdef class _IndexedGzipFile:
             raise NotCoveredError('Index must be completely built '
                                   'in order to seek from SEEK_END')
 
+        elif ret == zran.ZRAN_SEEK_CRC_ERROR:
+            raise CrcError('CRC/size validation failed - '
+                           'the GZIP data might be corrupt')
+
         elif ret not in (zran.ZRAN_SEEK_OK, zran.ZRAN_SEEK_EOF):
-            raise ZranError('zran_seek returned unknown code: {}'.format(ret))
+            raise ZranError('zran_seek returned error: {}'.format(ret))
 
         offset = self.tell()
 
@@ -657,10 +679,13 @@ cdef class _IndexedGzipFile:
                 with nogil:
                     ret = zran.zran_read(index, buffer, bufsz)
 
-                # see how the read went
-                if ret == zran.ZRAN_READ_FAIL:
-                    raise ZranError('zran_read returned error '
-                                    '({})'.format(ret))
+                # No bytes were read, and there are
+                # no more bytes to read. This will
+                # happen when the seek point was at
+                # or beyond EOF when zran_read was
+                # called
+                if ret == zran.ZRAN_READ_EOF:
+                    break
 
                 # This will happen if the current
                 # seek point is not covered by the
@@ -669,13 +694,16 @@ cdef class _IndexedGzipFile:
                     raise NotCoveredError('Index does not cover '
                                           'current offset')
 
-                # No bytes were read, and there are
-                # no more bytes to read. This will
-                # happen when the seek point was at
-                # or beyond EOF when zran_read was
-                # called
-                elif ret == zran.ZRAN_READ_EOF:
-                    break
+                # CRC or size check failed - data
+                # might be corrupt
+                elif ret == zran.ZRAN_READ_CRC_ERROR:
+                    raise CrcError('CRC/size validation failed - '
+                                   'the GZIP data might be corrupt')
+
+                # Unknown error
+                elif ret < 0:
+                    raise ZranError('zran_read returned error '
+                                    '({})'.format(ret))
 
                 nread  += ret
                 offset += ret
@@ -954,6 +982,8 @@ cdef class _IndexedGzipFile:
             if ret != zran.ZRAN_IMPORT_OK:
                 raise ZranError('import_index returned error: {}'.format(ret))
 
+            self.skip_crc_check = True
+
         finally:
             if close_file:
                 fileobj.close()
@@ -1052,6 +1082,14 @@ class ZranError(IOError):
     pass
 
 
+class CrcError(OSError):
+    """Exception raised by the :class:`_IndexedGzipFile` when a CRC/size
+    validation check fails, which suggests that the GZIP data might be
+    corrupt.
+    """
+    pass
+
+
 class NoHandleError(ValueError):
     """Exception raised by the :class:`_IndexedGzipFile` when
     ``drop_handles is True`` and an attempt is made to access the underlying
diff --git a/indexed_gzip/zran.c b/indexed_gzip/zran.c
index 3ab4082b..229a95a1 100644
--- a/indexed_gzip/zran.c
+++ b/indexed_gzip/zran.c
@@ -22,9 +22,10 @@
 #include "io.h"
 static int is_readonly(FILE *fd, PyObject *f)
 {
-    /* Can't find a way to do this correctly under Windows and
-       the check is not required anyway since the underlying
-       Python module checks it already */
+    /* Can't find a way to do this correctly under
+       Windows and the check is not required anyway
+       since the underlying Python module checks it
+       already */
     return 1;
 }
 #else
@@ -123,10 +124,11 @@ static uint64_t _zran_index_limit(
 
 
 /* Return codes for _zran_get_point_at */
+int ZRAN_GET_POINT_CRC_ERROR   =  -2;
 int ZRAN_GET_POINT_FAIL        =  -1;
-int ZRAN_GET_POINT_OK          =  0;
-int ZRAN_GET_POINT_NOT_COVERED =  1;
-int ZRAN_GET_POINT_EOF         =  2;
+int ZRAN_GET_POINT_OK          =   0;
+int ZRAN_GET_POINT_NOT_COVERED =   1;
+int ZRAN_GET_POINT_EOF         =   2;
 
 /*
  * Searches for the zran_point which preceeds the given offset. The offset
@@ -168,9 +170,13 @@ static int _zran_get_point_at(
  * be expanded to encompass it.
  *
  * The input arguments and return values are identical to the
- * _zran_get_point_at function, however if the index has been initialised
- * with the ZRAN_AUTO_BUILD flag, this function will never return
- * ZRAN_GET_POINT_NOT_COVERED.
+ * _zran_get_point_at function, however:
+ *
+ *   - if the index has been initialised with the ZRAN_AUTO_BUILD flag, this
+ *     function will never return ZRAN_GET_POINT_NOT_COVERED.
+ *
+ *   - If a CRC validation error occurs while the index is being expanded,
+ *     ZRAN_GET_POINT_CRC_ERROR is returned.
  */
 static int _zran_get_point_with_expand(
     zran_index_t  *index,      /* The index                           */
@@ -207,24 +213,47 @@ static uint64_t _zran_estimate_offset(
 
 
 /*
- * Used by _zran_expand_index, and _zran_read. Seeks to the specified
- * compressed data offset, and initialises zlib to start
- * decompressing/inflating from said offset.
+ * Used by _zran_inflate. Initialises zlib to start decompressing/inflating
+ * from either:
+ *
+ *  - the current seek location in the compressed data, or
+ *  - from a location denoted by a specific index point
+ *
+ * If an index point is provided, the function will seek to the specified
+ * compressed data offset before initialising zlib.
+ *
+ * Otherwise (no index point), inflation is initialised at the current seek
+ * location in the input data, and a GZIP header is expected at that location.
+ *
+ * The index->readbuf and readbuf_size, and the z_stream->avail_in, avail_out,
+ * next_in and next_out fields must all be set before this function is called.
  *
- * Returns 0 for success, non-0 on failure.
+ * Returns the number of bytes over the input data that were read (which could
+ * be 0), or a negative value on failure.
  */
 static int _zran_init_zlib_inflate(
-    zran_index_t *index,      /* The index */
+    zran_index_t *index,        /* The index */
 
-    z_stream     *stream,     /* Pointer to a z_stream struct */
+    z_stream     *stream,       /* Pointer to a z_stream struct */
 
-    zran_point_t *point       /* Pass in NULL to initialise for inflation from
-                                 the beginning of the stream. Or pass a
-                                 pointer to the index point corresponding to
-                                 the location to start from. */
+    zran_point_t *point         /* Pass in NULL to initialise for inflation
+                                   from the current location in the input file.
+                                   Or pass a pointer to the index point
+                                   corresponding to the location to start
+                                   from. */
 );
 
 
+/*
+ * Return codes for _zran_expand_index. These are currently
+ * assumed to have identical values to the ZRAN_BUILD_INDEX
+ * return codes.
+ */
+int ZRAN_EXPAND_INDEX_OK        =  0;
+int ZRAN_EXPAND_INDEX_FAIL      = -1;
+int ZRAN_EXPAND_INDEX_CRC_ERROR = -2;
+
+
 /*
  * Expands the index from its current end-point until the given offset (which
  * must be specified relative to the compressed data stream).
@@ -240,7 +269,9 @@ static int _zran_init_zlib_inflate(
  * be a long distance between block boundaries (longer than the desired index
  * point spacing).
  *
- * Returns 0 on success, non-0 on failure.
+ * Returns 0 on success. If a CRC check fails, returns
+ * ZRAN_EXPAND_INDEX_CRC_ERROR. For other types of failure, returns
+ * ZRAN_EXPAND_INDEX_FAIL.
  */
 static int _zran_expand_index(
     zran_index_t *index, /* The index                      */
@@ -277,23 +308,53 @@ static int _zran_add_point(
 );
 
 
+/* _zran_read_data return codes */
+int ZRAN_READ_DATA_EOF   = -1;
+int ZRAN_READ_DATA_ERROR = -2;
+
+/*
+ * This function is a sub-function of _zran_inflate, used to read data from
+ * the input file to be passed to zlib:inflate for decompression.
+ *
+ * Up to index->readbuf_size bytes are read from the input file into
+ * index->readbuf, and the z_stream counters/pointers are updated accordingly.
+ *
+ * On success, returns 0.
+ *
+ * If there are no more bytes left to read from the input file (i.e. we are at
+ * EOF), ZRAN_READ_DATA_EOF is returned. If an error occurs, returns
+ * ZRAN_READ_DATA_ERROR.
+ */
+static int _zran_read_data_from_file(
+    zran_index_t *index,        /* The index                               */
+    z_stream     *stream,       /* The z_stream struct                     */
+    uint64_t      cmp_offset,   /* Current offset in the compressed data   */
+    uint64_t      uncmp_offset, /* Current offset in the uncompressed data */
+    uint32_t      need_atleast  /* Skip read if the read buffer already has
+                                   this many bytes */
+);
+
+
 /* _zran_find_next_stream return codes */
 int ZRAN_FIND_STREAM_ERROR     = -2;
 int ZRAN_FIND_STREAM_NOT_FOUND = -1;
 
+
 /*
- * This function is used to search for concatenated compressed streams.  It
- * searches through the compressed data (pointed to by stream->next_in) to
- * find the location of the next compressed stream.
+ * This function is a sub-function of _zran_inflate, used to search for a new
+ * GZIP stream in a series of concatenated streams.  It searches through the
+ * compressed data (pointed to by stream->next_in) to find the location of the
+ * next compressed stream.
  *
  * If a new stream was found, the z_stream struct is re-initialised to
- * decompress data from the new stream. In this case, the function returns 0.
+ * decompress data from the new stream, using _zran_init_zlib_inflate. In
+ * this case, the value returned by that function is returned.
  *
  * Otherwise (if a compressed stream was not found), this function returns
  * ZRAN_FIND_STREAM_NOT_FOUND.
  *
- * The number of bytes that were read before the new stream was found is
- * stored in the provided offset pointer.
+ * The number of bytes that were skipped over before the new stream was found
+ * is added to the provided offset pointer.
  *
  * If an error occurs, ZRAN_FIND_STREAM_ERROR is returned.
  */
@@ -304,7 +365,35 @@ static int _zran_find_next_stream(
 );
 
 
+/* _zran_validate_stream return codes */
+int ZRAN_VALIDATE_STREAM_ERROR   = -2;
+int ZRAN_VALIDATE_STREAM_INVALID = -1;
+
+
+/*
+ * This function is a sub-function of _zran_inflate, called when the end of a
+ * gzip stream is reached. It reads the CRC32 and uncompressed file size from
+ * the end of the stream, and compares them to the CRC32 and size that was
+ * incrementally calculated by _zran_inflate (which are stored in
+ * index->stream_crc32 and index->stream_size),
+ *
+ * The number of bytes that were read before the new stream was found is
+ * added to the provided offset pointer.
+ *
+ * If ZRAN_SKIP_CRC_CHECK is active, this function returns immediately without
+ * doing anything.
+ *
+ * If an error occurs, ZRAN_VALIDATE_STREAM_ERROR is returned.
+ */
+static int _zran_validate_stream(
+    zran_index_t *index,  /* The index                                      */
+    z_stream     *stream, /* The z_stream struct                            */
+    int          *offset  /* Used to store the number of bytes skipped over */
+);
+
+
 /* _zran_inflate return codes */
+int ZRAN_INFLATE_CRC_ERROR      = -6;
 int ZRAN_INFLATE_ERROR          = -5;
 int ZRAN_INFLATE_NOT_COVERED    = -4;
 int ZRAN_INFLATE_OUTPUT_FULL    = -3;
@@ -346,6 +435,13 @@ uint32_t ZRAN_INFLATE_STOP_AT_BLOCK         = 64;
  * offset, inflates (a.k.a. decompresses) it, and copies the decompressed
  * data to the provided output buffer.
  *
+ * This function is complicated because it is used in three different
+ * situations:
+ *   - When generating the index (by zran_expand_index)
+ *   - When starting from an index seek point and discarding compressed data
+ *     to find a requested seek location (by zran_read)
+ *   - When actually reading and decompressing data (by zran_read).
+ *
  * This function may be used in a re-entrant or non-re-entrant manner,
  * depending on the flags which are used. In the latter (more likely) case,
  * various pieces of information representing the current inflation state are
@@ -360,24 +456,27 @@ uint32_t ZRAN_INFLATE_STOP_AT_BLOCK         = 64;
  *      ZRAN_INFLATE_USE_OFFSET is not active, index->inflate_cmp_offset and
  *      index->inflate_uncmp_offset are used as the starting point.
  *
- *   2. Initialises the z_stream struct, if
- *      ZRAN_INFLATE_INIT_Z_STREAM is active. Otherwise, the function assumes
- *      that the z_stream struct is already initialised and ready to be used.
- *
- *   3. Create a read buffer, if ZRAN_INFLATE_INIT_READBUF is active. A
+ *   2. Create a read buffer, if ZRAN_INFLATE_INIT_READBUF is active. A
  *      reference to the read buffer is stored at index->readbuf.  If
  *      ZRAN_INFLATE_INIT_READBUF is not set, the function assumes that the
  *      read buffer already exists.
  *
- *   4. If the ZRAN_INFLATE_CLEAR_READBUF_OFFSETS flag is active, the read
+ *   3. If the ZRAN_INFLATE_CLEAR_READBUF_OFFSETS flag is active, the read
  *      buffer offset (index->readbuf_offset) and length (index->readbuf_end)
  *      fields are both set to 0. Otherwise, the function assumes that the
  *      current offset/length values are valid.
+
+ *   4. Initialises the z_stream struct, if ZRAN_INFLATE_INIT_Z_STREAM is
+ *      active. Otherwise, the function assumes that the z_stream struct is
+ *      already initialised and ready to be used.
  *
  *   5. Read some compressed data from the file into the read buffer as needed.
  *
  *   6. Pass that data to the zlib inflate function, and store the resulting
- *      uncompressed data in the provided data buffer.
+ *      uncompressed data in the provided data buffer. If the end of a GZIP
+ *      stream is reached for the first time, it is validated against the
+ *      CRC/file size stored in the GZIP footer (unless ZRAN_SKIP_CRC_CHECK
+ *      is active).
  *
  *   7. Repeat steps 5 and 6 until one of the following is true:
  *
@@ -449,30 +548,39 @@ uint32_t ZRAN_INFLATE_STOP_AT_BLOCK         = 64;
  *
  *   - ZRAN_INFLATE_EOF:            The end of file has been reached.
  *
+ *   - ZRAN_INFLATE_CRC_ERROR:      The CRC or uncompressed data size in the
+ *                                  GZIP footer does not match the CRC/size
+ *                                  that was calculated.
+ *
  *   - ZRAN_INFLATE_ERROR:          A critical error has occurred.
  */
 static int _zran_inflate(
-    zran_index_t *index,          /* Pointer to the index. */
+    zran_index_t *index,            /* Pointer to the index. */
 
-    z_stream     *strm,           /* Pointer to a z_stream struct. */
+    z_stream     *strm,             /* Pointer to a z_stream struct. */
 
-    uint64_t      offset,         /* Compressed data offset to start inflation
-                                     from. */
+    uint64_t      offset,           /* Compressed data offset to start
+                                       inflation from. */
 
-    uint16_t      flags,          /* Control flags. */
+    uint16_t      flags,            /* Control flags. */
 
-    uint32_t     *total_consumed, /* Pointer which is updated to contain the
-                                     total number of bytes that were read
-                                     from the input file. */
+    uint32_t     *total_consumed,   /* Pointer which is updated to contain the
+                                       total number of bytes that were read
+                                       from the input file. */
 
-    uint32_t     *total_output,   /* Pointer which is updated to contain the
-                                     total number of bytes that were inflated,
-                                     and stored in data. */
+    uint32_t     *total_output,     /* Pointer which is updated to contain the
+                                       total number of bytes that were
+                                       inflated, and stored in data. */
 
-    uint32_t      len,            /* Maximum number of bytes to inflate. May
-                                     be 0. */
+    uint32_t      len,              /* Maximum number of bytes to inflate. May
+                                       be 0. */
 
-    uint8_t      *data            /* Place to store the inflated bytes. */
+    uint8_t      *data,             /* Place to store the inflated bytes. */
+
+    int           add_stream_points /* Add index points at the beginning of
+                                       every gzip stream, including the first
+                                       one at the beginning of the input file
+                                     */
 );
 
 
@@ -497,17 +605,34 @@ int zran_init(zran_index_t *index,
     if (readbuf_size == 0) readbuf_size = 16384;
 
     /*
-     * The zlib manual specifies that a window size of 32KB is 'always enough'
-     * to initialise inflation/deflation with a set dictionary. Less than
-     * that is not guaranteed to be enough.
-    */
+     * The zlib manual specifies that a window
+     * size of 32KB is 'always enough' to
+     * initialise inflation/deflation with a
+     * set dictionary. Less than that is not
+     * guaranteed to be enough.
+     */
     if (window_size < 32768)
         goto fail;
 
     /*
-     * window_size bytes of uncompressed data are stored with each seek point
-     * in the index. So it's a bit silly to have the distance between
-     * consecutive points less than the window size.
+     * Small read-buffers make code complicated.
+     * The absolute minimum we need is enough to
+     * store a GZIP footer, null padding bytes at
+     * the end of a stream, and the subsequent
+     * GZIP header. There are no bounds on the
+     * number of padding bytes, or the size of a
+     * GZIP header, so this constraint is
+     * arbitrary (but should be good enough).
+     */
+    if (readbuf_size < 128)
+        goto fail;
+
+    /*
+     * window_size bytes of uncompressed data are
+     * stored with each seek point in the index.
+     * So it's a bit silly to have the distance
+     * between consecutive points less than the
+     * window size.
      */
     if (spacing <= window_size)
       goto fail;
@@ -557,6 +682,10 @@ int zran_init(zran_index_t *index,
     index->uncmp_seek_offset    = 0;
     index->inflate_cmp_offset   = 0;
     index->inflate_uncmp_offset = 0;
+    index->validating           = 0;
+    index->last_stream_ended    = 0;
+    index->stream_size          = 0;
+    index->stream_crc32         = 0;
     index->list                 = point_list;
 
     return 0;
@@ -687,7 +816,7 @@ int zran_build_index(zran_index_t *index, uint64_t from, uint64_t until)
 {
 
     if (_zran_invalidate_index(index, from) != 0)
-        return -1;
+        return ZRAN_BUILD_INDEX_FAIL;
 
     if (until == 0)
       until = index->compressed_size;
@@ -725,28 +854,20 @@ int _zran_get_point_at(
         offset >= index->uncompressed_size)
         goto eof;
 
-    zran_log("_zran_get_point_at(%llu, c=%u)\n", offset, compressed);
+    if (index->npoints == 0)
+        goto not_covered;
 
-    /*
-     * Figure out how much of the compressed and
-     * uncompressed data the index currently covers.
-     *
-     * No points - no coverage.
-     */
-    if (index->npoints == 0) {
-        cmp_max   = 0;
-        uncmp_max = 0;
-    }
+    zran_log("_zran_get_point_at(%llu, c=%u)\n", offset, compressed);
 
     /*
-     * Otherwise the offsets of the
-     * last point in the index.
+     * Figure out how much of the compressed
+     * and uncompressed data the index currently
+     * covers -  the offsets of the last point
+     * in the index.
      */
-    else {
-        last      = &(index->list[index->npoints - 1]);
-        uncmp_max = last->uncmp_offset;
-        cmp_max   = last->cmp_offset;
-    }
+    last      = &(index->list[index->npoints - 1]);
+    uncmp_max = last->uncmp_offset;
+    cmp_max   = last->cmp_offset;
 
     if ( compressed && offset > cmp_max)   goto not_covered;
     if (!compressed && offset > uncmp_max) goto not_covered;
@@ -814,10 +935,6 @@ int _zran_get_point_with_expand(zran_index_t  *index,
              compressed,
              index->flags & ZRAN_AUTO_BUILD);
 
-    if ((index->flags & ZRAN_AUTO_BUILD) == 0) {
-        return _zran_get_point_at(index, offset, compressed, point);
-    }
-
     /*
      * See if there is an index point that
      * covers the specified offset. If there's
@@ -826,6 +943,14 @@ int _zran_get_point_with_expand(zran_index_t  *index,
      */
     result = _zran_get_point_at(index, offset, compressed, point);
 
+    /*
+     * Don't expand the index if
+     * auto_build is not active
+     */
+    if ((index->flags & ZRAN_AUTO_BUILD) == 0) {
+        return result;
+    }
+
     while (result == ZRAN_GET_POINT_NOT_COVERED) {
 
         /*
@@ -852,12 +977,15 @@ int _zran_get_point_with_expand(zran_index_t  *index,
         if (expand <= limit)
             expand = limit + 10;
 
+        zran_log("Estimated mapping from uncompresseed offset "
+                 "%lu into compressed data: %lu\n", offset, expand);
+
         /*
          * Expand the index
          */
-        if (_zran_expand_index(index, expand) != 0) {
-            goto fail;
-        }
+        result = _zran_expand_index(index, expand);
+        if      (result == ZRAN_EXPAND_INDEX_CRC_ERROR) { goto crcerror; }
+        else if (result != 0)                           { goto fail; }
 
         /*
          * Index has been expanded, so
@@ -882,6 +1010,8 @@ int _zran_get_point_with_expand(zran_index_t  *index,
 
     return result;
 
+crcerror:
+    return ZRAN_GET_POINT_CRC_ERROR;
 fail:
     return ZRAN_GET_POINT_FAIL;
 }
@@ -981,12 +1111,13 @@ int _zran_add_point(zran_index_t  *index,
     /*
      * Allocate memory to store the
      * uncompressed data (the "window")
-     * associated with this point. The
-     * first index point (where uncmp_offset == 0)
-     * has no data associated with it for
-     * obvious reasons.
+     * associated with this point. Index
+     * points corresponding to the beginning
+     * of a gzip stream (including at start
+     * of file) do not have any window data
+     * associated with them.
      */
-    if (uncmp_offset == 0) {
+    if (data == NULL) {
         point_data = NULL;
     }
     else {
@@ -1010,7 +1141,7 @@ int _zran_add_point(zran_index_t  *index,
      * window from the beginning of data. Does
      * that make sense?
      */
-    if (uncmp_offset > 0) {
+    if (data != NULL) {
         if (data_offset >= index->window_size) {
 
             memcpy(point_data,
@@ -1055,25 +1186,39 @@ int _zran_add_point(zran_index_t  *index,
 /* Initialise the given z_stream struct for decompression/inflation. */
 int _zran_init_zlib_inflate(zran_index_t *index,
                             z_stream     *stream,
-                            zran_point_t *point)
-{
+                            zran_point_t *point) {
+
+    int           ret;
+    int           windowBits;
+    int64_t       seek_loc;
+    unsigned long bytes_read;
+
+    bytes_read     = stream->avail_in;
+    windowBits     = index->log_window_size;
+    stream->zalloc = Z_NULL;
+    stream->zfree  = Z_NULL;
+    stream->opaque = Z_NULL;
 
-    int     ret;
-    int     windowBits;
-    int64_t seek_loc;
+    /*
+     * If we're starting from the the current location in
+     * the compressed data, we assume that there is a gzip
+     * header present. Initialise inflation, then read
+     * past the header.
+
+     * Below, we will re-initialise for raw inflation.
+     */
+    if (point == NULL) {
 
-    windowBits        = index->log_window_size;
-    stream->zalloc    = Z_NULL;
-    stream->zfree     = Z_NULL;
-    stream->opaque    = Z_NULL;
-    stream->avail_in  = 0;
-    stream->avail_out = 0;
-    stream->next_in   = Z_NULL;
+        zran_log("_zran_init_zlib_inflate from current "
+                 "seek location (expecting GZIP header)\n");
+        if (inflateInit2(stream, windowBits + 32) != Z_OK) { goto fail; }
+        if (inflate(stream, Z_BLOCK)              != Z_OK) { goto fail; }
+        if (inflateEnd(stream)                    != Z_OK) { goto fail; }
+    }
 
     /*
-     * Seek to the required location in the compressed
-     * data stream. If the provided index point is NULL,
-     * we start from the beginning of the file.
+     * If starting from an index point, seek to the
+     * required location in the compressed data stream.
      *
      * The compressed offset for index points correspond
      * to the first full byte of compressed data. So if
@@ -1081,47 +1226,42 @@ int _zran_init_zlib_inflate(zran_index_t *index,
      * need to seek to the previous byte, and tell zlib
      * about it (via the inflatePrime call below).
      */
-    if (point == NULL) seek_loc = 0;
-    else               seek_loc = point->cmp_offset - (point->bits > 0);
-
-    if (fseek_(index->fd, index->f, seek_loc, SEEK_SET) != 0)
-        goto fail;
+    else {
+        seek_loc = point->cmp_offset - (point->bits > 0);
 
-    /*
-     * If we're starting from the beginning
-     * of the file, we tell inflateInit2 to
-     * expect a file header
-     */
-    if (point == NULL) {
+        zran_log("_zran_init_zlib_inflate from index point (%li, %li, %li)\n",
+                 seek_loc,
+                 point->cmp_offset,
+                 point->uncmp_offset);
 
-        zran_log("zlib_init_zlib_inflate(0, n/a, n/a, %u + 32)\n", windowBits);
-        if (inflateInit2(stream, windowBits + 32) != Z_OK) {
+        if (fseek_(index->fd, index->f, seek_loc, SEEK_SET) != 0) {
             goto fail;
         }
     }
 
     /*
-     * Otherwise, we configure for raw inflation,
-     * and initialise the inflation dictionary
-     * from the uncompressed data associated with
-     * the index point.
+     * Now initialise for raw inflation. This tells zlib
+     * not to expect any GZIP headers, and not to read
+     * the GZIP footer (as we do our own CRC validation
+     * in _zran_inflate).
      */
-    else {
 
-        zran_log("_zran_init_zlib_inflate(%lld, %llu, %llu, -%u)\n",
-                 seek_loc,
-                 point->cmp_offset,
-                 point->uncmp_offset,
-                 windowBits);
+    if (inflateInit2(stream, -windowBits) != Z_OK) {
+        goto fail;
+    }
 
-        if (inflateInit2(stream, -windowBits) != Z_OK) {
-            goto fail;
-        }
+    /*
+     * If starting from an index point, initialise
+     * the inflation dictionary from the uncompressed
+     * data associated with the index point.
+     */
+    if (point != NULL && point->data != NULL) {
 
-        /* The starting index point is not
-         * byte-aligned, so we'll insert
-         * the initial bits into the inflate
-         * stream using inflatePrime
+        /*
+         * The starting index point is not byte-aligned,
+         * so we'll insert the initial bits into the
+         * inflate stream using inflatePrime (above,
+         * we seeked one byte back to accommodate this).
          */
         if (point->bits > 0) {
 
@@ -1140,21 +1280,150 @@ int _zran_init_zlib_inflate(zran_index_t *index,
          * Initialise the inflate stream
          * with the index point data.
          */
-        if (point->data != NULL) {
-            if (inflateSetDictionary(stream,
-                                     point->data,
-                                     index->window_size) != Z_OK)
-                goto fail;
-        }
+        if (inflateSetDictionary(stream,
+                                 point->data,
+                                 index->window_size) != Z_OK)
+            goto fail;
     }
 
-    return 0;
+    /*
+     * Reset CRC/size validation counters when
+     * we start reading a new gzip stream
+     */
+    index->validating   = (point == NULL);
+    index->stream_size  = 0;
+    index->stream_crc32 = 0;
+
+    zran_log("_zran_zlib_init_inflate: initialised, read %i bytes\n",
+             bytes_read - stream->avail_in);
+
+    /*
+     * Return the number of bytes of compressed
+     * data, if any that were read over
+     */
+    return bytes_read - stream->avail_in;
 
 fail:
     return -1;
 }
 
 
+/*
+ * Read data from the GZIP file, and copy it into the read buffer for
+ * decompression.
+ */
+static int _zran_read_data_from_file(zran_index_t *index,
+                                     z_stream     *stream,
+                                     uint64_t      cmp_offset,
+                                     uint64_t      uncmp_offset,
+                                     uint32_t      need_atleast) {
+
+    size_t f_ret;
+
+    if (stream->avail_in >= need_atleast) {
+        return 0;
+    }
+
+    /*
+     * If there are any unprocessed bytes
+     * left over, put them at the beginning
+     * of the read buffer.
+     *
+     * TODO: In times gone by, we would only
+     * attempt to read data (and therefore
+     * rotate memory here) when the read
+     * buffer was empty. But now, to keep
+     * the code in _zran_inflate clean-ish,
+     * we do this repeatedly, even when we
+     * are at EOF, to ensure that there is
+     * enough data to validate one stream,
+     * and find the next. We could improve
+     * things here, by only rotating memory
+     * here if needed.
+     */
+    if (stream->avail_in > 0) {
+        memmove(index->readbuf, stream->next_in, stream->avail_in);
+    }
+
+    zran_log("Reading from file %llu [== %llu?] "
+             "[into readbuf offset %u]\n",
+             ftell_(index->fd, index->f),
+             cmp_offset + stream->avail_in,
+             stream->avail_in);
+
+    /*
+     * Read a block of compressed data
+     * (offsetting past any left over
+     * bytes that we may have copied to
+     * the beginning of the read buffer
+     * above).
+     */
+    f_ret = fread_(index->readbuf + stream->avail_in,
+                   1,
+                   index->readbuf_size - stream->avail_in,
+                   index->fd,
+                   index->f);
+
+    if (ferror_(index->fd, index->f)) {
+        goto fail;
+    }
+
+    /*
+     * No bytes left to read, and there are
+     * only 8 bytes left to process (size of
+     * gzip footer) - we've reached EOF.
+     */
+    if (f_ret == 0 && stream->avail_in <= 8) {
+        if (feof_(index->fd, index->f, f_ret)) {
+
+            zran_log("End of file, stopping inflation\n");
+
+            /*
+             * we have uncompressed everything,
+             * so we now know its size.
+             */
+            if (index->uncompressed_size == 0) {
+                zran_log("Updating uncompressed data "
+                         "size: %llu\n", uncmp_offset);
+                index->uncompressed_size = uncmp_offset;
+            }
+            goto eof;
+        }
+        /*
+         * Or something went wrong (this
+         * should never happen if ferror
+         * does the right thing).
+         */
+        else {
+            goto fail;
+        }
+    }
+
+    zran_log("Read %lu bytes from file [c=%llu, u=%llu] "
+             "[%02x %02x %02x %02x ...]\n",
+             f_ret, cmp_offset, uncmp_offset,
+             index->readbuf[stream->avail_in],
+             index->readbuf[stream->avail_in + 1],
+             index->readbuf[stream->avail_in + 2],
+             index->readbuf[stream->avail_in + 3]);
+
+    /*
+     * Tell zlib about the block
+     * of compressed data that we
+     * just read in.
+     */
+    index->readbuf_end = f_ret + stream->avail_in;
+    stream->avail_in  += f_ret;
+    stream->next_in    = index->readbuf;
+
+    return 0;
+eof:
+    return ZRAN_READ_DATA_EOF;
+fail:
+    return ZRAN_READ_DATA_ERROR;
+}
+
+
 /*
  * Identify the location of the next compressed stream (if the file
  * contains concatenated streams).
@@ -1163,50 +1432,58 @@ int _zran_find_next_stream(zran_index_t *index,
                            z_stream     *stream,
                            int          *offset) {
 
+    int ret;
+    int found;
+
     /*
      * Search for the beginning of
      * the next stream. GZIP files
      * start with 0x1f8b.
      */
-    int found = 0;
-    *offset   = 0;
+    found = 0;
+
+    zran_log("Searching for a new stream [%u]\n", stream->avail_in);
 
-    while (stream->avail_in >= 2) {
+    while (stream->avail_in > 0) {
 
-        if (stream->next_in[0] == 0x1f &&
+        if (stream->avail_in   >= 2    &&
+            stream->next_in[0] == 0x1f &&
             stream->next_in[1] == 0x8b) {
             found = 1;
             break;
         }
 
-        *offset          += 2;
-        stream->next_in  += 2;
-        stream->avail_in -= 2;
+        *offset          += 1;
+        stream->next_in  += 1;
+        stream->avail_in -= 1;
     }
 
     /*
      * No header found for
      * the next stream.
      */
-    if (found == 0)
+    if (found == 0) {
+        zran_log("Could not find another stream [%u]\n", stream->avail_in);
         goto not_found;
+    }
 
-    zran_log("New stream found, re-initialising "
-             "inflation (offset: %u)\n", offset);
+    zran_log("New stream found, re-initialising inflation\n");
 
     /*
      * Re-configure for inflation
      * from the new stream.
      */
-    if (inflateEnd(stream) != Z_OK)
+    if (inflateEnd(stream) != Z_OK) {
         goto fail;
+    }
 
-    stream->zalloc = Z_NULL;
-    stream->zfree  = Z_NULL;
-    stream->opaque = Z_NULL;
+    ret = _zran_init_zlib_inflate(index, stream, NULL);
 
-    if (inflateInit2(stream, index->log_window_size + 32) != Z_OK)
+    if (ret < 0) {
         goto fail;
+    }
+
+    *offset += ret;
 
     return 0;
 
@@ -1218,6 +1495,52 @@ int _zran_find_next_stream(zran_index_t *index,
 }
 
 
+/* Validate the CRC32 and size of a GZIP stream. */
+static int _zran_validate_stream(zran_index_t *index,
+                                 z_stream     *stream,
+                                 int          *offset) {
+
+    uint32_t crc;
+    uint32_t size;
+
+    /* CRC validation is disabled */
+    if (index->flags & ZRAN_SKIP_CRC_CHECK) {
+        return 0;
+    }
+
+    /*
+     * A gzip stream should end with an 8 byte footer,
+     * which contains the CRC32 of the uncompressed
+     * data, and the uncompressed size modulo 2^32.
+     */
+    if (stream->avail_in < 8) {
+        return ZRAN_VALIDATE_STREAM_ERROR;
+    }
+
+    crc  = ((stream->next_in[0] << 0)  +
+            (stream->next_in[1] << 8)  +
+            (stream->next_in[2] << 16) +
+            (stream->next_in[3] << 24));
+    size = ((stream->next_in[4] << 0)  +
+            (stream->next_in[5] << 8)  +
+            (stream->next_in[6] << 16) +
+            (stream->next_in[7] << 24));
+
+    zran_log("Validating CRC32 and size [%8x == %8x, %u == %u]\n",
+             crc, index->stream_crc32, size, index->stream_size);
+
+    stream->avail_in -= 8;
+    stream->next_in  += 8;
+    *offset          += 8;
+
+    if (index->stream_crc32 != crc || index->stream_size != size) {
+        return ZRAN_VALIDATE_STREAM_INVALID;
+    }
+
+    return 0;
+}
+
+
 /* The workhorse. Inflate/decompress data from the file. */
 static int _zran_inflate(zran_index_t *index,
                          z_stream     *strm,
@@ -1226,21 +1549,21 @@ static int _zran_inflate(zran_index_t *index,
                          uint32_t     *total_consumed,
                          uint32_t     *total_output,
                          uint32_t      len,
-                         uint8_t      *data) {
+                         uint8_t      *data,
+                         int           add_stream_points) {
 
     /*
-     * Used to store and check return
-     * values. f_ret is for fread,
      * z_ret is for zlib/zran functions.
      * off is for storing offset of new
-     * stream (from _zran_find_next_stream).
-     * return_val is the return value for
-     * this function.
+     * stream (from _zran_validate_stream
+     * and _zran_find_next_stream).
+     * return_val/error_return_val is
+     * the return value for this function.
      */
-    size_t f_ret;
-    int    z_ret;
-    int    off;
-    int    return_val = ZRAN_INFLATE_OK;
+    int z_ret;
+    int off;
+    int return_val       = ZRAN_INFLATE_OK;
+    int error_return_val = ZRAN_INFLATE_ERROR;
 
     /*
      * Offsets into the compressed
@@ -1253,6 +1576,13 @@ static int _zran_inflate(zran_index_t *index,
     uint32_t _total_consumed = 0;
     uint32_t _total_output   = 0;
 
+    /*
+     * Number of bytes input/decompressed
+     * during a single call to zlib:inflate.
+     */
+    uint32_t bytes_consumed = 0;
+    uint32_t bytes_output   = 0;
+
     /*
      * Index point to start from
      * (if ZRAN_INFLATE_USE_OFFSET
@@ -1260,6 +1590,14 @@ static int _zran_inflate(zran_index_t *index,
      */
     zran_point_t *start = NULL;
 
+    /*
+     * Set all zstream_t fields to 0
+     * if we are initialising.
+     */
+    if (inflate_init_stream(flags)) {
+        memset(strm, 0, sizeof(z_stream));
+    }
+
     /*
      * If ZRAN_INFLATE_INIT_READBUF is not set,
      * make sure that a read buffer exists.
@@ -1351,27 +1689,11 @@ static int _zran_inflate(zran_index_t *index,
         uncmp_offset = index->inflate_uncmp_offset;
     }
 
-    zran_log("initialising to inflate from c=%llu, u=%llu\n",
+    zran_log("initialising to inflate from "
+             "cmp_offset=%llu, uncmp_offset=%llu\n",
              cmp_offset,
              uncmp_offset);
 
-    /*
-     * If ZRAN_INFLATE_INIT_Z_STREAM is active,
-     * initialise the zlib struct for inflation.
-     * The _zran_init_zlib_inflate function
-     * seeks to the correct location in the file
-     * for us.
-     *
-     * If ZRAN_INFLATE_INIT_Z_STREAM is not
-     * active, we assume that the file is
-     * already at the correct spot.
-     */
-    if (inflate_init_stream(flags)) {
-        if (_zran_init_zlib_inflate(index, strm, start) != 0) {
-            goto fail;
-        }
-    }
-
     /*
      * If ZRAN_INFLATE_INIT_READBUF,
      * allocate memory for reading
@@ -1400,7 +1722,7 @@ static int _zran_inflate(zran_index_t *index,
      * Otherwise, assume that there is already
      * some input (compressed) data in the
      * readbuf, and that index->readbuf_offset
-     * and index->readbuf_end were sert on a
+     * and index->readbuf_end were set on a
      * prior call.
      *
      *    - readbuf_offset tells us where in
@@ -1421,138 +1743,103 @@ static int _zran_inflate(zran_index_t *index,
     strm->next_out  = data;
 
     /*
-     * Keep going until we run out of space.
+     * If ZRAN_INFLATE_INIT_Z_STREAM is active,
+     * initialise the zlib struct for inflation.
+     *
+     * If ZRAN_INFLATE_INIT_Z_STREAM is not
+     * active, we assume that the inflation is
+     * already initialised.
      */
-    while (strm->avail_out > 0) {
+    if (inflate_init_stream(flags)) {
 
         /*
-         * We need to read in more data. We read in more
-         * data when strm->avail_in < 2, because a GZIP
-         * header is 2 bytes long, and when searching for
-         * the next stream in a sequence of concatenated
-         * streams, the _zran_find_next_stream function
-         * might leave a byte in the input without
-         * finding a new stream.
+         * No index point - we need to start reading
+         * from the beginning of the input file. If
+         * starting from an index point, the
+         * _zran_init_zlib_inflate function will seek
+         * to the correct location in the file for us.
          */
-        if (strm->avail_in < 2) {
-
-            /*
-             * If there are any unprocessed bytes
-             * left over, put them at the beginning
-             * of the read buffer
-             */
-            if (strm->avail_in > 0) {
-                memcpy(index->readbuf, strm->next_in, strm->avail_in);
+        if (start == NULL) {
+            if (fseek_(index->fd, index->f, 0, SEEK_SET) != 0) {
+                goto fail;
             }
 
-            zran_log("Reading from file %llu [== %llu?] "
-                     "[into readbuf offset %u]\n",
-                     ftell_(index->fd, index->f), cmp_offset, strm->avail_in);
-
             /*
-             * Read a block of compressed data
-             * (offsetting past any left over
-             * bytes that we may have copied to
-             * the beginning of the read buffer
-             * above).
+             * In this situation, _zran_init_zlib_inflate
+             * is going to expect a GZIP header, so make
+             * sure we have some data for it to look at.
              */
-            f_ret = fread_(index->readbuf + strm->avail_in,
-                          1,
-                          index->readbuf_size - strm->avail_in,
-                          index->fd,
-                          index->f);
-
-            if (ferror_(index->fd, index->f)) {
+            if (_zran_read_data_from_file(index,
+                                          strm,
+                                          cmp_offset,
+                                          uncmp_offset,
+                                          index->readbuf_size) != 0) {
                 goto fail;
             }
+        }
 
-            /*
-             * No bytes read - we've reached EOF
-             */
-            if (f_ret == 0) {
-                if (feof_(index->fd, index->f, f_ret)) {
-                    return_val = ZRAN_INFLATE_EOF;
-                    break;
-                }
-                /*
-                 * Or something went wrong (this
-                 * should never happen if ferror
-                 * does the right thing).
-                 */
-                else {
-                    goto fail;
-                }
+        /*
+         * If init_zlib_inflate skips over any input data
+         * (e.g. gzip header), it returns the number of
+         * bytes tyhat were read
+         */
+        z_ret = _zran_init_zlib_inflate(index, strm, start);
+        if (z_ret < 0) {
+            goto fail;
+        }
+        cmp_offset      += z_ret;
+        _total_consumed += z_ret;
+
+        if (start == NULL && add_stream_points) {
+            if (_zran_add_point(index, 0, cmp_offset, 0, 0, 0, NULL) != 0) {
+                goto fail;
             }
+        }
+    }
 
-            zran_log("Read %lu bytes from file [c=%llu, u=%llu] "
-                     "[%02x %02x %02x %02x ...]\n",
-                     f_ret, cmp_offset, uncmp_offset,
-                     index->readbuf[strm->avail_in],
-                     index->readbuf[strm->avail_in + 1],
-                     index->readbuf[strm->avail_in + 2],
-                     index->readbuf[strm->avail_in + 3]);
+    /*
+     * Keep going until we run out of space.
+     */
+    while (strm->avail_out > 0) {
 
-            /*
-             * Tell zlib about the block
-             * of compressed data that we
-             * just read in.
-             */
-            index->readbuf_end = f_ret + strm->avail_in;
-            strm->avail_in    += f_ret;
-            strm->next_in      = index->readbuf;
+        /*
+         * Make sure the input buffer contains
+         * some data to be decompressed.
+         */
+        z_ret = _zran_read_data_from_file(index,
+                                          strm,
+                                          cmp_offset,
+                                          uncmp_offset,
+                                          1);
+
+        /* EOF - there is no more data left to read */
+        if (z_ret == ZRAN_READ_DATA_EOF) {
+            return_val = ZRAN_INFLATE_EOF;
+            break;
+        }
+        else if (z_ret != 0) {
+            goto fail;
         }
 
         /*
-         * Decompress the block until it is
-         * gone (or we've read enough bytes)
+         * Decompress data until there's no data
+         * left, or we've read enough bytes
          */
         z_ret = Z_OK;
         while (strm->avail_in > 0) {
 
             /*
-             * Re-initialise inflation if we have
-             * hit a new compressed stream.
-             */
-            if (z_ret == Z_STREAM_END) {
-
-                zran_log("End of stream - searching for another stream\n");
-
-                z_ret = _zran_find_next_stream(index, strm, &off);
-
-                /*
-                 * _zran_find_next_stream will skip over
-                 * bytes in the input data while searching
-                 * for the next stream (e.g. the CRC32
-                 * and ISIZE fields at the end of a gzip
-                 * stream). It stores the number of skipped
-                 * bytes in the offset parameter.
-                 */
-                cmp_offset      += off;
-                _total_consumed += off;
-
-                /*
-                 * If _zran_find_next_stream can't find
-                 * a new stream, we are either out of
-                 * compressed input data, or at eof. In
-                 * either case, break and let the outer
-                 * loop deal with it.
-                 */
-                if      (z_ret == ZRAN_FIND_STREAM_NOT_FOUND) break;
-                else if (z_ret != 0)                          goto fail;
-            }
-
-            /*
-             * Optimistically update offsets -
-             * we will adjust them after the
-             * inflate call.
+             * Initialise counters to calculate
+             * how many bytes are input/output
+             * during this call to inflate.
              */
-            cmp_offset      += strm->avail_in;
-            uncmp_offset    += strm->avail_out;
-            _total_consumed += strm->avail_in;
-            _total_output   += strm->avail_out;
+            bytes_consumed = strm->avail_in;
+            bytes_output   = strm->avail_out;
 
-            zran_log("Before inflate - avail_in=%u, avail_out=%u\n",
-                     strm->avail_in, strm->avail_out);
+            zran_log("Before inflate - avail_in=%u, avail_out=%u, "
+                     "cmp_offset=%lu, uncmp_offset=%lu\n",
+                     strm->avail_in, strm->avail_out,
+                     cmp_offset, uncmp_offset);
 
             /*
              * Inflate the block - the decompressed
@@ -1565,20 +1852,29 @@ static int _zran_inflate(zran_index_t *index,
              * inflate stops when it comes to the end of a
              * stream, or it runs out of input or output.
              */
-            if (inflate_stop_at_block(flags)) z_ret = inflate(strm, Z_BLOCK);
-            else                              z_ret = inflate(strm, Z_NO_FLUSH);
-
-            zran_log("After inflate - avail_in=%u, avail_out=%u\n",
-                     strm->avail_in, strm->avail_out);
+            if (inflate_stop_at_block(flags)) {
+                z_ret = inflate(strm, Z_BLOCK);
+            }
+            else {
+                z_ret = inflate(strm, Z_NO_FLUSH);
+            }
 
             /*
              * Adjust our offsets according to what
              * was actually consumed/decompressed.
              */
-            cmp_offset      -= strm->avail_in;
-            uncmp_offset    -= strm->avail_out;
-            _total_consumed -= strm->avail_in;
-            _total_output   -= strm->avail_out;
+            bytes_consumed   = bytes_consumed - strm->avail_in;
+            bytes_output     = bytes_output   - strm->avail_out;
+            cmp_offset      += bytes_consumed;
+            _total_consumed += bytes_consumed;
+            uncmp_offset    += bytes_output;
+            _total_output   += bytes_output;
+
+            zran_log("After inflate - avail_in=%u, avail_out=%u, "
+                     "cmp_offset=%lu, uncmp_offset=%lu\n",
+                     strm->avail_in, strm->avail_out,
+                     cmp_offset, uncmp_offset);
+
 
             /*
              * Now we need to figure out what just happened.
@@ -1612,6 +1908,22 @@ static int _zran_inflate(zran_index_t *index,
                 goto fail;
             }
 
+            /*
+             * If we have not yet validated the current
+             * GZIP stream, update its size and crc so
+             * we can validate them against the size
+             * recorded in the stream footer when we
+             * get to it.
+             */
+            if ((uncmp_offset > index->last_stream_ended) &&
+                index->validating                         &&
+                !(index->flags & ZRAN_SKIP_CRC_CHECK)) {
+                index->stream_size +=       bytes_output;
+                index->stream_crc32 = crc32(index->stream_crc32,
+                                            strm->next_out - bytes_output,
+                                            bytes_output);
+            }
+
             /*
              * End of a block? If INFLATE_STOP_AT_BLOCK
              * is active, we want to stop at a compression
@@ -1632,70 +1944,169 @@ static int _zran_inflate(zran_index_t *index,
             }
 
             /*
-             * We've run out of space to
-             * store decompressed data
+             * We've found the end of file, or end of one
+             * gzip stream. Validate the uncompressed
+             * data (size/ CRC) against the gzip footer.
+             * Then search for a new stream and, if we
+             * find one, re-initialise inflation
              */
-            if (strm->avail_out == 0) {
+            if (z_ret == Z_STREAM_END) {
 
-                zran_log("Output buffer full - stopping inflation\n");
+                zran_log("End of gzip stream [%u]\n", strm->avail_in);
 
                 /*
-                 * We return OUTPUT_FULL if we haven't
-                 * decompressed the requested number of
-                 * bytes, or ZRAN_INFLATE_STOP_AT_BLOCK
-                 * is active and we haven't yet found a
-                 * block.
+                 * Make sure we have data in the input buffer
+                 * to read and vaildate the gzip footer and,
+                 * in case we are reading concatentaed
+                 * streams, to search for the next stream and
+                 * read its header.
+                 *
+                 * There is no way of knowing how much data we
+                 * need to read in here - there is no upper
+                 * bound on the amount of null padding bytes
+                 * that may be present in between, or at the
+                 * end of, a stream, and there is no upper
+                 * bound on the size of a gzip header.
+
+                 * So a critical assumption is made here, that
+                 * the size of the read buffer is large enough
+                 * to encompass all of:
+                 *
+                 *   - the footer of a gzip stream,
+                 *   - null padding after the end of a gzip
+                 *     stream, and
+                 *   - the header of the next gzip stream
+                 *
+                 * This assumption could be removed by changing
+                 * the way that data is loaded and buffered
+                 * from the file - e.g. we could load bytes in
+                 * one-by-one, skipping over null bytes, and
+                 * then parse the gzip header ourselves.  But
+                 * this is far too much work for what is a very
+                 * edge-casey scenario.
                  */
-                if (inflate_stop_at_block(flags) || _total_output < len) {
-                    return_val = ZRAN_INFLATE_OUTPUT_FULL;
+                z_ret = _zran_read_data_from_file(index,
+                                                  strm,
+                                                  cmp_offset,
+                                                  uncmp_offset,
+                                                  index->readbuf_size);
+                if (!((z_ret == 0) || (z_ret == ZRAN_READ_DATA_EOF))) {
+                    goto fail;
                 }
 
-                break;
-            }
+                /*
+                 * If there is no more data, the footer is
+                 * missing, so the data must be corrupt.
+                 */
+                if (strm->avail_in < 8) {
+                    goto fail;
+                }
 
-            /*
-             * End of file. The GZIP file
-             * footer takes up 8 bytes, which
-             * do not get processed by the
-             * inflate function.
-             *
-             * We use ftell rather than feof,
-             * as the EOF indicator only gets
-             * set on attempts to read past
-             * the end of a file, and this
-             * won't happen when the file
-             * size is an exact multiple of
-             # the read buffer size.
-             */
-            if ((uint64_t) ftell_(index->fd, index->f) >= index->compressed_size &&
-                strm->avail_in <= 8) {
+                /*
+                 * _validate_stream reads and checks in the
+                 * gzip stream footer (the CRC32 and ISIZE
+                 * fields at the end of a gzip stream), and
+                 * _find_next_stream will skip over any
+                 * remaining bytes (e.g. null padding bytes)
+                 * until eof, or a new stream is found.
+                 *
+                 * The number of bytes that were read/
+                 * skipped over are accumulated into off.
+                 */
+                off = 0;
+
+                /*
+                 * If we have not yet validated this stream,
+                 * check that the CRC and uncompressed size in
+                 * the footer match what we have calculated
+                 */
+                if (uncmp_offset > index->last_stream_ended &&
+                    index->validating) {
+                    z_ret = _zran_validate_stream(index, strm, &off);
+
+                    if (z_ret == ZRAN_VALIDATE_STREAM_INVALID) {
+                        error_return_val = ZRAN_INFLATE_CRC_ERROR;
+                        goto fail;
+                    }
+                    else if (z_ret != 0) {
+                        goto fail;
+                    }
+                    index->last_stream_ended = uncmp_offset;
+                    index->validating        = 0;
+                }
 
-                zran_log("End of file, stopping inflation\n");
+                /* Otherwise skip over the 8 byte GZIP footer */
+                else {
+                    strm->avail_in -= 8;
+                    strm->next_in  += 8;
+                    off            += 8;
+                }
+
+                /*
+                 * See if we have another concatenated gzip
+                 * stream.  If we run out of input data here,
+                 * bad things will happen. Refer to the long
+                 * comment regarding the input buffer, above.
+                 */
+                z_ret = _zran_find_next_stream(index, strm, &off);
 
-                return_val = ZRAN_INFLATE_EOF;
+                cmp_offset      += off;
+                _total_consumed += off;
 
                 /*
-                 * We now know how big the
-                 * uncompressed data is.
+                 * If _zran_find_next_stream can't find
+                 * a new stream, we are either out of
+                 * compressed input data, or at eof. In
+                 * either case, break and let the outer
+                 * loop deal with it.
                  */
-                if (index->uncompressed_size == 0) {
+                if (z_ret == ZRAN_FIND_STREAM_NOT_FOUND) {
+                    break;
+                }
+                else if (z_ret != 0) {
+                    goto fail;
+                }
 
-                    zran_log("Updating uncompressed data "
-                             "size: %llu\n", uncmp_offset);
-                    index->uncompressed_size = uncmp_offset;
+                if (add_stream_points) {
+                    if (_zran_add_point(index,
+                                        0,
+                                        cmp_offset,
+                                        uncmp_offset,
+                                        0,
+                                        0,
+                                        NULL) != 0) {
+                        goto fail;
+                    }
                 }
-                break;
             }
 
             /*
-             * Some of the code above has decided that
-             * it wants this _zran_inflate call to return.
+             * We've run out of space to store decompressed
+             * data - this is the responsibility of the caller,
+             * so bail out.
              */
-            if (return_val != ZRAN_INFLATE_OK) {
+            if (strm->avail_out == 0) {
+
+                zran_log("Output buffer full - stopping inflation\n");
+
+                /*
+                 * We return OUTPUT_FULL if we haven't
+                 * decompressed the requested number of
+                 * bytes, or ZRAN_INFLATE_STOP_AT_BLOCK
+                 * is active and we haven't yet found a
+                 * block.
+                 */
+                if (inflate_stop_at_block(flags) || _total_output < len) {
+                    return_val = ZRAN_INFLATE_OUTPUT_FULL;
+                }
                 break;
             }
         }
 
+        /*
+         * Some of the code above has decided that
+         * it wants this _zran_inflate call to return.
+         */
         if (return_val != ZRAN_INFLATE_OK) {
             break;
         }
@@ -1761,7 +2172,7 @@ static int _zran_inflate(zran_index_t *index,
         index->readbuf_end    = 0;
     }
 
-    return ZRAN_INFLATE_ERROR;
+    return error_return_val;
 }
 
 
@@ -1769,8 +2180,13 @@ static int _zran_inflate(zran_index_t *index,
  * Expands the index to encompass the
  * compressed offset specified by 'until'.
  */
-int _zran_expand_index(zran_index_t *index, uint64_t until)
-{
+int _zran_expand_index(zran_index_t *index, uint64_t until) {
+
+    /*
+     * Used to store return code when
+     * an error occurs.
+     */
+    int error_return_val = ZRAN_EXPAND_INDEX_FAIL;
 
     /*
      * Used to store and check return values
@@ -1957,12 +2373,22 @@ int _zran_expand_index(zran_index_t *index, uint64_t until)
                               &bytes_consumed,
                               &bytes_output,
                               data_size - data_offset,
-                              data      + data_offset);
+                              data      + data_offset,
+                              1);
 
         cmp_offset   += bytes_consumed;
         uncmp_offset += bytes_output;
         data_offset   = (data_offset + bytes_output) % data_size;
 
+        /*
+         * update the last created offset on every iteration,
+         * to catch any index points created by _zran_inflate
+         */
+        if (index->npoints > 0) {
+            last_created      = &index->list[index->npoints - 1];
+            last_uncmp_offset = last_created->uncmp_offset;
+        }
+
         /*
          * Has the output buffer been filled?
          * If so, we just continue - the
@@ -1981,28 +2407,29 @@ int _zran_expand_index(zran_index_t *index, uint64_t until)
          */
         else if (z_ret != ZRAN_INFLATE_EOF &&
                  z_ret != ZRAN_INFLATE_BLOCK_BOUNDARY) {
+            if (z_ret == ZRAN_INFLATE_CRC_ERROR) {
+                error_return_val = ZRAN_EXPAND_INDEX_CRC_ERROR;
+            }
             goto fail;
         }
 
         /*
-         * If we're at the beginning of the file
-         * (uncmp_offset == 0), or at the end of
-         * the file (z_ret == ZRAN_INFLATE_EOF),
-         * or at a compress block boundary,
-         * and index->spacing bytes have passed
-         * since the last index point that was
-         * created, we'll create a  new index
-         * point at this location.
+         * If we're at the end of the file (z_ret
+         * == ZRAN_INFLATE_EOF), or at a compress
+         * block boundary, and index->spacing bytes
+         * have passed since the last index point
+         * that was created, we'll create a new
+         * index point at this location.
+         *
+         * Note that the _zran_inflate function
+         * also adds index points at the beginning
+         * of the file, and at the beginning of all
+         * other gzip streams, in the case of
+         * concatenated streams (refer to its
+         * add_stream_points argument).
          */
         if (z_ret == ZRAN_INFLATE_EOF ||
-            uncmp_offset == 0         ||
             uncmp_offset - last_uncmp_offset >= index->spacing) {
-
-            // TODO If at start or EOF, you should
-            //      pass in  NULL for the window
-            //      data. You can then clean up
-            //      _zran_add_point a little bit.
-
             if (_zran_add_point(index,
                                 strm.data_type & 7,
                                 cmp_offset,
@@ -2012,14 +2439,14 @@ int _zran_expand_index(zran_index_t *index, uint64_t until)
                                 data) != 0) {
                 goto fail;
             }
-
             last_created      = &index->list[index->npoints - 1];
             last_uncmp_offset = uncmp_offset;
         }
 
         /* And if at EOF, we are done. */
-        if (z_ret == ZRAN_INFLATE_EOF)
+        if (z_ret == ZRAN_INFLATE_EOF) {
             break;
+        }
     }
 
     /*
@@ -2035,9 +2462,13 @@ int _zran_expand_index(zran_index_t *index, uint64_t until)
                           &bytes_consumed,
                           &bytes_output,
                           0,
-                          data);
+                          data,
+                          0);
 
     if (z_ret != ZRAN_INFLATE_OK && z_ret != ZRAN_INFLATE_EOF) {
+        if (z_ret == ZRAN_INFLATE_CRC_ERROR) {
+            error_return_val = ZRAN_EXPAND_INDEX_CRC_ERROR;
+        }
         goto fail;
     }
 
@@ -2054,12 +2485,11 @@ int _zran_expand_index(zran_index_t *index, uint64_t until)
              cmp_offset, last_created->cmp_offset);
 
     free(data);
-    return 0;
+    return ZRAN_EXPAND_INDEX_OK;
 
 fail:
     free(data);
-
-    return -1;
+    return error_return_val;
 }
 
 
@@ -2074,7 +2504,7 @@ int zran_seek(zran_index_t  *index,
 {
 
     int           result;
-    zran_point_t *seek_point;
+    zran_point_t *seek_point = NULL;
 
     zran_log("zran_seek(%lld, %i)\n", offset, whence);
 
@@ -2092,7 +2522,7 @@ int zran_seek(zran_index_t  *index,
      *
      * I am not currently taking into account
      * the overflow potential when converting
-     * from int64 to uint64.a
+     * from int64 to uint64.
      */
 
     /*
@@ -2117,24 +2547,40 @@ int zran_seek(zran_index_t  *index,
     }
 
     /*
-     * Get the index point that
-     * corresponds to this offset.
+     * We implicitly allow seek(0) - if
+     * not auto-building the index,
+     * seek(0) would otherwwise fail.
      */
-    result = _zran_get_point_with_expand(index, offset, 0, &seek_point);
+    if (offset == 0) {
+        index->uncmp_seek_offset = offset;
+    }
+    else {
 
-    if (result == ZRAN_GET_POINT_FAIL)        goto fail;
-    if (result == ZRAN_GET_POINT_NOT_COVERED) goto not_covered;
-    if (result == ZRAN_GET_POINT_EOF)         goto eof;
+        /*
+         * Get the index point that
+         * corresponds to this offset.
+         */
+        result = _zran_get_point_with_expand(index, offset, 0, &seek_point);
 
-    index->uncmp_seek_offset = offset;
-    offset                   = seek_point->cmp_offset;
+        if      (result == ZRAN_GET_POINT_EOF)         goto eof;
+        else if (result == ZRAN_GET_POINT_NOT_COVERED) goto not_covered;
+        else if (result == ZRAN_GET_POINT_CRC_ERROR)   goto crcerror;
+        else if (result != ZRAN_GET_POINT_OK)          goto fail;
 
-    /*
-     * This index point is not byte-aligned.
-     * Adjust the offset accordingly.
-     */
-    if (seek_point->bits > 0)
-        offset -= 1;
+        /*
+         * transform into an offset
+         * into the compresesd stream
+         */
+        index->uncmp_seek_offset = offset;
+        offset                   = seek_point->cmp_offset;
+
+        /*
+         * This index point is not byte-aligned.
+         * Adjust the offset accordingly.
+         */
+        if (seek_point->bits > 0)
+            offset -= 1;
+    }
 
     /*
      * The caller wants a ref to the
@@ -2150,6 +2596,7 @@ int zran_seek(zran_index_t  *index,
 
     return ZRAN_SEEK_OK;
 
+crcerror:        return ZRAN_SEEK_CRC_ERROR;
 fail:            return ZRAN_SEEK_FAIL;
 index_not_built: return ZRAN_SEEK_INDEX_NOT_BUILT;
 not_covered:     return ZRAN_SEEK_NOT_COVERED;
@@ -2174,6 +2621,12 @@ int64_t zran_read(zran_index_t *index,
     /* Used to store/check return values. */
     int ret;
 
+    /*
+     * Used to store error code for return
+     *   if an error occurs
+     */
+    int error_return_val = ZRAN_READ_FAIL;
+
     /*
      * Number of bytes we try to read, and
      * number of bytes actually read/output
@@ -2236,21 +2689,37 @@ int64_t zran_read(zran_index_t *index,
     if (len == 0)         return 0;
     if (len >  INT64_MAX) goto fail;
 
-    zran_log("zran_read(%llu)\n", len);
+    zran_log("zran_read(%llu, %lu)\n", len, index->uncmp_seek_offset);
 
     /*
-     * Search for the index point that
-     * corresponds to our current seek
-     * location in the uncompressed
-     * data stream.
+     * Search for the index point that corresponds to
+     * our current seek location in the uncompressed
+     * data stream. Reading from the start of file is
+     * always allowed, even if the index does not
+     * contain any points.
      */
-    ret = _zran_get_point_with_expand(index,
-                                      index->uncmp_seek_offset,
-                                      0,
-                                      &start);
+    if (index->uncmp_seek_offset == 0) {
+        cmp_offset   = 0;
+        uncmp_offset = 0;
+    }
+    else {
+        ret = _zran_get_point_with_expand(index,
+                                          index->uncmp_seek_offset,
+                                          0,
+                                          &start);
+
+        if      (ret == ZRAN_GET_POINT_EOF)         goto eof;
+        if      (ret == ZRAN_GET_POINT_NOT_COVERED) goto not_covered;
+        else if (ret != ZRAN_GET_POINT_OK) {
+            if (ret == ZRAN_GET_POINT_CRC_ERROR) {
+                error_return_val = ZRAN_READ_CRC_ERROR;
+            }
+            goto fail;
+        }
 
-    if (ret == ZRAN_GET_POINT_EOF)         goto eof;
-    if (ret == ZRAN_GET_POINT_NOT_COVERED) goto not_covered;
+        cmp_offset   = start->cmp_offset;
+        uncmp_offset = start->uncmp_offset;
+    }
 
     /*
      * We have to start decompressing from
@@ -2269,11 +2738,8 @@ int64_t zran_read(zran_index_t *index,
      * reach the current seek location
      * into the uncompresesd data stream.
      */
-    cmp_offset      = start->cmp_offset;
-    uncmp_offset    = start->uncmp_offset;
     first_inflate   = 1;
     total_discarded = 0;
-
     while (uncmp_offset < index->uncmp_seek_offset) {
 
         /*
@@ -2319,7 +2785,8 @@ int64_t zran_read(zran_index_t *index,
                             &bytes_consumed,
                             &bytes_output,
                             to_discard,
-                            discard);
+                            discard,
+                            0);
 
         /*
          * _zran_inflate should return 0 if
@@ -2331,8 +2798,12 @@ int64_t zran_read(zran_index_t *index,
          */
         if (ret != ZRAN_INFLATE_OUTPUT_FULL &&
             ret != ZRAN_INFLATE_EOF         &&
-            ret != ZRAN_INFLATE_OK)
+            ret != ZRAN_INFLATE_OK) {
+            if (ret == ZRAN_INFLATE_CRC_ERROR) {
+                error_return_val = ZRAN_READ_CRC_ERROR;
+            }
             goto fail;
+        }
 
         cmp_offset      += bytes_consumed;
         uncmp_offset    += bytes_output;
@@ -2399,7 +2870,8 @@ int64_t zran_read(zran_index_t *index,
                             &bytes_consumed,
                             &bytes_output,
                             bytes_to_read,
-                            (uint8_t *)(buf) + total_read);
+                            (uint8_t *)(buf) + total_read,
+                            0);
 
         cmp_offset   += bytes_consumed;
         uncmp_offset += bytes_output;
@@ -2418,8 +2890,12 @@ int64_t zran_read(zran_index_t *index,
                 break;
             }
         }
-        else if (ret != ZRAN_INFLATE_OK)
+        else if (ret != ZRAN_INFLATE_OK) {
+            if (ret == ZRAN_INFLATE_CRC_ERROR) {
+                error_return_val = ZRAN_READ_CRC_ERROR;
+            }
             goto fail;
+        }
 
         zran_log("Read %u bytes (%llu / %llu)\n",
                  bytes_output,
@@ -2440,9 +2916,13 @@ int64_t zran_read(zran_index_t *index,
                         &bytes_consumed,
                         &bytes_output,
                         0,
-                        discard);
+                        discard,
+                        0);
 
     if (ret != ZRAN_INFLATE_OK && ret != ZRAN_INFLATE_EOF) {
+        if (ret == ZRAN_INFLATE_CRC_ERROR) {
+            error_return_val = ZRAN_READ_CRC_ERROR;
+        }
         goto fail;
     }
 
@@ -2462,13 +2942,12 @@ int64_t zran_read(zran_index_t *index,
 
 not_covered: return ZRAN_READ_NOT_COVERED;
 eof:         return ZRAN_READ_EOF;
-
 fail:
 
     if (discard != NULL)
         free(discard);
 
-    return ZRAN_READ_FAIL;
+    return error_return_val;
 }
 
 
@@ -2688,6 +3167,9 @@ int zran_import_index(zran_index_t *index,
     uint32_t      npoints;
     zran_point_t *new_list = NULL;
 
+    /* CRC validation is currently not possible on an imported index */
+    index->flags |= ZRAN_SKIP_CRC_CHECK;
+
     /* Check if file is read only. */
     if (!is_readonly(fd, f)) goto fail;
 
diff --git a/indexed_gzip/zran.h b/indexed_gzip/zran.h
index 49b7a2d6..e74e81c8 100644
--- a/indexed_gzip/zran.h
+++ b/indexed_gzip/zran.h
@@ -26,8 +26,8 @@ typedef struct _zran_point zran_point_t;
  * They are specified as bit-masks, rather than bit locations.
  */
 enum {
-  ZRAN_AUTO_BUILD = 1,
-
+  ZRAN_AUTO_BUILD     = 1,
+  ZRAN_SKIP_CRC_CHECK = 2,
 };
 
 
@@ -40,60 +40,61 @@ struct _zran_index {
     /*
      * Handle to the compressed file.
      */
-    FILE         *fd;
+    FILE *fd;
 
     /*
      * Handle to the compressed file object.
      */
-    PyObject     *f;
+    PyObject *f;
 
     /*
      * Size of the compressed file. This
      * is calculated in zran_init.
      */
-    uint64_t      compressed_size;
+    uint64_t compressed_size;
 
     /*
      * Size of the uncompressed data. This is
      * only updated when it becomes known.
      */
-    uint64_t      uncompressed_size;
+    uint64_t uncompressed_size;
 
     /*
      * Spacing size in bytes, relative to the
      * uncompressed data stream, between adjacent
      * index points.
      */
-    uint32_t      spacing;
+    uint32_t spacing;
 
     /*
      * Number of bytes of uncompressed data to store
      * for each index point. This must be a minimum
      * of 32768 bytes.
      */
-    uint32_t      window_size;
+    uint32_t window_size;
 
     /*
      * Base2 logarithm of the window size - it
      * is needed to initialise zlib inflation.
      */
-    uint32_t      log_window_size;
+    uint32_t log_window_size;
 
     /*
      * Size, in bytes, of buffer used to store
      * compressed data read from disk.
      */
-    uint32_t      readbuf_size;
+    uint32_t readbuf_size;
 
     /*
      * Number of index points that have been created.
      */
-    uint32_t      npoints;
+    uint32_t npoints;
 
     /*
-     * Number of index points that can be stored.
+     * Number of index points that can be stored -
+     * i.e. the amount allocated to the "list" field.
      */
-    uint32_t      size;
+    uint32_t size;
 
     /*
      * List of index points.
@@ -107,12 +108,12 @@ struct _zran_index {
      * of where the calling code thinks it
      * is in the (uncompressed) file.
      */
-    uint64_t      uncmp_seek_offset;
+    uint64_t uncmp_seek_offset;
 
     /*
      * Flags passed to zran_init
      */
-    uint16_t      flags;
+    uint16_t flags;
 
     /*
      * All of the fields after this point are used
@@ -123,24 +124,51 @@ struct _zran_index {
      * Reference to a file input
      * buffer of size readbuf_size.
      */
-    uint8_t      *readbuf;
+    uint8_t *readbuf;
 
     /*
      * An offset into readbuf.
      */
-    uint32_t      readbuf_offset;
+    uint32_t readbuf_offset;
 
     /*
      * The current end of the readbuf contents.
      */
-    uint32_t      readbuf_end;
+    uint32_t readbuf_end;
 
     /*
      * Current offsets into the uncompressed and
      * compressed data streams.
      */
-    uint64_t      inflate_cmp_offset;
-    uint64_t      inflate_uncmp_offset;
+    uint64_t inflate_cmp_offset;
+    uint64_t inflate_uncmp_offset;
+
+    /*
+     * Uncompressed offset at the point that the
+     * last GZIP stream ended. This is updated as
+     * more data is read and uncompressed, and
+     * used to determine whether the CRC/size
+     * check for the current stream has already
+     * been performed.
+     */
+    uint64_t last_stream_ended;
+
+    /*
+     * CRC-32 checksum and size (number of
+     * bytes, modulo 2^32) of the uncompressed
+     * data in the current gzip stream, not
+     * used if ZRAN_SKIP_CRC_CHECK is active.
+     * The CRC and size are incrementally
+     * calculated as data is read in. When the
+     * end of a gzip stream is reached, the
+     * calculated CRC and size are compared
+     * against the CRC and size in the gzip
+     * footer, and an error is returned if
+     * they don't match.
+     */
+    uint32_t stream_crc32;
+    uint32_t stream_size;
+    uint8_t  validating;
 };
 
 
@@ -186,15 +214,27 @@ struct _zran_point {
  * Initialise a zran_index_t struct for use with the given file.
  *
  * Passing in 0 for the spacing, window_size and readbuf_size arguments
- * will result in the follwoing values being used:
+ * will result in the following values being used:
  *
  *    spacing:      1048576
  *    window_size:  32768
  *    readbuf_size: 16384
  *
+ * The read buffer must be at least the maximum expectedd size of a GZIP
+ * header. GZIP headers have a minimum size of 10 bytes, but there is no upper
+ * bound on their size, so using a very small read buffer would be unwise.  In
+ * the case of concatenated GZIP streams, the read buffer must be at least big
+ * enough to accommodate a GZIP footer of one stream, padding bytes in between
+ * two streams, and the GZIP header of the next stream.
+ *
  * The flags argument is a bit mask used to control the following options:
  *
- *     ZRAN_AUTO_BUILD: Build the index automatically on demand.
+ *     ZRAN_AUTO_BUILD:     Build the index automatically on demand.
+ *
+ *     ZRAN_SKIP_CRC_CHECK: Do not perform a CRC32 and file size check
+ *                          when the end of a GZIP stream is reached.
+ *                          This flag is automatically set when an index
+ *                          is imported from file using zran_import_index.
  */
 int  zran_init(
   zran_index_t *index,        /* The index                                  */
@@ -217,13 +257,24 @@ void zran_free(
   zran_index_t *index /* The index */
 );
 
+/*
+ * Return codes for zran_build_index.
+ */
+enum {
+    ZRAN_BUILD_INDEX_OK        =  0,
+    ZRAN_BUILD_INDEX_FAIL      = -1,
+    ZRAN_BUILD_INDEX_CRC_ERROR = -2,
+};
+
 
 /*
  * (Re-)Builds the index to cover the given range, which must be
  * specified relative to the compressed data stream. Pass in 0
  * for both offsets to re-build the full index.
  *
- * Returns 0 on success, non-0 on failure.
+ * Returns ZRAN_BUILD_INDEX_OK on success, ZRAN_BUILD_INDEX_CRC_ERROR
+ * if a CRC error is detected in a GZIP stream, or ZRAN_BUILD_INDEX_FAIL
+ * if some other type of error occurs.
  */
 int zran_build_index(
   zran_index_t *index, /* The index */
@@ -234,6 +285,7 @@ int zran_build_index(
 
 /* Return codes for zran_seek. */
 enum {
+    ZRAN_SEEK_CRC_ERROR       = -2,
     ZRAN_SEEK_FAIL            = -1,
     ZRAN_SEEK_OK              =  0,
     ZRAN_SEEK_NOT_COVERED     =  1,
@@ -264,6 +316,10 @@ enum {
  *    - ZRAN_SEEK_EOF to indicate that the requested offset
  *      is past the end of the uncompressed stream.
  *
+ *    - ZRAN_SEEK_CRC_ERROR to indicate that the CRC or file size
+ *      stored in the footer of a GZIP stream does not match the
+ *      data.
+ *
  *    - ZRAN_SEEK_FAIL to indicate failure of some sort.
  */
 int zran_seek(
@@ -287,7 +343,8 @@ uint64_t zran_tell(
 enum {
     ZRAN_READ_NOT_COVERED = -1,
     ZRAN_READ_EOF         = -2,
-    ZRAN_READ_FAIL        = -3
+    ZRAN_READ_FAIL        = -3,
+    ZRAN_READ_CRC_ERROR   = -4
 };
 
 /*
@@ -296,7 +353,8 @@ enum {
  * the ZRAN_AUTO_BUILD flag, it is expanded as needed.
  *
  * Returns:
- *   - Number of bytes read for success.
+ *   - Number of bytes read for success, or one of the following codes,
+ *     all of which are negative.
  *
  *   - ZRAN_READ_NOT_COVERED to indicate that the index does not
  *     cover the requested region (will never happen if
@@ -305,6 +363,10 @@ enum {
  *   - ZRAN_READ_EOF to indicate that the read could not be completed
  *     because the current uncompressed seek point is at EOF.
  *
+ *   - ZRAN_SEEK_CRC_ERROR to indicate that the CRC or file size
+ *     stored in the footer of a GZIP stream does not match the
+ *     data.
+ *
  *   - ZRAN_READ_FAIL to indicate that the read failed for some reason.
  */
 int64_t zran_read(
@@ -403,6 +465,10 @@ enum {
  * Updating an index file is not supported currently. To update an index file,
  * first import it, create new checkpoints, and then export it again.
  *
+ * CRC validation of uncompressed data from an imported index is not currently
+ * supported - this function will enable the ZRAN_SKIP_CRC_CHECK flag on the
+ * given zran_index_t struct.
+ *
  * See zran_export_index for exporting.
  *
  * Returns:
diff --git a/indexed_gzip/zran.pxd b/indexed_gzip/zran.pxd
index 44b4722f..82137339 100644
--- a/indexed_gzip/zran.pxd
+++ b/indexed_gzip/zran.pxd
@@ -28,21 +28,34 @@ cdef extern from "zran.h":
         uint8_t  *data;
 
     enum:
-        ZRAN_AUTO_BUILD           =  1,
+        # flags for zran_init
+        ZRAN_AUTO_BUILD     =  1,
+        ZRAN_SKIP_CRC_CHECK =  2,
 
+        # return codes for zran_build_index
+        ZRAN_BUILD_INDEX_OK        =  0,
+        ZRAN_BUILD_INDEX_FAIL      = -1,
+        ZRAN_BUILD_INDEX_CRC_ERROR = -2,
+
+        # return codes for zran_seek
+        ZRAN_SEEK_CRC_ERROR       = -2,
         ZRAN_SEEK_FAIL            = -1,
         ZRAN_SEEK_OK              =  0,
         ZRAN_SEEK_NOT_COVERED     =  1,
         ZRAN_SEEK_EOF             =  2,
         ZRAN_SEEK_INDEX_NOT_BUILT =  3,
 
+        # return codes for zran_read
         ZRAN_READ_NOT_COVERED = -1,
         ZRAN_READ_EOF         = -2,
         ZRAN_READ_FAIL        = -3,
+        ZRAN_READ_CRC_ERROR   = -4,
 
+        # return codes for zran_export_index
         ZRAN_EXPORT_OK          =  0,
         ZRAN_EXPORT_WRITE_ERROR = -1,
 
+        # return codes for zran_import_index
         ZRAN_IMPORT_OK             =  0,
         ZRAN_IMPORT_FAIL           = -1,
         ZRAN_IMPORT_EOF            = -2,
@@ -52,19 +65,19 @@ cdef extern from "zran.h":
         ZRAN_IMPORT_MEMORY_ERROR   = -6,
         ZRAN_IMPORT_UNKNOWN_FORMAT = -7
 
-    bint zran_init(zran_index_t *index,
-                   FILE         *fd,
-                   PyObject     *f,
-                   uint32_t      spacing,
-                   uint32_t      window_size,
-                   uint32_t      readbuf_size,
-                   uint16_t      flags)
+    int zran_init(zran_index_t *index,
+                  FILE         *fd,
+                  PyObject     *f,
+                  uint32_t      spacing,
+                  uint32_t      window_size,
+                  uint32_t      readbuf_size,
+                  uint16_t      flags)
 
     void zran_free(zran_index_t *index)
 
-    bint zran_build_index(zran_index_t *index,
-                          uint64_t      from_,
-                          uint64_t      until) nogil;
+    int zran_build_index(zran_index_t *index,
+                         uint64_t      from_,
+                         uint64_t      until) nogil;
 
     uint64_t zran_tell(zran_index_t *index);
 


Test patch:
diff --git a/indexed_gzip/tests/__init__.py b/indexed_gzip/tests/__init__.py
index bdb2276c..aac834c2 100644
--- a/indexed_gzip/tests/__init__.py
+++ b/indexed_gzip/tests/__init__.py
@@ -5,6 +5,7 @@
 # Author: Paul McCarthy <pauldmccarthy@gmail.com>
 #
 
+import                    io
 import                    os
 import                    sys
 import                    time
@@ -67,13 +68,11 @@ def compress_with_gzip_module():
 
         with open(infile, 'rb') as inf:
             while True:
-                with gzip.open(outfile, 'a') as outf:
-                    data = inf.read(buflen)
-
-                    if len(data) == 0:
-                        break
-
-                    outf.write(data)
+                data = inf.read(buflen)
+                if len(data) == 0:
+                    break
+                with open(outfile, 'ab') as outf:
+                    gzip.GzipFile(fileobj=outf).write(data)
 
     def compress_with_gzip_command():
 
@@ -124,6 +123,37 @@ def compress_with_gzip_command():
     poll(lambda : not cmpThread.is_alive())
 
 
+def compress_inmem(data, concat):
+    """Compress the given data (assumed to be bytes) and return a bytearray
+    containing the compressed data (including gzip header and footer).
+    Also returns offsets for the end of each separate stream.
+    """
+
+    f = io.BytesIO()
+    if concat: chunksize = len(data) // 10
+    else:      chunksize = len(data)
+
+    offsets    = []
+    compressed = 0
+    print('Generating compressed data {}, concat: {})'.format(
+        len(data), concat))
+    while compressed < len(data):
+        start = len(f.getvalue())
+        chunk = data[compressed:compressed + chunksize]
+        with gzip.GzipFile(mode='ab', fileobj=f) as gzf:
+            gzf.write(chunk)
+
+        end = len(f.getvalue())
+
+        print('  Wrote stream to {} - {} [{} bytes] ...'.format(
+            start, end, end - start))
+        offsets.append(end)
+        compressed += chunksize
+
+    print('  Final size: {}'.format(len(f.getvalue())))
+
+    f.seek(0)
+    return bytearray(f.read()), offsets
 
 
 def gen_test_data(filename, nelems, concat):
diff --git a/indexed_gzip/tests/ctest_indexed_gzip.pyx b/indexed_gzip/tests/ctest_indexed_gzip.pyx
index edb01ebd..6ac1017e 100644
--- a/indexed_gzip/tests/ctest_indexed_gzip.pyx
+++ b/indexed_gzip/tests/ctest_indexed_gzip.pyx
@@ -329,20 +329,28 @@ def test_manual_build():
 
         with igzip._IndexedGzipFile(fname, auto_build=False) as f:
 
-            # Seeking should fail
-            with pytest.raises(igzip.NotCoveredError):
-                f.seek(1234)
+            # Seeking to 0 should work, but
+            # anywhere else should fail
+            f.seek(0)
+            for off in [1, 2, 20, 200]:
+                with pytest.raises(igzip.NotCoveredError):
+                    f.seek(off)
 
             # Reading from beginning should work
             readval = read_element(f, 0, seek=False)
             assert readval == 0
 
+            # but subsequent reads should fail
+            # (n.b. this might change in the future)
+            with pytest.raises(igzip.NotCoveredError):
+                readval = read_element(f, 1, seek=False)
+
             # Seek should still fail even after read
-            f.seek(0)
             with pytest.raises(igzip.NotCoveredError):
                 f.seek(8)
 
             # But reading from beginning should still work
+            f.seek(0)
             readval = read_element(f, 0, seek=False)
             assert readval == 0
 
@@ -371,6 +379,40 @@ def test_read_all(testfile, nelems, use_mmap, drop):
     assert check_data_valid(data, 0)
 
 
+def test_simple_read_with_null_padding():
+
+
+    fileobj = BytesIO()
+
+    with gzip.GzipFile(fileobj=fileobj, mode='wb') as f:
+        f.write(b"hello world")
+
+    fileobj.write(b"\0" * 100)
+
+    with igzip._IndexedGzipFile(fileobj=fileobj) as f:
+        assert f.read() == b"hello world"
+        f.seek(3)
+        assert f.read() == b"lo world"
+        f.seek(20)
+        assert f.read() == b""
+
+
+def test_read_with_null_padding(testfile, nelems):
+
+    fileobj = BytesIO(open(testfile, "rb").read() + b"\0" * 100)
+
+    with igzip._IndexedGzipFile(fileobj=fileobj) as f:
+        data = f.read(nelems * 8)
+        # Read a bit further so we reach the zero-padded area.
+        # This line should not throw an exception.
+        f.read(1)
+
+    data = np.ndarray(shape=nelems, dtype=np.uint64, buffer=data)
+
+    # Check that every value is valid
+    assert check_data_valid(data, 0)
+
+
 def test_read_beyond_end(concat, drop):
     with tempdir() as tdir:
         nelems   = 65536
@@ -817,19 +859,38 @@ def test_wrapper_class():
             f.import_index(idxfname)
 
 
+def gcd(num):
+    if num <= 3:
+        return 1
+    candidates = list(range(int(np.ceil(np.sqrt(num))), 2, -2))
+    candidates.extend((2, 1))
+    for divisor in candidates:
+        if num % divisor == 0:
+            return divisor
+    return 1
+
+
 def test_size_multiple_of_readbuf():
 
     fname = 'test.gz'
 
     with tempdir():
-        data = np.random.randint(1, 1000, 10000, dtype=np.uint32)
 
-        with gzip.open(fname, 'wb') as f:
-            f.write(data.tobytes())
-        del f
-        f = None
+        while True:
+
+            data = np.random.randint(1, 1000, 100000, dtype=np.uint32)
+            with gzip.open(fname, 'wb') as f:
+                f.write(data.tobytes())
+            del f
+            f = None
 
-        fsize = op.getsize(fname)
+            # we need a file size that is divisible
+            # by the minimum readbuf size
+            fsize = op.getsize(fname)
+            if gcd(fsize) >= 128:
+                break
+
+        # readbuf size == file size
         bufsz = fsize
 
         with igzip.IndexedGzipFile(fname, readbuf_size=bufsz) as f:
@@ -838,18 +899,13 @@ def test_size_multiple_of_readbuf():
         f = None
 
         with igzip.IndexedGzipFile(fname, readbuf_size=bufsz) as f:
-            read = np.ndarray(shape=10000, dtype=np.uint32, buffer=f.read())
+            read = np.ndarray(shape=100000, dtype=np.uint32, buffer=f.read())
             assert np.all(read == data)
         del f
         f = None
 
-        # we're screwed if the
-        # file size is prime
-        for div in (2, 3, 5, 7, 11, 13, 17):
-            if div % fsize == 0:
-                break
-
-        bufsz = fsize / div
+        # Use a buf size that is a divisor of the file size
+        bufsz = gcd(fsize)
 
         with igzip.IndexedGzipFile(fname, readbuf_size=bufsz) as f:
             assert f.seek(fsize) == fsize
@@ -857,7 +913,7 @@ def test_size_multiple_of_readbuf():
         f = None
 
         with igzip.IndexedGzipFile(fname, readbuf_size=bufsz) as f:
-            read = np.ndarray(shape=10000, dtype=np.uint32, buffer=f.read())
+            read = np.ndarray(shape=100000, dtype=np.uint32, buffer=f.read())
             assert np.all(read == data)
         del f
         f = None
diff --git a/indexed_gzip/tests/ctest_zran.pyx b/indexed_gzip/tests/ctest_zran.pyx
index 09a220a5..ca9649ca 100644
--- a/indexed_gzip/tests/ctest_zran.pyx
+++ b/indexed_gzip/tests/ctest_zran.pyx
@@ -35,6 +35,8 @@ from libc.stdio  cimport (SEEK_SET,
                           fdopen,
                           fwrite)
 
+from libc.stdint cimport int64_t
+
 from libc.string cimport memset, memcmp
 
 from cpython.exc cimport (PyErr_Clear,
@@ -52,7 +54,7 @@ from posix.mman cimport (mmap,
                          MAP_SHARED)
 
 
-from . import poll, check_data_valid, tempdir
+from . import poll, check_data_valid, tempdir, compress_inmem
 
 
 cdef extern from "sys/mman.h":
@@ -799,13 +801,14 @@ def test_seek_then_read_block(testfile, no_fds, nelems, niters, seed, use_mmap):
     with open(testfile, 'rb') as pyfid:
         cfid = fdopen(pyfid.fileno(), 'rb')
 
-        assert not zran.zran_init(&index,
-                                  NULL if no_fds else cfid,
-                                  <PyObject*>pyfid if no_fds else NULL,
-                                  indexSpacing,
-                                  32768,
-                                  131072,
-                                  zran.ZRAN_AUTO_BUILD)
+        ret = zran.zran_init(&index,
+                             NULL if no_fds else cfid,
+                             <PyObject*>pyfid if no_fds else NULL,
+                             indexSpacing,
+                             32768,
+                             131072,
+                             zran.ZRAN_AUTO_BUILD)
+        assert not ret, ret
 
         for i, se in enumerate(seekelems):
 
@@ -1021,6 +1024,7 @@ def test_readbuf_spacing_sizes(testfile, no_fds, nelems, niters, seed):
 
 cdef _compare_indexes(zran.zran_index_t *index1,
                       zran.zran_index_t *index2):
+    """Check that two indexes are equivalent. """
     cdef zran.zran_point_t *p1
     cdef zran.zran_point_t *p2
 
@@ -1047,8 +1051,10 @@ cdef _compare_indexes(zran.zran_index_t *index1,
             assert not memcmp(p2.data, p1.data, ws), msg
 
 
-
 def test_export_then_import(testfile, no_fds):
+    """Export-import round trip . Test exporting an index, then importing it
+    back in.
+    """
 
     cdef zran.zran_index_t index1
     cdef zran.zran_index_t index2
@@ -1098,6 +1104,9 @@ def test_export_then_import(testfile, no_fds):
 
 
 def test_export_import_no_points(no_fds):
+    """Test exporting and importing an index which does not contain any
+    seek points.
+    """
     cdef zran.zran_index_t index
     cdef void             *buffer
 
@@ -1188,6 +1197,7 @@ def test_export_import_format_v0():
 
 
 cdef _write_index_file_v0(zran.zran_index_t *index, dest):
+    """Write the given index out to a file, index file version 0 format. """
 
     cdef zran.zran_point_t *point
 
@@ -1209,3 +1219,180 @@ cdef _write_index_file_v0(zran.zran_index_t *index, dest):
             point = &index.list[i]
             data  = <bytes>point.data[:index.window_size]
             f.write(data)
+
+
+def test_crc_validation(concat):
+    """Basic test of CRC validation. """
+
+    cdef zran.zran_index_t index
+    cdef void             *buffer
+    cdef int64_t           ret
+
+    # use uint32 so there are lots of zeros,
+    # and so there is something to compress
+    dsize             = 1048576 * 10
+    data              = np.random.randint(0, 255, dsize // 4, dtype=np.uint32)
+    cmpdata, strmoffs = compress_inmem(data.tobytes(), concat)
+    buf               = ReadBuffer(dsize)
+    buffer            = buf.buffer
+    f                 = [None]  # to prevent gc
+
+    def _zran_init(flags):
+        f[0] = BytesIO(cmpdata)
+        assert not zran.zran_init(&index,
+                                  NULL,
+                                  <PyObject*>f[0],
+                                  1048576,
+                                  32768,
+                                  131072,
+                                  flags)
+
+    def _run_crc_tests(shouldpass, flags=zran.ZRAN_AUTO_BUILD):
+        if shouldpass:
+            expect_build = zran.ZRAN_BUILD_INDEX_OK
+            expect_seek  = zran.ZRAN_SEEK_OK
+            expect_read  = dsize
+        else:
+            expect_build = zran.ZRAN_BUILD_INDEX_CRC_ERROR
+            expect_seek  = zran.ZRAN_SEEK_CRC_ERROR
+            expect_read  = zran.ZRAN_READ_CRC_ERROR
+
+        # CRC validation should occur on the first
+        # pass through a gzip stream, regardless
+        # of how that pass is initiated. Below we
+        # test the most common scenarios.
+
+        # Error if we try to build an index.  Note
+        # that an error here is not guaranteed, as
+        # the _zran_expand_index might need a few
+        # passes through the data to reach the end,
+        # which might cause inflation to be
+        # re-initialised, and therefore validation
+        # to be disabled.  It depends on the data,
+        # and on the constants used in
+        # _zran_estimate_offset
+        _zran_init(flags)
+        ret = zran.zran_build_index(&index, 0, 0)
+        assert ret == expect_build, ret
+        zran.zran_free(&index)
+
+        # error if we try to seek
+        _zran_init(flags)
+        ret = zran.zran_seek(&index, dsize - 1, SEEK_SET, NULL)
+        assert ret == expect_seek, ret
+        zran.zran_free(&index)
+
+        # error if we try to read
+        _zran_init(flags)
+        ret = zran.zran_read(&index, buffer, dsize)
+        assert ret == expect_read, ret
+        zran.zran_free(&index)
+
+        if shouldpass:
+            pybuf = <bytes>(<char *>buffer)[:dsize]
+            assert np.all(np.frombuffer(pybuf, dtype=np.uint32) == data)
+
+    # data/crc is good, all should be well
+    _run_crc_tests(True)
+
+    # corrupt the size, we should get an error
+    cmpdata[-1] += 1  # corrupt size
+    _run_crc_tests(False)
+
+    # corrupt the crc, we should get an error
+    cmpdata[-1] -= 1  # restore size to correct value
+    cmpdata[-5] += 1  # corrupt crc
+    _run_crc_tests(False)
+
+    # Corrupt a different stream, if we have more than one
+    cmpdata[-5] -= 1  # restore crc to correct value
+    if len(strmoffs) > 1:
+        for off in strmoffs[1:]:
+            cmpdata[off-1] += 1
+            _run_crc_tests(False)
+            cmpdata[off-1] -= 1
+
+    # Disable CRC, all should be well, even with a corrupt CRC/size
+    # First test with good data
+    _run_crc_tests(True, zran.ZRAN_AUTO_BUILD | zran.ZRAN_SKIP_CRC_CHECK)
+
+    cmpdata[-1] += 1  # corrupt size
+    _run_crc_tests(True, zran.ZRAN_AUTO_BUILD | zran.ZRAN_SKIP_CRC_CHECK)
+
+    cmpdata[-1] -= 1  # restore size to correct value
+    cmpdata[-5] -= 1  # corrupt crc
+    _run_crc_tests(True, zran.ZRAN_AUTO_BUILD | zran.ZRAN_SKIP_CRC_CHECK)
+
+
+def test_standard_usage_with_null_padding(concat):
+    """Make sure standard usage works with files that have null-padding after
+    the GZIP footer.
+
+    See https://www.gnu.org/software/gzip/manual/gzip.html#Tapes
+    """
+    cdef zran.zran_index_t index
+    cdef void             *buffer
+    cdef int64_t           ret
+
+    # use uint32 so there are lots of zeros,
+    # and so there is something to compress
+    dsize             = 1048576 * 10
+    data              = np.random.randint(0, 255, dsize // 4, dtype=np.uint32)
+    cmpdata, strmoffs = compress_inmem(data.tobytes(), concat)
+    buf               = ReadBuffer(dsize)
+    buffer            = buf.buffer
+    f                 = [None]  # to prevent gc
+
+    # random amount of padding for each stream
+    padding = np.random.randint(1, 100, len(strmoffs))
+
+    # new compressed data - bytearrays
+    # are initialised to contain all 0s
+    paddedcmpdata = bytearray(len(cmpdata) + padding.sum())
+
+    # copy old unpadded compressed data
+    # into new padded compressed data
+    padoff = 0  # offset into padded data
+    last   = 0  # offset to end of last copied stream in unpadded data
+    print('Padding streams [orig size: {}] ...'.format(len(cmpdata)))
+    for off, pad in zip(strmoffs, padding):
+
+        strm = cmpdata[last:off]
+
+        paddedcmpdata[padoff:padoff + len(strm)] = strm
+
+        print('  Copied stream from [{} - {}] to [{} - {}] ({} '
+              'padding bytes)'.format(
+                  last, off, padoff, padoff + len(strm), pad))
+
+        padoff += len(strm) + pad
+        last    = off
+
+
+    def _zran_init():
+        f[0] = BytesIO(paddedcmpdata)
+        assert not zran.zran_init(&index,
+                                  NULL,
+                                  <PyObject*>f[0],
+                                  1048576,
+                                  32768,
+                                  131072,
+                                  zran.ZRAN_AUTO_BUILD)
+
+    _zran_init()
+    ret = zran.zran_build_index(&index, 0, 0)
+    assert ret == zran.ZRAN_BUILD_INDEX_OK, ret
+    zran.zran_free(&index)
+
+    _zran_init()
+    ret = zran.zran_seek(&index, dsize - 1, SEEK_SET, NULL)
+    assert ret == zran.ZRAN_SEEK_OK, ret
+    zran.zran_free(&index)
+
+    _zran_init()
+    ret = zran.zran_read(&index, buffer, dsize)
+    assert ret == dsize, ret
+    zran.zran_free(&index)
+
+    pybuf = <bytes>(<char *>buffer)[:dsize]
+    assert np.all(np.frombuffer(pybuf, dtype=np.uint32) == data)
diff --git a/indexed_gzip/tests/test_indexed_gzip.py b/indexed_gzip/tests/test_indexed_gzip.py
index 183abfbb..e34c9142 100644
--- a/indexed_gzip/tests/test_indexed_gzip.py
+++ b/indexed_gzip/tests/test_indexed_gzip.py
@@ -82,6 +82,12 @@ def test_read_all(testfile, nelems, use_mmap):
 def test_read_all_drop_handles(testfile, nelems, use_mmap):
     ctest_indexed_gzip.test_read_all(testfile, nelems, use_mmap, True)
 
+def test_simple_read_with_null_padding():
+    ctest_indexed_gzip.test_simple_read_with_null_padding()
+
+def test_read_with_null_padding(testfile, nelems):
+    ctest_indexed_gzip.test_read_with_null_padding(testfile, nelems)
+
 def test_read_beyond_end(concat):
     ctest_indexed_gzip.test_read_beyond_end(concat, False)
 
@@ -154,7 +160,7 @@ def test_import_export_index():
 def test_wrapper_class():
     ctest_indexed_gzip.test_wrapper_class()
 
-def test_size_multiple_of_readbuf():
+def test_size_multiple_of_readbuf(seed):
     ctest_indexed_gzip.test_size_multiple_of_readbuf()
 
 @pytest.mark.slow_test
diff --git a/indexed_gzip/tests/test_zran.py b/indexed_gzip/tests/test_zran.py
index 815e07d2..5802f808 100644
--- a/indexed_gzip/tests/test_zran.py
+++ b/indexed_gzip/tests/test_zran.py
@@ -121,3 +121,9 @@ def test_export_import_no_points():
 
     def test_export_import_format_v0():
         ctest_zran.test_export_import_format_v0()
+
+    def test_crc_validation(concat):
+        ctest_zran.test_crc_validation(concat)
+
+    def test_standard_usage_with_null_padding(concat):
+        ctest_zran.test_standard_usage_with_null_padding(concat)