scipy/scipy — pull request #2812 by batterseapower (2013-08-29 14:45:16+00:00)
Title: Detect over-long compressed sequences in .mat files

We recently had a problem on our internal network where .mat files were served to users with corrupt content. Despite these files being bogus, they were silently accepted by Matlab and other .mat reading libraries.

Even though the .mat format does not contain any integrity-checking features per-se, it turns out that it is possible to perform substantial integrity checks on the contents by verifying the trailing ZLib checksums that get written as part of any miCOMPRESSED element. I've already raised a ticket with the Mathworks asking them to actually check these checksums in their internal .mat file loader -- this pull request tests and implements the same functionality for Scipy's .mat reader.

Scipy's reader would already validate the checksum correctly for many forms of corruption, but there was one missing check -- when the corruption caused the uncompressed stream to be *larger* than the length of the uncompressed stream that we are expecting. In this case Scipy's reader would consume only a prefix of the uncompressed data and so zlib would never get to see the trailing checksum on the compressed data stream.

This pull request verifies that that the uncompressed stream is entirely consumed by reading the compressed element, preventing this kind of corruption from going undetected. Furthermore, I have added tests for two kinds of corrupted file (one where the Zlib checksum is incorrect, and one where the uncompressed stream is overlong).