Package: compresso
Compresso: Efficient Compression of Segmentation Data For Connectomics (PyPI edition)importcompressoimportnumpyasnplabels=np.array(...)compressed_labels=compresso.compress(labels)# 3d numpy array -> compressed bytesreconstituted_labels=compresso.decompress(compressed_labels)# compressed bytes -> 3d numpy array# Adds an index and modifies the stream to enable# Random access to Z slices. Format Version 1.compressed_labels=compresso.compress(labels,random_access_z_index=True)reconstituted_labels=compresso.decompress(compressed_labels,z=3)# one z slicereconstituted_labels=compresso.decompress(compressed_labels,z=(1,5))# four slices# A convenience object that simulates an array# to efficiently extract image dataarr=compresso.CompressoArray(compressed_labels)img=arr[:,:,1:5]# same four slices as above# Returns header info as dict# Has array dimensions and data width information.header=compresso.header(compressed_labels)# Extract the unique labels from a stream without# decompressing to a full 3D array. Fast and low memory.uniq_labels=compresso.labels(compressed_labels)# Remap labels without decompressing. Could# be useful for e.g. proofreading.compressed_remapped=compresso.remap(compressed_labels,{1:2,2:3,...},preserve_missing_labels=True)# Checks if the stream appears to be valid.# This is a superficial check of headers.is_valid=compresso.valid(stream)# CLI compression of numpy data# Compresso is designed to use a second stage compressor# so use gzip, lzma, or others on the output file.$compressodata.npy# -> data.npy.cpso$compresso-ddata.npy.cpso# -> data.npy$compresso--helpNOTE: This is an extensive modification of the work by Matejek et al. which can be found here:https://github.com/VCG/compresso. It is not compatible with RhoANA streams.Recent advances in segmentation methods for connectomics and biomedical imaging produce very large datasets with labels that assign object classes to image pixels. The resulting label volumes are bigger than the raw image data and need compression for efficient storage and transfer. General-purpose compression methods are less effective because the label data consists of large low-frequency regions with structured boundaries unlike natural image data. We present Compresso, a new compression scheme for label data that outperforms existing approaches by using a sliding window to exploit redundancy across border regions in 2D and 3D. We compare our method to existing compression schemes and provide a detailed evaluation on eleven biomedical and image segmentation datasets. Our method provides a factor of 600-2200x compression for label volumes, with running times suitable for practice.Paper: Matejeket al., "Compresso: Efficient Compression of Segmentation Data For Connectomics", Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2017, 10-14. [CITE|PDF]In more concrete but simple terms, compresso represents the boundary between segments as a boolean bit packed field. Long runs of zeros are run length encoded. The 4-connected components within that field are mapped to a corresponding label. Boundary voxels are decoded with reference to their neighbors, or if that fails, by storing their label. A second stage of compression is then applied, such as gzip or lzma. There's a few more details but that's a reasonable overview.SetupRequires Python 3.6+pipinstallcompressoVersionsMajor VersionFormat VersionDescription1-Initial Release. Not usable due to bugs. No format versioning.20First major release.30,1Introduces random access to z slices in format version 1.Compresso Stream FormatSectionBytesDescriptionHeader36Metadata incl. length of fields.idsheader.data_width * header.id_sizeMap of CCL regions to labels.valueswindow_size * header.value_sizeValues of renumbered windows. Bitfields describing boundaries.locationsheader.data_width * header.locations_sizeSequence of 7 control codes and labels + 7 that describe how to decode indeterminate locations in the boundary.windowsThe rest of the stream.Sequence of numbers to be remapped from values. Describes the boundary structure of labels.z_index(optional tail) 2 * width * header.szOffsets into label values and locations to enable random access to slices. Format Version 1.window_sizeis the smallest data type that will containxstep * ystep * zstep. For example,steps=(4,4,1)uses uint16 whilesteps=(8,8,1)uses uint64.The byte width of thez_indexis the smallest unsigned integer type that will contain2 * sx * sy.Codec ChangesThe original codec has been updated and is no longer compatible with the original. Below are the important changes we made that differ from the code published alongside the paper.Implementation wise, we also fixed up several bugs, added guards against data corruption, did some performance tuning, and made sure that the entire codec is implemented in C++ and called by Python. Thus, the codec is usable in both C++ and Python as well as any languages, such as Web Assembly, that C++ can be transpiled to.Thank you to the original authors for publishing your code and algorithm from which this repo is derived.Updated HeaderThe previous header was 72 bytes. We updated the header to be only 35 bytes. It now includes the magic numbercpso, a version number, and the data width of the labels.
This additional information makes detecting valid compresso streams easier, allows for updating the format in the future, and allows us to assume smaller byte widths than 64-bit.AttributeValueTypeDescriptionmagiccpsochar[4]File magic number.format_version0 or 1u8Version of the compresso stream.data_width1,2,4,or 8u8Size of the labels in bytes.sx, sy, sz>= 0u16 x 3Size of array dimensions.xstep,ystep,zstep0 < product <= 64u8 x 3Size of structure grid.id_size>= 0u64Size of array mapping of CCL regions to labels.value_size>= 0u32Size of array mapping windows to renumbering.location_size>= 0u64Size of indeterminate locations array.connectivity4 or 6u8Connectivity for connected components.Char Byte StreamThe previous implementation treated the byte stream as uniform u64 little endian. We now emit the encoded stream asunsigned charand write each appropriate data type in little endian.Variable Data WidthsThe labels may assume any unsigned integer data width, which reduces the size of the ids and locations stream when appropriate. The encoded boundaries are reduced to the smallest size that fits. A 4x4x1 window is represented with u16, an 8x8x1 with u64. Less commonly used, but a 4x4x2 would be represented with u32, and a 4x2x1 would get a u8.Note that at this time only 4x4x1 and 8x8x1 are supported in this implementation, but the protocol will make those assumptions.Supports Full Integer Range in Indeterminate LocationsThe previous codec reserved 6 integers for instructions in the locations stream, but this meant that six segmentation labels were not representable. We added a seventh reserved instruction that indicates the next byte in the stream is the label and then we can use the full range of the integer to represent that number.This potentially expands the size of the compressed stream. However, we only use this instruction for non-representable numbers, so for most data it should cause zero increase and minimal increase so long as the non-representable numbers in indeterminate locations are rare. The upside is compresso now handles all possible inputs.Supports 4 and 6 Connected Components6-connected CCL seems like it would be a win because it would reduce the number of duplicated IDs that need to be stored. However, in an experiment we found that it did significantly decrease IDs, but at the expense of adding many more boundary voxels (since you need to consider the Z direction now) and increasing the number of indeterminate locations far more. It ended up being slower and larger on some connectomics segmentation we experimented with.However, we suspect that there are some images where 6 would do better. An obvious example is a solid
color image that has no boundaries. The images where 6 shines will probably have sparser and straighter boundaries so that fewer additional boundary voxels are introduced.Random Access to Z-Slices (Format Version 1)We make two changes to the codec in order to allow random access to Z slices. First, we disable indeterminate location codes 4 and 5 which refer to other slices, making each slice independently decodable. We also add a tail of size2 * index_width * szwhich contains unsigned 8, 16, 32, or 64 bit offsets into the labels and locations streams for each slice which are arranged as all the labels then all the locations (they are not interleaved). The streams are difference coded to reduce the magnitude of the integers. The byte width is determined by the smallest unsigned integer type that will be able to represent 2 * sx * sy which is a coarse upper bound for the locations stream.The overall impact of this change is a slight increase in the size of the compresso stream and a possible impact on the compressibility if the vertical references were heavily used, such as on a checkerboard type image.This feature can be disabled by settingcompress(..., random_access_z_index=False)which will emit a format version 0 stream. When this feature is enabled, it sets the format version to 1. This implementation can encode and decode both format versions.This feature is not supported whenconnectivity=6due to the required interdependence of the slices.Results From the PaperCompression PerformanceCompression ratios of general-purpose compression methods combined with Compresso and Neuroglancer. Compresso paired with LZMA yields the best compression ratios for all connectomics datasets (left) and in average (four out of five) for the others (right).