Compression library for Grid data logger - Page 1
Forum section: electronics / general

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: January 27, 2022, 08:22:14 am
content: Hi,The data logger would sample data from The Grid at 16Ksps, it would have 8 channels of 24bit ADC data,8 channels of 8bit ADC Data, and 32bit Digital Input and outputs too, I want to use a Cortex M7 processor for the Job.The ARM would handle lots of things including a GUI @ 720px720p and Ethernet connection too, so I need a lossless compression library which would preferably use low CPU usage, the Ram and Flash requirements are not that hard, because I would use external Flash and SDRAM. I prefer to reach a compression ratio better than 50% .A single channel ADC data is attached for your reference, (I have converted the data to ASCII) But I would use it on Binary data in the final Design.So which compression library would you recommend?

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: January 29, 2022, 09:27:23 am
content: Come on Guys, do you recommend any open source library?

author: Marco
author_rank: Super Contributor
timestamp: January 29, 2022, 12:18:04 pm
content: If memory isn't a problem then for 8 bit I'd try simply Huffman encoding the difference between the sample from 1/50th (or 1/60th) seconds before. For 24 bits, the same except only compress for a range of 2^x around the previous value, if the difference is more have a bit to bypass compression.Can use a more complex predictor if you want to take into account cycle time variance. In which case you might use a predictor based only on a couple of previous samples, so you don't need as much memory.

author: mariush
author_rank: Super Contributor
timestamp: January 29, 2022, 01:46:01 pm
content: Use something standard and well known ... deflate, lzo, lz4, whateverThe compression is less important, it's how you arrange the data that's important.Just grouping the bytes would help the compressor significantly. (ex take 16K readings and do 16 k bytes of first byte from every reading, then 16k of second byte from the next reading and so on)Just did an experiment with your data, and with first 16K records, the original 49152 byte file compressed using 7z lzma to 44266 bytes , and the one rearranged was reduced to 32260 bytes (67% of original size). Deflate does around 33500 bytes.You can see for yourself with the attached files that contain your data, and the script I've used to create them (in php)You can go further... if you know you have long runs of negative numbers, and then long runs of positive numbers ... you could just have a header to every chunk of records where you say N total samples, 100 negative, 500 positive, 1 negative, 1000 positive ... you may use 100 bytes or as many as needed, but you save 16k records x 1 sign bit or 2KB.In the attached example, there's 128 changes of sign in the first 16k samples if my code is correct, and most runs are up to 129 samples of same sign, so if you use 2 bytes for each group then that's 256 bytes header but you're saving 16 kbps or 2048 bytes... so overall 1792 bytes reduction.You could have other tricks like storing only difference between samples ... have a "keyframe sample" which is stored in 3 bytes, then have 16 or as many as you want that store the difference between samples and use 2 or 3 bytes for the next values ( first bit 0 = 2 bytes, first bit 1 = 3 bytes) dynamic size for each value (for example your number could be multiples of 4 bits (minimum 12 bits, maximum 28 bits) ... use 2 bits to signify length, one bit for sign, rest of the bits for difference

author: fcb
author_rank: Super Contributor
timestamp: January 29, 2022, 02:09:31 pm
content: Why bother for just 50% compression. Just double the size of the memory.When we've stored mains waveforms, we've either stored them unmolested or we've only stored the cycles that exceed a specified window (and during "normal" data just stored V/I/PF/etc.. per cycle).

author: Jeroen3
author_rank: Super Contributor
timestamp: January 29, 2022, 03:13:14 pm
content: Can you use an audio compression library?FLAC is free, takes about 30% of the size away.

author: Harjit
author_rank: Regular Contributor
timestamp: January 29, 2022, 05:46:56 pm
content: List I've saved off - in no particular order i.e. I think they will work for my embedded application but I haven't benchmarked them.I had posted this list in another thread and the person went with heatshrink.https://github.com/lz4/lz4https://github.com/atomicobject/heatshrinkhttps://github.com/richgel999/minizhttps://www.segger.com/products/compression/emcompress/emcompress-togo/https://github.com/dblalock/sprintzhttp://www.oberhumer.com/opensource/lzo/

author: Marco
author_rank: Super Contributor
timestamp: January 29, 2022, 06:00:54 pm
content: https://github.com/dblalock/sprintzForgot about that one, apart from FLAC it's the only ready made predictive coder mentioned.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: January 30, 2022, 09:13:30 am
content: Thanks guys for the Feedbacks, I should try some of the libraries that you mention, I have take a look at heatshrink, and it seems it's very easy to use.

author: mfro
author_rank: Regular Contributor
timestamp: January 30, 2022, 09:32:36 am
content: Your data appears scrambled (a trailing 1 sometimes)?I would fix that first before attempting to compress.

author: brucehoult
author_rank: Super Contributor
timestamp: January 30, 2022, 11:30:25 am
content: I had a go at your data file (after deleting any weird 2nd value on a line).I tried various combinations of a pre-processing stage, then an off the shelf compressor.Code:text file752792 raw633331 lz4446806 lz4 -9342929 gzip341785 gzip -9127860 bzip286559 zstd54196 xz24 bit binary307179 raw307198 lz4305236 lz4 -9284483 gzip284483 gzip -9100742 bzip271421 zstd53912 xzLEB128306495 raw290823 gzip -9100382 bzip273228 zstd57348 xzLEB128 1st diffs257751 raw65511 lz464077 lz4 -9232925 gzip232925 gzip -976949 bzip258545 zstd54508 xzLEB128 2nd diffs201742 raw51276 lz451261 lz4 -9191888 gzip64178 bzip248041 zstd43144 xzLEB128 3rd diffs201426 raw51196 lz451182 lz4 -963699 bzip247324 zstd42284 xzThe patterns:- the heavyweight compressors zstd and especially xz got pretty much the same results no matter what form the data started off in.- gzip for some unknown reason *really* doesn't like LEB128 encoding.- lz4 for some unknown reason likes LEB128 encoding a lot.- 1st diffs gets 75%-80% reduction in size.- 2nd diffs gets another 20% off.- 3rd diffs probably not worth it. It also risks blowing up on data that is less smooth.A combination of 2nd diffs and the very lightweight and simple lz4 compressor knocks 83% off the file size.Source code:Code:#include <stdio.h>#include <stdint.h>void LEB128(int32_t n){int more = 1, isNeg = n < 0;while (more) {int byte = n & 127;n >>= 7;if (isNeg) n |= (~0U << (32-7)); // in case was not arithetic shiftif (n == 0 && !(byte & 0x40) || n == -1 && (byte & 0x40)) {more = 0;} else {byte |= 0x80;}putchar(byte);}}int main(){int32_t d0 = 0, d1 = 0;int32_t new_d0, new_d1, new_d2;while (scanf("%d", &new_d0) != EOF){new_d1 = new_d0 - d0; d0 = new_d0;new_d2 = new_d1 - d1; d1 = new_d1;LEB128(new_d2);}return 0;}

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: February 01, 2022, 08:00:50 am
content: Bruce thanks for the tips, I never heard of xz, and it seems very impressive,where is the Github page for that? is it CPU intensive?Also Since you have a lot of experience in FPGA, I have about 5K free Lut's space in my Gowin FPGA, before sending the uncompressed data to the MCU, can we do something in the FPGA too?

author: brucehoult
author_rank: Super Contributor
timestamp: February 01, 2022, 08:23:13 am
content: Quote from: ali_asadzadeh on February 01, 2022, 08:00:50 amxz is the current benchmark compressor but it is *extremely* resource-intensive for compression. It can be ok to decompress it in embedded, but I don't think you could usefully do compression.xz is usually used where the same compressed file will be downloaded by millions of people and it's worth spending a lot of CPU time and RAM on compressing it to the maximum.Several Linux distros have switched to using zstd because it compresses much faster than xz (and also compresses much faster), but is only a little worse compression. But it's still very resource-intensive by embedded standards.THAT'S WHY my recommendation is to use the embedded-friendly lz4 compressor, with the custom preprocessing I showed.QuoteI'm not experienced in FPGAs.

author: ali_asadzadeh
author_rank: Super Contributor
timestamp: February 01, 2022, 09:51:27 am
content: Thanks, Where do you check the current benchmark compressor list?

author: brucehoult
author_rank: Super Contributor
timestamp: February 01, 2022, 11:20:31 am
content: Quote from: ali_asadzadeh on February 01, 2022, 09:51:27 amYou run different compressors on your own data.I suppose someone does this on some standard corpus from time to time. I don't know anything about that. Using your own data is better.Once upon a time I guess comp.compression was the place. I haven't looked there for a couple of decades.