jnovy/pxz — Issue #4: do not create temporary files
Author: eworm-de
Created: 2011-11-24 11:25:00+00:00
State: open

Ok, this is how pbzip2 does its job:

* read chunks of 900kB per process
* compress chunks
* write compressed chunks to the final file
* start with next chunk of 900kB

pxz does it differently:

* create temporary files in /tmp
* compress data and write to (already unlinked) files in /tmp
* write to final file if anything has been compressed

pbzip2 works fine for me, pxz does not. These are the problems:

* tmpfs is limited to RAM/2 by default. So pxz exits with error when the /tmp filesystem fills up.
* If I resize /tmp my swap gets filled up... That's not any better.
* The process takes even longer compared to "usual" xz compression when writing to slow block devices as data is written to the final file after the compression itself has finished.

Is there any reason why not to compress chunks of defined size in RAM and append to the final file directly?