Corvuscorax/treefiles

Hi Wouter,

I sent you a message regarding this on sourceforge on Friday, I don't know if you had a chance to read it yet :)
Anyway, I remembered that I do indeed have a github account already so I did a quick test if I these changes merge with your current master branch without issues, which they did :)
I also fixed a mistake I did in the trim() implementation, so please do not use the patch linked in that sourceforge email.

This branch adds the "treefiles" feaure. It allow the user to host a "network block device" which has a local representation as many small files (in fact one file per block) which are auto-generated on demand and stored in a hierarchical subdirectory tree. (to limit the amount of files/directories per subdirectory)

While likely not offering any performance benefit under normal circumstances, this might be a huge benefit when operating network block device from non-standard or esoteric filesystems (FUSE/... , ftpfs, sshfs, davfs, ...). Many of these network filesystems have only limited support to binary access parts of a file and instead require entire files to be sent back and forth to and from a server. To allow remote storage of a network block device in a cloud, it would be desirable that each block is sent back to the server once its written, and not only when the file handle is closed, as these filesystems often do, which would synchronize the entire block device only on unmount with a high risk of data-loss.
Storing one block per file, or one file per block does exactly that.

Treefiles support "sparse" storing, where untouched blocks simply do not exist - consequently the "exptrim()" request has been implemented in such a way that affected block files are deleted, such allowing sparse storage even on underlying filesystems that have no support for sparse storage themselves.

As such this feature can also be used for compressed storage of very large (but sparsely used) blockfiles that exceed the underlying storage space by far. As such it is also useful tool to simulate extremely large storage containers for tests of filesystem and other block device maintenance tools.

In READ_ONLY mode, no directories or block files are created (the underlying filesystem remains untouched) - non-existing blocks will always be simulated to read as all zeroes.
In RDRW mode, new blocks will be created on first access (as well as their containing directories) and truncated to BLOCKSIZE bytes (4096) - depending on the underlying file- and operatingsystem, thus spawned files may contain random data.


here too