opscode/bookshelf — pull request #26 by manderson26 (2014-10-13 20:02:24+00:00)
Title: Fix a directory leak when deleting files from bookshelf

No tests yet, because we don't seem to test deletion at all in bookshelf. Also there is some tech debt buried here, since we use the crypto_sha and crypto_md5, which are deprecated.

To avoid breaking filesystems with large directory issues, we create a hierarchy of directories to
hold the actual files stored. But we never delete the directories.

This creates a problem, because we potentially create a directory for every file we store in
bookshelf. This can grow pretty large (4 levels deep, with 256 entries per level, so 2^32 + 2^24 +
2^16 + 2^8 > 4 billion). If we never delete them we will blow out the inode table, and consume
considerable disk space. For example ext4 will most likely allocate a full block (default 4KiB) for
each directory we create due to the long file names for the files we store
(https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout)