Let's put this in it's own file, and also, let's think about a better name. DirectoryBatchedStore, maybe?