Checkpoint of directory-based dataset aggregation
And the rest ...
Added unit test for directory-based aggregation
Tidyups ... mostly cosmetic