(PUP-1699) Add caching for directory based environments

This also corrects a problem in the benchmark "many modules" - the
intention
was to measure directory based environments but it ended up measuring
legacy based.