Lock different evaluators from overwriting each others' directories.