in directory mode we use workers gem to parralelize the processing