make each job spawn multiple processes (shard per process)
better logging