Start parallelization at the beginning of training.