Mpi parallel

An example of using MPI for parameter sync. Another critical part for implementing parallel training is splitting training data to each worker or NFS.