Split config into a separate file and add sanity check and test

Chasing #1246 

/cc @philips @unihorn @xiangli-cmu 
well... we do not have to check if is not a new cluster. We explicitly say that we will ignore the cluster configuration if we restarts an existing cluster.