Split config into a separate file and add sanity check and test

Chasing #1246 

/cc @philips @unihorn @xiangli-cmu 
Happy to change it, but pointing out that it was a condition of the bug :)