Copy over old configurations.

Done as per the discussion on rails/rails#16179

cc @rafaelfranca