Add upgrade path

How's this look for upgrading? I still need to fix tests.

/cc @josh @NKoz rails/rails#11168