Add an alias plugin and set up the old URLs with it
Remove the alias plugin (moved to a separate repo)
Bring in the alias plugin as a submodule