alphagov/transition — pull request #76 by jamiecobbett (2013-08-21 11:49:15+00:00)
Title: Speed up load sh

We use this script to bring in the latest site, host and mapping data from redirector becuase it is currently the canonical source of mappings. At some point, the canonical source for that data will be Transition app.

Mappings need to be continuously available, but hits and totals can be missing if done out-of-hours.

Originally we dropped the database and inserted all the data. This was fast - ~1hr50m. Unfortunately it meant bouncer would stop working properly.

We then tried using 'robust' - updating everything in place. This took 11 hours, which is too much.

This mixed approach should get reasonable speed without breaking redirects.