alphagov/bouncer — pull request #117 by jamiecobbett (2014-12-18 11:07:49+00:00)
Title: Limit sitemaps to 50,000 entries

Bouncer's sitemaps include all redirect mappings from that site. This causes
three problems for sites with large numbers of redirect mappings:
* the request takes too long to process, so a 502 is served
* our memory usage jumps on each occasion for the largest sites
* we are violating the sitemap specification, which limits each file to 50,000
entries

The only downside to this change is that some redirects will not be included in
sitemaps, but we think this is better than not being able to serve a sitemap at
all, or Bouncer running out of memory.

If you need more than 50,000 entries, you should use multiple sitemap files,
like GOV.UK: https://www.gov.uk/sitemap.xml