Allow ability to serve static files
Move index tasks out into own rake file
Refactor file paths making project root accessible
Move sitemap generation into a Rake file.
Ignore generated .xml files
Create/update symlink to sitemap.xml.
Pull scan query scrolling out into its own class.
Add an enumerator for all document links.

This should mean we can iterate through the links to generate sitemaps
much more quickly, because we aren't loading in the entire document
bodies from elasticsearch and we aren't taking the time to construct
Document objects.
Re-factor sitemap generation

This will allow us to write tests and provides clearer separation of concerns.
Merge branch 'link-enumerator' into sitemap
Optimise retrieval of all_documents

Using an enumerator to avoid converting to an array.
Use all_document_links which returns links only, rather than the entire document.
Only include the site root once. Good.
Don't include logs
Cronjob to generate our sitemap everynight.
Serve sitemap.xml files from public/system folder.

When deploying rummager, we don't want to annihilate the folder that the sitemaps live in, nor do we want to have to regenerate them on every deploy. SO, let us use the system shared folder instead that is symlinked, this ensures it stays there.
Pesky extra space.
Refactor for tests, add tests, win at tests.
Merge branch 'master' into sitemap

Conflicts:
	lib/elasticsearch/index.rb
Merge pull request #85 from alphagov/sitemap

Sitemap