Slightly better handling of error fetching index.
Add route for robots.txt.
Use default index if index cannot be fetched from imgur.