LibraryOfCongress/chronam — pull request #111 by jechols (2014-11-12 21:46:33+00:00)
Title: Feature/reduce spidering

Simplifies the robots.txt rules and adds an index to disallow spidering of /images/tiles. In chronam core, /images/tiles isn't the only URL that generates tiles (see ticket #109 ), but I believe it is the primary URL for OpenSeadragon.

Unfortunately, any sites still using the old URL style (`/lccn/snxxxxxxxxx/YYYY-MM-DD/ed-#/seq-#/image_514x514_from_1022,4094_to_2050,5122.jpg`) won't benefit from this, but addressing #109, and having users update local themes, would allow this to work for everybody. (Or extra rules could be added to catch this style of URL, too)

It might seem strange to disallow bots from hitting the tiles, but Apache log analysis showed us that we were getting a huge amount of traffic from bots - including thumbnails and the dynamic tiles.