DSpace/DSpace — pull request #764 by bram-atmire (2014-11-27 10:46:17+00:00)
Title: DS-2335 Add more default blocks for certain spiders in robots.txt

- moving up the block to disallow /discover and /search-filter because it has to come immediately after the user-agent declaration. (whitelines break the user agent block)

- adding blocks for bad behaving bots, based on wikipedia's robots.txt file