DS-2335 Add more default blocks for certain spiders in robots.txt

- moving up the block to disallow /discover and /search-filter because it has to come immediately after the user-agent declaration. (whitelines break the user agent block)

- adding blocks for bad behaving bots, based on wikipedia's robots.txt file

https://jira.duraspace.org/browse/DS-2335