Add sample lists of spiders from DSpace 4.0
Disable user-agent filtering

The filtering by user-agent was not working, so temporarily disabled it.
Merge pull request #610 from ryscher/clean-spiders

Temporarily disable spider filtering by user-agent