Temporarily disable spider filtering by user-agent

The code here was not properly reading the user-agent strings from the spider files. I am disabling this section of the code and re-starting work under https://atmire.com/tickets-nescent2/view-ticket?id=2062