Filter out bots and search engines before collating