add method to check if user agent is a crawler