Crawler will test urls before add to queue