Added a do not crawl list based on regular expressions
Reduced computations done while visited_url cache is locked and added support for the re based do not crawl list