take care of "#.*" links and unlimited crawl
unlimited crawl