updated page comments
no longer try to crawl non crawlable links.. pdf, exe