only follow links that are dofollow
abort if page has an noindex tag