only follow links that are not nofollow
only index pages that may be indexed