crawler defaults to only domains and subdomains, not subpages