For an existing project, on the Tools menu, click Project Options , then click the General tab. Click the Verify Offsite Links option to select it. Set additional options on this tab, or click OK . To ignore robot protocols When Content Analyzer explores an entire site, it operates as a spider, or robot. That is, it requests many Web pages far more quickly than you could explore the pages individually.
Many popular sites use robot protocols to limit access by spiders.
This prevents the sites' servers from being overloaded with hits by spiders, which would slow browsing times for actual site visitors. By default, Content Analyzer will not explore any site areas that are set up to block spiders. To learn more about this option. For a new project, in the New Map dialog box, click the Options button, then click the General tab. For an existing project, on the Tools menu, click Project Options , then click the General tab.