Therefore, your documents should be captured and assimilated in a matter of days. If you are using a proxy server anything else that might cause your public documents to elude the spider, you will want to manually add your site by completing the Suggest A Site form. The spider will inventory all of your primary links and pages. This information will be compared to locator records in the IGI database. Replaced pages will be automatically added to our permanent public access database.
Illinois spider collects pages.
It begins by gathering information from a starting point page, typically the server's root page. It collects all the links on that page, then visits each of those pages. If the page meets our criteria, then that page is scanned for links, and those links are visited, and so on. This means that, for a page to be collected into our system, it must ultimately be available from the root, starting point page. The way to prevent an internal-use page from being spidered is to add meta tags that will block robot access.