In addition to HTML files, our spider indexes plain text, XML, PDF, and Postscript files. Most spiders will find text in alt tags if you include them, however. If your page depends on the browser interpreting some Java or Javascript code to locate the actual content it won't be found by spiders. Pages using frames can cause odd results. Pages using Flash and similar graphic-oriented products will require special care to insure that they are indexed appropriately.
Some spiders also look for meta tags containing additional information about a web page.
Indexing engines will only use a limited amount of text, so keep your description and keywords list concise. More complex metadata schemes have been proposed to allow more elaborate cataloging of web pages, although none have yet found wide acceptance. Most spiders follow a common Robots Exclusion Protocol which allows you to prevent some or all of your pages from being indexed. Our spider follows this protocol. You can use the Robots meta tag in a specific page to control how it is handled.