Then the spider and your server navigate a never-ending loop where a request for a page is met with a request for information. Getting a spider trapped inside your server is not just bad for the spider. The repeated requests for pages can crash the server. If you share server space with other Web sites and have a problem with site downtime , ask your Web host to check for CGI script problems on other sites. A page's actual URL address often poses a problem too because most dynamic page URL's contain query strings .
Search engines know about the problem, but most have shown very little interest in addressing it.
Infoseek and HotBot are the exception. Their search engine spiders can index dynamic page content, but don't do it automatically. You have to invite them in. You can also submit a static page that contains links to the dynamic URLs you need indexed. Infoseek's spider, called Slurp, will index dynamic pages that you submit, but won't crawl through your dynamic Web site by default.