Another common problem site developers have with search engines is caused by frames. Fortunately this one's solvable. Crawlers tend to pick up only parent frames or subframes, which prevents them from registering entire sites correctly. The solution here is fairly simple. If you aren't, you may want to start, because this'll take care of crawlers as well.
Search engines are useful tools, but they still have limitations.
As more features become available on new browsers, it'll be up to the designers of spiders to incorporate the technology into their engines. In the meantime, stick to the current robots.txt and make use of the handy dandy meta tag , which is currently supported by HotBot and AltaVista . Inc. All Rights Reserved.