If you manage a federal World Wide Web site, you either learn how to use bots and webcrawlers to your advantage, or you ignore them and risk abuse to your site. That sounds harsh, but it's true.
Abuse comes in two forms.
Second, Web bots may treat your site as if it doesn't exist. Although bots index your site, your pages might not show up very high in queries to Web search services such as Excite or AltaVista. If you want to play safe but get noticed, you must play by the bots' rules. For safety, develop your Web pages on a server other than the one used for public access. The development server should mirror the public server but be located inside your firewall.