Ideally, this selection should aim at improving the quality of the database by discarding junk and scanning only the premier web content. In reality, of course, this kind of discernment is impossible because there are no automatic programs smart enough to separate wheat from chaff. The only way to sort out anything is by placing some rather arbitrary restrictions. It's been claimed that the quota for Alta Vista's spider is not more than 600 documents per any single domain. It remains open to speculation whether other search engines employ similar sampling techniques or the size of their databases is limited only by their technical capacity.
All search engines allow users to add their URLs to the database for spidering.
Some of them retrieve submitted documents immediately, others schedule them for future scanning, but in any case this allows to at least make sure that your domain isn't missed. On the other hand, given the selective nature of spidering, it's not a bad idea to register at least all key pages of your site. Another important question is how often spiders update their databases by revisiting sites they've already indexed. This parameter varies significantly for different engines, with the update periods having been quoted from one week to several months. For example, HotBot returns nothing when queried with keywords in Russian, nor can it properly display summaries for documents in Russian.