Given those starting points a robot can select URLs to visit and index, and to parse and use as a source for new URLs. If an indexing robot knows about a document, it may decide to parse it, and insert it into its database. Some robots index the HTML Titles, or the first few paragraphs, or parse the entire HTML and index all words, with weightings depending on HTML constructs, etc. Some parse the META tag, or other special hidden tags. We hope that as the Web evolves more facilities becomes available to efficiently associate meta data such as indexing information with a document.
Most services have a link to a URL submission form on their search page.
You can check your server logs for sites that retrieve many documents, especially in a short time. First of all check if it is a problem by checking the load of your server, and monitoring your servers' error log, and concurrent connections if you can. If you have a medium or high performance server, it is quite likely to be able to cope a high load of even several requests per second, especially if the visits are quick. These problems manifest themselves in refused connections, a high load, performance slowdowns, or in extreme cases a system crash. If this happens, there are a few things you should do.