You will be asking a lot of the services they offer, and if something may go wrong they like to hear it from you first, not from external people. Don't run repeated tests on remote servers. Run a number of servers locally and use them first to test your robot. After doing a small run, analyze your performance and your results, and estimate how they will scale with thousands of documents. It may soon be obvious you don't have the resources to continue with your present method.
Robots consume a lot of resources.
Walk, Don't Run. Instead, put a sleep in, or if you're clever, rotate queries among different servers in a round-robin fashion. Retrieving one document per minute is much better than one per second. One every five minutes is better still. Use HEAD Where Possible.