Tim/sharepoint trade article

This while loop is something that you borrowed from your Parature FAQ work. In the absence of a listing of all documents that need to be imported, it guesses the URLs of the source docs, iterates over them and stops when it thinks it must have found them all. For Parature FAQ, it was the only solution that could work, but in general it's not a good solution. If the ID range grows beyond your 119 terminator, you'll miss docs. If you notice that docs are missing, you can only fix the issue with a code change (that needs to be deployed etc.). You also examine URLs of docs that may not exist, making the import process inefficient.

It would be way better if you could get a listing of all docs that exist on the S3 bucket, and iterate over them instead. I think that's possible to do, but I don't know how off the top of my head. I recommend that you look into that possibility, and if you're able to get such a list adjust the importer accordingly.