progress and logging is much better now...
some better handling of duplicates
progress, trying out a few things with queued requests, etc..