one possibility here is to not query for a specific job but a user and then use that as a way of determining which jobs have finished. this might speed up things quite a bit.

yes make throttled queries, but when you do process the output to record all jobs that have finished.