More fixes based on analysis of jobs running in production

Added a restart of the job if BQ reports an internal error
Added paging to the getBucketItems API; paging is used by the Job to gather more files if needed

@tejohnso @alexey-rise please review. Thanks.