Repository: chaoss/augur

Issue:
repo_info_worker rate limit exceeded error
**Describe the bug:**
When the rate limit on our GitHub API key is exceeded, the repo_info_error does not handle that and wait an hour to attempt again. 

It is notable that this error did occur during a time, second or 10th (not sure) run of the repo_info worker on a repository. 

**To Reproduce:**
Steps to reproduce the behavior:
1. Run the repo_info_worker using a GitHub API key that has exceeded its rate limit. 
2. The error appears to be a difference in how the differences in timestamps are specifically calculated in the worker if you look at the logs, below. 

[Example logs from github_worker](https://pastebin.com/pE71B71R)

**Expected behavior:**
1. Worker will pause for 60 minutes since its last request. 



Discussion:
@parthsharma2 and @gabe-heim : The issue does not insert "bad data", but there is a potential that we will miss data collection windows. I think the logic around waiting that's in the git_hub_worker for issues appears sound. If @parthsharma2 is able to look at that and refactor the repo_info_worker it could be beneficial.
I'm also noting the the repo_info_worker is hitting the graphql (API v4) endpoints. It *may* be the case that the data types returned are not the same for the v3 and v4 GitHub API implementations of date or timestamp, as that's the error being specifically thrown. 
@parthsharma2 : Update: While bad data is not inserted, the way the current endpoint is constructed for summaries by repo_group, by collecting only on a subset of repositories, the repo_group summary for the most recent collection is thrown off. @gabe-heim : That's the root cause.  I deleted the 42 records of 112 gathered in one instance most recently before we hit the rate limit and this fixed the issue. Temporarily.  Its still important that we get the worker logic sorted out. 
@sgoggins I think this is most probably happening because the v3 & v4 GitHub API have different rate limits.

Refer: https://developer.github.com/v3/rate_limit/

I think one solution is to only use the v4 API for gathering all data. The worker uses the v3 API for getting the `commit_count` and `commiters_count`. We could probably get `commit_count` from the v4 API but as far as I know, getting `commiters_count` from the v4 API is not possible yet.

Another solution could be to have the worker identify & maintain the rate limits of both the v3 & v4 APIs and go to sleep whenever any of the rate limits dips below 1.
IMO we should maintain both rate limits, otherwise the worker will skip past the v4 data we need while still thinking it completed the task

Resolving patch:
diff --git a/workers/repo_info_worker/repo_info_worker/worker.py b/workers/repo_info_worker/repo_info_worker/worker.py
--- a/workers/repo_info_worker/repo_info_worker/worker.py
+++ b/workers/repo_info_worker/repo_info_worker/worker.py
@@ -373,7 +373,7 @@ def query_committers_count(self, owner, repo):
                 else:
                     url = r.links['next']['url']
         except Exception:
-            logger.exceptioin('An error occured while querying contributor count')
+            logger.exception('An error occured while querying contributor count')
 
         return committers
 
@@ -414,7 +414,7 @@ def update_rate_limit(self, response):
         logger.info(f"[Rate Limit]: Updated rate limit, you have: {self.rate_limit} requests remaining")
         if self.rate_limit <= 0:
             reset_time = response.headers['X-RateLimit-Reset']
-            time_diff = datetime.datetime.fromtimestamp(int(reset_time)) - datetime.datetime.now()
+            time_diff = datetime.fromtimestamp(int(reset_time)) - datetime.now()
             logger.info(f"[Rate Limit]: Rate limit exceeded, waiting {time_diff.total_seconds()} seconds")
             time.sleep(time_diff.total_seconds())
             self.rate_limit = int(response.headers['X-RateLimit-Limit'])


Test patch: