Repository: chaoss/augur

Issue:
Unique Key on Pull Request Events is Blocking a Lot of Data from being Inserted
Please help us help you by filling out the following sections as thoroughly as you can.

**Description:**
The unique key of (pr_platform_event_id, platform_id) is not allowing for many Issue events to be inserted. The platform_id is fine to be in the key. Although I would comment that all the tables I have seen only include data from one platform (Github). The issue is that the pr_platform_event_id is the issue src id for that event. This means that every row in the table must have a unique issue src id, which is making the events table only include one event for every single pull request. Since all the events on a pull request will have the same issue src id and the same platform id. 

The way to fix this is to change the unique key to be (node_id, platform_id)



**Log files**
Attach the relevant log files here. Server and installation logs can be found in the `logs/` directory in the root `augur/` directory, and the logs for each worker are stored in their respective directories. If the logs are pretty long (> 50ish lines or just use your best judgement) please use a Gist or a [pastebin](https://pastebin.com/). **These logs file are required if you would like help solving your issue.**

**Software versions:**
 - Augur: (you can use `pip show augur` to find your version)
 - OS: (`sw_vers` for macOS, `lsb_release -a` on Linux)
 - Browser: (if applicable)


Discussion:

Resolving patch:
diff --git a/workers/worker_git_integration.py b/workers/worker_git_integration.py
--- a/workers/worker_git_integration.py
+++ b/workers/worker_git_integration.py
@@ -885,7 +885,7 @@ def query_gitlab_contributors(self, entry_info, repo_id):
             try:
                 cntrb_compressed_url = ("https://gitlab.com/api/v4/users?search=" + repo_contributor['email'])
                 self.logger.info("Hitting endpoint: " + cntrb_compressed_url + " ...\n")
-                r = requests.get(url=cntrb_compressed_url, headers=self.headers, timeout=(4.44, 40.01))
+                r = requests.get(url=cntrb_compressed_url, headers=self.headers, timeout=(30.44, 70.01))
                 contributor_compressed = r.json()
 
                 email = repo_contributor['email']
@@ -897,7 +897,7 @@ def query_gitlab_contributors(self, entry_info, repo_id):
 
                 cntrb_url = ("https://gitlab.com/api/v4/users/" + str(contributor_compressed[0]["id"]))
                 self.logger.info("Hitting end point to get complete contributor info now: " + cntrb_url + "...\n")
-                r = requests.get(url=cntrb_url, headers=self.headers, timeout=(4.44, 40.01))
+                r = requests.get(url=cntrb_url, headers=self.headers, timeout=(30.44, 70.01))
                 contributor = r.json()
 
                 cntrb = {
@@ -1170,7 +1170,7 @@ def paginate_endpoint(
             while num_attempts < 10:
                 self.logger.info(f"Hitting endpoint: {url.format(page_number)}...\n")
                 try:
-                    response = requests.get(url=url.format(page_number), headers=self.headers, timeout=(4.44, 40.01))
+                    response = requests.get(url=url.format(page_number), headers=self.headers, timeout=(20.24, 50.01))
                 except TimeoutError as e:
                     self.logger.info("Request timed out. Sleeping 10 seconds and trying again...\n")
                     time.sleep(10)
@@ -1352,7 +1352,7 @@ def paginate(self, url, duplicate_col_map, update_col_map, table, table_pkey, wh
             success = False
             while num_attempts < 3:
                 self.logger.info(f'Hitting endpoint: {url.format(i)}...\n')
-                r = requests.get(url=url.format(i), headers=self.headers, timeout=(4.44, 40.01))
+                r = requests.get(url=url.format(i), headers=self.headers, timeout=(10.59, 70.21))
 
                 self.update_rate_limit(r, platform=platform)
                 if 'last' not in r.links:


Test patch: