Repository: chaoss/augur

Issue:
Improved handling of redirects for orgs and repos
While the GitHub API automatically follows redirects and continues gathering data from the newly redirected location, this creates unexpected behavior for users. For example, someone looking in pivotal.osshealth.io for the repo group vmwarepivotallabs would think that the data is missing. However, they could find the data by looking for the pivotalservices repo group, which was the original name of the GitHub org. In this case, all of the repositories at https://github.com/pivotalservices have been redirected to https://github.com/vmwarepivotallabs

Where this gets tricky is that GitHub redirection seems to happen at the repository level, not at the org level. Because of the reirection, https://github.com/pivotalservices looks like an empty org, but you can see that if you go to one of the original repositories, like https://github.com/pivotalservices/binder, you'll be redirected to the corresponding repo in vmwarepivotallabs, in this case: https://github.com/vmwarepivotallabs/binder

In addition to displaying the new information correctly in the front end, the corresponding database fields will also need to be updated. For example: repo_groups.rg_name, repo_groups.description, repo_groups.website, repo.repo_path, repo.repo_name, repo.url, and probably others that I'm not noticing.

There are also a few different cases for redirection that should be considered. The above example talks about moving an entire org and all of the repos within it, which is probably the most complex case, but there are other simpler cases:

- A single repo moved to a new org. Example: http://github.com/Spring-projects/spring-batch-admin-samples to https://github.com/spring-attic/spring-batch-admin-samples
- A single repo renamed within the same org. Example: http://github.com/Pivotal-Field-Engineering/word_cloud-spring-kafka-k8s-workshop to https://github.com/Pivotal-Field-Engineering/word_cloud-spring-kafka-on-k8s




Discussion:
@sgoggins this is the other feature request we talked about. 

@ivanayov again, this is another one that you might be interested in working on?
Hi @geekygirldawn , Can I take this over?


@hqrshguptq : We would welcome the assistance! The development guide for Augur is being worked on as we speak, and there's a pull request open right now that @Nebrethar is reviewing (From @ccarterlandis ) that makes our docs a bit easier to follow. 

As long as the repository is only changing organizations, I think the process should: 
1. Queue the change in two tables in the `augur_operations` schema. 
2. These two tables should have the same structure as the `repo` and `repo_group` table in the `augur_data` schema PLUS one more value: `new_repo_git` in the `repo_change_queue` table (possible names for new tables) and `new_repo_group` in the `repo_group_change_queue`. 

The queue tables are necessary because, many workers are typically gathering data constantly, and unless we queue changes, we are likely to trip up workers if we handle this in real time. Augur doesn't maintain a central awareness of which worker is pulling data for which repo at any point in time, so in this way the WHEN we collect data is a non-deterministic function of new data size, and data collection order, and data collection frequency. 

This video is not Hollywood level, but it does give a nice overview of how the workers "work" : https://www.youtube.com/watch?v=0s--LS736Vg&feature=youtu.be

@gabe-heim , @ccarterlandis , @Nebrethar , and I are keen to support your activity here. And if you suggest another path to solution, we are completely open to it ... My aim here was only to surface factors to consider.  @geekygirldawn is the primary person to make happy here. :) THANK YOU!

@hqrshguptq : You should make a fork of augur, but work off the `dev` branch. The docs are bit more up to date there as well: 
https://oss-augur.readthedocs.io/en/dev/
Hi @hqrshguptq - I wanted to just check in to see if this was something you are still working on and see if you needed any additional info :) 
Assigning to @ivanayov since it looks like there hasn't been any progress, and she's offered to pick it up.
Hmmm, for some reason, I can't actually assign @ivanayov to this issue 🤷‍♀️ 
Weird, I can't either. Maybe she needs to be added to the CHAOSS organization, or given at least `Triage` access to Augur repo as an outside collaborator?
@ccarterlandis AFAIR I saw the invitation after it had expired, so couldn't join the org
@ivanayov just sent you another!
@ivanayov @sgoggins I was hoping to get a status update on this feature. This is something that's becoming pretty important for us because we've moved a bunch of pretty important projects from one org to another :) 

Resolving patch:
diff --git a/augur/config.py b/augur/config.py
--- a/augur/config.py
+++ b/augur/config.py
@@ -23,6 +23,10 @@
             "gitlab_api_key":"gitlab_api_key"
         },
         "Housekeeper": {
+            "update_redirects": {
+                "switch": 0,
+                "repo_group_id": 0
+            },
             "jobs": [
                 {
                     "all_focused": 1,
diff --git a/augur/housekeeper.py b/augur/housekeeper.py
--- a/augur/housekeeper.py
+++ b/augur/housekeeper.py
@@ -13,6 +13,7 @@
 from sqlalchemy import MetaData
 
 from augur.logging import AugurLogging
+from urllib.parse import urlparse
 
 import warnings
 warnings.filterwarnings('ignore')
@@ -27,6 +28,7 @@ def __init__(self, broker, augur_app):
         self._processes = []
         self.augur_logging = augur_app.logging
         self.jobs = deepcopy(augur_app.config.get_value("Housekeeper", "jobs"))
+        self.update_redirects = deepcopy(augur_app.config.get_value("Housekeeper", "update_redirects"))
         self.broker_host = augur_app.config.get_value("Server", "host")
         self.broker_port = augur_app.config.get_value("Server", "port")
         self.broker = broker
@@ -46,6 +48,10 @@ def __init__(self, broker, augur_app):
         rs = pd.read_sql(repoUrlSQL, self.db, params={})
         all_repos = rs['repo_git'].values.tolist()
 
+        # If enabled, updates all redirects of repositories 
+        # and organizations urls for configured repo_group_id
+        self.update_url_redirects()
+
         # List of tasks that need periodic updates
         self.schedule_updates()
 
@@ -336,3 +342,80 @@ def prep_jobs(self):
                 job['repos'] = rs
             # time.sleep(120)
 
+    def update_url_redirects(self):
+        if 'switch' in self.update_redirects and self.update_redirects['switch'] == 1 and 'repo_group_id' in self.update_redirects:
+            repos_urls = self.get_repos_urls(self.update_redirects['repo_group_id'])
+            for url in repos_urls:
+                r = requests.get(url)
+                check_for_update = url != r.url
+                if check_for_update:
+                    self.update_repo_url(url, r.url, self.update_redirects['repo_group_id'])
+
+    def get_repos_urls(self, repo_group_id):
+        repos_sql = s.sql.text("""
+                SELECT repo_git FROM repo
+                WHERE repo_group_id = ':repo_group_id'
+            """)
+
+        repos = pd.read_sql(repos_sql, self.db, params={'repo_group_id': repo_group_id})
+
+        if len(repos) == 0:
+            logger.info("Did not find any repositories stored in augur_database for repo_group_id {}\n".format(repo_group_id))
+
+        return repos['repo_git']
+
+    def update_repo_url(self, old_url, new_url, repo_group_id):
+        old_repo_path = Housekeeper.parseRepoName(old_url)
+        old_repo_group_name = old_repo_path[0]
+        new_repo_path = Housekeeper.parseRepoName(new_url)
+        new_repo_group_name = new_repo_path[0]
+
+        if old_repo_group_name != new_repo_group_name:
+            # verifying the old repo group name is available in the database
+            old_rg_name_sql = s.sql.text("""
+                SELECT rg_name FROM repo_groups
+                WHERE repo_group_id = ':repo_group_id'
+            """)
+            old_rg_name_from_DB = pd.read_sql(old_rg_name_sql, self.db, params={'repo_group_id': repo_group_id})
+            if len(old_rg_name_from_DB['rg_name']) > 0 and old_repo_group_name != old_rg_name_from_DB['rg_name'][0]:
+                logger.info("Incoming old repo group name doesn't match the DB record for repo_group_id {} . Incoming name: {} DB record: {} \n".format(repo_group_id, old_repo_group_name, old_rg_name_from_DB['rg_name'][0]))
+
+            # checking if the new repo group name already exists and
+            # inserting it in repo_groups if it doesn't
+            rg_name_check_sql = s.sql.text("""
+                    SELECT rg_name, repo_group_id FROM repo_groups
+                    WHERE rg_name = :new_repo_group_name
+                """)
+            rg_name_check = pd.read_sql(rg_name_check_sql, self.db, params={'new_repo_group_name': new_repo_group_name})
+            new_rg_name_already_exists = len(rg_name_check['rg_name']) > 0
+
+            if new_rg_name_already_exists:
+                new_repo_group_id = rg_name_check['repo_group_id'][0]
+            else:
+                insert_sql = s.sql.text("""
+                        INSERT INTO repo_groups("rg_name", "rg_description", "rg_website", "rg_recache", "rg_last_modified", "rg_type", "tool_source", "tool_version", "data_source", "data_collection_date")
+                        VALUES (:new_repo_group_name, '', '', 0, CURRENT_TIMESTAMP, 'Unknown', 'Loaded by user', '1.0', 'Git', CURRENT_TIMESTAMP) RETURNING repo_group_id;
+                    """)
+                new_repo_group_id = self.db.execute(insert_sql, new_repo_group_name=new_repo_group_name).fetchone()[0]
+                logger.info("Inserted repo group {} with id {}\n".format(new_repo_group_name, new_repo_group_id))
+
+            update_sql = s.sql.text("""
+                    UPDATE repo SET repo_git = :new_url, repo_path = NULL, repo_name = NULL, repo_status = 'New', repo_group_id = :new_repo_group_id
+                    WHERE repo_git = :old_url
+                """)
+            self.db.execute(update_sql, new_url=new_url, new_repo_group_id=new_repo_group_id, old_url=old_url)
+            logger.info("Updated repo url from {} to {}\n".format(new_url, old_url))
+
+        else:
+            update_sql = s.sql.text("""
+                UPDATE repo SET repo_git = :new_url, repo_path = NULL, repo_name = NULL, repo_status = 'New'
+                WHERE repo_git = :old_url
+            """)
+            self.db.execute(update_sql, new_url=new_url, old_url=old_url)
+            logger.info("Updated repo url from {} to {}\n".format(new_url, old_url))
+
+    def parseRepoName(repo_url):
+        path = urlparse(repo_url).path
+        parts = path.split('/')
+        return parts[1:]
+


Test patch: