Repository: mozilla/bugbug

Issue:
Store pipeline tasks in the index, and update Taskcluster URLs in scripts
The scripts are still using the old URLs from releng-services.
Store pipeline tasks in the index, and update Taskcluster URLs in scripts
The scripts are still using the old URLs from releng-services.


Discussion:
Are we speaking about these URLs? https://github.com/mozilla/bugbug/blob/master/scripts/trainer.py#L18

I see these URLs used to retrieve models, but I'm not sure where configure where we push the artifacts, is it the `artifacts` part of the data-pipeline.yml?
The artifacts part only specifies the artifacts the task uploads.

The index can either be populated programmatically, or using a route (e.g. see the static analysis hook, https://tools.taskcluster.net/hooks/project-releng/services-production-staticanalysis%2Fbot, which has index.project.releng.services.project.production.static_analysis_bot.latest in the routes, which results in https://tools.taskcluster.net/index/project.releng.services.project.production.static_analysis_bot/latest).
Are we speaking about these URLs? https://github.com/mozilla/bugbug/blob/master/scripts/trainer.py#L18

I see these URLs used to retrieve models, but I'm not sure where configure where we push the artifacts, is it the `artifacts` part of the data-pipeline.yml?
The artifacts part only specifies the artifacts the task uploads.

The index can either be populated programmatically, or using a route (e.g. see the static analysis hook, https://tools.taskcluster.net/hooks/project-releng/services-production-staticanalysis%2Fbot, which has index.project.releng.services.project.production.static_analysis_bot.latest in the routes, which results in https://tools.taskcluster.net/index/project.releng.services.project.production.static_analysis_bot/latest).

Resolving patch:
diff --git a/http_service/download_models.py b/http_service/download_models.py
--- a/http_service/download_models.py
+++ b/http_service/download_models.py
@@ -14,7 +14,7 @@
 logging.basicConfig(level=logging.INFO)
 LOGGER = logging.getLogger()
 
-BASE_URL = "https://index.taskcluster.net/v1/task/project.releng.services.project.testing.bugbug_train.latest/artifacts/public"
+BASE_URL = "https://index.taskcluster.net/v1/task/project.relman.bugbug.train_{}.latest/artifacts/public"
 
 MODELS_NAMES = ("defectenhancementtask", "component", "regression")
 MODELS_DIR = os.path.join(os.path.dirname(__file__), "models")
@@ -26,7 +26,8 @@ def retrieve_model(name):
     file_name = f"{name}model"
     file_path = os.path.join(MODELS_DIR, file_name)
 
-    model_url = f"{BASE_URL}/{file_name}.xz"
+    base_model_url = BASE_URL.format(name)
+    model_url = f"{base_model_url}/{file_name}.xz"
     LOGGER.info(f"Checking ETAG of {model_url}")
     r = requests.head(model_url, allow_redirects=True)
     r.raise_for_status()
diff --git a/scripts/trainer.py b/scripts/trainer.py
--- a/scripts/trainer.py
+++ b/scripts/trainer.py
@@ -15,7 +15,7 @@
 basicConfig(level=INFO)
 logger = getLogger(__name__)
 
-BASE_URL = "https://index.taskcluster.net/v1/task/project.releng.services.project.testing.bugbug_data.latest/artifacts/public"
+BASE_URL = "https://index.taskcluster.net/v1/task/project.relman.bugbug.data_{}.latest/artifacts/public"
 
 
 class Trainer(object):
@@ -68,13 +68,15 @@ def go(self, model):
 
         # Bugs.json
         logger.info("Downloading bugs database")
-        urlretrieve(f"{BASE_URL}/bugs.json.xz", "data/bugs.json.xz")
+        bugs_url = BASE_URL.format("bugs")
+        urlretrieve(f"{bugs_url}/bugs.json.xz", "data/bugs.json.xz")
         logger.info("Decompressing bugs database")
         self.decompress_file("data/bugs.json")
 
         # Commits.json
         logger.info("Downloading commits database")
-        urlretrieve(f"{BASE_URL}/commits.json.xz", "data/commits.json.xz")
+        commits_url = BASE_URL.format("commits")
+        urlretrieve(f"{commits_url}/commits.json.xz", "data/commits.json.xz")
         logger.info("Decompressing commits database")
         self.decompress_file("data/commits.json")
 


Test patch: