Repository: mozilla/bugbug

Issue:
Implementation for the hook to start the pipeline
Depends on #283.

After we have the hook, we have to implement its logic.

The logic will be kind of a decision task (similar to https://github.com/marco-c/breakpad-win-update-symbols/blob/3af1618c61f57f3ff87bf5f68e595d899316a8a2/run-taskcluster.py#L94), which will trigger a taskgraph (data retrieval -> training -> deployment of http service with trained models).
Pipeline hook should be also triggered when a new release is made
See also #283.


Discussion:

Here's a possible simple implementation:
- the deployment task sends a pulse message when it finishes;
- the hook adds the pulse message to its possible triggers.

Resolving patch:
diff --git a/infra/spawn_data_pipeline.py b/infra/spawn_data_pipeline.py
new file mode 100644
--- /dev/null
+++ b/infra/spawn_data_pipeline.py
@@ -0,0 +1,123 @@
+#!/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright 2019 Mozilla
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+This script triggers the data pipeline for the bugbug project
+"""
+
+import argparse
+import os
+import sys
+
+import jsone
+import requests.packages.urllib3
+import taskcluster
+import yaml
+
+requests.packages.urllib3.disable_warnings()
+
+TASKCLUSTER_DEFAULT_URL = "https://taskcluster.net"
+
+
+def get_taskcluster_options():
+    """
+    Helper to get the Taskcluster setup options
+    according to current environment (local or Taskcluster)
+    """
+    options = taskcluster.optionsFromEnvironment()
+    proxy_url = os.environ.get("TASKCLUSTER_PROXY_URL")
+
+    if proxy_url is not None:
+        # Always use proxy url when available
+        options["rootUrl"] = proxy_url
+
+    if "rootUrl" not in options:
+        # Always have a value in root url
+        options["rootUrl"] = TASKCLUSTER_DEFAULT_URL
+
+    return options
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Spawn tasks for bugbug data pipeline")
+    parser.add_argument("data_pipeline_json")
+
+    args = parser.parse_args()
+    decision_task_id = os.environ.get("TASK_ID")
+    options = get_taskcluster_options()
+    add_self = False
+    if decision_task_id:
+        add_self = True
+        task_group_id = decision_task_id
+    else:
+        task_group_id = taskcluster.utils.slugId()
+    keys = {"taskGroupId": task_group_id}
+
+    id_mapping = {}
+
+    # First pass, do the template rendering and dependencies resolution
+    tasks = []
+
+    with open(args.data_pipeline_json) as pipeline_file:
+        raw_tasks = yaml.load(pipeline_file.read())
+
+    for task in raw_tasks["tasks"]:
+        # Try render the task template
+        context = {}
+        payload = jsone.render(task, context)
+
+        # We need to generate new unique task ids for taskcluster to be happy
+        # but need to identify dependencies across tasks. So we create a
+        # mapping between an internal ID and the generate ID
+
+        task_id = taskcluster.utils.slugId()
+        task_internal_id = payload.pop("ID")
+
+        if task_internal_id in id_mapping:
+            raise ValueError("Conflicting IDs {}".format(task_internal_id))
+
+        id_mapping[task_internal_id] = task_id
+
+        for key, value in keys.items():
+            payload[key] = value
+
+        # Process the dependencies
+        new_dependencies = []
+        for dependency in payload.get("dependencies", []):
+            new_dependencies.append(id_mapping[dependency])
+
+        if add_self:
+            new_dependencies.append(decision_task_id)
+
+        payload["dependencies"] = new_dependencies
+
+        tasks.append((task_id, payload))
+
+    # Now sends them
+    queue = taskcluster.Queue(options)
+    try:
+        for task_id, task_payload in tasks:
+            queue.createTask(task_id, task_payload)
+
+        print("https://tools.taskcluster.net/task-group-inspector/#/" + task_group_id)
+    except taskcluster.exceptions.TaskclusterAuthFailure as e:
+        print("TaskclusterAuthFailure: {}".format(e.body), file=sys.stderr)
+        raise
+
+
+if __name__ == "__main__":
+    main()


Test patch: