Repository: jenkins-x/lighthouse
Pull request #813 (2020-06-21 18:54:15)

feat: Work towards supporting other engines besides jx
Switch to a common activity record that we'll convert `PipelineActivity`s to, and report back based on that. This will also end up allowing us to send activity records to external plugins.

Signed-off-by: Andrew Bayer <andrew.bayer@gmail.com>

jenkins-x-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **abayer**
You can assign the PR to them by writing `/assign @abayer` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=jenkins-x%2Flighthouse).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](jenkins-x/lighthouse/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["abayer"]} -->
jenkins-x-bot: @abayer: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
bbs | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/4) | `/test bbs`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jenkins-x-bot: @abayer: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
bbs | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/4) | `/test bbs`
gitlab | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/5) | `/test gitlab`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jenkins-x-bot: @abayer: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
bbs | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/4) | `/test bbs`
gitlab | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/5) | `/test gitlab`
github | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/3) | `/test github`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jenkins-x-bot: @abayer: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
bbs | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/4) | `/test bbs`
gitlab | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/5) | `/test gitlab`
github | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/3) | `/test github`
ghe | dd64c739442d505cf5381e2a14b60968e8a0d86e | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/2) | `/test ghe`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
abayer: /test github

Patch:
diff --git a/pkg/apis/lighthouse/v1alpha1/types.go b/pkg/apis/lighthouse/v1alpha1/types.go
index a124d263..31602af2 100644
--- a/pkg/apis/lighthouse/v1alpha1/types.go
+++ b/pkg/apis/lighthouse/v1alpha1/types.go
@@ -8,7 +8,6 @@ import (
 	"strings"
 	"time"
 
-	"github.com/jenkins-x/jx/v2/pkg/apis/jenkins.io/v1"
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
@@ -332,21 +331,3 @@ type ByNum []Pull
 func (prs ByNum) Len() int           { return len(prs) }
 func (prs ByNum) Swap(i, j int)      { prs[i], prs[j] = prs[j], prs[i] }
 func (prs ByNum) Less(i, j int) bool { return prs[i].Number < prs[j].Number }
-
-// ToPipelineState converts the PipelineActivity state to LighthouseJob state
-func ToPipelineState(status v1.ActivityStatusType) PipelineState {
-	switch status {
-	case v1.ActivityStatusTypePending:
-		return PendingState
-	case v1.ActivityStatusTypeAborted:
-		return AbortedState
-	case v1.ActivityStatusTypeRunning:
-		return RunningState
-	case v1.ActivityStatusTypeSucceeded:
-		return SuccessState
-	case v1.ActivityStatusTypeFailed, v1.ActivityStatusTypeError:
-		return FailureState
-	default:
-		return FailureState
-	}
-}
diff --git a/pkg/foghorn/controller.go b/pkg/foghorn/controller.go
index 59edd7e3..2a2f0feb 100644
--- a/pkg/foghorn/controller.go
+++ b/pkg/foghorn/controller.go
@@ -6,6 +6,7 @@ import (
 	"os/signal"
 	"reflect"
 	"strings"
+	"sync"
 	"syscall"
 	"text/template"
 	"time"
@@ -21,7 +22,9 @@ import (
 	clientset "github.com/jenkins-x/lighthouse/pkg/client/clientset/versioned"
 	lhinformers "github.com/jenkins-x/lighthouse/pkg/client/informers/externalversions/lighthouse/v1alpha1"
 	lhlisters "github.com/jenkins-x/lighthouse/pkg/client/listers/lighthouse/v1alpha1"
+	"github.com/jenkins-x/lighthouse/pkg/jx"
 	"github.com/jenkins-x/lighthouse/pkg/plugins"
+	"github.com/jenkins-x/lighthouse/pkg/record"
 	"github.com/jenkins-x/lighthouse/pkg/scmprovider"
 	"github.com/jenkins-x/lighthouse/pkg/scmprovider/reporter"
 	"github.com/jenkins-x/lighthouse/pkg/util"
@@ -68,6 +71,7 @@ type Controller struct {
 	jobConfig    *config.Agent
 	pluginConfig *plugins.ConfigAgent
 
+	wg     *sync.WaitGroup
 	logger *logrus.Entry
 	ns     string
 }
@@ -163,6 +167,8 @@ func NewController(kubeClient kubernetes.Interface, jxClient jxclient.Interface,
 		},
 	})
 
+	controller.wg = &sync.WaitGroup{}
+
 	return controller, nil
 }
 
@@ -269,7 +275,7 @@ func (c *Controller) syncHandler(key string) error {
 	}
 
 	// Get the PipelineActivity resource with this namespace/name
-	activity, err := c.activityLister.PipelineActivities(namespace).Get(name)
+	jxActivity, err := c.activityLister.PipelineActivities(namespace).Get(name)
 	if err != nil {
 		// The PipelineActivity resource may no longer exist, in which case we delete the associated LH job
 		// TODO: Actually delete.
@@ -282,10 +288,15 @@ func (c *Controller) syncHandler(key string) error {
 		return err
 	}
 
+	activityRecord, err := jx.ConvertPipelineActivity(jxActivity)
+	if err != nil {
+		return err
+	}
+
 	var job *v1alpha1.LighthouseJob
 
 	// Get all LighthouseJobs with the same owner/repo/branch/build/context
-	labelSelector, err := createLabelSelectorFromActivity(activity)
+	labelSelector, err := createLabelSelectorFromActivity(activityRecord)
 	possibleJobs, err := c.lhLister.LighthouseJobs(namespace).List(labelSelector)
 	if err != nil {
 		return err
@@ -298,7 +309,7 @@ func (c *Controller) syncHandler(key string) error {
 
 	// To be safe, find the job with the activity's name in its status.
 	for _, j := range possibleJobs {
-		if j.Status.ActivityName == activity.Name {
+		if j.Status.ActivityName == jxActivity.Name {
 			job = j
 		}
 	}
@@ -309,8 +320,8 @@ func (c *Controller) syncHandler(key string) error {
 
 	// Update the job's status for the activity.
 	jobCopy := job.DeepCopy()
-	c.updateJobStatusForActivity(activity, jobCopy)
-	c.reportStatus(namespace, activity, jobCopy)
+	c.updateJobStatusForActivity(activityRecord, jobCopy)
+	c.reportStatus(namespace, activityRecord, jobCopy)
 
 	currentJob, err := c.lhLister.LighthouseJobs(namespace).Get(jobCopy.Name)
 	if err != nil {
@@ -330,16 +341,15 @@ func (c *Controller) syncHandler(key string) error {
 	return nil
 }
 
-func (c *Controller) updateJobStatusForActivity(activity *jxv1.PipelineActivity, job *v1alpha1.LighthouseJob) {
-	activityState := v1alpha1.ToPipelineState(activity.Spec.Status)
-	if activityState != job.Status.State {
-		job.Status.State = activityState
+func (c *Controller) updateJobStatusForActivity(activity *record.ActivityRecord, job *v1alpha1.LighthouseJob) {
+	if activity.Status != job.Status.State {
+		job.Status.State = activity.Status
 	}
-	if activity.Spec.LastCommitSHA != job.Status.LastCommitSHA {
-		job.Status.LastCommitSHA = activity.Spec.LastCommitSHA
+	if activity.LastCommitSHA != job.Status.LastCommitSHA {
+		job.Status.LastCommitSHA = activity.LastCommitSHA
 	}
-	if activity.Spec.CompletedTimestamp != nil && activity.Spec.CompletedTimestamp != job.Status.CompletionTime {
-		job.Status.CompletionTime = activity.Spec.CompletedTimestamp
+	if activity.CompletionTime != nil && activity.CompletionTime != job.Status.CompletionTime {
+		job.Status.CompletionTime = activity.CompletionTime
 	}
 }
 
@@ -352,38 +362,35 @@ func RateLimiter() workqueue.RateLimitingInterface {
 	return workqueue.NewNamedRateLimitingQueue(rl, controllerName)
 }
 
-func createLabelSelectorFromActivity(activity *jxv1.PipelineActivity) (labels.Selector, error) {
+func createLabelSelectorFromActivity(activity *record.ActivityRecord) (labels.Selector, error) {
 	var selectors []string
 
-	if owner, ok := activity.Labels[util.ActivityOwnerLabel]; ok {
-		selectors = append(selectors, fmt.Sprintf("%s=%s", util.OrgLabel, owner))
+	if activity.Owner != "" {
+		selectors = append(selectors, fmt.Sprintf("%s=%s", util.OrgLabel, activity.Owner))
 	}
-	if repo, ok := activity.Labels[util.ActivityRepositoryLabel]; ok {
-		selectors = append(selectors, fmt.Sprintf("%s=%s", util.RepoLabel, repo))
+	if activity.Repo != "" {
+		selectors = append(selectors, fmt.Sprintf("%s=%s", util.RepoLabel, activity.Repo))
 	}
-	if branch, ok := activity.Labels[util.ActivityBranchLabel]; ok {
-		selectors = append(selectors, fmt.Sprintf("%s=%s", util.BranchLabel, branch))
+	if activity.Branch != "" {
+		selectors = append(selectors, fmt.Sprintf("%s=%s", util.BranchLabel, activity.Branch))
 	}
-	if buildNum, ok := activity.Labels[util.ActivityBuildLabel]; ok {
-		selectors = append(selectors, fmt.Sprintf("%s=%s", util.BuildNumLabel, buildNum))
+	if activity.BuildIdentifier != "" {
+		selectors = append(selectors, fmt.Sprintf("%s=%s", util.BuildNumLabel, activity.BuildIdentifier))
 	}
-	if context, ok := activity.Labels[util.ActivityContextLabel]; ok {
-		selectors = append(selectors, fmt.Sprintf("%s=%s", util.ContextLabel, context))
+	if activity.Context != "" {
+		selectors = append(selectors, fmt.Sprintf("%s=%s", util.ContextLabel, activity.Context))
 	}
 
 	return labels.Parse(strings.Join(selectors, ","))
 }
 
-func (c *Controller) reportStatus(ns string, activity *jxv1.PipelineActivity, job *v1alpha1.LighthouseJob) {
-	sha := activity.Spec.LastCommitSHA
-	if sha == "" && activity.Labels != nil {
-		sha = activity.Labels[jxv1.LabelLastCommitSha]
-	}
+func (c *Controller) reportStatus(ns string, activity *record.ActivityRecord, job *v1alpha1.LighthouseJob) {
+	sha := activity.LastCommitSHA
 
-	owner := activity.Spec.GitOwner
-	repo := activity.Spec.GitRepository
-	gitURL := activity.Spec.GitURL
-	activityStatus := activity.Spec.Status
+	owner := activity.Owner
+	repo := activity.Repo
+	gitURL := activity.GitURL
+	activityStatus := activity.Status
 	statusInfo := toScmStatusDescriptionRunningStages(activity, c.gitKind())
 
 	fields := map[string]interface{}{
@@ -393,10 +400,10 @@ func (c *Controller) reportStatus(ns string, activity *jxv1.PipelineActivity, jo
 		"gitRepo":     repo,
 		"gitSHA":      sha,
 		"gitURL":      gitURL,
-		"gitBranch":   activity.Spec.GitBranch,
+		"gitBranch":   activity.Branch,
 		"gitStatus":   statusInfo.scmStatus.String(),
-		"buildNumber": activity.Spec.Build,
-		"duration":    durationString(activity.Spec.StartedTimestamp, activity.Spec.CompletedTimestamp),
+		"buildNumber": activity.BuildIdentifier,
+		"duration":    durationString(activity.StartTime, activity.CompletionTime),
 	}
 	if gitURL == "" {
 		c.logger.WithFields(fields).Debugf("Cannot report pipeline %s as we have no git SHA", activity.Name)
@@ -435,7 +442,12 @@ func (c *Controller) reportStatus(ns string, activity *jxv1.PipelineActivity, jo
 		return
 	}
 
-	pipelineContext := activity.Spec.Context
+	// Trigger external plugins if appropriate
+	if external := util.ExternalPluginsForEvent(c.pluginConfig, util.LighthousePayloadTypeActivity, fmt.Sprintf("%s/%s", owner, repo)); len(external) > 0 {
+		go util.CallExternalPluginsWithActivityRecord(c.logger, external, activity, c.hmacToken(), c.wg)
+	}
+
+	pipelineContext := activity.Context
 	if pipelineContext == "" {
 		pipelineContext = "jenkins-x"
 	}
@@ -457,8 +469,8 @@ func (c *Controller) reportStatus(ns string, activity *jxv1.PipelineActivity, jo
 		targetURL := c.createReportTargetURL(defaultTargetURLTemplate, ReportParams{
 			Owner:      owner,
 			Repository: repo,
-			Branch:     activity.Spec.GitBranch,
-			Build:      activity.Spec.Build,
+			Branch:     activity.Branch,
+			Build:      activity.BuildIdentifier,
 			Context:    pipelineContext,
 			// TODO: Need to get the job URL base in here somehow. (apb)
 			BaseURL: strings.TrimRight(urlBase, "/"),
@@ -539,39 +551,36 @@ type reportStatusInfo struct {
 	runningStages string
 }
 
-func toScmStatusDescriptionRunningStages(activity *jxv1.PipelineActivity, gitKind string) reportStatusInfo {
+func toScmStatusDescriptionRunningStages(activity *record.ActivityRecord, gitKind string) reportStatusInfo {
 	info := reportStatusInfo{
 		description:   "",
 		runningStages: "",
 		scmStatus:     scm.StateUnknown,
 	}
-	switch activity.Spec.Status {
-	case jxv1.ActivityStatusTypeSucceeded:
+	switch activity.Status {
+	case v1alpha1.SuccessState:
 		info.scmStatus = scm.StateSuccess
 		info.description = "Pipeline successful"
-	case jxv1.ActivityStatusTypeRunning, jxv1.ActivityStatusTypePending:
+	case v1alpha1.RunningState, v1alpha1.PendingState:
 		info.scmStatus = scm.StateRunning
 		info.description = "Pipeline running"
-	case jxv1.ActivityStatusTypeError:
+	case v1alpha1.AbortedState:
 		info.scmStatus = scm.StateError
 		info.description = "Error executing pipeline"
-	case jxv1.ActivityStatusTypeNone:
-		info.scmStatus = scm.StatePending
-		info.description = "Pipeline triggered"
-	case jxv1.ActivityStatusTypeFailed:
+	case v1alpha1.FailureState:
 		info.scmStatus = scm.StateFailure
 		info.description = "Pipeline failed"
 	default:
 		info.scmStatus = scm.StateUnknown
 		info.description = "Pipeline in unknown state"
 	}
-	stagesByStatus := activity.StagesByStatus()
 
+	runningStages := activity.RunningStages()
 	// GitLab does not currently support updating description without changing state, so we need simple descriptions there.
 	// TODO: link to GitLab issue (apb)
-	if len(stagesByStatus[jxv1.ActivityStatusTypeRunning]) > 0 && gitKind != "gitlab" {
-		info.runningStages = strings.Join(stagesByStatus[jxv1.ActivityStatusTypeRunning], ",")
-		info.description = fmt.Sprintf("Pipeline running stage(s): %s", strings.Join(stagesByStatus[jxv1.ActivityStatusTypeRunning], ", "))
+	if len(runningStages) > 0 && gitKind != "gitlab" {
+		info.runningStages = strings.Join(runningStages, ",")
+		info.description = fmt.Sprintf("Pipeline running stage(s): %s", strings.Join(runningStages, ", "))
 		if len(info.description) > 63 {
 			info.description = info.description[:59] + "..."
 		}
@@ -650,6 +659,10 @@ func (c *Controller) createSCMToken(gitKind string) (string, error) {
 	return value, nil
 }
 
+func (c *Controller) hmacToken() string {
+	return os.Getenv("HMAC_TOKEN")
+}
+
 // stopper returns a channel that remains open until an interrupt is received.
 func stopper() chan struct{} {
 	stop := make(chan struct{})
diff --git a/pkg/jx/activity.go b/pkg/jx/activity.go
new file mode 100644
index 00000000..b51d9d17
--- /dev/null
+++ b/pkg/jx/activity.go
@@ -0,0 +1,89 @@
+package jx
+
+import (
+	"errors"
+
+	v1 "github.com/jenkins-x/jx/v2/pkg/apis/jenkins.io/v1"
+	"github.com/jenkins-x/lighthouse/pkg/apis/lighthouse/v1alpha1"
+	"github.com/jenkins-x/lighthouse/pkg/record"
+)
+
+// ToPipelineState converts the PipelineActivity state to LighthouseJob state
+func ToPipelineState(status v1.ActivityStatusType) v1alpha1.PipelineState {
+	switch status {
+	case v1.ActivityStatusTypePending, v1.ActivityStatusTypeNone:
+		return v1alpha1.PendingState
+	case v1.ActivityStatusTypeAborted:
+		return v1alpha1.AbortedState
+	case v1.ActivityStatusTypeRunning:
+		return v1alpha1.RunningState
+	case v1.ActivityStatusTypeSucceeded:
+		return v1alpha1.SuccessState
+	case v1.ActivityStatusTypeFailed, v1.ActivityStatusTypeError:
+		return v1alpha1.FailureState
+	default:
+		return v1alpha1.FailureState
+	}
+}
+
+// ConvertPipelineActivity converts a PipelineActivity from jx to an ActivityRecord
+func ConvertPipelineActivity(pa *v1.PipelineActivity) (*record.ActivityRecord, error) {
+	if pa == nil {
+		return nil, errors.New("pipeline activity is nil")
+	}
+
+	sha := pa.Spec.LastCommitSHA
+	if sha == "" && pa.Labels != nil {
+		sha = pa.Labels[v1.LabelLastCommitSha]
+	}
+
+	ar := &record.ActivityRecord{
+		Name:            pa.Name,
+		Owner:           pa.Spec.GitOwner,
+		Repo:            pa.Spec.GitRepository,
+		Branch:          pa.Spec.GitBranch,
+		BuildIdentifier: pa.Spec.Build,
+		LastCommitSHA:   sha,
+		BaseSHA:         pa.Spec.BaseSHA,
+		Context:         pa.Spec.Context,
+		GitURL:          pa.Spec.GitURL,
+		LogURL:          pa.Spec.BuildLogsURL,
+		Status:          ToPipelineState(pa.Spec.Status),
+		StartTime:       pa.Spec.StartedTimestamp,
+		CompletionTime:  pa.Spec.CompletedTimestamp,
+		Stages:          []*record.ActivityStageOrStep{},
+	}
+
+	for _, step := range pa.Spec.Steps {
+		if step.Kind == v1.ActivityStepKindTypeStage {
+			ar.Stages = append(ar.Stages, convertStage(step.Stage))
+		}
+	}
+
+	return ar, nil
+}
+
+func convertStage(paStage *v1.StageActivityStep) *record.ActivityStageOrStep {
+	stage := &record.ActivityStageOrStep{
+		Name:           paStage.Name,
+		Status:         ToPipelineState(paStage.Status),
+		StartTime:      paStage.StartedTimestamp,
+		CompletionTime: paStage.CompletedTimestamp,
+		Steps:          []*record.ActivityStageOrStep{},
+	}
+
+	for _, child := range paStage.Steps {
+		stage.Steps = append(stage.Steps, convertStep(child))
+	}
+
+	return stage
+}
+
+func convertStep(paStep v1.CoreActivityStep) *record.ActivityStageOrStep {
+	return &record.ActivityStageOrStep{
+		Name:           paStep.Name,
+		Status:         ToPipelineState(paStep.Status),
+		StartTime:      paStep.StartedTimestamp,
+		CompletionTime: paStep.CompletedTimestamp,
+	}
+}
diff --git a/pkg/launcher/helpers.go b/pkg/jx/helpers.go
similarity index 59%
rename from pkg/launcher/helpers.go
rename to pkg/jx/helpers.go
index 940c793f..92c9b48c 100644
--- a/pkg/launcher/helpers.go
+++ b/pkg/jx/helpers.go
@@ -1,22 +1,26 @@
-package launcher
+package jx
 
 import (
 	"fmt"
 	"os"
 	"runtime/debug"
 
+	jxclient "github.com/jenkins-x/jx/v2/pkg/client/clientset/versioned"
 	"github.com/jenkins-x/jx/v2/pkg/jxfactory"
 	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/jx/v2/pkg/util"
+	clientset "github.com/jenkins-x/lighthouse/pkg/client/clientset/versioned"
 	"github.com/jenkins-x/lighthouse/pkg/clients"
 	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
+	tektonclient "github.com/tektoncd/pipeline/pkg/client/clientset/versioned"
+	kubeclient "k8s.io/client-go/kubernetes"
 )
 
 // NewMetaPipelineClient creates a new client for the creation and application of meta pipelines.
 // The responsibility of the meta pipeline is to prepare the execution pipeline and to allow Apps to contribute
 // the this execution pipeline.
-func NewMetaPipelineClient(factory jxfactory.Factory) (metapipeline.Client, error) {
+func NewMetaPipelineClient(factory jxfactory.Factory) (metapipeline.Client, tektonclient.Interface, jxclient.Interface, kubeclient.Interface, clientset.Interface, string, error) {
 	if factory == nil {
 		logrus.Warnf("no jxfactory passed in to create metapipeline.Client: %s", string(debug.Stack()))
 		factory = jxfactory.NewFactory()
@@ -26,24 +30,16 @@ func NewMetaPipelineClient(factory jxfactory.Factory) (metapipeline.Client, erro
 	cfgHome := util.HomeDir()
 	err := os.MkdirAll(cfgHome, util.DefaultWritePermissions)
 	if err != nil {
-		return nil, errors.Wrapf(err, "failed to create jx home dir %s", cfgHome)
+		return nil, nil, nil, nil, nil, "", errors.Wrapf(err, "failed to create jx home dir %s", cfgHome)
 	}
 
-	tektonClient, jxClient, kubeClient, _, ns, err := clients.GetClientsAndNamespace(factory)
+	tektonClient, jxClient, kubeClient, lhClient, ns, err := clients.GetClientsAndNamespace(factory)
 	if err != nil {
-		return nil, err
+		return nil, nil, nil, nil, nil, "", err
 	}
-	/*
-		gitter := gits.NewGitCLI()
-		fileHandles := util.IOFileHandles{
-			Err: os.Stderr,
-			In:  os.Stdin,
-			Out: os.Stdout,
-		}
-	*/
 	client, err := metapipeline.NewMetaPipelineClientWithClientsAndNamespace(jxClient, tektonClient, kubeClient, ns)
 	if err == nil && client == nil {
-		return nil, fmt.Errorf("no metapipeline client created")
+		return nil, nil, nil, nil, nil, "", fmt.Errorf("no metapipeline client created")
 	}
-	return client, err
+	return client, tektonClient, jxClient, kubeClient, lhClient, ns, err
 }
diff --git a/pkg/launcher/launcher.go b/pkg/jx/launcher.go
similarity index 80%
rename from pkg/launcher/launcher.go
rename to pkg/jx/launcher.go
index 465ceee3..4d2f8f7c 100644
--- a/pkg/launcher/launcher.go
+++ b/pkg/jx/launcher.go
@@ -1,4 +1,4 @@
-package launcher
+package jx
 
 import (
 	"os"
@@ -6,9 +6,11 @@ import (
 
 	"github.com/jenkins-x/go-scm/scm"
 	jxclient "github.com/jenkins-x/jx/v2/pkg/client/clientset/versioned"
+	"github.com/jenkins-x/jx/v2/pkg/jxfactory"
 	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse/pkg/apis/lighthouse/v1alpha1"
 	clientset "github.com/jenkins-x/lighthouse/pkg/client/clientset/versioned"
+	launcher2 "github.com/jenkins-x/lighthouse/pkg/launcher"
 	"github.com/jenkins-x/lighthouse/pkg/util"
 	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
@@ -17,24 +19,31 @@ import (
 
 // launcher default launcher
 type launcher struct {
-	jxClient  jxclient.Interface
-	lhClient  clientset.Interface
-	namespace string
+	jxClient           jxclient.Interface
+	lhClient           clientset.Interface
+	metapipelineClient metapipeline.Client
+	namespace          string
 }
 
 // NewLauncher creates a new builder
-func NewLauncher(jxClient jxclient.Interface, lhClient clientset.Interface, namespace string) (PipelineLauncher, error) {
+func NewLauncher() (launcher2.PipelineLauncher, error) {
+	factory := jxfactory.NewFactory()
+	mpClient, _, jxClient, _, lhClient, namespace, err := NewMetaPipelineClient(factory)
+	if err != nil {
+		return nil, errors.Wrapf(err, "couldn't get metapipeline client")
+	}
 	b := &launcher{
-		jxClient:  jxClient,
-		lhClient:  lhClient,
-		namespace: namespace,
+		jxClient:           jxClient,
+		lhClient:           lhClient,
+		metapipelineClient: mpClient,
+		namespace:          namespace,
 	}
 	return b, nil
 }
 
 // Launch creates a pipeline
 // TODO: This should be moved somewhere else, probably, and needs some kind of unit testing (apb)
-func (b *launcher) Launch(request *v1alpha1.LighthouseJob, metapipelineClient metapipeline.Client, repository scm.Repository) (*v1alpha1.LighthouseJob, error) {
+func (b *launcher) Launch(request *v1alpha1.LighthouseJob, repository scm.Repository) (*v1alpha1.LighthouseJob, error) {
 	spec := &request.Spec
 
 	name := repository.Name
@@ -94,7 +103,7 @@ func (b *launcher) Launch(request *v1alpha1.LighthouseJob, metapipelineClient me
 		EnvVariables: spec.GetEnvVars(),
 	}
 
-	activityKey, tektonCRDs, err := metapipelineClient.Create(pipelineCreateParam)
+	activityKey, tektonCRDs, err := b.metapipelineClient.Create(pipelineCreateParam)
 	if err != nil {
 		return nil, errors.Wrap(err, "unable to create Tekton CRDs")
 	}
@@ -118,7 +127,7 @@ func (b *launcher) Launch(request *v1alpha1.LighthouseJob, metapipelineClient me
 		return nil, errors.Wrapf(err, "unable to set status on LighthouseJob %s", appliedJob.Name)
 	}
 
-	err = metapipelineClient.Apply(activityKey, tektonCRDs)
+	err = b.metapipelineClient.Apply(activityKey, tektonCRDs)
 	if err != nil {
 		return nil, errors.Wrap(err, "unable to apply Tekton CRDs")
 	}
diff --git a/pkg/keeper/githubapp/factory.go b/pkg/keeper/githubapp/factory.go
index 273d26a2..07a379ad 100644
--- a/pkg/keeper/githubapp/factory.go
+++ b/pkg/keeper/githubapp/factory.go
@@ -2,12 +2,11 @@ package githubapp
 
 import (
 	"github.com/jenkins-x/go-scm/scm/factory"
-	"github.com/jenkins-x/jx/v2/pkg/jxfactory"
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	"github.com/jenkins-x/lighthouse/pkg/clients"
 	"github.com/jenkins-x/lighthouse/pkg/git"
+	"github.com/jenkins-x/lighthouse/pkg/jx"
 	"github.com/jenkins-x/lighthouse/pkg/keeper"
-	"github.com/jenkins-x/lighthouse/pkg/launcher"
 	"github.com/jenkins-x/lighthouse/pkg/scmprovider"
 	"github.com/jenkins-x/lighthouse/pkg/util"
 	"github.com/pkg/errors"
@@ -16,14 +15,9 @@ import (
 // NewKeeperController creates a new controller; either regular or a GitHub App flavour
 // depending on the $GITHUB_APP_SECRET_DIR environment variable
 func NewKeeperController(configAgent *config.Agent, botName string, gitKind string, gitToken string, serverURL string, maxRecordsPerPool int, historyURI string, statusURI string) (keeper.Controller, error) {
-	clientFactory := jxfactory.NewFactory()
-	mpClient, err := launcher.NewMetaPipelineClient(clientFactory)
-	if err != nil {
-		return nil, errors.Wrap(err, "Error getting Kubernetes client.")
-	}
 	githubAppSecretDir := util.GetGitHubAppSecretDir()
 	if githubAppSecretDir != "" {
-		return NewGitHubAppKeeperController(githubAppSecretDir, configAgent, mpClient, botName, gitKind, maxRecordsPerPool, historyURI, statusURI)
+		return NewGitHubAppKeeperController(githubAppSecretDir, configAgent, botName, gitKind, maxRecordsPerPool, historyURI, statusURI)
 	}
 
 	scmClient, err := factory.NewClient(gitKind, serverURL, "")
@@ -40,14 +34,14 @@ func NewKeeperController(configAgent *config.Agent, botName string, gitKind stri
 		return []byte(gitToken)
 	})
 
-	tektonClient, jxClient, _, lhClient, ns, err := clients.GetClientsAndNamespace(nil)
+	tektonClient, _, _, lhClient, ns, err := clients.GetClientsAndNamespace(nil)
 	if err != nil {
 		return nil, errors.Wrap(err, "Error creating kubernetes resource clients.")
 	}
-	launcherClient, err := launcher.NewLauncher(jxClient, lhClient, ns)
+	launcherClient, err := jx.NewLauncher()
 	if err != nil {
 		return nil, errors.Wrap(err, "Error getting PipelineLauncher client.")
 	}
-	c, err := keeper.NewController(gitproviderClient, gitproviderClient, launcherClient, mpClient, tektonClient, lhClient, ns, configAgent.Config, gitClient, maxRecordsPerPool, historyURI, statusURI, nil)
+	c, err := keeper.NewController(gitproviderClient, gitproviderClient, launcherClient, tektonClient, lhClient, ns, configAgent.Config, gitClient, maxRecordsPerPool, historyURI, statusURI, nil)
 	return c, err
 }
diff --git a/pkg/keeper/githubapp/github_app_ctrl.go b/pkg/keeper/githubapp/github_app_ctrl.go
index db6153c1..0da1428b 100644
--- a/pkg/keeper/githubapp/github_app_ctrl.go
+++ b/pkg/keeper/githubapp/github_app_ctrl.go
@@ -9,13 +9,12 @@ import (
 	"github.com/jenkins-x/go-scm/scm/factory"
 	"github.com/jenkins-x/go-scm/scm/transport"
 	"github.com/jenkins-x/jx/v2/pkg/errorutil"
-	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	"github.com/jenkins-x/lighthouse/pkg/clients"
 	"github.com/jenkins-x/lighthouse/pkg/git"
+	"github.com/jenkins-x/lighthouse/pkg/jx"
 	"github.com/jenkins-x/lighthouse/pkg/keeper"
 	"github.com/jenkins-x/lighthouse/pkg/keeper/history"
-	"github.com/jenkins-x/lighthouse/pkg/launcher"
 	"github.com/jenkins-x/lighthouse/pkg/scmprovider"
 	"github.com/jenkins-x/lighthouse/pkg/util"
 	"github.com/pkg/errors"
@@ -28,7 +27,6 @@ type gitHubAppKeeperController struct {
 	gitServer          string
 	githubAppSecretDir string
 	configAgent        *config.Agent
-	mpClient           metapipeline.Client
 	botName            string
 	gitKind            string
 	maxRecordsPerPool  int
@@ -40,14 +38,13 @@ type gitHubAppKeeperController struct {
 
 // NewGitHubAppKeeperController creates a GitHub App style controller which needs to process each github owner
 // using a separate git provider client due to the way GitHub App tokens work
-func NewGitHubAppKeeperController(githubAppSecretDir string, configAgent *config.Agent, mpClient metapipeline.Client, botName string, gitKind string, maxRecordsPerPool int, historyURI string, statusURI string) (keeper.Controller, error) {
+func NewGitHubAppKeeperController(githubAppSecretDir string, configAgent *config.Agent, botName string, gitKind string, maxRecordsPerPool int, historyURI string, statusURI string) (keeper.Controller, error) {
 
 	gitServer := util.GithubServer
 	return &gitHubAppKeeperController{
 		ownerTokenFinder:  util.NewOwnerTokensDir(gitServer, githubAppSecretDir),
 		gitServer:         gitServer,
 		configAgent:       configAgent,
-		mpClient:          mpClient,
 		botName:           botName,
 		gitKind:           gitKind,
 		maxRecordsPerPool: maxRecordsPerPool,
@@ -170,15 +167,15 @@ func (g *gitHubAppKeeperController) createOwnerController(owner string, configGe
 	gitClient.SetCredentials(util.GitHubAppGitRemoteUsername, func() []byte {
 		return []byte(token)
 	})
-	tektonClient, jxClient, _, lhClient, ns, err := clients.GetClientsAndNamespace(nil)
+	tektonClient, _, _, lhClient, ns, err := clients.GetClientsAndNamespace(nil)
 	if err != nil {
 		return nil, errors.Wrap(err, "Error creating kubernetes resource clients.")
 	}
-	launcherClient, err := launcher.NewLauncher(jxClient, lhClient, ns)
+	launcherClient, err := jx.NewLauncher()
 	if err != nil {
 		return nil, errors.Wrap(err, "Error getting PipelineLauncher client.")
 	}
-	c, err := keeper.NewController(gitproviderClient, gitproviderClient, launcherClient, g.mpClient, tektonClient, lhClient, ns, configGetter, gitClient, g.maxRecordsPerPool, g.historyURI, g.statusURI, nil)
+	c, err := keeper.NewController(gitproviderClient, gitproviderClient, launcherClient, tektonClient, lhClient, ns, configGetter, gitClient, g.maxRecordsPerPool, g.historyURI, g.statusURI, nil)
 	return c, err
 }
 
diff --git a/pkg/keeper/keeper.go b/pkg/keeper/keeper.go
index e716017a..c983e525 100644
--- a/pkg/keeper/keeper.go
+++ b/pkg/keeper/keeper.go
@@ -31,7 +31,6 @@ import (
 	"time"
 
 	"github.com/jenkins-x/go-scm/scm"
-	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	"github.com/jenkins-x/lighthouse/pkg/apis/lighthouse/v1alpha1"
 	clientset "github.com/jenkins-x/lighthouse/pkg/client/clientset/versioned"
@@ -55,7 +54,7 @@ import (
 var sleep = time.Sleep
 
 type launcher interface {
-	Launch(*v1alpha1.LighthouseJob, metapipeline.Client, scm.Repository) (*v1alpha1.LighthouseJob, error)
+	Launch(*v1alpha1.LighthouseJob, scm.Repository) (*v1alpha1.LighthouseJob, error)
 }
 
 type scmProviderClient interface {
@@ -86,7 +85,6 @@ type DefaultController struct {
 	spc            scmProviderClient
 	launcherClient launcher
 	gc             git.Client
-	mpClient       metapipeline.Client
 	tektonClient   tektonclient.Interface
 	lhClient       clientset.Interface
 	ns             string
@@ -210,7 +208,7 @@ func init() {
 }
 
 // NewController makes a DefaultController out of the given clients.
-func NewController(spcSync, spcStatus *scmprovider.Client, launcherClient launcher, mpClient metapipeline.Client, tektonClient tektonclient.Interface, lighthouseClient clientset.Interface, ns string, cfg config.Getter, gc git.Client, maxRecordsPerPool int, historyURI, statusURI string, logger *logrus.Entry) (*DefaultController, error) {
+func NewController(spcSync, spcStatus *scmprovider.Client, launcherClient launcher, tektonClient tektonclient.Interface, lighthouseClient clientset.Interface, ns string, cfg config.Getter, gc git.Client, maxRecordsPerPool int, historyURI, statusURI string, logger *logrus.Entry) (*DefaultController, error) {
 	if logger == nil {
 		logger = logrus.NewEntry(logrus.StandardLogger())
 	}
@@ -231,7 +229,6 @@ func NewController(spcSync, spcStatus *scmprovider.Client, launcherClient launch
 		logger:         logger.WithField("controller", "sync"),
 		spc:            spcSync,
 		launcherClient: launcherClient,
-		mpClient:       mpClient,
 		tektonClient:   tektonClient,
 		lhClient:       lighthouseClient,
 		ns:             ns,
@@ -1116,7 +1113,7 @@ func (c *DefaultController) trigger(sp subpool, presubmits map[int][]config.Pres
 				Branch:    string(pr.BaseRef.Name),
 				Clone:     cloneURL,
 			}
-			if _, err := c.launcherClient.Launch(&pj, c.mpClient, repo); err != nil {
+			if _, err := c.launcherClient.Launch(&pj, repo); err != nil {
 				c.logger.WithField("duration", time.Since(start).String()).Debug("Failed to create pipeline on the cluster.")
 				return fmt.Errorf("failed to create a pipeline for job: %q, PRs: %v: %v", spec.Job, prNumbers(prs), err)
 			}
diff --git a/pkg/launcher/fake/fake.go b/pkg/launcher/fake/fake.go
index a327825b..7d20b3e0 100644
--- a/pkg/launcher/fake/fake.go
+++ b/pkg/launcher/fake/fake.go
@@ -4,7 +4,6 @@ import (
 	"errors"
 
 	"github.com/jenkins-x/go-scm/scm"
-	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse/pkg/apis/lighthouse/v1alpha1"
 	"github.com/jenkins-x/lighthouse/pkg/launcher"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -28,7 +27,7 @@ func NewLauncher() *Launcher {
 }
 
 // Launch creates a launcher job
-func (p *Launcher) Launch(po *v1alpha1.LighthouseJob, metapipelineClient metapipeline.Client, repo scm.Repository) (*v1alpha1.LighthouseJob, error) {
+func (p *Launcher) Launch(po *v1alpha1.LighthouseJob, repo scm.Repository) (*v1alpha1.LighthouseJob, error) {
 	if p.FailJobs.Has(po.Spec.Job) {
 		return po, errors.New("failed to create job")
 	}
diff --git a/pkg/launcher/interface.go b/pkg/launcher/interface.go
index 17db0d90..44781dee 100644
--- a/pkg/launcher/interface.go
+++ b/pkg/launcher/interface.go
@@ -2,12 +2,11 @@ package launcher
 
 import (
 	"github.com/jenkins-x/go-scm/scm"
-	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse/pkg/apis/lighthouse/v1alpha1"
 )
 
 // PipelineLauncher the interface is the service which creates Pipelines
 type PipelineLauncher interface {
-	// Launch creates new tekton pipelines
-	Launch(*v1alpha1.LighthouseJob, metapipeline.Client, scm.Repository) (*v1alpha1.LighthouseJob, error)
+	// Launch creates new pipelines
+	Launch(*v1alpha1.LighthouseJob, scm.Repository) (*v1alpha1.LighthouseJob, error)
 }
diff --git a/pkg/plugins/override/override.go b/pkg/plugins/override/override.go
index 23cdfa7a..872a4b1c 100644
--- a/pkg/plugins/override/override.go
+++ b/pkg/plugins/override/override.go
@@ -33,7 +33,6 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/util/sets"
 
-	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	"github.com/jenkins-x/lighthouse/pkg/pluginhelp"
 	"github.com/jenkins-x/lighthouse/pkg/plugins"
@@ -64,11 +63,10 @@ type overrideClient interface {
 }
 
 type client struct {
-	spc                scmProviderClient
-	jc                 config.JobConfig
-	clientFactory      jxfactory.Factory
-	metapipelineClient metapipeline.Client
-	lhClient           lighthouseclient.LighthouseJobInterface
+	spc           scmProviderClient
+	jc            config.JobConfig
+	clientFactory jxfactory.Factory
+	lhClient      lighthouseclient.LighthouseJobInterface
 }
 
 func (c client) createOverrideJob(job *v1alpha1.LighthouseJob) (*v1alpha1.LighthouseJob, error) {
@@ -145,10 +143,9 @@ func helpProvider(config *plugins.Configuration, enabledRepos []string) (*plugin
 
 func handleGenericComment(pc plugins.Agent, e scmprovider.GenericCommentEvent) error {
 	c := client{
-		spc:                pc.SCMProviderClient,
-		clientFactory:      pc.ClientFactory,
-		metapipelineClient: pc.MetapipelineClient,
-		lhClient:           pc.LighthouseClient,
+		spc:           pc.SCMProviderClient,
+		clientFactory: pc.ClientFactory,
+		lhClient:      pc.LighthouseClient,
 	}
 	if pc.Config != nil {
 		c.jc = pc.Config.JobConfig
diff --git a/pkg/plugins/plugins.go b/pkg/plugins/plugins.go
index 7f028bc7..cbecf31e 100644
--- a/pkg/plugins/plugins.go
+++ b/pkg/plugins/plugins.go
@@ -28,7 +28,6 @@ import (
 
 	"github.com/jenkins-x/go-scm/scm"
 	"github.com/jenkins-x/jx/v2/pkg/jxfactory"
-	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	lighthouseclient "github.com/jenkins-x/lighthouse/pkg/client/clientset/versioned/typed/lighthouse/v1alpha1"
 	"github.com/jenkins-x/lighthouse/pkg/commentpruner"
@@ -137,14 +136,13 @@ func RegisterGenericCommentHandler(name string, fn GenericCommentHandler, help H
 
 // Agent may be used concurrently, so each entry must be thread-safe.
 type Agent struct {
-	ClientFactory      jxfactory.Factory
-	SCMProviderClient  *scmprovider.Client
-	LauncherClient     launcher.PipelineLauncher
-	MetapipelineClient metapipeline.Client
-	GitClient          git2.Client
-	KubernetesClient   kubernetes.Interface
-	LighthouseClient   lighthouseclient.LighthouseJobInterface
-	ServerURL          *url.URL
+	ClientFactory     jxfactory.Factory
+	SCMProviderClient *scmprovider.Client
+	LauncherClient    launcher.PipelineLauncher
+	GitClient         git2.Client
+	KubernetesClient  kubernetes.Interface
+	LighthouseClient  lighthouseclient.LighthouseJobInterface
+	ServerURL         *url.URL
 	/*
 		SlackClient      *slack.Client
 	*/
@@ -164,18 +162,17 @@ type Agent struct {
 }
 
 // NewAgent bootstraps a new Agent struct from the passed dependencies.
-func NewAgent(clientFactory jxfactory.Factory, configAgent *config.Agent, pluginConfigAgent *ConfigAgent, clientAgent *ClientAgent, metapipelineClient metapipeline.Client, serverURL *url.URL, logger *logrus.Entry) Agent {
+func NewAgent(clientFactory jxfactory.Factory, configAgent *config.Agent, pluginConfigAgent *ConfigAgent, clientAgent *ClientAgent, serverURL *url.URL, logger *logrus.Entry) Agent {
 	prowConfig := configAgent.Config()
 	pluginConfig := pluginConfigAgent.Config()
 	scmClient := scmprovider.ToClient(clientAgent.SCMProviderClient, clientAgent.BotName)
 	return Agent{
-		ClientFactory:      clientFactory,
-		SCMProviderClient:  scmClient,
-		GitClient:          clientAgent.GitClient,
-		LauncherClient:     clientAgent.LauncherClient,
-		MetapipelineClient: metapipelineClient,
-		LighthouseClient:   clientAgent.LighthouseClient,
-		ServerURL:          serverURL,
+		ClientFactory:     clientFactory,
+		SCMProviderClient: scmClient,
+		GitClient:         clientAgent.GitClient,
+		LauncherClient:    clientAgent.LauncherClient,
+		LighthouseClient:  clientAgent.LighthouseClient,
+		ServerURL:         serverURL,
 
 		/*
 			SlackClient:   clientAgent.SlackClient,
@@ -214,11 +211,10 @@ type ClientAgent struct {
 	BotName           string
 	SCMProviderClient *scm.Client
 
-	KubernetesClient   kubernetes.Interface
-	GitClient          git2.Client
-	LauncherClient     launcher.PipelineLauncher
-	MetapipelineClient metapipeline.Client
-	LighthouseClient   lighthouseclient.LighthouseJobInterface
+	KubernetesClient kubernetes.Interface
+	GitClient        git2.Client
+	LauncherClient   launcher.PipelineLauncher
+	LighthouseClient lighthouseclient.LighthouseJobInterface
 
 	/*	SlackClient      *slack.Client
 	 */
diff --git a/pkg/plugins/trigger/push.go b/pkg/plugins/trigger/push.go
index a4a21869..962d248d 100644
--- a/pkg/plugins/trigger/push.go
+++ b/pkg/plugins/trigger/push.go
@@ -77,7 +77,7 @@ func handlePE(c Client, pe scm.PushHook) error {
 		labels[scmprovider.EventGUID] = pe.GUID
 		pj := jobutil.NewLighthouseJob(jobutil.PostsubmitSpec(j, refs), labels, j.Annotations)
 		c.Logger.WithFields(jobutil.LighthouseJobFields(&pj)).Info("Creating a new LighthouseJob.")
-		if _, err := c.LauncherClient.Launch(&pj, c.MetapipelineClient, pe.Repository()); err != nil {
+		if _, err := c.LauncherClient.Launch(&pj, pe.Repository()); err != nil {
 			return err
 		}
 	}
diff --git a/pkg/plugins/trigger/trigger.go b/pkg/plugins/trigger/trigger.go
index d5d72bfb..2d7a825d 100644
--- a/pkg/plugins/trigger/trigger.go
+++ b/pkg/plugins/trigger/trigger.go
@@ -21,7 +21,6 @@ import (
 	"strings"
 
 	"github.com/jenkins-x/go-scm/scm"
-	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	"github.com/jenkins-x/lighthouse/pkg/apis/lighthouse/v1alpha1"
 	"github.com/jenkins-x/lighthouse/pkg/errorutil"
@@ -112,18 +111,17 @@ type scmProviderClient interface {
 }
 
 type launcher interface {
-	Launch(*v1alpha1.LighthouseJob, metapipeline.Client, scm.Repository) (*v1alpha1.LighthouseJob, error)
+	Launch(*v1alpha1.LighthouseJob, scm.Repository) (*v1alpha1.LighthouseJob, error)
 }
 
 // Client holds the necessary structures to work with prow via logging, github, kubernetes and its configuration.
 //
 // TODO(fejta): consider exporting an interface rather than a struct
 type Client struct {
-	SCMProviderClient  scmProviderClient
-	LauncherClient     launcher
-	Config             *config.Config
-	Logger             *logrus.Entry
-	MetapipelineClient metapipeline.Client
+	SCMProviderClient scmProviderClient
+	LauncherClient    launcher
+	Config            *config.Config
+	Logger            *logrus.Entry
 }
 
 type trustedUserClient interface {
@@ -134,11 +132,10 @@ type trustedUserClient interface {
 
 func getClient(pc plugins.Agent) Client {
 	return Client{
-		SCMProviderClient:  pc.SCMProviderClient,
-		Config:             pc.Config,
-		LauncherClient:     pc.LauncherClient,
-		Logger:             pc.Logger,
-		MetapipelineClient: pc.MetapipelineClient,
+		SCMProviderClient: pc.SCMProviderClient,
+		Config:            pc.Config,
+		LauncherClient:    pc.LauncherClient,
+		Logger:            pc.Logger,
 	}
 }
 
@@ -260,7 +257,7 @@ func runRequested(c Client, pr *scm.PullRequest, requestedJobs []config.Presubmi
 		c.Logger.Infof("Starting %s build.", job.Name)
 		pj := jobutil.NewPresubmit(pr, baseSHA, job, eventGUID)
 		c.Logger.WithFields(jobutil.LighthouseJobFields(&pj)).Info("Creating a new LighthouseJob.")
-		if _, err := c.LauncherClient.Launch(&pj, c.MetapipelineClient, pr.Repository()); err != nil {
+		if _, err := c.LauncherClient.Launch(&pj, pr.Repository()); err != nil {
 			c.Logger.WithError(err).Error("Failed to create LighthouseJob.")
 			errors = append(errors, err)
 			if _, statusErr := c.SCMProviderClient.CreateStatus(pr.Base.Repo.Namespace, pr.Base.Repo.Name, pr.Head.Ref, failedStatusForMetapipelineCreation(job.Context, err)); statusErr != nil {
diff --git a/pkg/record/record.go b/pkg/record/record.go
new file mode 100644
index 00000000..503d81b3
--- /dev/null
+++ b/pkg/record/record.go
@@ -0,0 +1,48 @@
+package record
+
+import (
+	"github.com/jenkins-x/lighthouse/pkg/apis/lighthouse/v1alpha1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// ActivityRecord is a struct for reporting information on a pipeline, build, or other activity triggered by Lighthouse
+type ActivityRecord struct {
+	Name            string                 `json:"name"`
+	Owner           string                 `json:"owner,omitempty"`
+	Repo            string                 `json:"repo,omitempty"`
+	Branch          string                 `json:"branch,omitempty"`
+	BuildIdentifier string                 `json:"buildId,omitempty"`
+	Context         string                 `json:"context,omitempty"`
+	GitURL          string                 `json:"gitURL,omitempty"`
+	LogURL          string                 `json:"logURL,omitempty"`
+	LinkURL         string                 `json:"linkURL,omitempty"`
+	Status          v1alpha1.PipelineState `json:"status,omitempty"`
+	BaseSHA         string                 `json:"baseSHA,omitempty"`
+	LastCommitSHA   string                 `json:"lastCommitSHA,omitempty"`
+	StartTime       *metav1.Time           `json:"startTime,omitempty"`
+	CompletionTime  *metav1.Time           `json:"completionTime,omitempty"`
+	Stages          []*ActivityStageOrStep `json:"stages,omitempty"`
+	Steps           []*ActivityStageOrStep `json:"steps,omitEmpty"`
+}
+
+// ActivityStageOrStep represents a stage of an activity
+type ActivityStageOrStep struct {
+	Name           string                 `json:"name"`
+	Status         v1alpha1.PipelineState `json:"status"`
+	StartTime      *metav1.Time           `json:"startTime,omitempty"`
+	CompletionTime *metav1.Time           `json:"completionTime,omitempty"`
+	Stages         []*ActivityStageOrStep `json:"stages,omitempty"`
+	Steps          []*ActivityStageOrStep `json:"steps,omitempty"`
+}
+
+// RunningStages returns the list of stages currently running
+func (a *ActivityRecord) RunningStages() []string {
+	var running []string
+
+	for _, stage := range a.Stages {
+		if stage.Status == v1alpha1.RunningState {
+			running = append(running, stage.Name)
+		}
+	}
+	return running
+}
diff --git a/pkg/util/constants.go b/pkg/util/constants.go
index c30d52be..860209f2 100644
--- a/pkg/util/constants.go
+++ b/pkg/util/constants.go
@@ -84,4 +84,13 @@ const (
 
 	// LighthouseSignatureHeader is the header key used for the signature when relaying to external plugins
 	LighthouseSignatureHeader = "X-Lighthouse-Signature"
+
+	// LighthousePayloadTypeHeader is the header key displaying what type of payload this is, either "webhook" or "activity"
+	LighthousePayloadTypeHeader = "X-Lighthouse-Payload-Type"
+
+	// LighthousePayloadTypeWebhook is the webhook type
+	LighthousePayloadTypeWebhook = "webhook"
+
+	// LighthousePayloadTypeActivity is the activity type
+	LighthousePayloadTypeActivity = "activity"
 )
diff --git a/pkg/util/externalplugins.go b/pkg/util/externalplugins.go
index 7789fb74..968ae14a 100644
--- a/pkg/util/externalplugins.go
+++ b/pkg/util/externalplugins.go
@@ -1,24 +1,34 @@
 package util
 
 import (
+	"bytes"
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/hex"
 	"encoding/json"
 	"fmt"
 	"io"
 	"io/ioutil"
 	"net/http"
+	"strings"
+	"sync"
+	"time"
 
-	"github.com/jenkins-x/go-scm/pkg/hmac"
+	goscmhmac "github.com/jenkins-x/go-scm/pkg/hmac"
 	"github.com/jenkins-x/go-scm/scm"
+	"github.com/jenkins-x/lighthouse/pkg/plugins"
+	"github.com/jenkins-x/lighthouse/pkg/record"
+	"github.com/pkg/errors"
 	"github.com/sirupsen/logrus"
 )
 
-// ParseExternalPluginWebhook parses a webhook relayed to an external plugin
-func ParseExternalPluginWebhook(req *http.Request, secretToken string) (scm.Webhook, error) {
+// ParseExternalPluginEvent parses a webhook relayed to an external plugin
+func ParseExternalPluginEvent(req *http.Request, secretToken string) (scm.Webhook, *record.ActivityRecord, error) {
 	data, err := ioutil.ReadAll(
 		io.LimitReader(req.Body, 10000000),
 	)
 	if err != nil {
-		return nil, err
+		return nil, nil, err
 	}
 
 	log := logrus.WithFields(map[string]interface{}{
@@ -29,22 +39,50 @@ func ParseExternalPluginWebhook(req *http.Request, secretToken string) (scm.Webh
 
 	ua := req.Header.Get("User-Agent")
 	if ua != LighthouseUserAgent {
-		return nil, fmt.Errorf("unknown User-Agent %s, expected %s", ua, LighthouseUserAgent)
+		return nil, nil, fmt.Errorf("unknown User-Agent %s, expected %s", ua, LighthouseUserAgent)
 	}
 
 	sig := req.Header.Get(LighthouseSignatureHeader)
 	if sig == "" {
-		return nil, scm.ErrSignatureInvalid
+		return nil, nil, scm.ErrSignatureInvalid
 	}
-	if !hmac.ValidatePrefix(data, []byte(secretToken), sig) {
-		return nil, scm.ErrSignatureInvalid
+	if !goscmhmac.ValidatePrefix(data, []byte(secretToken), sig) {
+		return nil, nil, scm.ErrSignatureInvalid
 	}
 
+	payloadType := req.Header.Get(LighthousePayloadTypeHeader)
+	switch payloadType {
+	case LighthousePayloadTypeWebhook:
+		hook, err := parseWebhook(log, req, data)
+		if err != nil {
+			return nil, nil, errors.Wrap(err, "parsing webhook")
+		}
+		return hook, nil, nil
+	case LighthousePayloadTypeActivity:
+		ar := new(record.ActivityRecord)
+		err := json.Unmarshal(data, ar)
+		if err != nil {
+			return nil, nil, errors.Wrap(err, "parsing activity")
+		}
+		return nil, ar, nil
+	default:
+		return nil, nil, fmt.Errorf("unknown Lighthouse payload type %s", payloadType)
+	}
+}
+
+func parseActivity(l *logrus.Entry, req *http.Request, data []byte) (*record.ActivityRecord, error) {
+	ar := new(record.ActivityRecord)
+	err := json.Unmarshal(data, ar)
+	return ar, err
+}
+
+func parseWebhook(l *logrus.Entry, req *http.Request, data []byte) (scm.Webhook, error) {
 	kind := req.Header.Get(LighthouseWebhookKindHeader)
 	if kind == "" {
 		return nil, scm.MissingHeader{Header: LighthouseWebhookKindHeader}
 	}
 	var hook scm.Webhook
+	var err error
 	switch scm.WebhookKind(kind) {
 	case scm.WebhookKindBranch:
 		hook = new(scm.BranchHook)
@@ -116,7 +154,7 @@ func ParseExternalPluginWebhook(req *http.Request, secretToken string) (scm.Webh
 		hook = new(scm.WatchHook)
 		err = json.Unmarshal(data, hook)
 	default:
-		log.WithField("Kind", kind).Warnf("unknown webhook")
+		l.WithField("Kind", kind).Warnf("unknown webhook")
 		return nil, scm.UnknownWebhook{Event: kind}
 	}
 	if err != nil {
@@ -125,3 +163,117 @@ func ParseExternalPluginWebhook(req *http.Request, secretToken string) (scm.Webh
 
 	return hook, nil
 }
+
+// callExternalPlugins dispatches the provided payload to the external plugins.
+func callExternalPlugins(l *logrus.Entry, externalPlugins []plugins.ExternalPlugin, payload []byte, headers http.Header, hmacToken string, wg *sync.WaitGroup) {
+	headers.Set("User-Agent", LighthouseUserAgent)
+	mac := hmac.New(sha256.New, []byte(hmacToken))
+	_, err := mac.Write(payload)
+	if err != nil {
+		l.WithError(err).Error("Unable to generate signature for relayed payload")
+		return
+	}
+	sum := mac.Sum(nil)
+	signature := "sha256=" + hex.EncodeToString(sum)
+	headers.Set(LighthouseSignatureHeader, signature)
+	for _, p := range externalPlugins {
+		wg.Add(1)
+		go func(p plugins.ExternalPlugin) {
+			defer wg.Done()
+			if err := dispatch(p.Endpoint, payload, headers); err != nil {
+				l.WithError(err).WithField("external-plugin", p.Name).Error("Error dispatching event to external plugin.")
+			} else {
+				l.WithField("external-plugin", p.Name).Info("Dispatched event to external plugin")
+			}
+		}(p)
+	}
+}
+
+// CallExternalPluginsWithActivityRecord dispatches the provided activity record to the external plugins.
+func CallExternalPluginsWithActivityRecord(l *logrus.Entry, externalPlugins []plugins.ExternalPlugin, activity *record.ActivityRecord, hmacToken string, wg *sync.WaitGroup) {
+	headers := http.Header{}
+	headers.Set(LighthousePayloadTypeHeader, LighthousePayloadTypeActivity)
+	payload, err := json.Marshal(activity)
+	if err != nil {
+		l.WithError(err).Errorf("Unable to marshal activity for relaying to external plugins. Activity is: %v", activity)
+		return
+	}
+	callExternalPlugins(l, externalPlugins, payload, headers, hmacToken, wg)
+}
+
+// CallExternalPluginsWithWebhook dispatches the provided webhook to the external plugins.
+func CallExternalPluginsWithWebhook(l *logrus.Entry, externalPlugins []plugins.ExternalPlugin, webhook scm.Webhook, hmacToken string, wg *sync.WaitGroup) {
+	headers := http.Header{}
+	headers.Set(LighthouseWebhookKindHeader, string(webhook.Kind()))
+	headers.Set(LighthousePayloadTypeHeader, LighthousePayloadTypeWebhook)
+	payload, err := json.Marshal(webhook)
+	if err != nil {
+		l.WithError(err).Errorf("Unable to marshal webhook for relaying to external plugins. Webhook is: %v", webhook)
+		return
+	}
+	callExternalPlugins(l, externalPlugins, payload, headers, hmacToken, wg)
+}
+
+// dispatch creates a new request using the provided payload and headers
+// and dispatches the request to the provided endpoint.
+func dispatch(endpoint string, payload []byte, h http.Header) error {
+	req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewBuffer(payload))
+	if err != nil {
+		return err
+	}
+	req.Header = h
+	var resp *http.Response
+	backoff := 100 * time.Millisecond
+	maxRetries := 5
+
+	c := &http.Client{}
+	for retries := 0; retries < maxRetries; retries++ {
+		resp, err = c.Do(req)
+		if err == nil {
+			break
+		}
+		time.Sleep(backoff)
+		backoff *= 2
+	}
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	rb, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return err
+	}
+	if resp.StatusCode < 200 || resp.StatusCode > 299 {
+		return fmt.Errorf("response has status %q and body %q", resp.Status, string(rb))
+	}
+	return nil
+}
+
+// ExternalPluginsForEvent returns whether there are any external plugins that need to
+// get the present event.
+func ExternalPluginsForEvent(pluginConfig *plugins.ConfigAgent, eventKind string, srcRepo string) []plugins.ExternalPlugin {
+	var matching []plugins.ExternalPlugin
+	srcOrg := strings.Split(srcRepo, "/")[0]
+
+	for repo, extPlugins := range pluginConfig.Config().ExternalPlugins {
+		// Make sure the repositories match
+		if repo != srcRepo && repo != srcOrg {
+			continue
+		}
+
+		// Make sure the events match
+		for _, p := range extPlugins {
+			if len(p.Events) == 0 {
+				matching = append(matching, p)
+			} else {
+				for _, et := range p.Events {
+					if et == eventKind || et == eventKind+"s" {
+						matching = append(matching, p)
+						break
+					}
+				}
+			}
+		}
+	}
+	return matching
+}
diff --git a/pkg/webhook/events.go b/pkg/webhook/events.go
index 4b12d1d0..0efb4133 100644
--- a/pkg/webhook/events.go
+++ b/pkg/webhook/events.go
@@ -17,40 +17,27 @@ limitations under the License.
 package webhook
 
 import (
-	"bytes"
-	"crypto/hmac"
-	"crypto/sha256"
-	"encoding/hex"
-	"encoding/json"
-	"fmt"
-	"io/ioutil"
-	"net/http"
 	"net/url"
 	"strconv"
-	"strings"
 	"sync"
-	"time"
 
 	"github.com/jenkins-x/go-scm/scm"
 	"github.com/jenkins-x/jx/v2/pkg/jxfactory"
-	"github.com/jenkins-x/jx/v2/pkg/tekton/metapipeline"
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	"github.com/jenkins-x/lighthouse/pkg/plugins"
 	"github.com/jenkins-x/lighthouse/pkg/scmprovider"
-	"github.com/jenkins-x/lighthouse/pkg/util"
 	"github.com/sirupsen/logrus"
 )
 
 // Server keeps the information required to start a server
 type Server struct {
-	ClientFactory      jxfactory.Factory
-	MetapipelineClient metapipeline.Client
-	ClientAgent        *plugins.ClientAgent
-	Plugins            *plugins.ConfigAgent
-	ConfigAgent        *config.Agent
-	ServerURL          *url.URL
-	TokenGenerator     func() []byte
-	Metrics            *Metrics
+	ClientFactory  jxfactory.Factory
+	ClientAgent    *plugins.ClientAgent
+	Plugins        *plugins.ConfigAgent
+	ConfigAgent    *config.Agent
+	ServerURL      *url.URL
+	TokenGenerator func() []byte
+	Metrics        *Metrics
 
 	// Tracks running handlers for graceful shutdown
 	wg sync.WaitGroup
@@ -72,7 +59,7 @@ func (s *Server) HandleIssueCommentEvent(l *logrus.Entry, ic scm.IssueCommentHoo
 		s.wg.Add(1)
 		go func(p string, h plugins.IssueCommentHandler) {
 			defer s.wg.Done()
-			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.MetapipelineClient, s.ServerURL, l.WithField("plugin", p))
+			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.ServerURL, l.WithField("plugin", p))
 			agent.InitializeCommentPruner(
 				ic.Repo.Namespace,
 				ic.Repo.Name,
@@ -140,7 +127,7 @@ func (s *Server) handleGenericComment(l *logrus.Entry, ce *scmprovider.GenericCo
 		s.wg.Add(1)
 		go func(p string, h plugins.GenericCommentHandler) {
 			defer s.wg.Done()
-			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.MetapipelineClient, s.ServerURL, l.WithField("plugin", p))
+			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.ServerURL, l.WithField("plugin", p))
 			agent.InitializeCommentPruner(
 				ce.Repo.Namespace,
 				ce.Repo.Name,
@@ -169,7 +156,7 @@ func (s *Server) HandlePushEvent(l *logrus.Entry, pe *scm.PushHook) {
 		c++
 		go func(p string, h plugins.PushEventHandler) {
 			defer s.wg.Done()
-			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.MetapipelineClient, s.ServerURL, l.WithField("plugin", p))
+			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.ServerURL, l.WithField("plugin", p))
 			if err := h(agent, *pe); err != nil {
 				agent.Logger.WithError(err).Error("Error handling PushEvent.")
 			}
@@ -199,7 +186,7 @@ func (s *Server) HandlePullRequestEvent(l *logrus.Entry, pr *scm.PullRequestHook
 		c++
 		go func(p string, h plugins.PullRequestHandler) {
 			defer s.wg.Done()
-			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.MetapipelineClient, s.ServerURL, l.WithField("plugin", p))
+			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.ServerURL, l.WithField("plugin", p))
 			agent.InitializeCommentPruner(
 				pr.Repo.Namespace,
 				pr.Repo.Name,
@@ -255,7 +242,7 @@ func (s *Server) HandleReviewEvent(l *logrus.Entry, re scm.ReviewHook) {
 		s.wg.Add(1)
 		go func(p string, h plugins.ReviewEventHandler) {
 			defer s.wg.Done()
-			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.MetapipelineClient, s.ServerURL, l.WithField("plugin", p))
+			agent := plugins.NewAgent(s.ClientFactory, s.ConfigAgent, s.Plugins, s.ClientAgent, s.ServerURL, l.WithField("plugin", p))
 			agent.InitializeCommentPruner(
 				re.Repo.Namespace,
 				re.Repo.Name,
@@ -312,105 +299,3 @@ func actionRelatesToPullRequestComment(action scm.Action, l *logrus.Entry) bool
 		return false
 	}
 }
-
-// externalPluginsForEvent returns whether there are any external plugins that need to
-// get the present event.
-func (s *Server) externalPluginsForEvent(eventKind scm.WebhookKind, srcRepo string) []plugins.ExternalPlugin {
-	var matching []plugins.ExternalPlugin
-	srcOrg := strings.Split(srcRepo, "/")[0]
-
-	for repo, extPlugins := range s.Plugins.Config().ExternalPlugins {
-		// Make sure the repositories match
-		if repo != srcRepo && repo != srcOrg {
-			continue
-		}
-
-		// Make sure the events match
-		for _, p := range extPlugins {
-			if len(p.Events) == 0 {
-				matching = append(matching, p)
-			} else {
-				for _, et := range p.Events {
-					if et == string(eventKind) || et == string(eventKind)+"s" {
-						matching = append(matching, p)
-						break
-					}
-				}
-			}
-		}
-	}
-	return matching
-}
-
-// callExternalPlugins dispatches the provided payload to the external plugins.
-func (s *Server) callExternalPlugins(l *logrus.Entry, externalPlugins []plugins.ExternalPlugin, webhook scm.Webhook, hmacToken string) {
-	headers := http.Header{}
-	headers.Set("User-Agent", util.LighthouseUserAgent)
-	headers.Set(util.LighthouseWebhookKindHeader, string(webhook.Kind()))
-	payload, err := json.Marshal(webhook)
-	if err != nil {
-		l.WithError(err).Errorf("Unable to marshal webhook for relaying to external plugins. Webhook is: %v", webhook)
-		return
-	}
-	mac := hmac.New(sha256.New, []byte(hmacToken))
-	_, err = mac.Write(payload)
-	if err != nil {
-		l.WithError(err).Error("Unable to generate signature for relayed webhook")
-		return
-	}
-	sum := mac.Sum(nil)
-	signature := "sha256=" + hex.EncodeToString(sum)
-	headers.Set(util.LighthouseSignatureHeader, signature)
-	for _, p := range externalPlugins {
-		s.wg.Add(1)
-		go func(p plugins.ExternalPlugin) {
-			defer s.wg.Done()
-			if err := s.dispatch(p.Endpoint, payload, headers); err != nil {
-				l.WithError(err).WithField("external-plugin", p.Name).Error("Error dispatching event to external plugin.")
-			} else {
-				l.WithField("external-plugin", p.Name).Info("Dispatched event to external plugin")
-			}
-		}(p)
-	}
-}
-
-// dispatch creates a new request using the provided payload and headers
-// and dispatches the request to the provided endpoint.
-func (s *Server) dispatch(endpoint string, payload []byte, h http.Header) error {
-	req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewBuffer(payload))
-	if err != nil {
-		return err
-	}
-	req.Header = h
-	resp, err := s.do(req)
-	if err != nil {
-		return err
-	}
-	defer resp.Body.Close()
-	rb, err := ioutil.ReadAll(resp.Body)
-	if err != nil {
-		return err
-	}
-	if resp.StatusCode < 200 || resp.StatusCode > 299 {
-		return fmt.Errorf("response has status %q and body %q", resp.Status, string(rb))
-	}
-	return nil
-}
-
-func (s *Server) do(req *http.Request) (*http.Response, error) {
-	var resp *http.Response
-	var err error
-	backoff := 100 * time.Millisecond
-	maxRetries := 5
-
-	c := &http.Client{}
-	for retries := 0; retries < maxRetries; retries++ {
-		resp, err = c.Do(req)
-		if err == nil {
-			break
-		}
-		time.Sleep(backoff)
-		backoff *= 2
-	}
-	return resp, err
-}
diff --git a/pkg/webhook/webhook.go b/pkg/webhook/webhook.go
index 786aff15..405eee50 100644
--- a/pkg/webhook/webhook.go
+++ b/pkg/webhook/webhook.go
@@ -19,6 +19,7 @@ import (
 	"github.com/jenkins-x/lighthouse/pkg/clients"
 	"github.com/jenkins-x/lighthouse/pkg/cmd/helper"
 	"github.com/jenkins-x/lighthouse/pkg/git"
+	"github.com/jenkins-x/lighthouse/pkg/jx"
 	"github.com/jenkins-x/lighthouse/pkg/launcher"
 	"github.com/jenkins-x/lighthouse/pkg/logrusutil"
 	"github.com/jenkins-x/lighthouse/pkg/metrics"
@@ -127,13 +128,7 @@ func (o *Options) Run() error {
 
 	o.gitClient = gitClient
 
-	_, jxClient, _, lhClient, _, err := clients.GetClientsAndNamespace(nil)
-	if err != nil {
-		err = errors.Wrapf(err, "failed to create JX client")
-		logrus.Errorf("%s", err.Error())
-		return err
-	}
-	o.launcher, err = launcher.NewLauncher(jxClient, lhClient, o.namespace)
+	o.launcher, err = jx.NewLauncher()
 	if err != nil {
 		err = errors.Wrapf(err, "failed to create PipelineLauncher client")
 		logrus.Errorf("%s", err.Error())
@@ -274,8 +269,8 @@ func (o *Options) handleWebHookRequests(w http.ResponseWriter, r *http.Request)
 		responseHTTPError(w, http.StatusInternalServerError, fmt.Sprintf("500 Internal Server Error: %s", err.Error()))
 	}
 	// Demux events only to external plugins that require this event.
-	if external := o.server.externalPluginsForEvent(webhook.Kind(), webhook.Repository().FullName); len(external) > 0 {
-		go o.server.callExternalPlugins(l, external, webhook, o.hmacToken())
+	if external := util.ExternalPluginsForEvent(o.server.Plugins, string(webhook.Kind()), webhook.Repository().FullName); len(external) > 0 {
+		go util.CallExternalPluginsWithWebhook(l, external, webhook, o.hmacToken(), &o.server.wg)
 	}
 
 	_, err = w.Write([]byte(output))
@@ -550,22 +545,16 @@ func (o *Options) createHookServer() (*Server, error) {
 		logrus.Warn("no configAgent configuration")
 	}
 
-	metapipelineClient, err := launcher.NewMetaPipelineClient(clientFactory)
-	if err != nil {
-		return nil, errors.Wrap(err, "failed to create metapipeline client")
-	}
-
 	serverURL, err := url.Parse(o.gitServerURL)
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to parse server URL %s", o.gitServerURL)
 	}
 	server := &Server{
-		ClientFactory:      clientFactory,
-		ConfigAgent:        configAgent,
-		Plugins:            pluginAgent,
-		Metrics:            promMetrics,
-		MetapipelineClient: metapipelineClient,
-		ServerURL:          serverURL,
+		ClientFactory: clientFactory,
+		ConfigAgent:   configAgent,
+		Plugins:       pluginAgent,
+		Metrics:       promMetrics,
+		ServerURL:     serverURL,
 		//TokenGenerator: secretAgent.GetTokenGenerator(o.webhookSecretFile),
 	}
 	return server, nil


Test patch:
diff --git a/pkg/jx/activity_test.go b/pkg/jx/activity_test.go
new file mode 100644
index 00000000..794b5ac6
--- /dev/null
+++ b/pkg/jx/activity_test.go
@@ -0,0 +1,49 @@
+package jx_test
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"testing"
+
+	v1 "github.com/jenkins-x/jx/v2/pkg/apis/jenkins.io/v1"
+	"github.com/jenkins-x/lighthouse/pkg/apis/lighthouse/v1alpha1"
+	"github.com/jenkins-x/lighthouse/pkg/jx"
+	"github.com/jenkins-x/lighthouse/pkg/record"
+	"github.com/jenkins-x/lighthouse/pkg/util"
+	"github.com/stretchr/testify/assert"
+	"sigs.k8s.io/yaml"
+)
+
+func TestConvertPipelineActivity(t *testing.T) {
+	workDir, err := os.Getwd()
+	assert.NoError(t, err)
+	activityBytes, err := ioutil.ReadFile(fmt.Sprintf("%s/test_data/pipelineactivity.yaml", workDir))
+	assert.NoError(t, err)
+
+	activity := &v1.PipelineActivity{}
+	err = yaml.Unmarshal(activityBytes, activity)
+	assert.NoError(t, err)
+
+	jobBytes, err := ioutil.ReadFile(fmt.Sprintf("%s/test_data/lhjob.yaml", workDir))
+	assert.NoError(t, err)
+	job := &v1alpha1.LighthouseJob{}
+	err = yaml.Unmarshal(jobBytes, job)
+	assert.NoError(t, err)
+
+	converted, err := jx.ConvertPipelineActivity(activity)
+	assert.NoError(t, err)
+
+	assert.Equal(t, job.Labels[util.OrgLabel], converted.Owner)
+	assert.Equal(t, job.Labels[util.RepoLabel], converted.Repo)
+	assert.Equal(t, job.Labels[util.BranchLabel], converted.Branch)
+	assert.Equal(t, job.Labels[util.BuildNumLabel], converted.BuildIdentifier)
+	assert.Equal(t, job.Labels[util.ContextLabel], converted.Context)
+
+	expectedBytes, err := ioutil.ReadFile(fmt.Sprintf("%s/test_data/record.yaml", workDir))
+	assert.NoError(t, err)
+	expectedRecord := &record.ActivityRecord{}
+	err = yaml.Unmarshal(expectedBytes, expectedRecord)
+
+	assert.Equal(t, expectedRecord, converted)
+}
diff --git a/pkg/jx/test_data/lhjob.yaml b/pkg/jx/test_data/lhjob.yaml
new file mode 100644
index 00000000..938ec8c6
--- /dev/null
+++ b/pkg/jx/test_data/lhjob.yaml
@@ -0,0 +1,51 @@
+apiVersion: lighthouse.jenkins.io/v1alpha1
+kind: LighthouseJob
+metadata:
+  annotations:
+    lighthouse.jenkins-x.io/job: github
+  creationTimestamp: "2020-06-22T11:53:15Z"
+  generation: 1
+  labels:
+    created-by-lighthouse: "true"
+    event-GUID: "647468648"
+    lighthouse.jenkins-x.io/branch: PR-813
+    lighthouse.jenkins-x.io/buildNum: "6"
+    lighthouse.jenkins-x.io/context: github
+    lighthouse.jenkins-x.io/job: github
+    lighthouse.jenkins-x.io/refs.org: jenkins-x
+    lighthouse.jenkins-x.io/refs.pull: "813"
+    lighthouse.jenkins-x.io/refs.repo: lighthouse
+    lighthouse.jenkins-x.io/type: presubmit
+  name: f46327af-b47e-11ea-b797-9256b7b8d9b0
+  namespace: jx
+  resourceVersion: "196519432"
+  selfLink: /apis/lighthouse.jenkins.io/v1alpha1/namespaces/jx/lighthousejobs/f46327af-b47e-11ea-b797-9256b7b8d9b0
+  uid: f4b1580b-b47e-11ea-b02a-42010a8401ad
+spec:
+  context: github
+  job: github
+  namespace: jx
+  refs:
+    base_link: https://github.com/jenkins-x/lighthouse/commit/e8d56b5ee9671599c75644af574a251dd3b94a5c
+    base_ref: master
+    base_sha: e8d56b5ee9671599c75644af574a251dd3b94a5c
+    org: jenkins-x
+    pulls:
+    - author: abayer
+      author_link: https://github.com/abayer
+      commit_link: https://github.com/jenkins-x/lighthouse/pull/813/commits/dd64c739442d505cf5381e2a14b60968e8a0d86e
+      link: https://github.com/jenkins-x/lighthouse/pull/813.diff
+      number: 813
+      sha: dd64c739442d505cf5381e2a14b60968e8a0d86e
+    repo: lighthouse
+    repo_link: https://github.com/jenkins-x/lighthouse
+  rerun_command: /test github
+  type: presubmit
+status:
+  activityName: jenkins-x-lighthouse-pr-813-6
+  description: 'Pipeline running stage(s): ci'
+  lastCommitSHA: dd64c739442d505cf5381e2a14b60968e8a0d86e
+  lastReportState: running
+  reportURL: https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-813/6
+  startTime: "2020-06-22T11:53:15Z"
+  state: running
diff --git a/pkg/jx/test_data/pipelineactivity.yaml b/pkg/jx/test_data/pipelineactivity.yaml
new file mode 100644
index 00000000..356604ce
--- /dev/null
+++ b/pkg/jx/test_data/pipelineactivity.yaml
@@ -0,0 +1,127 @@
+apiVersion: jenkins.io/v1
+kind: PipelineActivity
+metadata:
+  creationTimestamp: "2020-06-22T11:53:16Z"
+  generation: 24
+  labels:
+    branch: PR-813
+    build: "6"
+    context: github
+    lastCommitSha: dd64c739442d505cf5381e2a14b60968e8a0d86e
+    owner: jenkins-x
+    provider: github
+    repository: lighthouse
+  name: jenkins-x-lighthouse-pr-813-6
+  namespace: jx
+  resourceVersion: "196520840"
+  selfLink: /apis/jenkins.io/v1/namespaces/jx/pipelineactivities/jenkins-x-lighthouse-pr-813-6
+  uid: f53aa32b-b47e-11ea-b02a-42010a8401ad
+spec:
+  author: abayer
+  authorAvatarURL: https://avatars2.githubusercontent.com/u/120218?v=4
+  authorURL: https://github.com/abayer
+  baseSHA: e8d56b5ee9671599c75644af574a251dd3b94a5c
+  batchPipelineActivity: {}
+  build: "6"
+  context: github
+  gitBranch: PR-813
+  gitOwner: jenkins-x
+  gitRepository: lighthouse
+  gitUrl: https://github.com/jenkins-x/lighthouse.git
+  lastCommitSHA: dd64c739442d505cf5381e2a14b60968e8a0d86e
+  pipeline: jenkins-x/lighthouse/PR-813
+  pullTitle: 'WIP: feat: Work towards supporting other engines besides jx'
+  startedTimestamp: "2020-06-22T11:53:17Z"
+  status: Running
+  steps:
+  - kind: Stage
+    stage:
+      completedTimestamp: "2020-06-22T11:53:49Z"
+      name: meta pipeline
+      startedTimestamp: "2020-06-22T11:53:17Z"
+      status: Succeeded
+      steps:
+      - completedTimestamp: "2020-06-22T11:53:17Z"
+        name: Credential Initializer C5wrl
+        startedTimestamp: "2020-06-22T11:53:17Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:17Z"
+        name: Working Dir Initializer Vjz4t
+        startedTimestamp: "2020-06-22T11:53:17Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:18Z"
+        name: Place Tools
+        startedTimestamp: "2020-06-22T11:53:17Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:25Z"
+        description: https://github.com/jenkins-x/lighthouse.git
+        name: Git Source Meta Jenkins X Lighthouse Pr 81 7gtvf Vxn85
+        startedTimestamp: "2020-06-22T11:53:18Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:29Z"
+        name: Git Merge
+        startedTimestamp: "2020-06-22T11:53:25Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:32Z"
+        name: Merge Pull Refs
+        startedTimestamp: "2020-06-22T11:53:29Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:42Z"
+        name: Create Effective Pipeline
+        startedTimestamp: "2020-06-22T11:53:32Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:49Z"
+        name: Create Tekton Crds
+        startedTimestamp: "2020-06-22T11:53:42Z"
+        status: Succeeded
+  - kind: Stage
+    stage:
+      name: ci
+      startedTimestamp: "2020-06-22T11:53:50Z"
+      status: Running
+      steps:
+      - completedTimestamp: "2020-06-22T11:53:50Z"
+        name: Credential Initializer 52cws
+        startedTimestamp: "2020-06-22T11:53:50Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:51Z"
+        name: Working Dir Initializer Z594z
+        startedTimestamp: "2020-06-22T11:53:50Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:52Z"
+        name: Place Tools
+        startedTimestamp: "2020-06-22T11:53:51Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:53:59Z"
+        description: https://github.com/jenkins-x/lighthouse.git
+        name: Git Source Jenkins X Lighthouse Pr 813 Git R8sdb F64b8
+        startedTimestamp: "2020-06-22T11:53:52Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:54:02Z"
+        name: Git Merge
+        startedTimestamp: "2020-06-22T11:53:59Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:55:37Z"
+        name: Build Binaries
+        startedTimestamp: "2020-06-22T11:54:02Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:55:54Z"
+        name: Build And Push Webhooks
+        startedTimestamp: "2020-06-22T11:55:37Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:56:10Z"
+        name: Build And Push Keeper
+        startedTimestamp: "2020-06-22T11:55:54Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:56:26Z"
+        name: Build And Push Foghorn
+        startedTimestamp: "2020-06-22T11:56:10Z"
+        status: Succeeded
+      - completedTimestamp: "2020-06-22T11:56:41Z"
+        name: Build And Push Gc Jobs
+        startedTimestamp: "2020-06-22T11:56:26Z"
+        status: Succeeded
+      - name: Runci
+        startedTimestamp: "2020-06-22T11:56:41Z"
+        status: Running
+status: {}
diff --git a/pkg/jx/test_data/record.yaml b/pkg/jx/test_data/record.yaml
new file mode 100644
index 00000000..486a63a1
--- /dev/null
+++ b/pkg/jx/test_data/record.yaml
@@ -0,0 +1,97 @@
+baseSHA: e8d56b5ee9671599c75644af574a251dd3b94a5c
+branch: PR-813
+buildId: "6"
+context: github
+gitURL: https://github.com/jenkins-x/lighthouse.git
+lastCommitSHA: dd64c739442d505cf5381e2a14b60968e8a0d86e
+name: jenkins-x-lighthouse-pr-813-6
+owner: jenkins-x
+repo: lighthouse
+stages:
+  - completionTime: "2020-06-22T11:53:49Z"
+    name: meta pipeline
+    startTime: "2020-06-22T11:53:17Z"
+    status: success
+    steps:
+      - completionTime: "2020-06-22T11:53:17Z"
+        name: Credential Initializer C5wrl
+        startTime: "2020-06-22T11:53:17Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:17Z"
+        name: Working Dir Initializer Vjz4t
+        startTime: "2020-06-22T11:53:17Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:18Z"
+        name: Place Tools
+        startTime: "2020-06-22T11:53:17Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:25Z"
+        name: Git Source Meta Jenkins X Lighthouse Pr 81 7gtvf Vxn85
+        startTime: "2020-06-22T11:53:18Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:29Z"
+        name: Git Merge
+        startTime: "2020-06-22T11:53:25Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:32Z"
+        name: Merge Pull Refs
+        startTime: "2020-06-22T11:53:29Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:42Z"
+        name: Create Effective Pipeline
+        startTime: "2020-06-22T11:53:32Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:49Z"
+        name: Create Tekton Crds
+        startTime: "2020-06-22T11:53:42Z"
+        status: success
+  - name: ci
+    startTime: "2020-06-22T11:53:50Z"
+    status: running
+    steps:
+      - completionTime: "2020-06-22T11:53:50Z"
+        name: Credential Initializer 52cws
+        startTime: "2020-06-22T11:53:50Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:51Z"
+        name: Working Dir Initializer Z594z
+        startTime: "2020-06-22T11:53:50Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:52Z"
+        name: Place Tools
+        startTime: "2020-06-22T11:53:51Z"
+        status: success
+      - completionTime: "2020-06-22T11:53:59Z"
+        name: Git Source Jenkins X Lighthouse Pr 813 Git R8sdb F64b8
+        startTime: "2020-06-22T11:53:52Z"
+        status: success
+      - completionTime: "2020-06-22T11:54:02Z"
+        name: Git Merge
+        startTime: "2020-06-22T11:53:59Z"
+        status: success
+      - completionTime: "2020-06-22T11:55:37Z"
+        name: Build Binaries
+        startTime: "2020-06-22T11:54:02Z"
+        status: success
+      - completionTime: "2020-06-22T11:55:54Z"
+        name: Build And Push Webhooks
+        startTime: "2020-06-22T11:55:37Z"
+        status: success
+      - completionTime: "2020-06-22T11:56:10Z"
+        name: Build And Push Keeper
+        startTime: "2020-06-22T11:55:54Z"
+        status: success
+      - completionTime: "2020-06-22T11:56:26Z"
+        name: Build And Push Foghorn
+        startTime: "2020-06-22T11:56:10Z"
+        status: success
+      - completionTime: "2020-06-22T11:56:41Z"
+        name: Build And Push Gc Jobs
+        startTime: "2020-06-22T11:56:26Z"
+        status: success
+      - name: Runci
+        startTime: "2020-06-22T11:56:41Z"
+        status: running
+startTime: "2020-06-22T11:53:17Z"
+status: running
+steps: nil
diff --git a/pkg/webhook/webhook_test.go b/pkg/webhook/webhook_test.go
index 59f41c50..847b9d6c 100644
--- a/pkg/webhook/webhook_test.go
+++ b/pkg/webhook/webhook_test.go
@@ -254,7 +254,7 @@ func TestNeedDemux(t *testing.T) {
 			})
 			s := &Server{Plugins: pa}
 
-			gotPlugins := s.externalPluginsForEvent(test.eventType, test.srcRepo)
+			gotPlugins := util.ExternalPluginsForEvent(s.Plugins, string(test.eventType), test.srcRepo)
 			if len(gotPlugins) != len(test.expected) {
 				t.Fatalf("expected plugins: %+v, got: %+v", test.expected, gotPlugins)
 			}