Repository: jenkins-x/lighthouse
Pull request #805 (2020-06-17 16:15:03)

chore: Add external plugin support like Prow
For whatever reason, this functionality wasn't included originally, so let's add it.

Signed-off-by: Andrew Bayer <andrew.bayer@gmail.com>

jenkins-x-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **abayer**
You can assign the PR to them by writing `/assign @abayer` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=jenkins-x%2Flighthouse).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](jenkins-x/lighthouse/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["abayer"]} -->
abayer: Trying to figure out what more is needed besides just this (and tests, obviously).
jenkins-x-bot: @abayer: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ghe | 2f6f6ec1359b37263c1bcd550af6d122ae37e9e5 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/7) | `/test ghe`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jenkins-x-bot: @abayer: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ghe | 2f6f6ec1359b37263c1bcd550af6d122ae37e9e5 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/7) | `/test ghe`
gitlab | 2f6f6ec1359b37263c1bcd550af6d122ae37e9e5 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/10) | `/test gitlab`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jenkins-x-bot: @abayer: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ghe | 2f6f6ec1359b37263c1bcd550af6d122ae37e9e5 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/7) | `/test ghe`
gitlab | 2f6f6ec1359b37263c1bcd550af6d122ae37e9e5 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/10) | `/test gitlab`
bbs | 2f6f6ec1359b37263c1bcd550af6d122ae37e9e5 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/9) | `/test bbs`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
abayer: /retest
jenkins-x-bot: @abayer: The following tests **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
ghe | 2f6f6ec1359b37263c1bcd550af6d122ae37e9e5 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/7) | `/test ghe`
gitlab | 2f6f6ec1359b37263c1bcd550af6d122ae37e9e5 | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/10) | `/test gitlab`
bbs | da95af6b1166834103ac2a803a6fa192d042be3b | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/4) | `/test bbs`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
jenkins-x-bot: @abayer: The following test **failed**, say `/retest` to rerun them all:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pr-build | 9504b09825d75c199d8dd572cee3c18b46e83d2a | [link](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805/14) | `/test pr-build`

[View all Builds for this Pull Request](https://dashboard-jx.jenkins-x.live/teams/jx/projects/jenkins-x/lighthouse/PR-805) 

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/pkg/util/constants.go b/pkg/util/constants.go
index acc81b83..c30d52be 100644
--- a/pkg/util/constants.go
+++ b/pkg/util/constants.go
@@ -75,4 +75,13 @@ const (
 
 	// LighthouseCommandPrefix is an optional prefix for commands to deal with things like GitLab hijacking /approve
 	LighthouseCommandPrefix = "lh-"
+
+	// LighthouseUserAgent is the header value for User-Agent when Lighthouse calls external plugins
+	LighthouseUserAgent = "LighthouseHook"
+
+	// LighthouseWebhookKindHeader is the header key used for the webhook kind when relaying to external plugins
+	LighthouseWebhookKindHeader = "X-Lighthouse-Webhook-Kind"
+
+	// LighthouseSignatureHeader is the header key used for the signature when relaying to external plugins
+	LighthouseSignatureHeader = "X-Lighthouse-Signature"
 )
diff --git a/pkg/util/externalplugins.go b/pkg/util/externalplugins.go
new file mode 100644
index 00000000..7789fb74
--- /dev/null
+++ b/pkg/util/externalplugins.go
@@ -0,0 +1,127 @@
+package util
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+
+	"github.com/jenkins-x/go-scm/pkg/hmac"
+	"github.com/jenkins-x/go-scm/scm"
+	"github.com/sirupsen/logrus"
+)
+
+// ParseExternalPluginWebhook parses a webhook relayed to an external plugin
+func ParseExternalPluginWebhook(req *http.Request, secretToken string) (scm.Webhook, error) {
+	data, err := ioutil.ReadAll(
+		io.LimitReader(req.Body, 10000000),
+	)
+	if err != nil {
+		return nil, err
+	}
+
+	log := logrus.WithFields(map[string]interface{}{
+		"URL":     req.URL,
+		"Headers": req.Header,
+		"Body":    string(data),
+	})
+
+	ua := req.Header.Get("User-Agent")
+	if ua != LighthouseUserAgent {
+		return nil, fmt.Errorf("unknown User-Agent %s, expected %s", ua, LighthouseUserAgent)
+	}
+
+	sig := req.Header.Get(LighthouseSignatureHeader)
+	if sig == "" {
+		return nil, scm.ErrSignatureInvalid
+	}
+	if !hmac.ValidatePrefix(data, []byte(secretToken), sig) {
+		return nil, scm.ErrSignatureInvalid
+	}
+
+	kind := req.Header.Get(LighthouseWebhookKindHeader)
+	if kind == "" {
+		return nil, scm.MissingHeader{Header: LighthouseWebhookKindHeader}
+	}
+	var hook scm.Webhook
+	switch scm.WebhookKind(kind) {
+	case scm.WebhookKindBranch:
+		hook = new(scm.BranchHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindCheckRun:
+		hook = new(scm.CheckRunHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindCheckSuite:
+		hook = new(scm.CheckSuiteHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindDeploy:
+		hook = new(scm.DeployHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindDeploymentStatus:
+		hook = new(scm.DeploymentStatusHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindFork:
+		hook = new(scm.ForkHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindInstallation:
+		hook = new(scm.InstallationHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindInstallationRepository:
+		hook = new(scm.InstallationRepositoryHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindIssue:
+		hook = new(scm.IssueHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindIssueComment:
+		hook = new(scm.IssueCommentHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindLabel:
+		hook = new(scm.LabelHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindPing:
+		hook = new(scm.PingHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindPullRequest:
+		hook = new(scm.PullRequestHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindPullRequestComment:
+		hook = new(scm.PullRequestCommentHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindPush:
+		hook = new(scm.PushHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindRelease:
+		hook = new(scm.ReleaseHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindRepository:
+		hook = new(scm.RepositoryHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindReview:
+		hook = new(scm.ReviewHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindReviewCommentHook:
+		hook = new(scm.ReviewCommentHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindStar:
+		hook = new(scm.StarHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindStatus:
+		hook = new(scm.StatusHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindTag:
+		hook = new(scm.TagHook)
+		err = json.Unmarshal(data, hook)
+	case scm.WebhookKindWatch:
+		hook = new(scm.WatchHook)
+		err = json.Unmarshal(data, hook)
+	default:
+		log.WithField("Kind", kind).Warnf("unknown webhook")
+		return nil, scm.UnknownWebhook{Event: kind}
+	}
+	if err != nil {
+		return nil, err
+	}
+
+	return hook, nil
+}
diff --git a/pkg/webhook/events.go b/pkg/webhook/events.go
index 5ade2531..4b12d1d0 100644
--- a/pkg/webhook/events.go
+++ b/pkg/webhook/events.go
@@ -17,9 +17,19 @@ limitations under the License.
 package webhook
 
 import (
+	"bytes"
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"net/http"
 	"net/url"
 	"strconv"
+	"strings"
 	"sync"
+	"time"
 
 	"github.com/jenkins-x/go-scm/scm"
 	"github.com/jenkins-x/jx/v2/pkg/jxfactory"
@@ -27,6 +37,7 @@ import (
 	"github.com/jenkins-x/lighthouse-config/pkg/config"
 	"github.com/jenkins-x/lighthouse/pkg/plugins"
 	"github.com/jenkins-x/lighthouse/pkg/scmprovider"
+	"github.com/jenkins-x/lighthouse/pkg/util"
 	"github.com/sirupsen/logrus"
 )
 
@@ -301,3 +312,105 @@ func actionRelatesToPullRequestComment(action scm.Action, l *logrus.Entry) bool
 		return false
 	}
 }
+
+// externalPluginsForEvent returns whether there are any external plugins that need to
+// get the present event.
+func (s *Server) externalPluginsForEvent(eventKind scm.WebhookKind, srcRepo string) []plugins.ExternalPlugin {
+	var matching []plugins.ExternalPlugin
+	srcOrg := strings.Split(srcRepo, "/")[0]
+
+	for repo, extPlugins := range s.Plugins.Config().ExternalPlugins {
+		// Make sure the repositories match
+		if repo != srcRepo && repo != srcOrg {
+			continue
+		}
+
+		// Make sure the events match
+		for _, p := range extPlugins {
+			if len(p.Events) == 0 {
+				matching = append(matching, p)
+			} else {
+				for _, et := range p.Events {
+					if et == string(eventKind) || et == string(eventKind)+"s" {
+						matching = append(matching, p)
+						break
+					}
+				}
+			}
+		}
+	}
+	return matching
+}
+
+// callExternalPlugins dispatches the provided payload to the external plugins.
+func (s *Server) callExternalPlugins(l *logrus.Entry, externalPlugins []plugins.ExternalPlugin, webhook scm.Webhook, hmacToken string) {
+	headers := http.Header{}
+	headers.Set("User-Agent", util.LighthouseUserAgent)
+	headers.Set(util.LighthouseWebhookKindHeader, string(webhook.Kind()))
+	payload, err := json.Marshal(webhook)
+	if err != nil {
+		l.WithError(err).Errorf("Unable to marshal webhook for relaying to external plugins. Webhook is: %v", webhook)
+		return
+	}
+	mac := hmac.New(sha256.New, []byte(hmacToken))
+	_, err = mac.Write(payload)
+	if err != nil {
+		l.WithError(err).Error("Unable to generate signature for relayed webhook")
+		return
+	}
+	sum := mac.Sum(nil)
+	signature := "sha256=" + hex.EncodeToString(sum)
+	headers.Set(util.LighthouseSignatureHeader, signature)
+	for _, p := range externalPlugins {
+		s.wg.Add(1)
+		go func(p plugins.ExternalPlugin) {
+			defer s.wg.Done()
+			if err := s.dispatch(p.Endpoint, payload, headers); err != nil {
+				l.WithError(err).WithField("external-plugin", p.Name).Error("Error dispatching event to external plugin.")
+			} else {
+				l.WithField("external-plugin", p.Name).Info("Dispatched event to external plugin")
+			}
+		}(p)
+	}
+}
+
+// dispatch creates a new request using the provided payload and headers
+// and dispatches the request to the provided endpoint.
+func (s *Server) dispatch(endpoint string, payload []byte, h http.Header) error {
+	req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewBuffer(payload))
+	if err != nil {
+		return err
+	}
+	req.Header = h
+	resp, err := s.do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	rb, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return err
+	}
+	if resp.StatusCode < 200 || resp.StatusCode > 299 {
+		return fmt.Errorf("response has status %q and body %q", resp.Status, string(rb))
+	}
+	return nil
+}
+
+func (s *Server) do(req *http.Request) (*http.Response, error) {
+	var resp *http.Response
+	var err error
+	backoff := 100 * time.Millisecond
+	maxRetries := 5
+
+	c := &http.Client{}
+	for retries := 0; retries < maxRetries; retries++ {
+		resp, err = c.Do(req)
+		if err == nil {
+			break
+		}
+		time.Sleep(backoff)
+		backoff *= 2
+	}
+	return resp, err
+}
diff --git a/pkg/webhook/webhook.go b/pkg/webhook/webhook.go
index 5882b67b..786aff15 100644
--- a/pkg/webhook/webhook.go
+++ b/pkg/webhook/webhook.go
@@ -1,7 +1,9 @@
 package webhook
 
 import (
+	"bytes"
 	"fmt"
+	"io/ioutil"
 	"net/http"
 	"net/url"
 	"os"
@@ -191,6 +193,21 @@ func (o *Options) handleWebHookRequests(w http.ResponseWriter, r *http.Request)
 	}
 	logrus.Debug("about to parse webhook")
 
+	bodyBytes, err := ioutil.ReadAll(r.Body)
+	if err != nil {
+		logrus.Errorf("failed to Read Body: %s", err.Error())
+		responseHTTPError(w, http.StatusInternalServerError, fmt.Sprintf("500 Internal Server Error: Read Body: %s", err.Error()))
+		return
+	}
+
+	err = r.Body.Close() // must close
+	if err != nil {
+		logrus.Errorf("failed to Close Body: %s", err.Error())
+		responseHTTPError(w, http.StatusInternalServerError, fmt.Sprintf("500 Internal Server Error: Read Close: %s", err.Error()))
+		return
+	}
+
+	r.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes))
 	scmClient, serverURL, err := o.createSCMClient()
 	if err != nil {
 		logrus.Errorf("failed to create SCM scmClient: %s", err.Error())
@@ -256,6 +273,11 @@ func (o *Options) handleWebHookRequests(w http.ResponseWriter, r *http.Request)
 	if err != nil {
 		responseHTTPError(w, http.StatusInternalServerError, fmt.Sprintf("500 Internal Server Error: %s", err.Error()))
 	}
+	// Demux events only to external plugins that require this event.
+	if external := o.server.externalPluginsForEvent(webhook.Kind(), webhook.Repository().FullName); len(external) > 0 {
+		go o.server.callExternalPlugins(l, external, webhook, o.hmacToken())
+	}
+
 	_, err = w.Write([]byte(output))
 	if err != nil {
 		l.Debugf("failed to process the webhook: %v", err)
@@ -412,8 +434,12 @@ func (o *Options) GetFactory() jxfactory.Factory {
 	return o.factory
 }
 
+func (o *Options) hmacToken() string {
+	return os.Getenv("HMAC_TOKEN")
+}
+
 func (o *Options) secretFn(webhook scm.Webhook) (string, error) {
-	return os.Getenv("HMAC_TOKEN"), nil
+	return o.hmacToken(), nil
 }
 
 func (o *Options) createSCMClient() (*scm.Client, string, error) {


Test patch:
diff --git a/pkg/webhook/webhook_test.go b/pkg/webhook/webhook_test.go
index 516f8294..59f41c50 100644
--- a/pkg/webhook/webhook_test.go
+++ b/pkg/webhook/webhook_test.go
@@ -4,6 +4,7 @@ import (
 	"fmt"
 	"io/ioutil"
 	"os"
+	"reflect"
 	"testing"
 
 	"github.com/jenkins-x/go-scm/scm"
@@ -175,6 +176,108 @@ func (suite *WebhookTestSuite) SetupSuite() {
 	}
 }
 
+func TestNeedDemux(t *testing.T) {
+	tests := []struct {
+		name string
+
+		eventType scm.WebhookKind
+		srcRepo   string
+		plugins   map[string][]plugins.ExternalPlugin
+
+		expected []plugins.ExternalPlugin
+	}{
+		{
+			name: "no external plugins",
+
+			eventType: scm.WebhookKindIssueComment,
+			srcRepo:   "kubernetes/test-infra",
+			plugins:   nil,
+
+			expected: nil,
+		},
+		{
+			name: "we have variety",
+
+			eventType: scm.WebhookKindIssueComment,
+			srcRepo:   "kubernetes/test-infra",
+			plugins: map[string][]plugins.ExternalPlugin{
+				"kubernetes/test-infra": {
+					{
+						Name:   "sandwich",
+						Events: []string{"pull_request"},
+					},
+					{
+						Name: "coffee",
+					},
+				},
+				"kubernetes/kubernetes": {
+					{
+						Name:   "gumbo",
+						Events: []string{"issue_comment"},
+					},
+				},
+				"kubernetes": {
+					{
+						Name:   "chicken",
+						Events: []string{"push"},
+					},
+					{
+						Name: "water",
+					},
+					{
+						Name:   "chocolate",
+						Events: []string{"pull_request", "issue_comment", "issues"},
+					},
+				},
+			},
+
+			expected: []plugins.ExternalPlugin{
+				{
+					Name: "coffee",
+				},
+				{
+					Name: "water",
+				},
+				{
+					Name:   "chocolate",
+					Events: []string{"pull_request", "issue_comment", "issues"},
+				},
+			},
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			pa := &plugins.ConfigAgent{}
+			pa.Set(&plugins.Configuration{
+				ExternalPlugins: test.plugins,
+			})
+			s := &Server{Plugins: pa}
+
+			gotPlugins := s.externalPluginsForEvent(test.eventType, test.srcRepo)
+			if len(gotPlugins) != len(test.expected) {
+				t.Fatalf("expected plugins: %+v, got: %+v", test.expected, gotPlugins)
+			}
+			for _, expected := range test.expected {
+				var found bool
+				for _, got := range gotPlugins {
+					if got.Name != expected.Name {
+						continue
+					}
+					if !reflect.DeepEqual(expected, got) {
+						t.Errorf("expected plugin: %+v, got: %+v", expected, got)
+					}
+					found = true
+				}
+				if !found {
+					t.Errorf("expected plugins: %+v, got: %+v", test.expected, gotPlugins)
+					break
+				}
+			}
+		})
+	}
+}
+
 func TestWebhookTestSuite(t *testing.T) {
 	os.Setenv("GIT_TOKEN", "abc123")
 	suite.Run(t, new(WebhookTestSuite))