Repository: openshift-pipelines/pipelines-as-code
Pull request #146 (2021-06-22 17:18:14)

Add pipelinesascode.tekton.dev/max-keep-runs annotation
There can be a lot of PipelineRuns into the user namespace. If the
PipelineRun has the an annotation :

```yaml
pipelinesascode.tekton.dev/max-keep-runs: "maxNumber"
```

after each Successfull Run Pipelinerun as Code will cleanup the old
PipelineRun attached to the same owner/repository and event type
(i.e: pull_request or push) and target branch (i.e: main) .

It will skip the Running PipelineRuns but will not skip the PipelineRuns
with `Unknown` status.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>


Patch:
diff --git a/.golangci.yml b/.golangci.yml
index 26b6d0f0..fbd0a9d1 100644
--- a/.golangci.yml
+++ b/.golangci.yml
@@ -44,7 +44,7 @@ linters:
   - unparam
   - gochecknoinits
   - goconst
-  - gocyclo
+    # - gocyclo
     # - goerr113
   - gofmt
   - goheader
diff --git a/Makefile b/Makefile
index 7a78d50b..7c2a279f 100644
--- a/Makefile
+++ b/Makefile
@@ -3,6 +3,7 @@ QUAY_REPOSITORY=quay.io/openshift-pipeline/pipelines-as-code
 QUAY_REPOSITORY_BRANCH=main
 GO_TEST_FLAGS=-v -cover
 GOLANGCI_LINT=golangci-lint
+GOFUMPT=gofumpt
 
 YAML_FILES := $(shell find . -type f -regex ".*y[a]ml" -print)
 
@@ -73,6 +74,12 @@ clean: ## clean build artifacts
 fmt:
 	@go fmt `go list ./... | grep -v /vendor/`
 
+.PHONY: fumpt ## formats the GO code with gofumpt(excludes vendors dir)
+fumpt:
+	@$(GOFUMPT) -w pkg/**/*go
+
+
+
 .PHONY: help
 help: ## print this help
 	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z0-9_-]+:.*?## / {gsub("\\\\n",sprintf("\n%22c",""), $$2);printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)
diff --git a/README.md b/README.md
index acdabf80..410957cd 100644
--- a/README.md
+++ b/README.md
@@ -1,8 +1,8 @@
-[![Container Repository on Quay](https://quay.io/repository/openshift-pipeline/pipelines-as-code/status "Container Repository on Quay")](https://quay.io/repository/openshift-pipeline/pipelines-as-code) [![codecov](https://codecov.io/gh/openshift-pipelines/pipelines-as-code/branch/main/graph/badge.svg)](https://codecov.io/gh/openshift-pipelines/pipelines-as-code) [![Go Report Card](https://goreportcard.com/badge/google/ko)](https://goreportcard.com/report/openshift-pipelines/pipelines-as-code)
-
 # Pipelines as Code
 
-Pipelines as Code is an opinionated CI based on OpenShift Pipelines / Tekton.
+[![Container Repository on Quay](https://quay.io/repository/openshift-pipeline/pipelines-as-code/status "Container Repository on Quay")](https://quay.io/repository/openshift-pipeline/pipelines-as-code) [![codecov](https://codecov.io/gh/openshift-pipelines/pipelines-as-code/branch/main/graph/badge.svg)](https://codecov.io/gh/openshift-pipelines/pipelines-as-code) [![Go Report Card](https://goreportcard.com/badge/google/ko)](https://goreportcard.com/report/openshift-pipelines/pipelines-as-code)
+
+Pipelines as Code -- An opinionated CI based on OpenShift Pipelines / Tekton.
 
 ## Introduction
 
@@ -15,11 +15,13 @@ to have your pipelines "sits and live" inside the same repository where your
 code is.
 
 The goal of Pipelines as Code is to let you define your
-[Tekton](https://tekton.cd) templates inside your source code repository and have the pipeline run and report the status of the execution when triggered by a Pull Request or a branch push.
+[Tekton](https://tekton.cd) templates inside your source code repository and
+have the pipeline run and report the status of the execution when triggered by a
+Pull Request or a branch push.
 
 ## Components
 
-Pipelines as Code is built on the following technologies :
+Pipelines as Code is built on the following technologies:
 
 - [Tekton Triggers](github.com/tektoncd/triggers): A Tekton Triggers
   EventListener is spun up in a central namespace (`pipelines-as-code`). The
@@ -124,7 +126,7 @@ on the `PipelineRun`. For example when you have these metadatas in your `Pipelin
 
 ```yaml
  metadata:
-    name: pipeline-pr-main
+ name: pipeline-pr-main
  annotations:
     pipelinesascode.tekton.dev/on-target-branch: "[main]"
     pipelinesascode.tekton.dev/on-event: "[pull_request]"
@@ -175,6 +177,23 @@ If there is multiple pipeline matching an event, it will match the first one.
 We are currently not supporting multiple PipelineRuns on a single event but
 this may be something we can consider to implement in the future.
 
+### PipelineRuns Cleanups
+
+There can be a lot of PipelineRuns into an user namespace and Pipelines as Code
+has the ability to only keep a number of PipelineRuns that matches an event.
+
+For example if the PipelineRun has this annotation :
+
+```yaml
+pipelinesascode.tekton.dev/max-keep-runs: "maxNumber"
+```
+
+Pipelines as Code sees this and will start cleaning up right after it finishes a
+successful execution keeping only the maxNumber of PipelineRuns.
+
+It will skip the `Running` PipelineRuns but will not skip the PipelineRuns with
+`Unknown` status.
+
 #### Pipelines as Code resolver
 
 If `Pipelines as Code` sees a PipelineRun with a reference to a `Task` or a
@@ -262,11 +281,11 @@ If the object fetched cannot be parsed as a Tekton `Task` it will error out.
 
 ### Running the Pipeline
 
-* A user create a Pull Request.
+- A user create a Pull Request.
 
-* If the user sending the Pull Request is not the owner of the repository or not a public member of the organization where the repository belong to, `Pipelines as Code` will not run.
+- If the user sending the Pull Request is not the owner of the repository or not a public member of the organization where the repository belong to, `Pipelines as Code` will not run.
 
-* If the user sending the Pull Request is inside an OWNER file located in the repository root in the main branch (the main branch as defined in the Github configuration for the repo) in the `approvers` or `reviewers` section like this :
+- If the user sending the Pull Request is inside an OWNER file located in the repository root in the main branch (the main branch as defined in the Github configuration for the repo) in the `approvers` or `reviewers` section like this :
 
 ```yaml
 approvers:
@@ -302,7 +321,7 @@ with the string `/retest` to ask Pipeline as Code to retest the current PR.
 
 Example :
 
-```
+```text
 Thanks for contributing! This is a much needed bugfix! ❤️
 The failure is not with your PR but seems to be an infra issue.
 
diff --git a/pkg/cli/interface.go b/pkg/cli/interface.go
index feedd1cc..7390639e 100644
--- a/pkg/cli/interface.go
+++ b/pkg/cli/interface.go
@@ -43,5 +43,7 @@ type Params interface {
 type KubeInteractionIntf interface {
 	GetConsoleUI(context.Context, string, string) (string, error)
 	GetNamespace(context.Context, string) error
+	// TODO: we don't need tektonv1beta1client stuff here
 	WaitForPipelineRunSucceed(context.Context, tektonv1beta1client.TektonV1beta1Interface, *v1beta1.PipelineRun, time.Duration) error
+	CleanupPipelines(context.Context, string, *webvcs.RunInfo, int) error
 }
diff --git a/pkg/config/annotation_matcher.go b/pkg/config/annotation_matcher.go
index 8548fa69..2c3f860d 100644
--- a/pkg/config/annotation_matcher.go
+++ b/pkg/config/annotation_matcher.go
@@ -19,6 +19,7 @@ const (
 	onTargetBranchAnnotation = "on-target-branch"
 	onTargetNamespace        = "target-namespace"
 	reValidateTag            = `^\[(.*)\]$`
+	maxKeepRuns              = "max-keep-runs"
 )
 
 // TODO: move to another file since it's common to all annotations_* files
@@ -43,7 +44,7 @@ func getAnnotationValues(annotation string) ([]string, error) {
 }
 
 func MatchPipelinerunByAnnotation(ctx context.Context, pruns []*v1beta1.PipelineRun, cs *cli.Clients,
-	runinfo *webvcs.RunInfo) (*v1beta1.PipelineRun, *apipac.Repository, error) {
+	runinfo *webvcs.RunInfo) (*v1beta1.PipelineRun, *apipac.Repository, map[string]string, error) {
 	configurations := map[string]map[string]string{}
 	repo := &apipac.Repository{}
 	for _, prun := range pruns {
@@ -53,6 +54,11 @@ func MatchPipelinerunByAnnotation(ctx context.Context, pruns []*v1beta1.Pipeline
 			continue
 		}
 
+		if maxPrNumber, ok := prun.GetObjectMeta().GetAnnotations()[pipelinesascode.
+			GroupName+"/"+maxKeepRuns]; ok {
+			configurations[prun.GetGenerateName()]["max-keep-runs"] = maxPrNumber
+		}
+
 		if targetNS, ok := prun.GetObjectMeta().GetAnnotations()[pipelinesascode.
 			GroupName+"/"+onTargetNamespace]; ok {
 			configurations[prun.GetGenerateName()]["target-namespace"] = targetNS
@@ -68,7 +74,7 @@ func MatchPipelinerunByAnnotation(ctx context.Context, pruns []*v1beta1.Pipeline
 			matched, err := matchOnAnnotation(targetEvent, runinfo.EventType, false)
 			configurations[prun.GetGenerateName()]["target-event"] = targetEvent
 			if err != nil {
-				return nil, nil, err
+				return nil, nil, map[string]string{}, err
 			}
 			if !matched {
 				continue
@@ -80,14 +86,14 @@ func MatchPipelinerunByAnnotation(ctx context.Context, pruns []*v1beta1.Pipeline
 			matched, err := matchOnAnnotation(targetBranch, runinfo.BaseBranch, true)
 			configurations[prun.GetGenerateName()]["target-branch"] = targetBranch
 			if err != nil {
-				return nil, nil, err
+				return nil, nil, map[string]string{}, err
 			}
 			if !matched {
 				continue
 			}
 		}
 
-		return prun, repo, nil
+		return prun, repo, configurations[prun.GetGenerateName()], nil
 	}
 
 	cs.Log.Infof("cannot match between event and pipelineRuns: URL=%s baseBranch=%s, "+
@@ -102,7 +108,7 @@ func MatchPipelinerunByAnnotation(ctx context.Context, pruns []*v1beta1.Pipeline
 	}
 
 	// TODO: more descriptive error message
-	return nil, nil, fmt.Errorf("cannot match pipeline from webhook to pipelineruns")
+	return nil, nil, map[string]string{}, fmt.Errorf("cannot match pipeline from webhook to pipelineruns")
 }
 
 func matchOnAnnotation(annotations string, runinfoValue string, branchMatching bool) (bool, error) {
diff --git a/pkg/kubeinteraction/cleanups.go b/pkg/kubeinteraction/cleanups.go
new file mode 100644
index 00000000..d27d6e56
--- /dev/null
+++ b/pkg/kubeinteraction/cleanups.go
@@ -0,0 +1,56 @@
+package kubeinteraction
+
+import (
+	"context"
+	"fmt"
+	"sort"
+
+	"github.com/openshift-pipelines/pipelines-as-code/pkg/webvcs"
+	"github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"knative.dev/pkg/apis"
+)
+
+// From tekton cli prsort package
+type prByCompletionTime []v1beta1.PipelineRun
+
+func (prs prByCompletionTime) Len() int      { return len(prs) }
+func (prs prByCompletionTime) Swap(i, j int) { prs[i], prs[j] = prs[j], prs[i] }
+func (prs prByCompletionTime) Less(i, j int) bool {
+	if prs[j].Status.CompletionTime == nil {
+		return false
+	}
+	if prs[i].Status.CompletionTime == nil {
+		return true
+	}
+	return prs[j].Status.CompletionTime.Before(prs[i].Status.CompletionTime)
+}
+
+func (k Interaction) CleanupPipelines(ctx context.Context, namespace string, runinfo *webvcs.RunInfo, maxKeep int) error {
+	labelSelector := fmt.Sprintf("tekton.dev/pipeline-ascode-owner=%s,"+
+		"tekton.dev/pipeline-ascode-repository=%s, tekton.dev/pipeline-ascode-event-type=%s, "+
+		"tekton.dev/pipeline-ascode-branch=%s",
+		runinfo.Owner, runinfo.Repository, runinfo.EventType, runinfo.BaseBranch)
+
+	pruns, err := k.Clients.Tekton.TektonV1beta1().PipelineRuns(namespace).List(ctx, metav1.ListOptions{
+		LabelSelector: labelSelector,
+	})
+
+	sort.Sort(prByCompletionTime(pruns.Items))
+
+	for c, v := range pruns.Items {
+		if v.GetStatusCondition().GetCondition(apis.ConditionSucceeded).GetReason() == "Running" {
+			k.Clients.Log.Infof("Skipping Cleanining up %s since it is currently Running", v.GetName())
+			continue
+		}
+		if c >= maxKeep {
+			k.Clients.Log.Infof("Cleaning old PipelineRun: %s", v.GetName())
+			err := k.Clients.Tekton.TektonV1beta1().PipelineRuns(namespace).Delete(ctx, v.GetName(), metav1.DeleteOptions{})
+			if err != nil {
+				return err
+			}
+		}
+	}
+
+	return err
+}
diff --git a/pkg/pipelineascode/pipelineascode.go b/pkg/pipelineascode/pipelineascode.go
index a3e58153..ef2b8c55 100644
--- a/pkg/pipelineascode/pipelineascode.go
+++ b/pkg/pipelineascode/pipelineascode.go
@@ -3,6 +3,7 @@ package pipelineascode
 import (
 	"context"
 	"fmt"
+	"strconv"
 	"strings"
 	"time"
 
@@ -146,10 +147,11 @@ func Run(ctx context.Context, cs *cli.Clients, k8int cli.KubeInteractionIntf, ru
 	}
 
 	// Match the pipelinerun with annotation
-	pipelineRun, annotationRepo, err := config.MatchPipelinerunByAnnotation(ctx, pipelineRuns, cs, runinfo)
+	pipelineRun, annotationRepo, config, err := config.MatchPipelinerunByAnnotation(ctx, pipelineRuns, cs, runinfo)
 	if err != nil {
 		return err
 	}
+
 	if annotationRepo.Spec.Namespace != "" {
 		repo = annotationRepo
 	}
@@ -195,6 +197,19 @@ func Run(ctx context.Context, cs *cli.Clients, k8int cli.KubeInteractionIntf, ru
 		cs.Log.Info("PipelineRun has failed.")
 	}
 
+	// Do cleanups
+	if keepMaxPipeline, ok := config["max-keep-runs"]; ok {
+		max, err := strconv.Atoi(keepMaxPipeline)
+		if err != nil {
+			return err
+		}
+
+		err = k8int.CleanupPipelines(ctx, repo.Spec.Namespace, runinfo, max)
+		if err != nil {
+			return err
+		}
+	}
+
 	// Post the final status to GitHub check status with a nice breakdown and
 	// tekton cli describe output.
 	newPr, err := postFinalStatus(ctx, cs, k8int, runinfo, pr.Name, repo.Spec.Namespace)
@@ -223,6 +238,7 @@ func Run(ctx context.Context, cs *cli.Clients, k8int cli.KubeInteractionIntf, ru
 		copy(lastrepo.Status, lastrepo.Status[len(lastrepo.Status)-maxPipelineRunStatusRun+1:])
 		lastrepo.Status = lastrepo.Status[:maxPipelineRunStatusRun-1]
 	}
+
 	lastrepo.Status = append(lastrepo.Status, repoStatus)
 	nrepo, err := cs.PipelineAsCode.PipelinesascodeV1alpha1().Repositories(lastrepo.Namespace).Update(
 		ctx, lastrepo, metav1.UpdateOptions{})


Test patch:
diff --git a/pkg/config/annotation_matcher_test.go b/pkg/config/annotation_matcher_test.go
index 097d5f7d..cf6e3495 100644
--- a/pkg/config/annotation_matcher_test.go
+++ b/pkg/config/annotation_matcher_test.go
@@ -29,6 +29,7 @@ func TestMatchPipelinerunAnnotationAndRepositories(t *testing.T) {
 				pipelinesascode.GroupName + "/" + onTargetNamespace:        targetNamespace,
 				pipelinesascode.GroupName + "/" + onEventAnnotation:        "[pull_request]",
 				pipelinesascode.GroupName + "/" + onTargetBranchAnnotation: fmt.Sprintf("[%s]", mainBranch),
+				pipelinesascode.GroupName + "/" + maxKeepRuns:              "2",
 			},
 		},
 	}
@@ -94,7 +95,7 @@ func TestMatchPipelinerunAnnotationAndRepositories(t *testing.T) {
 			observer, log := zapobserver.New(zap.InfoLevel)
 			logger := zap.New(observer).Sugar()
 			client := &cli.Clients{PipelineAsCode: cs.PipelineAsCode, Log: logger}
-			got, repo, err := MatchPipelinerunByAnnotation(ctx,
+			got, repo, _, err := MatchPipelinerunByAnnotation(ctx,
 				tt.args.pruns,
 				client, tt.args.runinfo)
 
@@ -321,7 +322,7 @@ func TestMatchPipelinerunByAnnotation(t *testing.T) {
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			ctx, _ := rtesting.SetupFakeContext(t)
-			got, _, err := MatchPipelinerunByAnnotation(ctx, tt.args.pruns, cs, tt.args.runinfo)
+			got, _, _, err := MatchPipelinerunByAnnotation(ctx, tt.args.pruns, cs, tt.args.runinfo)
 			if (err != nil) != tt.wantErr {
 				t.Errorf("MatchPipelinerunByAnnotation() error = %v, wantErr %v", err, tt.wantErr)
 				return
diff --git a/pkg/pipelineascode/pipelineascode_test.go b/pkg/pipelineascode/pipelineascode_test.go
index 9f67e7c5..a16a9893 100644
--- a/pkg/pipelineascode/pipelineascode_test.go
+++ b/pkg/pipelineascode/pipelineascode_test.go
@@ -106,6 +106,7 @@ func TestRun(t *testing.T) {
 		finalLogText                 string
 		repositories                 []*v1alpha1.Repository
 		skipReplyingOrgPublicMembers bool
+		expectedNumberofCleanups     int
 	}{
 		{
 			name: "pull request",
@@ -123,6 +124,7 @@ func TestRun(t *testing.T) {
 			finalStatus:  "neutral",
 			finalLogText: "<th>Status</th><th>Duration</th><th>Name</th>",
 		},
+
 		{
 			name: "No match",
 			runinfo: &webvcs.RunInfo{
@@ -264,6 +266,23 @@ func TestRun(t *testing.T) {
 			finalLogText:                 "is not allowed to run CI on this repo",
 			skipReplyingOrgPublicMembers: true,
 		},
+		{
+			name: "Keep max number of pipelineruns",
+			runinfo: &webvcs.RunInfo{
+				SHA:        "principale",
+				Owner:      "organizationes",
+				Repository: "lagaffe",
+				URL:        "https://service/documentation",
+				HeadBranch: "press",
+				BaseBranch: "main",
+				Sender:     "fantasio",
+				EventType:  "pull_request",
+			},
+			tektondir:                "testdata/max-keep-runs",
+			finalStatus:              "neutral",
+			finalLogText:             "<th>Status</th><th>Duration</th><th>Name</th>",
+			expectedNumberofCleanups: 10,
+		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
@@ -306,7 +325,8 @@ func TestRun(t *testing.T) {
 				Dynamic:        dc,
 			}
 			k8int := &kitesthelper.KinterfaceTest{
-				ConsoleURL: "https://console.url",
+				ConsoleURL:               "https://console.url",
+				ExpectedNumberofCleanups: tt.expectedNumberofCleanups,
 			}
 			err := Run(ctx, cs, k8int, tt.runinfo)
 
diff --git a/pkg/pipelineascode/testdata/max-keep-runs/run.yaml b/pkg/pipelineascode/testdata/max-keep-runs/run.yaml
new file mode 100644
index 00000000..897c896a
--- /dev/null
+++ b/pkg/pipelineascode/testdata/max-keep-runs/run.yaml
@@ -0,0 +1,18 @@
+---
+apiVersion: tekton.dev/v1beta1
+kind: PipelineRun
+metadata:
+  name: max-keep-runs
+  annotations:
+    pipelinesascode.tekton.dev/on-target-branch: "[main]"
+    pipelinesascode.tekton.dev/on-event: "[pull_request]"
+    pipelinesascode.tekton.dev/max-keep-runs: "10"
+spec:
+  pipelineSpec:
+    tasks:
+      - name: max
+        taskSpec:
+          steps:
+            - name: success
+              image: registry.access.redhat.com/ubi8/ubi-minimal:8.3
+              script: 'exit 0'
diff --git a/pkg/test/kubernetestint/kubernetesint.go b/pkg/test/kubernetestint/kubernetesint.go
index 29301296..2b1d064c 100644
--- a/pkg/test/kubernetestint/kubernetesint.go
+++ b/pkg/test/kubernetestint/kubernetesint.go
@@ -3,15 +3,18 @@ package kubernetestint
 import (
 	"context"
 	"errors"
+	"fmt"
 	"time"
 
+	"github.com/openshift-pipelines/pipelines-as-code/pkg/webvcs"
 	"github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1"
 	tektonv1beta1client "github.com/tektoncd/pipeline/pkg/client/clientset/versioned/typed/pipeline/v1beta1"
 )
 
 type KinterfaceTest struct {
-	ConsoleURL     string
-	NamespaceError bool
+	ConsoleURL               string
+	NamespaceError           bool
+	ExpectedNumberofCleanups int
 }
 
 func (k *KinterfaceTest) GetConsoleUI(ctx context.Context, ns string, pr string) (string, error) {
@@ -28,3 +31,10 @@ func (k *KinterfaceTest) GetNamespace(ctx context.Context, ns string) error {
 func (k *KinterfaceTest) WaitForPipelineRunSucceed(ctx context.Context, tektonbeta1 tektonv1beta1client.TektonV1beta1Interface, pr *v1beta1.PipelineRun, polltimeout time.Duration) error {
 	return nil
 }
+
+func (k *KinterfaceTest) CleanupPipelines(ctx context.Context, namespace string, runinfo *webvcs.RunInfo, maxKeep int) error {
+	if k.ExpectedNumberofCleanups != maxKeep {
+		return fmt.Errorf("we wanted %d and we got %d", k.ExpectedNumberofCleanups, maxKeep)
+	}
+	return nil
+}