Repository: kubesphere/ks-devops
Pull request #535 (2022-04-06 13:32:27)

Fix the conflicts with duplicated PipelineRuns
<!-- Thanks for sending a pull request! Here are some tips for you:

1. If you want **faster** PR reviews, read how: https://github.com/kubesphere/community/blob/master/developer-guide/development/the-pr-author-guide-to-getting-through-code-review.md
2. In case you want to know how your PR got reviewed, read: https://github.com/kubesphere/community/blob/master/developer-guide/development/code-review-guide.md
3. Here are some coding conventions followed by the KubeSphere community: https://github.com/kubesphere/community/blob/master/developer-guide/development/coding-conventions.md
4. Additional open-source best practice: https://github.com/LinuxSuRen/open-source-best-practice
-->

### What type of PR is this?
<!-- 
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design
/kind chore

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

### What this PR does / why we need it:

### Which issue(s) this PR fixes:
<!--
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
Please leave it or change # to be None if there is no corresponding issue that exists
-->
Fixes #

Please feel free to use the following images to test:
```
kubespheredev/devops-controller:dev-v3.2.1-ce8fbda
```

```
kubespheredev/devops-apiserver:dev-v3.2.1-ce8fbda
```

### Special notes for reviewers:
<!--
You can use the following command to let the DevOps SIG members help you to review your PR.
/cc @kubesphere/sig-devops 
And please avoid cc any individual.
-->
Please check the following list before waiting reviewers:

- [ ] Already committed the CRD files to [the Helm Chart](https://github.com/kubesphere-sigs/ks-devops-helm-chart/) if you created some new CRDs
- [ ] Already [added the permission](https://github.com/kubesphere/ks-installer/blob/9e063b085a0e43fdb3d0d9e3e7f4149146f14b9c/roles/ks-core/prepare/files/ks-init/role-templates.yaml) for the new API
- [ ] Already added the RBAC markers for the new controllers

### Does this PR introduce a user-facing change??
<!--
If no, just write "None" in the release-note block below.
If yes, a release note is required:
Enter your extended-release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

Please keep the note be same as your PR title if you believe it should be in the release notes.
-->
```release-note

```


ks-ci-bot: @LinuxSuRen: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our [release note process](https://git.k8s.io/community/contributors/guide/release-notes.md) to remove it.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
codecov[bot]: # [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/535?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) Report
> Merging [#535](https://codecov.io/gh/kubesphere/ks-devops/pull/535?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (ce8fbda) into [master](https://codecov.io/gh/kubesphere/ks-devops/commit/591dd1788a78f76f3a7199d0fd9e4637774d1b26?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (591dd17) will **decrease** coverage by `0.01%`.
> The diff coverage is `70.96%`.

```diff
@@            Coverage Diff             @@
##           master     #535      +/-   ##
==========================================
- Coverage   77.50%   77.48%   -0.02%     
==========================================
  Files         142      143       +1     
  Lines       23153    23173      +20     
==========================================
+ Hits        17945    17956      +11     
- Misses       5004     5010       +6     
- Partials      204      207       +3     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `77.48% <70.96%> (-0.02%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/kubesphere/ks-devops/pull/535?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) | Coverage Δ | |
|---|---|---|
| [...rs/jenkins/pipelinerun/pipelinerun\_synchronizer.go](https://codecov.io/gh/kubesphere/ks-devops/pull/535/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvamVua2lucy9waXBlbGluZXJ1bi9waXBlbGluZXJ1bl9zeW5jaHJvbml6ZXIuZ28=) | `40.81% <66.66%> (+1.68%)` | :arrow_up: |
| [...pis/devops/v1alpha3/webhook/workflowrun\_handler.go](https://codecov.io/gh/kubesphere/ks-devops/pull/535/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL2thcGlzL2Rldm9wcy92MWFscGhhMy93ZWJob29rL3dvcmtmbG93cnVuX2hhbmRsZXIuZ28=) | `80.00% <72.00%> (-6.21%)` | :arrow_down: |
| [pkg/api/devops/v1alpha3/template\_types.go](https://codecov.io/gh/kubesphere/ks-devops/pull/535/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL2FwaS9kZXZvcHMvdjFhbHBoYTMvdGVtcGxhdGVfdHlwZXMuZ28=) | `50.00% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/535?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/535?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere). Last update [591dd17...ce8fbda](https://codecov.io/gh/kubesphere/ks-devops/pull/535?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).


Patch:
diff --git a/controllers/jenkins/pipelinerun/pipelinerun_synchronizer.go b/controllers/jenkins/pipelinerun/pipelinerun_synchronizer.go
index cd97b71..52f40ec 100644
--- a/controllers/jenkins/pipelinerun/pipelinerun_synchronizer.go
+++ b/controllers/jenkins/pipelinerun/pipelinerun_synchronizer.go
@@ -71,6 +71,12 @@ func (r *SyncReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 		return ctrl.Result{}, err
 	}
 
+	// skip if there are any PipelineRuns don't have v1alpha3.JenkinsPipelineRunIDAnnoKey
+	// wait it until all PipelineRuns have the run id from Jenkins
+	if hasPendingPipelineRuns(prList.Items) {
+		return ctrl.Result{}, nil
+	}
+
 	boClient := job.BlueOceanClient{
 		JenkinsCore:  r.JenkinsCore,
 		Organization: "jenkins",
@@ -105,6 +111,16 @@ func (r *SyncReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 	return ctrl.Result{}, nil
 }
 
+func hasPendingPipelineRuns(items []v1alpha3.PipelineRun) bool {
+	for i := range items {
+		item := items[i]
+		if id, ok := item.Annotations[v1alpha3.JenkinsPipelineRunIDAnnoKey]; !ok || id == "" {
+			return true
+		}
+	}
+	return false
+}
+
 func createBarePipelineRunsIfNotPresent(finder pipelineRunFinder, pipeline *v1alpha3.Pipeline, jobRuns []job.PipelineRun) []v1alpha3.PipelineRun {
 	var pipelineRunsToBeCreated []v1alpha3.PipelineRun
 	for i := range jobRuns {
diff --git a/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler.go b/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler.go
index d25fe09..ae09c3a 100644
--- a/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler.go
+++ b/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler.go
@@ -18,11 +18,15 @@ package webhook
 
 import (
 	"context"
+	"errors"
 	"fmt"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/client-go/util/retry"
+	"k8s.io/klog"
 	"kubesphere.io/devops/pkg/event/workflowrun"
 	"strings"
+	"time"
 
-	"k8s.io/klog"
 	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
 	"kubesphere.io/devops/pkg/kapis/devops/v1alpha3/pipelinerun"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -94,29 +98,51 @@ func (handler *Handler) handleWorkflowRunInitialize(workflowRunData *workflowrun
 		return nil
 	}
 
-	// TODO Execute process below asynchronously
+	go func() {
+		pipelineRunList := &v1alpha3.PipelineRunList{}
+		if err := handler.List(context.Background(), pipelineRunList,
+			client.InNamespace(identifier.namespaceName),
+			client.MatchingFields{v1alpha3.PipelineRunIdentifierIndexerName: identifier.String()}); err != nil {
+			return
+		}
 
-	pipelineRunList := &v1alpha3.PipelineRunList{}
-	if err := handler.List(context.Background(), pipelineRunList,
-		client.InNamespace(identifier.namespaceName),
-		client.MatchingFields{v1alpha3.PipelineRunIdentifierIndexerName: identifier.String()}); err != nil {
-		return err
-	}
+		// TODO Currently, no good way to check if this event originally comes from Jenkins.
+		// Let's wait for the PipelineRun controller to set the Jenkins job run ID
+		pipelineRunListErr := handler.retryCheckPipelineRunList(identifier)
+		if pipelineRunListErr != nil {
+			parameters, err := workflowRunData.Actions.GetParameters()
+			if err != nil {
+				return
+			}
+
+			pipelineRun, err := handler.createPipelineRun(identifier, convertParameters(parameters))
+			if err != nil {
+				return
+			}
+			klog.Infof("Created a PipelineRun: %s/%s", pipelineRun.Namespace, pipelineRun.Name)
+		}
+	}()
+	return nil
+}
 
-	if len(pipelineRunList.Items) == 0 {
-		parameters, err := workflowRunData.Actions.GetParameters()
-		if err != nil {
+func (handler *Handler) retryCheckPipelineRunList(id *pipelineRunIdentifier) error {
+	return retry.OnError(retry.DefaultRetry, func(err error) bool {
+		return true
+	}, func() error {
+		pipelineRunList := &v1alpha3.PipelineRunList{}
+		if err := handler.List(context.Background(), pipelineRunList,
+			client.InNamespace(id.namespaceName),
+			client.MatchingFields{v1alpha3.PipelineRunIdentifierIndexerName: id.String()}); err != nil {
 			return err
 		}
 
-		pipelineRun, err := handler.createPipelineRun(identifier, convertParameters(parameters))
-		if err != nil {
-			return err
+		if len(pipelineRunList.Items) > 0 {
+			return nil
 		}
-		klog.Infof("Created a PipelineRun: %s/%s", pipelineRun.Namespace, pipelineRun.Name)
-		return nil
-	}
-	return nil
+
+		time.Sleep(time.Second)
+		return errors.New(string(metav1.StatusReasonNotFound))
+	})
 }
 
 func (handler *Handler) createPipelineRun(identifier *pipelineRunIdentifier, parameters []v1alpha3.Parameter) (*v1alpha3.PipelineRun, error) {


Test patch:
diff --git a/controllers/jenkins/pipelinerun/pipelinerun_synchronizer_test.go b/controllers/jenkins/pipelinerun/pipelinerun_synchronizer_test.go
index eac99da..948add1 100644
--- a/controllers/jenkins/pipelinerun/pipelinerun_synchronizer_test.go
+++ b/controllers/jenkins/pipelinerun/pipelinerun_synchronizer_test.go
@@ -433,3 +433,43 @@ func Test_createBarePipelineRunsIfNotPresent(t *testing.T) {
 		})
 	}
 }
+
+func Test_hasPendingPipelineRuns(t *testing.T) {
+	type args struct {
+		items []v1alpha3.PipelineRun
+	}
+	tests := []struct {
+		name string
+		args args
+		want bool
+	}{{
+		name: "have pending PipelineRuns",
+		args: args{
+			items: []v1alpha3.PipelineRun{{
+				ObjectMeta: v1.ObjectMeta{
+					Annotations: map[string]string{
+						"a": "1",
+					},
+				},
+			}},
+		},
+		want: true,
+	}, {
+		name: "not have pending PipelineRuns",
+		args: args{
+			items: []v1alpha3.PipelineRun{{
+				ObjectMeta: v1.ObjectMeta{
+					Annotations: map[string]string{
+						v1alpha3.JenkinsPipelineRunIDAnnoKey: "1",
+					},
+				},
+			}},
+		},
+		want: false,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, tt.want, hasPendingPipelineRuns(tt.args.items), "hasPendingPipelineRuns(%v)", tt.args.items)
+		})
+	}
+}
diff --git a/pkg/kapis/devops/v1alpha3/webhook/register_test.go b/pkg/kapis/devops/v1alpha3/webhook/register_test.go
index cd96d0a..1a62e06 100644
--- a/pkg/kapis/devops/v1alpha3/webhook/register_test.go
+++ b/pkg/kapis/devops/v1alpha3/webhook/register_test.go
@@ -18,13 +18,17 @@ package webhook
 
 import (
 	"context"
+	"errors"
+	"fmt"
 	"io"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	"k8s.io/client-go/kubernetes/scheme"
+	"k8s.io/client-go/util/retry"
 	"net/http"
 	"net/http/httptest"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/emicklei/go-restful"
 	"github.com/stretchr/testify/assert"
@@ -112,9 +116,20 @@ func TestJenkinsWebhook(t *testing.T) {
 			}`,
 		},
 		assertion: func(t *testing.T, c client.Client) {
-			pipelineruns := &v1alpha3.PipelineRunList{}
-			assert.Nil(t, c.List(context.Background(), pipelineruns))
-			assert.Equal(t, 1, len(pipelineruns.Items))
+			_ = retry.OnError(retry.DefaultRetry, func(err error) bool {
+				return true
+			}, func() error {
+				pipelineRuns := &v1alpha3.PipelineRunList{}
+				_ = c.List(context.Background(), pipelineRuns)
+				if len(pipelineRuns.Items) > 0 {
+					return nil
+				}
+				time.Sleep(time.Second * 2)
+				return errors.New("not found")
+			})
+			pipelineRuns := &v1alpha3.PipelineRunList{}
+			_ = c.List(context.Background(), pipelineRuns)
+			assert.Equal(t, 1, len(pipelineRuns.Items))
 		},
 	}, {
 		name: "Should not create any PipelineRuns when type doesn't start with run",
@@ -164,3 +179,13 @@ func TestJenkinsWebhook(t *testing.T) {
 		})
 	}
 }
+
+func Test(t *testing.T) {
+	err := retry.OnError(retry.DefaultRetry, func(err error) bool {
+		return true
+	}, func() error {
+		time.Sleep(time.Second)
+		return errors.New("")
+	})
+	fmt.Println(err)
+}
diff --git a/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler_test.go b/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler_test.go
index 6c43fd0..0cac79f 100644
--- a/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler_test.go
+++ b/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler_test.go
@@ -18,9 +18,12 @@ package webhook
 
 import (
 	"context"
+	"errors"
+	"k8s.io/client-go/util/retry"
 	"kubesphere.io/devops/pkg/event/workflowrun"
 	"reflect"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -235,6 +238,17 @@ func TestHandler_handleWorkflowRunInitialize(t *testing.T) {
 		},
 		wantErr: false,
 		assertion: func(t *testing.T, c client.Client) {
+			_ = retry.OnError(retry.DefaultRetry, func(err error) bool {
+				return true
+			}, func() error {
+				pipelineRuns := &v1alpha3.PipelineRunList{}
+				_ = c.List(context.Background(), pipelineRuns)
+				if len(pipelineRuns.Items) > 0 {
+					return nil
+				}
+				time.Sleep(time.Second * 2)
+				return errors.New("not found")
+			})
 			pipelineRuns := &v1alpha3.PipelineRunList{}
 			_ = c.List(context.Background(), pipelineRuns)
 			assert.Equal(t, 1, len(pipelineRuns.Items))
@@ -244,7 +258,7 @@ func TestHandler_handleWorkflowRunInitialize(t *testing.T) {
 		args: args{
 			workflowRunData: createWorkflowRun("fake-namepsace", "fake-pipeline", "1", false),
 		},
-		wantErr: true,
+		wantErr: false,
 	}, {
 		name: "Should create nothing if WorkflowRunData is invalid",
 		args: args{