Repository: kubesphere/ks-devops
Pull request #530 (2022-03-31 07:26:39)

Avoid to return the secret text from API
<!-- Thanks for sending a pull request! Here are some tips for you:

1. If you want **faster** PR reviews, read how: https://github.com/kubesphere/community/blob/master/developer-guide/development/the-pr-author-guide-to-getting-through-code-review.md
2. In case you want to know how your PR got reviewed, read: https://github.com/kubesphere/community/blob/master/developer-guide/development/code-review-guide.md
3. Here are some coding conventions followed by the KubeSphere community: https://github.com/kubesphere/community/blob/master/developer-guide/development/coding-conventions.md
4. Additional open-source best practice: https://github.com/LinuxSuRen/open-source-best-practice
-->

### What type of PR is this?
<!-- 
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design
/kind chore

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

### What this PR does / why we need it:

### Which issue(s) this PR fixes:
<!--
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
Please leave it or change # to be None if there is no corresponding issue that exists
-->
Fixes #

### Special notes for reviewers:
<!--
You can use the following command to let the DevOps SIG members help you to review your PR.
/cc @kubesphere/sig-devops 
And please avoid cc any individual.
-->
Please check the following list before waiting reviewers:

- [ ] Already committed the CRD files to [the Helm Chart](https://github.com/kubesphere-sigs/ks-devops-helm-chart/) if you created some new CRDs
- [ ] Already [added the permission](https://github.com/kubesphere/ks-installer/blob/9e063b085a0e43fdb3d0d9e3e7f4149146f14b9c/roles/ks-core/prepare/files/ks-init/role-templates.yaml) for the new API
- [ ] Already added the RBAC markers for the new controllers

### Does this PR introduce a user-facing change??
<!--
If no, just write "None" in the release-note block below.
If yes, a release note is required:
Enter your extended-release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

Please keep the note be same as your PR title if you believe it should be in the release notes.
-->
```release-note
None
```


ks-ci-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubesphere/ks-devops/pull/530#" title="Author self-approved">LinuxSuRen</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubesphere%2Fks-devops).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[pkg/models/devops/OWNERS](https://github.com/kubesphere/ks-devops/blob/master/pkg/models/devops/OWNERS)~~ [LinuxSuRen]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
codecov[bot]: # [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/530?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) Report
> Merging [#530](https://codecov.io/gh/kubesphere/ks-devops/pull/530?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (d572da0) into [master](https://codecov.io/gh/kubesphere/ks-devops/commit/edbd79e3f725e2f900894f2f3309c66b90952615?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (edbd79e) will **not change** coverage.
> The diff coverage is `0.00%`.

```diff
@@           Coverage Diff           @@
##           master     #530   +/-   ##
=======================================
  Coverage   77.43%   77.43%           
=======================================
  Files         142      142           
  Lines       23144    23144           
=======================================
  Hits        17921    17921           
  Misses       5017     5017           
  Partials      206      206           
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `77.43% <0.00%> (ø)` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/kubesphere/ks-devops/pull/530?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) | Coverage Δ | |
|---|---|---|
| [pkg/models/devops/devops.go](https://codecov.io/gh/kubesphere/ks-devops/pull/530/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL21vZGVscy9kZXZvcHMvZGV2b3BzLmdv) | `19.33% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/530?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/530?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere). Last update [edbd79e...d572da0](https://codecov.io/gh/kubesphere/ks-devops/pull/530?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).

LinuxSuRen: /hold

Patch:
diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml
index 561bab3..0a1ca3c 100644
--- a/.github/workflows/lint.yaml
+++ b/.github/workflows/lint.yaml
@@ -49,9 +49,8 @@ jobs:
             exit 1
           fi
       - name: golint
-        uses: golangci/golangci-lint-action@v2
+        uses: golangci/golangci-lint-action@v3.1.0
         with:
           # Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
           version: v1.41.1
-          skip-build-cache: true
-          skip-pkg-cache: true
+          skip-cache: true
diff --git a/cmd/controller/app/controllers.go b/cmd/controller/app/controllers.go
index 1281460..51ced88 100644
--- a/cmd/controller/app/controllers.go
+++ b/cmd/controller/app/controllers.go
@@ -97,7 +97,8 @@ func getAllControllers(mgr manager.Manager, client k8s.Client, informerFactory i
 	s *options.DevOpsControllerManagerOptions) map[string]func(mgr manager.Manager) error {
 
 	argocdReconciler := &argocd.Reconciler{
-		Client: mgr.GetClient(),
+		Client:        mgr.GetClient(),
+		ArgoNamespace: s.ArgoCDOption.Namespace,
 	}
 	argocdAppReconciler := &argocd.ApplicationReconciler{
 		Client: mgr.GetClient(),
diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml
index ba903d1..4ae43f3 100644
--- a/config/rbac/role.yaml
+++ b/config/rbac/role.yaml
@@ -38,8 +38,10 @@ rules:
   - secrets
   verbs:
   - create
+  - delete
   - get
   - list
+  - patch
   - update
   - watch
 - apiGroups:
@@ -56,6 +58,7 @@ rules:
   - applications
   verbs:
   - create
+  - delete
   - get
   - list
   - update
@@ -154,8 +157,11 @@ rules:
   resources:
   - gitrepositories
   verbs:
+  - create
+  - delete
   - get
   - list
+  - patch
   - update
   - watch
 - apiGroups:
diff --git a/controllers/argocd/application_controller.go b/controllers/argocd/application_controller.go
index 36af299..3deb71c 100644
--- a/controllers/argocd/application_controller.go
+++ b/controllers/argocd/application_controller.go
@@ -25,11 +25,15 @@ import (
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/tools/record"
+	"k8s.io/client-go/util/retry"
 	"kubesphere.io/devops/pkg/api/gitops/v1alpha1"
 	"kubesphere.io/devops/pkg/utils/k8sutil"
+	"reflect"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/event"
 	"sigs.k8s.io/controller-runtime/pkg/predicate"
+	"strings"
 )
 
 //+kubebuilder:rbac:groups=gitops.kubesphere.io,resources=applications,verbs=get;list;update
@@ -95,14 +99,14 @@ func (r *ApplicationReconciler) reconcileArgoApplication(app *v1alpha1.Applicati
 
 		if err == nil {
 			k8sutil.RemoveFinalizer(&app.ObjectMeta, v1alpha1.ApplicationFinalizerName)
+			k8sutil.RemoveFinalizer(&app.ObjectMeta, v1alpha1.ArgoCDResourcesFinalizer)
 			err = r.Update(context.TODO(), app)
 		}
 		return
 	}
-	if k8sutil.AddFinalizer(&app.ObjectMeta, v1alpha1.ApplicationFinalizerName) {
-		if err = r.Update(context.TODO(), app); err != nil {
-			return
-		}
+
+	if app.Spec.ArgoApp.Spec.Project != app.Namespace {
+		app.Spec.ArgoApp.Spec.Project = app.Namespace // update the cache as well
 	}
 
 	if err = r.Get(ctx, types.NamespacedName{
@@ -149,10 +153,29 @@ func (r *ApplicationReconciler) reconcileArgoApplication(app *v1alpha1.Applicati
 			var newArgoApp *unstructured.Unstructured
 			if newArgoApp, err = createUnstructuredApplication(app); err == nil {
 				argoApp.Object["spec"] = newArgoApp.Object["spec"]
-				err = r.Update(ctx, argoApp)
+				argoApp.SetFinalizers(newArgoApp.GetFinalizers())
+				err = retry.RetryOnConflict(retry.DefaultRetry, func() (err error) {
+					latestArgoApp := createBareArgoCDApplicationObject()
+					if err = r.Get(context.Background(), types.NamespacedName{
+						Namespace: argoApp.GetNamespace(),
+						Name:      argoApp.GetName(),
+					}, latestArgoApp); err != nil {
+						return
+					}
+
+					argoApp.SetResourceVersion(latestArgoApp.GetResourceVersion())
+					err = r.Update(ctx, argoApp)
+					return
+				})
 			}
 		}
 	}
+
+	if err == nil {
+		if err = r.setArgoProject(app); err != nil {
+			return
+		}
+	}
 	return
 }
 
@@ -181,6 +204,17 @@ func createUnstructuredApplication(app *v1alpha1.Application) (result *unstructu
 	newArgoApp.SetName(app.GetName())
 	newArgoApp.SetNamespace(app.GetNamespace())
 
+	// copy all potential finalizers
+	finalizers := app.GetFinalizers()
+	targetFinalizers := make([]string, 0)
+	for i := range finalizers {
+		finalizer := finalizers[i]
+		if strings.HasSuffix(finalizer, "argocd.argoproj.io") {
+			targetFinalizers = append(targetFinalizers, finalizer)
+		}
+	}
+	newArgoApp.SetFinalizers(targetFinalizers)
+
 	if err := SetNestedField(newArgoApp.Object, argoApp.Spec, "spec"); err != nil {
 		return nil, err
 	}
@@ -209,6 +243,37 @@ func (r *ApplicationReconciler) addArgoAppNameIntoLabels(namespace, name, argoAp
 	return
 }
 
+// TODO it might be better to do it as a mutating hook
+func (r *ApplicationReconciler) setArgoProject(app *v1alpha1.Application) (err error) {
+	if app.Spec.ArgoApp == nil {
+		return
+	}
+
+	latestApp := &v1alpha1.Application{}
+	ctx := context.Background()
+	if err = r.Get(ctx, types.NamespacedName{
+		Namespace: app.Namespace,
+		Name:      app.Name,
+	}, latestApp); err != nil {
+		return client.IgnoreNotFound(err)
+	}
+	err = r.Update(ctx, latestApp)
+
+	// there is a appProject in the same namespace
+	needUpdate := false
+	if latestApp.Spec.ArgoApp.Spec.Project != latestApp.Namespace {
+		latestApp.Spec.ArgoApp.Spec.Project = latestApp.Namespace
+		needUpdate = true
+	}
+
+	if k8sutil.AddFinalizer(&latestApp.ObjectMeta, v1alpha1.ApplicationFinalizerName) || needUpdate {
+		if err = r.Update(context.TODO(), latestApp); err != nil {
+			return
+		}
+	}
+	return
+}
+
 // GetName returns the name of this reconciler
 func (r *ApplicationReconciler) GetName() string {
 	return "ApplicationController"
@@ -219,13 +284,34 @@ func (r *ApplicationReconciler) GetGroupName() string {
 	return controllerGroupName
 }
 
+type finalizersChangedPredicate struct {
+	predicate.Funcs
+}
+
+// Update implements default UpdateEvent filter for validating finalizers change
+func (finalizersChangedPredicate) Update(e event.UpdateEvent) bool {
+	if e.MetaOld == nil {
+		return false
+	}
+	if e.ObjectOld == nil {
+		return false
+	}
+	if e.ObjectNew == nil {
+		return false
+	}
+	if e.MetaNew == nil {
+		return false
+	}
+	return !reflect.DeepEqual(e.MetaNew.GetFinalizers(), e.MetaOld.GetFinalizers())
+}
+
 // SetupWithManager setups the reconciler with a manager
 // setup the logger, recorder
 func (r *ApplicationReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	r.log = ctrl.Log.WithName(r.GetName())
 	r.recorder = mgr.GetEventRecorderFor(r.GetName())
 	return ctrl.NewControllerManagedBy(mgr).
-		WithEventFilter(predicate.GenerationChangedPredicate{}).
+		WithEventFilter(predicate.Or(predicate.GenerationChangedPredicate{}, finalizersChangedPredicate{})).
 		For(&v1alpha1.Application{}).
 		Complete(r)
 }
diff --git a/controllers/argocd/git-repository-controller.go b/controllers/argocd/git-repository-controller.go
index 3dd2dad..4782892 100644
--- a/controllers/argocd/git-repository-controller.go
+++ b/controllers/argocd/git-repository-controller.go
@@ -30,7 +30,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/predicate"
 )
 
-//+kubebuilder:rbac:groups=devops.kubesphere.io,resources=gitrepositories,verbs=get;list;watch;update
+//+kubebuilder:rbac:groups=devops.kubesphere.io,resources=gitrepositories,verbs=get;list;watch;update;delete
 //+kubebuilder:rbac:groups="",resources=secrets,verbs=create;update;patch;delete
 //+kubebuilder:rbac:groups="",resources=events,verbs=create;patch
 
@@ -86,6 +86,11 @@ func (c *GitRepositoryController) handleFinalizer(repo *v1alpha3.GitRepository)
 	} else {
 		err = c.Delete(ctx, secret)
 	}
+
+	if err == nil {
+		k8sutil.RemoveFinalizer(&repo.ObjectMeta, v1alpha3.GitRepoFinalizerName)
+		err = c.Update(context.TODO(), repo)
+	}
 	return
 }
 
@@ -106,6 +111,7 @@ func (c *GitRepositoryController) handleArgoGitRepo(repo *v1alpha3.GitRepository
 		secret.SetNamespace(ns)
 		secret.SetName(name)
 		c.setArgoGitRepoFields(repo, secret)
+		c.log.Info(fmt.Sprintf("create secret for ArgoCD: %s/%s", ns, name))
 		err = c.Create(ctx, secret)
 	} else {
 		c.setArgoGitRepoFields(repo, secret)
@@ -140,7 +146,7 @@ func (c *GitRepositoryController) setArgoGitRepoAuth(secret *v1.Secret, ref *v1.
 		Name:      ref.Name,
 	}, authSecret); err == nil {
 		switch authSecret.Type {
-		case v1.SecretTypeBasicAuth:
+		case v1.SecretTypeBasicAuth, v1alpha3.SecretTypeBasicAuth:
 			secret.Data["username"] = authSecret.Data[v1.BasicAuthUsernameKey]
 			secret.Data["password"] = authSecret.Data[v1.BasicAuthPasswordKey]
 		default:
diff --git a/controllers/argocd/project_controller.go b/controllers/argocd/project_controller.go
index 70a9381..346fad3 100644
--- a/controllers/argocd/project_controller.go
+++ b/controllers/argocd/project_controller.go
@@ -42,6 +42,8 @@ type Reconciler struct {
 	client.Client
 	log      logr.Logger
 	recorder record.EventRecorder
+
+	ArgoNamespace string
 }
 
 // Reconcile makes sure the ArgoAppProject can be maintained which comes from the DevOpsProject
@@ -55,10 +57,25 @@ func (r *Reconciler) Reconcile(req ctrl.Request) (result ctrl.Result, err error)
 		return
 	}
 
-	if argo := project.Spec.Argo; argo != nil {
-		// we only handle the project that have the Argo settings
-		err = r.reconcileArgoProject(project)
+	if project.Spec.Argo == nil {
+		// give it a default setting which is same with the default setting in Argo CD
+		project.Spec.Argo = &v1alpha3.Argo{
+			SourceRepos: []string{"*"},
+			Destinations: []v1alpha3.ApplicationDestination{{
+				Namespace: "*",
+				Server:    "*",
+			}},
+			ClusterResourceWhitelist: []metav1.GroupKind{{
+				Group: "*",
+				Kind:  "*",
+			}},
+		}
 	}
+	if err = r.Update(ctx, project); err != nil {
+		return
+	}
+
+	err = r.reconcileArgoProject(project)
 	return
 }
 
@@ -70,21 +87,21 @@ func (r *Reconciler) reconcileArgoProject(project *v1alpha3.DevOpsProject) (err
 	argoAppProject.SetAPIVersion("argoproj.io/v1alpha1")
 
 	if err = r.Client.Get(ctx, types.NamespacedName{
-		Namespace: project.Name,
+		Namespace: r.ArgoNamespace,
 		Name:      project.Name,
 	}, argoAppProject); err != nil {
 		if !apierrors.IsNotFound(err) {
 			return
 		}
 
-		if argoAppProject, err = createUnstructuredObject(project); err != nil {
+		if argoAppProject, err = createUnstructuredObject(project, r.ArgoNamespace); err != nil {
 			return
 		}
 
 		err = r.Client.Create(ctx, argoAppProject)
 	} else {
 		var newProject *unstructured.Unstructured
-		if newProject, err = createUnstructuredObject(project); err == nil {
+		if newProject, err = createUnstructuredObject(project, r.ArgoNamespace); err == nil {
 			argoAppProject.Object["spec"] = newProject.Object["spec"]
 			k8sutil.AddOwnerReference(argoAppProject, project.TypeMeta, project.ObjectMeta)
 			err = r.Client.Update(ctx, argoAppProject)
@@ -93,7 +110,7 @@ func (r *Reconciler) reconcileArgoProject(project *v1alpha3.DevOpsProject) (err
 	return
 }
 
-func createUnstructuredObject(project *v1alpha3.DevOpsProject) (result *unstructured.Unstructured, err error) {
+func createUnstructuredObject(project *v1alpha3.DevOpsProject, argocdNamespace string) (result *unstructured.Unstructured, err error) {
 	project = project.DeepCopy()
 	if project.Spec.Argo == nil {
 		err = fmt.Errorf("no argo found from the spec")
@@ -127,7 +144,7 @@ func createUnstructuredObject(project *v1alpha3.DevOpsProject) (result *unstruct
 	if err = tpl.Execute(buffer, argo); err == nil {
 		if result, err = GetObjectFromYaml(buffer.String()); err == nil {
 			result.SetName(project.GetName())
-			result.SetNamespace(project.GetName())
+			result.SetNamespace(argocdNamespace)
 			k8sutil.AddOwnerReference(result, project.TypeMeta, project.ObjectMeta)
 		}
 	}
diff --git a/controllers/jenkins/pipelinerun/pipelinerun_synchronizer.go b/controllers/jenkins/pipelinerun/pipelinerun_synchronizer.go
index cd97b71..52f40ec 100644
--- a/controllers/jenkins/pipelinerun/pipelinerun_synchronizer.go
+++ b/controllers/jenkins/pipelinerun/pipelinerun_synchronizer.go
@@ -71,6 +71,12 @@ func (r *SyncReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 		return ctrl.Result{}, err
 	}
 
+	// skip if there are any PipelineRuns don't have v1alpha3.JenkinsPipelineRunIDAnnoKey
+	// wait it until all PipelineRuns have the run id from Jenkins
+	if hasPendingPipelineRuns(prList.Items) {
+		return ctrl.Result{}, nil
+	}
+
 	boClient := job.BlueOceanClient{
 		JenkinsCore:  r.JenkinsCore,
 		Organization: "jenkins",
@@ -105,6 +111,16 @@ func (r *SyncReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 	return ctrl.Result{}, nil
 }
 
+func hasPendingPipelineRuns(items []v1alpha3.PipelineRun) bool {
+	for i := range items {
+		item := items[i]
+		if id, ok := item.Annotations[v1alpha3.JenkinsPipelineRunIDAnnoKey]; !ok || id == "" {
+			return true
+		}
+	}
+	return false
+}
+
 func createBarePipelineRunsIfNotPresent(finder pipelineRunFinder, pipeline *v1alpha3.Pipeline, jobRuns []job.PipelineRun) []v1alpha3.PipelineRun {
 	var pipelineRunsToBeCreated []v1alpha3.PipelineRun
 	for i := range jobRuns {
diff --git a/pkg/api/gitops/v1alpha1/constants.go b/pkg/api/gitops/v1alpha1/constants.go
index 4ea92a4..861cae9 100644
--- a/pkg/api/gitops/v1alpha1/constants.go
+++ b/pkg/api/gitops/v1alpha1/constants.go
@@ -28,3 +28,6 @@ const (
 
 // ApplicationFinalizerName is the name of PipelineRun finalizer
 const ApplicationFinalizerName = "application." + GroupName
+
+// ArgoCDResourcesFinalizer is the name of Argo CD resource finalizer
+const ArgoCDResourcesFinalizer = "resources-finalizer.argocd.argoproj.io"
diff --git a/pkg/client/git/client.go b/pkg/client/git/client.go
index e4e5387..202612d 100644
--- a/pkg/client/git/client.go
+++ b/pkg/client/git/client.go
@@ -27,6 +27,7 @@ import (
 	v1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
+	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
 	"net/http"
 )
 
@@ -84,11 +85,11 @@ func (c *ClientFactory) getTokenFromSecret(secretRef *v1.SecretReference) (token
 	}
 
 	switch gitSecret.Type {
-	case v1.SecretTypeBasicAuth, "credential.devops.kubesphere.io/basic-auth":
+	case v1.SecretTypeBasicAuth, v1alpha3.SecretTypeBasicAuth:
 		token = string(gitSecret.Data[v1.BasicAuthPasswordKey])
 	case v1.SecretTypeOpaque:
 		token = string(gitSecret.Data[v1.ServiceAccountTokenKey])
-	case "credential.devops.kubesphere.io/secret-text":
+	case v1alpha3.SecretTypeSecretText:
 		token = string(gitSecret.Data["secret"])
 	}
 	return
diff --git a/pkg/kapis/devops/v1alpha3/register.go b/pkg/kapis/devops/v1alpha3/register.go
index cf2d984..d5fda16 100644
--- a/pkg/kapis/devops/v1alpha3/register.go
+++ b/pkg/kapis/devops/v1alpha3/register.go
@@ -97,8 +97,9 @@ func registerRoutersForCredentials(handler *devopsHandler, ws *restful.WebServic
 	ws.Route(ws.POST("/devops/{devops}/credentials").
 		To(handler.CreateCredential).
 		Param(ws.PathParameter("devops", "devops name")).
+		Reads(v1.Secret{}).
 		Doc("create the credential of the specified devops for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1alpha3.Pipeline{}).
+		Returns(http.StatusOK, api.StatusOK, v1.Secret{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 
 	ws.Route(ws.GET("/devops/{devops}/credentials/{credential}").
@@ -106,15 +107,16 @@ func registerRoutersForCredentials(handler *devopsHandler, ws *restful.WebServic
 		Param(ws.PathParameter("devops", "project name")).
 		Param(ws.PathParameter("credential", "pipeline name")).
 		Doc("get the credential of the specified devops for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1.Secret{}).
+		Returns(http.StatusOK, api.StatusOK, v1.Secret{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 
 	ws.Route(ws.PUT("/devops/{devops}/credentials/{credential}").
 		To(handler.UpdateCredential).
 		Param(ws.PathParameter("devops", "project name")).
 		Param(ws.PathParameter("credential", "credential name")).
+		Reads(v1.Secret{}).
 		Doc("put the credential of the specified devops for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1.Secret{}).
+		Returns(http.StatusOK, api.StatusOK, v1.Secret{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 
 	ws.Route(ws.DELETE("/devops/{devops}/credentials/{credential}").
@@ -122,7 +124,7 @@ func registerRoutersForCredentials(handler *devopsHandler, ws *restful.WebServic
 		Param(ws.PathParameter("devops", "project name")).
 		Param(ws.PathParameter("credential", "credential name")).
 		Doc("delete the credential of the specified devops for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1.Secret{}).
+		Returns(http.StatusOK, api.StatusOK, v1.Secret{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsPipelineTag}))
 }
 
@@ -141,8 +143,9 @@ func registerRoutersForPipelines(handler *devopsHandler, ws *restful.WebService)
 	ws.Route(ws.POST("/devops/{devops}/pipelines").
 		To(handler.CreatePipeline).
 		Param(ws.PathParameter("devops", "devops name")).
+		Reads(v1alpha3.Pipeline{}).
 		Doc("create the pipeline of the specified devops for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1alpha3.Pipeline{}).
+		Returns(http.StatusOK, api.StatusOK, v1alpha3.Pipeline{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 
 	ws.Route(ws.GET("/devops/{devops}/pipelines/{pipeline}").
@@ -151,7 +154,7 @@ func registerRoutersForPipelines(handler *devopsHandler, ws *restful.WebService)
 		Param(ws.PathParameter("devops", "project name")).
 		Param(ws.PathParameter("pipeline", "pipeline name")).
 		Doc("get the pipeline of the specified devops for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1alpha3.Pipeline{}).
+		Returns(http.StatusOK, api.StatusOK, v1alpha3.Pipeline{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 
 	ws.Route(ws.PUT("/devops/{devops}/pipelines/{pipeline}").
@@ -159,7 +162,7 @@ func registerRoutersForPipelines(handler *devopsHandler, ws *restful.WebService)
 		Param(ws.PathParameter("devops", "project name")).
 		Param(ws.PathParameter("pipeline", "pipeline name")).
 		Doc("put the pipeline of the specified devops for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1alpha3.Pipeline{}).
+		Returns(http.StatusOK, api.StatusOK, v1alpha3.Pipeline{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 
 	ws.Route(ws.DELETE("/devops/{devops}/pipelines/{pipeline}").
@@ -167,7 +170,7 @@ func registerRoutersForPipelines(handler *devopsHandler, ws *restful.WebService)
 		Param(ws.PathParameter("devops", "project name")).
 		Param(ws.PathParameter("pipeline", "pipeline name")).
 		Doc("delete the pipeline of the specified devops for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1alpha3.Pipeline{}).
+		Returns(http.StatusOK, api.StatusOK, v1alpha3.Pipeline{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsPipelineTag}))
 }
 
@@ -185,8 +188,9 @@ func registerRoutersForWorkspace(handler *devopsHandler, ws *restful.WebService)
 	ws.Route(ws.POST("/workspaces/{workspace}/devops").
 		To(handler.CreateDevOpsProject).
 		Param(ws.PathParameter("workspace", "workspace name")).
+		Reads(v1alpha3.DevOpsProject{}).
 		Doc("Create the devopsproject of the specified workspace for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1alpha3.DevOpsProject{}).
+		Returns(http.StatusOK, api.StatusOK, v1alpha3.DevOpsProject{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 
 	ws.Route(ws.GET("/workspaces/{workspace}/devops/{devops}").
@@ -202,8 +206,9 @@ func registerRoutersForWorkspace(handler *devopsHandler, ws *restful.WebService)
 		To(handler.UpdateDevOpsProject).
 		Param(ws.PathParameter("workspace", "workspace name")).
 		Param(ws.PathParameter("devops", "project name")).
+		Reads(v1alpha3.DevOpsProject{}).
 		Doc("Put the devopsproject of the specified workspace for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1alpha3.DevOpsProject{}).
+		Returns(http.StatusOK, api.StatusOK, v1alpha3.DevOpsProject{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 
 	ws.Route(ws.DELETE("/workspaces/{workspace}/devops/{devops}").
@@ -211,6 +216,6 @@ func registerRoutersForWorkspace(handler *devopsHandler, ws *restful.WebService)
 		Param(ws.PathParameter("workspace", "workspace name")).
 		Param(ws.PathParameter("devops", "project name")).
 		Doc("Get the devopsproject of the specified workspace for the current user").
-		Returns(http.StatusOK, api.StatusOK, []v1alpha3.DevOpsProject{}).
+		Returns(http.StatusOK, api.StatusOK, v1alpha3.DevOpsProject{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 }
diff --git a/pkg/kapis/devops/v1alpha3/scm/gitrepository_handler.go b/pkg/kapis/devops/v1alpha3/scm/gitrepository_handler.go
index ef6bb24..4f42cff 100644
--- a/pkg/kapis/devops/v1alpha3/scm/gitrepository_handler.go
+++ b/pkg/kapis/devops/v1alpha3/scm/gitrepository_handler.go
@@ -79,20 +79,19 @@ func (h *handler) updateGitRepositories(req *restful.Request, res *restful.Respo
 	ctx := context.Background()
 
 	patchRepo := &v1alpha3.GitRepository{}
-	repo := &v1alpha3.GitRepository{}
-
 	err := req.ReadEntity(patchRepo)
 	if err == nil {
+		repo := &v1alpha3.GitRepository{}
 		err = h.Get(ctx, types.NamespacedName{
 			Namespace: namespace,
 			Name:      repoName,
 		}, repo)
 		if err == nil {
-			repo.Spec = patchRepo.Spec
-			err = h.Update(ctx, repo)
+			patchRepo.ResourceVersion = repo.ResourceVersion
+			err = h.Update(ctx, patchRepo)
 		}
 	}
-	common.Response(req, res, repo, err)
+	common.Response(req, res, patchRepo, err)
 }
 
 func (h *handler) deleteGitRepositories(req *restful.Request, res *restful.Response) {
diff --git a/pkg/kapis/devops/v1alpha3/scm/route.go b/pkg/kapis/devops/v1alpha3/scm/route.go
index 2c2d076..b5b4688 100644
--- a/pkg/kapis/devops/v1alpha3/scm/route.go
+++ b/pkg/kapis/devops/v1alpha3/scm/route.go
@@ -26,6 +26,8 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 )
 
+//+kubebuilder:rbac:groups=devops.kubesphere.io,resources=gitrepositories,verbs=get;list;update;delete;create;watch;patch
+
 var (
 	pathParameterSCM          = restful.PathParameter("scm", "the SCM type")
 	pathParameterOrganization = restful.PathParameter("organization",
diff --git a/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler.go b/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler.go
index d25fe09..ae09c3a 100644
--- a/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler.go
+++ b/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler.go
@@ -18,11 +18,15 @@ package webhook
 
 import (
 	"context"
+	"errors"
 	"fmt"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/client-go/util/retry"
+	"k8s.io/klog"
 	"kubesphere.io/devops/pkg/event/workflowrun"
 	"strings"
+	"time"
 
-	"k8s.io/klog"
 	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
 	"kubesphere.io/devops/pkg/kapis/devops/v1alpha3/pipelinerun"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -94,29 +98,51 @@ func (handler *Handler) handleWorkflowRunInitialize(workflowRunData *workflowrun
 		return nil
 	}
 
-	// TODO Execute process below asynchronously
+	go func() {
+		pipelineRunList := &v1alpha3.PipelineRunList{}
+		if err := handler.List(context.Background(), pipelineRunList,
+			client.InNamespace(identifier.namespaceName),
+			client.MatchingFields{v1alpha3.PipelineRunIdentifierIndexerName: identifier.String()}); err != nil {
+			return
+		}
 
-	pipelineRunList := &v1alpha3.PipelineRunList{}
-	if err := handler.List(context.Background(), pipelineRunList,
-		client.InNamespace(identifier.namespaceName),
-		client.MatchingFields{v1alpha3.PipelineRunIdentifierIndexerName: identifier.String()}); err != nil {
-		return err
-	}
+		// TODO Currently, no good way to check if this event originally comes from Jenkins.
+		// Let's wait for the PipelineRun controller to set the Jenkins job run ID
+		pipelineRunListErr := handler.retryCheckPipelineRunList(identifier)
+		if pipelineRunListErr != nil {
+			parameters, err := workflowRunData.Actions.GetParameters()
+			if err != nil {
+				return
+			}
+
+			pipelineRun, err := handler.createPipelineRun(identifier, convertParameters(parameters))
+			if err != nil {
+				return
+			}
+			klog.Infof("Created a PipelineRun: %s/%s", pipelineRun.Namespace, pipelineRun.Name)
+		}
+	}()
+	return nil
+}
 
-	if len(pipelineRunList.Items) == 0 {
-		parameters, err := workflowRunData.Actions.GetParameters()
-		if err != nil {
+func (handler *Handler) retryCheckPipelineRunList(id *pipelineRunIdentifier) error {
+	return retry.OnError(retry.DefaultRetry, func(err error) bool {
+		return true
+	}, func() error {
+		pipelineRunList := &v1alpha3.PipelineRunList{}
+		if err := handler.List(context.Background(), pipelineRunList,
+			client.InNamespace(id.namespaceName),
+			client.MatchingFields{v1alpha3.PipelineRunIdentifierIndexerName: id.String()}); err != nil {
 			return err
 		}
 
-		pipelineRun, err := handler.createPipelineRun(identifier, convertParameters(parameters))
-		if err != nil {
-			return err
+		if len(pipelineRunList.Items) > 0 {
+			return nil
 		}
-		klog.Infof("Created a PipelineRun: %s/%s", pipelineRun.Namespace, pipelineRun.Name)
-		return nil
-	}
-	return nil
+
+		time.Sleep(time.Second)
+		return errors.New(string(metav1.StatusReasonNotFound))
+	})
 }
 
 func (handler *Handler) createPipelineRun(identifier *pipelineRunIdentifier, parameters []v1alpha3.Parameter) (*v1alpha3.PipelineRun, error) {
diff --git a/pkg/kapis/gitops/v1alpha1/argocd/handler.go b/pkg/kapis/gitops/v1alpha1/argocd/handler.go
index 6d5c1b9..1369317 100644
--- a/pkg/kapis/gitops/v1alpha1/argocd/handler.go
+++ b/pkg/kapis/gitops/v1alpha1/argocd/handler.go
@@ -28,6 +28,7 @@ import (
 	"kubesphere.io/devops/pkg/config"
 	"kubesphere.io/devops/pkg/kapis/common"
 	"kubesphere.io/devops/pkg/models/resources/v1alpha3"
+	"kubesphere.io/devops/pkg/utils/k8sutil"
 	"net/http"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 )
@@ -171,14 +172,31 @@ func (h *handler) updateOperation(namespace, name string, operation *v1alpha1.Op
 func (h *handler) delApplication(req *restful.Request, res *restful.Response) {
 	namespace := common.GetPathParameter(req, common.NamespacePathParameter)
 	name := common.GetPathParameter(req, pathParameterApplication)
+	cascade := common.GetQueryParameter(req, cascadeQueryParam)
 
+	ctx := context.Background()
 	application := &v1alpha1.Application{}
-	err := h.Get(context.Background(), types.NamespacedName{
+	objectKey := types.NamespacedName{
 		Namespace: namespace,
 		Name:      name,
-	}, application)
+	}
+	err := h.Get(ctx, objectKey, application)
 	if err == nil {
-		err = h.Delete(context.Background(), application)
+		// add the Argo CD resources finalizer if cascade is true
+		if cascade == "true" {
+			if k8sutil.AddFinalizer(&application.ObjectMeta, v1alpha1.ArgoCDResourcesFinalizer) {
+				if err = h.Update(ctx, application); err != nil {
+					common.Response(req, res, application, err)
+					return
+				}
+
+				if err = h.Get(ctx, objectKey, application); err != nil {
+					common.Response(req, res, application, err)
+					return
+				}
+			}
+		}
+		err = h.Delete(ctx, application)
 	}
 	common.Response(req, res, application, err)
 }
@@ -228,7 +246,11 @@ func (h *handler) getClusters(req *restful.Request, res *restful.Response) {
 		"argocd.argoproj.io/secret-type": "cluster",
 	})
 
-	argoClusters := make([]v1alpha1.ApplicationDestination, 0)
+	// always have on cluster at least
+	argoClusters := []v1alpha1.ApplicationDestination{{
+		Server: "https://kubernetes.default.svc",
+		Name:   "in-cluster",
+	}}
 	if err == nil && secrets != nil && len(secrets.Items) > 0 {
 		for i := range secrets.Items {
 			secret := secrets.Items[i]
diff --git a/pkg/kapis/gitops/v1alpha1/argocd/route.go b/pkg/kapis/gitops/v1alpha1/argocd/route.go
index 05d0e69..d31f697 100644
--- a/pkg/kapis/gitops/v1alpha1/argocd/route.go
+++ b/pkg/kapis/gitops/v1alpha1/argocd/route.go
@@ -29,6 +29,9 @@ var (
 	pathParameterApplication = restful.PathParameter("application", "The application name")
 	syncStatusQueryParam     = restful.QueryParameter("syncStatus", `Filter by sync status. Available values: "Unknown", "Synced" and "OutOfSync"`)
 	healthStatusQueryParam   = restful.QueryParameter("healthStatus", `Filter by health status. Available values: "Unknown", "Progressing", "Healthy", "Suspended", "Degraded" and "Missing"`)
+	cascadeQueryParam        = restful.QueryParameter("cascade",
+		"Delete both the app and its resources, rather than only the application if cascade is true").
+		DefaultValue("false").DataType("bool")
 )
 
 // ApplicationPageResult is the model of page result of Applications.
@@ -105,6 +108,7 @@ func RegisterRoutes(service *restful.WebService, options *common.Options, argoOp
 		To(handler.delApplication).
 		Param(common.NamespacePathParameter).
 		Param(pathParameterApplication).
+		Param(cascadeQueryParam).
 		Doc("Delete a particular application").
 		Returns(http.StatusOK, api.StatusOK, v1alpha1.Application{}))
 
diff --git a/pkg/models/devops/devops.go b/pkg/models/devops/devops.go
index d67f1a5..ea3f383 100644
--- a/pkg/models/devops/devops.go
+++ b/pkg/models/devops/devops.go
@@ -318,7 +318,7 @@ func (d devopsOperator) ListPipelineObj(projectName string, query *query.Query)
 	return *resourcesV1alpha3.DefaultList(result, query, resourcesV1alpha3.DefaultCompare(), resourcesV1alpha3.DefaultFilter()), nil
 }
 
-//credentialobj in crd
+// CreateCredentialObj creates a secret
 func (d devopsOperator) CreateCredentialObj(projectName string, secret *v1.Secret) (*v1.Secret, error) {
 	projectObj, err := d.ksclient.DevopsV1alpha3().DevOpsProjects().Get(d.context, projectName, metav1.GetOptions{})
 	if err != nil {
@@ -342,9 +342,7 @@ func (d devopsOperator) GetCredentialObj(projectName string, secretName string)
 	if secret, err := d.k8sclient.CoreV1().Secrets(projectObj.Status.AdminNamespace).Get(d.context, secretName, metav1.GetOptions{}); err != nil {
 		return nil, err
 	} else {
-		// TODO Mask the secret if there is no place to use plain secret.
-		// return secretutil.MaskCredential(secret), nil
-		return secret, nil
+		return secretutil.MaskCredential(secret), nil
 	}
 }
 


Test patch:
diff --git a/controllers/argocd/application_controller_test.go b/controllers/argocd/application_controller_test.go
index 77c3363..273651f 100644
--- a/controllers/argocd/application_controller_test.go
+++ b/controllers/argocd/application_controller_test.go
@@ -27,6 +27,8 @@ import (
 	"kubesphere.io/devops/pkg/api/gitops/v1alpha1"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/controller-runtime/pkg/event"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
 	"testing"
 )
 
@@ -156,7 +158,7 @@ func TestApplicationReconciler_reconcileArgoApplication(t *testing.T) {
 			}, app)
 			assert.Nil(t, err)
 			project, _, _ := unstructured.NestedString(app.Object, "spec", "project")
-			assert.Equal(t, "project", project)
+			assert.Equal(t, "fake", project)
 			prune, _, _ := unstructured.NestedBool(app.Object, "spec", "syncPolicy", "automated", "prune")
 			assert.True(t, prune)
 		},
@@ -179,7 +181,7 @@ func TestApplicationReconciler_reconcileArgoApplication(t *testing.T) {
 			}, app)
 			assert.Nil(t, err)
 			project, _, _ := unstructured.NestedString(app.Object, "spec", "project")
-			assert.Equal(t, "project", project)
+			assert.Equal(t, "fake", project)
 		},
 	}}
 	for _, tt := range tests {
@@ -194,3 +196,96 @@ func TestApplicationReconciler_reconcileArgoApplication(t *testing.T) {
 		})
 	}
 }
+
+func Test_finalizersChangedPredicate_Update(t *testing.T) {
+	type fields struct {
+		Funcs predicate.Funcs
+	}
+	type args struct {
+		e event.UpdateEvent
+	}
+	tests := []struct {
+		name   string
+		fields fields
+		args   args
+		want   bool
+	}{{
+		name: "different finalizers",
+		args: args{
+			e: event.UpdateEvent{
+				MetaOld: &metav1.ObjectMeta{
+					Finalizers: []string{"b"},
+				},
+				ObjectOld: &v1alpha1.Application{},
+				MetaNew: &metav1.ObjectMeta{
+					Finalizers: []string{"a"},
+				},
+				ObjectNew: &v1alpha1.Application{},
+			},
+		},
+		want: true,
+	}, {
+		name: "different order with the finalizers",
+		args: args{
+			e: event.UpdateEvent{
+				MetaOld: &metav1.ObjectMeta{
+					Finalizers: []string{"b", "a"},
+				},
+				ObjectOld: &v1alpha1.Application{},
+				MetaNew: &metav1.ObjectMeta{
+					Finalizers: []string{"a", "b"},
+				},
+				ObjectNew: &v1alpha1.Application{},
+			},
+		},
+		want: true,
+	}, {
+		name: "ObjectOld is nil",
+		args: args{
+			e: event.UpdateEvent{
+				MetaOld:   &metav1.ObjectMeta{},
+				MetaNew:   &metav1.ObjectMeta{},
+				ObjectNew: &v1alpha1.Application{},
+			},
+		},
+		want: false,
+	}, {
+		name: "MetaNew is nil",
+		args: args{
+			e: event.UpdateEvent{
+				ObjectOld: &v1alpha1.Application{},
+				MetaOld:   &metav1.ObjectMeta{},
+				ObjectNew: &v1alpha1.Application{},
+			},
+		},
+		want: false,
+	}, {
+		name: "ObjectNew is nil",
+		args: args{
+			e: event.UpdateEvent{
+				MetaOld:   &metav1.ObjectMeta{},
+				ObjectOld: &v1alpha1.Application{},
+				MetaNew:   &metav1.ObjectMeta{},
+			},
+		},
+		want: false,
+	}, {
+		name: "metaOld is nil",
+		args: args{
+			e: event.UpdateEvent{
+				ObjectOld: &v1alpha1.Application{},
+				MetaNew:   &metav1.ObjectMeta{},
+				ObjectNew: &v1alpha1.Application{},
+			},
+		},
+		want: false,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			fi := finalizersChangedPredicate{
+				Funcs: tt.fields.Funcs,
+			}
+			assert.Equalf(t, tt.want, fi.Update(tt.args.e), "Update(%v)", tt.args.e)
+		})
+	}
+}
diff --git a/controllers/argocd/project_controller_test.go b/controllers/argocd/project_controller_test.go
index e774f02..aa0a0d8 100644
--- a/controllers/argocd/project_controller_test.go
+++ b/controllers/argocd/project_controller_test.go
@@ -32,9 +32,10 @@ func TestCreateUnstructuredObject(t *testing.T) {
 	project := &v1alpha3.DevOpsProject{}
 	project.SetName("name")
 	project.SetNamespace("namespace")
+	argocdNs := "argocd"
 
 	// test against the nil situation
-	_, err := createUnstructuredObject(project)
+	_, err := createUnstructuredObject(project, argocdNs)
 	assert.NotNil(t, err, "should not do anything if there is no spec.argo")
 
 	var obj *unstructured.Unstructured
@@ -42,7 +43,7 @@ func TestCreateUnstructuredObject(t *testing.T) {
 	// test against the default value situation
 	p1 := project.DeepCopy()
 	p1.Spec.Argo = &v1alpha3.Argo{}
-	obj, err = createUnstructuredObject(p1)
+	obj, err = createUnstructuredObject(p1, argocdNs)
 	assert.Nil(t, err)
 	result, _, _ := unstructured.NestedStringSlice(obj.Object, "spec", "sourceRepos")
 	assert.Equal(t, []string{"*"}, result)
@@ -71,7 +72,7 @@ func TestCreateUnstructuredObject(t *testing.T) {
 			Kind:  "kind2",
 		}},
 	}
-	obj, err = createUnstructuredObject(p2)
+	obj, err = createUnstructuredObject(p2, argocdNs)
 	assert.Nil(t, err)
 	p2Repos, _, _ := unstructured.NestedStringSlice(obj.Object, "spec", "sourceRepos")
 	assert.Equal(t, []string{"repo1", "repo2"}, p2Repos)
@@ -104,7 +105,7 @@ func TestReconcileArgoProject(t *testing.T) {
 	appProject.SetAPIVersion("argoproj.io/v1alpha1")
 	appProject.SetKind("AppProject")
 	appProject.SetName("fake")
-	appProject.SetNamespace("fake")
+	appProject.SetNamespace("argocd")
 
 	tests := []struct {
 		name    string
@@ -135,7 +136,7 @@ func TestReconcileArgoProject(t *testing.T) {
 			appProject := appProject.DeepCopy()
 
 			err = c.Get(context.Background(), types.NamespacedName{
-				Namespace: "fake",
+				Namespace: "argocd",
 				Name:      "fake",
 			}, appProject)
 			assert.Nil(t, err)
@@ -157,7 +158,7 @@ func TestReconcileArgoProject(t *testing.T) {
 			appProject := appProject.DeepCopy()
 
 			err = c.Get(context.Background(), types.NamespacedName{
-				Namespace: "fake",
+				Namespace: "argocd",
 				Name:      "fake",
 			}, appProject)
 			assert.Nil(t, err)
@@ -168,7 +169,8 @@ func TestReconcileArgoProject(t *testing.T) {
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			r := &Reconciler{
-				Client: tt.c,
+				Client:        tt.c,
+				ArgoNamespace: "argocd",
 			}
 			err := r.reconcileArgoProject(tt.project())
 			tt.verify(t, err, tt.c)
diff --git a/controllers/jenkins/pipelinerun/pipelinerun_synchronizer_test.go b/controllers/jenkins/pipelinerun/pipelinerun_synchronizer_test.go
index eac99da..948add1 100644
--- a/controllers/jenkins/pipelinerun/pipelinerun_synchronizer_test.go
+++ b/controllers/jenkins/pipelinerun/pipelinerun_synchronizer_test.go
@@ -433,3 +433,43 @@ func Test_createBarePipelineRunsIfNotPresent(t *testing.T) {
 		})
 	}
 }
+
+func Test_hasPendingPipelineRuns(t *testing.T) {
+	type args struct {
+		items []v1alpha3.PipelineRun
+	}
+	tests := []struct {
+		name string
+		args args
+		want bool
+	}{{
+		name: "have pending PipelineRuns",
+		args: args{
+			items: []v1alpha3.PipelineRun{{
+				ObjectMeta: v1.ObjectMeta{
+					Annotations: map[string]string{
+						"a": "1",
+					},
+				},
+			}},
+		},
+		want: true,
+	}, {
+		name: "not have pending PipelineRuns",
+		args: args{
+			items: []v1alpha3.PipelineRun{{
+				ObjectMeta: v1.ObjectMeta{
+					Annotations: map[string]string{
+						v1alpha3.JenkinsPipelineRunIDAnnoKey: "1",
+					},
+				},
+			}},
+		},
+		want: false,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, tt.want, hasPendingPipelineRuns(tt.args.items), "hasPendingPipelineRuns(%v)", tt.args.items)
+		})
+	}
+}
diff --git a/pkg/kapis/devops/v1alpha3/webhook/register_test.go b/pkg/kapis/devops/v1alpha3/webhook/register_test.go
index cd96d0a..1a62e06 100644
--- a/pkg/kapis/devops/v1alpha3/webhook/register_test.go
+++ b/pkg/kapis/devops/v1alpha3/webhook/register_test.go
@@ -18,13 +18,17 @@ package webhook
 
 import (
 	"context"
+	"errors"
+	"fmt"
 	"io"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	"k8s.io/client-go/kubernetes/scheme"
+	"k8s.io/client-go/util/retry"
 	"net/http"
 	"net/http/httptest"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/emicklei/go-restful"
 	"github.com/stretchr/testify/assert"
@@ -112,9 +116,20 @@ func TestJenkinsWebhook(t *testing.T) {
 			}`,
 		},
 		assertion: func(t *testing.T, c client.Client) {
-			pipelineruns := &v1alpha3.PipelineRunList{}
-			assert.Nil(t, c.List(context.Background(), pipelineruns))
-			assert.Equal(t, 1, len(pipelineruns.Items))
+			_ = retry.OnError(retry.DefaultRetry, func(err error) bool {
+				return true
+			}, func() error {
+				pipelineRuns := &v1alpha3.PipelineRunList{}
+				_ = c.List(context.Background(), pipelineRuns)
+				if len(pipelineRuns.Items) > 0 {
+					return nil
+				}
+				time.Sleep(time.Second * 2)
+				return errors.New("not found")
+			})
+			pipelineRuns := &v1alpha3.PipelineRunList{}
+			_ = c.List(context.Background(), pipelineRuns)
+			assert.Equal(t, 1, len(pipelineRuns.Items))
 		},
 	}, {
 		name: "Should not create any PipelineRuns when type doesn't start with run",
@@ -164,3 +179,13 @@ func TestJenkinsWebhook(t *testing.T) {
 		})
 	}
 }
+
+func Test(t *testing.T) {
+	err := retry.OnError(retry.DefaultRetry, func(err error) bool {
+		return true
+	}, func() error {
+		time.Sleep(time.Second)
+		return errors.New("")
+	})
+	fmt.Println(err)
+}
diff --git a/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler_test.go b/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler_test.go
index 6c43fd0..0cac79f 100644
--- a/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler_test.go
+++ b/pkg/kapis/devops/v1alpha3/webhook/workflowrun_handler_test.go
@@ -18,9 +18,12 @@ package webhook
 
 import (
 	"context"
+	"errors"
+	"k8s.io/client-go/util/retry"
 	"kubesphere.io/devops/pkg/event/workflowrun"
 	"reflect"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -235,6 +238,17 @@ func TestHandler_handleWorkflowRunInitialize(t *testing.T) {
 		},
 		wantErr: false,
 		assertion: func(t *testing.T, c client.Client) {
+			_ = retry.OnError(retry.DefaultRetry, func(err error) bool {
+				return true
+			}, func() error {
+				pipelineRuns := &v1alpha3.PipelineRunList{}
+				_ = c.List(context.Background(), pipelineRuns)
+				if len(pipelineRuns.Items) > 0 {
+					return nil
+				}
+				time.Sleep(time.Second * 2)
+				return errors.New("not found")
+			})
 			pipelineRuns := &v1alpha3.PipelineRunList{}
 			_ = c.List(context.Background(), pipelineRuns)
 			assert.Equal(t, 1, len(pipelineRuns.Items))
@@ -244,7 +258,7 @@ func TestHandler_handleWorkflowRunInitialize(t *testing.T) {
 		args: args{
 			workflowRunData: createWorkflowRun("fake-namepsace", "fake-pipeline", "1", false),
 		},
-		wantErr: true,
+		wantErr: false,
 	}, {
 		name: "Should create nothing if WorkflowRunData is invalid",
 		args: args{
diff --git a/pkg/kapis/gitops/v1alpha1/argocd/route_test.go b/pkg/kapis/gitops/v1alpha1/argocd/route_test.go
index b6b2d28..0f5c89c 100644
--- a/pkg/kapis/gitops/v1alpha1/argocd/route_test.go
+++ b/pkg/kapis/gitops/v1alpha1/argocd/route_test.go
@@ -192,6 +192,25 @@ func TestAPIs(t *testing.T) {
 			assert.Equal(t, "app", name)
 			assert.Nil(t, err)
 		},
+	}, {
+		name: "delete an application by cascade",
+		request: request{
+			method: http.MethodDelete,
+			uri:    "/namespaces/ns/applications/app?cascade=true",
+		},
+		k8sclient:    fake.NewFakeClientWithScheme(schema, app.DeepCopy()),
+		responseCode: http.StatusOK,
+		verify: func(t *testing.T, body []byte) {
+			list := &unstructured.Unstructured{}
+			err := yaml.Unmarshal(body, list)
+			assert.Nil(t, err)
+
+			name, _, err := unstructured.NestedString(list.Object, "metadata", "name")
+			assert.Equal(t, "app", name)
+			finalizers, _, err := unstructured.NestedSlice(list.Object, "metadata", "finalizers")
+			assert.Equal(t, []interface{}{"resources-finalizer.argocd.argoproj.io"}, finalizers)
+			assert.Nil(t, err)
+		},
 	}, {
 		name: "create an application",
 		request: request{
@@ -279,7 +298,12 @@ func TestAPIs(t *testing.T) {
 		k8sclient:    fake.NewFakeClientWithScheme(schema, nonArgoClusterSecret.DeepCopy()),
 		responseCode: http.StatusOK,
 		verify: func(t *testing.T, body []byte) {
-			assert.Equal(t, "[]", string(body))
+			assert.Equal(t, `[
+ {
+  "server": "https://kubernetes.default.svc",
+  "name": "in-cluster"
+ }
+]`, string(body))
 		},
 	}, {
 		name: "get clusters, have invalid data",
@@ -290,7 +314,12 @@ func TestAPIs(t *testing.T) {
 		k8sclient:    fake.NewFakeClientWithScheme(schema, invalidArgoClusterSecret.DeepCopy()),
 		responseCode: http.StatusOK,
 		verify: func(t *testing.T, body []byte) {
-			assert.Equal(t, "[]", string(body))
+			assert.Equal(t, `[
+ {
+  "server": "https://kubernetes.default.svc",
+  "name": "in-cluster"
+ }
+]`, string(body))
 		},
 	}, {
 		name: "get clusters, have the expected data",
@@ -302,6 +331,10 @@ func TestAPIs(t *testing.T) {
 		responseCode: http.StatusOK,
 		verify: func(t *testing.T, body []byte) {
 			assert.Equal(t, `[
+ {
+  "server": "https://kubernetes.default.svc",
+  "name": "in-cluster"
+ },
  {
   "server": "server",
   "name": "name"