Repository: kubesphere/ks-devops
Pull request #748 (2022-07-22 09:14:06)

Support to get Jenkins agent labels from ConfigMap
<!-- Thanks for sending a pull request! Here are some tips for you:

1. If you want **faster** PR reviews, read how: https://github.com/kubesphere/community/blob/master/developer-guide/development/the-pr-author-guide-to-getting-through-code-review.md
2. In case you want to know how your PR got reviewed, read: https://github.com/kubesphere/community/blob/master/developer-guide/development/code-review-guide.md
3. Here are some coding conventions followed by the KubeSphere community: https://github.com/kubesphere/community/blob/master/developer-guide/development/coding-conventions.md
4. Additional open-source best practice: https://github.com/LinuxSuRen/open-source-best-practice
-->

### What type of PR is this?
<!-- 
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind design
/kind chore

Optionally add one or more of the following kinds if applicable:
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->
/kind feature
/kind api-change

### What this PR does / why we need it:
As I mentioned in the title.

### Which issue(s) this PR fixes:
<!--
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
Please leave it or change # to be None if there is no corresponding issue that exists
-->
related #738

### Special notes for reviewers:
<!--
You can use the following command to let the DevOps SIG members help you to review your PR.
/cc @kubesphere/sig-devops 
And please avoid cc any individual.
-->
Please check the following list before waiting reviewers:

- [ ] Already committed the CRD files to [the Helm Chart](https://github.com/kubesphere-sigs/ks-devops-helm-chart/) if you created some new CRDs
- [ ] Already [added the permission](https://github.com/kubesphere/ks-installer/blob/9e063b085a0e43fdb3d0d9e3e7f4149146f14b9c/roles/ks-core/prepare/files/ks-init/role-templates.yaml) for the new API
- [ ] Already added the RBAC markers for the new controllers

### Does this PR introduce a user-facing change??
<!--
If no, just write "None" in the release-note block below.
If yes, a release note is required:
Enter your extended-release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

Please keep the note be same as your PR title if you believe it should be in the release notes.
-->
```release-note
Support to getting Jenkins agent labels when it's offline
```


ks-ci-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please ask for approval from **linuxsuren** after the PR has been reviewed.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubesphere%2Fks-devops).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubesphere/ks-devops/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["linuxsuren"]} -->
codecov[bot]: # [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/748?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) Report
> Merging [#748](https://codecov.io/gh/kubesphere/ks-devops/pull/748?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (1cc5ad0) into [master](https://codecov.io/gh/kubesphere/ks-devops/commit/40ec4b7cfef083b9b023f90277b23754877ea250?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (40ec4b7) will **increase** coverage by `0.34%`.
> The diff coverage is `89.56%`.

> :exclamation: Current head 1cc5ad0 differs from pull request most recent head 752af50. Consider uploading reports for the commit 752af50 to get more accurate results

```diff
@@            Coverage Diff             @@
##           master     #748      +/-   ##
==========================================
+ Coverage   78.25%   78.60%   +0.34%     
==========================================
  Files         157      159       +2     
  Lines       23811    23915     +104     
==========================================
+ Hits        18633    18798     +165     
+ Misses       4925     4859      -66     
- Partials      253      258       +5     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `78.60% <89.56%> (+0.34%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/kubesphere/ks-devops/pull/748?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) | Coverage Δ | |
|---|---|---|
| [pkg/models/devops/common.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL21vZGVscy9kZXZvcHMvY29tbW9uLmdv) | `100.00% <ø> (+91.66%)` | :arrow_up: |
| [pkg/kapis/devops/v1alpha3/handler.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL2thcGlzL2Rldm9wcy92MWFscGhhMy9oYW5kbGVyLmdv) | `79.00% <77.77%> (-0.06%)` | :arrow_down: |
| [pkg/models/devops/devops.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL21vZGVscy9kZXZvcHMvZGV2b3BzLmdv) | `25.36% <83.33%> (+0.94%)` | :arrow_up: |
| [controllers/jenkins/config/labels\_controller.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvamVua2lucy9jb25maWcvbGFiZWxzX2NvbnRyb2xsZXIuZ28=) | `88.15% <88.15%> (ø)` | |
| [controllers/core/fake.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvY29yZS9mYWtlLmdv) | `100.00% <100.00%> (ø)` | |
| [controllers/core/fake\_manager.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvY29yZS9mYWtlX21hbmFnZXIuZ28=) | `100.00% <100.00%> (ø)` | |
| [pkg/kapis/devops/v1alpha3/register.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL2thcGlzL2Rldm9wcy92MWFscGhhMy9yZWdpc3Rlci5nbw==) | `100.00% <100.00%> (ø)` | |
| [...ontrollers/jenkins/pipeline/pipeline\_controller.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvamVua2lucy9waXBlbGluZS9waXBlbGluZV9jb250cm9sbGVyLmdv) | `43.18% <0.00%> (+3.78%)` | :arrow_up: |
| [...kg/models/devops/project\_pipeline\_sonar\_handler.go](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL21vZGVscy9kZXZvcHMvcHJvamVjdF9waXBlbGluZV9zb25hcl9oYW5kbGVyLmdv) | `4.61% <0.00%> (+4.61%)` | :arrow_up: |
| ... and [9 more](https://codecov.io/gh/kubesphere/ks-devops/pull/748/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/748?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/748?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere). Last update [40ec4b7...752af50](https://codecov.io/gh/kubesphere/ks-devops/pull/748?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).

LinuxSuRen: I have tested it manually with the following images:

```
kubespheredev/devops-apiserver:dev--
```

```
kubespheredev/devops-controller:dev--
```

test result:
![image](https://user-images.githubusercontent.com/1450685/180692545-586b4948-23d0-48ba-af63-d7b3a1b58159.png)


Patch:
diff --git a/cmd/controller/app/controllers.go b/cmd/controller/app/controllers.go
index f7d1a45..6710643 100644
--- a/cmd/controller/app/controllers.go
+++ b/cmd/controller/app/controllers.go
@@ -121,6 +121,13 @@ func getAllControllers(mgr manager.Manager, client k8s.Client, informerFactory i
 	fluxcdGitRepoReconciler := &fluxcd.GitRepositoryReconciler{
 		Client: mgr.GetClient(),
 	}
+	tokenIssuer := token.NewTokenIssuer(s.JWTOptions.Secret, s.JWTOptions.MaximumClockSkew)
+	jenkinsAgentLabelsReconciler := config.AgentLabelsReconciler{
+		Client:          mgr.GetClient(),
+		TargetNamespace: "kubesphere-devops-system",
+		TokenIssuer:     tokenIssuer,
+		JenkinsClient:   jenkinsCore,
+	}
 
 	return map[string]func(mgr manager.Manager) error{
 		gitRepoReconcilers.GetName(): func(mgr manager.Manager) error {
@@ -169,7 +176,6 @@ func getAllControllers(mgr manager.Manager, client k8s.Client, informerFactory i
 					informerFactory.KubeSphereSharedInformerFactory().Devops().V1alpha3().Pipelines()))
 			}
 
-			tokenIssuer := token.NewTokenIssuer(s.JWTOptions.Secret, s.JWTOptions.MaximumClockSkew)
 			if err == nil {
 				jenkinsfileReconciler := &jenkinspipeline.JenkinsfileReconciler{
 					Client:      mgr.GetClient(),
@@ -178,6 +184,9 @@ func getAllControllers(mgr manager.Manager, client k8s.Client, informerFactory i
 				}
 				err = jenkinsfileReconciler.SetupWithManager(mgr)
 			}
+			if err == nil {
+				err = jenkinsAgentLabelsReconciler.SetupWithManager(mgr)
+			}
 			return err
 		},
 		argocdReconciler.GetGroupName(): func(mgr manager.Manager) (err error) {
diff --git a/controllers/core/fake.go b/controllers/core/fake.go
index 405440d..c194c88 100644
--- a/controllers/core/fake.go
+++ b/controllers/core/fake.go
@@ -18,6 +18,7 @@ package core
 
 import (
 	"errors"
+	"github.com/stretchr/testify/assert"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 )
@@ -49,3 +50,9 @@ func (f *FakeGroupedReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	}
 	return nil
 }
+
+// NoErrors represents no errors
+func NoErrors(t assert.TestingT, err error, i ...interface{}) bool {
+	assert.Nil(t, err)
+	return true
+}
diff --git a/controllers/core/fake_manager.go b/controllers/core/fake_manager.go
new file mode 100644
index 0000000..4b56c73
--- /dev/null
+++ b/controllers/core/fake_manager.go
@@ -0,0 +1,123 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package core
+
+import (
+	"github.com/go-logr/logr"
+	"k8s.io/apimachinery/pkg/api/meta"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/rest"
+	"k8s.io/client-go/tools/record"
+	"net/http"
+	"sigs.k8s.io/controller-runtime/pkg/cache"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/healthz"
+	"sigs.k8s.io/controller-runtime/pkg/log"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	"sigs.k8s.io/controller-runtime/pkg/webhook"
+)
+
+// FakeManager is for the test purpose
+type FakeManager struct {
+	Client client.Client
+	Scheme *runtime.Scheme
+}
+
+// Add is a fake method
+func (f *FakeManager) Add(manager.Runnable) error {
+	return nil
+}
+
+// Elected is a fake method
+func (f *FakeManager) Elected() <-chan struct{} {
+	return nil
+}
+
+// SetFields is a fake method
+func (f *FakeManager) SetFields(interface{}) error {
+	return nil
+}
+
+// AddMetricsExtraHandler is a fake method
+func (f *FakeManager) AddMetricsExtraHandler(path string, handler http.Handler) error {
+	return nil
+}
+
+// AddHealthzCheck is a fake method
+func (f *FakeManager) AddHealthzCheck(string, healthz.Checker) error {
+	return nil
+}
+
+// AddReadyzCheck is a fake method
+func (f *FakeManager) AddReadyzCheck(string, healthz.Checker) error {
+	return nil
+}
+
+// Start is a fake method
+func (f *FakeManager) Start(<-chan struct{}) error {
+	return nil
+}
+
+// GetConfig is a fake method
+func (f *FakeManager) GetConfig() *rest.Config {
+	return nil
+}
+
+// GetScheme is a fake method
+func (f *FakeManager) GetScheme() *runtime.Scheme {
+	return f.Scheme
+}
+
+// GetClient is a fake method
+func (f *FakeManager) GetClient() client.Client {
+	return f.Client
+}
+
+// GetFieldIndexer is a fake method
+func (f *FakeManager) GetFieldIndexer() client.FieldIndexer {
+	return nil
+}
+
+// GetCache is a fake method
+func (f *FakeManager) GetCache() cache.Cache {
+	return nil
+}
+
+// GetEventRecorderFor is a fake method
+func (f *FakeManager) GetEventRecorderFor(name string) record.EventRecorder {
+	return nil
+}
+
+// GetRESTMapper is a fake method
+func (f *FakeManager) GetRESTMapper() meta.RESTMapper {
+	return meta.FirstHitRESTMapper{}
+}
+
+// GetAPIReader is a fake method
+func (f *FakeManager) GetAPIReader() client.Reader {
+	return f.Client
+}
+
+// GetWebhookServer is a fake method
+func (f *FakeManager) GetWebhookServer() *webhook.Server {
+	return nil
+}
+
+// GetLogger is a fake method
+func (f *FakeManager) GetLogger() logr.Logger {
+	return log.NullLogger{}
+}
diff --git a/controllers/jenkins/config/labels_controller.go b/controllers/jenkins/config/labels_controller.go
new file mode 100644
index 0000000..e996caa
--- /dev/null
+++ b/controllers/jenkins/config/labels_controller.go
@@ -0,0 +1,189 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package config
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"github.com/go-logr/logr"
+	"github.com/jenkins-zh/jenkins-client/pkg/core"
+	v1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/apiserver/pkg/authentication/user"
+	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/pkg/api/devops"
+	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
+	"kubesphere.io/devops/pkg/jwt/token"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
+	"strings"
+	"time"
+)
+
+// tokenExpireIn indicates that the temporary token issued by controller will be expired in some time.
+const tokenExpireIn time.Duration = 5 * time.Minute
+
+// AgentLabelsReconciler responsible for the Jenkins agent labels sync
+type AgentLabelsReconciler struct {
+	// TargetNamespace indicate which namespace the target ConfigMap located in
+	TargetNamespace string
+	JenkinsClient   core.JenkinsCore
+	TokenIssuer     token.Issuer
+
+	targetName string
+	client.Client
+	log      logr.Logger
+	recorder record.EventRecorder
+}
+
+// Reconcile makes sure the target ConfigMap has all the Jenkins agent labels
+func (r *AgentLabelsReconciler) Reconcile(req ctrl.Request) (result ctrl.Result, err error) {
+	var cm *v1.ConfigMap
+	if cm, err = r.getConfigMap(); err != nil {
+		if apierrors.IsNotFound(err) {
+			err = r.initConfigMap()
+		}
+
+		if err != nil {
+			return
+		}
+	}
+
+	var labels []string
+	if labels, err = r.getLabels(); err != nil {
+		err = fmt.Errorf("failed to get labels, error: %v", err)
+		return
+	}
+
+	if changed := setLabelsToConfigMap(labels, cm); changed {
+		err = r.Update(context.Background(), cm)
+	}
+
+	// make sure we could always get the latest Jenkins labels
+	result = ctrl.Result{
+		RequeueAfter: time.Minute * 5,
+	}
+	return
+}
+
+func (r *AgentLabelsReconciler) getLabels() (labels []string, err error) {
+	// set up the Jenkins client
+	var c *core.JenkinsCore
+	if c, err = r.getOrCreateJenkinsCore(map[string]string{
+		v1alpha3.PipelineRunCreatorAnnoKey: "admin",
+	}); err != nil {
+		err = fmt.Errorf("failed to create Jenkins client, error: %v", err)
+		return
+	}
+	c.RoundTripper = r.JenkinsClient.RoundTripper
+	coreClient := core.Client{JenkinsCore: *c}
+
+	var labelRes *core.LabelsResponse
+	if labelRes, err = coreClient.GetLabels(); err != nil {
+		err = fmt.Errorf("failed to get lables from Jenkins, error: %v", err)
+		return
+	}
+	labels = labelRes.GetLabels()
+	return
+}
+
+func (r *AgentLabelsReconciler) getOrCreateJenkinsCore(annotations map[string]string) (*core.JenkinsCore, error) {
+	creator, ok := annotations[v1alpha3.PipelineRunCreatorAnnoKey]
+	if !ok || creator == "" {
+		return &r.JenkinsClient, nil
+	}
+	// create a new JenkinsCore for current creator
+	accessToken, err := r.TokenIssuer.IssueTo(&user.DefaultInfo{Name: creator}, token.AccessToken, tokenExpireIn)
+	if err != nil {
+		return nil, fmt.Errorf("failed to issue access token for creator %s, error was %v", creator, err)
+	}
+	jenkinsCore := &core.JenkinsCore{
+		URL:      r.JenkinsClient.URL,
+		UserName: creator,
+		Token:    accessToken,
+	}
+	return jenkinsCore, nil
+}
+
+func setLabelsToConfigMap(labels []string, cm *v1.ConfigMap) (changed bool) {
+	if cm.Data == nil {
+		cm.Data = map[string]string{}
+	}
+	if cm.Data[devops.JenkinsAgentLabelsKey] != strings.Join(labels, ",") {
+		cm.Data[devops.JenkinsAgentLabelsKey] = strings.Join(labels, ",")
+		changed = true
+	}
+	return
+}
+
+func (r *AgentLabelsReconciler) initConfigMap() (err error) {
+	var labels []string
+	if labels, err = r.getLabels(); err == nil {
+		cm := &v1.ConfigMap{}
+		setLabelsToConfigMap(labels, cm)
+		cm.SetNamespace(r.TargetNamespace)
+		cm.SetName(r.targetName)
+
+		err = r.Create(context.Background(), cm)
+	}
+	return
+}
+
+func (r *AgentLabelsReconciler) getConfigMap() (cm *v1.ConfigMap, err error) {
+	cm = &v1.ConfigMap{}
+	err = r.Get(context.Background(), types.NamespacedName{
+		Namespace: r.TargetNamespace,
+		Name:      r.targetName,
+	}, cm)
+	return
+}
+
+// GetName returns the name of this reconciler
+func (r *AgentLabelsReconciler) GetName() string {
+	return "JenkinsAgentLabelReconciler"
+}
+
+// GetGroupName returns the group name of this reconciler
+func (r *AgentLabelsReconciler) GetGroupName() string {
+	return "jenkins"
+}
+
+func getSpecificConfigMapPredicate(name, namespace string) predicate.Funcs {
+	return predicate.NewPredicateFuncs(func(meta metav1.Object, object runtime.Object) (ok bool) {
+		ok = meta.GetName() == name && meta.GetNamespace() == namespace
+		return
+	})
+}
+
+// SetupWithManager setups the all necessary fields
+func (r *AgentLabelsReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	if r.targetName == "" {
+		r.targetName = "jenkins-agent-config"
+	}
+	if r.TargetNamespace == "" {
+		return errors.New("the target namespace is required")
+	}
+
+	r.log = ctrl.Log.WithName(r.GetName())
+	r.recorder = mgr.GetEventRecorderFor(r.GetName())
+	return ctrl.NewControllerManagedBy(mgr).
+		WithEventFilter(getSpecificConfigMapPredicate(r.targetName, r.TargetNamespace)).
+		For(&v1.ConfigMap{}).
+		Complete(r)
+}
diff --git a/go.mod b/go.mod
index 1480877..c86e285 100644
--- a/go.mod
+++ b/go.mod
@@ -12,7 +12,6 @@ require (
 	github.com/emicklei/go-restful v2.9.6+incompatible
 	github.com/emicklei/go-restful-openapi v1.4.1
 	github.com/emicklei/go-restful-swagger12 v0.0.0-20201014110547-68ccff494617
-	github.com/fatih/structs v1.1.0
 	github.com/form3tech-oss/jwt-go v3.2.2+incompatible
 	github.com/go-logr/logr v0.3.0
 	github.com/go-logr/zapr v0.2.0 // indirect
@@ -25,7 +24,7 @@ require (
 	github.com/google/go-querystring v1.1.0 // indirect
 	github.com/h2non/gock v1.0.9
 	github.com/jenkins-x/go-scm v1.11.4
-	github.com/jenkins-zh/jenkins-client v0.0.10-0.20220706065616-22f8c7675234
+	github.com/jenkins-zh/jenkins-client v0.0.11
 	github.com/kubesphere/sonargo v0.0.2
 	github.com/onsi/ginkgo v1.16.4
 	github.com/onsi/gomega v1.15.0
diff --git a/go.sum b/go.sum
index 33d4b55..e783523 100644
--- a/go.sum
+++ b/go.sum
@@ -135,8 +135,6 @@ github.com/evanphx/json-patch v4.5.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLi
 github.com/evanphx/json-patch v4.9.0+incompatible h1:kLcOMZeuLAJvL2BPWLMIj5oaZQobrkAqrL+WFZwQses=
 github.com/evanphx/json-patch v4.9.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
-github.com/fatih/structs v1.1.0 h1:Q7juDM0QtcnhCpeyLGQKyg4TOIghuNXrkL32pHAUMxo=
-github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
 github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
 github.com/form3tech-oss/jwt-go v3.2.2+incompatible h1:TcekIExNqud5crz4xD2pavyTgWiPvpYe4Xau31I0PRk=
 github.com/form3tech-oss/jwt-go v3.2.2+incompatible/go.mod h1:pbq4aXjuKjdthFRnoDwaVPLA+WlJuPGy+QneDUgJi2k=
@@ -341,8 +339,8 @@ github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NH
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
 github.com/jenkins-zh/jenkins-cli v0.0.32/go.mod h1:uE1mH9PNITrg0sugv6HXuM/CSddg0zxXoYu3w57I3JY=
-github.com/jenkins-zh/jenkins-client v0.0.10-0.20220706065616-22f8c7675234 h1:gO2Ca6t/V7l7SD9H6mAy6UkcHVT0Ey3sZ6o0jIKPUHg=
-github.com/jenkins-zh/jenkins-client v0.0.10-0.20220706065616-22f8c7675234/go.mod h1:ICBk7OOoTafVP//f/VfKZ34c0ff8vJwVnOsF9btiMYU=
+github.com/jenkins-zh/jenkins-client v0.0.11 h1:g7f4WhI4iA6JA6l3fW5E/vSsBNYYr3/Ey+wwLxPTKeQ=
+github.com/jenkins-zh/jenkins-client v0.0.11/go.mod h1:ICBk7OOoTafVP//f/VfKZ34c0ff8vJwVnOsF9btiMYU=
 github.com/jenkins-zh/jenkins-formulas v0.0.5/go.mod h1:zS8fm8u5L6FcjZM0QznXsLV9T2UtSVK+hT6Sm76iUZ4=
 github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
 github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4=
diff --git a/pkg/api/devops/constants.go b/pkg/api/devops/constants.go
new file mode 100644
index 0000000..9572ed1
--- /dev/null
+++ b/pkg/api/devops/constants.go
@@ -0,0 +1,22 @@
+// Copyright 2022 KubeSphere Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
+package devops
+
+const (
+	// JenkinsAgentLabelsKey is the key of Jenkins agent labels. For example, you could find it in ConfigMap
+	// kubesphere-devops-system/jenkins-agent-config
+	JenkinsAgentLabelsKey = "agent.labels"
+)
diff --git a/pkg/kapis/devops/v1alpha3/handler.go b/pkg/kapis/devops/v1alpha3/handler.go
index de1bea6..d0fdd8d 100644
--- a/pkg/kapis/devops/v1alpha3/handler.go
+++ b/pkg/kapis/devops/v1alpha3/handler.go
@@ -221,6 +221,21 @@ func NewSuccessResponse() *GenericResponse {
 	}
 }
 
+// GenericArrayResponse represents a generic array response
+type GenericArrayResponse struct {
+	Status  string   `json:"status"`
+	Data    []string `json:"data"`
+	Message string   `json:"message"`
+}
+
+// NewSuccessGenericArrayResponse creates a generic array response
+func NewSuccessGenericArrayResponse(data []string) *GenericArrayResponse {
+	return &GenericArrayResponse{
+		Status: "success",
+		Data:   data,
+	}
+}
+
 func (h *devopsHandler) UpdateJenkinsfile(request *restful.Request, response *restful.Response) {
 	projectName := request.PathParameter("devops")
 	pipelineName := request.PathParameter("pipeline")
@@ -346,6 +361,21 @@ func (h *devopsHandler) DeleteCredential(request *restful.Request, response *res
 	}
 }
 
+func (h *devopsHandler) getJenkinsLabels(request *restful.Request, response *restful.Response) {
+	client, err := h.getDevOps(request)
+	if err != nil {
+		kapis.HandleBadRequest(response, request, err)
+		return
+	}
+
+	var labels []string
+	if labels, err = client.GetJenkinsAgentLabels(); err != nil {
+		kapis.HandleBadRequest(response, request, err)
+	} else {
+		errorHandle(request, response, NewSuccessGenericArrayResponse(labels), nil)
+	}
+}
+
 func (h *devopsHandler) getDevOps(request *restful.Request) (operator devops.DevopsOperator, err error) {
 	ctx := request.Request.Context()
 	token := ctx.Value(constants.K8SToken).(constants.ContextKeyK8SToken)
diff --git a/pkg/kapis/devops/v1alpha3/register.go b/pkg/kapis/devops/v1alpha3/register.go
index ff3a931..5809977 100644
--- a/pkg/kapis/devops/v1alpha3/register.go
+++ b/pkg/kapis/devops/v1alpha3/register.go
@@ -85,6 +85,7 @@ func registerRoutes(devopsClient devopsClient.Interface, k8sClient k8s.Client, c
 	registerRoutersForPipelines(handler, ws)
 	registerRoutersForWorkspace(handler, ws)
 	scm.RegisterRoutersForSCM(client, ws)
+	registerRoutersForCI(handler, ws)
 }
 
 func registerRoutersForCredentials(handler *devopsHandler, ws *restful.WebService) {
@@ -234,3 +235,10 @@ func registerRoutersForWorkspace(handler *devopsHandler, ws *restful.WebService)
 		Returns(http.StatusOK, api.StatusOK, v1alpha3.DevOpsProject{}).
 		Metadata(restfulspec.KeyOpenAPITags, []string{constants.DevOpsProjectTag}))
 }
+
+func registerRoutersForCI(handler *devopsHandler, ws *restful.WebService) {
+	ws.Route(ws.GET("/ci/nodelabels").
+		To(handler.getJenkinsLabels).
+		Doc("Get the all labels of the Jenkins").
+		Returns(http.StatusOK, api.StatusOK, GenericArrayResponse{}))
+}
diff --git a/pkg/models/devops/common.go b/pkg/models/devops/common.go
index 078da7e..3ea40fa 100644
--- a/pkg/models/devops/common.go
+++ b/pkg/models/devops/common.go
@@ -17,35 +17,11 @@ limitations under the License.
 package devops
 
 import (
-	"fmt"
 	"time"
 
-	"github.com/fatih/structs"
-
 	"kubesphere.io/devops/pkg/client/devops"
-	"kubesphere.io/devops/pkg/utils/stringutils"
 )
 
-func GetColumnsFromStruct(s interface{}) []string {
-	names := structs.Names(s)
-	for i, name := range names {
-		names[i] = stringutils.CamelCaseToUnderscore(name)
-	}
-	return names
-}
-
-func GetColumnsFromStructWithPrefix(prefix string, s interface{}) []string {
-	names := structs.Names(s)
-	for i, name := range names {
-		names[i] = WithPrefix(prefix, stringutils.CamelCaseToUnderscore(name))
-	}
-	return names
-}
-
-func WithPrefix(prefix, str string) string {
-	return prefix + "." + str
-}
-
 const (
 	StatusActive     = "active"
 	StatusDeleted    = "deleted"
@@ -299,27 +275,7 @@ var JenkinsPipelinePermissionMap = map[string]devops.ProjectPermissionIds{
 	},
 }
 
-// get roleName of the project
-func GetProjectRoleName(projectId, role string) string {
-	return fmt.Sprintf("%s-%s-project", projectId, role)
-}
-
-// get roleName of the pipeline
-func GetPipelineRoleName(projectId, role string) string {
-	return fmt.Sprintf("%s-%s-pipeline", projectId, role)
-}
-
-// get pattern string of the project
-func GetProjectRolePattern(projectId string) string {
-	return fmt.Sprintf("^%s$", projectId)
-}
-
-// get pattern string of the project
-func GetPipelineRolePattern(projectId string) string {
-	return fmt.Sprintf("^%s/.*", projectId)
-}
-
-// get unified sync current time
+// GetSyncNowTime returns unified sync current time
 func GetSyncNowTime() string {
 	return time.Now().String()
 }
diff --git a/pkg/models/devops/devops.go b/pkg/models/devops/devops.go
index 81693f6..9798cec 100644
--- a/pkg/models/devops/devops.go
+++ b/pkg/models/devops/devops.go
@@ -42,6 +42,7 @@ import (
 	"kubesphere.io/devops/pkg/utils/secretutil"
 
 	"kubesphere.io/devops/pkg/api"
+	devopsapi "kubesphere.io/devops/pkg/api/devops"
 	"kubesphere.io/devops/pkg/apiserver/query"
 	kubesphere "kubesphere.io/devops/pkg/client/clientset/versioned"
 	"kubesphere.io/devops/pkg/client/devops"
@@ -121,6 +122,8 @@ type DevopsOperator interface {
 
 	ToJenkinsfile(req *http.Request) (*devops.ResJenkinsfile, error)
 	ToJSON(req *http.Request) (map[string]interface{}, error)
+
+	GetJenkinsAgentLabels() ([]string, error)
 }
 
 type devopsOperator struct {
@@ -998,6 +1001,16 @@ func (d devopsOperator) ToJSON(req *http.Request) (map[string]interface{}, error
 	return res, err
 }
 
+func (d devopsOperator) GetJenkinsAgentLabels() (labels []string, err error) {
+	var cm *v1.ConfigMap
+	if cm, err = d.k8sclient.CoreV1().ConfigMaps("kubesphere-devops-system").
+		Get(context.Background(), "jenkins-agent-config", metav1.GetOptions{}); err == nil {
+		labelsInStr := cm.Data[devopsapi.JenkinsAgentLabelsKey]
+		labels = strings.Split(labelsInStr, ",")
+	}
+	return
+}
+
 func (d devopsOperator) isGenerateNameUnique(workspace, generateName string) (bool, error) {
 	projects, err := d.ksclient.DevopsV1alpha3().DevOpsProjects().List(d.context, metav1.ListOptions{})
 	if err != nil {


Test patch:
diff --git a/controllers/argocd/application_controller_test.go b/controllers/argocd/application_controller_test.go
index f3cdc22..b7ce5aa 100644
--- a/controllers/argocd/application_controller_test.go
+++ b/controllers/argocd/application_controller_test.go
@@ -18,13 +18,17 @@ package argocd
 
 import (
 	"context"
+	"fmt"
 	"github.com/go-logr/logr"
 	"github.com/stretchr/testify/assert"
+	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/controllers/core"
 	"kubesphere.io/devops/pkg/api/gitops/v1alpha1"
+	controllerruntime "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 	"sigs.k8s.io/controller-runtime/pkg/event"
@@ -613,3 +617,46 @@ func Test_specificAnnotationsOrLabelsChangedPredicate_Update(t *testing.T) {
 		})
 	}
 }
+
+func TestApplicationReconciler_SetupWithManager(t *testing.T) {
+	schema, err := v1alpha1.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+	err = v1alpha1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	type fields struct {
+		Client   client.Client
+		log      logr.Logger
+		recorder record.EventRecorder
+	}
+	type args struct {
+		mgr controllerruntime.Manager
+	}
+	tests := []struct {
+		name    string
+		fields  fields
+		args    args
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "normal",
+		args: args{
+			mgr: &core.FakeManager{
+				Client: fake.NewFakeClientWithScheme(schema),
+				Scheme: schema,
+			},
+		},
+		wantErr: core.NoErrors,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &ApplicationReconciler{
+				Client:   tt.fields.Client,
+				log:      tt.fields.log,
+				recorder: tt.fields.recorder,
+			}
+			tt.wantErr(t, r.SetupWithManager(tt.args.mgr), fmt.Sprintf("SetupWithManager(%v)", tt.args.mgr))
+		})
+	}
+}
diff --git a/controllers/argocd/argocd-application-status-controller_test.go b/controllers/argocd/argocd-application-status-controller_test.go
index f02e6a2..1df5475 100644
--- a/controllers/argocd/argocd-application-status-controller_test.go
+++ b/controllers/argocd/argocd-application-status-controller_test.go
@@ -19,13 +19,14 @@ package argocd
 import (
 	"context"
 	"fmt"
+	"github.com/go-logr/logr"
 	"github.com/stretchr/testify/assert"
 	"io/ioutil"
 	v1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/tools/record"
-	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
+	"kubesphere.io/devops/controllers/core"
 	"kubesphere.io/devops/pkg/api/gitops/v1alpha1"
 	controllerruntime "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -35,7 +36,7 @@ import (
 )
 
 func Test_getArgoCDApplication(t *testing.T) {
-	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	schema, err := v1alpha1.SchemeBuilder.Register().Build()
 	assert.Nil(t, err)
 	err = v1.SchemeBuilder.AddToScheme(schema)
 	assert.Nil(t, err)
@@ -107,7 +108,7 @@ func Test_getArgoCDApplicationObject(t *testing.T) {
 }
 
 func TestArgoCDApplicationStatusReconciler_Reconcile(t *testing.T) {
-	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	schema, err := v1alpha1.SchemeBuilder.Register().Build()
 	assert.Nil(t, err)
 	err = v1.SchemeBuilder.AddToScheme(schema)
 	assert.Nil(t, err)
@@ -311,3 +312,44 @@ func Test_parseArgoStatus(t *testing.T) {
 		})
 	}
 }
+
+func TestApplicationStatusReconciler_SetupWithManager(t *testing.T) {
+	schema, err := v1alpha1.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	type fields struct {
+		Client   client.Client
+		log      logr.Logger
+		recorder record.EventRecorder
+	}
+	type args struct {
+		mgr controllerruntime.Manager
+	}
+	tests := []struct {
+		name    string
+		fields  fields
+		args    args
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "normal",
+		args: args{
+			mgr: &core.FakeManager{
+				Scheme: schema,
+				Client: fake.NewFakeClientWithScheme(schema),
+			},
+		},
+		wantErr: core.NoErrors,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &ApplicationStatusReconciler{
+				Client:   tt.fields.Client,
+				log:      tt.fields.log,
+				recorder: tt.fields.recorder,
+			}
+			tt.wantErr(t, r.SetupWithManager(tt.args.mgr), fmt.Sprintf("SetupWithManager(%v)", tt.args.mgr))
+		})
+	}
+}
diff --git a/controllers/argocd/git-repository-controller_test.go b/controllers/argocd/git-repository-controller_test.go
index b28ae72..b462244 100644
--- a/controllers/argocd/git-repository-controller_test.go
+++ b/controllers/argocd/git-repository-controller_test.go
@@ -18,11 +18,13 @@ package argocd
 
 import (
 	"fmt"
+	"github.com/go-logr/logr"
 	"github.com/stretchr/testify/assert"
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/controllers/core"
 	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
 	controllerruntime "sigs.k8s.io/controller-runtime"
 	ctrl "sigs.k8s.io/controller-runtime"
@@ -204,3 +206,46 @@ func TestGitRepositoryController_Reconcile(t *testing.T) {
 		})
 	}
 }
+
+func TestGitRepositoryController_SetupWithManager(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	type fields struct {
+		Client        client.Client
+		log           logr.Logger
+		recorder      record.EventRecorder
+		ArgoNamespace string
+	}
+	type args struct {
+		mgr ctrl.Manager
+	}
+	tests := []struct {
+		name    string
+		fields  fields
+		args    args
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "normal",
+		args: args{
+			mgr: &core.FakeManager{
+				Client: fake.NewFakeClientWithScheme(schema),
+				Scheme: schema,
+			},
+		},
+		wantErr: core.NoErrors,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			c := &GitRepositoryController{
+				Client:        tt.fields.Client,
+				log:           tt.fields.log,
+				recorder:      tt.fields.recorder,
+				ArgoNamespace: tt.fields.ArgoNamespace,
+			}
+			tt.wantErr(t, c.SetupWithManager(tt.args.mgr), fmt.Sprintf("SetupWithManager(%v)", tt.args.mgr))
+		})
+	}
+}
diff --git a/controllers/argocd/image-updater-controller_test.go b/controllers/argocd/image-updater-controller_test.go
index 5529fb1..1e37c00 100644
--- a/controllers/argocd/image-updater-controller_test.go
+++ b/controllers/argocd/image-updater-controller_test.go
@@ -19,11 +19,13 @@ package argocd
 import (
 	"context"
 	"fmt"
+	"github.com/go-logr/logr"
 	"github.com/stretchr/testify/assert"
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/controllers/core"
 	"kubesphere.io/devops/pkg/api/gitops/v1alpha1"
 	controllerruntime "sigs.k8s.io/controller-runtime"
 	ctrl "sigs.k8s.io/controller-runtime"
@@ -349,3 +351,44 @@ func Test_setImagePreference(t *testing.T) {
 		})
 	}
 }
+
+func TestImageUpdaterReconciler_SetupWithManager(t *testing.T) {
+	schema, err := v1alpha1.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	type fields struct {
+		Client   client.Client
+		log      logr.Logger
+		recorder record.EventRecorder
+	}
+	type args struct {
+		mgr ctrl.Manager
+	}
+	tests := []struct {
+		name    string
+		fields  fields
+		args    args
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "normal",
+		args: args{
+			mgr: &core.FakeManager{
+				Client: fake.NewFakeClientWithScheme(schema),
+				Scheme: schema,
+			},
+		},
+		wantErr: core.NoErrors,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &ImageUpdaterReconciler{
+				Client:   tt.fields.Client,
+				log:      tt.fields.log,
+				recorder: tt.fields.recorder,
+			}
+			tt.wantErr(t, r.SetupWithManager(tt.args.mgr), fmt.Sprintf("SetupWithManager(%v)", tt.args.mgr))
+		})
+	}
+}
diff --git a/controllers/argocd/multi-cluster-controller_test.go b/controllers/argocd/multi-cluster-controller_test.go
index 919f5be..23f6cda 100644
--- a/controllers/argocd/multi-cluster-controller_test.go
+++ b/controllers/argocd/multi-cluster-controller_test.go
@@ -27,6 +27,8 @@ import (
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/controllers/core"
+	"kubesphere.io/devops/pkg/api/devops/v1alpha1"
 	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
 	controllerruntime "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -480,3 +482,46 @@ users:
 		})
 	}
 }
+
+func TestMultiClusterReconciler_SetupWithManager(t *testing.T) {
+	schema, err := v1alpha1.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	type fields struct {
+		Client          client.Client
+		log             logr.Logger
+		recorder        record.EventRecorder
+		argocdNamespace string
+	}
+	type args struct {
+		mgr controllerruntime.Manager
+	}
+	tests := []struct {
+		name    string
+		fields  fields
+		args    args
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "normal",
+		args: args{
+			mgr: &core.FakeManager{
+				Client: fake.NewFakeClientWithScheme(schema),
+				Scheme: schema,
+			},
+		},
+		wantErr: core.NoErrors,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &MultiClusterReconciler{
+				Client:          tt.fields.Client,
+				log:             tt.fields.log,
+				recorder:        tt.fields.recorder,
+				argocdNamespace: tt.fields.argocdNamespace,
+			}
+			tt.wantErr(t, r.SetupWithManager(tt.args.mgr), fmt.Sprintf("SetupWithManager(%v)", tt.args.mgr))
+		})
+	}
+}
diff --git a/controllers/argocd/project_controller_test.go b/controllers/argocd/project_controller_test.go
index aa0a0d8..d6ab52d 100644
--- a/controllers/argocd/project_controller_test.go
+++ b/controllers/argocd/project_controller_test.go
@@ -18,11 +18,17 @@ package argocd
 
 import (
 	"context"
+	"fmt"
+	"github.com/go-logr/logr"
 	"github.com/stretchr/testify/assert"
-	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/controllers/core"
 	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
+	controllerruntime "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 	"testing"
@@ -64,7 +70,7 @@ func TestCreateUnstructuredObject(t *testing.T) {
 			Namespace: "ns2",
 		}},
 		Description: "description",
-		ClusterResourceWhitelist: []v1.GroupKind{{
+		ClusterResourceWhitelist: []metav1.GroupKind{{
 			Group: "group1",
 			Kind:  "kind1",
 		}, {
@@ -177,3 +183,46 @@ func TestReconcileArgoProject(t *testing.T) {
 		})
 	}
 }
+
+func TestReconciler_SetupWithManager(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	type fields struct {
+		Client        client.Client
+		log           logr.Logger
+		recorder      record.EventRecorder
+		ArgoNamespace string
+	}
+	type args struct {
+		mgr controllerruntime.Manager
+	}
+	tests := []struct {
+		name    string
+		fields  fields
+		args    args
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "normal",
+		args: args{
+			mgr: &core.FakeManager{
+				Client: fake.NewFakeClientWithScheme(schema),
+				Scheme: schema,
+			},
+		},
+		wantErr: core.NoErrors,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &Reconciler{
+				Client:        tt.fields.Client,
+				log:           tt.fields.log,
+				recorder:      tt.fields.recorder,
+				ArgoNamespace: tt.fields.ArgoNamespace,
+			}
+			tt.wantErr(t, r.SetupWithManager(tt.args.mgr), fmt.Sprintf("SetupWithManager(%v)", tt.args.mgr))
+		})
+	}
+}
diff --git a/controllers/core/fake_manager_test.go b/controllers/core/fake_manager_test.go
new file mode 100644
index 0000000..caa75a0
--- /dev/null
+++ b/controllers/core/fake_manager_test.go
@@ -0,0 +1,57 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package core
+
+import (
+	"github.com/stretchr/testify/assert"
+	v1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/meta"
+	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
+	clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/controller-runtime/pkg/log"
+	"testing"
+)
+
+func TestFakeManager(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+	client := clientfake.NewFakeClientWithScheme(schema)
+
+	fake := FakeManager{
+		Client: client,
+		Scheme: schema,
+	}
+	assert.Nil(t, fake.Add(nil))
+	assert.Nil(t, fake.Elected())
+	assert.Nil(t, fake.SetFields(nil))
+	assert.Nil(t, fake.AddMetricsExtraHandler("", nil))
+	assert.Nil(t, fake.AddHealthzCheck("", nil))
+	assert.Nil(t, fake.AddReadyzCheck("", nil))
+	assert.Nil(t, fake.Start(nil))
+	assert.Nil(t, fake.GetConfig())
+	assert.Equal(t, schema, fake.GetScheme())
+	assert.Equal(t, client, fake.GetClient())
+	assert.Nil(t, fake.GetFieldIndexer())
+	assert.Nil(t, fake.GetCache())
+	assert.Nil(t, fake.GetEventRecorderFor(""))
+	assert.Equal(t, meta.FirstHitRESTMapper{}, fake.GetRESTMapper())
+	assert.Equal(t, client, fake.GetAPIReader())
+	assert.Nil(t, fake.GetWebhookServer())
+	assert.Equal(t, log.NullLogger{}, fake.GetLogger())
+}
diff --git a/controllers/core/fake_test.go b/controllers/core/fake_test.go
new file mode 100644
index 0000000..77128a4
--- /dev/null
+++ b/controllers/core/fake_test.go
@@ -0,0 +1,26 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package core
+
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func TestNoErrors(t *testing.T) {
+	assert.True(t, NoErrors(t, nil))
+}
diff --git a/controllers/jenkins/config/interface_test.go b/controllers/jenkins/config/interface_test.go
new file mode 100644
index 0000000..fc63259
--- /dev/null
+++ b/controllers/jenkins/config/interface_test.go
@@ -0,0 +1,50 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package config
+
+import (
+	"github.com/stretchr/testify/assert"
+	"kubesphere.io/devops/controllers/core"
+	"testing"
+)
+
+func TestInterfaceImplement(t *testing.T) {
+	type interInstance struct {
+		NamedReconciler core.NamedReconciler
+		GroupReconciler core.GroupReconciler
+	}
+
+	tests := []struct {
+		name     string
+		instance interInstance
+	}{{
+		name: "AgentLabelsReconciler",
+		instance: interInstance{
+			NamedReconciler: &AgentLabelsReconciler{},
+			GroupReconciler: &AgentLabelsReconciler{},
+		},
+	}}
+	for i := range tests {
+		tt := tests[i]
+		t.Run(tt.name, func(t *testing.T) {
+			assert.NotNil(t, tt.instance.NamedReconciler)
+			assert.NotEmpty(t, tt.instance.NamedReconciler.GetName())
+			assert.NotNil(t, tt.instance.GroupReconciler)
+			assert.NotEmpty(t, tt.instance.GroupReconciler.GetGroupName())
+		})
+	}
+}
diff --git a/controllers/jenkins/config/labels_controller_test.go b/controllers/jenkins/config/labels_controller_test.go
new file mode 100644
index 0000000..7d90b49
--- /dev/null
+++ b/controllers/jenkins/config/labels_controller_test.go
@@ -0,0 +1,320 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package config
+
+import (
+	"fmt"
+	"github.com/go-logr/logr"
+	"github.com/golang/mock/gomock"
+	"github.com/jenkins-zh/jenkins-client/pkg/core"
+	"github.com/jenkins-zh/jenkins-client/pkg/mock/mhttp"
+	"github.com/stretchr/testify/assert"
+	v1 "k8s.io/api/core/v1"
+	v12 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/tools/record"
+	mgrcore "kubesphere.io/devops/controllers/core"
+	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
+	"kubesphere.io/devops/pkg/jwt/token"
+	controllerruntime "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/controller-runtime/pkg/event"
+	"sigs.k8s.io/controller-runtime/pkg/log"
+	"testing"
+	"time"
+)
+
+func Test_setLabelsToConfigMap(t *testing.T) {
+	emptyConfigMap := &v1.ConfigMap{}
+
+	oneItemConfigMap := &v1.ConfigMap{Data: map[string]string{"name": "good"}}
+
+	twoItemsConfigMap := &v1.ConfigMap{
+		Data: map[string]string{
+			"name": "good,bad",
+		},
+	}
+
+	type args struct {
+		labels []string
+		cm     *v1.ConfigMap
+	}
+	tests := []struct {
+		name        string
+		args        args
+		wantChanged bool
+	}{{
+		name: "set labels to an empty",
+		args: args{
+			labels: []string{"good"},
+			cm:     emptyConfigMap.DeepCopy(),
+		},
+		wantChanged: true,
+	}, {
+		name: "one item in the ConfigMap",
+		args: args{
+			labels: []string{"good"},
+			cm:     oneItemConfigMap.DeepCopy(),
+		},
+		wantChanged: true,
+	}, {
+		name: "different ordered items",
+		args: args{
+			labels: []string{"bad", "good"},
+			cm:     twoItemsConfigMap.DeepCopy(),
+		},
+		wantChanged: true,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, tt.wantChanged, setLabelsToConfigMap(tt.args.labels, tt.args.cm), "setLabelsToConfigMap(%v, %v)", tt.args.labels, tt.args.cm)
+		})
+	}
+}
+
+func TestAgentLabelsReconciler_Reconcile(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	defaultReq := controllerruntime.Request{
+		NamespacedName: types.NamespacedName{
+			Namespace: "ns",
+			Name:      "fake",
+		},
+	}
+
+	cmWithoutLabels := &v1.ConfigMap{}
+	cmWithoutLabels.Namespace = "ns"
+	cmWithoutLabels.Name = "fake"
+
+	cmWithLabels := cmWithoutLabels.DeepCopy()
+	cmWithLabels.Data = map[string]string{
+		"agent.labels": "a,b",
+	}
+
+	type fields struct {
+		TargetNamespace string
+		JenkinsClient   core.JenkinsCore
+		TokenIssuer     token.Issuer
+		targetName      string
+		Client          client.Client
+		log             logr.Logger
+		recorder        record.EventRecorder
+	}
+	type args struct {
+		req controllerruntime.Request
+	}
+	tests := []struct {
+		name       string
+		fields     fields
+		args       args
+		prepare    func(t *testing.T, c *core.JenkinsCore)
+		wantResult controllerruntime.Result
+		wantErr    assert.ErrorAssertionFunc
+	}{{
+		name: "not found ConfigMap",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema),
+			JenkinsClient: core.JenkinsCore{
+				URL: "http://localhost",
+			},
+			log:             log.NullLogger{},
+			TokenIssuer:     &token.FakeIssuer{},
+			targetName:      "fake",
+			TargetNamespace: "ns",
+		},
+		prepare: func(t *testing.T, c *core.JenkinsCore) {
+			ctrl := gomock.NewController(t)
+			roundTripper := mhttp.NewMockRoundTripper(ctrl)
+			c.RoundTripper = roundTripper
+
+			core.PrepareForToGetLabels(roundTripper, "http://localhost", "", "")
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.NotNil(t, err)
+			return true
+		},
+		args: args{req: defaultReq},
+	}, {
+		name: "have the specific ConfigMap",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema, cmWithoutLabels),
+			JenkinsClient: core.JenkinsCore{
+				URL: "http://localhost",
+			},
+			log:             log.NullLogger{},
+			TokenIssuer:     &token.FakeIssuer{},
+			targetName:      "fake",
+			TargetNamespace: "ns",
+		},
+		prepare: func(t *testing.T, c *core.JenkinsCore) {
+			ctrl := gomock.NewController(t)
+			roundTripper := mhttp.NewMockRoundTripper(ctrl)
+			c.RoundTripper = roundTripper
+
+			core.PrepareForToGetLabels(roundTripper, "http://localhost", "", "")
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+		wantResult: controllerruntime.Result{RequeueAfter: time.Minute * 5},
+		args:       args{req: defaultReq},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if tt.prepare != nil {
+				tt.prepare(t, &tt.fields.JenkinsClient)
+			}
+			r := &AgentLabelsReconciler{
+				TargetNamespace: tt.fields.TargetNamespace,
+				JenkinsClient:   tt.fields.JenkinsClient,
+				TokenIssuer:     tt.fields.TokenIssuer,
+				targetName:      tt.fields.targetName,
+				Client:          tt.fields.Client,
+				log:             tt.fields.log,
+				recorder:        tt.fields.recorder,
+			}
+			err = r.SetupWithManager(&mgrcore.FakeManager{
+				Scheme: schema,
+				Client: tt.fields.Client,
+			})
+			assert.Nil(t, err)
+			gotResult, err := r.Reconcile(tt.args.req)
+			if !tt.wantErr(t, err, fmt.Sprintf("Reconcile(%v)", tt.args.req)) {
+				return
+			}
+			assert.Equalf(t, tt.wantResult, gotResult, "Reconcile(%v)", tt.args.req)
+		})
+	}
+}
+
+func TestAgentLabelsReconciler_SetupWithManager(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	type fields struct {
+		TargetNamespace string
+		JenkinsClient   core.JenkinsCore
+		TokenIssuer     token.Issuer
+		targetName      string
+		Client          client.Client
+		log             logr.Logger
+		recorder        record.EventRecorder
+	}
+	tests := []struct {
+		name    string
+		fields  fields
+		verify  func(*testing.T, *AgentLabelsReconciler)
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "target name is empty",
+		fields: fields{
+			TargetNamespace: "ns",
+			targetName:      "",
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+		verify: func(t *testing.T, c *AgentLabelsReconciler) {
+			assert.Equal(t, "jenkins-agent-config", c.targetName)
+		},
+	}, {
+		name: "target namespace is empty",
+		fields: fields{
+			TargetNamespace: "",
+			targetName:      "",
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.NotNil(t, err)
+			return true
+		},
+		verify: func(t *testing.T, c *AgentLabelsReconciler) {
+			assert.Equal(t, "jenkins-agent-config", c.targetName)
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &AgentLabelsReconciler{
+				TargetNamespace: tt.fields.TargetNamespace,
+				JenkinsClient:   tt.fields.JenkinsClient,
+				TokenIssuer:     tt.fields.TokenIssuer,
+				targetName:      tt.fields.targetName,
+				Client:          tt.fields.Client,
+				log:             tt.fields.log,
+				recorder:        tt.fields.recorder,
+			}
+			mgr := &mgrcore.FakeManager{
+				Client: tt.fields.Client,
+				Scheme: schema,
+			}
+			tt.wantErr(t, r.SetupWithManager(mgr), fmt.Sprintf("SetupWithManager(%v)", mgr))
+			if tt.verify != nil {
+				tt.verify(t, r)
+			}
+		})
+	}
+}
+
+func Test_getSpecificConfigMapPredicate(t *testing.T) {
+	type args struct {
+		name            string
+		namespace       string
+		targetName      string
+		targetNamespace string
+	}
+	tests := []struct {
+		name string
+		args args
+		want bool
+	}{{
+		name: "normal",
+		args: args{
+			name:            "name",
+			namespace:       "ns",
+			targetName:      "name",
+			targetNamespace: "ns",
+		},
+		want: true,
+	}, {
+		name: "different name",
+		args: args{
+			name:            "name",
+			namespace:       "ns",
+			targetName:      "name-1",
+			targetNamespace: "ns-1",
+		},
+		want: false,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			funcs := getSpecificConfigMapPredicate(tt.args.name, tt.args.namespace)
+			result := funcs.Generic(event.GenericEvent{
+				Meta: &v1.ConfigMap{
+					ObjectMeta: v12.ObjectMeta{
+						Name:      tt.args.targetName,
+						Namespace: tt.args.targetNamespace,
+					},
+				},
+			})
+			assert.Equal(t, tt.want, result)
+		})
+	}
+}
diff --git a/pkg/kapis/devops/v1alpha3/handler_test.go b/pkg/kapis/devops/v1alpha3/handler_test.go
new file mode 100644
index 0000000..bf50ec5
--- /dev/null
+++ b/pkg/kapis/devops/v1alpha3/handler_test.go
@@ -0,0 +1,47 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha3
+
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func TestNewSuccessGenericArrayResponse(t *testing.T) {
+	type args struct {
+		data []string
+	}
+	tests := []struct {
+		name string
+		args args
+		want *GenericArrayResponse
+	}{{
+		name: "normal",
+		args: args{
+			data: []string{"good", "bad"},
+		},
+		want: &GenericArrayResponse{
+			Status: "success",
+			Data:   []string{"good", "bad"},
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, tt.want, NewSuccessGenericArrayResponse(tt.args.data), "NewSuccessGenericArrayResponse(%v)", tt.args.data)
+		})
+	}
+}
diff --git a/pkg/kapis/devops/v1alpha3/register_test.go b/pkg/kapis/devops/v1alpha3/register_test.go
index 969384c..0e4ced6 100644
--- a/pkg/kapis/devops/v1alpha3/register_test.go
+++ b/pkg/kapis/devops/v1alpha3/register_test.go
@@ -231,6 +231,13 @@ func TestAPIsExist(t *testing.T) {
 		},
 		body:       `{"data":"fake-jenkinsfile"}`,
 		expectCode: 404,
+	}, {
+		name: "get Jenkins labels",
+		args: args{
+			method: http.MethodGet,
+			uri:    "/ci/nodelabels",
+		},
+		expectCode: http.StatusBadRequest,
 	}}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
diff --git a/pkg/models/devops/common_test.go b/pkg/models/devops/common_test.go
new file mode 100644
index 0000000..452c0f4
--- /dev/null
+++ b/pkg/models/devops/common_test.go
@@ -0,0 +1,26 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package devops
+
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func TestGetSyncNowTime(t *testing.T) {
+	assert.NotEmpty(t, GetSyncNowTime())
+}
diff --git a/pkg/models/devops/devops_test.go b/pkg/models/devops/devops_test.go
index d4dce68..e5fd1bd 100644
--- a/pkg/models/devops/devops_test.go
+++ b/pkg/models/devops/devops_test.go
@@ -19,7 +19,9 @@ package devops
 import (
 	"context"
 	"fmt"
+	v12 "k8s.io/api/core/v1"
 	"k8s.io/client-go/kubernetes"
+	k8sfake "k8s.io/client-go/kubernetes/fake"
 	"net/http"
 	"testing"
 
@@ -544,3 +546,61 @@ func Test_devopsOperator_UpdatePipelineObj(t *testing.T) {
 		})
 	}
 }
+
+func Test_devopsOperator_GetJenkinsAgentLabels(t *testing.T) {
+	cm := &v12.ConfigMap{
+		Data: map[string]string{
+			"agent.labels": "good,bad",
+		},
+	}
+	cm.SetName("jenkins-agent-config")
+	cm.SetNamespace("kubesphere-devops-system")
+
+	type fields struct {
+		devopsClient devops.Interface
+		k8sclient    kubernetes.Interface
+		ksclient     versioned.Interface
+		context      context.Context
+	}
+	tests := []struct {
+		name       string
+		fields     fields
+		wantLabels []string
+		wantErr    assert.ErrorAssertionFunc
+	}{{
+		name: "not found ConfigMap",
+		fields: fields{
+			k8sclient: k8sfake.NewSimpleClientset(),
+		},
+		wantLabels: nil,
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.NotNil(t, err)
+			return true
+		},
+	}, {
+		name: "normal",
+		fields: fields{
+			k8sclient: k8sfake.NewSimpleClientset(cm.DeepCopy()),
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+		wantLabels: []string{"good", "bad"},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			d := devopsOperator{
+				devopsClient: tt.fields.devopsClient,
+				k8sclient:    tt.fields.k8sclient,
+				ksclient:     tt.fields.ksclient,
+				context:      tt.fields.context,
+			}
+			gotLabels, err := d.GetJenkinsAgentLabels()
+			if !tt.wantErr(t, err, fmt.Sprintf("GetJenkinsAgentLabels()")) {
+				return
+			}
+			assert.Equalf(t, tt.wantLabels, gotLabels, "GetJenkinsAgentLabels()")
+		})
+	}
+}
diff --git a/pkg/models/devops/jkerror_test.go b/pkg/models/devops/jkerror_test.go
new file mode 100644
index 0000000..788e31f
--- /dev/null
+++ b/pkg/models/devops/jkerror_test.go
@@ -0,0 +1,27 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package devops
+
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func TestJkError_Error(t *testing.T) {
+	err := &JkError{Message: "message"}
+	assert.Equal(t, "message", err.Error())
+}
diff --git a/pkg/models/devops/project_credential_handler_test.go b/pkg/models/devops/project_credential_handler_test.go
new file mode 100644
index 0000000..0cafe2b
--- /dev/null
+++ b/pkg/models/devops/project_credential_handler_test.go
@@ -0,0 +1,96 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package devops
+
+import (
+	"fmt"
+	"github.com/stretchr/testify/assert"
+	v1 "k8s.io/api/core/v1"
+	"kubesphere.io/devops/pkg/client/devops"
+	"kubesphere.io/devops/pkg/client/devops/fake"
+	"testing"
+)
+
+func TestNewProjectCredentialOperator(t *testing.T) {
+	operator := NewProjectCredentialOperator(nil)
+	assert.NotNil(t, operator)
+}
+
+func Test_projectCredentialGetter_GetProjectCredentialUsage(t *testing.T) {
+	type fields struct {
+		devopsClient devops.Interface
+	}
+	type args struct {
+		projectID    string
+		credentialID string
+	}
+	tests := []struct {
+		name    string
+		fields  fields
+		args    args
+		want    *devops.Credential
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "not found",
+		fields: fields{
+			devopsClient: &fake.Devops{},
+		},
+		args: args{
+			projectID:    "projectID",
+			credentialID: "credentialID",
+		},
+		want: nil,
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.NotNil(t, err)
+			return true
+		},
+	}, {
+		name: "normal",
+		fields: fields{
+			devopsClient: &fake.Devops{
+				Credentials: map[string]map[string]*v1.Secret{
+					"projectID": {
+						"credentialID": nil,
+					},
+				},
+			},
+		},
+		args: args{
+			projectID:    "projectID",
+			credentialID: "credentialID",
+		},
+		want: &devops.Credential{
+			Id: "credentialID",
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			o := &projectCredentialGetter{
+				devopsClient: tt.fields.devopsClient,
+			}
+			got, err := o.GetProjectCredentialUsage(tt.args.projectID, tt.args.credentialID)
+			if !tt.wantErr(t, err, fmt.Sprintf("GetProjectCredentialUsage(%v, %v)", tt.args.projectID, tt.args.credentialID)) {
+				return
+			}
+			assert.Equalf(t, tt.want, got, "GetProjectCredentialUsage(%v, %v)", tt.args.projectID, tt.args.credentialID)
+		})
+	}
+}
diff --git a/pkg/models/devops/project_pipeline_sonar_handler_test.go b/pkg/models/devops/project_pipeline_sonar_handler_test.go
new file mode 100644
index 0000000..212c271
--- /dev/null
+++ b/pkg/models/devops/project_pipeline_sonar_handler_test.go
@@ -0,0 +1,27 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package devops
+
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func TestNewPipelineSonarGetter(t *testing.T) {
+	getter := NewPipelineSonarGetter(nil, nil)
+	assert.NotNil(t, getter)
+}
diff --git a/pkg/models/devops/s2ibinary_handler_test.go b/pkg/models/devops/s2ibinary_handler_test.go
index 529d8d8..e7b064a 100644
--- a/pkg/models/devops/s2ibinary_handler_test.go
+++ b/pkg/models/devops/s2ibinary_handler_test.go
@@ -1,5 +1,5 @@
 /*
-Copyright 2020 The KubeSphere Authors.
+Copyright 2020,2022 The KubeSphere Authors.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -16,6 +16,11 @@ limitations under the License.
 
 package devops
 
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
 //
 //import (
 //	"fmt"
@@ -132,3 +137,8 @@ package devops
 //		Status: v1alpha1.S2iBinaryStatus{},
 //	}
 //}
+
+func TestNewS2iBinaryUploader(t *testing.T) {
+	uploader := NewS2iBinaryUploader(nil, nil, nil, nil)
+	assert.NotNil(t, uploader)
+}