Repository: kubesphere/ks-devops
Pull request #764 (2022-08-03 06:55:57)

Support to manage pod templates automatically
<!-- Thanks for sending a pull request! Here are some tips for you:

1. If you want **faster** PR reviews, read how: https://github.com/kubesphere/community/blob/master/developer-guide/development/the-pr-author-guide-to-getting-through-code-review.md
2. In case you want to know how your PR got reviewed, read: https://github.com/kubesphere/community/blob/master/developer-guide/development/code-review-guide.md
3. Here are some coding conventions followed by the KubeSphere community: https://github.com/kubesphere/community/blob/master/developer-guide/development/coding-conventions.md
4. Additional open-source best practice: https://github.com/LinuxSuRen/open-source-best-practice
-->

### What type of PR is this?
<!-- 
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind design
/kind chore

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->
/kind feature

### What this PR does / why we need it:

### Which issue(s) this PR fixes:
<!--
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
Please leave it or change # to be None if there is no corresponding issue that exists
-->
Fixes #

### Special notes for reviewers:
<!--
You can use the following command to let the DevOps SIG members help you to review your PR.
/cc @kubesphere/sig-devops 
And please avoid cc any individual.
-->
Please check the following list before waiting reviewers:

- [ ] Already committed the CRD files to [the Helm Chart](https://github.com/kubesphere-sigs/ks-devops-helm-chart/) if you created some new CRDs
- [ ] Already [added the permission](https://github.com/kubesphere/ks-installer/blob/9e063b085a0e43fdb3d0d9e3e7f4149146f14b9c/roles/ks-core/prepare/files/ks-init/role-templates.yaml) for the new API
- [ ] Already added the RBAC markers for the new controllers

### Does this PR introduce a user-facing change??
<!--
If no, just write "None" in the release-note block below.
If yes, a release note is required:
Enter your extended-release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

Please keep the note be same as your PR title if you believe it should be in the release notes.
-->
```release-note
Support manage Jenkins agent via PodTemplate
```


ks-ci-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please ask for approval from **linuxsuren** after the PR has been reviewed.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubesphere%2Fks-devops).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubesphere/ks-devops/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["linuxsuren"]} -->
codecov[bot]: # [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/764?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) Report
> Merging [#764](https://codecov.io/gh/kubesphere/ks-devops/pull/764?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (842de5c) into [master](https://codecov.io/gh/kubesphere/ks-devops/commit/7d384b242038a503e797da23999be78a08ec8bac?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (7d384b2) will **increase** coverage by `0.13%`.
> The diff coverage is `93.33%`.

> :exclamation: Current head 842de5c differs from pull request most recent head 4561558. Consider uploading reports for the commit 4561558 to get more accurate results

```diff
@@            Coverage Diff             @@
##           master     #764      +/-   ##
==========================================
+ Coverage   78.94%   79.08%   +0.13%     
==========================================
  Files         159      165       +6     
  Lines       23938    24019      +81     
==========================================
+ Hits        18899    18995      +96     
+ Misses       4774     4761      -13     
+ Partials      265      263       -2     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `79.08% <93.33%> (+0.13%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/kubesphere/ks-devops/pull/764?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) | Coverage Δ | |
|---|---|---|
| [pkg/utils/net/net.go](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL3V0aWxzL25ldC9uZXQuZ28=) | `100.00% <ø> (ø)` | |
| [controllers/predicate/label.go](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvcHJlZGljYXRlL2xhYmVsLmdv) | `75.00% <75.00%> (ø)` | |
| [pkg/utils/hashutil/MD5.go](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL3V0aWxzL2hhc2h1dGlsL01ENS5nbw==) | `90.00% <80.00%> (ø)` | |
| [...ntrollers/jenkins/config/podtemplate\_controller.go](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvamVua2lucy9jb25maWcvcG9kdGVtcGxhdGVfY29udHJvbGxlci5nbw==) | `95.23% <95.23%> (ø)` | |
| [...ers/argocd/argocd-application-status-controller.go](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvYXJnb2NkL2FyZ29jZC1hcHBsaWNhdGlvbi1zdGF0dXMtY29udHJvbGxlci5nbw==) | `95.38% <100.00%> (+5.67%)` | :arrow_up: |
| [controllers/jenkins/config/labels\_controller.go](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvamVua2lucy9jb25maWcvbGFiZWxzX2NvbnRyb2xsZXIuZ28=) | `88.15% <100.00%> (ø)` | |
| [pkg/utils/readerutils/MD5Reader.go](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL3V0aWxzL3JlYWRlcnV0aWxzL01ENVJlYWRlci5nbw==) | `100.00% <100.00%> (ø)` | |
| [pkg/utils/stringutils/string.go](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-cGtnL3V0aWxzL3N0cmluZ3V0aWxzL3N0cmluZy5nbw==) | `100.00% <100.00%> (ø)` | |
| ... and [5 more](https://codecov.io/gh/kubesphere/ks-devops/pull/764/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).


Patch:
diff --git a/.golangci.yml b/.golangci.yml
index ebecb43..a073e3d 100644
--- a/.golangci.yml
+++ b/.golangci.yml
@@ -95,7 +95,6 @@ run:
     - pkg/server/options/options.go
     - pkg/server/params/params.go
     - pkg/utils/idutils/id_utils.go
-    - pkg/utils/idutils/id_utils_test.go
     - pkg/client/sonarqube/options.go
     - pkg/client/sonarqube/sonarqube.go
     - pkg/client/cache/simple_cache.go
@@ -121,16 +120,12 @@ run:
     - pkg/utils/reflectutils/reflect.go
     - pkg/api/devops/v1alpha1/s2ibinary_type.go
     - pkg/client/devops/interface.go
-    - pkg/utils/stringutils/string.go
     - pkg/models/resources/v1alpha3/devops/devops.go
     - pkg/models/resources/v1alpha3/interface.go
     - pkg/apiserver/query/field.go
-    - pkg/utils/readerutils/MD5Reader.go
     - cmd/controller/app/options/options.go
     - cmd/tools/jwt/app/kubernetes.go
     - cmd/apiserver/app/options/options.go
-    - pkg/utils/net/net.go
-    - pkg/utils/hashutil/MD5.go
     - pkg/kapis/oauth/register.go
     - pkg/apiserver/authentication/authenticators/bearertoken/bearertoken.go
     - pkg/apiserver/authentication/request/anonymous/anonymous.go
diff --git a/cmd/controller/app/controllers.go b/cmd/controller/app/controllers.go
index 6710643..f2be746 100644
--- a/cmd/controller/app/controllers.go
+++ b/cmd/controller/app/controllers.go
@@ -34,48 +34,48 @@ import (
 	"kubesphere.io/devops/controllers/jenkins/pipelinerun"
 	"kubesphere.io/devops/pkg/client/devops"
 	"kubesphere.io/devops/pkg/client/k8s"
-	"kubesphere.io/devops/pkg/client/s3"
 	"kubesphere.io/devops/pkg/informers"
 	"sigs.k8s.io/controller-runtime/pkg/manager"
 )
 
 func addControllers(mgr manager.Manager, client k8s.Client, informerFactory informers.InformerFactory,
-	devopsClient devops.Interface, jenkinsCore core.JenkinsCore, s3Client s3.Interface,
-	s *options.DevOpsControllerManagerOptions, stopCh <-chan struct{}) error {
+	devopsClient devops.Interface, jenkinsCore core.JenkinsCore,
+	s *options.DevOpsControllerManagerOptions) error {
 	if devopsClient == nil {
 		return errors.New("devopsClient should not be nil")
 	}
 
-	tokenIssuer := token.NewTokenIssuer(s.JWTOptions.Secret, s.JWTOptions.MaximumClockSkew)
-	// add PipelineRun controller
-	if err := (&pipelinerun.Reconciler{
-		Client:       mgr.GetClient(),
-		Scheme:       mgr.GetScheme(),
-		DevOpsClient: devopsClient,
-		JenkinsCore:  jenkinsCore,
-		TokenIssuer:  tokenIssuer,
-	}).SetupWithManager(mgr); err != nil {
-		klog.Errorf("unable to create pipelinerun-controller, err: %v", err)
-		return err
-	}
+	reconcilers := getAllControllers(mgr, client, informerFactory, devopsClient, s, jenkinsCore)
+	reconcilers["pipeline"] = func(mgr manager.Manager) (err error) {
+		tokenIssuer := token.NewTokenIssuer(s.JWTOptions.Secret, s.JWTOptions.MaximumClockSkew)
+		// add PipelineRun controller
+		if err = (&pipelinerun.Reconciler{
+			Client:       mgr.GetClient(),
+			Scheme:       mgr.GetScheme(),
+			DevOpsClient: devopsClient,
+			JenkinsCore:  jenkinsCore,
+			TokenIssuer:  tokenIssuer,
+		}).SetupWithManager(mgr); err != nil {
+			klog.Errorf("unable to create pipelinerun-controller, err: %v", err)
+			return
+		}
 
-	// add PipelineRun Synchronizer
-	if err := (&pipelinerun.SyncReconciler{
-		Client:      mgr.GetClient(),
-		JenkinsCore: jenkinsCore,
-	}).SetupWithManager(mgr); err != nil {
-		klog.Errorf("unable to create pipelinerun-synchronizer, err: %v", err)
-		return err
-	}
+		// add PipelineRun Synchronizer
+		if err = (&pipelinerun.SyncReconciler{
+			Client:      mgr.GetClient(),
+			JenkinsCore: jenkinsCore,
+		}).SetupWithManager(mgr); err != nil {
+			klog.Errorf("unable to create pipelinerun-synchronizer, err: %v", err)
+			return
+		}
 
-	// add Pipeline metadata controller
-	if err := (&jenkinspipeline.Reconciler{
-		Client:      mgr.GetClient(),
-		JenkinsCore: jenkinsCore,
-	}).SetupWithManager(mgr); err != nil {
-		return err
+		// add Pipeline metadata controller
+		err = (&jenkinspipeline.Reconciler{
+			Client:      mgr.GetClient(),
+			JenkinsCore: jenkinsCore,
+		}).SetupWithManager(mgr)
+		return
 	}
-	reconcilers := getAllControllers(mgr, client, informerFactory, devopsClient, s, jenkinsCore)
 
 	// Add all controllers into manager.
 	for name, ok := range s.FeatureOptions.GetControllers() {
@@ -124,10 +124,14 @@ func getAllControllers(mgr manager.Manager, client k8s.Client, informerFactory i
 	tokenIssuer := token.NewTokenIssuer(s.JWTOptions.Secret, s.JWTOptions.MaximumClockSkew)
 	jenkinsAgentLabelsReconciler := config.AgentLabelsReconciler{
 		Client:          mgr.GetClient(),
-		TargetNamespace: "kubesphere-devops-system",
+		TargetNamespace: s.FeatureOptions.SystemNamespace,
 		TokenIssuer:     tokenIssuer,
 		JenkinsClient:   jenkinsCore,
 	}
+	jenkinsPodTemplate := config.PodTemplateReconciler{
+		Client:                   mgr.GetClient(),
+		TargetConfigMapNamespace: s.FeatureOptions.SystemNamespace,
+	}
 
 	return map[string]func(mgr manager.Manager) error{
 		gitRepoReconcilers.GetName(): func(mgr manager.Manager) error {
@@ -144,6 +148,9 @@ func getAllControllers(mgr manager.Manager, client k8s.Client, informerFactory i
 			}
 			return err
 		},
+		"jenkinsagent": func(mgr manager.Manager) error {
+			return jenkinsPodTemplate.SetupWithManager(mgr)
+		},
 		"jenkinsconfig": func(mgr manager.Manager) error {
 			return mgr.Add(config.NewController(&config.ControllerOptions{
 				LimitRangeClient:    client.Kubernetes().CoreV1(),
diff --git a/cmd/controller/app/options/feature.go b/cmd/controller/app/options/feature.go
index a2de3ac..cebb039 100644
--- a/cmd/controller/app/options/feature.go
+++ b/cmd/controller/app/options/feature.go
@@ -26,7 +26,8 @@ import (
 
 // FeatureOptions provide some feature options, such as specifying the controller to be enabled.
 type FeatureOptions struct {
-	Controllers map[string]bool
+	Controllers     map[string]bool
+	SystemNamespace string
 }
 
 // GetControllers returns the controllers map
@@ -35,7 +36,9 @@ func (o *FeatureOptions) GetControllers() map[string]bool {
 	defaultMap := map[string]bool{
 		"jenkins":       true,
 		"jenkinsconfig": true,
+		"jenkinsagent":  true,
 		"gitrepository": true,
+		"pipeline":      true,
 	}
 
 	// support to only enable the specific controllers
@@ -72,6 +75,8 @@ func (o *FeatureOptions) ApplyTo(options *FeatureOptions) {
 func (o *FeatureOptions) AddFlags(fs *pflag.FlagSet, c *FeatureOptions) {
 	fs.Var(cliflag.NewMapStringBool(&o.Controllers), "enabled-controllers", "A set of key=value pairs that describe feature options for controllers. "+
 		"Options are:\n"+strings.Join(c.knownControllers(), "\n"))
+	fs.StringVarP(&o.SystemNamespace, "system-namespace", "", "kubesphere-devops-system",
+		"The system namespace that contains ConfigMap, Secrets e.g.")
 }
 
 func (o *FeatureOptions) knownControllers() []string {
diff --git a/cmd/controller/app/server.go b/cmd/controller/app/server.go
index 61b32a5..efbb983 100644
--- a/cmd/controller/app/server.go
+++ b/cmd/controller/app/server.go
@@ -26,7 +26,6 @@ import (
 	"kubesphere.io/devops/pkg/client/devops"
 	"kubesphere.io/devops/pkg/client/devops/jclient"
 	"kubesphere.io/devops/pkg/client/k8s"
-	"kubesphere.io/devops/pkg/client/s3"
 	"kubesphere.io/devops/pkg/config"
 	"kubesphere.io/devops/pkg/indexers"
 	"kubesphere.io/devops/pkg/informers"
@@ -177,15 +176,6 @@ func Run(s *options.DevOpsControllerManagerOptions, stopCh <-chan struct{}) erro
 	apis.AddToScheme(mgr.GetScheme())
 	_ = apiextensions.AddToScheme(mgr.GetScheme())
 
-	// Init s3 client
-	var s3Client s3.Interface
-	if s.S3Options != nil && len(s.S3Options.Endpoint) != 0 {
-		s3Client, err = s3.NewS3Client(s.S3Options)
-		if err != nil {
-			return fmt.Errorf("failed to connect to s3, please check s3 service status, error: %v", err)
-		}
-	}
-
 	// register common meta types into schemas.
 	metav1.AddToGroupVersion(mgr.GetScheme(), metav1.SchemeGroupVersion)
 
@@ -194,13 +184,11 @@ func Run(s *options.DevOpsControllerManagerOptions, stopCh <-chan struct{}) erro
 		informerFactory,
 		devopsClient,
 		jenkinsCore,
-		s3Client,
-		s,
-		stopCh); err != nil {
+		s); err != nil {
 		return fmt.Errorf("unable to register controllers to the manager: %v", err)
 	}
 
-	if err := indexers.CreatePipelineRunSCMRefNameIndexer(mgr.GetCache()); err != nil {
+	if err = indexers.CreatePipelineRunSCMRefNameIndexer(mgr.GetCache()); err != nil {
 		return err
 	}
 
diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml
index c4f06c9..f376f95 100644
--- a/config/rbac/role.yaml
+++ b/config/rbac/role.yaml
@@ -32,6 +32,15 @@ rules:
   - list
   - update
   - watch
+- apiGroups:
+  - ""
+  resources:
+  - podtemplates
+  verbs:
+  - get
+  - list
+  - update
+  - watch
 - apiGroups:
   - ""
   resources:
diff --git a/controllers/argocd/argocd-application-status-controller.go b/controllers/argocd/argocd-application-status-controller.go
index 5d925d5..36403d3 100644
--- a/controllers/argocd/argocd-application-status-controller.go
+++ b/controllers/argocd/argocd-application-status-controller.go
@@ -21,17 +21,15 @@ import (
 	"encoding/json"
 	"fmt"
 	"github.com/go-logr/logr"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
-	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/controllers/predicate"
 	"kubesphere.io/devops/pkg/api/gitops/v1alpha1"
 	"reflect"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
-	"sigs.k8s.io/controller-runtime/pkg/predicate"
 	"strings"
 )
 
@@ -165,10 +163,7 @@ func (r *ApplicationStatusReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	argoApp := createBareArgoCDApplicationObject()
 	r.log = ctrl.Log.WithName(r.GetName())
 	r.recorder = mgr.GetEventRecorderFor(r.GetName())
-	var withLabelPredicate = predicate.NewPredicateFuncs(func(meta metav1.Object, object runtime.Object) (ok bool) {
-		_, ok = meta.GetLabels()[v1alpha1.ArgoCDAppControlByLabelKey]
-		return
-	})
+	var withLabelPredicate = predicate.NewPredicateFuncs(predicate.NewFilterHasLabel(v1alpha1.ArgoCDAppControlByLabelKey))
 	return ctrl.NewControllerManagedBy(mgr).
 		For(argoApp).
 		WithEventFilter(withLabelPredicate).
diff --git a/controllers/jenkins/config/constant.go b/controllers/jenkins/config/constant.go
index 4f33e15..d992d8b 100644
--- a/controllers/jenkins/config/constant.go
+++ b/controllers/jenkins/config/constant.go
@@ -31,3 +31,7 @@ const (
 	// FormulaLow is a formula name - low
 	FormulaLow = "low"
 )
+
+const reconcilerGroupName = "jenkins"
+
+const podTemplateFinalizer = "podtemplate.devops.kubesphere.io/finalizer"
diff --git a/controllers/jenkins/config/labels_controller.go b/controllers/jenkins/config/labels_controller.go
index e996caa..bbccb21 100644
--- a/controllers/jenkins/config/labels_controller.go
+++ b/controllers/jenkins/config/labels_controller.go
@@ -161,7 +161,7 @@ func (r *AgentLabelsReconciler) GetName() string {
 
 // GetGroupName returns the group name of this reconciler
 func (r *AgentLabelsReconciler) GetGroupName() string {
-	return "jenkins"
+	return reconcilerGroupName
 }
 
 func getSpecificConfigMapPredicate(name, namespace string) predicate.Funcs {
diff --git a/controllers/jenkins/config/podtemplate_controller.go b/controllers/jenkins/config/podtemplate_controller.go
new file mode 100644
index 0000000..b054f24
--- /dev/null
+++ b/controllers/jenkins/config/podtemplate_controller.go
@@ -0,0 +1,136 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package config
+
+import (
+	"context"
+	"github.com/go-logr/logr"
+	k8s "github.com/jenkins-zh/jenkins-client/pkg/k8s"
+	v1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/controllers/predicate"
+	"kubesphere.io/devops/pkg/utils/k8sutil"
+	"kubesphere.io/devops/pkg/utils/stringutils"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"strings"
+	"time"
+)
+
+//+kubebuilder:rbac:groups="",resources=configmaps,verbs=get;update
+//+kubebuilder:rbac:groups="",resources=podtemplates,verbs=get;list;watch;update
+
+// PodTemplateReconciler responsible for the Jenkins podTemplate sync
+type PodTemplateReconciler struct {
+	LabelSelector            string
+	TargetConfigMapName      string
+	TargetConfigMapNamespace string
+	TargetConfigMapKey       string
+	Interval                 time.Duration
+
+	client.Client
+	log      logr.Logger
+	recorder record.EventRecorder
+}
+
+// Reconcile is the entrypoint of this reconciler
+func (r *PodTemplateReconciler) Reconcile(req ctrl.Request) (result ctrl.Result, err error) {
+	r.log.Info("start to reconcile PodTemplate", "resource", req)
+	ctx := context.Background()
+
+	podTemplate := &v1.PodTemplate{}
+	if err = r.Get(ctx, req.NamespacedName, podTemplate); err != nil {
+		err = client.IgnoreNotFound(err)
+		return
+	}
+
+	// make sure all the PodTemplates have finalizer
+	if k8sutil.AddFinalizer(&podTemplate.ObjectMeta, podTemplateFinalizer) {
+		if err = r.Update(ctx, podTemplate); err != nil {
+			return
+		}
+	}
+
+	// get the Jenkins CasC data that we will manipulate
+	cm := &v1.ConfigMap{}
+	if err = r.Get(ctx, types.NamespacedName{
+		Namespace: r.TargetConfigMapNamespace,
+		Name:      r.TargetConfigMapName,
+	}, cm); err != nil {
+		// we will handle it only when the cm exists
+		err = client.IgnoreNotFound(err)
+		return
+	}
+	data := strings.TrimSpace(cm.Data[r.TargetConfigMapKey])
+	if data == "" {
+		r.log.V(7).Info("skip update cm due to expect key is empty", "resource", req)
+		return
+	}
+
+	casc := k8s.JenkinsConfig{
+		Config: []byte(data),
+	}
+
+	// manipulate the data
+	if podTemplate.DeletionTimestamp.IsZero() {
+		if err = casc.ReplaceOrAddPodTemplate(podTemplate); err == nil {
+			cm.Data[r.TargetConfigMapKey] = casc.GetConfigAsString()
+
+			// write back the data
+			err = r.Update(ctx, cm)
+		}
+	} else {
+		if err = casc.RemovePodTemplate(podTemplate.Name); err == nil {
+			cm.Data[r.TargetConfigMapKey] = casc.GetConfigAsString()
+			k8sutil.RemoveFinalizer(&podTemplate.ObjectMeta, podTemplateFinalizer)
+			if err = r.Update(ctx, podTemplate); err == nil {
+				// write back the data
+				err = r.Update(ctx, cm)
+			}
+		}
+	}
+
+	if err == nil {
+		// make sure the PodTemplates always could be in the Jenkins CasC
+		result = ctrl.Result{RequeueAfter: r.Interval}
+	}
+	return
+}
+
+// GetName returns the name of this reconcile
+func (r *PodTemplateReconciler) GetName() string {
+	return "pod-template"
+}
+
+// GetGroupName ret
+func (r *PodTemplateReconciler) GetGroupName() string {
+	return reconcilerGroupName
+}
+
+// SetupWithManager setups the reconciler
+func (r *PodTemplateReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	r.log = ctrl.Log.WithName(r.GetName())
+	r.recorder = mgr.GetEventRecorderFor(r.GetName())
+	r.LabelSelector = stringutils.SetOrDefault(r.LabelSelector, "jenkins.agent.pod")
+	r.TargetConfigMapName = stringutils.SetOrDefault(r.TargetConfigMapName, "jenkins-casc-config")
+	r.TargetConfigMapNamespace = stringutils.SetOrDefault(r.TargetConfigMapNamespace, "kubesphere-devops-system")
+	r.TargetConfigMapKey = stringutils.SetOrDefault(r.TargetConfigMapKey, "jenkins_user.yaml")
+	if r.Interval == 0 {
+		r.Interval = 5 * time.Minute
+	}
+
+	var withLabelPredicate = predicate.NewPredicateFuncs(predicate.NewFilterHasLabel(r.LabelSelector))
+	return ctrl.NewControllerManagedBy(mgr).WithEventFilter(withLabelPredicate).For(&v1.PodTemplate{}).Complete(r)
+}
diff --git a/controllers/predicate/label.go b/controllers/predicate/label.go
new file mode 100644
index 0000000..16d2d6c
--- /dev/null
+++ b/controllers/predicate/label.go
@@ -0,0 +1,37 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package predicate
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+
+	k8spredicate "sigs.k8s.io/controller-runtime/pkg/predicate"
+)
+
+// Filter is a reconciler filter function
+type Filter func(meta metav1.Object, object runtime.Object) (ok bool)
+
+// NewFilterHasLabel creats a filter that contains the specific label
+func NewFilterHasLabel(label string) Filter {
+	return func(meta metav1.Object, object runtime.Object) (ok bool) {
+		_, ok = meta.GetLabels()[label]
+		return
+	}
+}
+
+// NewPredicateFuncs creates a filter function
+func NewPredicateFuncs(filter Filter) k8spredicate.Funcs {
+	return k8spredicate.NewPredicateFuncs(filter)
+}
diff --git a/docs/pod-template.md b/docs/pod-template.md
new file mode 100644
index 0000000..e69de29
diff --git a/go.mod b/go.mod
index c86e285..92062ca 100644
--- a/go.mod
+++ b/go.mod
@@ -24,7 +24,8 @@ require (
 	github.com/google/go-querystring v1.1.0 // indirect
 	github.com/h2non/gock v1.0.9
 	github.com/jenkins-x/go-scm v1.11.4
-	github.com/jenkins-zh/jenkins-client v0.0.11
+	github.com/jenkins-zh/jenkins-client v0.0.12
+	github.com/jenkins-zh/jenkins-client/pkg/k8s v0.0.0-20220803063034-4f2584fd4a90
 	github.com/kubesphere/sonargo v0.0.2
 	github.com/onsi/ginkgo v1.16.4
 	github.com/onsi/gomega v1.15.0
diff --git a/go.sum b/go.sum
index e783523..27b6a14 100644
--- a/go.sum
+++ b/go.sum
@@ -339,8 +339,10 @@ github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NH
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
 github.com/jenkins-zh/jenkins-cli v0.0.32/go.mod h1:uE1mH9PNITrg0sugv6HXuM/CSddg0zxXoYu3w57I3JY=
-github.com/jenkins-zh/jenkins-client v0.0.11 h1:g7f4WhI4iA6JA6l3fW5E/vSsBNYYr3/Ey+wwLxPTKeQ=
-github.com/jenkins-zh/jenkins-client v0.0.11/go.mod h1:ICBk7OOoTafVP//f/VfKZ34c0ff8vJwVnOsF9btiMYU=
+github.com/jenkins-zh/jenkins-client v0.0.12 h1:rtFhug88U2TSedgX6gUFkODid/E6qnfJxj7OwjLjBU4=
+github.com/jenkins-zh/jenkins-client v0.0.12/go.mod h1:ICBk7OOoTafVP//f/VfKZ34c0ff8vJwVnOsF9btiMYU=
+github.com/jenkins-zh/jenkins-client/pkg/k8s v0.0.0-20220803063034-4f2584fd4a90 h1:QMUbSp4/FSA0rGcWu85wOJ4FHQHFbxNpQArgdE90N+k=
+github.com/jenkins-zh/jenkins-client/pkg/k8s v0.0.0-20220803063034-4f2584fd4a90/go.mod h1:zetE8JfNmxkot7OCNXJyS/QwxFnfcUFzrs/jg4U2gAw=
 github.com/jenkins-zh/jenkins-formulas v0.0.5/go.mod h1:zS8fm8u5L6FcjZM0QznXsLV9T2UtSVK+hT6Sm76iUZ4=
 github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
 github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4=
@@ -393,6 +395,8 @@ github.com/linuxsuren/http-downloader v0.0.2-0.20201207132639-19888a6beaec/go.mo
 github.com/linuxsuren/http-downloader v0.0.6/go.mod h1:xxgh2OE7WGL9TwDE9L8Gh7Lqq9fFPuHbh5tofUitEfE=
 github.com/linuxsuren/http-downloader v0.0.29 h1:GJUoybR4SViarPfX2xipq0unUssiuB9qrSSoDDW1jao=
 github.com/linuxsuren/http-downloader v0.0.29/go.mod h1:bniEqpLyyydtCTDyjo3nco6LD2F90ruPH87fuqAznQg=
+github.com/linuxsuren/unstructured v0.0.1 h1:ilUA8MUYbR6l9ebo/YPV2bKqlf62bzQursDSE+j00iU=
+github.com/linuxsuren/unstructured v0.0.1/go.mod h1:KH6aTj+FegzGBzc1vS6mzZx3/duhTUTEVyW5sO7p4as=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/magiconair/properties v1.8.5 h1:b6kJs+EmPFMYGkow9GiUyCyOvIwYetYJ3fSaWak/Gls=
diff --git a/pkg/utils/hashutil/MD5.go b/pkg/utils/hashutil/MD5.go
index 4b401d1..651a792 100644
--- a/pkg/utils/hashutil/MD5.go
+++ b/pkg/utils/hashutil/MD5.go
@@ -1,5 +1,5 @@
 /*
-Copyright 2019 The KubeSphere Authors.
+Copyright 2019-2022 The KubeSphere Authors.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -21,28 +21,24 @@ import (
 	"io"
 
 	"code.cloudfoundry.org/bytefmt"
-	"k8s.io/klog"
-
 	"kubesphere.io/devops/pkg/utils/readerutils"
 )
 
-func GetMD5(reader io.ReadCloser) (string, error) {
+// GetMD5 returns the md5 value from a reader
+func GetMD5(reader io.ReadCloser) (result string, err error) {
 	md5reader := readerutils.NewMD5Reader(reader)
 	data := make([]byte, bytefmt.KILOBYTE)
 	for {
-		_, err := md5reader.Read(data)
+		_, err = md5reader.Read(data)
 		if err != nil {
 			if err == io.EOF {
 				break
 			}
-			klog.Error(err)
-			return "", err
+			return
 		}
 	}
-	err := reader.Close()
-	if err != nil {
-		return "", err
+	if err = reader.Close(); err == nil {
+		result = hex.EncodeToString(md5reader.MD5())
 	}
-
-	return hex.EncodeToString(md5reader.MD5()), nil
+	return
 }
diff --git a/pkg/utils/net/net.go b/pkg/utils/net/net.go
index 2b67338..a6da555 100644
--- a/pkg/utils/net/net.go
+++ b/pkg/utils/net/net.go
@@ -1,5 +1,5 @@
 /*
-Copyright 2019 The KubeSphere Authors.
+Copyright 2019-2022 The KubeSphere Authors.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -23,11 +23,12 @@ import (
 )
 
 // IsValidPort check if given number is an valid port
-// 0 is considered as an inalid port
+// 0 is considered as an invalid port
 func IsValidPort(port int) bool {
 	return port > 0 && port < 65535
 }
 
+// GetRequestIP returns the real ip from a HTTP request
 func GetRequestIP(req *http.Request) string {
 	address := strings.Trim(req.Header.Get("X-Real-Ip"), " ")
 	if address != "" {
diff --git a/pkg/utils/readerutils/MD5Reader.go b/pkg/utils/readerutils/MD5Reader.go
index 281ea13..68aea4d 100644
--- a/pkg/utils/readerutils/MD5Reader.go
+++ b/pkg/utils/readerutils/MD5Reader.go
@@ -1,5 +1,5 @@
 /*
-Copyright 2019 The KubeSphere Authors.
+Copyright 2019-2022 The KubeSphere Authors.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -22,23 +22,27 @@ import (
 	"io"
 )
 
+// MD5Reader is a reader to read the dm5
 type MD5Reader struct {
 	md5  hash.Hash
 	body io.Reader
 }
 
-func (reader *MD5Reader) Read(b []byte) (int, error) {
-	n, err := reader.body.Read(b)
-	if err != nil {
-		return n, err
+// Read reads the data
+func (reader *MD5Reader) Read(b []byte) (n int, err error) {
+	n, err = reader.body.Read(b)
+	if err == nil {
+		n, err = reader.md5.Write(b[:n])
 	}
-	return reader.md5.Write(b[:n])
+	return
 }
 
+// MD5 returns the data
 func (reader *MD5Reader) MD5() []byte {
 	return reader.md5.Sum(nil)
 }
 
+// NewMD5Reader creates a md5 reader
 func NewMD5Reader(reader io.Reader) *MD5Reader {
 	return &MD5Reader{
 		md5:  md5.New(),
diff --git a/pkg/utils/stringutils/string.go b/pkg/utils/stringutils/string.go
index ad7445d..c15c790 100644
--- a/pkg/utils/stringutils/string.go
+++ b/pkg/utils/stringutils/string.go
@@ -1,5 +1,5 @@
 /*
-Copyright 2018 The KubeSphere Authors.
+Copyright 2018-2022 The KubeSphere Authors.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -17,63 +17,10 @@ limitations under the License.
 package stringutils
 
 import (
-	"regexp"
-	"strings"
 	"unicode/utf8"
-
-	"github.com/asaskevich/govalidator"
 )
 
-const ansi = "[\u001B\u009B][[\\]()#;?]*(?:(?:(?:[a-zA-Z\\d]*(?:;[a-zA-Z\\d]*)*)?\u0007)|(?:(?:\\d{1,4}(?:;\\d{0,4})*)?[\\dA-PRZcf-ntqry=><~]))"
-
-var re = regexp.MustCompile(ansi)
-
-// Diff creates an slice of slice values not included in the other given slice.
-func Diff(base, exclude []string) (result []string) {
-	excludeMap := make(map[string]bool)
-	for _, s := range exclude {
-		excludeMap[s] = true
-	}
-	for _, s := range base {
-		if !excludeMap[s] {
-			result = append(result, s)
-		}
-	}
-	return result
-}
-
-func Unique(ss []string) (result []string) {
-	smap := make(map[string]bool)
-	for _, s := range ss {
-		smap[s] = true
-	}
-	for s := range smap {
-		result = append(result, s)
-	}
-	return result
-}
-
-func CamelCaseToUnderscore(str string) string {
-	return govalidator.CamelCaseToUnderscore(str)
-}
-
-func UnderscoreToCamelCase(str string) string {
-	return govalidator.UnderscoreToCamelCase(str)
-}
-
-func FindString(array []string, str string) int {
-	for index, s := range array {
-		if str == s {
-			return index
-		}
-	}
-	return -1
-}
-
-func StringIn(str string, array []string) bool {
-	return FindString(array, str) > -1
-}
-
+// Reverse makes the string reversed
 func Reverse(s string) string {
 	size := len(s)
 	buf := make([]byte, size)
@@ -85,20 +32,10 @@ func Reverse(s string) string {
 	return string(buf)
 }
 
-func Split(str string, sep string) []string {
-	if str == "" {
-		return nil
-	}
-	return strings.Split(str, sep)
-}
-
-func StripAnsi(str string) string {
-	return re.ReplaceAllString(str, "")
-}
-
-func ShortenString(str string, n int) string {
-	if len(str) <= n {
-		return str
+// SetOrDefault uses a default value or the original
+func SetOrDefault(val string, defVal string) string {
+	if val == "" {
+		return defVal
 	}
-	return str[:n]
+	return val
 }


Test patch:
diff --git a/cmd/controller/app/options/feature_test.go b/cmd/controller/app/options/feature_test.go
index 2e0cc5c..9f7e089 100644
--- a/cmd/controller/app/options/feature_test.go
+++ b/cmd/controller/app/options/feature_test.go
@@ -37,7 +37,9 @@ func TestFeatureOptions_GetControllers(t *testing.T) {
 		want: map[string]bool{
 			"jenkins":       true,
 			"jenkinsconfig": true,
+			"jenkinsagent":  true,
 			"gitrepository": true,
+			"pipeline":      true,
 		},
 	}, {
 		name: "no input (be nil) from users",
@@ -47,7 +49,9 @@ func TestFeatureOptions_GetControllers(t *testing.T) {
 		want: map[string]bool{
 			"jenkins":       true,
 			"jenkinsconfig": true,
+			"jenkinsagent":  true,
 			"gitrepository": true,
+			"pipeline":      true,
 		},
 	}, {
 		name: "merge with the input from users",
@@ -59,7 +63,9 @@ func TestFeatureOptions_GetControllers(t *testing.T) {
 		want: map[string]bool{
 			"jenkins":       true,
 			"jenkinsconfig": true,
+			"jenkinsagent":  true,
 			"gitrepository": true,
+			"pipeline":      true,
 			"fake":          true,
 		},
 	}, {
diff --git a/controllers/jenkins/config/interface_test.go b/controllers/jenkins/config/interface_test.go
index fc63259..6034d59 100644
--- a/controllers/jenkins/config/interface_test.go
+++ b/controllers/jenkins/config/interface_test.go
@@ -37,6 +37,12 @@ func TestInterfaceImplement(t *testing.T) {
 			NamedReconciler: &AgentLabelsReconciler{},
 			GroupReconciler: &AgentLabelsReconciler{},
 		},
+	}, {
+		name: "PodTemplateReconciler",
+		instance: interInstance{
+			NamedReconciler: &PodTemplateReconciler{},
+			GroupReconciler: &PodTemplateReconciler{},
+		},
 	}}
 	for i := range tests {
 		tt := tests[i]
diff --git a/controllers/jenkins/config/podtemplate_controller_test.go b/controllers/jenkins/config/podtemplate_controller_test.go
new file mode 100644
index 0000000..f26c6ac
--- /dev/null
+++ b/controllers/jenkins/config/podtemplate_controller_test.go
@@ -0,0 +1,199 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package config
+
+import (
+	"context"
+	"fmt"
+	"github.com/stretchr/testify/assert"
+	"io/ioutil"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	mgrcore "kubesphere.io/devops/controllers/core"
+	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
+	controllerruntime "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/controller-runtime/pkg/log"
+	"testing"
+	"time"
+)
+
+func TestPodTemplateReconciler_SetupWithManager(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	tests := []struct {
+		name    string
+		wantErr assert.ErrorAssertionFunc
+	}{{
+		name: "normal",
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &PodTemplateReconciler{}
+			mgr := &mgrcore.FakeManager{
+				Scheme: schema,
+			}
+			tt.wantErr(t, r.SetupWithManager(mgr), fmt.Sprintf("SetupWithManager(%v)", mgr))
+			assert.NotEmpty(t, r.LabelSelector)
+			assert.NotEmpty(t, r.TargetConfigMapName)
+			assert.NotEmpty(t, r.TargetConfigMapNamespace)
+			assert.NotEmpty(t, r.TargetConfigMapKey)
+			assert.Equal(t, r.Interval, 5*time.Minute)
+		})
+	}
+}
+
+func TestPodTemplateReconciler_Reconcile(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	req := controllerruntime.Request{
+		NamespacedName: types.NamespacedName{
+			Namespace: "ns",
+			Name:      "pod-template",
+		},
+	}
+
+	now := metav1.Now()
+	podT := v1.PodTemplate{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: req.Namespace,
+			Name:      req.Name,
+		},
+	}
+	deletingPodT := podT.DeepCopy()
+	deletingPodT.DeletionTimestamp = &now
+
+	cascData, err := ioutil.ReadFile("testdata/casc.yaml")
+	assert.Nil(t, err)
+
+	cm := v1.ConfigMap{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: "kubesphere-devops-system",
+			Name:      "jenkins-casc-config",
+		},
+		Data: map[string]string{
+			"jenkins_user.yaml": string(cascData),
+		},
+	}
+	cmWithoutKey := cm.DeepCopy()
+	cmWithoutKey.Data = map[string]string{}
+
+	type fields struct {
+		Client client.Client
+	}
+	type args struct {
+		req controllerruntime.Request
+	}
+	tests := []struct {
+		name       string
+		fields     fields
+		args       args
+		wantResult controllerruntime.Result
+		wantErr    assert.ErrorAssertionFunc
+		verify     func(*testing.T, client.Client)
+	}{{
+		name:   "not found podtemplate",
+		fields: fields{Client: fake.NewFakeClientWithScheme(schema)},
+		args:   args{req: req},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+	}, {
+		name:   "no related ConfigMap exist",
+		fields: fields{Client: fake.NewFakeClientWithScheme(schema, podT.DeepCopy())},
+		args:   args{req: req},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+	}, {
+		name:   "no expect key in specific ConfigMap",
+		fields: fields{Client: fake.NewFakeClientWithScheme(schema, podT.DeepCopy(), cmWithoutKey.DeepCopy())},
+		args:   args{req: req},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+	}, {
+		name:   "normal case",
+		fields: fields{Client: fake.NewFakeClientWithScheme(schema, podT.DeepCopy(), cm.DeepCopy())},
+		args:   args{req: req},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+		wantResult: controllerruntime.Result{RequeueAfter: 5 * time.Minute},
+		verify: func(t *testing.T, c client.Client) {
+			podT := v1.PodTemplate{}
+			err := c.Get(context.Background(), types.NamespacedName{
+				Namespace: "ns",
+				Name:      "pod-template",
+			}, &podT)
+			assert.Nil(t, err)
+			assert.NotEmpty(t, podT.Finalizers)
+		},
+	}, {
+		name:   "handle a deleting podTemplate",
+		fields: fields{Client: fake.NewFakeClientWithScheme(schema, deletingPodT.DeepCopy(), cm.DeepCopy())},
+		args:   args{req: req},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			assert.Nil(t, err)
+			return true
+		},
+		wantResult: controllerruntime.Result{RequeueAfter: 5 * time.Minute},
+		verify: func(t *testing.T, c client.Client) {
+			podT := v1.PodTemplate{}
+			err := c.Get(context.Background(), types.NamespacedName{
+				Namespace: "ns",
+				Name:      "pod-template",
+			}, &podT)
+			assert.Nil(t, err)
+			assert.Empty(t, podT.Finalizers)
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &PodTemplateReconciler{
+				Client: tt.fields.Client,
+				log:    log.NullLogger{},
+			}
+			mgr := &mgrcore.FakeManager{
+				Scheme: schema,
+			}
+			err = r.SetupWithManager(mgr)
+			assert.Nil(t, err)
+			gotResult, err := r.Reconcile(tt.args.req)
+			if !tt.wantErr(t, err, fmt.Sprintf("Reconcile(%v)", tt.args.req)) {
+				return
+			}
+			assert.Equalf(t, tt.wantResult, gotResult, "Reconcile(%v)", tt.args.req)
+			if tt.verify != nil {
+				tt.verify(t, tt.fields.Client)
+			}
+		})
+	}
+}
diff --git a/controllers/jenkins/config/testdata/casc.yaml b/controllers/jenkins/config/testdata/casc.yaml
new file mode 100644
index 0000000..0cefe83
--- /dev/null
+++ b/controllers/jenkins/config/testdata/casc.yaml
@@ -0,0 +1,86 @@
+jenkins:
+  clouds:
+    - kubernetes:
+        connectTimeout: "60"
+        containerCapStr: "4"
+        credentialsId: k8s-service-account
+        jenkinsTunnel: devops-jenkins-agent.kubesphere-devops-system:50000
+        jenkinsUrl: http://devops-jenkins.kubesphere-devops-system:80
+        maxRequestsPerHostStr: "32"
+        name: kubernetes
+        namespace: kubesphere-devops-worker
+        readTimeout: "60"
+        serverUrl: https://kubernetes.default
+        skipTlsVerify: true
+        templates:
+          - containers:
+              - args: ""
+                command: cat
+                image: kubesphere/builder-go:v3.2.0
+                name: go
+                privileged: false
+                resourceLimitCpu: 4000m
+                resourceLimitMemory: 8192Mi
+                resourceRequestCpu: 100m
+                resourceRequestMemory: 100Mi
+                ttyEnabled: true
+              - args: ^${computer.jnlpmac} ^${computer.name}
+                command: jenkins-slave
+                image: jenkins/jnlp-slave:3.27-1
+                name: jnlp
+                resourceLimitCpu: 500m
+                resourceLimitMemory: 1536Mi
+                resourceRequestCpu: 50m
+                resourceRequestMemory: 400Mi
+            idleMinutes: 0
+            label: go
+            name: go
+            namespace: kubesphere-devops-worker
+            nodeUsageMode: EXCLUSIVE
+            volumes:
+              - hostPathVolume:
+                  hostPath: /var/run/docker.sock
+                  mountPath: /var/run/docker.sock
+              - hostPathVolume:
+                  hostPath: /var/data/jenkins_go_cache
+                  mountPath: /home/jenkins/go/pkg
+              - hostPathVolume:
+                  hostPath: /var/data/jenkins_sonar_cache
+                  mountPath: /root/.sonar/cache
+            workspaceVolume:
+              emptyDirWorkspaceVolume:
+                memory: false
+            yaml: ""
+  disableRememberMe: true
+  mode: EXCLUSIVE
+  numExecutors: 0
+  scmCheckoutRetryCount: 2
+  securityRealm:
+    ldap:
+      configurations:
+        - displayNameAttributeName: uid
+          groupSearchBase: ou=Groups
+          groupSearchFilter: (&(objectClass=posixGroup)(cn={0}))
+          inhibitInferRootDN: false
+          mailAddressAttributeName: mail
+          managerDN: cn=admin,dc=kubesphere,dc=io
+          managerPasswordSecret: admin
+          rootDN: dc=kubesphere,dc=io
+          server: ldap://openldap.kubesphere-system.svc:389
+          userSearch: (&(objectClass=inetOrgPerson)(|(uid={0})(mail={0})))
+          userSearchBase: ou=Users
+      disableMailAddressResolver: false
+      disableRolePrefixing: true
+unclassified:
+  gitLabServers:
+    servers:
+      - name: https://gitlab.com
+        serverUrl: https://gitlab.com
+  kubespheretokenauthglobalconfiguration:
+    cacheConfiguration:
+      size: 20
+      ttl: 300
+    enabled: true
+    server: http://devops-apiserver.kubesphere-devops-system:9090/
+  location:
+    url: jenkins.devops.kubesphere.local
diff --git a/controllers/predicate/label_test.go b/controllers/predicate/label_test.go
new file mode 100644
index 0000000..26a2253
--- /dev/null
+++ b/controllers/predicate/label_test.go
@@ -0,0 +1,35 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package predicate
+
+import (
+	"github.com/stretchr/testify/assert"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"testing"
+)
+
+func TestNewFilterHasLabel(t *testing.T) {
+	filter := NewFilterHasLabel("fake")
+	assert.NotNil(t, filter)
+
+	ok := filter(&v1.ConfigMap{ObjectMeta: metav1.ObjectMeta{
+		Labels: map[string]string{
+			"fake": "good",
+		},
+	}}, nil)
+	assert.True(t, ok)
+	ok = filter(&v1.ConfigMap{}, nil)
+	assert.False(t, ok)
+}
diff --git a/pkg/utils/hashutil/MD5_test.go b/pkg/utils/hashutil/MD5_test.go
new file mode 100644
index 0000000..10e2a4d
--- /dev/null
+++ b/pkg/utils/hashutil/MD5_test.go
@@ -0,0 +1,50 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package hashutil
+
+import (
+	"bytes"
+	"io"
+	"io/ioutil"
+	"testing"
+)
+
+func TestGetMD5(t *testing.T) {
+	type args struct {
+		reader io.ReadCloser
+	}
+	tests := []struct {
+		name    string
+		args    args
+		want    string
+		wantErr bool
+	}{{
+		name:    "normal",
+		args:    args{reader: ioutil.NopCloser(bytes.NewBufferString("abc"))},
+		want:    "900150983cd24fb0d6963f7d28e17f72",
+		wantErr: false,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			got, err := GetMD5(tt.args.reader)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("GetMD5() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			if got != tt.want {
+				t.Errorf("GetMD5() got = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}
diff --git a/pkg/utils/idutils/id_utils_test.go b/pkg/utils/idutils/id_utils_test.go
index 80f6b86..fc2c4a9 100644
--- a/pkg/utils/idutils/id_utils_test.go
+++ b/pkg/utils/idutils/id_utils_test.go
@@ -33,8 +33,8 @@ func TestGetUuid36(t *testing.T) {
 func TestGetManyUuid(t *testing.T) {
 	var strSlice []string
 	for i := 0; i < 10000; i++ {
-		testId := GetUuid("")
-		strSlice = append(strSlice, testId)
+		testID := GetUuid("")
+		strSlice = append(strSlice, testID)
 	}
 	sort.Strings(strSlice)
 }
diff --git a/pkg/utils/net/net_test.go b/pkg/utils/net/net_test.go
new file mode 100644
index 0000000..8a8875e
--- /dev/null
+++ b/pkg/utils/net/net_test.go
@@ -0,0 +1,98 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package net
+
+import (
+	"net/http"
+	"testing"
+)
+
+func TestIsValidPort(t *testing.T) {
+	type args struct {
+		port int
+	}
+	tests := []struct {
+		name string
+		args args
+		want bool
+	}{{
+		name: "valid port",
+		args: args{port: 22},
+		want: true,
+	}, {
+		name: "negative port",
+		args: args{port: -1},
+		want: false,
+	}, {
+		name: "very big port number",
+		args: args{port: 65536},
+		want: false,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := IsValidPort(tt.args.port); got != tt.want {
+				t.Errorf("IsValidPort() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestGetRequestIP(t *testing.T) {
+	type args struct {
+		req *http.Request
+	}
+	tests := []struct {
+		name string
+		args args
+		want string
+	}{{
+		name: "have X-Real-Ip",
+		args: args{req: &http.Request{Header: map[string][]string{
+			"X-Real-Ip":       {"0.0.0.0"},
+			"X-Forwarded-For": {"1.1.1.1"},
+		}}},
+		want: "0.0.0.0",
+	}, {
+		name: "have X-Real-Ip",
+		args: args{req: &http.Request{
+			Header: map[string][]string{
+				"X-Forwarded-For": {"0.0.0.0"},
+			},
+			RemoteAddr: "2.2.2.2:22",
+		}},
+		want: "0.0.0.0",
+	}, {
+		name: "get ip from remote-address",
+		args: args{req: &http.Request{
+			RemoteAddr: "2.2.2.2:22",
+		}},
+		want: "2.2.2.2",
+	}, {
+		name: "get ip from remote-address that does not have the port",
+		args: args{req: &http.Request{
+			RemoteAddr: "2.2.2.2",
+		}},
+		want: "2.2.2.2",
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := GetRequestIP(tt.args.req); got != tt.want {
+				t.Errorf("GetRequestIP() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}
diff --git a/pkg/utils/readerutils/MD5Reader_test.go b/pkg/utils/readerutils/MD5Reader_test.go
new file mode 100644
index 0000000..bfdfb95
--- /dev/null
+++ b/pkg/utils/readerutils/MD5Reader_test.go
@@ -0,0 +1,30 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package readerutils
+
+import (
+	"bytes"
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func TestNewMD5Reader(t *testing.T) {
+	buf := bytes.NewBufferString("abc")
+	reader := NewMD5Reader(buf)
+	assert.NotNil(t, reader)
+	assert.Equal(t, "\xd4\x1d\x8cُ\x00\xb2\x04\xe9\x80\t\x98\xec\xf8B~", string(reader.MD5()))
+	count, err := reader.Read([]byte("abc"))
+	assert.Equal(t, 3, count)
+	assert.Nil(t, err)
+}
diff --git a/pkg/utils/stringutils/string_test.go b/pkg/utils/stringutils/string_test.go
new file mode 100644
index 0000000..a5d2690
--- /dev/null
+++ b/pkg/utils/stringutils/string_test.go
@@ -0,0 +1,71 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package stringutils
+
+import "testing"
+
+func TestSetOrDefault(t *testing.T) {
+	type args struct {
+		val    string
+		defVal string
+	}
+	tests := []struct {
+		name string
+		args args
+		want string
+	}{{
+		name: "empty string",
+		args: args{
+			val:    "",
+			defVal: "abc",
+		},
+		want: "abc",
+	}, {
+		name: "not empty string",
+		args: args{
+			val:    "def",
+			defVal: "abc",
+		},
+		want: "def",
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := SetOrDefault(tt.args.val, tt.args.defVal); got != tt.want {
+				t.Errorf("SetOrDefault() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestReverse(t *testing.T) {
+	type args struct {
+		s string
+	}
+	tests := []struct {
+		name string
+		args args
+		want string
+	}{{
+		name: "normal",
+		args: args{s: "abcd"},
+		want: "dcba",
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := Reverse(tt.args.s); got != tt.want {
+				t.Errorf("Reverse() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}