Repository: kubesphere/ks-devops
Pull request #469 (2022-02-23 01:10:10)

Support sync kubesphere multi-cluster to argo multi-cluster
<!-- Thanks for sending a pull request! Here are some tips for you:

1. If you want **faster** PR reviews, read how: https://github.com/kubesphere/community/blob/master/developer-guide/development/the-pr-author-guide-to-getting-through-code-review.md
2. In case you want to know how your PR got reviewed, read: https://github.com/kubesphere/community/blob/master/developer-guide/development/code-review-guide.md
3. Here are some coding conventions followed by the KubeSphere community: https://github.com/kubesphere/community/blob/master/developer-guide/development/coding-conventions.md
4. Additional open-source best practice: https://github.com/LinuxSuRen/open-source-best-practice
-->

### What type of PR is this?
<!-- 
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design
/kind chore

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

### What this PR does / why we need it:
/kind feature

this is part of #63 

With help from this PR, users don't need to add the [ArgoCD cluster](https://argo-cd.readthedocs.io/en/stable/user-guide/commands/argocd_cluster/) manually. They just need to follow [this document](https://kubesphere.com.cn/docs/multicluster-management/introduction/overview/) to add a member cluster, the controller will add the corresponding ArgoCD cluster.

### Which issue(s) this PR fixes:
<!--
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
Please leave it or change # to be None if there is no corresponding issue that exists
-->
Fixes #

### Special notes for reviewers:
<!--
You can use the following command to let the DevOps SIG members help you to review your PR.
/cc @kubesphere/sig-devops 
And please avoid cc any individual.
-->
You can use the following image if you want to help to test this PR.
```
surenpi/devops-controller:dev-v3.2.1-50bb80b
```

### Does this PR introduce a user-facing change??
<!--
If no, just write "None" in the release-note block below.
If yes, a release note is required:
Enter your extended-release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

Please keep the note be same as your PR title if you believe it should be in the release notes.
-->
```release-note
None
```


ks-ci-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubesphere/ks-devops/pull/469#" title="Author self-approved">LinuxSuRen</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubesphere%2Fks-devops).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubesphere/ks-devops/blob/master/OWNERS)~~ [LinuxSuRen]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
codecov[bot]: # [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/469?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) Report
> Merging [#469](https://codecov.io/gh/kubesphere/ks-devops/pull/469?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (50bb80b) into [master](https://codecov.io/gh/kubesphere/ks-devops/commit/12018e7d66089999bbd016cb1a3cd380ab4eeda0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) (12018e7) will **increase** coverage by `0.02%`.
> The diff coverage is `88.00%`.

[![Impacted file tree graph](https://codecov.io/gh/kubesphere/ks-devops/pull/469/graphs/tree.svg?width=650&height=150&src=pr&token=qSK4RBbyNg&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere)](https://codecov.io/gh/kubesphere/ks-devops/pull/469?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere)

```diff
@@            Coverage Diff             @@
##           master     #469      +/-   ##
==========================================
+ Coverage   76.89%   76.92%   +0.02%     
==========================================
  Files         130      132       +2     
  Lines       22556    22656     +100     
==========================================
+ Hits        17345    17428      +83     
- Misses       5024     5039      +15     
- Partials      187      189       +2     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `76.92% <88.00%> (+0.02%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/kubesphere/ks-devops/pull/469?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere) | Coverage Δ | |
|---|---|---|
| [controllers/argocd/multi-cluster-controller.go](https://codecov.io/gh/kubesphere/ks-devops/pull/469/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvYXJnb2NkL211bHRpLWNsdXN0ZXItY29udHJvbGxlci5nbw==) | `87.50% <87.50%> (ø)` | |
| [controllers/argocd/types.go](https://codecov.io/gh/kubesphere/ks-devops/pull/469/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvYXJnb2NkL3R5cGVzLmdv) | `100.00% <100.00%> (ø)` | |
| [...ns/devopscredential/devopscredential\_controller.go](https://codecov.io/gh/kubesphere/ks-devops/pull/469/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere#diff-Y29udHJvbGxlcnMvamVua2lucy9kZXZvcHNjcmVkZW50aWFsL2Rldm9wc2NyZWRlbnRpYWxfY29udHJvbGxlci5nbw==) | `33.33% <0.00%> (-3.41%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/469?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kubesphere/ks-devops/pull/469?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere). Last update [12018e7...50bb80b](https://codecov.io/gh/kubesphere/ks-devops/pull/469?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubesphere).


Patch:
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 83399ac..5db1834 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -40,8 +40,11 @@ You can use the following command to let the DevOps SIG members help you to revi
 /cc @kubesphere/sig-devops 
 And please avoid cc any individual.
 -->
-```
-```
+Please check the following list before waiting reviewers:
+
+- [ ] Already committed the CRD files to [the Helm Chart](https://github.com/kubesphere-sigs/ks-devops-helm-chart/) if you created some new CRDs
+- [ ] Already [added the permission](https://github.com/kubesphere/ks-installer/blob/9e063b085a0e43fdb3d0d9e3e7f4149146f14b9c/roles/ks-core/prepare/files/ks-init/role-templates.yaml) for the new API
+- [ ] Already added the RBAC markers for the new controllers
 
 ### Does this PR introduce a user-facing change??
 <!--
diff --git a/README.md b/README.md
index 6daa851..a8e485b 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,6 @@
 [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/kubesphere/ks-devops)
 [![](https://goreportcard.com/badge/kubesphere/ks-devops)](https://goreportcard.com/report/kubesphere/ks-devops)
+<a href="https://github.com/search?q=user%3Akubesphere-sigs+label%3A%22good+first+issue%22+state%3Aopen+repo%3Akubesphere%2Fks-devops+repo%3Akubesphere-sigs%2Fks-devops-helm-chart&type=Issues&ref=advsearch&l=&l="><img src="https://img.shields.io/github/issues/badges/shields/good%20first%20issue" alt="good first"></a>
 [![codecov](https://codecov.io/gh/kubesphere/ks-devops/branch/master/graph/badge.svg?token=XS8g2CjdNL)](https://codecov.io/gh/kubesphere/ks-devops)
 [![Contributors](https://img.shields.io/github/contributors/kubesphere/ks-devops.svg)](https://github.com/kubesphere/ks-devops/graphs/contributors)
 
diff --git a/cmd/controller/app/controllers.go b/cmd/controller/app/controllers.go
index aae8fef..56c2e3d 100644
--- a/cmd/controller/app/controllers.go
+++ b/cmd/controller/app/controllers.go
@@ -105,6 +105,9 @@ func getAllControllers(mgr manager.Manager, client k8s.Client, informerFactory i
 	argocdAppReconciler := &argocd.ApplicationReconciler{
 		Client: mgr.GetClient(),
 	}
+	argocdClusterReconciler := &argocd.MultiClusterReconciler{
+		Client: mgr.GetClient(),
+	}
 
 	return map[string]func(mgr manager.Manager) error{
 		"gitrepository": func(mgr manager.Manager) error {
@@ -175,9 +178,12 @@ func getAllControllers(mgr manager.Manager, client k8s.Client, informerFactory i
 				kubesphereInformer.Devops().V1alpha1().S2iBinaries(),
 				kubesphereInformer.Devops().V1alpha1().S2iRuns()))
 		},
-		argocdReconciler.GetGroupName(): func(mgr manager.Manager) error {
-			if err := argocdReconciler.SetupWithManager(mgr); err != nil {
-				return err
+		argocdReconciler.GetGroupName(): func(mgr manager.Manager) (err error) {
+			if err = argocdReconciler.SetupWithManager(mgr); err != nil {
+				return
+			}
+			if err = argocdClusterReconciler.SetupWithManager(mgr); err != nil {
+				return
 			}
 			return argocdAppReconciler.SetupWithManager(mgr)
 		},
diff --git a/controllers/argocd/multi-cluster-controller.go b/controllers/argocd/multi-cluster-controller.go
new file mode 100644
index 0000000..453ccad
--- /dev/null
+++ b/controllers/argocd/multi-cluster-controller.go
@@ -0,0 +1,223 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package argocd
+
+import (
+	"context"
+	"encoding/base64"
+	"encoding/json"
+	"github.com/go-logr/logr"
+	v1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/pkg/utils/k8sutil"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
+)
+
+//+kubebuilder:rbac:groups=cluster.kubesphere.io,resources=clusters,verbs=get;list;watch
+//+kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;create;update
+
+// MultiClusterReconciler represents a controller to sync cluster to ArgoCD cluster
+type MultiClusterReconciler struct {
+	client.Client
+	log      logr.Logger
+	recorder record.EventRecorder
+
+	argocdNamespace string
+}
+
+// Reconcile is the entrypoint of the controller
+func (r *MultiClusterReconciler) Reconcile(req ctrl.Request) (result ctrl.Result, err error) {
+	var cluster *unstructured.Unstructured
+	if cluster, err = getCluster(r.Client, req.NamespacedName); err != nil {
+		err = client.IgnoreNotFound(err)
+		return
+	}
+
+	// find the namespace of ArgoCD, and cache it
+	if r.argocdNamespace == "" {
+		if r.argocdNamespace = r.findArgoCDNamespace(); r.argocdNamespace == "" {
+			r.log.V(6).Info("cannot found the namespace of ArgoCD")
+			return
+		}
+	}
+
+	// ignore the host cluster
+	if ignore(cluster) {
+		return
+	}
+
+	ownerRef := getOwnerReference(cluster)
+	argoCluster := createArgoCluster(cluster)
+	argoCluster.SetNamespace(r.argocdNamespace)
+	err = r.updateOrCreate(argoCluster, ownerRef)
+	return
+}
+
+func getOwnerReference(object *unstructured.Unstructured) (ref metav1.OwnerReference) {
+	ref = metav1.OwnerReference{
+		APIVersion: object.GetAPIVersion(),
+		Kind:       object.GetKind(),
+		Name:       object.GetName(),
+		UID:        object.GetUID(),
+	}
+	return
+}
+
+func (r *MultiClusterReconciler) updateOrCreate(cluster *v1.Secret, ownerRef metav1.OwnerReference) (err error) {
+	ctx := context.Background()
+	existingArgoCluster := cluster.DeepCopy()
+	if err = r.Get(ctx, types.NamespacedName{
+		Namespace: cluster.Namespace,
+		Name:      cluster.Name,
+	}, existingArgoCluster); err != nil {
+		if apierrors.IsNotFound(err) {
+			// create the argo cluster
+			k8sutil.SetOwnerReference(cluster, ownerRef)
+			err = r.Create(ctx, cluster)
+		}
+		return
+	}
+	existingArgoCluster.Data = cluster.Data
+	k8sutil.SetOwnerReference(existingArgoCluster, ownerRef)
+	err = r.Update(ctx, existingArgoCluster)
+	return
+}
+
+// let's assume there is only one ArgoCD existing
+// TODO consider to have a better solution to figure the namespace of ArgoCD
+func (r *MultiClusterReconciler) findArgoCDNamespace() string {
+	secretList := &v1.SecretList{}
+	if err := r.List(context.Background(), secretList); err == nil {
+		for i := range secretList.Items {
+			secret := secretList.Items[i]
+			if secret.Name == "argocd-secret" {
+				return secret.Namespace
+			}
+		}
+	}
+	return ""
+}
+
+// createArgoCluster creates an object which represents the argo cluster\
+// see also https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#clusters
+func createArgoCluster(cluster *unstructured.Unstructured) (secret *v1.Secret) {
+	name := cluster.GetName()
+	secret = &v1.Secret{
+		Type: "Opaque",
+	}
+	secret.SetName(name)
+	secret.SetLabels(map[string]string{
+		"argocd.argoproj.io/secret-type": "cluster",
+	})
+
+	config, _, _ := unstructured.NestedString(cluster.Object, "spec", "connection", "kubeconfig")
+	server, _, _ := unstructured.NestedString(cluster.Object, "spec", "connection", "kubernetesAPIEndpoint")
+
+	secret.Data = map[string][]byte{
+		"name":   []byte(name),
+		"config": getArgoClusterConfigData(config),
+		"server": []byte(server),
+	}
+	return
+}
+
+// getArgoClusterConfigFormat returns a ArgoCD cluster format config from a KubeSpher cluster config format
+// the parameter `config` is base64 encoded
+func getArgoClusterConfigData(config string) (result []byte) {
+	clusterConfigObject := getArgoClusterConfigObject(config)
+	result, _ = json.Marshal(clusterConfigObject)
+	return
+}
+
+// only support one config
+func getArgoClusterConfigObject(configText string) (object *ClusterConfig) {
+	rawDecodedconfig, _ := base64.StdEncoding.DecodeString(configText)
+
+	var kubeconfig *config
+	if kubeconfig = parseKubeConfig(rawDecodedconfig); kubeconfig == nil {
+		return
+	}
+
+	if len(kubeconfig.Clusters) == 0 || len(kubeconfig.Users) == 0 {
+		return
+	}
+
+	cluster := kubeconfig.Clusters[0]
+	user := kubeconfig.Users[0]
+
+	object = &ClusterConfig{
+		BearerToken: user.Auth.Token,
+		TLSClientConfig: TLSClientConfig{
+			Insecure: cluster.Cluster.SkipTLS,
+			CAData:   []byte(cluster.Cluster.CA),
+			CertData: []byte(user.Auth.ClientCert),
+			KeyData:  []byte(user.Auth.ClientKey),
+		},
+	}
+	return
+}
+
+func getCluster(client client.Reader, namespacedName types.NamespacedName) (cluster *unstructured.Unstructured, err error) {
+	cluster = getClusterObject()
+
+	if err = client.Get(context.Background(), namespacedName, cluster); err != nil {
+		cluster = nil
+	}
+	return
+}
+
+func ignore(cluster *unstructured.Unstructured) bool {
+	if cluster != nil {
+		_, ok := cluster.GetLabels()["cluster-role.kubesphere.io/host"]
+		return ok
+	}
+	return true
+}
+
+// GetName returns the name of this controller
+func (r *MultiClusterReconciler) GetName() string {
+	return "ArgoCDMultiClusterController"
+}
+
+// GetGroupName returns the group name of this controller
+func (r *MultiClusterReconciler) GetGroupName() string {
+	return controllerGroupName
+}
+
+func getClusterObject() *unstructured.Unstructured {
+	cluster := &unstructured.Unstructured{}
+	cluster.SetKind("Cluster")
+	cluster.SetAPIVersion("cluster.kubesphere.io/v1alpha1")
+	return cluster.DeepCopy()
+}
+
+// SetupWithManager init the logger, recorder and filters
+func (r *MultiClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	cluster := getClusterObject()
+	r.log = ctrl.Log.WithName(r.GetName())
+	r.recorder = mgr.GetEventRecorderFor(r.GetName())
+	return ctrl.NewControllerManagedBy(mgr).
+		For(cluster).
+		WithEventFilter(predicate.GenerationChangedPredicate{}).
+		Complete(r)
+}
diff --git a/controllers/argocd/types.go b/controllers/argocd/types.go
new file mode 100644
index 0000000..0c592b7
--- /dev/null
+++ b/controllers/argocd/types.go
@@ -0,0 +1,93 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Parts of the following structs originally come from https://github.com/argoproj/argo-cd/
+
+package argocd
+
+import (
+	"fmt"
+	"gopkg.in/yaml.v2"
+)
+
+// ClusterConfig is the configuration attributes. This structure is subset of the go-client
+// rest.Config with annotations added for marshalling.
+type ClusterConfig struct {
+	// Server requires Basic authentication
+	Username string `json:"username,omitempty"`
+	Password string `json:"password,omitempty"`
+
+	// Server requires Bearer authentication. This client will not attempt to use
+	// refresh tokens for an OAuth2 flow.
+	// TODO: demonstrate an OAuth2 compatible client.
+	BearerToken string `json:"bearerToken,omitempty"`
+
+	// TLSClientConfig contains settings to enable transport layer security
+	TLSClientConfig `json:"tlsClientConfig"`
+}
+
+// TLSClientConfig contains settings to enable transport layer security
+type TLSClientConfig struct {
+	// Insecure specifies that the server should be accessed without verifying the TLS certificate. For testing only.
+	Insecure bool `json:"insecure"`
+	// ServerName is passed to the server for SNI and is used in the client to check server
+	// certificates against. If ServerName is empty, the hostname used to contact the
+	// server is used.
+	ServerName string `json:"serverName,omitempty"`
+	// CertData holds PEM-encoded bytes (typically read from a client certificate file).
+	// CertData takes precedence over CertFile
+	CertData []byte `json:"certData,omitempty"`
+	// KeyData holds PEM-encoded bytes (typically read from a client certificate key file).
+	// KeyData takes precedence over KeyFile
+	KeyData []byte `json:"keyData,omitempty"`
+	// CAData holds PEM-encoded bytes (typically read from a root certificates bundle).
+	// CAData takes precedence over CAFile
+	CAData []byte `json:"caData,omitempty"`
+}
+
+func parseKubeConfig(data []byte) (cfg *config) {
+	cfg = &config{}
+	err := yaml.Unmarshal(data, cfg)
+	fmt.Println(err)
+	return
+}
+
+type config struct {
+	Clusters []clusterConfig     `yaml:"clusters,omitempty"`
+	Users    []clusterConfigUser `yaml:"users,omitempty"`
+}
+
+type clusterConfig struct {
+	Name    string                  `yaml:"name,omitempty"`
+	Cluster clusterConfigConnection `yaml:"cluster,omitempty"`
+}
+
+type clusterConfigConnection struct {
+	SkipTLS bool   `yaml:"insecure-skip-tls-verify,omitempty"`
+	Server  string `yaml:"server,omitempty"`
+	CA      string `yaml:"certificate-authority-data:,omitempty"`
+}
+
+type clusterConfigUser struct {
+	Name string                `yaml:"name,omitempty"`
+	Auth clusterConfigUserAuth `yaml:"user,omitempty"`
+}
+
+type clusterConfigUserAuth struct {
+	Token      string `yaml:"token,omitempty"`
+	ClientCert string `yaml:"client-certificate-data,omitempty"`
+	ClientKey  string `yaml:"client-key-data,omitempty"`
+}


Test patch:
diff --git a/controllers/argocd/multi-cluster-controller_test.go b/controllers/argocd/multi-cluster-controller_test.go
new file mode 100644
index 0000000..4aa0cbf
--- /dev/null
+++ b/controllers/argocd/multi-cluster-controller_test.go
@@ -0,0 +1,504 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package argocd
+
+import (
+	"context"
+	"encoding/base64"
+	"fmt"
+	"github.com/go-logr/logr"
+	"github.com/stretchr/testify/assert"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/tools/record"
+	"kubesphere.io/devops/pkg/api/devops/v1alpha3"
+	controllerruntime "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"testing"
+)
+
+func Test_ignore(t *testing.T) {
+	sampleCluster := &unstructured.Unstructured{}
+	sampleCluster.SetKind("Cluster")
+	sampleCluster.SetAPIVersion("cluster.kubesphere.io/v1alpha1")
+
+	type args struct {
+		cluster func() *unstructured.Unstructured
+	}
+	tests := []struct {
+		name string
+		args args
+		want bool
+	}{{
+		name: "cluster is nil",
+		want: true,
+	}, {
+		name: "cluster without the particular label",
+		args: args{cluster: func() *unstructured.Unstructured {
+			return sampleCluster.DeepCopy()
+		}},
+		want: false,
+	}, {
+		name: "with the particular label",
+		args: args{cluster: func() *unstructured.Unstructured {
+			cluster := sampleCluster.DeepCopy()
+			cluster.SetLabels(map[string]string{
+				"cluster-role.kubesphere.io/host": "",
+			})
+			return cluster
+		}},
+		want: true,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			var cluster *unstructured.Unstructured
+			if tt.args.cluster != nil {
+				cluster = tt.args.cluster()
+			}
+			assert.Equalf(t, tt.want, ignore(cluster), "ignore(%v)", tt.args.cluster)
+		})
+	}
+}
+
+func Test_createArgoCluster(t *testing.T) {
+	defaultCluster := &unstructured.Unstructured{}
+	defaultCluster.SetKind("Cluster")
+	defaultCluster.SetAPIVersion("cluster.kubesphere.io/v1alpha1")
+	defaultCluster.SetNamespace("ns")
+	defaultCluster.SetName("name")
+	_ = unstructured.SetNestedField(defaultCluster.Object, base64.StdEncoding.EncodeToString([]byte(`clusters:
+- cluster:
+    insecure-skip-tls-verify: true
+    server: server
+  name: name
+users:
+- name: name
+  user:
+    token: token`)), "spec", "connection", "kubeconfig")
+	_ = unstructured.SetNestedField(defaultCluster.Object, "server", "spec", "connection", "kubernetesAPIEndpoint")
+
+	defaultArgoCluster := &v1.Secret{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: "name",
+			Labels: map[string]string{
+				"argocd.argoproj.io/secret-type": "cluster",
+			},
+		},
+		Type: "Opaque",
+		Data: map[string][]byte{
+			"name":   []byte("name"),
+			"config": []byte(`{"bearerToken":"token","tlsClientConfig":{"insecure":true}}`),
+			"server": []byte("server"),
+		},
+	}
+
+	type args struct {
+		cluster *unstructured.Unstructured
+	}
+	tests := []struct {
+		name       string
+		args       args
+		wantSecret *v1.Secret
+	}{{
+		name: "normal",
+		args: args{
+			cluster: defaultCluster.DeepCopy(),
+		},
+		wantSecret: defaultArgoCluster.DeepCopy(),
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, tt.wantSecret, createArgoCluster(tt.args.cluster), "createArgoCluster(%v)", tt.args.cluster)
+		})
+	}
+}
+
+func TestInterfaceImplement(t *testing.T) {
+	tests := []struct {
+		name   string
+		verify func(t *testing.T)
+	}{{
+		name: "NamedReconciler",
+		verify: func(t *testing.T) {
+			reconciler := &MultiClusterReconciler{}
+			assert.NotEmpty(t, reconciler.GetName())
+		},
+	}, {
+		name: "GroupedReconciler",
+		verify: func(t *testing.T) {
+			reconciler := &MultiClusterReconciler{}
+			assert.NotEmpty(t, reconciler.GetGroupName())
+		},
+	}}
+	for i := range tests {
+		tt := tests[i]
+		t.Run(tt.name, func(t *testing.T) {
+			tt.verify(t)
+		})
+	}
+}
+
+func Test_getCluster(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+
+	cluster := &unstructured.Unstructured{}
+	cluster.SetKind("Cluster")
+	cluster.SetAPIVersion("cluster.kubesphere.io/v1alpha1")
+	cluster.SetName("name")
+	cluster.SetNamespace("namespace")
+
+	defaultNamespacedName := types.NamespacedName{
+		Namespace: "namespace",
+		Name:      "name",
+	}
+
+	type args struct {
+		client         client.Reader
+		namespacedName types.NamespacedName
+	}
+	tests := []struct {
+		name   string
+		args   args
+		verify func(t *testing.T, obj *unstructured.Unstructured, err error)
+	}{{
+		name: "normal case",
+		args: args{
+			client:         fake.NewFakeClientWithScheme(schema, cluster.DeepCopy()),
+			namespacedName: defaultNamespacedName,
+		},
+		verify: func(t *testing.T, obj *unstructured.Unstructured, err error) {
+			assert.Nil(t, err)
+			assert.NotNil(t, obj)
+			assert.Equal(t, cluster, obj)
+		},
+	}, {
+		name: "not found",
+		args: args{
+			client:         fake.NewFakeClientWithScheme(schema),
+			namespacedName: defaultNamespacedName,
+		},
+		verify: func(t *testing.T, obj *unstructured.Unstructured, err error) {
+			assert.NotNil(t, err)
+			assert.Nil(t, obj)
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			gotResult, err := getCluster(tt.args.client, tt.args.namespacedName)
+			tt.verify(t, gotResult, err)
+		})
+	}
+}
+
+func TestMultiClusterReconciler_updateOrCreate(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	defaultArgoCluster := &v1.Secret{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: "ns",
+			Name:      "name",
+		},
+		StringData: map[string]string{
+			"key": "value",
+		},
+	}
+
+	defaultOwnerRef := metav1.OwnerReference{}
+
+	type fields struct {
+		Client   client.Client
+		log      logr.Logger
+		recorder record.EventRecorder
+	}
+	type args struct {
+		cluster *v1.Secret
+	}
+	tests := []struct {
+		name   string
+		fields fields
+		args   args
+		verify func(t *testing.T, c client.Client, err error)
+	}{{
+		name: "create when the cluster does not exist",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema),
+		},
+		args: args{
+			cluster: defaultArgoCluster.DeepCopy(),
+		},
+		verify: func(t *testing.T, c client.Client, err error) {
+			assert.Nil(t, err)
+
+			secret := &v1.Secret{}
+			err = c.Get(context.Background(), types.NamespacedName{
+				Namespace: "ns",
+				Name:      "name",
+			}, secret)
+			assert.Nil(t, err)
+			assert.Equal(t, "value", secret.StringData["key"])
+			assert.True(t, len(secret.GetOwnerReferences()) == 1)
+			assert.Equal(t, defaultOwnerRef, secret.GetOwnerReferences()[0])
+		},
+	}, {
+		name: "update when the corresponding cluster does not exist",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema, defaultArgoCluster.DeepCopy()),
+		},
+		args: args{
+			cluster: defaultArgoCluster.DeepCopy(),
+		},
+		verify: func(t *testing.T, c client.Client, err error) {
+			assert.Nil(t, err)
+
+			secret := &v1.Secret{}
+			err = c.Get(context.Background(), types.NamespacedName{
+				Namespace: "ns",
+				Name:      "name",
+			}, secret)
+			assert.Nil(t, err)
+			assert.Equal(t, "value", secret.StringData["key"])
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &MultiClusterReconciler{
+				Client:   tt.fields.Client,
+				log:      tt.fields.log,
+				recorder: tt.fields.recorder,
+			}
+			err := r.updateOrCreate(tt.args.cluster, defaultOwnerRef)
+			tt.verify(t, tt.fields.Client, err)
+		})
+	}
+}
+
+func Test_getOwnerReference(t *testing.T) {
+	defaultObject := &unstructured.Unstructured{}
+	defaultObject.SetName("name")
+	defaultObject.SetKind("kind")
+	defaultObject.SetUID("uid")
+	defaultObject.SetAPIVersion("apiversion")
+
+	type args struct {
+		object *unstructured.Unstructured
+	}
+	tests := []struct {
+		name    string
+		args    args
+		wantRef metav1.OwnerReference
+	}{{
+		name: "normal",
+		args: args{
+			object: defaultObject.DeepCopy(),
+		},
+		wantRef: metav1.OwnerReference{
+			APIVersion: "apiversion",
+			Kind:       "kind",
+			Name:       "name",
+			UID:        "uid",
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, tt.wantRef, getOwnerReference(tt.args.object), "getOwnerReference(%v)", tt.args.object)
+		})
+	}
+}
+
+func TestMultiClusterReconciler_Reconcile(t *testing.T) {
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	defaultRequest := controllerruntime.Request{
+		NamespacedName: types.NamespacedName{
+			Namespace: "ns",
+			Name:      "name",
+		},
+	}
+
+	defaultCluster := &unstructured.Unstructured{}
+	defaultCluster.SetKind("Cluster")
+	defaultCluster.SetAPIVersion("cluster.kubesphere.io/v1alpha1")
+	defaultCluster.SetName("name")
+	defaultCluster.SetNamespace("ns")
+
+	ignoreCluster := defaultCluster.DeepCopy()
+	ignoreCluster.SetLabels(map[string]string{
+		"cluster-role.kubesphere.io/host": "",
+	})
+
+	argoSecret := &v1.Secret{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: "ns",
+			Name:      "argocd-secret",
+		},
+	}
+
+	type fields struct {
+		Client   client.Client
+		log      logr.Logger
+		recorder record.EventRecorder
+	}
+	type args struct {
+		req controllerruntime.Request
+	}
+	tests := []struct {
+		name       string
+		fields     fields
+		args       args
+		wantResult controllerruntime.Result
+		wantErr    assert.ErrorAssertionFunc
+		verify     func(t *testing.T, c client.Client)
+	}{{
+		name: "not found",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema),
+		},
+		args: args{
+			req: defaultRequest,
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			return true
+		},
+		wantResult: controllerruntime.Result{},
+	}, {
+		name: "ignore changes case",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema, ignoreCluster.DeepCopy(), argoSecret.DeepCopy()),
+		},
+		args: args{
+			req: defaultRequest,
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			return false
+		},
+		wantResult: controllerruntime.Result{},
+	}, {
+		name: "no argo cluster existing",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema, defaultCluster.DeepCopy(), argoSecret.DeepCopy()),
+		},
+		args: args{
+			req: defaultRequest,
+		},
+		wantErr: func(t assert.TestingT, err error, i ...interface{}) bool {
+			return false
+		},
+		wantResult: controllerruntime.Result{},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &MultiClusterReconciler{
+				Client:   tt.fields.Client,
+				log:      tt.fields.log,
+				recorder: tt.fields.recorder,
+			}
+			gotResult, err := r.Reconcile(tt.args.req)
+			if !tt.wantErr(t, err, fmt.Sprintf("Reconcile(%v)", tt.args.req)) {
+				return
+			}
+			assert.Equalf(t, tt.wantResult, gotResult, "Reconcile(%v)", tt.args.req)
+			if tt.verify != nil {
+				tt.verify(t, tt.fields.Client)
+			}
+		})
+	}
+}
+
+func TestMultiClusterReconciler_findArgoCDNamespace(t *testing.T) {
+	const expectedNamespace = "namespace"
+	schema, err := v1alpha3.SchemeBuilder.Register().Build()
+	assert.Nil(t, err)
+	err = v1.SchemeBuilder.AddToScheme(schema)
+	assert.Nil(t, err)
+
+	argoSecret := &v1.Secret{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: expectedNamespace,
+			Name:      "argocd-secret",
+		},
+	}
+
+	type fields struct {
+		Client client.Client
+	}
+	tests := []struct {
+		name   string
+		fields fields
+		want   string
+	}{{
+		name: "no expected secret",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema),
+		},
+		want: "",
+	}, {
+		name: "normal case, have the expected secret",
+		fields: fields{
+			Client: fake.NewFakeClientWithScheme(schema, argoSecret.DeepCopy()),
+		},
+		want: expectedNamespace,
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			r := &MultiClusterReconciler{
+				Client: tt.fields.Client,
+			}
+			assert.Equalf(t, tt.want, r.findArgoCDNamespace(), "findArgoCDNamespace()")
+		})
+	}
+}
+
+func Test_getArgoClusterConfigFormat(t *testing.T) {
+	configData := base64.StdEncoding.EncodeToString([]byte(`clusters:
+- cluster:
+    insecure-skip-tls-verify: true
+    server: server
+  name: name
+users:
+- name: name
+  user:
+    token: token`))
+	type args struct {
+		config string
+	}
+	tests := []struct {
+		name string
+		args args
+		want []byte
+	}{{
+		name: "normal",
+		args: args{
+			config: configData,
+		},
+		want: []byte(`{"bearerToken":"token","tlsClientConfig":{"insecure":true}}`),
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, string(tt.want), string(getArgoClusterConfigData(tt.args.config)), "getArgoClusterConfigFormat(%v)", tt.args.config)
+		})
+	}
+}
diff --git a/controllers/argocd/types_test.go b/controllers/argocd/types_test.go
new file mode 100644
index 0000000..f83a3e0
--- /dev/null
+++ b/controllers/argocd/types_test.go
@@ -0,0 +1,67 @@
+/*
+Copyright 2022 The KubeSphere Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package argocd
+
+import (
+	"github.com/stretchr/testify/assert"
+	"testing"
+)
+
+func Test_parseKubeConfig(t *testing.T) {
+	type args struct {
+		data []byte
+	}
+	tests := []struct {
+		name string
+		args args
+		want *config
+	}{{
+		name: "with token",
+		args: args{
+			data: []byte(`
+clusters:
+- cluster:
+    insecure-skip-tls-verify: true
+    server: server
+  name: name
+users:
+- name: name
+  user:
+    token: token`),
+		},
+		want: &config{
+			Clusters: []clusterConfig{{
+				Name: "name",
+				Cluster: clusterConfigConnection{
+					SkipTLS: true,
+					Server:  "server",
+				},
+			}},
+			Users: []clusterConfigUser{{
+				Name: "name",
+				Auth: clusterConfigUserAuth{
+					Token: "token",
+				},
+			}},
+		},
+	}}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, tt.want, parseKubeConfig(tt.args.data), "parseKubeConfig(%v)", tt.args.data)
+		})
+	}
+}