Repository: kubeclipper/kubeclipper
Pull request #412 (2022-11-18 08:09:28)

Refactor add-or-remove cluster node operation feature
<!-- Thanks for sending a pull request! Here are some tips for you:

1. If you want **faster** PR reviews, read how: https://github.com/kubeclipper/community/blob/master/developer-guide/development/the-pr-author-guide-to-getting-through-code-review.md
2. In case you want to know how your PR got reviewed, read: https://github.com/kubeclipper/community/blob/master/developer-guide/development/code-review-guide.md
3. Here are some coding convetions followed by Kubeclipper community: https://github.com/kubeclipper/community/blob/master/developer-guide/development/coding-conventions.md
-->

### What type of PR is this?
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->
/kind feature

### What this PR does / why we need it:

### Which issue(s) this PR fixes:
<!--
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

### Special notes for reviewers:
```
```

### Does this PR introduced a user-facing change?
<!--
If no, just write "None" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md
-->
```release-note

```

### Additional documentation, usage docs, etc.:
<!--
This section can be blank if this pull request does not require a release note.
Please use the following format for linking documentation or pass the
section below:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```

kubeclipper-bot: @qinyer: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our [release note process](https://git.k8s.io/community/contributors/guide/release-notes.md) to remove it.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
kubeclipper-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubeclipper/kubeclipper/pull/412#" title="Author self-approved">qinyer</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **wu-wenxiang** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @wu-wenxiang` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubeclipper%2Fkubeclipper).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubeclipper/kubeclipper/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["wu-wenxiang"]} -->
codecov-commenter: # [Codecov](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) Report
> Merging [#412](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) (1843e8c) into [master](https://codecov.io/gh/kubeclipper/kubeclipper/commit/31dd99dcb5130cf8692a418ddacc7b581f12060f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) (31dd99d) will **decrease** coverage by `0.14%`.
> The diff coverage is `27.77%`.

[![Impacted file tree graph](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/graphs/tree.svg?width=650&height=150&src=pr&token=3PWFG2MHTE&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper)](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper)

```diff
@@            Coverage Diff             @@
##           master     #412      +/-   ##
==========================================
- Coverage   12.17%   12.03%   -0.15%     
==========================================
  Files         103      106       +3     
  Lines       16969    17138     +169     
==========================================
- Hits         2066     2062       -4     
- Misses      14656    14828     +172     
- Partials      247      248       +1     
```


| [Impacted Files](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) | Coverage Δ | |
|---|---|---|
| [pkg/apis/core/v1/handler.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2FwaXMvY29yZS92MS9oYW5kbGVyLmdv) | `0.33% <0.00%> (+<0.01%)` | :arrow_up: |
| [pkg/apis/core/v1/registry.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2FwaXMvY29yZS92MS9yZWdpc3RyeS5nbw==) | `0.00% <0.00%> (ø)` | |
| [pkg/apis/core/v1/schema.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2FwaXMvY29yZS92MS9zY2hlbWEuZ28=) | `59.70% <ø> (-10.59%)` | :arrow_down: |
| [pkg/apis/core/v1/utils.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2FwaXMvY29yZS92MS91dGlscy5nbw==) | `52.33% <0.00%> (-1.91%)` | :arrow_down: |
| [pkg/clusteroperation/operation.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2NsdXN0ZXJvcGVyYXRpb24vb3BlcmF0aW9uLmdv) | `0.00% <0.00%> (ø)` | |
| [pkg/scheme/core/v1/k8s/node.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL3NjaGVtZS9jb3JlL3YxL2s4cy9ub2RlLmdv) | `0.33% <0.00%> (ø)` | |
| [pkg/clusteroperation/common.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2NsdXN0ZXJvcGVyYXRpb24vY29tbW9uLmdv) | `35.29% <35.29%> (ø)` | |
| [pkg/clusteroperation/node.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/412/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2NsdXN0ZXJvcGVyYXRpb24vbm9kZS5nbw==) | `64.70% <64.70%> (ø)` | |

x893675: /cc @zhuzhenfan 

Patch:
diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 0000000..3c8e08e
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,3 @@
+.git
+.idea
+dist
diff --git a/.github/workflows/build-kc-image.yml b/.github/workflows/build-kc-image.yml
new file mode 100644
index 0000000..a7b8061
--- /dev/null
+++ b/.github/workflows/build-kc-image.yml
@@ -0,0 +1,69 @@
+name: build-kc-image
+
+on:
+  push:
+    branches:
+      - 'master'
+    tags:
+      - 'v*'
+
+jobs:
+  build:
+    name: build-kc-image
+    runs-on: ubuntu-latest
+    env:
+      GO111MODULE: on
+    steps:
+      - name: Set up Go 1.19
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.19
+        id: go
+
+      - name: Check out code into the Go module directory
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+
+      - name: Extract Tag
+        id: extract_tag
+        run:  echo ::set-output name=version::${GITHUB_REF/refs\/tags\//}
+
+      - name: Extract branch name
+        shell: bash
+        run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/})"
+        id: extract_branch
+
+      - name: Set up docker buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v2
+        with:
+          version: latest
+
+      - name: Available platforms
+        run: echo ${{ steps.buildx.outputs.platforms }}
+
+      - name: Docker login
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
+          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+        run: |
+          echo "${DOCKER_PASSWORD}" | docker login --username "${DOCKER_USERNAME}" --password-stdin
+
+      - name: Run buildx
+        run: |
+          tag=latest
+          if [[ "${{ steps.extract_branch.outputs.branch }}" == "master" ]];then 
+             tag=latest
+          elif [[ "${{ steps.extract_tag.outputs.version }}" == v* ]]; then
+            tag="${{ steps.extract_tag.outputs.version }}"
+          fi
+     
+          echo extract_tag   ${{ steps.extract_tag.outputs.version }}
+          echo extract_branch  ${{ steps.extract_branch.outputs.branch }}
+          echo current tag is ${tag}
+          
+          docker buildx build --platform linux/amd64,linux/arm64 --push --file ./Dockerfile -t kubeclipper/kc-server:${tag} .
+
+
diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 0000000..378926d
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,15 @@
+FROM --platform=${BUILDPLATFORM}  golang:1.19 as build
+ARG TARGETPLATFORM
+ARG TARGETOS
+ARG TARGETARCH
+ENV GOPROXY=https://goproxy.cn
+WORKDIR /kubeclipper
+COPY . /kubeclipper
+RUN KUBE_BUILD_PLATFORMS="${TARGETPLATFORM}" KUBE_ALL_WITH_PREFIX="true" make build
+
+FROM alpine:3.16.3
+ARG TARGETOS
+ARG TARGETARCH
+WORKDIR /root
+COPY --from=build /kubeclipper/dist/${TARGETOS}_${TARGETARCH}/kubeclipper-server .
+ENTRYPOINT  ["./kubeclipper-server","serve"]
diff --git a/api/openapi-spec/swagger.json b/api/openapi-spec/swagger.json
index 4a64a2a..93ce826 100644
--- a/api/openapi-spec/swagger.json
+++ b/api/openapi-spec/swagger.json
@@ -1483,6 +1483,13 @@
             "name": "name",
             "in": "path",
             "required": true
+          },
+          {
+            "type": "boolean",
+            "default": false,
+            "description": "use kubeClipper proxy",
+            "name": "proxy",
+            "in": "query"
           }
         ],
         "responses": {
@@ -4602,6 +4609,76 @@
         }
       }
     },
+    "/api/core.kubeclipper.io/v1/projects/{project}/regions": {
+      "get": {
+        "produces": [
+          "application/json"
+        ],
+        "tags": [
+          "Core-Region"
+        ],
+        "summary": "List regions in project.",
+        "operationId": "ListRegionsInProject",
+        "parameters": [
+          {
+            "type": "string",
+            "description": "project name",
+            "name": "project",
+            "in": "path",
+            "required": true
+          },
+          {
+            "type": "string",
+            "format": "limit=%d,page=%d",
+            "default": "limit=10,page=1",
+            "description": "paging query, e.g. limit=100,page=1",
+            "name": "paging",
+            "in": "query"
+          },
+          {
+            "type": "string",
+            "format": "labelSelector=%s=%s",
+            "description": "resource filter by metadata label",
+            "name": "labelSelector",
+            "in": "query"
+          },
+          {
+            "type": "string",
+            "format": "fieldSelector=%s=%s",
+            "description": "resource filter by field",
+            "name": "fieldSelector",
+            "in": "query"
+          },
+          {
+            "type": "boolean",
+            "description": "resource sort reverse or not",
+            "name": "reverse",
+            "in": "query"
+          },
+          {
+            "type": "boolean",
+            "description": "watch request",
+            "name": "watch",
+            "in": "query"
+          },
+          {
+            "type": "integer",
+            "default": 60,
+            "description": "watch timeout seconds",
+            "name": "timeoutSeconds",
+            "in": "query"
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "schema": {
+              "$ref": "#/definitions/models.PageableResponse"
+            }
+          }
+        }
+      }
+    },
     "/api/core.kubeclipper.io/v1/projects/{project}/registries": {
       "get": {
         "produces": [
@@ -5317,47 +5394,6 @@
           }
         }
       },
-      "put": {
-        "produces": [
-          "application/json"
-        ],
-        "tags": [
-          "Core-IAM"
-        ],
-        "summary": "Update member info in specified project",
-        "operationId": "UpdateProjectMember",
-        "parameters": [
-          {
-            "name": "body",
-            "in": "body",
-            "required": true,
-            "schema": {
-              "$ref": "#/definitions/v1.Member"
-            }
-          },
-          {
-            "type": "string",
-            "description": "project name",
-            "name": "project",
-            "in": "path",
-            "required": true
-          }
-        ],
-        "responses": {
-          "200": {
-            "description": "OK",
-            "schema": {
-              "$ref": "#/definitions/v1.ProjectRoleBinding"
-            }
-          },
-          "500": {
-            "description": "Internal Server Error",
-            "schema": {
-              "$ref": "#/definitions/errors.HTTPError"
-            }
-          }
-        }
-      },
       "post": {
         "produces": [
           "application/json"
@@ -5531,38 +5567,38 @@
         }
       }
     },
-    "/api/iam.kubeclipper.io/v1/projects/{project}/projectroles": {
-      "get": {
+    "/api/iam.kubeclipper.io/v1/projects/{project}/projectmembers/{projectmember}": {
+      "put": {
         "produces": [
           "application/json"
         ],
         "tags": [
           "Core-IAM"
         ],
-        "summary": "List roles under specified project",
-        "operationId": "ListProjectRole",
+        "summary": "Update member info in specified project",
+        "operationId": "UpdateProjectMember",
         "parameters": [
+          {
+            "name": "body",
+            "in": "body",
+            "required": true,
+            "schema": {
+              "$ref": "#/definitions/v1.Member"
+            }
+          },
           {
             "type": "string",
             "description": "project name",
             "name": "project",
             "in": "path",
             "required": true
-          },
-          {
-            "type": "string",
-            "format": "limit=%d,page=%d",
-            "default": "limit=10,page=1",
-            "description": "paging query, e.g. limit=100,page=1",
-            "name": "paging",
-            "in": "query"
           }
         ],
         "responses": {
           "200": {
             "description": "OK",
             "schema": {
-              "$ref": "#/definitions/models.PageableResponse"
+              "$ref": "#/definitions/v1.ProjectRoleBinding"
             }
           },
           "500": {
@@ -5572,38 +5608,40 @@
             }
           }
         }
-      },
-      "put": {
+      }
+    },
+    "/api/iam.kubeclipper.io/v1/projects/{project}/projectroles": {
+      "get": {
         "produces": [
           "application/json"
         ],
         "tags": [
           "Core-IAM"
         ],
-        "summary": "Update role info",
-        "operationId": "UpdateProjectRole",
+        "summary": "List roles under specified project",
+        "operationId": "ListProjectRole",
         "parameters": [
-          {
-            "name": "body",
-            "in": "body",
-            "required": true,
-            "schema": {
-              "$ref": "#/definitions/v1.ProjectRole"
-            }
-          },
           {
             "type": "string",
             "description": "project name",
             "name": "project",
             "in": "path",
             "required": true
+          },
+          {
+            "type": "string",
+            "format": "limit=%d,page=%d",
+            "default": "limit=10,page=1",
+            "description": "paging query, e.g. limit=100,page=1",
+            "name": "paging",
+            "in": "query"
           }
         ],
         "responses": {
           "200": {
             "description": "OK",
             "schema": {
-              "$ref": "#/definitions/v1.ProjectRole"
+              "$ref": "#/definitions/models.PageableResponse"
             }
           },
           "500": {
@@ -5697,6 +5735,47 @@
           }
         }
       },
+      "put": {
+        "produces": [
+          "application/json"
+        ],
+        "tags": [
+          "Core-IAM"
+        ],
+        "summary": "Update projectrole info",
+        "operationId": "UpdateProjectRole",
+        "parameters": [
+          {
+            "name": "body",
+            "in": "body",
+            "required": true,
+            "schema": {
+              "$ref": "#/definitions/v1.ProjectRole"
+            }
+          },
+          {
+            "type": "string",
+            "description": "project name",
+            "name": "project",
+            "in": "path",
+            "required": true
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "OK",
+            "schema": {
+              "$ref": "#/definitions/v1.ProjectRole"
+            }
+          },
+          "500": {
+            "description": "Internal Server Error",
+            "schema": {
+              "$ref": "#/definitions/errors.HTTPError"
+            }
+          }
+        }
+      },
       "delete": {
         "produces": [
           "application/json"
@@ -7875,6 +7954,9 @@
           "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources",
           "type": "string"
         },
+        "description": {
+          "type": "string"
+        },
         "fsConfig": {
           "$ref": "#/definitions/v1.FsConfig"
         },
@@ -7923,35 +8005,35 @@
     "v1.CNI": {
       "required": [
         "type",
-        "Version",
-        "CriType",
-        "Offline",
+        "version",
+        "criType",
+        "offline",
         "namespace"
       ],
       "properties": {
-        "CriType": {
-          "type": "string"
-        },
-        "Offline": {
-          "type": "boolean"
-        },
-        "Version": {
-          "type": "string"
-        },
         "calico": {
           "$ref": "#/definitions/v1.Calico"
         },
+        "criType": {
+          "type": "string"
+        },
         "localRegistry": {
           "type": "string"
         },
         "namespace": {
           "type": "string"
         },
+        "offline": {
+          "type": "boolean"
+        },
         "type": {
           "type": "string",
           "enum": [
             "calico"
           ]
+        },
+        "version": {
+          "type": "string"
         }
       }
     },
@@ -8133,6 +8215,12 @@
         "etcd": {
           "$ref": "#/definitions/v1.Etcd"
         },
+        "externalCaCert": {
+          "type": "string"
+        },
+        "externalCaKey": {
+          "type": "string"
+        },
         "kind": {
           "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds",
           "type": "string"
@@ -8179,16 +8267,7 @@
       }
     },
     "v1.ClusterStatus": {
-      "required": [
-        "Registries"
-      ],
       "properties": {
-        "Registries": {
-          "type": "array",
-          "items": {
-            "$ref": "#/definitions/v1.RegistrySpec"
-          }
-        },
         "certifications": {
           "type": "array",
           "items": {
@@ -8210,6 +8289,12 @@
         "phase": {
           "type": "string"
         },
+        "registries": {
+          "type": "array",
+          "items": {
+            "$ref": "#/definitions/v1.RegistrySpec"
+          }
+        },
         "versions": {
           "$ref": "#/definitions/v1.ClusterVersionsStatus"
         }
@@ -8589,9 +8674,6 @@
       "properties": {
         "backupRootDir": {
           "type": "string"
-        },
-        "description": {
-          "type": "string"
         }
       }
     },
diff --git a/deploy/etcd-sts.yaml b/deploy/etcd-sts.yaml
new file mode 100644
index 0000000..073dcfa
--- /dev/null
+++ b/deploy/etcd-sts.yaml
@@ -0,0 +1,204 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: kc-etcd
+  labels:
+    app: kc-etcd
+spec:
+  ports:
+    - port: 2380
+      name: kc-etcd-server
+    - port: 2379
+      name: kc-etcd-client
+  clusterIP: None
+  selector:
+    app: kc-etcd
+  publishNotReadyAddresses: true
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: kc-etcd-pdb
+  labels:
+    pdb: kc-etcd
+spec:
+  minAvailable: 2
+  selector:
+    matchLabels:
+      app: kc-etcd
+---
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  labels:
+    app: kc-etcd
+  name: kc-etcd
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: kc-etcd
+  serviceName: kc-etcd
+  template:
+    metadata:
+      labels:
+        app: kc-etcd
+    spec:
+      containers:
+        - name: kc-etcd
+          image: caas4/etcd:3.5.5
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 2380
+              name: peer
+              protocol: TCP
+            - containerPort: 2379
+              name: client
+              protocol: TCP
+          env:
+            - name: INITIAL_CLUSTER_SIZE
+              value: "3"
+            - name: MY_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+            - name: POD_IP
+              valueFrom:
+                fieldRef:
+                  fieldPath: status.podIP
+            - name: SET_NAME
+              value: "kc-etcd"
+          command:
+            - /bin/sh
+            - -ec
+            - |
+              HOSTNAME=$(hostname)
+
+              eps() {
+                  EPS=""
+                  for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
+                      EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379"
+                  done
+                  echo ${EPS}
+              }
+
+              member_hash() {
+                 ETCDCTL_API=3 etcdctl member list | grep -w "$HOSTNAME" | awk '{ print $1}' | awk -F "," '{ print $1}'
+              }
+
+              initial_peers() {
+                  PEERS=""
+                  for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
+                    PEERS="${PEERS}${PEERS:+,}${SET_NAME}-${i}=http://${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2380"
+                  done
+                  echo ${PEERS}
+              }
+
+              # etcd-SET_ID
+              SET_ID=${HOSTNAME##*-}
+
+              # adding a new member to existing cluster (assuming all initial pods are available)
+              if [ "${SET_ID}" -ge ${INITIAL_CLUSTER_SIZE} ]; then
+                  # export ETCDCTL_ENDPOINTS=$(eps)
+                  # member already added?
+
+                  MEMBER_HASH=$(member_hash)
+                  if [ -n "${MEMBER_HASH}" ]; then
+                      # the member hash exists but for some reason etcd failed
+                      # as the datadir has not be created, we can remove the member
+                      # and retrieve new hash
+                      echo "Remove member ${MEMBER_HASH}"
+                    ETCDCTL_API=3 etcdctl --endpoints=$(eps) member remove ${MEMBER_HASH}
+                  fi
+
+                  echo "Adding new member"
+
+                  echo "ETCDCTL_API=3 etcdctl --endpoints=$(eps) member add ${HOSTNAME} --peer-urls=http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2380"
+                  ETCDCTL_API=3 etcdctl member --endpoints=$(eps) add ${HOSTNAME} --peer-urls=http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2380 | grep "^ETCD_" > /var/run/etcd/new_member_envs
+
+                  if [ $? -ne 0 ]; then
+                      echo "member add ${HOSTNAME} error."
+                      rm -f /var/run/etcd/new_member_envs
+                      exit 1
+                  fi
+
+                  echo "==> Loading env vars of existing cluster..."
+                  sed -ie "s/^/export /" /var/run/etcd/new_member_envs
+                  cat /var/run/etcd/new_member_envs
+                  . /var/run/etcd/new_member_envs
+
+                  echo "etcd --name ${HOSTNAME} --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} --listen-peer-urls http://${POD_IP}:2380 --listen-client-urls http://${POD_IP}:2379,http://127.0.0.1:2379 --advertise-client-urls http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379 --data-dir /var/run/etcd/default.etcd --initial-cluster ${ETCD_INITIAL_CLUSTER} --initial-cluster-state ${ETCD_INITIAL_CLUSTER_STATE}"
+
+                  exec etcd --listen-peer-urls http://${POD_IP}:2380 \
+                      --listen-client-urls http://${POD_IP}:2379,http://127.0.0.1:2379 \
+                      --advertise-client-urls http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379 \
+                      --data-dir /var/run/etcd/default.etcd
+              fi
+
+              for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
+                  while true; do
+                      echo "Waiting for ${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local to come up"
+                      ping -W 1 -c 1 ${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local > /dev/null && break
+                      sleep 1s
+                  done
+              done
+
+              echo "join member ${HOSTNAME}"
+              # join member
+              exec etcd --name ${HOSTNAME} \
+                  --initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2380 \
+                  --listen-peer-urls http://${POD_IP}:2380 \
+                  --listen-client-urls http://${POD_IP}:2379,http://127.0.0.1:2379 \
+                  --advertise-client-urls http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379 \
+                  --initial-cluster-token etcd-cluster-1 \
+                  --data-dir /var/run/etcd/default.etcd \
+                  --initial-cluster $(initial_peers) \
+                  --initial-cluster-state new
+          lifecycle:
+            preStop:
+              exec:
+                command:
+                  - /bin/sh
+                  - -ec
+                  - |
+                    HOSTNAME=$(hostname)
+
+                    member_hash() {
+                       ETCDCTL_API=3 etcdctl member list | grep -w "$HOSTNAME" | awk '{ print $1}' | awk -F "," '{ print $1}'
+                    }
+
+                    eps() {
+                        EPS=""
+                        for i in $(seq 0 $((${INITIAL_CLUSTER_SIZE} - 1))); do
+                            EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379"
+                        done
+                        echo ${EPS}
+                    }
+
+                    export ETCDCTL_ENDPOINTS=$(eps)
+                    SET_ID=${HOSTNAME##*-}
+
+                    # Removing member from cluster
+                    if [ "${SET_ID}" -ge ${INITIAL_CLUSTER_SIZE} ]; then
+                        echo "Removing ${HOSTNAME} from etcd cluster"
+                        ETCDCTL_API=3 etcdctl member remove $(member_hash)
+                        if [ $? -eq 0 ]; then
+                            # Remove everything otherwise the cluster will no longer scale-up
+                            rm -rf /var/run/etcd/*
+                        fi
+                    fi
+          volumeMounts:
+            - mountPath: /var/run/etcd
+              name: datadir
+  volumeClaimTemplates:
+    - metadata:
+        name: datadir
+      spec:
+        # specify a sc for this sts
+        storageClassName: nfs-sc
+        accessModes:
+          - "ReadWriteOnce"
+        resources:
+          requests:
+            # upstream recommended max is 700M
+            storage: 1Gi
\ No newline at end of file
diff --git a/deploy/kc-server.yaml b/deploy/kc-server.yaml
new file mode 100644
index 0000000..36fa28f
--- /dev/null
+++ b/deploy/kc-server.yaml
@@ -0,0 +1,60 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: kc-server
+  labels:
+    app: kc-server
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: kc-server
+  template:
+    metadata:
+      labels:
+        app: kc-server
+    spec:
+      volumes:
+        - name: host-time
+          hostPath:
+            path: /etc/localtime
+            type: ''
+        - name: kc-server
+          configMap:
+            name: kc-server
+      containers:
+        - name: kc-server
+          image: caas4/kc-server:v0.0.1
+          imagePullPolicy: Always
+          ports:
+            - containerPort: 80
+          volumeMounts:
+            - name: host-time
+              readOnly: true
+              mountPath: /etc/localtime
+            - name: kc-server
+              mountPath: /etc/kubeclipper-server
+          resources:
+            requests:
+              memory: "100Mi"
+              cpu: "100m"
+            limits:
+              memory: "500Mi"
+              cpu: "500m"
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: kc-server
+spec:
+  selector:
+    app: kc-server
+  type: NodePort
+#  type: LoadBalancer
+  ports:
+    - name: web
+      protocol: TCP
+      port: 80
+      #targetPort: 80
+
+
diff --git a/hack/lib/golang.sh b/hack/lib/golang.sh
index dcf1620..5c51df9 100644
--- a/hack/lib/golang.sh
+++ b/hack/lib/golang.sh
@@ -268,7 +268,7 @@ kube::golang::outfile_for_binary() {
   local output_path="${KUBE_GOPATH}"
   local bin
   bin=$(basename "${binary}")
-  if [[ "${platform}" != "${host_platform}" ]]; then
+  if [[ "${platform}" != "${host_platform}" || ${KUBE_ALL_WITH_PREFIX:-"false"} == "true" ]]; then
     output_path="${output_path}/${platform//\//_}"
   fi
   if [[ ${GOOS} == "windows" ]]; then
diff --git a/pkg/apis/core/v1/handler.go b/pkg/apis/core/v1/handler.go
index 16aaf96..8d10a63 100644
--- a/pkg/apis/core/v1/handler.go
+++ b/pkg/apis/core/v1/handler.go
@@ -31,6 +31,8 @@ import (
 	"strings"
 	"time"
 
+	"github.com/kubeclipper/kubeclipper/pkg/clusteroperation"
+
 	"github.com/kubeclipper/kubeclipper/pkg/controller/cronbackupcontroller"
 
 	"github.com/emicklei/go-restful"
@@ -179,7 +181,7 @@ func (h *handler) DescribeCluster(request *restful.Request, response *restful.Re
 }
 
 func (h *handler) AddOrRemoveNodes(request *restful.Request, response *restful.Response) {
-	pn := &PatchNodes{}
+	pn := &clusteroperation.PatchNodes{}
 	if err := request.ReadEntity(pn); err != nil {
 		restplus.HandleBadRequest(response, request, err)
 		return
@@ -215,22 +217,26 @@ func (h *handler) AddOrRemoveNodes(request *restful.Request, response *restful.R
 		restplus.HandleBadRequest(response, request, err)
 		return
 	}
-	// backing up old masters and workers
-	nodeSet := c.GetAllNodes()
 
-	// We need IP addresses of all master nodes later.
-	extraMeta, err := h.getClusterMetadata(ctx, c, false)
+	operationType := v1.OperationAddNodes
+	if pn.Operation == clusteroperation.NodesOperationRemove {
+		operationType = v1.OperationRemoveNodes
+	}
+	executable, err := clusteroperation.Executable(ctx, operationType, clu, h.opOperator)
 	if err != nil {
-		if apimachineryErrors.IsNotFound(err) || err == ErrNodesRegionDifferent {
-			restplus.HandleBadRequest(response, request, err)
-			return
-		}
-		restplus.HandleInternalError(response, request, err)
+		restplus.HandleBadRequest(response, request, err)
+		return
+	}
+	if !executable {
+		restplus.HandleBadRequest(response, request, fmt.Errorf("%s does not support concurrent execution", operationType))
 		return
 	}
 
+	// backing up old masters and workers
+	nodeSet := c.GetAllNodes()
+
 	if err = pn.MakeCompare(c); err != nil {
-		if errors.Is(err, ErrInvalidNodesOperation) || errors.Is(err, ErrInvalidNodesRole) {
+		if errors.Is(err, clusteroperation.ErrInvalidNodesOperation) || errors.Is(err, clusteroperation.ErrInvalidNodesRole) {
 			restplus.HandleBadRequest(response, request, err)
 			return
 		}
@@ -242,21 +248,14 @@ func (h *handler) AddOrRemoveNodes(request *restful.Request, response *restful.R
 	// Replace cluster worker nodes with nodes to be operated.
 	nodes, err := h.getNodeInfo(ctx, pn.Nodes, false)
 	if err != nil {
-		if err == ErrNodesRegionDifferent {
-			restplus.HandleBadRequest(response, request, err)
-			return
-		}
 		restplus.HandleInternalError(response, request, err)
 		return
 	}
-
-	if pn.Role == common.NodeRoleWorker {
-		extraMeta.Workers = append(extraMeta.Workers, nodes...)
-	}
+	pn.ConvertNodes = nodes
 
 	if len(nodes) == 0 {
 		err = fmt.Errorf("nodes is already in use")
-		if pn.Operation == NodesOperationRemove {
+		if pn.Operation == clusteroperation.NodesOperationRemove {
 			err = fmt.Errorf("nodes is already removed")
 		}
 		restplus.HandleBadRequest(response, request, err)
@@ -265,7 +264,7 @@ func (h *handler) AddOrRemoveNodes(request *restful.Request, response *restful.R
 
 	for _, n := range nodes {
 		switch pn.Operation {
-		case NodesOperationAdd:
+		case clusteroperation.NodesOperationAdd:
 			if n.Disable {
 				restplus.HandleBadRequest(response, request, fmt.Errorf("this node(%s) is disabled", n.IPv4))
 				return
@@ -274,11 +273,11 @@ func (h *handler) AddOrRemoveNodes(request *restful.Request, response *restful.R
 				restplus.HandleBadRequest(response, request, fmt.Errorf("this node(%s) is already in use", n.IPv4))
 				return
 			}
-			if n.Region != extraMeta.Masters[0].Region {
+			if n.Region != c.Labels[common.LabelTopologyRegion] {
 				restplus.HandleBadRequest(response, request, fmt.Errorf("the node(%s) belongs to different region", n.IPv4))
 				return
 			}
-		case NodesOperationRemove:
+		case clusteroperation.NodesOperationRemove:
 			if !nodeSet.Has(n.ID) {
 				restplus.HandleBadRequest(response, request, fmt.Errorf("the node(%s) is not part of this cluster and cannot be removed", n.IPv4))
 				return
@@ -286,22 +285,6 @@ func (h *handler) AddOrRemoveNodes(request *restful.Request, response *restful.R
 		}
 	}
 
-	op, err := pn.MakeOperation(*extraMeta, c)
-	if err != nil {
-		if errors.Is(err, ErrZeroNode) {
-			// No node needs to be operated.
-			_ = response.WriteHeaderAndEntity(http.StatusOK, c)
-			return
-		} else if errors.Is(err, ErrInvalidNodesOperation) || errors.Is(err, ErrInvalidNodesRole) {
-			restplus.HandleBadRequest(response, request, err)
-			return
-		}
-		restplus.HandleInternalError(response, request, err)
-		return
-	}
-
-	op.Labels[common.LabelTimeoutSeconds] = timeoutSecs
-	op.Status.Status = v1.OperationStatusRunning
 	if !dryRun {
 		for _, node := range pn.Nodes {
 			_, err = MarkToOriginNode(ctx, h.clusterOperator, node.ID)
@@ -310,20 +293,20 @@ func (h *handler) AddOrRemoveNodes(request *restful.Request, response *restful.R
 				return
 			}
 		}
-		c.Status.Phase = v1.ClusterUpdating
-		if c, err = h.clusterOperator.UpdateCluster(ctx, c); err != nil {
+		pendingOperation, err := buildPendingOperation(operationType, timeoutSecs, c.ResourceVersion, pn)
+		if err != nil {
 			restplus.HandleInternalError(response, request, err)
 			return
 		}
-		if op, err = h.opOperator.CreateOperation(ctx, op); err != nil {
+
+		c.Status.Phase = v1.ClusterUpdating
+		c.PendingOperations = append(c.PendingOperations, pendingOperation)
+		if c, err = h.clusterOperator.UpdateCluster(ctx, c); err != nil {
 			restplus.HandleInternalError(response, request, err)
 			return
 		}
 	}
 
-	// distribute tasks
-	go h.doOperation(context.TODO(), op, &service.Options{DryRun: dryRun})
-
 	_ = response.WriteHeaderAndEntity(http.StatusOK, c)
 }
 
@@ -1067,9 +1050,9 @@ func mergeNodes(p *tenantv1.Project, pn *nodeJoinProject) (*tenantv1.Project, er
 	set := sets.NewString(p.Spec.Nodes...)
 
 	switch pn.OP {
-	case NodesOperationAdd:
+	case clusteroperation.NodesOperationAdd:
 		set.Insert(pn.Node)
-	case NodesOperationRemove:
+	case clusteroperation.NodesOperationRemove:
 		set.Delete(pn.Node)
 	default:
 		return nil, fmt.Errorf("invalid operation %s", pn.OP)
@@ -1173,6 +1156,7 @@ func (h *handler) watchOperations(req *restful.Request, resp *restful.Response,
 	restplus.ServeWatch(watcher, v1.SchemeGroupVersion.WithKind("Operation"), req, resp, timeout)
 }
 
+// TODO: it will be deprecated in the future
 func (h *handler) getClusterMetadata(ctx context.Context, c *v1.Cluster, skipNodeNotFound bool) (*component.ExtraMetadata, error) {
 	meta := &component.ExtraMetadata{
 		ClusterName:        c.Name,
diff --git a/pkg/apis/core/v1/registry.go b/pkg/apis/core/v1/registry.go
index 8fc1694..aa489e7 100644
--- a/pkg/apis/core/v1/registry.go
+++ b/pkg/apis/core/v1/registry.go
@@ -21,6 +21,8 @@ package v1
 import (
 	"net/http"
 
+	"github.com/kubeclipper/kubeclipper/pkg/clusteroperation"
+
 	"github.com/kubeclipper/kubeclipper/pkg/authentication/auth"
 	"github.com/kubeclipper/kubeclipper/pkg/models/core"
 	"github.com/kubeclipper/kubeclipper/pkg/models/tenant"
@@ -279,7 +281,7 @@ func SetupWebService(h *handler) *restful.WebService {
 		To(h.AddOrRemoveNodes).
 		Metadata(restfulspec.KeyOpenAPITags, []string{CoreClusterTag}).
 		Doc("Add or remove cluster node.").
-		Reads(PatchNodes{}).
+		Reads(clusteroperation.PatchNodes{}).
 		Param(webservice.PathParameter(query.ParameterName, "cluster name").
 			Required(true).
 			DataType("string")).
@@ -289,7 +291,7 @@ func SetupWebService(h *handler) *restful.WebService {
 		To(h.AddOrRemoveNodes).
 		Metadata(restfulspec.KeyOpenAPITags, []string{CoreClusterTag}).
 		Doc("Add or remove cluster node.").
-		Reads(PatchNodes{}).
+		Reads(clusteroperation.PatchNodes{}).
 		Param(webservice.PathParameter("project", "project name")).
 		Param(webservice.PathParameter(query.ParameterName, "cluster name").
 			Required(true).
diff --git a/pkg/apis/core/v1/schema.go b/pkg/apis/core/v1/schema.go
index c88b9df..fc4cc05 100644
--- a/pkg/apis/core/v1/schema.go
+++ b/pkg/apis/core/v1/schema.go
@@ -19,224 +19,20 @@
 package v1
 
 import (
-	"context"
 	"encoding/json"
-	"errors"
 	"fmt"
 
-	"github.com/kubeclipper/kubeclipper/pkg/clustermanage/kubeadm"
-	"github.com/kubeclipper/kubeclipper/pkg/scheme/common"
-	"github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1/k8s"
-
-	"github.com/google/uuid"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	"github.com/kubeclipper/kubeclipper/pkg/component"
 	corev1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
 )
 
-type (
-	NodesPatchOperation string
-)
-
-const (
-	NodesOperationAdd    = "add"
-	NodesOperationRemove = "remove"
-)
-
-type PatchNodes struct {
-	Operation NodesPatchOperation   `json:"operation"`
-	Nodes     corev1.WorkerNodeList `json:"nodes"`
-	Role      common.NodeRole       `json:"role"`
-}
-
 type PatchComponents struct {
 	Uninstall bool           `json:"uninstall"`
 	Addons    []corev1.Addon `json:"addons"`
 }
 
-var (
-	ErrInvalidNodesOperation      = errors.New("invalid nodes patch operation")
-	ErrInvalidNodesRole           = errors.New("invalid node role")
-	ErrZeroNode                   = errors.New("zero node")
-	ErrUninstallNotExistComponent = errors.New("the component is not installed in the current cluster")
-	ErrInstallExistingComponent   = errors.New("the component has been installed in the current cluster")
-)
-
-// MakeCompare compares and filters node to be operated with master/worker nodes in cluster already,
-// it also modifies master/worker nodes in cluster object.
-func (p *PatchNodes) MakeCompare(cluster *corev1.Cluster) error {
-	switch p.Role {
-	case common.NodeRoleMaster:
-		return p.makeMasterCompare(cluster)
-	case common.NodeRoleWorker:
-		return p.makeWorkerCompare(cluster)
-	default:
-		return ErrInvalidNodesRole
-	}
-}
-
-func (p *PatchNodes) makeMasterCompare(cluster *corev1.Cluster) error {
-	// support later
-	return ErrInvalidNodesRole
-}
-
-func (p *PatchNodes) makeWorkerCompare(cluster *corev1.Cluster) error {
-	// Compare between nodes unfiltered and existed worker nodes in cluster.
-	switch p.Operation {
-	case NodesOperationAdd:
-		// Add nodes to cluster.
-		// Check nodes in cluster already.
-		// Filter out nodes to be added.
-		p.Nodes = p.Nodes.Complement(cluster.Workers...)
-		cluster.Workers = append(cluster.Workers, p.Nodes...)
-	case NodesOperationRemove:
-		// Remove nodes from cluster.
-		// Filter out nodes in cluster already.
-		// Filter out nodes to be removed.
-		// TODO: if len(p.nodes)==0, should return error
-		p.Nodes = cluster.Workers.Intersect(p.Nodes...)
-		cluster.Workers = cluster.Workers.Complement(p.Nodes...)
-	default:
-		return ErrInvalidNodesOperation
-	}
-	return nil
-}
-
-// Must be called after MakeCompare.
-func (p *PatchNodes) MakeOperation(extra component.ExtraMetadata, cluster *corev1.Cluster) (*corev1.Operation, error) {
-	pType, ok := cluster.Labels[common.LabelClusterProviderType]
-	if ok {
-		switch pType {
-		case kubeadm.ProviderKubeadm:
-			return p.doMakeOperation(extra, cluster)
-		default:
-			return nil, errors.New("invalid provider type")
-		}
-	}
-	return p.doMakeOperation(extra, cluster)
-}
-
-func (p *PatchNodes) doMakeOperation(extra component.ExtraMetadata, cluster *corev1.Cluster) (*corev1.Operation, error) {
-	switch p.Role {
-	case common.NodeRoleMaster:
-		return nil, ErrInvalidNodesRole
-	case common.NodeRoleWorker:
-		return p.makeWorkerOperation(extra, cluster)
-	default:
-		return nil, ErrInvalidNodesRole
-	}
-}
-
-func (p *PatchNodes) makeWorkerOperation(extra component.ExtraMetadata, cluster *corev1.Cluster) (*corev1.Operation, error) {
-	// no node need to be operated
-	if len(p.Nodes) == 0 {
-		return nil, ErrZeroNode
-	}
-
-	// make operation for adding worker nodes to cluster
-	op := &corev1.Operation{}
-	op.Name = uuid.New().String()
-	op.Labels = map[string]string{
-		common.LabelClusterName: cluster.Name,
-		// v1.LabelTopologyRegion ???
-	}
-	ctx := context.TODO()
-	ctx = component.WithExtraMetadata(ctx, extra)
-	stepNodes := []corev1.StepNode{}
-	workerIPs := extra.GetWorkerNodeIP()
-	for _, nodeID := range p.Nodes.GetNodeIDs() {
-		stepNode := corev1.StepNode{
-			ID:       nodeID,
-			IPv4:     workerIPs[nodeID],
-			Hostname: extra.GetWorkerHostname(nodeID),
-		}
-		stepNodes = append(stepNodes, stepNode)
-	}
-
-	var action corev1.StepAction
-	switch p.Operation {
-	case NodesOperationAdd:
-		// add nodes to cluster
-		// check nodes in cluster
-		// nodes to be added
-
-		// p.Nodes = cluster.Workers.Complement(p.Nodes...)
-		// cluster.Workers = append(cluster.Workers, p.Nodes...)
-		action = corev1.ActionInstall
-		op.Labels[common.LabelOperationAction] = corev1.OperationAddNodes
-
-		// container runtime
-		steps, err := getCriStep(ctx, cluster, action, stepNodes)
-		if err != nil {
-			return nil, err
-		}
-		op.Steps = append(op.Steps, steps...)
-
-		// kubernetes
-		steps, err = p.getPackageSteps(cluster, action, stepNodes)
-		if err != nil {
-			return nil, err
-		}
-		op.Steps = append(op.Steps, steps...)
-
-		// join component
-		gen := k8s.GenNode{}
-		err = gen.InitStepper(&extra, cluster, p.Role.String()).MakeInstallSteps(&extra, stepNodes, p.Role.String())
-		if err != nil {
-			return nil, err
-		}
-		op.Steps = append(op.Steps, gen.GetSteps(action)...)
-	case NodesOperationRemove:
-		// // remove nodes from cluster
-		// // filter nodes in cluster
-		// // nodes to be removed
-		// p.Nodes = cluster.Workers.Intersect(p.Nodes...)
-		// cluster.Workers = cluster.Workers.Complement(p.Nodes...)
-		action = corev1.ActionUninstall
-		op.Labels[common.LabelOperationAction] = corev1.OperationRemoveNodes
-
-		gen := k8s.GenNode{}
-		err := gen.InitStepper(&extra, cluster, p.Role.String()).MakeUninstallSteps(&extra, stepNodes)
-		if err != nil {
-			return nil, err
-		}
-		op.Steps = append(op.Steps, gen.GetSteps(action)...)
-
-		// kubernetes
-		steps, err := p.getPackageSteps(cluster, action, stepNodes)
-		if err != nil {
-			return nil, err
-		}
-		op.Steps = append(op.Steps, steps...)
-
-		// container runtime
-		steps, err = getCriStep(ctx, cluster, action, stepNodes)
-		if err != nil {
-			return nil, err
-		}
-		op.Steps = append(op.Steps, steps...)
-	default:
-		return nil, ErrInvalidNodesOperation
-	}
-
-	return op, nil
-}
-
-func (p *PatchNodes) getPackageSteps(cluster *corev1.Cluster, action corev1.StepAction, pNodes []corev1.StepNode) ([]corev1.Step, error) {
-	pack := &k8s.Package{}
-	pack = pack.InitStepper(cluster)
-
-	switch action {
-	case corev1.ActionInstall:
-		return pack.InstallSteps(pNodes)
-	case corev1.ActionUninstall:
-		return pack.UninstallSteps(pNodes)
-	}
-
-	return nil, fmt.Errorf("packageSteps dose not support action: %s", action)
-}
-
 // checkComponent check whether the component is installed in the current cluster
 func (p *PatchComponents) checkComponents(cluster *corev1.Cluster) error {
 	for _, v := range p.Addons {
diff --git a/pkg/apis/core/v1/utils.go b/pkg/apis/core/v1/utils.go
index f5cf999..5fb5fce 100644
--- a/pkg/apis/core/v1/utils.go
+++ b/pkg/apis/core/v1/utils.go
@@ -24,6 +24,8 @@ import (
 	"fmt"
 	"path/filepath"
 
+	"github.com/kubeclipper/kubeclipper/pkg/utils/strutil"
+
 	"k8s.io/apimachinery/pkg/util/sets"
 
 	"github.com/google/uuid"
@@ -436,3 +438,23 @@ func CreateBasic(serverURL, clusterName, userName string, caCert []byte) *client
 		CurrentContext: contextName,
 	}
 }
+
+// BuildPendingOperation build PendingOperation struct
+// @param operationType Operation type
+// @param timeout The timeout time of the entire operation
+// @param clusterResourceVersion The resource version of the cluster at the time the request was submitted
+// @param extra Additional data necessary to create the operation
+func buildPendingOperation(operationType, timeout, clusterResourceVersion string, extra interface{}) (v1.PendingOperation, error) {
+	extraData, err := json.Marshal(extra)
+	if err != nil {
+		return v1.PendingOperation{}, nil
+	}
+
+	return v1.PendingOperation{
+		OperationID:            strutil.GetUUID(),
+		OperationType:          operationType,
+		Timeout:                timeout,
+		ClusterResourceVersion: clusterResourceVersion,
+		ExtraData:              extraData,
+	}, nil
+}
diff --git a/pkg/clustermanage/kubeadm/provider.go b/pkg/clustermanage/kubeadm/provider.go
index 279d2cd..21a1cc6 100644
--- a/pkg/clustermanage/kubeadm/provider.go
+++ b/pkg/clustermanage/kubeadm/provider.go
@@ -10,6 +10,17 @@ import (
 	"strings"
 
 	"github.com/google/uuid"
+	"github.com/pkg/errors"
+	"gopkg.in/yaml.v2"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	apimachineryErrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	kuberuntime "k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/selection"
+	"k8s.io/client-go/kubernetes"
+
 	"github.com/kubeclipper/kubeclipper/cmd/kcctl/app/options"
 	agentconfig "github.com/kubeclipper/kubeclipper/pkg/agent/config"
 	"github.com/kubeclipper/kubeclipper/pkg/cli/config"
@@ -22,16 +33,6 @@ import (
 	"github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1/k8s"
 	"github.com/kubeclipper/kubeclipper/pkg/utils/sshutils"
 	tmplutil "github.com/kubeclipper/kubeclipper/pkg/utils/template"
-	"github.com/pkg/errors"
-	"gopkg.in/yaml.v2"
-	corev1 "k8s.io/api/core/v1"
-	rbacv1 "k8s.io/api/rbac/v1"
-	apimachineryErrors "k8s.io/apimachinery/pkg/api/errors"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/labels"
-	kuberuntime "k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/selection"
-	"k8s.io/client-go/kubernetes"
 )
 
 func init() {
diff --git a/pkg/clustermanage/kubeadm/wrapper.go b/pkg/clustermanage/kubeadm/wrapper.go
index 01d0f37..f21af80 100644
--- a/pkg/clustermanage/kubeadm/wrapper.go
+++ b/pkg/clustermanage/kubeadm/wrapper.go
@@ -7,14 +7,15 @@ import (
 	"fmt"
 	"strings"
 
-	"github.com/kubeclipper/kubeclipper/pkg/scheme/common"
-	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
 	"gopkg.in/yaml.v2"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/client-go/kubernetes"
 	"k8s.io/client-go/tools/clientcmd"
+
+	"github.com/kubeclipper/kubeclipper/pkg/scheme/common"
+	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
 )
 
 type Wrapper struct {
diff --git a/pkg/clusteroperation/common.go b/pkg/clusteroperation/common.go
new file mode 100644
index 0000000..29fc7c4
--- /dev/null
+++ b/pkg/clusteroperation/common.go
@@ -0,0 +1,29 @@
+package clusteroperation
+
+import (
+	"context"
+	"fmt"
+
+	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
+	"github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1/cri"
+)
+
+func getCriStep(ctx context.Context, c *v1.Cluster, action v1.StepAction, nodes []v1.StepNode) ([]v1.Step, error) {
+	switch c.ContainerRuntime.Type {
+	case v1.CRIDocker:
+		r := cri.DockerRunnable{}
+		err := r.InitStep(ctx, c, nodes)
+		if err != nil {
+			return nil, err
+		}
+		return r.GetActionSteps(action), nil
+	case v1.CRIContainerd:
+		r := cri.ContainerdRunnable{}
+		err := r.InitStep(ctx, c, nodes)
+		if err != nil {
+			return nil, err
+		}
+		return r.GetActionSteps(action), nil
+	}
+	return nil, fmt.Errorf("%v type CRI is not supported", c.ContainerRuntime.Type)
+}
diff --git a/pkg/clusteroperation/node.go b/pkg/clusteroperation/node.go
new file mode 100644
index 0000000..739a502
--- /dev/null
+++ b/pkg/clusteroperation/node.go
@@ -0,0 +1,228 @@
+package clusteroperation
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"github.com/kubeclipper/kubeclipper/pkg/clustermanage/kubeadm"
+	"github.com/kubeclipper/kubeclipper/pkg/component"
+	"github.com/kubeclipper/kubeclipper/pkg/scheme/common"
+	corev1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
+	"github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1/k8s"
+)
+
+const (
+	NodesOperationAdd    = "add"
+	NodesOperationRemove = "remove"
+)
+
+var (
+	ErrInvalidNodesOperation = errors.New("invalid nodes patch operation")
+	ErrInvalidNodesRole      = errors.New("invalid node role")
+	ErrZeroNode              = errors.New("zero node")
+)
+
+type NodesPatchOperation string
+
+type PatchNodes struct {
+	Operation    NodesPatchOperation   `json:"operation"`
+	Nodes        corev1.WorkerNodeList `json:"nodes"`
+	ConvertNodes []component.Node      `json:"convertNodes"`
+	Role         common.NodeRole       `json:"role"`
+}
+
+var _ Interface = (*NodeOperation)(nil)
+
+type NodeOperation struct {
+	Options
+}
+
+func NewNodeOperation(options Options) *NodeOperation {
+	return &NodeOperation{options}
+}
+
+func (n *NodeOperation) Builder() (*corev1.Operation, error) {
+	var pn PatchNodes
+	if err := json.Unmarshal(n.pendingOperation.ExtraData, &pn); err != nil {
+		return nil, err
+	}
+	// add the worker node to extra
+	if pn.Role == common.NodeRoleWorker {
+		n.extra.Workers = append(n.extra.Workers, pn.ConvertNodes...)
+	}
+	op, err := pn.MakeOperation(*n.extra, n.cluster)
+	if err != nil {
+		return nil, err
+	}
+
+	op.Labels[common.LabelTimeoutSeconds] = n.pendingOperation.Timeout
+	op.Status.Status = corev1.OperationStatusPending
+
+	return op, nil
+}
+
+// MakeCompare compares and filters node to be operated with master/worker nodes in cluster already,
+// it also modifies master/worker nodes in cluster object.
+func (p *PatchNodes) MakeCompare(cluster *corev1.Cluster) error {
+	switch p.Role {
+	case common.NodeRoleMaster:
+		return p.makeMasterCompare(cluster)
+	case common.NodeRoleWorker:
+		return p.makeWorkerCompare(cluster)
+	default:
+		return ErrInvalidNodesRole
+	}
+}
+
+func (p *PatchNodes) makeMasterCompare(cluster *corev1.Cluster) error {
+	// support later
+	return ErrInvalidNodesRole
+}
+
+func (p *PatchNodes) makeWorkerCompare(cluster *corev1.Cluster) error {
+	// Compare between nodes unfiltered and existed worker nodes in cluster.
+	switch p.Operation {
+	case NodesOperationAdd:
+		// Add nodes to cluster.
+		// Check nodes in cluster already.
+		// Filter out nodes to be added.
+		p.Nodes = p.Nodes.Complement(cluster.Workers...)
+		cluster.Workers = append(cluster.Workers, p.Nodes...)
+	case NodesOperationRemove:
+		// Remove nodes from cluster.
+		// Filter out nodes in cluster already.
+		// Filter out nodes to be removed.
+		p.Nodes = cluster.Workers.Intersect(p.Nodes...)
+		cluster.Workers = cluster.Workers.Complement(p.Nodes...)
+	default:
+		return ErrInvalidNodesOperation
+	}
+	return nil
+}
+
+// MakeOperation Must be called after MakeCompare.
+func (p *PatchNodes) MakeOperation(extra component.ExtraMetadata, cluster *corev1.Cluster) (*corev1.Operation, error) {
+	pType, ok := cluster.Labels[common.LabelClusterProviderType]
+	if ok {
+		switch pType {
+		case kubeadm.ProviderKubeadm:
+			return p.doMakeOperation(extra, cluster)
+		default:
+			return nil, errors.New("invalid provider type")
+		}
+	}
+	return p.doMakeOperation(extra, cluster)
+}
+
+func (p *PatchNodes) doMakeOperation(extra component.ExtraMetadata, cluster *corev1.Cluster) (*corev1.Operation, error) {
+	switch p.Role {
+	case common.NodeRoleMaster:
+		return nil, ErrInvalidNodesRole
+	case common.NodeRoleWorker:
+		return p.makeWorkerOperation(extra, cluster)
+	default:
+		return nil, ErrInvalidNodesRole
+	}
+}
+
+func (p *PatchNodes) makeWorkerOperation(extra component.ExtraMetadata, cluster *corev1.Cluster) (*corev1.Operation, error) {
+	// no node need to be operated
+	if len(p.Nodes) == 0 {
+		return nil, ErrZeroNode
+	}
+
+	// make operation for adding worker nodes to cluster
+	op := &corev1.Operation{}
+	// use pass-through operationID
+	op.Name = extra.OperationID
+	op.Labels = map[string]string{
+		common.LabelClusterName: cluster.Name,
+	}
+	// pass extra metadata in context
+	ctx := component.WithExtraMetadata(context.TODO(), extra)
+	// nodes to be added or removed
+	var stepNodes []corev1.StepNode
+	workerIPs := extra.GetWorkerNodeIP()
+	for _, nodeID := range p.Nodes.GetNodeIDs() {
+		stepNode := corev1.StepNode{
+			ID:       nodeID,
+			IPv4:     workerIPs[nodeID],
+			Hostname: extra.GetWorkerHostname(nodeID),
+		}
+		stepNodes = append(stepNodes, stepNode)
+	}
+
+	var action corev1.StepAction
+	switch p.Operation {
+	case NodesOperationAdd:
+		action = corev1.ActionInstall
+		op.Labels[common.LabelOperationAction] = corev1.OperationAddNodes
+
+		// container runtime
+		steps, err := getCriStep(ctx, cluster, action, stepNodes)
+		if err != nil {
+			return nil, err
+		}
+		op.Steps = append(op.Steps, steps...)
+
+		// kubernetes
+		steps, err = p.getPackageSteps(cluster, action, stepNodes)
+		if err != nil {
+			return nil, err
+		}
+		op.Steps = append(op.Steps, steps...)
+
+		// join node
+		gen := k8s.GenNode{}
+		err = gen.InitStepper(&extra, cluster, p.Role.String()).MakeInstallSteps(&extra, stepNodes, p.Role.String())
+		if err != nil {
+			return nil, err
+		}
+		op.Steps = append(op.Steps, gen.GetSteps(action)...)
+	case NodesOperationRemove:
+		action = corev1.ActionUninstall
+		op.Labels[common.LabelOperationAction] = corev1.OperationRemoveNodes
+
+		// drain node
+		gen := k8s.GenNode{}
+		err := gen.InitStepper(&extra, cluster, p.Role.String()).MakeUninstallSteps(&extra, stepNodes)
+		if err != nil {
+			return nil, err
+		}
+		op.Steps = append(op.Steps, gen.GetSteps(action)...)
+
+		// kubernetes
+		steps, err := p.getPackageSteps(cluster, action, stepNodes)
+		if err != nil {
+			return nil, err
+		}
+		op.Steps = append(op.Steps, steps...)
+
+		// container runtime
+		steps, err = getCriStep(ctx, cluster, action, stepNodes)
+		if err != nil {
+			return nil, err
+		}
+		op.Steps = append(op.Steps, steps...)
+	default:
+		return nil, ErrInvalidNodesOperation
+	}
+
+	return op, nil
+}
+
+func (p *PatchNodes) getPackageSteps(cluster *corev1.Cluster, action corev1.StepAction, pNodes []corev1.StepNode) ([]corev1.Step, error) {
+	pack := &k8s.Package{}
+	pack = pack.InitStepper(cluster)
+
+	switch action {
+	case corev1.ActionInstall:
+		return pack.InstallSteps(pNodes)
+	case corev1.ActionUninstall:
+		return pack.UninstallSteps(pNodes)
+	}
+
+	return nil, fmt.Errorf("packageSteps dose not support action: %s", action)
+}
diff --git a/pkg/clusteroperation/operation.go b/pkg/clusteroperation/operation.go
new file mode 100644
index 0000000..2847dc3
--- /dev/null
+++ b/pkg/clusteroperation/operation.go
@@ -0,0 +1,225 @@
+package clusteroperation
+
+import (
+	"context"
+	"fmt"
+	"strconv"
+
+	"github.com/kubeclipper/kubeclipper/pkg/component"
+	"github.com/kubeclipper/kubeclipper/pkg/models/operation"
+	"github.com/kubeclipper/kubeclipper/pkg/query"
+	"github.com/kubeclipper/kubeclipper/pkg/scheme/common"
+	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
+)
+
+// RetryTimes max number of operation retry
+// TODO: Later, if necessary, can set a separate retry number for each operation ?
+const RetryTimes = 3
+
+type Options struct {
+	cluster           *v1.Cluster
+	pendingOperation  v1.PendingOperation
+	extra             *component.ExtraMetadata
+	additionalOptions *AdditionalOptions // additional information
+}
+
+// AdditionalOptions some types of operations require additional information, which can be put here
+type AdditionalOptions struct{}
+
+// Interface all operation types must implement this interface
+type Interface interface {
+	Builder() (*v1.Operation, error)
+}
+
+type OptionInterface interface {
+}
+
+// BuildOperationAdapter create an operation based on the operation type
+func BuildOperationAdapter(
+	cluster *v1.Cluster, pendingOp v1.PendingOperation,
+	extra *component.ExtraMetadata, addition *AdditionalOptions,
+) (*v1.Operation, error) {
+	options := Options{
+		cluster:           cluster,
+		pendingOperation:  pendingOp,
+		extra:             extra,
+		additionalOptions: addition,
+	}
+
+	var instance Interface
+	switch pendingOp.OperationType {
+	case v1.OperationAddNodes, v1.OperationRemoveNodes:
+		instance = NewNodeOperation(options)
+	case v1.OperationInstallComponents, v1.OperationUninstallComponents:
+	case v1.OperationCreateCluster:
+	case v1.OperationDeleteCluster:
+	case v1.OperationUpgradeCluster:
+	case v1.OperationBackupCluster:
+	case v1.OperationDeleteBackup:
+	case v1.OperationRecoverCluster:
+	case v1.OperationUpdateCertification:
+		// TODO support all operations
+	default:
+		return &v1.Operation{}, fmt.Errorf("unsupported %s operation type", pendingOp.OperationType)
+	}
+
+	return instance.Builder()
+}
+
+// IsRetry whether the operation supports retry
+func IsRetry(opType string) bool {
+	switch opType {
+	case v1.OperationBackupCluster, v1.OperationRecoverCluster, v1.OperationUpgradeCluster:
+		return false
+	}
+	return true
+}
+
+// AutomaticRetry whether automatic retry is supported
+// Automatic retry conditions:
+// 1. operation support retry.
+// 2. the maximum number of retries has not been exceeded.
+// 3. the operation is the latest in the cluster.
+// TODO: better retry judgment, perhaps it is made up of global and custom rules ?
+func AutomaticRetry(ctx context.Context, op *v1.Operation, operator operation.Operator) (bool, error) {
+	if !IsRetry(op.Labels[common.LabelOperationAction]) {
+		return false, nil
+	}
+	times, _ := strconv.Atoi(op.Labels[common.LabelOperationRetry])
+	// maximum number of retries exceeded
+	if times > RetryTimes {
+		return false, nil
+	}
+	q := query.New()
+	q.LabelSelector = fmt.Sprintf("%s=%s", common.LabelClusterName, op.Labels[common.LabelClusterName])
+	q.Pagination.Offset = 0
+	q.Pagination.Limit = 1
+	// get the latest operations of the cluster
+	resp, err := operator.ListOperationsEx(ctx, q)
+	if err != nil {
+		return false, err
+	}
+	if resp.TotalCount == 0 {
+		return false, fmt.Errorf("no operation record is queried in %s cluster", op.Labels[common.LabelClusterName])
+	}
+	latestOp := resp.Items[0].(*v1.Operation)
+	// if the current operation is not up-to-date, then failure will not be retried
+	if latestOp.Name != op.Name {
+		return false, nil
+	}
+	return true, nil
+}
+
+// Retry operation retry
+func Retry(op *v1.Operation) (context.Context, *v1.Operation, []v1.Step, error) {
+	if op.Status.Status == v1.OperationStatusSuccessful || op.Status.Status == v1.OperationStatusRunning {
+		return nil, nil, nil, fmt.Errorf("only the latest faild operation can do a retry")
+	}
+
+	// error step index
+	failedIndex := len(op.Status.Conditions) - 1
+	ctx := component.WithRetry(context.TODO(), true)
+
+	// if an installation error occurs, proceed directly from the current step
+	var continueSteps []v1.Step
+	if op.Steps[0].Action == v1.ActionInstall {
+		findStepNode := func(nodes []v1.StepNode, nodeID string) v1.StepNode {
+			for _, v := range nodes {
+				if v.ID == nodeID {
+					return v
+				}
+			}
+			return v1.StepNode{}
+		}
+		var failedNodes []v1.StepNode
+		successStatus := make([]v1.StepStatus, 0)
+		for _, status := range op.Status.Conditions[failedIndex].Status {
+			if status.Status == v1.StepStatusFailed {
+				// select the nodes whose execution fails
+				if node := findStepNode(op.Steps[0].Nodes, status.Node); node.ID != "" {
+					failedNodes = append(failedNodes, node)
+				}
+				continue
+			}
+			successStatus = append(successStatus, status)
+		}
+		// the step to continue
+		continueSteps = op.Steps[failedIndex:]
+
+		// the node that failed to execute the task
+		continueSteps[0].Nodes = failedNodes
+		// set the current step to the auto retry flag so that the step log file can be cleared later
+		continueSteps[0].AutomaticRetry = true
+
+		if len(successStatus) != 0 {
+			// failed status
+			failedStepStatus := op.Status.Conditions[failedIndex]
+			// retain successful status
+			failedStepStatus.Status = successStatus
+			// Remove the failed status and keep the successful status
+			op.Status.Conditions = op.Status.Conditions[0:failedIndex]
+			op.Status.Conditions = append(op.Status.Conditions, failedStepStatus)
+		} else {
+			op.Status.Conditions = op.Status.Conditions[0:failedIndex]
+		}
+		if failedIndex > 0 && op.Status.Conditions[failedIndex-1].Status[0].Response != nil {
+			ctx = component.WithExtraData(ctx, op.Status.Conditions[failedIndex-1].Status[0].Response)
+		}
+	}
+
+	// if an uninstallation error occurs, start over
+	if op.Steps[0].Action == v1.ActionUninstall {
+		continueSteps = op.Steps
+		op.Status.Conditions = make([]v1.OperationCondition, 0)
+	}
+	return ctx, op, continueSteps, nil
+}
+
+// SupportConcurrent whether concurrency is supported
+func SupportConcurrent(opType string) bool {
+	switch opType {
+	case v1.OperationAddNodes, v1.OperationRemoveNodes:
+		return true
+	default:
+		return false
+	}
+}
+
+// Executable the operation is committed for execution
+func Executable(ctx context.Context, opType, cluName string, operator operation.Operator) (bool, error) {
+	q := query.New()
+	q.LabelSelector = fmt.Sprintf("%s=%s,%s=%s", common.LabelClusterName, cluName, common.LabelOperationAction, opType)
+	q.FieldSelector = fmt.Sprintf("status.status=%s", v1.OperationStatusRunning)
+	q.Pagination.Offset = 0
+	q.Pagination.Limit = 1
+	// gets whether the same type of operations are being performed in the cluster
+	resp, err := operator.ListOperationsEx(ctx, q)
+	if err != nil {
+		return false, err
+	}
+	if resp.TotalCount == 0 {
+		return true, nil
+	}
+	return SupportConcurrent(opType), nil
+}
+
+// GetClusterPhase get the cluster phase based on the type of operation
+func GetClusterPhase(opType string) v1.ClusterPhase {
+	switch opType {
+	case v1.OperationCreateCluster:
+		return v1.ClusterInstalling
+	case v1.OperationDeleteCluster:
+		return v1.ClusterTerminating
+	case v1.OperationUpgradeCluster:
+		return v1.ClusterUpgrading
+	case v1.OperationBackupCluster, v1.OperationDeleteBackup:
+		return v1.ClusterBackingUp
+	case v1.OperationRecoverCluster:
+		return v1.ClusterRestoring
+	default:
+		// OperationAddNodes,OperationRemoveNodes
+		// OperationInstallComponents,OperationUninstallComponents
+		// OperationUpdateCertification
+		return v1.ClusterUpdating
+	}
+}
diff --git a/pkg/component/common/image.go b/pkg/component/common/image.go
index 8805938..d6ac640 100644
--- a/pkg/component/common/image.go
+++ b/pkg/component/common/image.go
@@ -7,14 +7,15 @@ import (
 	"runtime"
 	"time"
 
+	"go.uber.org/zap"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
 	"github.com/kubeclipper/kubeclipper/pkg/component"
 	"github.com/kubeclipper/kubeclipper/pkg/component/utils"
 	"github.com/kubeclipper/kubeclipper/pkg/logger"
 	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
 	"github.com/kubeclipper/kubeclipper/pkg/simple/downloader"
 	"github.com/kubeclipper/kubeclipper/pkg/utils/strutil"
-	"go.uber.org/zap"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
 const (
diff --git a/pkg/component/ctx.go b/pkg/component/ctx.go
index e8943b2..b84aee3 100644
--- a/pkg/component/ctx.go
+++ b/pkg/component/ctx.go
@@ -42,6 +42,7 @@ type ExtraMetadata struct {
 	CRI                string
 	ClusterName        string
 	KubeVersion        string
+	OperationID        string
 	OperationType      string
 	KubeletDataDir     string
 	ControlPlaneStatus []v1.ControlPlaneHealth
diff --git a/pkg/controller/clustercontroller/controller.go b/pkg/controller/clustercontroller/controller.go
index 0117d89..eadf93c 100644
--- a/pkg/controller/clustercontroller/controller.go
+++ b/pkg/controller/clustercontroller/controller.go
@@ -26,6 +26,8 @@ import (
 	"strings"
 	"time"
 
+	"github.com/kubeclipper/kubeclipper/pkg/clusteroperation"
+
 	"go.uber.org/zap"
 	"k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -64,9 +66,11 @@ type ClusterReconciler struct {
 	mgr                 manager.Manager
 	ClusterLister       listerv1.ClusterLister
 	ClusterWriter       cluster.ClusterWriter
+	ClusterOperator     cluster.Operator
 	RegistryLister      listerv1.RegistryLister
 	NodeLister          listerv1.NodeLister
 	NodeWriter          cluster.NodeWriter
+	OperationOperator   operation.Operator
 	OperationWriter     operation.Writer
 	CronBackupWriter    cluster.CronBackupWriter
 	CloudProviderLister listerv1.CloudProviderLister
@@ -209,6 +213,10 @@ func (r *ClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 	if err = r.syncClusterClient(ctx, log, clu); err != nil {
 		return ctrl.Result{}, nil
 	}
+	if err = r.processPendingOperations(ctx, log, clu); err != nil {
+		log.Error("process pending operation error", zap.Error(err))
+		return ctrl.Result{}, nil
+	}
 	return ctrl.Result{}, r.updateCRIRegistries(ctx, clu)
 }
 
@@ -599,3 +607,68 @@ users:
 func getKubeConfig(clusterName string, address string, user string, token string) string {
 	return fmt.Sprintf(kubeconfigFormat, address, clusterName, clusterName, user, user, clusterName, user, clusterName, user, token)
 }
+
+func (r *ClusterReconciler) processPendingOperations(ctx context.Context, log logger.Logging, c *v1.Cluster) error {
+	log.Debug("process pending operations")
+
+	// all pending operations are processed
+	clean := true
+
+	for _, pendingOperation := range c.PendingOperations {
+		log.Debugf("pending operation info is %+v, extraData is %+v", pendingOperation, string(pendingOperation.ExtraData))
+		// the operation has been processed ?
+		op, err := r.OperationOperator.GetOperationEx(ctx, pendingOperation.OperationID, "0")
+		// ignore IsNotFound error
+		if err != nil && !errors.IsNotFound(err) {
+			// if an error occurs, the deletion operation will not be performed
+			clean = false
+			continue
+		}
+		// if no, the operation will be created
+		if op == nil {
+			clean = false
+
+			// get cluster information about the specified resource version when the operation is performed
+			clu, err := r.ClusterOperator.GetClusterEx(ctx, c.Name, pendingOperation.ClusterResourceVersion)
+			if err != nil {
+				log.Error(fmt.Sprintf("get resource-version-%s cluster info failed", pendingOperation.ClusterResourceVersion),
+					zap.String("cluster", c.Name), zap.String("operation-id", pendingOperation.OperationID), zap.Error(err))
+				continue
+			}
+
+			// restore and assemble the cluster metadata, passing it through
+			extraMeta, err := r.assembleClusterExtraMetadata(ctx, clu)
+			if err != nil {
+				log.Error("get cluster extra metadata failed", zap.String("cluster", c.Name), zap.String("operation-id", pendingOperation.OperationID), zap.Error(err))
+				continue
+			}
+
+			// pass-through operationID
+			extraMeta.OperationID = pendingOperation.OperationID
+
+			// build the operation structure based on the type of operation
+			newOperation, err := clusteroperation.BuildOperationAdapter(clu, pendingOperation, extraMeta, nil)
+			if err != nil {
+				log.Error("create operation struct failed", zap.String("cluster", c.Name), zap.String("operation-id", pendingOperation.OperationID), zap.Error(err))
+				continue
+			}
+
+			log.Debugf("create operation struct successful", zap.String("cluster", c.Name), zap.String("operation-id", pendingOperation.OperationID))
+			_, err = r.OperationWriter.CreateOperation(ctx, newOperation)
+			if err != nil {
+				log.Error("create operation failed", zap.String("cluster", c.Name), zap.String("operation-id", pendingOperation.OperationID), zap.Error(err))
+				continue
+			}
+		}
+	}
+
+	if clean && len(c.PendingOperations) > 0 {
+		log.Debugf("clean pending operation in %s cluster", c.Name)
+		// clean pending operations
+		c.PendingOperations = nil
+		if _, err := r.ClusterWriter.UpdateCluster(ctx, c); err != nil {
+			return err
+		}
+	}
+	return nil
+}
diff --git a/pkg/controller/clustercontroller/metadata.go b/pkg/controller/clustercontroller/metadata.go
new file mode 100644
index 0000000..95d9384
--- /dev/null
+++ b/pkg/controller/clustercontroller/metadata.go
@@ -0,0 +1,59 @@
+package clustercontroller
+
+import (
+	"context"
+
+	"github.com/kubeclipper/kubeclipper/pkg/component"
+	"github.com/kubeclipper/kubeclipper/pkg/scheme/common"
+	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
+)
+
+// assembleClusterExtraMetadata assemble cluster extra metadata, used to share data between the various steps
+func (r *ClusterReconciler) assembleClusterExtraMetadata(ctx context.Context, c *v1.Cluster) (*component.ExtraMetadata, error) {
+	meta := &component.ExtraMetadata{
+		ClusterName:        c.Name,
+		Offline:            c.Offline(),
+		LocalRegistry:      c.LocalRegistry,
+		CRI:                c.ContainerRuntime.Type,
+		KubeVersion:        c.KubernetesVersion,
+		KubeletDataDir:     c.Kubelet.RootDir,
+		ControlPlaneStatus: c.Status.ControlPlaneHealth,
+		CNI:                c.CNI.Type,
+		CNINamespace:       c.CNI.Namespace,
+	}
+	meta.Addons = append(meta.Addons, c.Addons...)
+
+	masters, err := r.convertNodes(ctx, c.Masters)
+	if err != nil {
+		return nil, err
+	}
+	meta.Masters = append(meta.Masters, masters...)
+	workers, err := r.convertNodes(ctx, c.Workers)
+	if err != nil {
+		return nil, err
+	}
+	meta.Workers = append(meta.Workers, workers...)
+	return meta, nil
+}
+
+// convertNodes convert nodes information, this is especially important for handling node information in subsequent operations
+func (r *ClusterReconciler) convertNodes(ctx context.Context, nodes v1.WorkerNodeList) ([]component.Node, error) {
+	var meta []component.Node
+	for _, node := range nodes {
+		n, err := r.ClusterOperator.GetNodeEx(ctx, node.ID, "0")
+		if err != nil {
+			return nil, err
+		}
+		item := component.Node{
+			ID:       n.Name,
+			IPv4:     n.Status.Ipv4DefaultIP,
+			Region:   n.Labels[common.LabelTopologyRegion],
+			Hostname: n.Labels[common.LabelHostname],
+			Role:     n.Labels[common.LabelNodeRole],
+		}
+		_, item.Disable = n.Labels[common.LabelNodeDisable]
+		meta = append(meta, item)
+	}
+
+	return meta, nil
+}
diff --git a/pkg/controller/operationcontroller/controller.go b/pkg/controller/operationcontroller/controller.go
index 3bbb570..0ffacb2 100644
--- a/pkg/controller/operationcontroller/controller.go
+++ b/pkg/controller/operationcontroller/controller.go
@@ -21,6 +21,13 @@ package operationcontroller
 import (
 	"context"
 	"fmt"
+	"strconv"
+
+	"k8s.io/client-go/util/retry"
+
+	"github.com/kubeclipper/kubeclipper/pkg/clusteroperation"
+	"github.com/kubeclipper/kubeclipper/pkg/models/cluster"
+	"github.com/kubeclipper/kubeclipper/pkg/service"
 
 	"go.uber.org/zap"
 	"k8s.io/apimachinery/pkg/api/errors"
@@ -40,9 +47,12 @@ import (
 )
 
 type OperationReconciler struct {
-	ClusterLister   listerv1.ClusterLister
-	OperationLister listerv1.OperationLister
-	OperationWriter operation.Writer
+	CmdDelivery       service.CmdDelivery
+	ClusterLister     listerv1.ClusterLister
+	ClusterOperator   cluster.Operator
+	OperationLister   listerv1.OperationLister
+	OperationWriter   operation.Writer
+	OperationOperator operation.Operator
 }
 
 func (r *OperationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
@@ -101,6 +111,12 @@ func (r *OperationReconciler) Reconcile(ctx context.Context, req ctrl.Request) (
 		log.Error("Failed to get cluster with name", zap.String("cluster", cluName), zap.Error(err))
 		return ctrl.Result{}, err
 	}
+
+	if err = r.distributeOperation(ctx, log, op); err != nil {
+		log.Error("Failed to distribute operation", zap.String("cluster", cluName), zap.String("operation", op.Name), zap.Error(err))
+		return ctrl.Result{}, err
+	}
+
 	return ctrl.Result{}, nil
 }
 
@@ -121,3 +137,69 @@ func (r *OperationReconciler) SetupWithManager(mgr manager.Manager, cache inform
 	mgr.AddRunnable(c)
 	return nil
 }
+
+func (r *OperationReconciler) distributeOperation(ctx context.Context, log logger.Logging, op *v1.Operation) error {
+	cluName := op.Labels[common.LabelClusterName]
+	if op.Status.Status == v1.OperationStatusPending {
+		op.Status.Status = v1.OperationStatusRunning
+		if _, err := r.OperationWriter.UpdateOperation(ctx, op); err != nil {
+			return err
+		}
+
+		go func() {
+			if err := r.CmdDelivery.DeliverTaskOperation(context.TODO(), op, nil); err != nil {
+				log.Error("distribute task error", zap.Error(err))
+			}
+		}()
+	}
+	if op.Status.Status == v1.OperationStatusFailed {
+		times, _ := strconv.Atoi(op.Labels[common.LabelOperationRetry])
+		op.Labels[common.LabelOperationRetry] = strconv.Itoa(times + 1)
+		automaticRetry, err := clusteroperation.AutomaticRetry(ctx, op, r.OperationOperator)
+		if err != nil {
+			log.Error("automatic retry failure", zap.String("cluster", cluName), zap.String("operation", op.Name), zap.Error(err))
+			return err
+		}
+		log.Infof("automatic retry result: %v, current-times: %s", automaticRetry, strconv.Itoa(times+1))
+		// automatic retry
+		if automaticRetry {
+			// get retry operation step
+			deliverCtx, newOp, continueSteps, err := clusteroperation.Retry(op)
+			if err != nil {
+				log.Error("automatic retry failure, processing retry step error", zap.String("cluster", cluName), zap.String("operation", op.Name), zap.Error(err))
+				return err
+			}
+			// TODO: update operation and cluster is non-atomic
+			if err = retry.RetryOnConflict(retry.DefaultRetry, func() error {
+				// get the latest cluster info
+				clu, err := r.ClusterOperator.GetClusterEx(ctx, cluName, "0")
+				if err != nil {
+					return err
+				}
+				// update cluster status
+				clu.Status.Phase = clusteroperation.GetClusterPhase(op.Labels[common.LabelOperationAction])
+				if _, err = r.ClusterOperator.UpdateCluster(ctx, clu.DeepCopy()); err != nil {
+					return err
+				}
+				return nil
+			}); err != nil {
+				log.Error("automatic retry, update cluster failed", zap.String("cluster", cluName), zap.String("operation", op.Name), zap.Error(err))
+				return err
+			}
+			// update operation
+			newOp.Status.Status = v1.OperationStatusRunning
+			if _, err = r.OperationWriter.UpdateOperation(ctx, newOp); err != nil {
+				log.Error("retry on conflict, update operation failed", zap.String("cluster", cluName), zap.String("operation", op.Name), zap.Error(err))
+				return err
+			}
+			// process delivery steps
+			newOp.Steps = continueSteps
+			go func() {
+				if err = r.CmdDelivery.DeliverTaskOperation(deliverCtx, newOp, nil); err != nil {
+					log.Error("automatic retry, distribute task error", zap.String("cluster", cluName), zap.String("operation", op.Name), zap.Error(err))
+				}
+			}()
+		}
+	}
+	return nil
+}
diff --git a/pkg/scheme/common/well_known_labels.go b/pkg/scheme/common/well_known_labels.go
index d9addd1..d1f92a0 100644
--- a/pkg/scheme/common/well_known_labels.go
+++ b/pkg/scheme/common/well_known_labels.go
@@ -33,6 +33,7 @@ const (
 	LabelRecoveryName      = "kubeclipper.io/recovery"
 	LabelOperationAction   = "kubeclipper.io/operation"
 	LabelOperationName     = "kubeclipper.io/operation-name"
+	LabelOperationRetry    = "kubeclipper.io/operation-retry-times"
 	LabelTimeoutSeconds    = "kubeclipper.io/timeout"
 	LabelRoleTemplate      = "kubeclipper.io/role-template"
 	LabelHidden            = "kubeclipper.io/hidden"
diff --git a/pkg/scheme/core/v1/cluster_types.go b/pkg/scheme/core/v1/cluster_types.go
index d339edc..f628f1a 100644
--- a/pkg/scheme/core/v1/cluster_types.go
+++ b/pkg/scheme/core/v1/cluster_types.go
@@ -38,23 +38,24 @@ type Cluster struct {
 	metav1.ObjectMeta `json:"metadata,omitempty"`
 	// move offline to metadata annotation
 	// Offline           bool   `json:"offline" optional:"true"`
-	LocalRegistry     string           `json:"localRegistry,omitempty" optional:"true"`
-	Masters           WorkerNodeList   `json:"masters"`
-	Workers           WorkerNodeList   `json:"workers" optional:"true"`
-	KubernetesVersion string           `json:"kubernetesVersion" enum:"v1.20.13"`
-	CertSANs          []string         `json:"certSANs,omitempty" optional:"true"`
-	ExternalCaCert    string           `json:"externalCaCert,omitempty" optional:"true"`
-	ExternalCaKey     string           `json:"externalCaKey,omitempty" optional:"true"`
-	KubeProxy         KubeProxy        `json:"kubeProxy,omitempty" optional:"true"`
-	Etcd              Etcd             `json:"etcd,omitempty" optional:"true"`
-	Kubelet           Kubelet          `json:"kubelet,omitempty" optional:"true"`
-	Networking        Networking       `json:"networking"`
-	ContainerRuntime  ContainerRuntime `json:"containerRuntime"`
-	CNI               CNI              `json:"cni"`
-	KubeConfig        []byte           `json:"kubeConfig,omitempty"`
-	Addons            []Addon          `json:"addons" optional:"true"`
-	Description       string           `json:"description,omitempty" optional:"true"`
-	Status            ClusterStatus    `json:"status,omitempty" optional:"true"`
+	LocalRegistry     string             `json:"localRegistry,omitempty" optional:"true"`
+	Masters           WorkerNodeList     `json:"masters"`
+	Workers           WorkerNodeList     `json:"workers" optional:"true"`
+	KubernetesVersion string             `json:"kubernetesVersion" enum:"v1.20.13"`
+	CertSANs          []string           `json:"certSANs,omitempty" optional:"true"`
+	ExternalCaCert    string             `json:"externalCaCert,omitempty" optional:"true"`
+	ExternalCaKey     string             `json:"externalCaKey,omitempty" optional:"true"`
+	KubeProxy         KubeProxy          `json:"kubeProxy,omitempty" optional:"true"`
+	Etcd              Etcd               `json:"etcd,omitempty" optional:"true"`
+	Kubelet           Kubelet            `json:"kubelet,omitempty" optional:"true"`
+	Networking        Networking         `json:"networking"`
+	ContainerRuntime  ContainerRuntime   `json:"containerRuntime"`
+	CNI               CNI                `json:"cni"`
+	KubeConfig        []byte             `json:"kubeConfig,omitempty"`
+	Addons            []Addon            `json:"addons" optional:"true"`
+	Description       string             `json:"description,omitempty" optional:"true"`
+	Status            ClusterStatus      `json:"status,omitempty" optional:"true"`
+	PendingOperations []PendingOperation `json:"pendingOperations,omitempty" optional:"true"`
 }
 
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
@@ -129,7 +130,7 @@ type ClusterStatus struct {
 
 	Certifications []Certification `json:"certifications,omitempty"`
 	// Registries all CRI registry
-	Registries []RegistrySpec
+	Registries []RegistrySpec `json:"registries,omitempty"`
 	// ControlPlane Health
 	ControlPlaneHealth []ControlPlaneHealth `json:"controlPlaneHealth,omitempty"`
 }
@@ -236,10 +237,10 @@ var (
 type CNI struct {
 	LocalRegistry string `json:"localRegistry" optional:"true"`
 	// TODO: Cluster multiple cni plugins are not supported at this time
-	Type      string `json:"type" enum:"calico"`
-	Version   string
-	CriType   string
-	Offline   bool
+	Type      string  `json:"type" enum:"calico"`
+	Version   string  `json:"version"`
+	CriType   string  `json:"criType"`
+	Offline   bool    `json:"offline"`
 	Namespace string  `json:"namespace"`
 	Calico    *Calico `json:"calico" optional:"true"`
 }
diff --git a/pkg/scheme/core/v1/operation_types.go b/pkg/scheme/core/v1/operation_types.go
index a6eb42f..213641e 100644
--- a/pkg/scheme/core/v1/operation_types.go
+++ b/pkg/scheme/core/v1/operation_types.go
@@ -64,6 +64,7 @@ const DefaultOperationTimeoutSecs = "5400"
 type OperationStatusType string
 
 const (
+	OperationStatusPending    OperationStatusType = "pending"
 	OperationStatusRunning    OperationStatusType = "running"
 	OperationStatusFailed     OperationStatusType = "failed"
 	OperationStatusUnknown    OperationStatusType = "unknown"
@@ -111,6 +112,7 @@ type Step struct {
 	BeforeRunCommands []Command       `json:"beforeRunCommands,omitempty"`
 	AfterRunCommands  []Command       `json:"afterRunCommands,omitempty"`
 	RetryTimes        int32           `json:"retryTimes,omitempty"`
+	AutomaticRetry    bool            `json:"automaticRetry"`
 }
 
 type StepNode struct {
@@ -166,3 +168,11 @@ type StepStatus struct {
 	Message  string `json:"message,omitempty"`
 	Response []byte `json:"response,omitempty"`
 }
+
+type PendingOperation struct {
+	OperationID            string `json:"operationID"`
+	OperationType          string `json:"operationType"`
+	Timeout                string `json:"timeout"`
+	ClusterResourceVersion string `json:"clusterResourceVersion"`
+	ExtraData              []byte `json:"extraData,omitempty"`
+}
diff --git a/pkg/scheme/core/v1/zz_generated.deepcopy.go b/pkg/scheme/core/v1/zz_generated.deepcopy.go
index c935214..d42fff3 100644
--- a/pkg/scheme/core/v1/zz_generated.deepcopy.go
+++ b/pkg/scheme/core/v1/zz_generated.deepcopy.go
@@ -432,6 +432,13 @@ func (in *Cluster) DeepCopyInto(out *Cluster) {
 		}
 	}
 	in.Status.DeepCopyInto(&out.Status)
+	if in.PendingOperations != nil {
+		in, out := &in.PendingOperations, &out.PendingOperations
+		*out = make([]PendingOperation, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
 	return
 }
 
@@ -1424,6 +1431,27 @@ func (in *ParseRecord) DeepCopy() *ParseRecord {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *PendingOperation) DeepCopyInto(out *PendingOperation) {
+	*out = *in
+	if in.ExtraData != nil {
+		in, out := &in.ExtraData, &out.ExtraData
+		*out = make([]byte, len(*in))
+		copy(*out, *in)
+	}
+	return
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PendingOperation.
+func (in *PendingOperation) DeepCopy() *PendingOperation {
+	if in == nil {
+		return nil
+	}
+	out := new(PendingOperation)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *PlatformSetting) DeepCopyInto(out *PlatformSetting) {
 	*out = *in
@@ -1763,7 +1791,7 @@ func (in *Step) DeepCopyInto(out *Step) {
 		*out = make([]StepNode, len(*in))
 		copy(*out, *in)
 	}
-	out.Timeout = in.Timeout
+	in.Timeout.DeepCopyInto(&out.Timeout)
 	if in.Commands != nil {
 		in, out := &in.Commands, &out.Commands
 		*out = make([]Command, len(*in))
diff --git a/pkg/server/server.go b/pkg/server/server.go
index 0aa6e5e..d7df5b2 100644
--- a/pkg/server/server.go
+++ b/pkg/server/server.go
@@ -441,9 +441,11 @@ func (s *APIServer) SetupController(mgr manager.Manager, informerFactory informe
 		CmdDelivery:         mgr.GetCmdDelivery(),
 		ClusterLister:       informerFactory.Core().V1().Clusters().Lister(),
 		ClusterWriter:       clusterOperator,
+		ClusterOperator:     clusterOperator,
 		RegistryLister:      informerFactory.Core().V1().Registries().Lister(),
 		NodeLister:          informerFactory.Core().V1().Nodes().Lister(),
 		NodeWriter:          clusterOperator,
+		OperationOperator:   opOperator,
 		OperationWriter:     opOperator,
 		CronBackupWriter:    clusterOperator,
 		CloudProviderLister: informerFactory.Core().V1().CloudProviders().Lister(),
@@ -473,9 +475,12 @@ func (s *APIServer) SetupController(mgr manager.Manager, informerFactory informe
 		return err
 	}
 	if err = (&operationcontroller.OperationReconciler{
-		ClusterLister:   informerFactory.Core().V1().Clusters().Lister(),
-		OperationLister: informerFactory.Core().V1().Operations().Lister(),
-		OperationWriter: opOperator,
+		CmdDelivery:       mgr.GetCmdDelivery(),
+		ClusterLister:     informerFactory.Core().V1().Clusters().Lister(),
+		ClusterOperator:   clusterOperator,
+		OperationLister:   informerFactory.Core().V1().Operations().Lister(),
+		OperationWriter:   opOperator,
+		OperationOperator: opOperator,
 	}).SetupWithManager(mgr, informerFactory); err != nil {
 		return err
 	}
diff --git a/pkg/service/task/handler.go b/pkg/service/task/handler.go
index d27ff0b..02aac57 100644
--- a/pkg/service/task/handler.go
+++ b/pkg/service/task/handler.go
@@ -59,6 +59,17 @@ func (s *Service) runTaskStep(ctx context.Context, payload *service.MsgPayload,
 			zap.Bool("retry", payload.Retry),
 		)
 	}
+	if payload.Step.AutomaticRetry {
+		// the automatic retry step clears the log file
+		if err := s.oplog.TruncateStepLogFile(payload.OperationIdentity, stepKey); err != nil {
+			logger.Error("truncate step log file failed: "+err.Error(),
+				zap.String("operation", payload.OperationIdentity),
+				zap.String("step", stepKey),
+				zap.String("cmd", "run task step"),
+				zap.Bool("retry", payload.Retry),
+			)
+		}
+	}
 
 	cmds := make([]v1.Command, len(payload.Step.BeforeRunCommands)+len(payload.Step.Commands)+len(payload.Step.AfterRunCommands))
 	cmds = append(cmds, payload.Step.BeforeRunCommands...)
diff --git a/pkg/simple/client/kc/api.go b/pkg/simple/client/kc/api.go
index fd3c9b4..d557ade 100644
--- a/pkg/simple/client/kc/api.go
+++ b/pkg/simple/client/kc/api.go
@@ -24,11 +24,13 @@ import (
 	"fmt"
 	"net/url"
 
+	corev1 "github.com/kubeclipper/kubeclipper/pkg/apis/core/v1"
+	"github.com/kubeclipper/kubeclipper/pkg/clusteroperation"
+
 	apimachineryversion "k8s.io/apimachinery/pkg/version"
 
 	tenantv1 "github.com/kubeclipper/kubeclipper/pkg/scheme/tenant/v1"
 
-	corev1 "github.com/kubeclipper/kubeclipper/pkg/apis/core/v1"
 	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
 	iamv1 "github.com/kubeclipper/kubeclipper/pkg/scheme/iam/v1"
 )
@@ -206,7 +208,7 @@ func (cli *Client) CreateUser(ctx context.Context, user *iamv1.User) (*UsersList
 	return &users, err
 }
 
-func (cli *Client) AddOrRemoveNode(ctx context.Context, patchNode *corev1.PatchNodes, clusterName string) (*ClustersList, error) {
+func (cli *Client) AddOrRemoveNode(ctx context.Context, patchNode *clusteroperation.PatchNodes, clusterName string) (*ClustersList, error) {
 	serverResp, err := cli.put(ctx, fmt.Sprintf("%s/%s/%s", clustersPath, clusterName, "nodes"), nil, patchNode, nil)
 	defer ensureReaderClosed(serverResp)
 	if err != nil {


Test patch:
diff --git a/pkg/apis/core/v1/schema_test.go b/pkg/apis/core/v1/schema_test.go
index 57002cc..ae11b2f 100644
--- a/pkg/apis/core/v1/schema_test.go
+++ b/pkg/apis/core/v1/schema_test.go
@@ -89,25 +89,6 @@ var (
 			WorkerNodeVip: "169.254.169.100",
 		},
 	}
-	master = v1.WorkerNodeList{
-		{
-			ID: "1e3ea00f-1403-46e5-a486-70e4cb29d541",
-		},
-		{
-			ID: "43ed594a-a76f-4370-a14d-551e7b6153de",
-		},
-		{
-			ID: "c7a91d86-cd53-4c3f-85b0-fbc657778067",
-		},
-	}
-	worker = v1.WorkerNodeList{
-		{
-			ID: "4cf1ad74-704c-4290-a523-e524e930245d",
-		},
-		{
-			ID: "ae4ba282-27f9-4a93-8fe9-63f786781d48",
-		},
-	}
 	addons = []v1.Addon{
 		{
 			Name:    "nfs-provisioner",
@@ -123,148 +104,6 @@ func init() {
 
 }
 
-func Test_MakeCompare(t *testing.T) {
-	type args struct {
-		cluster   *v1.Cluster
-		patchNode *PatchNodes
-	}
-	tests := []struct {
-		name    string
-		arg     args
-		wantErr error
-	}{
-		{
-			name: "addWorker",
-			arg: args{
-				cluster: c2,
-				patchNode: &PatchNodes{
-					Operation: "add",
-					Nodes:     worker,
-					Role:      "worker",
-				},
-			},
-			wantErr: nil,
-		},
-		{
-			name: "removeWorker",
-			arg: args{
-				cluster: c2,
-				patchNode: &PatchNodes{
-					Operation: "remove",
-					Nodes:     worker,
-					Role:      "worker",
-				},
-			},
-			wantErr: nil,
-		},
-		{
-			name: "addMaster",
-			arg: args{
-				cluster: c2,
-				patchNode: &PatchNodes{
-					Operation: "add",
-					Nodes:     master,
-					Role:      "master",
-				},
-			},
-			wantErr: ErrInvalidNodesRole,
-		},
-		{
-			name: "removeMaster",
-			arg: args{
-				cluster: nil,
-				patchNode: &PatchNodes{
-					Operation: "remove",
-					Nodes:     master,
-					Role:      "master",
-				},
-			},
-			wantErr: ErrInvalidNodesRole,
-		},
-	}
-	for _, test := range tests {
-		t.Run(test.name, func(t *testing.T) {
-			if err := test.arg.patchNode.MakeCompare(test.arg.cluster); err != nil && err != test.wantErr {
-				t.Errorf(" MakeCompare() err: %v ", err)
-			}
-		})
-	}
-}
-
-func Test_MakeOperation(t *testing.T) {
-	type args struct {
-		cluster    *v1.Cluster
-		meta       component.ExtraMetadata
-		patchNodes *PatchNodes
-	}
-	tests := []struct {
-		name    string
-		arg     args
-		wantErr error
-	}{
-		{
-			name: "test add worker node operation",
-			arg: args{
-				cluster: c2,
-				meta:    *extraMeta,
-				patchNodes: &PatchNodes{
-					Operation: "add",
-					Nodes: []v1.WorkerNode{
-						{
-							ID: "6b8456e8-2489-4321-bbb0-f8d75c065384",
-						},
-					},
-					Role: "worker",
-				},
-			},
-			wantErr: nil,
-		},
-		{
-			name: "test remove worker node operation",
-			arg: args{
-				cluster: c2,
-				meta:    *extraMeta,
-				patchNodes: &PatchNodes{
-					Operation: "remove",
-					Nodes: []v1.WorkerNode{
-						{
-							ID: "4cf1ad74-704c-4290-a523-e524e930245d",
-						},
-					},
-					Role: "worker",
-				},
-			},
-			wantErr: nil,
-		},
-		{
-			name: "test add master node operation",
-			arg: args{
-				cluster: c2,
-				meta:    *extraMeta,
-				patchNodes: &PatchNodes{
-					Operation: "remove",
-					Nodes: []v1.WorkerNode{
-						{
-							ID: "1e3ea00f-1403-46e5-a486-70e4cb29d541",
-						},
-					},
-					Role: "master",
-				},
-			},
-			wantErr: ErrInvalidNodesRole,
-		},
-	}
-	for _, test := range tests {
-		t.Run(test.name, func(t *testing.T) {
-			_, err := test.arg.patchNodes.MakeOperation(test.arg.meta, test.arg.cluster)
-			if err != nil && err != test.wantErr {
-				t.Errorf(" MakeOperation() error: %v ", err)
-			}
-		})
-	}
-
-}
-
 func Test_checkComponents(t *testing.T) {
 	type args struct {
 		cluster *v1.Cluster
diff --git a/pkg/clusteroperation/node_test.go b/pkg/clusteroperation/node_test.go
new file mode 100644
index 0000000..e73ee91
--- /dev/null
+++ b/pkg/clusteroperation/node_test.go
@@ -0,0 +1,258 @@
+package clusteroperation
+
+import (
+	"testing"
+
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	"github.com/kubeclipper/kubeclipper/pkg/component"
+	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
+)
+
+var (
+	c2 = &v1.Cluster{
+		TypeMeta: metav1.TypeMeta{},
+		ObjectMeta: metav1.ObjectMeta{
+			Name: "demo",
+		},
+
+		Masters: v1.WorkerNodeList{
+			{
+				ID: "1e3ea00f-1403-46e5-a486-70e4cb29d541",
+			},
+			{
+				ID: "43ed594a-a76f-4370-a14d-551e7b6153de",
+			},
+			{
+				ID: "c7a91d86-cd53-4c3f-85b0-fbc657778067",
+			},
+		},
+		Workers: v1.WorkerNodeList{
+			{
+				ID: "4cf1ad74-704c-4290-a523-e524e930245d",
+			},
+			{
+				ID: "ae4ba282-27f9-4a93-8fe9-63f786781d48",
+			},
+		},
+		ContainerRuntime: v1.ContainerRuntime{
+			Type:        "docker",
+			Version:     "19.03.12",
+			DataRootDir: "/var/lib/docker",
+		},
+		KubeProxy: v1.KubeProxy{},
+		Etcd:      v1.Etcd{},
+		CNI: v1.CNI{
+			LocalRegistry: "172.20.150.138:5000",
+			Type:          "calico",
+			Version:       "v3.21.2",
+			Calico: &v1.Calico{
+				IPv4AutoDetection: "first-found",
+				IPv6AutoDetection: "first-found",
+				Mode:              "Overlay-Vxlan-All",
+				IPManger:          true,
+				MTU:               1440,
+			},
+		},
+		Networking: v1.Networking{
+			IPFamily:      v1.IPFamilyIPv4,
+			Services:      v1.NetworkRanges{CIDRBlocks: []string{"10.96.0.0/16"}},
+			Pods:          v1.NetworkRanges{CIDRBlocks: []string{"172.25.0.0/24"}},
+			DNSDomain:     "cluster.local",
+			ProxyMode:     "ipvs",
+			WorkerNodeVip: "169.254.169.100",
+		},
+	}
+	master = v1.WorkerNodeList{
+		{
+			ID: "1e3ea00f-1403-46e5-a486-70e4cb29d541",
+		},
+		{
+			ID: "43ed594a-a76f-4370-a14d-551e7b6153de",
+		},
+		{
+			ID: "c7a91d86-cd53-4c3f-85b0-fbc657778067",
+		},
+	}
+	worker = v1.WorkerNodeList{
+		{
+			ID: "4cf1ad74-704c-4290-a523-e524e930245d",
+		},
+		{
+			ID: "ae4ba282-27f9-4a93-8fe9-63f786781d48",
+		},
+	}
+	extraMeta = &component.ExtraMetadata{
+		Masters: []component.Node{
+			{
+				ID:     "1e3ea00f-1403-46e5-a486-70e4cb29d541",
+				IPv4:   "192.168.1.1",
+				Region: "default",
+			},
+			{
+				ID:     "43ed594a-a76f-4370-a14d-551e7b6153de",
+				IPv4:   "192.168.1.2",
+				Region: "default",
+			},
+			{
+				ID:     "c7a91d86-cd53-4c3f-85b0-fbc657778067",
+				IPv4:   "192.168.1.3",
+				Region: "default",
+			},
+		},
+		Workers: []component.Node{
+			{
+				ID:     "4cf1ad74-704c-4290-a523-e524e930245d",
+				IPv4:   "192.168.1.4",
+				Region: "default",
+			},
+			{
+				ID:     "ae4ba282-27f9-4a93-8fe9-63f786781d48",
+				IPv4:   "192.168.1.5",
+				Region: "default",
+			},
+		},
+	}
+)
+
+func Test_MakeCompare(t *testing.T) {
+	type args struct {
+		cluster   *v1.Cluster
+		patchNode *PatchNodes
+	}
+	tests := []struct {
+		name    string
+		arg     args
+		wantErr error
+	}{
+		{
+			name: "addWorker",
+			arg: args{
+				cluster: c2,
+				patchNode: &PatchNodes{
+					Operation: "add",
+					Nodes:     worker,
+					Role:      "worker",
+				},
+			},
+			wantErr: nil,
+		},
+		{
+			name: "removeWorker",
+			arg: args{
+				cluster: c2,
+				patchNode: &PatchNodes{
+					Operation: "remove",
+					Nodes:     worker,
+					Role:      "worker",
+				},
+			},
+			wantErr: nil,
+		},
+		{
+			name: "addMaster",
+			arg: args{
+				cluster: c2,
+				patchNode: &PatchNodes{
+					Operation: "add",
+					Nodes:     master,
+					Role:      "master",
+				},
+			},
+			wantErr: ErrInvalidNodesRole,
+		},
+		{
+			name: "removeMaster",
+			arg: args{
+				cluster: nil,
+				patchNode: &PatchNodes{
+					Operation: "remove",
+					Nodes:     master,
+					Role:      "master",
+				},
+			},
+			wantErr: ErrInvalidNodesRole,
+		},
+	}
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			if err := test.arg.patchNode.MakeCompare(test.arg.cluster); err != nil && err != test.wantErr {
+				t.Errorf(" MakeCompare() err: %v ", err)
+			}
+		})
+	}
+}
+
+func Test_MakeOperation(t *testing.T) {
+	type args struct {
+		cluster    *v1.Cluster
+		meta       component.ExtraMetadata
+		patchNodes *PatchNodes
+	}
+	tests := []struct {
+		name    string
+		arg     args
+		wantErr error
+	}{
+		{
+			name: "test add worker node operation",
+			arg: args{
+				cluster: c2,
+				meta:    *extraMeta,
+				patchNodes: &PatchNodes{
+					Operation: "add",
+					Nodes: []v1.WorkerNode{
+						{
+							ID: "6b8456e8-2489-4321-bbb0-f8d75c065384",
+						},
+					},
+					Role: "worker",
+				},
+			},
+			wantErr: nil,
+		},
+		{
+			name: "test remove worker node operation",
+			arg: args{
+				cluster: c2,
+				meta:    *extraMeta,
+				patchNodes: &PatchNodes{
+					Operation: "remove",
+					Nodes: []v1.WorkerNode{
+						{
+							ID: "4cf1ad74-704c-4290-a523-e524e930245d",
+						},
+					},
+					Role: "worker",
+				},
+			},
+			wantErr: nil,
+		},
+		{
+			name: "test add master node operation",
+			arg: args{
+				cluster: c2,
+				meta:    *extraMeta,
+				patchNodes: &PatchNodes{
+					Operation: "remove",
+					Nodes: []v1.WorkerNode{
+						{
+							ID: "1e3ea00f-1403-46e5-a486-70e4cb29d541",
+						},
+					},
+					Role: "master",
+				},
+			},
+			wantErr: ErrInvalidNodesRole,
+		},
+	}
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			_, err := test.arg.patchNodes.MakeOperation(test.arg.meta, test.arg.cluster)
+			if err != nil && err != test.wantErr {
+				t.Errorf(" MakeOperation() error: %v ", err)
+			}
+		})
+	}
+
+}
diff --git a/test/e2e/cluster/add_node.go b/test/e2e/cluster/add_node.go
index b2c3747..161dd76 100644
--- a/test/e2e/cluster/add_node.go
+++ b/test/e2e/cluster/add_node.go
@@ -4,10 +4,11 @@ import (
 	"context"
 	"errors"
 
+	"github.com/kubeclipper/kubeclipper/pkg/clusteroperation"
+
 	"github.com/onsi/ginkgo"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
-	v1 "github.com/kubeclipper/kubeclipper/pkg/apis/core/v1"
 	"github.com/kubeclipper/kubeclipper/pkg/query"
 	"github.com/kubeclipper/kubeclipper/pkg/scheme/common"
 	corev1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
@@ -49,7 +50,7 @@ var _ = SIGDescribe("[Slow] [Serial] Add a node to the cluster", func() {
 
 	ginkgo.It("add a node to the cluster and ensure the node is added", func() {
 		ginkgo.By("add node")
-		_, err := f.Client.AddOrRemoveNode(context.TODO(), initPatchNode(v1.NodesOperationAdd, nodeID), clu.Name)
+		_, err := f.Client.AddOrRemoveNode(context.TODO(), initPatchNode(clusteroperation.NodesOperationAdd, nodeID), clu.Name)
 		framework.ExpectNoError(err)
 
 		ginkgo.By("check cluster status is running")
@@ -87,7 +88,7 @@ var _ = SIGDescribe("[Slow] [Serial] Remove a node from the cluster", func() {
 
 	ginkgo.It("remove a node from the cluster and ensure the node is removed", func() {
 		ginkgo.By("remove node")
-		_, err := f.Client.AddOrRemoveNode(context.TODO(), initPatchNode(v1.NodesOperationRemove, clu.Workers[0].ID), clu.Name)
+		_, err := f.Client.AddOrRemoveNode(context.TODO(), initPatchNode(clusteroperation.NodesOperationRemove, clu.Workers[0].ID), clu.Name)
 		framework.ExpectNoError(err)
 
 		ginkgo.By("check cluster status is running")
@@ -157,9 +158,9 @@ func initClusterWithWorkNode(clusterName string, nodeID []string) *corev1.Cluste
 	}
 }
 
-func initPatchNode(operation, node string) *v1.PatchNodes {
-	return &v1.PatchNodes{
-		Operation: v1.NodesPatchOperation(operation),
+func initPatchNode(operation, node string) *clusteroperation.PatchNodes {
+	return &clusteroperation.PatchNodes{
+		Operation: clusteroperation.NodesPatchOperation(operation),
 		Role:      "worker",
 		Nodes: corev1.WorkerNodeList{
 			{