Repository: kubeclipper/kubeclipper
Pull request #325 (2022-11-02 02:58:30)

fix: worker node CRI registry init
<!-- Thanks for sending a pull request! Here are some tips for you:

1. If you want **faster** PR reviews, read how: https://github.com/kubeclipper/community/blob/master/developer-guide/development/the-pr-author-guide-to-getting-through-code-review.md
2. In case you want to know how your PR got reviewed, read: https://github.com/kubeclipper/community/blob/master/developer-guide/development/code-review-guide.md
3. Here are some coding convetions followed by Kubeclipper community: https://github.com/kubeclipper/community/blob/master/developer-guide/development/coding-conventions.md
-->

/kind bug

### What type of PR is this?
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->


### What this PR does / why we need it:

### Which issue(s) this PR fixes:
<!--
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

### Special notes for reviewers:
```
```

### Does this PR introduced a user-facing change?
<!--
If no, just write "None" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md
-->
```release-note
None
```

### Additional documentation, usage docs, etc.:
<!--
This section can be blank if this pull request does not require a release note.
Please use the following format for linking documentation or pass the
section below:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```

kubeclipper-bot: Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.

:memo: **Please follow instructions in the [contributing guide](https://github.com/kubeclipper/kubeclipper/blob/master/CONTRIBUTING.md) to update your commits with the DCO**

Full details of the Developer Certificate of Origin can be found at [developercertificate.org](https://developercertificate.org/).

**The list of commits missing DCO signoff**:

- [d55782a](https://github.com/kubeclipper/kubeclipper/commits/d55782a8c7d6dca844a9b55e7cc69723d3981c85) fix: worker node CRI registry init

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>

kubeclipper-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubeclipper/kubeclipper/pull/325#" title="Author self-approved">lyp256</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **wu-wenxiang** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @wu-wenxiang` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubeclipper%2Fkubeclipper).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubeclipper/kubeclipper/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["wu-wenxiang"]} -->
codecov-commenter: # [Codecov](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) Report
> Merging [#325](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) (48a7a43) into [master](https://codecov.io/gh/kubeclipper/kubeclipper/commit/5c656562f7b31c6e3222130aebf7891eb764c0ea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) (5c65656) will **increase** coverage by `0.02%`.
> The diff coverage is `38.88%`.

[![Impacted file tree graph](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/graphs/tree.svg?width=650&height=150&src=pr&token=3PWFG2MHTE&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper)](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper)

```diff
@@            Coverage Diff             @@
##           master     #325      +/-   ##
==========================================
+ Coverage   12.96%   12.98%   +0.02%     
==========================================
  Files          99      102       +3     
  Lines       15561    15601      +40     
==========================================
+ Hits         2017     2026       +9     
- Misses      13301    13332      +31     
  Partials      243      243              
```


| [Impacted Files](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) | Coverage Δ | |
|---|---|---|
| [pkg/scheme/core/v1/cri/containerd.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL3NjaGVtZS9jb3JlL3YxL2NyaS9jb250YWluZXJkLmdv) | `12.74% <0.00%> (+1.85%)` | :arrow_up: |
| [pkg/scheme/core/v1/cri/cri.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL3NjaGVtZS9jb3JlL3YxL2NyaS9jcmkuZ28=) | `52.94% <ø> (ø)` | |
| [pkg/scheme/core/v1/cri/docker.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL3NjaGVtZS9jb3JlL3YxL2NyaS9kb2NrZXIuZ28=) | `0.00% <0.00%> (ø)` | |
| [pkg/apis/core/v1/utils.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2FwaXMvY29yZS92MS91dGlscy5nbw==) | `54.27% <75.00%> (+0.15%)` | :arrow_up: |
| [pkg/apis/core/v1/schema.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2FwaXMvY29yZS92MS9zY2hlbWEuZ28=) | `69.72% <100.00%> (ø)` | |
| [pkg/scheme/core/v1/k8s/kubeadm.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL3NjaGVtZS9jb3JlL3YxL2s4cy9rdWJlYWRtLmdv) | `1.57% <0.00%> (-1.11%)` | :arrow_down: |
| [pkg/scheme/core/v1/k8s/node.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL3NjaGVtZS9jb3JlL3YxL2s4cy9ub2RlLmdv) | `0.35% <0.00%> (-0.02%)` | :arrow_down: |
| [pkg/apis/core/v1/handler.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2FwaXMvY29yZS92MS9oYW5kbGVyLmdv) | `0.31% <0.00%> (-0.01%)` | :arrow_down: |
| [pkg/cli/registry/registry.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL2NsaS9yZWdpc3RyeS9yZWdpc3RyeS5nbw==) | `0.00% <0.00%> (ø)` | |
| [pkg/scheme/core/v1/k8s/kubeadm\_step.go](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper#diff-cGtnL3NjaGVtZS9jb3JlL3YxL2s4cy9rdWJlYWRtX3N0ZXAuZ28=) | `0.00% <0.00%> (ø)` | |
| ... and [4 more](https://codecov.io/gh/kubeclipper/kubeclipper/pull/325/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubeclipper) | |

lyp256: /assign @x893675 

Patch:
diff --git a/api/openapi-spec/swagger.json b/api/openapi-spec/swagger.json
index e89274c..3e3e4b3 100644
--- a/api/openapi-spec/swagger.json
+++ b/api/openapi-spec/swagger.json
@@ -1279,6 +1279,13 @@
             "in": "path",
             "required": true
           },
+          {
+            "type": "boolean",
+            "default": false,
+            "description": "force delete cluster, will ignore operation error",
+            "name": "force",
+            "in": "query"
+          },
           {
             "type": "boolean",
             "description": "dry run delete clusters",
@@ -5550,7 +5557,8 @@
         "type",
         "Version",
         "CriType",
-        "Offline"
+        "Offline",
+        "namespace"
       ],
       "properties": {
         "CriType": {
@@ -5568,6 +5576,9 @@
         "localRegistry": {
           "type": "string"
         },
+        "namespace": {
+          "type": "string"
+        },
         "type": {
           "type": "string",
           "enum": [
@@ -5577,10 +5588,6 @@
       }
     },
     "v1.CRIRegistry": {
-      "required": [
-        "insecureRegistry",
-        "registryRef"
-      ],
       "properties": {
         "insecureRegistry": {
           "type": "string"
@@ -5826,6 +5833,12 @@
             "$ref": "#/definitions/v1.ComponentConditions"
           }
         },
+        "controlPlaneHealth": {
+          "type": "array",
+          "items": {
+            "$ref": "#/definitions/v1.ControlPlaneHealth"
+          }
+        },
         "phase": {
           "type": "string"
         },
@@ -5987,6 +6000,22 @@
         }
       }
     },
+    "v1.ControlPlaneHealth": {
+      "properties": {
+        "address": {
+          "type": "string"
+        },
+        "hostname": {
+          "type": "string"
+        },
+        "id": {
+          "type": "string"
+        },
+        "status": {
+          "type": "string"
+        }
+      }
+    },
     "v1.CronBackup": {
       "required": [
         "spec"
diff --git a/pkg/apis/core/v1/handler.go b/pkg/apis/core/v1/handler.go
index 7cd71d2..101177f 100644
--- a/pkg/apis/core/v1/handler.go
+++ b/pkg/apis/core/v1/handler.go
@@ -304,6 +304,7 @@ func (h *handler) DeleteCluster(request *restful.Request, response *restful.Resp
 		timeoutSecs = v1.DefaultOperationTimeoutSecs
 	}
 	dryRun := query.GetBoolValueWithDefault(request, query.ParamDryRun, false)
+	force := query.GetBoolValueWithDefault(request, query.ParameterForce, false)
 	c, err := h.clusterOperator.GetClusterEx(request.Request.Context(), name, "0")
 	if err != nil {
 		if apimachineryErrors.IsNotFound(err) {
@@ -366,7 +367,7 @@ func (h *handler) DeleteCluster(request *restful.Request, response *restful.Resp
 		if err := h.delivery.DeliverTaskOperation(context.TODO(), o, opts); err != nil {
 			logger.Error("delivery task error", zap.Error(err))
 		}
-	}(op, &service.Options{DryRun: dryRun})
+	}(op, &service.Options{DryRun: dryRun, ForceSkipError: force})
 	response.WriteHeader(http.StatusOK)
 }
 
diff --git a/pkg/apis/core/v1/registry.go b/pkg/apis/core/v1/registry.go
index ae37669..a01f540 100644
--- a/pkg/apis/core/v1/registry.go
+++ b/pkg/apis/core/v1/registry.go
@@ -121,6 +121,8 @@ func SetupWebService(h *handler) *restful.WebService {
 		Metadata(restfulspec.KeyOpenAPITags, []string{CoreClusterTag}).
 		Doc("Delete clusters.").
 		Param(webservice.PathParameter("name", "cluster name")).
+		Param(webservice.QueryParameter(query.ParameterForce, "force delete cluster, will ignore operation error").
+			Required(false).DataType("boolean").DefaultValue("false")).
 		Param(webservice.QueryParameter(query.ParamDryRun, "dry run delete clusters").
 			Required(false).DataType("boolean")).
 		Returns(http.StatusOK, http.StatusText(http.StatusOK), nil))
diff --git a/pkg/apis/core/v1/schema.go b/pkg/apis/core/v1/schema.go
index 654749d..3850b00 100644
--- a/pkg/apis/core/v1/schema.go
+++ b/pkg/apis/core/v1/schema.go
@@ -161,13 +161,13 @@ func (p *PatchNodes) makeWorkerOperation(extra component.ExtraMetadata, cluster
 		// check nodes in cluster
 		// nodes to be added
 
-		//p.Nodes = cluster.Workers.Complement(p.Nodes...)
-		//cluster.Workers = append(cluster.Workers, p.Nodes...)
+		// p.Nodes = cluster.Workers.Complement(p.Nodes...)
+		// cluster.Workers = append(cluster.Workers, p.Nodes...)
 		action = corev1.ActionInstall
 		op.Labels[common.LabelOperationAction] = corev1.OperationAddNodes
 
 		// container runtime
-		steps, err := getCriStep(ctx, &cluster.ContainerRuntime, action, stepNodes)
+		steps, err := getCriStep(ctx, cluster, action, stepNodes)
 		if err != nil {
 			return nil, err
 		}
@@ -209,7 +209,7 @@ func (p *PatchNodes) makeWorkerOperation(extra component.ExtraMetadata, cluster
 		op.Steps = append(op.Steps, steps...)
 
 		// container runtime
-		steps, err = getCriStep(ctx, &cluster.ContainerRuntime, action, stepNodes)
+		steps, err = getCriStep(ctx, cluster, action, stepNodes)
 		if err != nil {
 			return nil, err
 		}
diff --git a/pkg/apis/core/v1/utils.go b/pkg/apis/core/v1/utils.go
index 07cb282..caf1642 100644
--- a/pkg/apis/core/v1/utils.go
+++ b/pkg/apis/core/v1/utils.go
@@ -46,8 +46,8 @@ func reverseComponents(components []v1.Addon) {
 	}
 }
 
-func getCriStep(ctx context.Context, c *v1.ContainerRuntime, action v1.StepAction, nodes []v1.StepNode) ([]v1.Step, error) {
-	switch c.Type {
+func getCriStep(ctx context.Context, c *v1.Cluster, action v1.StepAction, nodes []v1.StepNode) ([]v1.Step, error) {
+	switch c.ContainerRuntime.Type {
 	case v1.CRIDocker:
 		r := cri.DockerRunnable{}
 		err := r.InitStep(ctx, c, nodes)
@@ -63,7 +63,7 @@ func getCriStep(ctx context.Context, c *v1.ContainerRuntime, action v1.StepActio
 		}
 		return r.GetActionSteps(action), nil
 	}
-	return nil, fmt.Errorf("%v type CRI is not supported", c.Type)
+	return nil, fmt.Errorf("%v type CRI is not supported", c.ContainerRuntime.Type)
 }
 
 func getK8sSteps(ctx context.Context, c *v1.Cluster, action v1.StepAction) ([]v1.Step, error) {
@@ -103,7 +103,7 @@ func (h *handler) parseOperationFromCluster(extraMetadata *component.ExtraMetada
 	// Container runtime should be installed on all nodes.
 	ctx := component.WithExtraMetadata(context.TODO(), *extraMetadata)
 	stepNodes := utils.UnwrapNodeList(extraMetadata.GetAllNodes())
-	cSteps, err := getCriStep(ctx, &c.ContainerRuntime, action, stepNodes)
+	cSteps, err := getCriStep(ctx, c, action, stepNodes)
 	if err != nil {
 		return nil, err
 	}
@@ -224,10 +224,11 @@ func (h *handler) parseRecoverySteps(c *v1.Cluster, b *v1.Backup, restoreDir str
 
 func getRecoveryStep(c *v1.Cluster, bp *v1.BackupPoint, b *v1.Backup, restoreDir string, masters, workers []component.Node, nodeNames, nodeIPs []string, action v1.StepAction) (steps []v1.Step, err error) {
 	meta := component.ExtraMetadata{
-		ClusterName: c.Name,
-		Masters:     masters,
-		Workers:     workers,
-		CNI:         c.CNI.Type,
+		ClusterName:  c.Name,
+		Masters:      masters,
+		Workers:      workers,
+		CNI:          c.CNI.Type,
+		CNINamespace: c.CNI.Namespace,
 	}
 	ctx := component.WithExtraMetadata(context.TODO(), meta)
 
diff --git a/pkg/component/ctx.go b/pkg/component/ctx.go
index 4c1bac1..a644f5a 100644
--- a/pkg/component/ctx.go
+++ b/pkg/component/ctx.go
@@ -45,8 +45,9 @@ type ExtraMetadata struct {
 	OperationType      string
 	KubeletDataDir     string
 	ControlPlaneStatus []v1.ControlPlaneHealth
-	CNI                string
-	CNINamespace       string
+
+	CNI          string
+	CNINamespace string
 }
 
 type Node struct {
@@ -142,6 +143,15 @@ func (e ExtraMetadata) GetAvailableMasterNodes() []string {
 	return nodes
 }
 
+func (e ExtraMetadata) IsAllMasterAvailable() bool {
+	for _, node := range e.ControlPlaneStatus {
+		if node.Status != v1.ComponentHealthy {
+			return false
+		}
+	}
+	return true
+}
+
 func WithExtraData(ctx context.Context, data []byte) context.Context {
 	return context.WithValue(ctx, extraKey{}, data)
 }
diff --git a/pkg/controller/cluster_status.go b/pkg/controller/cluster_status.go
index f1e8433..a896318 100644
--- a/pkg/controller/cluster_status.go
+++ b/pkg/controller/cluster_status.go
@@ -26,6 +26,7 @@ import (
 	"time"
 
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/util/sets"
 
 	"github.com/kubeclipper/kubeclipper/pkg/clustermanage/kubeadm"
 	"github.com/kubeclipper/kubeclipper/pkg/utils/httputil"
@@ -52,6 +53,10 @@ const (
 	clusterStatusMonitorPeriod = 3 * time.Minute
 )
 
+var (
+	exemptStatus = sets.NewString(string(v1.ClusterInstalling), string(v1.ClusterInstallFailed), string(v1.ClusterTerminating), string(v1.ClusterTerminateFailed))
+)
+
 type ClusterStatusMon struct {
 	ClusterWriter       cluster.ClusterWriter
 	ClusterLister       listerv1.ClusterLister
@@ -77,15 +82,18 @@ func (s *ClusterStatusMon) monitorClusterStatus() {
 		return
 	}
 	for _, clu := range clusters {
-		cc, exist := s.mgr.GetClusterClientSet(clu.Name)
-		if !exist {
-			s.log.Debug("clientset not exist, clientset may have not been finished", zap.String("cluster", clu.Name))
+		if exemptStatus.Has(string(clu.Status.Phase)) {
 			continue
 		}
 		err = s.updateClusterControlPlaneStatus(clu.DeepCopy())
 		if err != nil {
 			s.log.Error("update cluster control plane status failed", zap.Error(err))
 		}
+		cc, exist := s.mgr.GetClusterClientSet(clu.Name)
+		if !exist {
+			s.log.Debug("clientset not exist, clientset may have not been finished", zap.String("cluster", clu.Name))
+			continue
+		}
 		err = s.updateClusterCertification(clu.Name)
 		if err != nil {
 			s.log.Error("update cluster certification failed", zap.Error(err))
@@ -295,11 +303,13 @@ func (s *ClusterStatusMon) updateClusterControlPlaneStatus(clu *v1.Cluster) erro
 		}
 		controlplanestatus[index] = health
 	}
+
 	if isControlPlaneHealthChange(clu.Status.ControlPlaneHealth, controlplanestatus) {
 		clu.Status.ControlPlaneHealth = controlplanestatus
 		_, err := s.ClusterWriter.UpdateCluster(context.TODO(), clu)
 		return err
 	}
+
 	s.log.Debug("control plane status has no change, skip update", zap.String("cluster", clu.Name))
 	return nil
 }
diff --git a/pkg/query/query.go b/pkg/query/query.go
index ab556a9..5726da3 100644
--- a/pkg/query/query.go
+++ b/pkg/query/query.go
@@ -54,6 +54,7 @@ const (
 	ParamOffline                  = "offline"
 	ParameterSubDomain            = "subdomain"
 	ParameterFuzzySearch          = "fuzzy"
+	ParameterForce                = "force"
 )
 
 const (
diff --git a/pkg/scheme/core/v1/cri/containerd.go b/pkg/scheme/core/v1/cri/containerd.go
index 8ba37c3..6f1f7f9 100644
--- a/pkg/scheme/core/v1/cri/containerd.go
+++ b/pkg/scheme/core/v1/cri/containerd.go
@@ -56,13 +56,14 @@ type ContainerdRunnable struct {
 	upgradeSteps   []v1.Step
 }
 
-func (runnable *ContainerdRunnable) InitStep(ctx context.Context, containerd *v1.ContainerRuntime, nodes []v1.StepNode) error {
+func (runnable *ContainerdRunnable) InitStep(ctx context.Context, cluster *v1.Cluster, nodes []v1.StepNode) error {
 	metadata := component.GetExtraMetadata(ctx)
-	runnable.Version = containerd.Version
+	runnable.Version = cluster.ContainerRuntime.Version
 	runnable.Offline = metadata.Offline
-	runnable.DataRootDir = strutil.StringDefaultIfEmpty(containerdDefaultConfigDir, containerd.DataRootDir)
+	runnable.DataRootDir = strutil.StringDefaultIfEmpty(containerdDefaultConfigDir, cluster.ContainerRuntime.DataRootDir)
 	runnable.LocalRegistry = metadata.LocalRegistry
-	runnable.InsecureRegistry = containerd.InsecureRegistry
+	runnable.Registies = cluster.Status.Registries
+
 	runnable.PauseVersion = runnable.matchPauseVersion(metadata.KubeVersion)
 	runtimeBytes, err := json.Marshal(runnable)
 	if err != nil {
@@ -212,10 +213,6 @@ func (runnable *ContainerdRunnable) setupContainerdConfig(ctx context.Context, d
 	if runnable.RegistryConfigDir == "" {
 		runnable.RegistryConfigDir = ContainerdDefaultRegistryConfigDir
 	}
-	if !runnable.Offline && runnable.LocalRegistry == "" {
-		runnable.LocalRegistry = component.GetRepoMirror(ctx)
-		logger.Info("render containerd config, the default repo mirror proxy will be used", zap.String("local_registry", runnable.LocalRegistry))
-	}
 	cf := filepath.Join(containerdDefaultConfigDir, "config.toml")
 	if err := os.MkdirAll(containerdDefaultConfigDir, 0755); err != nil {
 		return err
@@ -265,50 +262,7 @@ func (runnable *ContainerdRunnable) renderRegistryConfig(dryRun bool) error {
 	if dryRun {
 		return nil
 	}
-	regCfgs := make([]ContainerdRegistry, 0, len(runnable.InsecureRegistry)+1)
-
-	for _, regHost := range runnable.InsecureRegistry {
-		regCfgs = append(regCfgs, ContainerdRegistry{
-			Server: regHost,
-			Hosts: []ContainerdHost{
-				{
-					Scheme:       "http",
-					Host:         regHost,
-					Capabilities: []string{CapabilityPull, CapabilityResolve},
-				},
-				{
-					Scheme:       "https",
-					Host:         regHost,
-					Capabilities: []string{CapabilityPull, CapabilityResolve},
-					SkipVerify:   true,
-				},
-			},
-		},
-		)
-	}
-
-	if runnable.LocalRegistry != "" {
-		regCfgs = append(
-			regCfgs,
-			ContainerdRegistry{
-				Server: runnable.LocalRegistry,
-				Hosts: []ContainerdHost{
-					{
-						Scheme:       "http",
-						Host:         runnable.LocalRegistry,
-						Capabilities: []string{CapabilityPull, CapabilityResolve},
-					},
-					{
-						Scheme:       "https",
-						Host:         runnable.LocalRegistry,
-						Capabilities: []string{CapabilityPull, CapabilityResolve},
-						SkipVerify:   true,
-					},
-				},
-			},
-		)
-	}
-
+	regCfgs := ToContainerdRegistryConfig(runnable.Registies)
 	for _, cfg := range regCfgs {
 		if err := cfg.renderConfigs(runnable.RegistryConfigDir); err != nil {
 			return err
diff --git a/pkg/scheme/core/v1/cri/cri.go b/pkg/scheme/core/v1/cri/cri.go
index 0d8dddc..daf9e33 100644
--- a/pkg/scheme/core/v1/cri/cri.go
+++ b/pkg/scheme/core/v1/cri/cri.go
@@ -22,6 +22,7 @@ import (
 	"fmt"
 
 	"github.com/kubeclipper/kubeclipper/pkg/component"
+	v1 "github.com/kubeclipper/kubeclipper/pkg/scheme/core/v1"
 )
 
 func init() {
@@ -91,8 +92,8 @@ var _ component.StepRunnable = (*ContainerdRunnable)(nil)
 var _ component.StepRunnable = (*DockerRunnable)(nil)
 
 type Base struct {
-	Version          string   `json:"version,omitempty"`
-	Offline          bool     `json:"offline"`
-	DataRootDir      string   `json:"rootDir"`
-	InsecureRegistry []string `json:"insecureRegistry,omitempty"`
+	Version     string            `json:"version,omitempty"`
+	Offline     bool              `json:"offline"`
+	DataRootDir string            `json:"rootDir"`
+	Registies   []v1.RegistrySpec `json:"registry,omitempty"`
 }
diff --git a/pkg/scheme/core/v1/cri/docker.go b/pkg/scheme/core/v1/cri/docker.go
index 24d65db..71817b5 100644
--- a/pkg/scheme/core/v1/cri/docker.go
+++ b/pkg/scheme/core/v1/cri/docker.go
@@ -52,13 +52,13 @@ type DockerRunnable struct {
 	upgradeSteps     []v1.Step
 }
 
-func (runnable *DockerRunnable) InitStep(ctx context.Context, cri *v1.ContainerRuntime, nodes []v1.StepNode) error {
+func (runnable *DockerRunnable) InitStep(ctx context.Context, cluster *v1.Cluster, nodes []v1.StepNode) error {
 	metadata := component.GetExtraMetadata(ctx)
 
-	runnable.Version = cri.Version
+	runnable.Version = cluster.ContainerRuntime.Version
 	runnable.Offline = metadata.Offline
-	runnable.DataRootDir = cri.DataRootDir
-	runnable.InsecureRegistry = cri.InsecureRegistry
+	runnable.DataRootDir = cluster.ContainerRuntime.DataRootDir
+	runnable.InsecureRegistry = ToDockerInsecureRegistry(cluster.Status.Registries)
 
 	runtimeBytes, err := json.Marshal(runnable)
 	if err != nil {
diff --git a/pkg/scheme/core/v1/k8s/cluster.go b/pkg/scheme/core/v1/k8s/cluster.go
index 4baa48d..0e36483 100644
--- a/pkg/scheme/core/v1/k8s/cluster.go
+++ b/pkg/scheme/core/v1/k8s/cluster.go
@@ -666,7 +666,7 @@ func (stepper *Recovery) InitSteps(ctx context.Context) error {
 	}
 
 	if len(stepper.installSteps) == 0 {
-		if err := stepper.MakeInstallSteps(&extraMetadata); err != nil {
+		if err := stepper.MakeInstallSteps(ctx, &extraMetadata); err != nil {
 			return err
 		}
 	}
@@ -695,7 +695,10 @@ func (stepper *Recovery) GetInstallSteps() []v1.Step {
 	return stepper.installSteps
 }
 
-func (stepper *Recovery) MakeInstallSteps(metadata *component.ExtraMetadata) error {
+func (stepper *Recovery) MakeInstallSteps(ctx context.Context, metadata *component.ExtraMetadata) error {
+	if !metadata.IsAllMasterAvailable() {
+		return errors.New("there is an unavailable master node in the cluster, please check the cluster master node status")
+	}
 	rBytes, err := json.Marshal(stepper)
 	if err != nil {
 		return err
@@ -735,42 +738,91 @@ func (stepper *Recovery) MakeInstallSteps(metadata *component.ExtraMetadata) err
 				},
 			})
 	}
-	stepper.installSteps = append(stepper.installSteps,
-		v1.Step{
-			ID:         strutil.GetUUID(),
-			Name:       "restartCniAndKubeProxy",
-			Timeout:    metav1.Duration{Duration: 5 * time.Minute},
-			ErrIgnore:  false,
-			RetryTimes: 1,
-			Nodes:      []v1.StepNode{utils.UnwrapNodeList(metadata.Masters)[0]},
-			Action:     v1.ActionInstall,
-			Commands: []v1.Command{
-				{
-					Type:         v1.CommandShell,
-					ShellCommand: []string{"/bin/bash", "-c", "while true; do kubectl get po -n kube-system && break;sleep 5; done"},
-				},
-				{
-					Type:         v1.CommandShell,
-					ShellCommand: []string{"/bin/bash", "-c", "while true; do if [ 0 == $(kubectl get po -n kube-system | grep kube | grep -v Running | wc -l) ]; then break; else sleep 5 && kubectl get po -n kube-system | grep kube | grep -v Running ;fi ; done"},
-				},
-				{
-					Type:         v1.CommandShell,
-					ShellCommand: []string{"/bin/bash", "-c", "kubectl rollout restart ds calico-node -n kube-system && kubectl rollout restart ds kube-proxy -n kube-system"},
-				},
-				{
-					Type:         v1.CommandShell,
-					ShellCommand: []string{"/bin/bash", "-c", "while true; do if [ 0 == $(kubectl get po -n kube-system | grep calico | grep -v Running | wc -l) ]; then break; else sleep 5 && kubectl get po -n kube-system | grep calico | grep -v Running ; fi ; done"},
-				},
-				{
-					Type:         v1.CommandShell,
-					ShellCommand: []string{"/bin/bash", "-c", "while true; do if [ 0 == $(kubectl get po -n kube-system | grep kube-proxy | grep -v Running | wc -l) ]; then break; else sleep 5 && kubectl get po -n kube-system | grep kube-proxy | grep -v Running ; fi ; done"},
-				},
+
+	restart := v1.Step{
+		ID:         strutil.GetUUID(),
+		Name:       "restartCniAndKubeProxy",
+		Timeout:    metav1.Duration{Duration: 5 * time.Minute},
+		ErrIgnore:  false,
+		RetryTimes: 1,
+		Nodes:      []v1.StepNode{utils.UnwrapNodeList(metadata.Masters)[0]},
+		Action:     v1.ActionInstall,
+		Commands: []v1.Command{
+			{
+				Type:         v1.CommandShell,
+				ShellCommand: []string{"/bin/bash", "-c", "while true; do kubectl get po -n kube-system && break;sleep 5; done"},
 			},
+			{
+				Type:         v1.CommandShell,
+				ShellCommand: []string{"/bin/bash", "-c", "while true; do if [ 0 == $(kubectl get po -n kube-system | grep kube | grep -v Running | wc -l) ]; then break; else sleep 5 && kubectl get po -n kube-system | grep kube | grep -v Running ;fi ; done"},
+			},
+		},
+	}
+
+	unSupport, cmdList := stepper.RecoveryCNICmd(metadata)
+
+	if !unSupport {
+		restart.Commands = append(restart.Commands, v1.Command{
+			Type:         v1.CommandShell,
+			ShellCommand: []string{"/bin/bash", "-c", cmdList["restart"]},
 		})
+	}
+
+	restart.Commands = append(restart.Commands, v1.Command{
+		Type:         v1.CommandShell,
+		ShellCommand: []string{"/bin/bash", "-c", "kubectl rollout restart ds kube-proxy -n kube-system"},
+	})
+
+	if !unSupport {
+		restart.Commands = append(restart.Commands, v1.Command{
+			Type: v1.CommandShell,
+			// since the daemon-set controller restarts the pods asynchronously, we try to get the pod status running 3 times in a row and the restart is considered complete
+			// for((i=1;i<=3;i++));do sleep 5;while true; do if [ 0 == $(kubectl get po -n kube-system | grep calico | grep -v Running | wc -l) ]; then break; else sleep 5 && kubectl get po -n kube-system | grep calico | grep -v Running ; fi ; done;done;
+			ShellCommand: []string{"/bin/bash", "-c", fmt.Sprintf("for((i=1;i<=3;i++));do sleep 5;while true; do if [ 0 == $(%s | grep -v Running | wc -l) ]; then break; else sleep 5 && %s | grep -v Running ; fi ; done;done;", cmdList["get"], cmdList["get"])},
+		})
+	}
+
+	restart.Commands = append(restart.Commands, v1.Command{
+		Type: v1.CommandShell,
+		// Since the daemon-set controller restarts the pods asynchronously, we try to get the pod status running 3 times in a row and the restart is considered complete
+		ShellCommand: []string{"/bin/bash", "-c", "for((i=1;i<=3;i++));do sleep 5;while true; do if [ 0 == $(kubectl get po -n kube-system | grep kube-proxy | grep -v Running | wc -l) ]; then break; else sleep 5 && kubectl get po -n kube-system | grep kube-proxy | grep -v Running ; fi ; done;done;"},
+	})
+
+	stepper.installSteps = append(stepper.installSteps, restart)
+
+	if unSupport {
+		if ok := stepper.appendFile(ctx, metadata.CNI, restart.Name); ok == nil {
+			return nil
+		}
+	}
 
 	return nil
 }
 
+func (stepper *Recovery) RecoveryCNICmd(metadata *component.ExtraMetadata) (unSupport bool, cmdList map[string]string) {
+	cmdList = make(map[string]string)
+	switch metadata.CNI {
+	case CniCalico:
+		cmdList["get"] = fmt.Sprintf("kubectl get po -n %s | grep calico", metadata.CNINamespace)
+		cmdList["restart"] = fmt.Sprintf("kubectl rollout restart ds calico-node -n %s", metadata.CNINamespace)
+	default:
+		unSupport = true
+	}
+
+	return
+}
+
+func (stepper *Recovery) appendFile(ctx context.Context, cni string, stepName string) error {
+	stepKey := fmt.Sprintf("%s-%s", component.GetStepID(ctx), stepName)
+	operationID := component.GetOperationID(ctx)
+	opLog := component.GetOplog(ctx)
+
+	note := fmt.Sprintf("\n ===== The daemon-set of this cni(%s) cannot be restarted, please use kubectl to restart the daemon-set service of this cni(%s) ===== \n\n", cni, cni)
+	err := opLog.CreateStepLogFileAndAppend(operationID, stepKey, []byte(note))
+	logger.Debugf("Ccreate step log file error: %v", err)
+	return err
+}
+
 func (stepper *Recovery) Install(ctx context.Context, opts component.Options) ([]byte, error) {
 	_, err := stepper.Download(ctx, opts)
 	if err != nil {
diff --git a/pkg/service/delivery/delivery.go b/pkg/service/delivery/delivery.go
index c3422c2..11b1bd3 100644
--- a/pkg/service/delivery/delivery.go
+++ b/pkg/service/delivery/delivery.go
@@ -446,7 +446,7 @@ func (s *Service) DeliverTaskOperation(ctx context.Context, operation *v1.Operat
 		logger.Debug("after delivery task step", zap.Error(err))
 		if err != nil {
 			logger.Error("delivery task step error", zap.Error(err), zap.String("step", step.Name))
-			if step.ErrIgnore {
+			if step.ErrIgnore || opts.ForceSkipError {
 				logger.Debug("delivery task step, ignore the error", zap.Error(err), zap.String("step", step.Name))
 				// reset error
 				err = nil
diff --git a/pkg/service/msg.go b/pkg/service/msg.go
index a4542d7..876600b 100644
--- a/pkg/service/msg.go
+++ b/pkg/service/msg.go
@@ -82,5 +82,6 @@ type LogOperation struct {
 }
 
 type Options struct {
-	DryRun bool
+	DryRun         bool
+	ForceSkipError bool
 }


Test patch:
diff --git a/pkg/scheme/core/v1/cri/containerd_test.go b/pkg/scheme/core/v1/cri/containerd_test.go
index ffb9544..9aec409 100644
--- a/pkg/scheme/core/v1/cri/containerd_test.go
+++ b/pkg/scheme/core/v1/cri/containerd_test.go
@@ -34,10 +34,19 @@ func TestContainerdRunnable_renderTo(t *testing.T) {
 			name: "set insecure registry",
 			fields: fields{
 				Base: Base{
-					Version:          "1.6.4",
-					Offline:          true,
-					DataRootDir:      "/var/lib/containerd",
-					InsecureRegistry: []string{"127.0.0.1:5000", "127.0.0.1:6000"},
+					Version:     "1.6.4",
+					Offline:     true,
+					DataRootDir: "/var/lib/containerd",
+					Registies: []v1.RegistrySpec{
+						{
+							Scheme: "http",
+							Host:   "127.0.0.1:5000",
+						},
+						{
+							Scheme: "http",
+							Host:   "127.0.0.1:6000",
+						},
+					},
 				},
 				LocalRegistry: "",
 				KubeVersion:   "1.23.6",
@@ -48,10 +57,15 @@ func TestContainerdRunnable_renderTo(t *testing.T) {
 			name: "use local registry",
 			fields: fields{
 				Base: Base{
-					Version:          "1.6.4",
-					Offline:          true,
-					DataRootDir:      "/var/lib/containerd",
-					InsecureRegistry: []string{"127.0.0.1:5000"},
+					Version:     "1.6.4",
+					Offline:     true,
+					DataRootDir: "/var/lib/containerd",
+					Registies: []v1.RegistrySpec{
+						{
+							Scheme: "http",
+							Host:   "127.0.0.1:5000",
+						},
+					},
 				},
 				LocalRegistry: "127.0.0.1:5000",
 				KubeVersion:   "1.23.6",