Repository: apecloud/kubeblocks

Issue:
[BUG] clickhouse keeper scalein hang: no template found for pod demo/ch-cluster-ch-keeper-1 
**Describe the bug**
This bug is happened on main branch, when scaleIn pod>=2.

e.g. clickhouse keeper 3->1, hang on 2 replica.

When scaling in from 3 → 1, the third node is terminated (k get pod disappears). However, when reconciling to statusReconciler, a problem occurs in isPodUpdated(): this function skips offloadInstances (i.e., the second node), but at this point, the second pod still exists.

```bash
Oct 30 11:36:39.000 |INFO| reconcile reconcileID=21650eb4-074d-4bc0-887e-8d537a6d6aa9
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=23f31a6c-2e92-4d2d-9b76-9c575a0e991f error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |ERRO| Reconciler error reconcileID=a20c13fd-5410-4b54-a654-653e0465dc87
Oct 30 11:36:39.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=c893e1c3-2eed-4a7f-95b8-544c4eec2605
Oct 30 11:36:39.000 |INFO| reconcile object *v1.ConfigMap with action UPDATE OK 
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=73125ee5-bce8-4541-b18e-9e4c5d0bc122 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| reconcile object *v1.InstanceSet with action UPDATE OK namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=c893e1c3-2eed-4a7f-95b8-544c4eec2605
Oct 30 11:36:39.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false name=ch-cluster-ch-keeper Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=823f538e-7a98-43e1-aecd-87024d2f600e
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=3e42a3a3-3b7c-4ed6-a07e-881f6fd1e9b7 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| reconcile object *v1.ConfigMap with action UPDATE OK namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=c893e1c3-2eed-4a7f-95b8-544c4eec2605
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=7c722c65-ef5d-471d-a9ab-f77d5fc8ca5c
Oct 30 11:36:39.000 |INFO| execute error: Operation cannot be fulfilled on instancesets.workloads.kubeblocks.io "ch-cluster-ch-keeper": the object has been modified; please apply your changes to the latest version and try again controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=823f538e-7a98-43e1-aecd-87024d2f600e
Oct 30 11:36:39.000 |INFO| reconcile object *v1.Component with action STATUS OK name=ch-cluster-clickhouse-rsc Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=c893e1c3-2eed-4a7f-95b8-544c4eec2605
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=fc023738-b6ad-4b24-a785-662a35291121 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| reconcile namespace=demo name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=f1f4f8f2-42be-47f3-b993-8538a0e02e61
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=568373dd-1b39-4dc9-9535-f2f17576de8c error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| reconcile namespace=demo name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=eb0af09a-ee6c-498c-b1b6-d1dd6754c063
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=17b0ed26-5ab6-4651-9386-0a5b87a8f8bf error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |ERRO| Reconciler error reconcileID=e77b3f6b-0ffe-4397-907c-7f5dfc0599c8
Oct 30 11:36:39.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| reconcile object *v1.ConfigMap with action UPDATE OK 
Oct 30 11:36:39.000 |INFO| status conditions, creating: false, its running: false, has failure: false, updating: false name=ch-cluster-clickhouse-rsc Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=52d0947e-8d7e-442b-bf7d-02c235d7a3d8
Oct 30 11:36:39.000 |INFO| reconcile object *v1.Component with action STATUS OK 
Oct 30 11:36:39.000 |INFO| reconcile object *v1.InstanceSet with action UPDATE OK name=ch-cluster-ch-keeper Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| cluster component phase transition:  -> Updating (cluster component ch-keeper is Updating) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster reconcileID=9582d05a-86ed-4d2a-90b1-a4cdde964298
Oct 30 11:36:39.000 |INFO| cluster sharding phase transition:  -> Updating (cluster sharding clickhouse is Updating) 
Oct 30 11:36:39.000 |INFO| reconcile object *v1.ConfigMap with action UPDATE OK controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| InstanceSet demo/ch-cluster-ch-keeper instances are not aligned controller=instanceset InstanceSet.namespace=demo InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=3b10c444-06ac-47e8-bafa-2d223dc8d9bd
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=dd1d320f-cdf8-4ac7-b4f3-1e2262700f4c
Oct 30 11:36:39.000 |INFO| reconcile object *v1.Component with action PATCH OK controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=39688056-be9d-40a9-9f75-a21e203ad83a
Oct 30 11:36:39.000 |INFO| reconcile object *v1.Component with action STATUS OK controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| cluster component phase transition:  -> Failed (cluster component ch-keeper is Failed) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster reconcileID=37d0bab1-e00a-42e6-b83f-15ab9d9dc9f5
Oct 30 11:36:39.000 |INFO| cluster sharding phase transition:  -> Updating (cluster sharding clickhouse is Updating) 
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=db79b5a3-8178-4a92-af04-513678823037
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=6dfe5301-5c9c-4843-8359-4a1fa3494bc5 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| cluster component phase transition:  -> Failed (cluster component ch-keeper is Failed) namespace=demo name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster controllerGroup=apps.kubeblocks.io reconcileID=c70a0373-1ae4-41fd-b548-d25ddf79bd5d
Oct 30 11:36:39.000 |INFO| cluster sharding phase transition:  -> Updating (cluster sharding clickhouse is Updating) 
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=5ad7d9f4-547f-40df-b31a-2c7eb22a6d96
Oct 30 11:36:39.000 |INFO| cluster component phase transition:  -> Failed (cluster component ch-keeper is Failed) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster controllerGroup=apps.kubeblocks.io reconcileID=5f98e759-9c15-4b40-bc33-a356d8f3ca5a
Oct 30 11:36:39.000 |INFO| cluster sharding phase transition:  -> Updating (cluster sharding clickhouse is Updating) 
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=9161834b-cd8f-481f-b975-97c375f77cb6
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=abf92a0d-4ace-49a7-864e-7ac00869cf7e error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=d5a9922e-5396-4e05-b958-92bcfc325331
Oct 30 11:36:40.000 |INFO| execute error: Operation cannot be fulfilled on components.apps.kubeblocks.io "ch-cluster-clickhouse-rsc": the object has been modified; please apply your changes to the latest version and try again 
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=0d404486-7a7f-4467-9ccc-22a7b6660f76 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=64d52312-630c-447a-afe1-48d5a59f2599
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action PATCH OK 
Oct 30 11:36:40.000 |INFO| execute error: Operation cannot be fulfilled on components.apps.kubeblocks.io "ch-cluster-ch-keeper": the object has been modified; please apply your changes to the latest version and try again 
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=8b2fbc8a-366b-4052-bc3a-cea00312e91b error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=7ffe03ec-c056-4ed1-a173-62ec3f03471b
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK 
Oct 30 11:36:40.000 |INFO| cluster component phase transition:  -> Failed (cluster component ch-keeper is Failed) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster reconcileID=2c42a240-858f-4fb2-ba73-fe70fa62eedc
Oct 30 11:36:40.000 |INFO| cluster sharding phase transition:  -> Running (cluster sharding clickhouse is Running) 
Oct 30 11:36:40.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=5c9e4268-d6b8-47e8-83a1-642d1547632d
Oct 30 11:36:40.000 |INFO| cluster component phase transition:  -> Failed (cluster component ch-keeper is Failed) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster controllerGroup=apps.kubeblocks.io reconcileID=cff666aa-acfa-4a03-817f-4a01b4d4f577
Oct 30 11:36:40.000 |INFO| cluster sharding phase transition:  -> Running (cluster sharding clickhouse is Running) 
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=c6e5d57d-f5f5-4e4c-829d-a0219b6afcc6 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=e644ecd3-58cb-4e71-ba73-cebc9b485e40
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK 
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=21f6b36a-ae47-459d-8ff9-01f34e701a35 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=681dd563-d69c-499a-a807-56c440692c67
Oct 30 11:36:40.000 |INFO| execute error: Operation cannot be fulfilled on components.apps.kubeblocks.io "ch-cluster-clickhouse-rsc": the object has been modified; please apply your changes to the latest version and try again 
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false name=ch-cluster-ch-keeper Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=d7afbacb-d713-4465-975f-09edc0d412c7
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK 
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false name=ch-cluster-clickhouse-rsc Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=2306d427-c04c-4d9f-ab52-402c6dc9087d
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK 
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false reconcileID=77df2453-73ae-4109-858a-683671dc69a3
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=3dd10a06-6d39-4e12-b3cf-3ae679ae2e3a error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=77df2453-73ae-4109-858a-683671dc69a3
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=115f53e7-6a33-41ff-b493-66c772460850 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                           
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                        
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                              
    /Users/loomt/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=227b61d4-46e1-473c-9664-92bd48e83cae
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=a9ed1170-cb72-48e4-b3e9-92279c5c47f6
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=3a50794e-a150-4b9a-beb2-b22ee7421b94
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=55acebb4-fc50-4edb-9364-456d1b53ccb9
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=60118cee-a554-407d-9d30-f01c306ed4fe
Oct 30 11:36:44.000 |ERRO| Reconciler error reconcileID=7a5e0f33-0685-45c9-9429-6ac535cfd11b
```

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.


Hints from issue comments:
@shanshanying: reproduced on KB 1.1 alpha 3.  besdies component  status goes from RUNNING to FAILED , on scale-in. @github-actions: This issue has been marked as stale because it has been open for 30 days with no activity

Solution patch:
diff --git a/pkg/controller/instanceset/in_place_update_util.go b/pkg/controller/instanceset/in_place_update_util.go
index c0b677c11..58b8c9576 100644
--- a/pkg/controller/instanceset/in_place_update_util.go
+++ b/pkg/controller/instanceset/in_place_update_util.go
@@ -25,6 +25,7 @@ import (
 	"slices"
 	"strings"
 
+	"github.com/pkg/errors"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
 	"k8s.io/utils/ptr"
@@ -44,6 +45,10 @@ const (
 	inPlaceUpdatePolicy podUpdatePolicy = "inPlaceUpdate"
 )
 
+var (
+	errTemplateNotFound = fmt.Errorf("no template found for pod")
+)
+
 func supportPodVerticalScaling() bool {
 	return viper.GetBool(constant.FeatureGateInPlacePodVerticalScaling)
 }
@@ -314,7 +319,7 @@ func getPodUpdatePolicy(its *workloads.InstanceSet, pod *corev1.Pod) (podUpdateP
 		return templateName == templateExt.Name
 	})
 	if index < 0 {
-		return noOpsPolicy, "", fmt.Errorf("no corresponding template found for instance %s", pod.Name)
+		return noOpsPolicy, "", errors.Wrapf(errTemplateNotFound, "pod: %s/%s", pod.Namespace, pod.Name)
 	}
 	newPod, err := buildInstancePodByTemplate(pod.Name, templateList[index], its, getPodRevision(pod))
 	if err != nil {
@@ -371,13 +376,19 @@ func getTemplateNameByPod(itsExt *instancetemplate.InstanceSetExt, pod *corev1.P
 	if ok {
 		return tplExt.Name, nil
 	}
-	return "", fmt.Errorf("no template found for pod %s/%s", pod.Namespace, pod.Name)
+	return "", errors.Wrapf(errTemplateNotFound, "pod: %s/%s", pod.Namespace, pod.Name)
 }
 
-// IsPodUpdated tells whether the pod's spec is as expected in the InstanceSet.
+// isPodUpdated tells whether the pod's spec is as expected in the InstanceSet.
 // This function is meant to replace the old fashion `GetPodRevision(pod) == updateRevision`,
 // as the pod template revision has been redefined in instanceset.
-func IsPodUpdated(its *workloads.InstanceSet, pod *corev1.Pod) (bool, error) {
+func isPodUpdated(its *workloads.InstanceSet, pod *corev1.Pod) (bool, error) {
 	policy, _, err := getPodUpdatePolicy(its, pod)
+	if err != nil {
+		if errors.Is(err, errTemplateNotFound) {
+			return true, nil
+		}
+		return false, err
+	}
 	return policy == noOpsPolicy, err
 }
diff --git a/pkg/controller/instanceset/reconciler_revision_update.go b/pkg/controller/instanceset/reconciler_revision_update.go
index 081f6b0f1..6166bd630 100644
--- a/pkg/controller/instanceset/reconciler_revision_update.go
+++ b/pkg/controller/instanceset/reconciler_revision_update.go
@@ -106,7 +106,7 @@ func calculateUpdatedReplicas(its *workloads.InstanceSet, pods []client.Object)
 	updatedReplicas := int32(0)
 	for i := range pods {
 		pod, _ := pods[i].(*corev1.Pod)
-		updated, err := IsPodUpdated(its, pod)
+		updated, err := isPodUpdated(its, pod)
 		if err != nil {
 			return 0, nil
 		}
diff --git a/pkg/controller/instanceset/reconciler_status.go b/pkg/controller/instanceset/reconciler_status.go
index fb71b4be3..73a73d1da 100644
--- a/pkg/controller/instanceset/reconciler_status.go
+++ b/pkg/controller/instanceset/reconciler_status.go
@@ -124,7 +124,7 @@ func (r *statusReconciler) Reconcile(tree *kubebuilderx.ObjectTree) (kubebuilder
 			}
 		}
 		if isCreated(pod) && !isTerminating(pod) {
-			isPodUpdated, err := IsPodUpdated(its, pod)
+			isPodUpdated, err := isPodUpdated(its, pod)
 			if err != nil {
 				return kubebuilderx.Continue, err
 			}
diff --git a/pkg/kbagent/service/probe.go b/pkg/kbagent/service/probe.go
index a4c1c009f..1337eaace 100644
--- a/pkg/kbagent/service/probe.go
+++ b/pkg/kbagent/service/probe.go
@@ -40,6 +40,11 @@ const (
 	defaultProbePeriodSeconds = 60
 )
 
+var (
+	defaultRetrySendEventInterval = 1 * time.Minute
+	retrySendEventInterval        = defaultRetrySendEventInterval
+)
+
 func newProbeService(logger logr.Logger, actionService *actionService, probes []proto.Probe) (*probeService, error) {
 	sp := &probeService{
 		logger:        logger,
@@ -58,10 +63,11 @@ func newProbeService(logger logr.Logger, actionService *actionService, probes []
 }
 
 type probeService struct {
-	logger        logr.Logger
-	actionService *actionService
-	probes        map[string]*proto.Probe
-	runners       map[string]*probeRunner
+	logger               logr.Logger
+	actionService        *actionService
+	probes               map[string]*proto.Probe
+	runners              map[string]*probeRunner
+	sendEventWithMessage func(logger *logr.Logger, reason string, message string, sync bool) error
 }
 
 var _ Service = &probeService{}
@@ -77,9 +83,10 @@ func (s *probeService) URI() string {
 func (s *probeService) Start() error {
 	for name := range s.probes {
 		runner := &probeRunner{
-			logger:        s.logger.WithValues("probe", name),
-			actionService: s.actionService,
-			latestEvent:   make(chan proto.ProbeEvent, 1),
+			logger:               s.logger.WithValues("probe", name),
+			actionService:        s.actionService,
+			latestEvent:          make(chan proto.ProbeEvent, 1),
+			sendEventWithMessage: s.sendEventWithMessage,
 		}
 		go runner.run(s.probes[name])
 		s.runners[name] = runner
@@ -87,22 +94,23 @@ func (s *probeService) Start() error {
 	return nil
 }
 
-func (s *probeService) HandleConn(ctx context.Context, conn net.Conn) error {
+func (s *probeService) HandleConn(context.Context, net.Conn) error {
 	return nil
 }
 
-func (s *probeService) HandleRequest(ctx context.Context, payload []byte) ([]byte, error) {
+func (s *probeService) HandleRequest(context.Context, []byte) ([]byte, error) {
 	return nil, errors.Wrapf(proto.ErrNotImplemented, "service %s does not support request handling", s.Kind())
 }
 
 type probeRunner struct {
-	logger        logr.Logger
-	actionService *actionService
-	ticker        *time.Ticker
-	succeedCount  int64
-	failedCount   int64
-	latestOutput  []byte
-	latestEvent   chan proto.ProbeEvent
+	logger               logr.Logger
+	actionService        *actionService
+	ticker               *time.Ticker
+	succeedCount         int64
+	failedCount          int64
+	latestOutput         []byte
+	latestEvent          chan proto.ProbeEvent
+	sendEventWithMessage func(logger *logr.Logger, reason string, message string, sync bool) error
 }
 
 func (r *probeRunner) run(probe *proto.Probe) {
@@ -112,23 +120,23 @@ func (r *probeRunner) run(probe *proto.Probe) {
 		time.Sleep(time.Duration(probe.InitialDelaySeconds) * time.Second)
 	}
 
-	// launch report loop first
+	// launch the report loop first
 	r.launchReportLoop(probe)
 
-	r.launchRunLoop(probe)
+	r.launchProbeLoop(probe)
 }
 
-func (r *probeRunner) launchRunLoop(probe *proto.Probe) {
+func (r *probeRunner) launchProbeLoop(probe *proto.Probe) {
 	if probe.PeriodSeconds <= 0 {
 		probe.PeriodSeconds = defaultProbePeriodSeconds
 	}
 	r.ticker = time.NewTicker(time.Duration(probe.PeriodSeconds) * time.Second)
 	defer r.ticker.Stop()
 
-	r.runLoop(probe)
+	r.probeLoop(probe)
 }
 
-func (r *probeRunner) runLoop(probe *proto.Probe) {
+func (r *probeRunner) probeLoop(probe *proto.Probe) {
 	once := func() {
 		output, err := r.actionService.handleRequest(context.Background(), &proto.ActionRequest{Action: probe.Action})
 		if err == nil {
@@ -154,45 +162,16 @@ func (r *probeRunner) runLoop(probe *proto.Probe) {
 	}
 }
 
-func (r *probeRunner) launchReportLoop(probe *proto.Probe) {
-	if probe.ReportPeriodSeconds <= 0 {
-		return
-	}
-
-	if probe.ReportPeriodSeconds < probe.PeriodSeconds {
-		probe.ReportPeriodSeconds = probe.PeriodSeconds
-	}
-
-	go func() {
-		ticker := time.NewTicker(time.Duration(probe.ReportPeriodSeconds) * time.Second)
-		defer ticker.Stop()
-
-		var latestReportedEvent *proto.ProbeEvent
-		for range ticker.C {
-			latestEvent := gather(r.latestEvent)
-			if latestEvent == nil && latestReportedEvent != nil {
-				latestEvent = latestReportedEvent
-			}
-			if latestEvent != nil {
-				r.logger.Info("report probe event periodically",
-					"code", latestEvent.Code, "output", outputPrefix(latestEvent.Output), "message", latestEvent.Message)
-				r.sendEvent(latestEvent)
-			}
-			latestReportedEvent = latestEvent
-		}
-	}()
-}
-
 func (r *probeRunner) report(probe *proto.Probe, output []byte, err error) {
 	var latestEvent *proto.ProbeEvent
 
 	succeed, thresholdPoint := r.succeed(probe)
 	if succeed && thresholdPoint ||
 		succeed && !thresholdPoint && !reflect.DeepEqual(output, r.latestOutput) {
-		latestEvent = r.buildNSendEvent(probe.Instance, probe.Action, 0, output, "")
+		latestEvent = r.buildEvent(probe.Instance, probe.Action, 0, output, "")
 	}
 	if r.fail(probe) {
-		latestEvent = r.buildNSendEvent(probe.Instance, probe.Action, -1, r.latestOutput, err.Error())
+		latestEvent = r.buildEvent(probe.Instance, probe.Action, -1, r.latestOutput, err.Error())
 	}
 
 	if latestEvent != nil {
@@ -227,29 +206,88 @@ func (r *probeRunner) fail(probe *proto.Probe) bool {
 	return false
 }
 
-func (r *probeRunner) buildNSendEvent(instance, probe string, code int32, output []byte, message string) *proto.ProbeEvent {
-	r.logger.Info("send probe event", "probe", probe, "code", code, "output", outputPrefix(output), "message", message)
-	event := &proto.ProbeEvent{
+func (r *probeRunner) buildEvent(instance, probe string, code int32, output []byte, message string) *proto.ProbeEvent {
+	return &proto.ProbeEvent{
 		Instance: instance,
 		Probe:    probe,
 		Code:     code,
 		Output:   output,
 		Message:  message,
 	}
-	r.sendEvent(event)
-	return event
 }
 
-func (r *probeRunner) sendEvent(event *proto.ProbeEvent) {
-	msg, err := json.Marshal(&event)
-	if err == nil {
-		_ = util.SendEventWithMessage(&r.logger, event.Probe, string(msg), false)
-	} else {
-		r.logger.Error(err, fmt.Sprintf("failed to marshal probe event, probe: %s", event.Probe))
-	}
-}
+func (r *probeRunner) launchReportLoop(probe *proto.Probe) {
+	go func() {
+		var reportChan <-chan time.Time
+		if probe.ReportPeriodSeconds > 0 {
+			if probe.ReportPeriodSeconds < probe.PeriodSeconds {
+				probe.ReportPeriodSeconds = probe.PeriodSeconds
+			}
+			ticker := time.NewTicker(time.Duration(probe.ReportPeriodSeconds) * time.Second)
+			defer ticker.Stop()
+			reportChan = ticker.C
+		}
+
+		retryTicker := time.NewTicker(retrySendEventInterval)
+		defer retryTicker.Stop()
+
+		var event proto.ProbeEvent
+		var hasEvent bool
+
+		output := func() string {
+			prefixLen := min(len(event.Output), 32)
+			return string(event.Output[:prefixLen])
+		}
+
+		log := func(err error, msg string, retry, periodically bool) {
+			if err == nil {
+				r.logger.Info(msg, "retry", retry, "periodically", periodically,
+					"probe", probe.Action, "code", event.Code, "output", output(), "message", event.Message)
+			} else {
+				r.logger.Error(err, msg, "retry", retry, "periodically", periodically,
+					"probe", probe.Action, "code", event.Code, "output", output(), "message", event.Message)
+			}
+		}
+
+		trySend := func(retry, periodically bool) bool {
+			if !hasEvent {
+				return true
+			}
 
-func outputPrefix(output []byte) string {
-	prefixLen := min(len(output), 32)
-	return string(output[:prefixLen])
+			msg, err := json.Marshal(event)
+			if err != nil {
+				log(err, "failed to marshal the probe event", retry, periodically)
+				return true
+			}
+
+			if r.sendEventWithMessage == nil {
+				r.sendEventWithMessage = util.SendEventWithMessage
+			}
+			err = r.sendEventWithMessage(&r.logger, event.Probe, string(msg), true)
+			if err == nil {
+				log(nil, "succeed to send the probe event", retry, periodically)
+			} else {
+				log(err, "failed to send the probe event, will retry later", retry, periodically)
+			}
+			return err == nil
+		}
+
+		needsRetry := false
+		for {
+			select {
+			case latest := <-r.latestEvent:
+				event = latest
+				hasEvent = true
+				needsRetry = !trySend(false, false)
+
+			case <-reportChan:
+				needsRetry = !trySend(false, true)
+
+			case <-retryTicker.C:
+				if needsRetry {
+					needsRetry = !trySend(true, false)
+				}
+			}
+		}
+	}()
 }
diff --git a/pkg/kbagent/util/event.go b/pkg/kbagent/util/event.go
index aaffd8a43..315a7eb7e 100644
--- a/pkg/kbagent/util/event.go
+++ b/pkg/kbagent/util/event.go
@@ -38,25 +38,26 @@ import (
 )
 
 const (
-	maxRetryAttempts = 30
-	retryInterval    = 10 * time.Second
+	defaultRetryInterval    = 10 * time.Second
+	defaultMaxRetryAttempts = 30
+
+	syncRetryInterval    = 200 * time.Millisecond
+	syncMaxRetryAttempts = 3
 )
 
 func SendEventWithMessage(logger *logr.Logger, reason string, message string, sync bool) error {
-	send := func() error {
-		err := createOrUpdateEvent(reason, message)
-		if logger != nil && err != nil {
-			logger.Error(err, "failed to send event",
-				"reason", reason,
-				"message", message)
+	send := func(retryInterval time.Duration, retryAttempts int32) error {
+		err := createOrUpdateEvent(reason, message, retryInterval, retryAttempts)
+		if err != nil && logger != nil {
+			logger.Error(err, "failed to send event", "reason", reason, "message", message)
 		}
 		return err
 	}
 	if sync {
-		return send()
+		return send(syncRetryInterval, syncMaxRetryAttempts)
 	}
 	go func() {
-		_ = send()
+		_ = send(defaultRetryInterval, defaultMaxRetryAttempts)
 	}()
 	return nil
 }
@@ -92,7 +93,7 @@ func newEvent(reason string, message string) *corev1.Event {
 	}
 }
 
-func createOrUpdateEvent(reason, message string) error {
+func createOrUpdateEvent(reason, message string, retryInterval time.Duration, retryAttempts int32) error {
 	clientSet, err := getK8sClientSet()
 	if err != nil {
 		return err
@@ -101,30 +102,28 @@ func createOrUpdateEvent(reason, message string) error {
 	eventName := generateEventName(reason, message)
 
 	var event *corev1.Event
-	for i := 0; i < maxRetryAttempts; i++ {
+	attempts := max(retryAttempts, 1)
+	for i := int32(0); i < attempts; i++ {
 		event, err = eventsClient.Get(context.Background(), eventName, metav1.GetOptions{})
 		if err == nil {
-			// update
 			event.Count++
 			// the granularity of lastTimestamp is second and it is not enough for the event.
 			// there may multiple events in the same second, so we need to use EventTime here.
 			// event.LastTimestamp = metav1.Now()
 			event.EventTime = metav1.NowMicro()
 			_, err = eventsClient.Update(context.Background(), event, metav1.UpdateOptions{})
-			if err == nil {
-				return nil
-			}
 		} else if k8serrors.IsNotFound(err) {
-			// create
 			event = newEvent(reason, message)
 			_, err = eventsClient.Create(context.Background(), event, metav1.CreateOptions{})
-			if err == nil {
-				return nil
-			}
 		}
-		time.Sleep(retryInterval)
+		if err == nil {
+			return nil
+		}
+		if retryInterval > 0 && i < attempts-1 {
+			time.Sleep(retryInterval)
+		}
 	}
-	return errors.Wrapf(err, "failed to handle event after %d attempts", maxRetryAttempts)
+	return errors.Wrapf(err, "failed to create or update event after %d attempts", retryAttempts)
 }
 
 func getK8sClientSet() (*kubernetes.Clientset, error) {



Test patch:
diff --git a/pkg/kbagent/service/probe_test.go b/pkg/kbagent/service/probe_test.go
index cb69fd274..87fdab662 100644
--- a/pkg/kbagent/service/probe_test.go
+++ b/pkg/kbagent/service/probe_test.go
@@ -20,6 +20,8 @@ along with this program.  If not, see <http://www.gnu.org/licenses/>.
 package service
 
 import (
+	"encoding/json"
+	"fmt"
 	"time"
 
 	. "github.com/onsi/ginkgo/v2"
@@ -34,9 +36,10 @@ import (
 var _ = Describe("probe", func() {
 	Context("probe", func() {
 		var (
-			actions = []proto.Action{
+			probeName = "roleProbe"
+			actions   = []proto.Action{
 				{
-					Name: "roleProbe",
+					Name: probeName,
 					Exec: &proto.ExecAction{
 						Commands: []string{"/bin/bash", "-c", "echo -n leader"},
 					},
@@ -44,7 +47,7 @@ var _ = Describe("probe", func() {
 			}
 			probes = []proto.Probe{
 				{
-					Action:              "roleProbe",
+					Action:              probeName,
 					InitialDelaySeconds: 0,
 					PeriodSeconds:       1,
 					SuccessThreshold:    1,
@@ -90,6 +93,8 @@ var _ = Describe("probe", func() {
 
 		It("initial delay seconds", func() {
 			probes[0].InitialDelaySeconds = 60
+			defer func() { probes[0].InitialDelaySeconds = 0 }()
+
 			service, err := newProbeService(logr.New(nil), actionSvc, probes)
 			Expect(err).Should(BeNil())
 			Expect(service).ShouldNot(BeNil())
@@ -97,11 +102,107 @@ var _ = Describe("probe", func() {
 			Expect(service.Start()).Should(Succeed())
 
 			time.Sleep(1 * time.Second)
-			r := service.runners["roleProbe"]
+			r := service.runners[probeName]
 			Expect(r).ShouldNot(BeNil())
 			Expect(r.ticker).Should(BeNil())
 		})
 
+		It("send event", func() {
+			By("create probe service")
+			service, err := newProbeService(logr.New(nil), actionSvc, probes)
+			Expect(err).Should(BeNil())
+			Expect(service).ShouldNot(BeNil())
+
+			By("mock send event function")
+			eventChan := make(chan struct {
+				reason  string
+				message string
+			}, 128)
+			service.sendEventWithMessage = func(_ *logr.Logger, reason string, message string, _ bool) error {
+				eventChan <- struct{ reason, message string }{reason, message}
+				return nil
+			}
+
+			By("start probe service")
+			Expect(service.Start()).Should(Succeed())
+
+			By("check received event")
+			var receivedData struct{ reason, message string }
+			Eventually(eventChan).Should(Receive(&receivedData))
+			Expect(receivedData.reason).Should(Equal(probeName))
+			var event proto.ProbeEvent
+			Expect(json.Unmarshal([]byte(receivedData.message), &event)).Should(Succeed())
+			Eventually(event.Probe).Should(Equal(probeName))
+			Eventually(event.Code).Should(Equal(int32(0)))
+			Eventually(event.Output).Should(Equal([]byte("leader")))
+		})
+
+		It("send event - API server error", func() {
+			By("create probe service")
+			service, err := newProbeService(logr.New(nil), actionSvc, probes)
+			Expect(err).Should(BeNil())
+			Expect(service).ShouldNot(BeNil())
+
+			By("mock send event function with error")
+			var (
+				count = 0
+			)
+			service.sendEventWithMessage = func(_ *logr.Logger, reason string, message string, _ bool) error {
+				count += 1
+				return fmt.Errorf("API server error")
+			}
+			retrySendEventInterval = 1 * time.Second
+			defer func() { retrySendEventInterval = defaultRetrySendEventInterval }()
+
+			By("start probe service")
+			Expect(service.Start()).Should(Succeed())
+
+			By("wait for probe to send event error")
+			Eventually(func() int { return count }, 2*retrySendEventInterval).Should(BeNumerically(">", 1))
+		})
+
+		It("send event - after API server recover", func() {
+			By("create probe service")
+			service, err := newProbeService(logr.New(nil), actionSvc, probes)
+			Expect(err).Should(BeNil())
+			Expect(service).ShouldNot(BeNil())
+
+			By("mock send event function with temporary error")
+			var (
+				count     = 0
+				eventChan = make(chan struct {
+					reason  string
+					message string
+				}, 128)
+			)
+			service.sendEventWithMessage = func(_ *logr.Logger, reason string, message string, _ bool) error {
+				count += 1
+				if count <= 2 {
+					return fmt.Errorf("API server error")
+				}
+				eventChan <- struct{ reason, message string }{reason, message}
+				return nil
+			}
+			retrySendEventInterval = 1 * time.Second
+			defer func() { retrySendEventInterval = defaultRetrySendEventInterval }()
+
+			By("start probe service")
+			Expect(service.Start()).Should(Succeed())
+
+			By("wait for probe to send event error")
+			Eventually(func() int { return count }, 2*retrySendEventInterval).Should(BeNumerically(">", 1))
+
+			By("check received event after recover")
+			var receivedData struct{ reason, message string }
+			Eventually(eventChan, 2*retrySendEventInterval).Should(Receive(&receivedData))
+			Expect(receivedData.reason).Should(Equal(probeName))
+			var event proto.ProbeEvent
+			Expect(json.Unmarshal([]byte(receivedData.message), &event)).Should(Succeed())
+			Eventually(event.Probe).Should(Equal(probeName))
+			Eventually(event.Code).Should(Equal(int32(0)))
+			Eventually(event.Output).Should(Equal([]byte("leader")))
+		})
+
 		// TODO: more test cases
 	})
 })