Repository: caas-team/GoKubeDownscaler
Pull request #74: Feat/retries on conflict error
Author: login: samuel-esp
Labels: name: enhancement
description: New feature or request
Created: 2024-12-14T17:30:12Z | Merged: 2025-03-12T09:46:34Z

## Motivation

Give the user the possibility to choose how to handle HTTP 409 conflict errors. Such conflicts typically occur when another entity (such as an HPA, CI/CD pipeline, or manual intervention) modifies a resource just before KubeDownscaler processes it

See #68 or https://github.com/caas-team/py-kube-downscaler/issues/111

## Changes

- Introduced `--max-retries-on-conflict` argument like in Py-Kube-Downscaler
- Introduced GetWorkload() function to handle the use case when the downscaler needs to retrieve a single Kubernetes resource (before it was only possible to get a list of resources, i.e `kubectl get deploy -n default`). the old GetWorkload() was renamed to GetWorkloads() to reflect the changes
- Introduced a new function GetResourceType() that returns the resource type (string)
- Refactored the main loop to be able to use `--max-retries-on-conflict`

## Tests done

- Unit Tests

## TODO

- [x] I've assigned myself to this PR
- [x] Refactored docs
- [ ] Added more unit tests on this specific use case


Diff:
diff --git a/README.md b/README.md
index eed8900f..9cc32ca1 100644
--- a/README.md
+++ b/README.md
@@ -171,6 +171,10 @@ Runtime Configuration:
   - string key of an annotation on the workload containing a [RFC3339 Timestamp](https://datatracker.ietf.org/doc/html/rfc3339)
   - when set grace-period will use the timestamp in the annotation instead of the creation time of the workload
   - default: none (uses the workloads creation time)
+- <span id="--max-retries-on-conflict">--max-retries-on-conflict</span>:
+  - integer
+  - sets the maximum number of retries for handling HTTP 409 conflict errors, which occur when another entity modifies a resource that the downscaler is currently processing
+  - default: 0
 
 ### Environment Variables
 
diff --git a/cmd/kubedownscaler/main.go b/cmd/kubedownscaler/main.go
index b910af28..afe71e88 100644
--- a/cmd/kubedownscaler/main.go
+++ b/cmd/kubedownscaler/main.go
@@ -7,6 +7,7 @@ import (
 	"log/slog"
 	"os"
 	"os/signal"
+	"strings"
 	"sync"
 	"syscall"
 	"time"
@@ -16,6 +17,7 @@ import (
 	"github.com/caas-team/gokubedownscaler/internal/pkg/scalable"
 	"github.com/caas-team/gokubedownscaler/internal/pkg/util"
 	"github.com/caas-team/gokubedownscaler/internal/pkg/values"
+	"k8s.io/apiserver/pkg/registry/generic/registry"
 	"k8s.io/client-go/tools/leaderelection"
 )
 
@@ -161,19 +163,15 @@ func startScanning(
 		for _, workload := range workloads {
 			waitGroup.Add(1)
 
-			go func() {
-				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
-
+			go func(workload scalable.Workload) {
 				defer waitGroup.Done()
 
-				err := scanWorkload(workload, client, ctx, layerDefault, layerCli, layerEnv, config)
+				err = attemptScan(client, ctx, layerDefault, layerCli, layerEnv, config, workload)
 				if err != nil {
 					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
 					return
 				}
-
-				slog.Debug("successfully scanned workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
-			}()
+			}(workload)
 		}
 
 		waitGroup.Wait()
@@ -191,6 +189,42 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerDefault, layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	workload scalable.Workload,
+) error {
+	slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+	for retry := range config.MaxRetriesOnConflict + 1 {
+		err := scanWorkload(workload, client, ctx, layerDefault, layerCli, layerEnv, config)
+		if err != nil {
+			if !(strings.Contains(err.Error(), registry.OptimisticLockErrorMsg)) {
+				return fmt.Errorf("failed to scan workload: %w", err)
+			}
+
+			slog.Warn("workload modified, retrying", "attempt", retry+1, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+			err := client.RegetWorkload(workload, ctx)
+			if err != nil {
+				return fmt.Errorf("failed to fetch updated workload: %w", err)
+			}
+
+			continue
+		}
+
+		slog.Debug("successfully scanned workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+		return nil
+	}
+
+	slog.Error("failed to scan workload", "attempts", config.MaxRetriesOnConflict+1)
+
+	return nil
+}
+
 // scanWorkload runs a scan on the worklod, determining the scaling and scaling the workload.
 func scanWorkload(
 	workload scalable.Workload,
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 31c9aaf0..830b7a12 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -18,7 +18,7 @@ Causes:
 Fixes:
 
 - do not run multiple downscalers on the same resources
-- it should just scale in the next scan cycle so there are probably no changes needed
+- the `--max-retries-on-conflict` argument enables users to specify the number of retries for the downscaler when a conflict occurs. While the affected resource will likely be scaled in the next cycle without this optional argument, it is highly recommended to use it in conjunction with the `--once` argument
 
 > [!Note]
 > this is a pretty unavoidable issue due to there being no easy way to lock the resource from being edited while the downscaler is scaling it. The py-kube-downscaler solved this by just overwriting the changes made during scaling
diff --git a/go.mod b/go.mod
index a5116cab..755a0ddf 100644
--- a/go.mod
+++ b/go.mod
@@ -11,31 +11,43 @@ require (
 	github.com/zalando-incubator/stackset-controller v1.4.98
 	k8s.io/api v0.32.2
 	k8s.io/apimachinery v0.32.2
+	k8s.io/apiserver v0.32.2
 	k8s.io/client-go v0.32.2
 )
 
 require (
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/blang/semver/v4 v4.0.0 // indirect
+	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/coreos/go-semver v0.3.1 // indirect
+	github.com/coreos/go-systemd/v22 v22.5.0 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/emicklei/go-restful/v3 v3.12.2 // indirect
 	github.com/evanphx/json-patch/v5 v5.9.11 // indirect
 	github.com/expr-lang/expr v1.16.9 // indirect
+	github.com/felixge/httpsnoop v1.0.4 // indirect
 	github.com/fsnotify/fsnotify v1.8.0 // indirect
 	github.com/fxamacker/cbor/v2 v2.7.0 // indirect
 	github.com/go-logr/logr v1.4.2 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-openapi/jsonpointer v0.21.0 // indirect
 	github.com/go-openapi/jsonreference v0.21.0 // indirect
 	github.com/go-openapi/swag v0.23.0 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/protobuf v1.5.4 // indirect
+	github.com/google/btree v1.1.2 // indirect
 	github.com/google/gnostic-models v0.6.9 // indirect
 	github.com/google/go-cmp v0.7.0 // indirect
 	github.com/google/gofuzz v1.2.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
+	github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/klauspost/compress v1.18.0 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
 	github.com/mailru/easyjson v0.9.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
@@ -47,10 +59,26 @@ require (
 	github.com/prometheus/common v0.62.0 // indirect
 	github.com/prometheus/procfs v0.15.1 // indirect
 	github.com/rogpeppe/go-internal v1.14.1 // indirect
+	github.com/spf13/cobra v1.8.1 // indirect
 	github.com/spf13/pflag v1.0.6 // indirect
 	github.com/stretchr/objx v0.5.2 // indirect
 	github.com/szuecs/routegroup-client v0.28.2 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
+	go.etcd.io/etcd/api/v3 v3.5.17 // indirect
+	go.etcd.io/etcd/client/pkg/v3 v3.5.17 // indirect
+	go.etcd.io/etcd/client/v3 v3.5.17 // indirect
+	go.opentelemetry.io/auto/sdk v1.1.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
+	go.opentelemetry.io/otel v1.34.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 // indirect
+	go.opentelemetry.io/otel/metric v1.34.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.34.0 // indirect
+	go.opentelemetry.io/otel/trace v1.34.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.4.0 // indirect
+	go.uber.org/multierr v1.11.0 // indirect
+	go.uber.org/zap v1.27.0 // indirect
 	golang.org/x/exp v0.0.0-20250210185358-939b2ce775ac // indirect
 	golang.org/x/net v0.35.0 // indirect
 	golang.org/x/oauth2 v0.27.0 // indirect
@@ -59,15 +87,20 @@ require (
 	golang.org/x/text v0.22.0 // indirect
 	golang.org/x/time v0.10.0 // indirect
 	gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20250207221924-e9438ea467c6 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 // indirect
+	google.golang.org/grpc v1.70.0 // indirect
 	google.golang.org/protobuf v1.36.5 // indirect
 	gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	k8s.io/apiextensions-apiserver v0.32.2 // indirect
+	k8s.io/component-base v0.32.2 // indirect
 	k8s.io/klog/v2 v2.130.1 // indirect
 	k8s.io/kube-openapi v0.0.0-20241212222426-2c72e554b1e7 // indirect
 	k8s.io/utils v0.0.0-20241210054802-24370beab758 // indirect
 	knative.dev/pkg v0.0.0-20250226145529-0372c089c78f // indirect
+	sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.0 // indirect
 	sigs.k8s.io/controller-runtime v0.20.3 // indirect
 	sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
 	sigs.k8s.io/structured-merge-diff/v4 v4.6.0 // indirect
diff --git a/go.sum b/go.sum
index b6fc2594..71dec106 100644
--- a/go.sum
+++ b/go.sum
@@ -6,16 +6,27 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
+github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
+github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
+github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
+github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
 github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
 github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
+github.com/coreos/go-semver v0.3.1 h1:yi21YpKnrx1gt5R+la8n5WgS0kCrsPp33dmEyHReZr4=
+github.com/coreos/go-semver v0.3.1/go.mod h1:irMmmIw/7yzSRPWryHsK7EYSg09caPQL03VsM8rvUec=
+github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
+github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
+github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
+github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
 github.com/emicklei/go-restful/v3 v3.8.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
 github.com/emicklei/go-restful/v3 v3.9.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
 github.com/emicklei/go-restful/v3 v3.12.2 h1:DhwDP0vY3k8ZzE0RunuJy8GhNpPL6zqLkDf9B/a0/xU=
@@ -27,6 +38,8 @@ github.com/evanphx/json-patch/v5 v5.9.11 h1:/8HVnzMq13/3x9TPvjG08wUGqBTmZBsCWzjT
 github.com/evanphx/json-patch/v5 v5.9.11/go.mod h1:3j+LviiESTElxA4p3EMKAB9HXj3/XEtnUf6OZxqIQTM=
 github.com/expr-lang/expr v1.16.9 h1:WUAzmR0JNI9JCiF0/ewwHB1gmcGw5wW7nWt8gc6PpCI=
 github.com/expr-lang/expr v1.16.9/go.mod h1:8/vRC7+7HBzESEqt5kKpYXxrxkr31SaO8r40VO/1IT4=
+github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
+github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
@@ -36,10 +49,13 @@ github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXE
 github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
 github.com/go-logr/logr v0.2.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU=
 github.com/go-logr/logr v1.2.0/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
+github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
 github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
 github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ=
 github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg=
 github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
@@ -57,8 +73,12 @@ github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEe
 github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls=
 github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
 github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
+github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
 github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
+github.com/golang-jwt/jwt v3.2.2+incompatible h1:IfV12K8xAKAnZqdXVzCZ+TOjboZ2keLg81eXfW3O+oY=
+github.com/golang-jwt/jwt/v4 v4.5.1 h1:JdqV9zKUdtaa9gdPlywC3aeoEsR681PlKC+4F5gQgeo=
+github.com/golang-jwt/jwt/v4 v4.5.1/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -74,6 +94,8 @@ github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiu
 github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
 github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
 github.com/google/btree v1.0.1/go.mod h1:xXMiIv4Fb/0kKde4SpL7qlzvu5cMJDRkFDxJfI9uaxA=
+github.com/google/btree v1.1.2 h1:xf4v41cLI2Z6FxbKm+8Bu+m8ifhj15JuZ9sa0jZCMUU=
+github.com/google/btree v1.1.2/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
 github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U=
 github.com/google/gnostic-models v0.6.9 h1:MU/8wDLif2qCXZmzncUQ/BOfxWfthHi63KqpoNbWqVw=
 github.com/google/gnostic-models v0.6.9/go.mod h1:CiWsm0s6BSQd1hRn8/QmxqB6BesYcbSZxsz9b0KuDBw=
@@ -98,10 +120,24 @@ github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
+github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
+github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
+github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 h1:UH//fgunKIs4JdUbpDl1VZCDaL56wXCB/5+wF6uHfaI=
+github.com/grpc-ecosystem/go-grpc-middleware v1.4.0/go.mod h1:g5qyo/la0ALbONm6Vbp88Yd8NsDy6rZz+RcrMPxvld8=
+github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 h1:Ovs26xHkKqVztRpIrF/92BcuyuQ/YW4NSIpoGtfXNho=
+github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
+github.com/grpc-ecosystem/grpc-gateway v1.16.0 h1:gmcG1KaJ57LophUzW0Hy8NmPhnMZb4M0+kPpLofRdBo=
+github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 h1:e9Rjr40Z98/clHv5Yg79Is0NtosR5LXRvdr7o/6NwbA=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1/go.mod h1:tIxuGz/9mpox++sgp9fJjHO0+q1X9/UOWd798aAm22M=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
 github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
 github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
+github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
+github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/jonboulle/clockwork v0.4.0 h1:p4Cf1aMWXnXAUh8lVfewRBx1zaTSYKrKMF2g3ST4RZ4=
+github.com/jonboulle/clockwork v0.4.0/go.mod h1:xgRqUGwRcjKCO1vbZUEtSLrqKoPSsUpK7fnezOII0kc=
 github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
 github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
@@ -196,7 +232,14 @@ github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncj
 github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
 github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
 github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
+github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
+github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
+github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
+github.com/soheilhy/cmux v0.1.5 h1:jjzc5WVemNEDTLwv9tlmemhC73tI08BNOIGwBOo10Js=
+github.com/soheilhy/cmux v0.1.5/go.mod h1:T7TcVDs9LWfQgPlPsdngu6I6QIoyIFZDDC6sNE1GqG0=
 github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
+github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM=
+github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
 github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
@@ -216,14 +259,56 @@ github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOf
 github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
 github.com/szuecs/routegroup-client v0.28.2 h1:Dk9D6VqhtYM0IVRkik0fpZ5IbVrf1mHssYmAyRrwehU=
 github.com/szuecs/routegroup-client v0.28.2/go.mod h1:QpI/XGdncIAYIE03Nwjq0w+NXlIfV/n56BI1uR2a2Do=
+github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 h1:6fotK7otjonDflCTK0BCfls4SPy3NcCVb5dqqmbRknE=
+github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75/go.mod h1:KO6IkyS8Y3j8OdNO85qEYBsRPuteD+YciPomcXdrMnk=
 github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
 github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
+github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 h1:S2dVYn90KE98chqDkyE9Z4N61UnQd+KOfgp5Iu53llk=
+github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
 github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
 github.com/zalando-incubator/stackset-controller v1.4.98 h1:xGTKSRbW7uFM1HxfLhWHTq69k81/CqN0OuF5MK9QTlk=
 github.com/zalando-incubator/stackset-controller v1.4.98/go.mod h1:3LAv4SQPDPlLMlqGQngJZFXEJ4z5y6jDun62Kr9RAZI=
+go.etcd.io/bbolt v1.3.11 h1:yGEzV1wPz2yVCLsD8ZAiGHhHVlczyC9d1rP43/VCRJ0=
+go.etcd.io/bbolt v1.3.11/go.mod h1:dksAq7YMXoljX0xu6VF5DMZGbhYYoLUalEiSySYAS4I=
+go.etcd.io/etcd/api/v3 v3.5.17 h1:cQB8eb8bxwuxOilBpMJAEo8fAONyrdXTHUNcMd8yT1w=
+go.etcd.io/etcd/api/v3 v3.5.17/go.mod h1:d1hvkRuXkts6PmaYk2Vrgqbv7H4ADfAKhyJqHNLJCB4=
+go.etcd.io/etcd/client/pkg/v3 v3.5.17 h1:XxnDXAWq2pnxqx76ljWwiQ9jylbpC4rvkAeRVOUKKVw=
+go.etcd.io/etcd/client/pkg/v3 v3.5.17/go.mod h1:4DqK1TKacp/86nJk4FLQqo6Mn2vvQFBmruW3pP14H/w=
+go.etcd.io/etcd/client/v2 v2.305.16 h1:kQrn9o5czVNaukf2A2At43cE9ZtWauOtf9vRZuiKXow=
+go.etcd.io/etcd/client/v2 v2.305.16/go.mod h1:h9YxWCzcdvZENbfzBTFCnoNumr2ax3F19sKMqHFmXHE=
+go.etcd.io/etcd/client/v3 v3.5.17 h1:o48sINNeWz5+pjy/Z0+HKpj/xSnBkuVhVvXkjEXbqZY=
+go.etcd.io/etcd/client/v3 v3.5.17/go.mod h1:j2d4eXTHWkT2ClBgnnEPm/Wuu7jsqku41v9DZ3OtjQo=
+go.etcd.io/etcd/pkg/v3 v3.5.16 h1:cnavs5WSPWeK4TYwPYfmcr3Joz9BH+TZ6qoUtz6/+mc=
+go.etcd.io/etcd/pkg/v3 v3.5.16/go.mod h1:+lutCZHG5MBBFI/U4eYT5yL7sJfnexsoM20Y0t2uNuY=
+go.etcd.io/etcd/raft/v3 v3.5.16 h1:zBXA3ZUpYs1AwiLGPafYAKKl/CORn/uaxYDwlNwndAk=
+go.etcd.io/etcd/raft/v3 v3.5.16/go.mod h1:P4UP14AxofMJ/54boWilabqqWoW9eLodl6I5GdGzazI=
+go.etcd.io/etcd/server/v3 v3.5.16 h1:d0/SAdJ3vVsZvF8IFVb1k8zqMZ+heGcNfft71ul9GWE=
+go.etcd.io/etcd/server/v3 v3.5.16/go.mod h1:ynhyZZpdDp1Gq49jkUg5mfkDWZwXnn3eIqCqtJnrD/s=
+go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
+go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
+go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 h1:r6I7RJCN86bpD/FQwedZ0vSixDpwuWREjW9oRMsmqDc=
+go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0/go.mod h1:B9yO6b04uB80CzjedvewuqDhxJxi11s7/GtiGa8bAjI=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 h1:TT4fX+nBOA/+LUkobKGW1ydGcn+G3vRw9+g5HwCphpk=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0/go.mod h1:L7UH0GbB0p47T4Rri3uHjbpCFYrVrwc1I25QhNPiGK8=
+go.opentelemetry.io/otel v1.34.0 h1:zRLXxLCgL1WyKsPVrgbSdMN4c0FMkDAskSTQP+0hdUY=
+go.opentelemetry.io/otel v1.34.0/go.mod h1:OWFPOQ+h4G8xpyjgqo4SxJYdDQ/qmRH+wivy7zzx9oI=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0 h1:3Q/xZUyC1BBkualc9ROb4G8qkH90LXEIICcs5zv1OYY=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0/go.mod h1:s75jGIWA9OfCMzF0xr+ZgfrB5FEbbV7UuYo32ahUiFI=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 h1:qFffATk0X+HD+f1Z8lswGiOQYKHRlzfmdJm0wEaVrFA=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0/go.mod h1:MOiCmryaYtc+V0Ei+Tx9o5S1ZjA7kzLucuVuyzBZloQ=
+go.opentelemetry.io/otel/metric v1.34.0 h1:+eTR3U0MyfWjRDhmFMxe2SsW64QrZ84AOhvqS7Y+PoQ=
+go.opentelemetry.io/otel/metric v1.34.0/go.mod h1:CEDrp0fy2D0MvkXE+dPV7cMi8tWZwX3dmaIhwPOaqHE=
+go.opentelemetry.io/otel/sdk v1.34.0 h1:95zS4k/2GOy069d321O8jWgYsW3MzVV+KuSPKp7Wr1A=
+go.opentelemetry.io/otel/sdk v1.34.0/go.mod h1:0e/pNiaMAqaykJGKbi+tSjWfNNHMTxoC9qANsCzbyxU=
+go.opentelemetry.io/otel/sdk/metric v1.33.0 h1:Gs5VK9/WUJhNXZgn8MR6ITatvAmKeIuCtNbsP3JkNqU=
+go.opentelemetry.io/otel/sdk/metric v1.33.0/go.mod h1:dL5ykHZmm1B1nVRk9dDjChwDmt81MjVp3gLkQRwKf/Q=
+go.opentelemetry.io/otel/trace v1.34.0 h1:+ouXS2V8Rd4hp4580a8q23bg0azF2nI8cqLYnC8mh/k=
+go.opentelemetry.io/otel/trace v1.34.0/go.mod h1:Svm7lSjQD7kG7KJ/MUHPVXSDGz2OX4h0M2jHBhmSfRE=
+go.opentelemetry.io/proto/otlp v1.4.0 h1:TA9WRvW6zMwP+Ssb6fLoUIuirti1gGbP28GcKG1jgeg=
+go.opentelemetry.io/proto/otlp v1.4.0/go.mod h1:PPBWZIP98o2ElSqI35IHfu7hIhSwvc5N38Jw8pXuGFY=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
 go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
@@ -239,6 +324,8 @@ golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliY
 golang.org/x/crypto v0.16.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4=
 golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
 golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs=
+golang.org/x/crypto v0.33.0 h1:IOBPskki6Lysi0lo9qQvbxiQ+FvsCC/YWOecCHAixus=
+golang.org/x/crypto v0.33.0/go.mod h1:bVdXmD7IV/4GdElGPozy6U7lWdRXA4qyRVGJV57uQ5M=
 golang.org/x/exp v0.0.0-20250210185358-939b2ce775ac h1:l5+whBCLH3iH2ZNHYLbAe58bo7yrN4mVcnkHDYz5vvs=
 golang.org/x/exp v0.0.0-20250210185358-939b2ce775ac/go.mod h1:hH+7mtFmImwwcMvScyxUhjuVHR3HGaDPMn9rMSUUbxo=
 golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
@@ -388,6 +475,14 @@ golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNq
 gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw=
 gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY=
 google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
+google.golang.org/genproto v0.0.0-20241118233622-e639e219e697 h1:ToEetK57OidYuqD4Q5w+vfEnPvPpuTwedCNVohYJfNk=
+google.golang.org/genproto v0.0.0-20241118233622-e639e219e697/go.mod h1:JJrvXBWRZaFMxBufik1a4RpFw4HhgVtBBWQeQgUj2cc=
+google.golang.org/genproto/googleapis/api v0.0.0-20250207221924-e9438ea467c6 h1:L9JNMl/plZH9wmzQUHleO/ZZDSN+9Gh41wPczNy+5Fk=
+google.golang.org/genproto/googleapis/api v0.0.0-20250207221924-e9438ea467c6/go.mod h1:iYONQfRdizDB8JJBybql13nArx91jcUk7zCXEsOofM4=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 h1:2duwAxN2+k0xLNpjnHTXoMUgnv6VPSp5fiqTuwSxjmI=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
+google.golang.org/grpc v1.70.0 h1:pWFv03aZoHzlRKHWicjsZytKAiYCtNS0dHbXnIdq7jQ=
+google.golang.org/grpc v1.70.0/go.mod h1:ofIJqVKDXx/JiXrwr2IG4/zwdH9txy3IlF40RmcJSQw=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
@@ -411,6 +506,8 @@ gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWM
 gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
 gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc=
+gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -429,10 +526,14 @@ k8s.io/apiextensions-apiserver v0.32.2/go.mod h1:GPwf8sph7YlJT3H6aKUWtd0E+oyShk/
 k8s.io/apimachinery v0.28.7/go.mod h1:QFNX/kCl/EMT2WTSz8k4WLCv2XnkOLMaL8GAVRMdpsA=
 k8s.io/apimachinery v0.32.2 h1:yoQBR9ZGkA6Rgmhbp/yuT9/g+4lxtsGYwW6dR6BDPLQ=
 k8s.io/apimachinery v0.32.2/go.mod h1:GpHVgxoKlTxClKcteaeuF1Ul/lDVb74KpZcxcmLDElE=
+k8s.io/apiserver v0.32.2 h1:WzyxAu4mvLkQxwD9hGa4ZfExo3yZZaYzoYvvVDlM6vw=
+k8s.io/apiserver v0.32.2/go.mod h1:PEwREHiHNU2oFdte7BjzA1ZyjWjuckORLIK/wLV5goM=
 k8s.io/client-go v0.28.7/go.mod h1:xIoEaDewZ+EwWOo1/F1t0IOKMPe1rwBZhLu9Es6y0tE=
 k8s.io/client-go v0.32.2 h1:4dYCD4Nz+9RApM2b/3BtVvBHw54QjMFUl1OLcJG5yOA=
 k8s.io/client-go v0.32.2/go.mod h1:fpZ4oJXclZ3r2nDOv+Ux3XcJutfrwjKTCHz2H3sww94=
 k8s.io/code-generator v0.28.7/go.mod h1:IaYGMqYjgj0zE3L9mnHo7hIL9GkY08GvGyyracaIxTA=
+k8s.io/component-base v0.32.2 h1:1aUL5Vdmu7qNo4ZsE+569PV5zFatM9hl+lb3dEea2zU=
+k8s.io/component-base v0.32.2/go.mod h1:PXJ61Vx9Lg+P5mS8TLd7bCIr+eMJRQTyXe8KvkrvJq0=
 k8s.io/gengo v0.0.0-20210813121822-485abfe95c7c/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E=
 k8s.io/gengo v0.0.0-20220902162205-c0856e24416d/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E=
 k8s.io/klog/v2 v2.0.0/go.mod h1:PBfzABfn139FHAV07az/IF9Wp1bkk3vpT2XSJ76fSDE=
@@ -450,6 +551,8 @@ k8s.io/utils v0.0.0-20241210054802-24370beab758 h1:sdbE21q2nlQtFh65saZY+rRM6x6aJ
 k8s.io/utils v0.0.0-20241210054802-24370beab758/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
 knative.dev/pkg v0.0.0-20250226145529-0372c089c78f h1:udznvloLcWaixDGrENQ9HRcj1fiMlE6cP8L0V3K4V74=
 knative.dev/pkg v0.0.0-20250226145529-0372c089c78f/go.mod h1:ijkSiVyNl0yTjfRsGx3infOo9ohlyE1lTCri9iUHR80=
+sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.0 h1:CPT0ExVicCzcpeN4baWEV2ko2Z/AsiZgEdwgcfwLgMo=
+sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.0/go.mod h1:Ve9uj1L+deCXFrPOk1LpFXqTg7LCFzFso6PA48q/XZw=
 sigs.k8s.io/controller-runtime v0.19.4 h1:SUmheabttt0nx8uJtoII4oIP27BVVvAKFvdvGFwV/Qo=
 sigs.k8s.io/controller-runtime v0.19.4/go.mod h1:iRmWllt8IlaLjvTTDLhRBXIEtkCK6hwVBJJsYS9Ajf4=
 sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0=
diff --git a/internal/api/kubernetes/client.go b/internal/api/kubernetes/client.go
index df8a059f..4b5b52c5 100644
--- a/internal/api/kubernetes/client.go
+++ b/internal/api/kubernetes/client.go
@@ -31,6 +31,8 @@ type Client interface {
 	GetNamespaceAnnotations(namespace string, ctx context.Context) (map[string]string, error)
 	// GetWorkloads gets all workloads of the specified resources for the specified namespaces
 	GetWorkloads(namespaces []string, resourceTypes []string, ctx context.Context) ([]scalable.Workload, error)
+	// RegetWorkload gets the workload again to ensure the latest state
+	RegetWorkload(workload scalable.Workload, ctx context.Context) error
 	// DownscaleWorkload downscales the workload to the specified replicas
 	DownscaleWorkload(replicas int32, workload scalable.Workload, ctx context.Context) error
 	// UpscaleWorkload upscales the workload to the original replicas
@@ -130,6 +132,16 @@ func (c client) GetWorkloads(namespaces, resourceTypes []string, ctx context.Con
 	return results, nil
 }
 
+// RegetWorkload gets the workload again to ensure the latest state.
+func (c client) RegetWorkload(workload scalable.Workload, ctx context.Context) error {
+	err := workload.Reget(c.clientsets, ctx)
+	if err != nil {
+		return fmt.Errorf("failed to get workload: %w", err)
+	}
+
+	return nil
+}
+
 // DownscaleWorkload downscales the workload to the specified replicas.
 func (c client) DownscaleWorkload(replicas int32, workload scalable.Workload, ctx context.Context) error {
 	err := workload.ScaleDown(replicas)
diff --git a/internal/pkg/scalable/cronjobs.go b/internal/pkg/scalable/cronjobs.go
index c3415b2d..d531ba8a 100644
--- a/internal/pkg/scalable/cronjobs.go
+++ b/internal/pkg/scalable/cronjobs.go
@@ -1,4 +1,4 @@
-//nolint:dupl // this code is very similar for every resource, but its not really abstractable to avoid more duplication
+// nolint:dupl // necessary to handle different workload types separately
 package scalable
 
 import (
@@ -29,6 +29,18 @@ type cronJob struct {
 	*batch.CronJob
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (c *cronJob) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	c.CronJob, err = clientsets.Kubernetes.BatchV1().CronJobs(c.Namespace).Get(ctx, c.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get cronjob: %w", err)
+	}
+
+	return nil
+}
+
 // setSuspend sets the value of the suspend field on the cronJob.
 func (c *cronJob) setSuspend(suspend bool) {
 	c.Spec.Suspend = &suspend
diff --git a/internal/pkg/scalable/daemonsets.go b/internal/pkg/scalable/daemonsets.go
index 3e2bae0d..7b3f16c4 100644
--- a/internal/pkg/scalable/daemonsets.go
+++ b/internal/pkg/scalable/daemonsets.go
@@ -1,3 +1,4 @@
+// nolint:dupl // necessary to handle different workload types separately
 package scalable
 
 import (
@@ -49,6 +50,18 @@ func (d *daemonSet) ScaleDown(_ int32) error {
 	return nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (d *daemonSet) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	d.DaemonSet, err = clientsets.Kubernetes.AppsV1().DaemonSets(d.Namespace).Get(ctx, d.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get cronjob: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (d *daemonSet) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Kubernetes.AppsV1().DaemonSets(d.Namespace).Update(ctx, d.DaemonSet, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/deployments.go b/internal/pkg/scalable/deployments.go
index 347a4ca0..959c703b 100644
--- a/internal/pkg/scalable/deployments.go
+++ b/internal/pkg/scalable/deployments.go
@@ -45,6 +45,18 @@ func (d *deployment) getReplicas() (int32, error) {
 	return *d.Spec.Replicas, nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (d *deployment) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	d.Deployment, err = clientsets.Kubernetes.AppsV1().Deployments(d.Namespace).Get(ctx, d.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get cronjob: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (d *deployment) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Kubernetes.AppsV1().Deployments(d.Namespace).Update(ctx, d.Deployment, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/horizontalpodautoscalers.go b/internal/pkg/scalable/horizontalpodautoscalers.go
index 0311e625..f76e4d52 100644
--- a/internal/pkg/scalable/horizontalpodautoscalers.go
+++ b/internal/pkg/scalable/horizontalpodautoscalers.go
@@ -1,3 +1,4 @@
+// nolint:dupl // necessary to handle different workload types separately
 package scalable
 
 import (
@@ -52,6 +53,19 @@ func (h *horizontalPodAutoscaler) getReplicas() (int32, error) {
 	return *h.Spec.MinReplicas, nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (h *horizontalPodAutoscaler) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	h.HorizontalPodAutoscaler, err = clientsets.
+		Kubernetes.AutoscalingV2().HorizontalPodAutoscalers(h.Namespace).Get(ctx, h.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get horizontalpodautoscaler: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (h *horizontalPodAutoscaler) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Kubernetes.AutoscalingV2().HorizontalPodAutoscalers(h.Namespace).Update(
diff --git a/internal/pkg/scalable/jobs.go b/internal/pkg/scalable/jobs.go
index ad165c5a..89dd0fda 100644
--- a/internal/pkg/scalable/jobs.go
+++ b/internal/pkg/scalable/jobs.go
@@ -1,4 +1,4 @@
-//nolint:dupl // this code is very similar for every resource, but its not really abstractable to avoid more duplication
+// nolint:dupl // necessary to handle different workload types separately
 package scalable
 
 import (
@@ -34,6 +34,18 @@ func (j *job) setSuspend(suspend bool) {
 	j.Spec.Suspend = &suspend
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (j *job) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	j.Job, err = clientsets.Kubernetes.BatchV1().Jobs(j.Namespace).Get(ctx, j.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get job: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (j *job) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Kubernetes.BatchV1().Jobs(j.Namespace).Update(ctx, j.Job, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/poddisruptionbudgets.go b/internal/pkg/scalable/poddisruptionbudgets.go
index 588c5baf..0ba4eb40 100644
--- a/internal/pkg/scalable/poddisruptionbudgets.go
+++ b/internal/pkg/scalable/poddisruptionbudgets.go
@@ -1,3 +1,4 @@
+// nolint:dupl // necessary to handle different workload types separately
 package scalable
 
 import (
@@ -137,6 +138,18 @@ func (p *podDisruptionBudget) ScaleDown(downscaleReplicas int32) error {
 	return nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (p *podDisruptionBudget) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	p.PodDisruptionBudget, err = clientsets.Kubernetes.PolicyV1().PodDisruptionBudgets(p.Namespace).Get(ctx, p.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get poddisruptionbudget: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (p *podDisruptionBudget) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Kubernetes.PolicyV1().PodDisruptionBudgets(p.Namespace).Update(ctx, p.PodDisruptionBudget, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/prometheuses.go b/internal/pkg/scalable/prometheuses.go
index a9822227..d36fc767 100644
--- a/internal/pkg/scalable/prometheuses.go
+++ b/internal/pkg/scalable/prometheuses.go
@@ -1,4 +1,3 @@
-//nolint:dupl // this code is very similar for every resource, but its not really abstractable to avoid more duplication
 package scalable
 
 import (
@@ -45,6 +44,18 @@ func (p *prometheus) getReplicas() (int32, error) {
 	return *p.Spec.Replicas, nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (p *prometheus) Reget(clientsets *Clientsets, ctx context.Context) error {
+	singlePrometheus, err := clientsets.Monitoring.MonitoringV1().Prometheuses(p.Namespace).Get(ctx, p.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get prometheus: %w", err)
+	}
+
+	p.Prometheus = singlePrometheus
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (p *prometheus) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Monitoring.MonitoringV1().Prometheuses(p.Namespace).Update(ctx, p.Prometheus, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/rollouts.go b/internal/pkg/scalable/rollouts.go
index 79d398a8..d093871b 100644
--- a/internal/pkg/scalable/rollouts.go
+++ b/internal/pkg/scalable/rollouts.go
@@ -1,4 +1,4 @@
-//nolint:dupl // this code is very similar for every resource, but its not really abstractable to avoid more duplication
+//nolint:dupl // necessary to handle different workload types separately
 package scalable
 
 import (
@@ -45,6 +45,18 @@ func (r *rollout) getReplicas() (int32, error) {
 	return *r.Spec.Replicas, nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (r *rollout) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	r.Rollout, err = clientsets.Argo.ArgoprojV1alpha1().Rollouts(r.Namespace).Get(ctx, r.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get rollout: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (r *rollout) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Argo.ArgoprojV1alpha1().Rollouts(r.Namespace).Update(ctx, r.Rollout, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/scaledobjects.go b/internal/pkg/scalable/scaledobjects.go
index ce0af1b1..63c27ae2 100644
--- a/internal/pkg/scalable/scaledobjects.go
+++ b/internal/pkg/scalable/scaledobjects.go
@@ -1,3 +1,4 @@
+// nolint:dupl // necessary to handle different workload types separately
 package scalable
 
 import (
@@ -66,6 +67,18 @@ func (s *scaledObject) getReplicas() (int32, error) {
 	return int32(pausedReplicas), nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (s *scaledObject) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	s.ScaledObject, err = clientsets.Keda.KedaV1alpha1().ScaledObjects(s.Namespace).Get(ctx, s.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get scaledObject: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (s *scaledObject) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Keda.KedaV1alpha1().ScaledObjects(s.Namespace).Update(ctx, s.ScaledObject, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/stacks.go b/internal/pkg/scalable/stacks.go
index 1f0fae38..00b87e02 100644
--- a/internal/pkg/scalable/stacks.go
+++ b/internal/pkg/scalable/stacks.go
@@ -45,6 +45,18 @@ func (s *stack) getReplicas() (int32, error) {
 	return *s.Spec.Replicas, nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (s *stack) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	s.Stack, err = clientsets.Zalando.ZalandoV1().Stacks(s.Namespace).Get(ctx, s.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get stack: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (s *stack) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Zalando.ZalandoV1().Stacks(s.Namespace).Update(ctx, s.Stack, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/statefulsets.go b/internal/pkg/scalable/statefulsets.go
index 84113c5b..6c66cdce 100644
--- a/internal/pkg/scalable/statefulsets.go
+++ b/internal/pkg/scalable/statefulsets.go
@@ -1,4 +1,4 @@
-//nolint:dupl // this code is very similar for every resource, but its not really abstractable to avoid more duplication
+//nolint:dupl // necessary to handle different workload types separately
 package scalable
 
 import (
@@ -45,6 +45,18 @@ func (s *statefulSet) getReplicas() (int32, error) {
 	return *s.Spec.Replicas, nil
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (s *statefulSet) Reget(clientsets *Clientsets, ctx context.Context) error {
+	var err error
+
+	s.StatefulSet, err = clientsets.Kubernetes.AppsV1().StatefulSets(s.Namespace).Get(ctx, s.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get statefulset: %w", err)
+	}
+
+	return nil
+}
+
 // Update updates the resource with all changes made to it. It should only be called once on a resource.
 func (s *statefulSet) Update(clientsets *Clientsets, ctx context.Context) error {
 	_, err := clientsets.Kubernetes.AppsV1().StatefulSets(s.Namespace).Update(ctx, s.StatefulSet, metav1.UpdateOptions{})
diff --git a/internal/pkg/scalable/workload.go b/internal/pkg/scalable/workload.go
index 0ce52043..9d12a1bf 100644
--- a/internal/pkg/scalable/workload.go
+++ b/internal/pkg/scalable/workload.go
@@ -70,6 +70,8 @@ type scalableResource interface {
 	SetAnnotations(annotations map[string]string)
 	// GroupVersionKind gets the group version kind of the workload
 	GroupVersionKind() schema.GroupVersionKind
+	// Reget regets the workload to ensure the latest state
+	Reget(clientsets *Clientsets, ctx context.Context) error
 }
 
 // Workload provides all functions needed to scale the workload.
diff --git a/internal/pkg/util/config.go b/internal/pkg/util/config.go
index 85f66fa5..6675236f 100644
--- a/internal/pkg/util/config.go
+++ b/internal/pkg/util/config.go
@@ -31,6 +31,8 @@ type RuntimeConfiguration struct {
 	IncludeLabels RegexList
 	// TimeAnnotation sets the annotation used for grace-period instead of creation time.
 	TimeAnnotation string
+	// MaxRetriesOnError sets the maximum number of retries when encountering Kubernetes HTTP 409 conflict error.
+	MaxRetriesOnConflict int
 	// Kubeconfig sets an optional kubeconfig to use for testing purposes instead of the in-cluster config.
 	Kubeconfig string
 }
@@ -119,6 +121,12 @@ func (c *RuntimeConfiguration) ParseConfigFlags() {
 		"",
 		"the annotation to use instead of creation time for grace period (optional)",
 	)
+	flag.IntVar(
+		&c.MaxRetriesOnConflict,
+		"max-retries-on-conflict",
+		0,
+		"the annotation to use instead of creation time for grace period (optional)",
+	)
 }
 
 // ParseConfigEnvVars parses all environment variables for the runtime configuration.


Reviews:
user: login: jonathan-mayer
body: Also just so you know, the workflows were broken for forks (thats why they were failing). We've fixed it but it will now run every workflow twice in this and the other pr. If you want you can rebase the branches with main and the errors will go away.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-01-07T09:44:41Z
user: login: jonathan-mayer
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-01-07T12:31:56Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-19T20:52:40Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-19T22:36:21Z
user: login: jonathan-mayer
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-02-25T07:05:07Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-25T19:35:32Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-25T20:18:27Z
user: login: jonathan-mayer
body: looked at main and clientgo
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-02-26T10:59:34Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-26T11:06:53Z
user: login: jonathan-mayer
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-02-28T06:19:48Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-28T11:31:47Z
user: login: jonathan-mayer
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-03T06:21:38Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T17:20:21Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T17:35:29Z
user: login: jonathan-mayer
body: I think this should be good to go
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-04T07:03:18Z
user: login: jonathan-mayer
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-05T10:49:54Z
user: login: jonathan-mayer
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-10T13:23:02Z
user: login: jonathan-mayer
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T11:42:18Z
user: login: samuel-esp
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-11T14:42:35Z
user: login: jonathan-mayer
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-12T06:42:39Z
user: login: jonathan-mayer
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-12T07:06:52Z

Review comments:
diff_hunk: @@ -121,18 +125,40 @@ func main() {
 		var wg sync.WaitGroup
 		for _, workload := range workloads {
 			wg.Add(1)
-			go func() {
+			go func(workload scalable.Workload) {
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: great catch
created_at: 2025-01-07T09:59:06Z
author_association: MEMBER
diff_hunk: @@ -121,18 +125,40 @@ func main() {
 		var wg sync.WaitGroup
 		for _, workload := range workloads {
 			wg.Add(1)
-			go func() {
+			go func(workload scalable.Workload) {
 				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
 				defer wg.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
-				if err != nil {
-					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
-					return
+				attempts := 0
+				for {
+					err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
+					if err != nil {
+						if strings.Contains(err.Error(), "the object has been modified") {
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: I think we should avoid checking the error string here, since it can change at any time. I think it would be better to see if we can get the underlying error type and check if the error is an instance of it.
created_at: 2025-01-07T10:02:51Z
author_association: MEMBER
diff_hunk: @@ -121,18 +125,40 @@ func main() {
 		var wg sync.WaitGroup
 		for _, workload := range workloads {
 			wg.Add(1)
-			go func() {
+			go func(workload scalable.Workload) {
 				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
 				defer wg.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
-				if err != nil {
-					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
-					return
+				attempts := 0
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: 
```suggestion
				var attempts int
```
created_at: 2025-01-07T10:03:08Z
author_association: MEMBER
diff_hunk: @@ -121,18 +125,40 @@ func main() {
 		var wg sync.WaitGroup
 		for _, workload := range workloads {
 			wg.Add(1)
-			go func() {
+			go func(workload scalable.Workload) {
 				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
 				defer wg.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
-				if err != nil {
-					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
-					return
+				attempts := 0
+				for {
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: i would change this to a for i as we can break out either way and then dont have to handle incrementing nor declaring the attempts variable
created_at: 2025-01-07T10:08:35Z
author_association: MEMBER
diff_hunk: @@ -121,18 +125,40 @@ func main() {
 		var wg sync.WaitGroup
 		for _, workload := range workloads {
 			wg.Add(1)
-			go func() {
+			go func(workload scalable.Workload) {
 				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
 				defer wg.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
-				if err != nil {
-					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
-					return
+				attempts := 0
+				for {
+					err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
+					if err != nil {
+						if strings.Contains(err.Error(), "the object has been modified") {
+							if attempts >= maxRetriesOnConflict {
+								if maxRetriesOnConflict > 0 {
+									slog.Error("max retries reached, will try again in the next cycle", "workload", workload.GetName(), "namespace", workload.GetNamespace())
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: 
```suggestion
									slog.Error("max retries reached, will try again in the next scan", "workload", workload.GetName(), "namespace", workload.GetNamespace())
```
created_at: 2025-01-07T10:10:00Z
author_association: MEMBER
diff_hunk: @@ -121,18 +125,40 @@ func main() {
 		var wg sync.WaitGroup
 		for _, workload := range workloads {
 			wg.Add(1)
-			go func() {
+			go func(workload scalable.Workload) {
 				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
 				defer wg.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
-				if err != nil {
-					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
-					return
+				attempts := 0
+				for {
+					err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
+					if err != nil {
+						if strings.Contains(err.Error(), "the object has been modified") {
+							if attempts >= maxRetriesOnConflict {
+								if maxRetriesOnConflict > 0 {
+									slog.Error("max retries reached, will try again in the next cycle", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+								} else {
+									slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+								}
+								return
+							}
+							slog.Warn("workload modified, retrying", "attempt", attempts+1, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+							updatedWorkload, err := client.GetWorkload(workload.GetName(), workload.GetNamespace(), workload.GetResourceType(), ctx)
+							if err != nil {
+								slog.Error("failed to fetch latest workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+								return
+							}
+							workload = updatedWorkload
+						} else {
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: use [guard clauses](https://en.wikipedia.org/wiki/Guard_(computer_science)#Flatter_code_with_less_nesting) where applicable
created_at: 2025-01-07T10:12:09Z
author_association: MEMBER
diff_hunk: @@ -121,18 +125,40 @@ func main() {
 		var wg sync.WaitGroup
 		for _, workload := range workloads {
 			wg.Add(1)
-			go func() {
+			go func(workload scalable.Workload) {
 				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
 				defer wg.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
-				if err != nil {
-					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
-					return
+				attempts := 0
+				for {
+					err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
+					if err != nil {
+						if strings.Contains(err.Error(), "the object has been modified") {
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: ok ive looked and there is no higher level underlying error type we could use for this. What i did find is where the message is coming from which means, that we can reference that instead of using just a string. With that said, I would change this to:

```
import "k8s.io/apiserver/pkg/registry/generic/registry"
```
```suggestion
						if strings.Contains(err.Error(), registry.OptimisticLockErrorMsg) {
```
created_at: 2025-01-07T11:55:05Z
author_association: MEMBER
diff_hunk: @@ -21,11 +21,27 @@ func getCronJobs(namespace string, clientsets *Clientsets, ctx context.Context)
 	return results, nil
 }
 
+// getCronJob is the getResourceFunc for a single CronJob
+func getCronJob(name string, namespace string, clientsets *Clientsets, ctx context.Context) (Workload, error) {
+	var result Workload
+	cronjob, err := clientsets.Kubernetes.BatchV1().CronJobs(namespace).Get(ctx, name, metav1.GetOptions{})
+	if err != nil {
+		return result, fmt.Errorf("failed to get cronjob: %w", err)
+	}
+	result = &suspendScaledWorkload{&cronJob{cronjob}}
+	return result, nil
+}
path: internal/pkg/scalable/cronjobs.go
user: login: jonathan-mayer
body: I really think we shouldn't have a whole other function just to get just 1 of the resource.

```suggestion
// getCronJobs is the getResourceFunc for CronJobs
func getCronJobs(name, namespace string, clientsets *Clientsets, ctx context.Context) ([]Workload, error) {
    var results []Workload
    if name != "" {
        cronjob, err := clientsets.Kubernetes.BatchV1().CronJobs(namespace).Get(ctx, name, metav1.GetOptions{})
        if err != nil {
            return result, fmt.Errorf("failed to get cronjob: %w", err)
        }
        results = append(results, &suspendScaledWorkload{&cronJob{cronjob}})
        return results
    }

    cronjobs, err := clientsets.Kubernetes.BatchV1().CronJobs(namespace).List(ctx, metav1.ListOptions{TimeoutSeconds: &timeout})
    if err != nil {
        return nil, fmt.Errorf("failed to get cronjobs: %w", err)
    }
    for _, item := range cronjobs.Items {
        results = append(results, &suspendScaledWorkload{&cronJob{&item}})
    }
    return results, nil
}
```

I think we should keep the client.GetWorkload and GetWorkloads in single functions. This will also help abstract this configuration away.
created_at: 2025-01-07T12:31:29Z
author_association: MEMBER
diff_hunk: @@ -121,18 +125,40 @@ func main() {
 		var wg sync.WaitGroup
 		for _, workload := range workloads {
 			wg.Add(1)
-			go func() {
+			go func(workload scalable.Workload) {
 				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
 				defer wg.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
-				if err != nil {
-					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
-					return
+				attempts := 0
+				for {
+					err := scanWorkload(workload, client, ctx, layerCli, layerEnv)
+					if err != nil {
+						if strings.Contains(err.Error(), "the object has been modified") {
path: cmd/kubedownscaler/main.go
user: login: samuel-esp
body: That's elegant jonathan, will refactor to this. Great find!
created_at: 2025-02-19T20:52:40Z
author_association: COLLABORATOR
diff_hunk: @@ -21,11 +21,27 @@ func getCronJobs(namespace string, clientsets *Clientsets, ctx context.Context)
 	return results, nil
 }
 
+// getCronJob is the getResourceFunc for a single CronJob
+func getCronJob(name string, namespace string, clientsets *Clientsets, ctx context.Context) (Workload, error) {
+	var result Workload
+	cronjob, err := clientsets.Kubernetes.BatchV1().CronJobs(namespace).Get(ctx, name, metav1.GetOptions{})
+	if err != nil {
+		return result, fmt.Errorf("failed to get cronjob: %w", err)
+	}
+	result = &suspendScaledWorkload{&cronJob{cronjob}}
+	return result, nil
+}
path: internal/pkg/scalable/cronjobs.go
user: login: samuel-esp
body: Refactored in this way
created_at: 2025-02-19T22:36:21Z
author_association: COLLABORATOR
diff_hunk: @@ -215,6 +205,64 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	waitGroup *sync.WaitGroup,
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: we might just want to pass in the .Done() func

or we could wrap calling the attemptScan function in a anonymous func which handles waitGroup.Done and error handling
created_at: 2025-02-25T06:14:54Z
author_association: MEMBER
diff_hunk: @@ -215,6 +205,64 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	waitGroup *sync.WaitGroup,
+	workload scalable.Workload,
+) {
+	slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+	defer waitGroup.Done()
+
+	scanSucceded := false
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: 
```suggestion
	var scanSucceded bool
```
created_at: 2025-02-25T06:16:33Z
author_association: MEMBER
diff_hunk: @@ -215,6 +205,64 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	waitGroup *sync.WaitGroup,
+	workload scalable.Workload,
+) {
+	slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+	defer waitGroup.Done()
+
+	scanSucceded := false
+
+	for retry := 0; retry <= config.MaxRetriesOnConflict; retry++ {
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: ```
	for retry := range config.MaxRetriesOnConflict+1 {
```
created_at: 2025-02-25T06:18:49Z
author_association: MEMBER
diff_hunk: @@ -215,6 +205,64 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	waitGroup *sync.WaitGroup,
+	workload scalable.Workload,
+) {
+	slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+	defer waitGroup.Done()
+
+	scanSucceded := false
+
+	for retry := 0; retry <= config.MaxRetriesOnConflict; retry++ {
+		err := scanWorkload(workload, client, ctx, layerCli, layerEnv, config)
+		if err != nil {
+			if !(strings.Contains(err.Error(), registry.OptimisticLockErrorMsg)) {
+				slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: if we wrap the function call in another function, we should definitely return the errors instead of logging them directly. This will also spare use from some verbose log messages
created_at: 2025-02-25T06:41:24Z
author_association: MEMBER
diff_hunk: @@ -102,6 +104,12 @@ func (c *RuntimeConfiguration) ParseConfigFlags() {
 		"",
 		"the annotation to use instead of creation time for grace period (optional)",
 	)
+	flag.IntVar(
+		&c.MaxRetriesOnConflict,
+		"max-retries-on-error",
path: internal/pkg/util/config.go
user: login: jonathan-mayer
body: in the docs it says `max-retries-on-conflict`
created_at: 2025-02-25T07:02:41Z
author_association: MEMBER
diff_hunk: @@ -102,6 +104,12 @@ func (c *RuntimeConfiguration) ParseConfigFlags() {
 		"",
 		"the annotation to use instead of creation time for grace period (optional)",
 	)
+	flag.IntVar(
+		&c.MaxRetriesOnConflict,
+		"max-retries-on-error",
path: internal/pkg/util/config.go
user: login: samuel-esp
body: nice catch!
created_at: 2025-02-25T19:35:32Z
author_association: COLLABORATOR
diff_hunk: @@ -215,6 +205,64 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	waitGroup *sync.WaitGroup,
+	workload scalable.Workload,
+) {
+	slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+	defer waitGroup.Done()
+
+	scanSucceded := false
+
+	for retry := 0; retry <= config.MaxRetriesOnConflict; retry++ {
+		err := scanWorkload(workload, client, ctx, layerCli, layerEnv, config)
+		if err != nil {
+			if !(strings.Contains(err.Error(), registry.OptimisticLockErrorMsg)) {
+				slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
path: cmd/kubedownscaler/main.go
user: login: samuel-esp
body: I agree on the risk on logging verbosity but I think it could be a correct choice here. We are not expecting to handle errors outside this function because if the workload couldn't be fetched we are not going to take action elsewhere to solve this problem, rather we'll wait for the next cycle

Maybe you mean returning the errors to log that outside `attemptScan`?

I'll leave that open to further discuss this @jonathan-mayer 
created_at: 2025-02-25T20:18:27Z
author_association: COLLABORATOR
diff_hunk: @@ -209,59 +210,65 @@ func attemptScan(
 	ctx context.Context,
 	layerCli, layerEnv *values.Layer,
 	config *util.RuntimeConfiguration,
-	waitGroup *sync.WaitGroup,
+	deferFunc func(),
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: ```suggestion
	doneFunc func(),
```
created_at: 2025-02-26T06:48:47Z
author_association: MEMBER
diff_hunk: @@ -30,7 +30,9 @@ type Client interface {
 	// GetNamespaceAnnotations gets the annotations of the workload's namespace
 	GetNamespaceAnnotations(namespace string, ctx context.Context) (map[string]string, error)
 	// GetWorkloads gets all workloads of the specified resources for the specified namespaces
-	GetWorkloads(name string, namespaces []string, resourceTypes []string, ctx context.Context) ([]scalable.Workload, error)
+	GetWorkloads(namespaces []string, resourceTypes []string, ctx context.Context) ([]scalable.Workload, error)
+	// RegetWorkload gets the workload again to ensure the latest state
+	RegetWorkload(name, namespace, resourceType string, ctx context.Context) (scalable.Workload, error)
path: internal/api/kubernetes/client.go
user: login: jonathan-mayer
body: we should be able to avoid the linter if we re-get the workload directly into the same workload object. just overwriting it in memory
created_at: 2025-02-26T06:53:22Z
author_association: MEMBER
diff_hunk: @@ -130,6 +132,18 @@ func (c client) GetWorkloads(name string, namespaces, resourceTypes []string, ct
 	return results, nil
 }
 
+// RegetWorkload gets the workload again to ensure the latest state.
+func (c client) RegetWorkload(name, namespace, resourceType string, ctx context.Context) (scalable.Workload, error) {
path: internal/api/kubernetes/client.go
user: login: jonathan-mayer
body: we shouldnt need all this. the code in main should just be able to do: client.RegetWorkload(workload, ctx)

then the only purpose of this function is to be a wrapper which calls workload.Reget(clientset, ctx)
created_at: 2025-02-26T06:55:33Z
author_association: MEMBER
diff_hunk: @@ -130,6 +132,18 @@ func (c client) GetWorkloads(name string, namespaces, resourceTypes []string, ct
 	return results, nil
 }
 
+// RegetWorkload gets the workload again to ensure the latest state.
+func (c client) RegetWorkload(name, namespace, resourceType string, ctx context.Context) (scalable.Workload, error) {
path: internal/api/kubernetes/client.go
user: login: samuel-esp
body: fine, I'll refactor this
created_at: 2025-02-26T11:06:53Z
author_association: COLLABORATOR
diff_hunk: @@ -215,6 +205,64 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	waitGroup *sync.WaitGroup,
+	workload scalable.Workload,
+) {
+	slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+	defer waitGroup.Done()
+
+	scanSucceded := false
+
+	for retry := 0; retry <= config.MaxRetriesOnConflict; retry++ {
+		err := scanWorkload(workload, client, ctx, layerCli, layerEnv, config)
+		if err != nil {
+			if !(strings.Contains(err.Error(), registry.OptimisticLockErrorMsg)) {
+				slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: well, but with every error where we just return after either way i dont see why we wouldn't handle it further up 
created_at: 2025-02-27T11:49:36Z
author_association: MEMBER
diff_hunk: @@ -215,6 +205,65 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	doneFunc func(),
+	workload scalable.Workload,
+) {
+	slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+	defer doneFunc()
+
+	var scanSucceded bool
+
+	for retry := range config.MaxRetriesOnConflict + 1 {
+		err := scanWorkload(workload, client, ctx, layerCli, layerEnv, config)
+		if err != nil {
+			if !(strings.Contains(err.Error(), registry.OptimisticLockErrorMsg)) {
+				slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+				return
+			}
+
+			slog.Warn("workload modified, retrying", "attempt", retry+1, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+			err := client.RegetWorkload(workload, ctx)
+			if err != nil {
+				slog.Error("failed to fetch updated workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+				return
+			}
+
+			continue
+		}
+
+		slog.Debug("successfully scanned workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+		scanSucceded = true
+
+		break
+	}
+
+	if !scanSucceded && config.MaxRetriesOnConflict > 0 {
+		slog.Warn("max retries reached, will try again in the next scan", "workload",
+			workload.GetName(),
+			"namespace",
+			workload.GetNamespace())
+
+		return
+	}
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: im not too much of a fan of putting logs in if statements and this message wouldn't fit into the case where `once` is specified (the main proposed use case of the retries argument). so i would try to change up these messages so that it is a bit more generic. 

`slog.Error(failed to scan workload, "tries/attempts", config.MaxRetriesOnConflict+1, ...)`

\*i would choose "tries" or "attempts", not both


we should be able to put this after the for loop. then instead of having the `scanSucceeded` bool we can just return where the `scanSucceeded = true` is now
created_at: 2025-02-28T06:13:22Z
author_association: MEMBER
diff_hunk: @@ -29,6 +28,18 @@ type cronJob struct {
 	*batch.CronJob
 }
 
+// Reget regets the resource from the Kubernetes API.
+func (c *cronJob) Reget(clientsets *Clientsets, ctx context.Context) error {
+	singleCronJob, err := clientsets.Kubernetes.BatchV1().CronJobs(c.Namespace).Get(ctx, c.Name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to get cronjob: %w", err)
+	}
+
+	c.CronJob = singleCronJob
+
+	return nil
path: internal/pkg/scalable/cronjobs.go
user: login: jonathan-mayer
body: 
```suggestion
    var err error
	c.CronJob, err = clientsets.Kubernetes.BatchV1().CronJobs(c.Namespace).Get(ctx, c.Name, metav1.GetOptions{})
	if err != nil {
		return fmt.Errorf("failed to get cronjob: %w", err)
	}

	return nil
```

this should be done with every resource
created_at: 2025-02-28T06:16:00Z
author_association: MEMBER
diff_hunk: @@ -81,6 +81,8 @@ type Workload interface {
 	ScaleUp() error
 	// ScaleDown scales down the workload
 	ScaleDown(downscaleReplicas int32) error
+	// Reget regets the workload to ensure the latest state
+	Reget(clientsets *Clientsets, ctx context.Context) error
path: internal/pkg/scalable/workload.go
user: login: jonathan-mayer
body: i think this should be in the scalableResource interface
created_at: 2025-02-28T06:19:27Z
author_association: MEMBER
diff_hunk: @@ -215,6 +205,64 @@ func startScanning(
 	return nil
 }
 
+func attemptScan(
+	client kubernetes.Client,
+	ctx context.Context,
+	layerCli, layerEnv *values.Layer,
+	config *util.RuntimeConfiguration,
+	waitGroup *sync.WaitGroup,
+	workload scalable.Workload,
+) {
+	slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
+
+	defer waitGroup.Done()
+
+	scanSucceded := false
+
+	for retry := 0; retry <= config.MaxRetriesOnConflict; retry++ {
+		err := scanWorkload(workload, client, ctx, layerCli, layerEnv, config)
+		if err != nil {
+			if !(strings.Contains(err.Error(), registry.OptimisticLockErrorMsg)) {
+				slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
path: cmd/kubedownscaler/main.go
user: login: samuel-esp
body: I got your point, I refactored that accordingly
created_at: 2025-02-28T11:31:46Z
author_association: COLLABORATOR
diff_hunk: @@ -187,7 +187,13 @@ func startScanning(
 		for _, workload := range workloads {
 			waitGroup.Add(1)
 
-			go attemptScan(client, ctx, layerCli, layerEnv, config, waitGroup.Done, workload)
+			go func() {
+				err = attemptScan(client, ctx, layerCli, layerEnv, config, waitGroup.Done, workload)
+				if err != nil {
+					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+					return
+				}
+			}()
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: oh, can you actually redo what you did [here](https://github.com/caas-team/GoKubeDownscaler/pull/74#discussion_r1905199891)
created_at: 2025-03-03T06:03:46Z
author_association: MEMBER
diff_hunk: @@ -186,17 +188,11 @@ func startScanning(
 			waitGroup.Add(1)
 
 			go func() {
-				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
-
-				defer waitGroup.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv, config)
+				err = attemptScan(client, ctx, layerCli, layerEnv, config, waitGroup.Done, workload)
 				if err != nil {
 					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
 					return
 				}
path: cmd/kubedownscaler/main.go
user: login: jonathan-mayer
body: from https://github.com/caas-team/GoKubeDownscaler/pull/74#discussion_r1969015079
> or we could wrap calling the attemptScan function in a anonymous func which handles waitGroup.Done and error handling
created_at: 2025-03-03T06:19:26Z
author_association: MEMBER
diff_hunk: @@ -186,17 +188,11 @@ func startScanning(
 			waitGroup.Add(1)
 
 			go func() {
-				slog.Debug("scanning workload", "workload", workload.GetName(), "namespace", workload.GetNamespace())
-
-				defer waitGroup.Done()
-
-				err := scanWorkload(workload, client, ctx, layerCli, layerEnv, config)
+				err = attemptScan(client, ctx, layerCli, layerEnv, config, waitGroup.Done, workload)
 				if err != nil {
 					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
 					return
 				}
path: cmd/kubedownscaler/main.go
user: login: samuel-esp
body: I don't know if I understood it correctly, I refactored the `waitGroup.Done()` outside the `attemptScan`
created_at: 2025-03-03T17:20:21Z
author_association: COLLABORATOR
diff_hunk: @@ -187,7 +187,13 @@ func startScanning(
 		for _, workload := range workloads {
 			waitGroup.Add(1)
 
-			go attemptScan(client, ctx, layerCli, layerEnv, config, waitGroup.Done, workload)
+			go func() {
+				err = attemptScan(client, ctx, layerCli, layerEnv, config, waitGroup.Done, workload)
+				if err != nil {
+					slog.Error("failed to scan workload", "error", err, "workload", workload.GetName(), "namespace", workload.GetNamespace())
+					return
+				}
+			}()
path: cmd/kubedownscaler/main.go
user: login: samuel-esp
body: yes!
created_at: 2025-03-03T17:35:29Z
author_association: COLLABORATOR
diff_hunk: @@ -14,7 +15,7 @@ const (
 	annotationKedaPausedReplicas = "autoscaling.keda.sh/paused-replicas"
 )
 
-// getScaledObjects is the getResourceFunc for Keda ScaledObjects.
+// getScaledObjects is the getResourceFunc for Keda ScaledObjects. //nolint:dupl.
path: internal/pkg/scalable/scaledobjects.go
user: login: jonathan-mayer
body: do we still need this no lint here if we have the whole file nolinted fo rthis linter

```suggestion
// getScaledObjects is the getResourceFunc for Keda ScaledObjects.
```
created_at: 2025-03-10T13:21:33Z
author_association: MEMBER
diff_hunk: @@ -9,7 +9,9 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
-// getStacks is the getResourceFunc for Zalando Stacks.
+// getStacks is the getResourceFunc for Zalando Stacks. //nolint:dupl.
+//
+//nolint:dupl // necessary to handle different workload types separately
path: internal/pkg/scalable/stacks.go
user: login: jonathan-mayer
body: ```suggestion
// getStacks is the getResourceFunc for Zalando Stacks.
//nolint:dupl // necessary to handle different workload types separately
```
created_at: 2025-03-10T13:22:34Z
author_association: MEMBER
diff_hunk: @@ -9,7 +9,9 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
-// getStacks is the getResourceFunc for Zalando Stacks.
+// getStacks is the getResourceFunc for Zalando Stacks. //nolint:dupl.
+//
+//nolint:dupl // necessary to handle different workload types separately
path: internal/pkg/scalable/stacks.go
user: login: jonathan-mayer
body: oh acutally, can you remove the //nolint:dupl on that func entirely. the file already has the nolint for this
created_at: 2025-03-11T11:42:18Z
author_association: MEMBER
diff_hunk: @@ -9,7 +9,9 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
-// getStacks is the getResourceFunc for Zalando Stacks.
+// getStacks is the getResourceFunc for Zalando Stacks. //nolint:dupl.
+//
+//nolint:dupl // necessary to handle different workload types separately
path: internal/pkg/scalable/stacks.go
user: login: samuel-esp
body: done!
created_at: 2025-03-11T14:42:34Z
author_association: COLLABORATOR

PR comments:
user: login: samuel-esp
created_at: 2025-02-19T23:34:08Z
body: @jonathan-mayer rebased and included the previous suggestions. Just a couple of things to note:

1. Pre-Commit failed because the specific `getResourceFunc` for each resource has duplicate code" detected (which is true because the logic is shared across them). So, eventually, we should try to think about a strategy to tackle that
2. Pre-Commit also suggested to lower a bit `startScanning` complexity. What would you refactor inside that?
author_association: COLLABORATOR
user: login: jonathan-mayer
created_at: 2025-02-20T06:16:11Z
body: Ill have a look over it. 

> 1. Pre-Commit failed because the specific getResourceFunc for each resource has duplicate code" detected (which is true because the logic is shared across them). So, eventually, we should try to think about a strategy to tackle that

I do get that it detects it as duplicate, but im not entirely sure how to avoid it. I think in theory it is avoidable by just having list and get in separate functions. Another way would also be to refactor to use the dynamic client again, although i would still like to avoid that.

> 2. Pre-Commit also suggested to lower a bit startScanning complexity. What would you refactor inside that?

I think we could refactor the function called in the go routine out. 
author_association: MEMBER
user: login: samuel-esp
created_at: 2025-02-20T18:43:10Z
body: 1. I perfectly agree with you, I don't see anything wrong with having them duplicate. I would nolint them
2. I'll extract the go routine from the function
author_association: COLLABORATOR
user: login: samuel-esp
created_at: 2025-02-20T19:56:33Z
body: @jonathan-mayer I refactored the go func out of the `startScanning` function however I'm not sure what could be the best name for the that. `attemptScan` was the best I came up with
author_association: COLLABORATOR
user: login: jonathan-mayer
created_at: 2025-02-21T05:58:49Z
body: > @jonathan-mayer I refactored the go func out of the `startScanning` function however I'm not sure what could be the best name for the that. `attemptScan` was the best I came up with

i think the current function name should be good for now. Although i would not put a anonymous function in the attemptScan function, but instead just run the attemptScan function in a goroutine so `go attemptScan()`. Also just a heads up, it might take some time until i get around to reviewing this.
author_association: MEMBER
user: login: samuel-esp
created_at: 2025-02-21T07:23:12Z
body: Don't worry Jonathan, absolutely no rush take your time
author_association: COLLABORATOR
user: login: jonathan-mayer
created_at: 2025-02-25T06:56:39Z
body: I think we should change up the GetResource funcs again. They should only handle listing and then every scalableResouce should have a re-get function which regets itself from kubernetes. it could like something like this:

```go
func (c *cronJob) Reget(clientsets *Clientsets, ctx context.Context) {
	*c, err = clientsets.Kubernetes.BatchV1().CronJobs(c.Namespace).Get(c.Name, ctx,, metav1.GetOptions{})
	if err != nil {
		// TODO handle error
	}
}
```
author_association: MEMBER
user: login: samuel-esp
created_at: 2025-02-25T21:15:57Z
body: > I think we should change up the GetResource funcs again. They should only handle listing and then every scalableResouce should have a re-get function which regets itself from kubernetes. it could like something like this:
> 
> ```go
> func (c *cronJob) Reget(clientsets *Clientsets, ctx context.Context) {
> 	*c, err = clientsets.Kubernetes.BatchV1().CronJobs(c.Namespace).Get(c.Name, ctx,, metav1.GetOptions{})
> 	if err != nil {
> 		// TODO handle error
> 	}
> }
> ```

Implemented, the linter still complains about 2 things:
- deployment/statefulset class duplication
- RegetWorkload and its specific methods returns an interface type

I mean the first one could be true, for the second one to me it seems legit to return an interface type in that case
author_association: COLLABORATOR
user: login: samuel-esp
created_at: 2025-02-26T21:03:51Z
body: refactored the way you suggested, still having "duplicate" suggestion on linting. I think we can't do much about that
author_association: COLLABORATOR
user: login: jonathan-mayer
created_at: 2025-03-04T11:52:04Z
body: Oh actually add the nolints for the dupl linter suggestion, then we can merge
author_association: MEMBER
user: login: samuel-esp
created_at: 2025-03-04T17:30:00Z
body: Done
author_association: COLLABORATOR
user: login: jonathan-mayer
created_at: 2025-03-05T10:50:12Z
body: feel free to merge this whenever

author_association: MEMBER
user: login: samuel-esp
created_at: 2025-03-07T16:16:04Z
body: @jonathan-mayer rebased and cleaned history
author_association: COLLABORATOR
user: login: samuel-esp
created_at: 2025-03-11T14:43:19Z
body: I think everthing is ready now
author_association: COLLABORATOR
user: login: jonathan-mayer
created_at: 2025-03-12T07:07:49Z
body: Resolved merge conflicts. Feel free to merge whenever
author_association: MEMBER