Repository: jcmoraisjr/haproxy-ingress

Issue:
Haproxy ingress doesn't report correct error messages or have a retry when an reload issue happens
<!--
Welcome to HAProxy Ingress and thank you for contributing to the project!

Please fill and submit this issue only if you have a problem running
HAProxy Ingress. Use Slack, mailing list or Stack Overflow instead if
you need support or if you are in doubt if you found a bug.
-->

**Description of the problem**

<!-- A clear and concise description of what's happening but shouldn't happen -->
We noticed that when haproxy reload faces an issue, especially when multiple reloads happen at the same time
-> haproxy ingress logs show that the communication to the master is refused:
`error reloading server: error sending reload to master socket: error connecting to /var/run/haproxy/master.sock: dial unix /var/run/haproxy/master.sock: connect: connection refused`
but also: 
`10 controller.go:366] finish haproxy reload id=2: reload_haproxy=5000.375786ms total=5000.375786ms`, 
which is very confusing, sounds like the haproxy reload finished successfully in 5 sec. 
But actually, the reload was not successful and there was no retry.
-> In this case, the failed/latest haproxy config update cannot be triggered successfully, they are not able to reach out to the backends, until the next reload succeeds again.

**Expected behavior**

<!-- A description of what you expected to happen -->

- Once a newer haproxy reload is refused because the last haproxy reload has not finished, the failed reload should be reported and a retry should be triggered between [the lines](https://github.com/jcmoraisjr/haproxy-ingress/blob/release-0.14/pkg/controller/controller.go#L365-L366). There could be some messages like `haproxy reload failed, will retry again`
- Making the [timeout setting for sending the socket](https://github.com/jcmoraisjr/haproxy-ingress/blob/release-0.14/pkg/haproxy/socket/socket.go#L52) configurable 


**Steps to reproduce the problem**

In our cases, the reload issues happen normally when a second reload is triggered when another reload takes longer to get a response and has not finished...
Therefore, we reproduce the issue by:
1. Delaying the startup of each new haproxy process (for example, by adding a sleep statement [here](https://github.com/haproxy/haproxy/blob/v2.9.0/src/haproxy.c#L3470)
2. Tigger the first reload process, and before it finishes, trigger the second haproxy reload
3. haproxy reload logs then show `error reloading server: error sending reload to master socket: error connecting to /var/run/haproxy/master.sock: dial unix /var/run/haproxy/master.sock: connect: connection refused`, and the haproxy connections cannot be routed to the backends, until a following reload succeeds again

**Environment information**

<!-- Edit v0.xx below with the version of your HAProxy Ingress -->

HAProxy Ingress version: `v0.14` and HAProxy version we are using: `v2.9.0`

<!-- Put the actual configuration inside code blocks ``` -->

Command-line options:

```
(paste command-line options here)
```

Global options:

```
(paste global configmap options here)
```

Ingress objects:

```
(paste ingress objects here)
```


Hints from issue comments:
@jcmoraisjr: Hi, can you confirm if this happens with a haproxy version older than `v2.9`, some `v2.8` maybe? It seems we have some compatibility issue with newer versions which is missing to be addressed.@danjiawork: Hi @jcmoraisjr, thanks for your response and sorry for the late answer!
Yes, this issue happens with the haproxy version v2.9.@jcmoraisjr: Good catch, thanks for sharing! In fact we didn't try to restart in the case of failure because historically the only reason for a failure was a misconfiguration, so changing the bad configuration would be responsible for triggering the restart. Now we'll retry a restart every 30s in the case of a failure.

Fixed on v0.15 and merged to v0.14. Closing.

Solution patch:
diff --git a/docs/content/en/docs/configuration/command-line.md b/docs/content/en/docs/configuration/command-line.md
index c1253b8e..db4aea43 100644
--- a/docs/content/en/docs/configuration/command-line.md
+++ b/docs/content/en/docs/configuration/command-line.md
@@ -53,8 +53,9 @@ The following command-line options are supported:
 | [`--publish-address`](#publish-address)                 | list of hostname/IP        |                         | v0.15 |
 | [`--publish-service`](#publish-service)                 | namespace/servicename      |                         |       |
 | [`--rate-limit-update`](#rate-limit-update)             | uploads per second (float) | `0.5`                   |       |
-| [`--reload-interval`](#reload-interval)                 | time                       | `0`                     | v0.13 |
 | [`--ready-check-path`](#stats)                          | path                       | `/readyz`               | v0.15 |
+| [`--reload-interval`](#reload-interval)                 | time                       | `0`                     | v0.13 |
+| [`--reload-retry`](#reload-retry)                       | time                       | `30s`                   | v0.15 |
 | [`--reload-strategy`](#reload-strategy)                 | [native\|reusesocket]      | `reusesocket`           |       |
 | [`--report-node-internal-ip-address`](#report-node-internal-ip-address) | [true\|false] | `false`              |       |
 | [`--sort-backends`](#sort-backends)                     | [true\|false]              | `false`                 |       |
@@ -495,6 +496,14 @@ one will delay up to the configured duration to be reflected by HAProxy.
 
 ---
 
+## reload-retry
+
+* `--reload-retry`
+
+How long HAProxy Ingress should wait before trying to reload HAProxy if an error happens. Defaults to `30s`.
+
+---
+
 ## reload-strategy
 
 * `--reload-strategy`
diff --git a/go.mod b/go.mod
index 61a5fbef..40f84bf9 100644
--- a/go.mod
+++ b/go.mod
@@ -12,12 +12,12 @@ require (
 	github.com/jinzhu/copier v0.4.0
 	github.com/kylelemons/godebug v1.1.0
 	github.com/mitchellh/mapstructure v1.5.0
-	github.com/prometheus/client_golang v1.21.0
+	github.com/prometheus/client_golang v1.21.1
 	github.com/spf13/pflag v1.0.6
 	github.com/stretchr/testify v1.10.0
 	go.uber.org/zap v1.27.0
-	golang.org/x/crypto v0.35.0
-	golang.org/x/sync v0.11.0
+	golang.org/x/crypto v0.36.0
+	golang.org/x/sync v0.12.0
 	gopkg.in/go-playground/pool.v3 v3.1.1
 	gopkg.in/yaml.v2 v2.4.0
 	k8s.io/api v0.32.2
@@ -26,7 +26,7 @@ require (
 	k8s.io/client-go v0.32.2
 	k8s.io/klog/v2 v2.130.1
 	k8s.io/utils v0.0.0-20241210054802-24370beab758
-	sigs.k8s.io/controller-runtime v0.20.2
+	sigs.k8s.io/controller-runtime v0.20.3
 	sigs.k8s.io/gateway-api v1.0.0
 )
 
@@ -76,11 +76,11 @@ require (
 	go.opentelemetry.io/otel v1.34.0 // indirect
 	go.opentelemetry.io/otel/trace v1.34.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
-	golang.org/x/net v0.35.0 // indirect
+	golang.org/x/net v0.36.0 // indirect
 	golang.org/x/oauth2 v0.27.0 // indirect
-	golang.org/x/sys v0.30.0 // indirect
-	golang.org/x/term v0.29.0 // indirect
-	golang.org/x/text v0.22.0 // indirect
+	golang.org/x/sys v0.31.0 // indirect
+	golang.org/x/term v0.30.0 // indirect
+	golang.org/x/text v0.23.0 // indirect
 	golang.org/x/time v0.10.0 // indirect
 	gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
 	google.golang.org/protobuf v1.36.5 // indirect
diff --git a/go.sum b/go.sum
index 2d67f0e6..2491cd63 100644
--- a/go.sum
+++ b/go.sum
@@ -103,8 +103,8 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prometheus/client_golang v1.21.0 h1:DIsaGmiaBkSangBgMtWdNfxbMNdku5IK6iNhrEqWvdA=
-github.com/prometheus/client_golang v1.21.0/go.mod h1:U9NM32ykUErtVBxdvD3zfi+EuFkkaBvMb09mIfe0Zgg=
+github.com/prometheus/client_golang v1.21.1 h1:DOvXXTqVzvkIewV/CDPFdejpMCGeMcbGCQ8YOmu+Ibk=
+github.com/prometheus/client_golang v1.21.1/go.mod h1:U9NM32ykUErtVBxdvD3zfi+EuFkkaBvMb09mIfe0Zgg=
 github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
 github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
 github.com/prometheus/common v0.62.0 h1:xasJaQlnWAeyHdUBeGjXmutelfJHWMRr+Fg4QszZ2Io=
@@ -143,34 +143,34 @@ go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/crypto v0.35.0 h1:b15kiHdrGCHrP6LvwaQ3c03kgNhhiMgvlhxHQhmg2Xs=
-golang.org/x/crypto v0.35.0/go.mod h1:dy7dXNW32cAb/6/PRuTNsix8T+vJAqvuIy5Bli/x0YQ=
+golang.org/x/crypto v0.36.0 h1:AnAEvhDddvBdpY+uR+MyHmuZzzNqXSe/GvuDeob5L34=
+golang.org/x/crypto v0.36.0/go.mod h1:Y4J0ReaxCR1IMaabaSMugxJES1EpwhBHhv2bDHklZvc=
 golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/net v0.35.0 h1:T5GQRQb2y08kTAByq9L4/bz8cipCdA8FbRTXewonqY8=
-golang.org/x/net v0.35.0/go.mod h1:EglIi67kWsHKlRzzVMUD93VMSWGFOMSZgxFjparz1Qk=
+golang.org/x/net v0.36.0 h1:vWF2fRbw4qslQsQzgFqZff+BItCvGFQqKzKIzx1rmoA=
+golang.org/x/net v0.36.0/go.mod h1:bFmbeoIPfrw4sMHNhb4J9f6+tPziuGjq7Jk/38fxi1I=
 golang.org/x/oauth2 v0.27.0 h1:da9Vo7/tDv5RH/7nZDz1eMGS/q1Vv1N/7FCrBhI9I3M=
 golang.org/x/oauth2 v0.27.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.11.0 h1:GGz8+XQP4FvTTrjZPzNKTMFtSXH80RAzG+5ghFPgK9w=
-golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
+golang.org/x/sync v0.12.0 h1:MHc5BpPuC30uJk597Ri8TV3CNZcTLu6B6z4lJy+g6Jw=
+golang.org/x/sync v0.12.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
-golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/term v0.29.0 h1:L6pJp37ocefwRRtYPKSWOWzOtWSxVajvz2ldH/xi3iU=
-golang.org/x/term v0.29.0/go.mod h1:6bl4lRlvVuDgSf3179VpIxBF0o10JUpXWOnI7nErv7s=
+golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
+golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+golang.org/x/term v0.30.0 h1:PQ39fJZ+mfadBm0y5WlL4vlM7Sx1Hgf13sMIY2+QS9Y=
+golang.org/x/term v0.30.0/go.mod h1:NYYFdzHoI5wRh/h5tDMdMqCqPJZEuNqVR5xJLd/n67g=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.22.0 h1:bofq7m3/HAFvbF51jz3Q9wLg3jkvSPuiZu/pD1XwgtM=
-golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
+golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY=
+golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4=
 golang.org/x/time v0.10.0 h1:3usCWA8tQn0L8+hFJQNgzpWbd89begxN66o1Ojdn5L4=
 golang.org/x/time v0.10.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -220,8 +220,8 @@ k8s.io/kube-openapi v0.0.0-20241212222426-2c72e554b1e7 h1:hcha5B1kVACrLujCKLbr8X
 k8s.io/kube-openapi v0.0.0-20241212222426-2c72e554b1e7/go.mod h1:GewRfANuJ70iYzvn+i4lezLDAFzvjxZYK1gn1lWcfas=
 k8s.io/utils v0.0.0-20241210054802-24370beab758 h1:sdbE21q2nlQtFh65saZY+rRM6x6aJJI8IUa1AmH/qa0=
 k8s.io/utils v0.0.0-20241210054802-24370beab758/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
-sigs.k8s.io/controller-runtime v0.20.2 h1:/439OZVxoEc02psi1h4QO3bHzTgu49bb347Xp4gW1pc=
-sigs.k8s.io/controller-runtime v0.20.2/go.mod h1:xg2XB0K5ShQzAgsoujxuKN4LNXR2LfwwHsPj7Iaw+XY=
+sigs.k8s.io/controller-runtime v0.20.3 h1:I6Ln8JfQjHH7JbtCD2HCYHoIzajoRxPNuvhvcDbZgkI=
+sigs.k8s.io/controller-runtime v0.20.3/go.mod h1:xg2XB0K5ShQzAgsoujxuKN4LNXR2LfwwHsPj7Iaw+XY=
 sigs.k8s.io/gateway-api v1.0.0 h1:iPTStSv41+d9p0xFydll6d7f7MOBGuqXM6p2/zVYMAs=
 sigs.k8s.io/gateway-api v1.0.0/go.mod h1:4cUgr0Lnp5FZ0Cdq8FdRwCvpiWws7LVhLHGIudLlf4c=
 sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=
diff --git a/pkg/controller/config/config.go b/pkg/controller/config/config.go
index 1648daf2..b4446225 100644
--- a/pkg/controller/config/config.go
+++ b/pkg/controller/config/config.go
@@ -506,6 +506,7 @@ func CreateWithConfig(ctx context.Context, restConfig *rest.Config, opt *Options
 		RateLimitUpdate:          opt.RateLimitUpdate,
 		ReadyzURL:                opt.ReadyzURL,
 		ReloadInterval:           opt.ReloadInterval,
+		ReloadRetry:              opt.ReloadRetry,
 		ReloadStrategy:           opt.ReloadStrategy,
 		ResyncPeriod:             &opt.ResyncPeriod,
 		RootContext:              rootcontext,
@@ -689,6 +690,7 @@ type Config struct {
 	RateLimitUpdate          float64
 	ReadyzURL                string
 	ReloadInterval           time.Duration
+	ReloadRetry              time.Duration
 	ReloadStrategy           string
 	ResyncPeriod             *time.Duration
 	RootContext              context.Context
diff --git a/pkg/controller/config/options.go b/pkg/controller/config/options.go
index c07d4e19..db808688 100644
--- a/pkg/controller/config/options.go
+++ b/pkg/controller/config/options.go
@@ -24,6 +24,7 @@ func NewOptions() *Options {
 		AnnPrefix:               "haproxy-ingress.github.io,ingress.kubernetes.io",
 		RateLimitUpdate:         0.5,
 		WaitBeforeUpdate:        200 * time.Millisecond,
+		ReloadRetry:             30 * time.Second,
 		ResyncPeriod:            10 * time.Hour,
 		WatchNamespace:          corev1.NamespaceAll,
 		StatsCollectProcPeriod:  500 * time.Millisecond,
@@ -71,6 +72,7 @@ type Options struct {
 	AnnPrefix                string
 	RateLimitUpdate          float64
 	ReloadInterval           time.Duration
+	ReloadRetry              time.Duration
 	WaitBeforeUpdate         time.Duration
 	ResyncPeriod             time.Duration
 	WatchNamespace           string
@@ -288,6 +290,10 @@ func (o *Options) AddFlags(fs *flag.FlagSet) {
 		"one, applying every new configuration changes made between this interval.",
 	)
 
+	fs.DurationVar(&o.ReloadRetry, "reload-retry", o.ReloadRetry, ""+
+		"How long HAProxy Ingress should wait before trying to reload HAProxy if an error "+
+		"happens.")
+
 	fs.DurationVar(&o.WaitBeforeUpdate, "wait-before-update", o.WaitBeforeUpdate, ""+
 		"Amount of time to wait before start a reconciliation and update haproxy, giving "+
 		"the time to receive all/most of the changes of a batch update.",
diff --git a/pkg/controller/legacy/controller.go b/pkg/controller/legacy/controller.go
index 2645ed26..1afcb4ef 100644
--- a/pkg/controller/legacy/controller.go
+++ b/pkg/controller/legacy/controller.go
@@ -345,8 +345,13 @@ func (hc *HAProxyController) syncIngress(item interface{}) {
 	// update proxy
 	//
 	hc.instance.AcmeUpdate()
-	hc.instance.HAProxyUpdate(timer)
-	hc.logger.Info("finish haproxy update id=%d: %s", hc.updateCount, timer.AsString("total"))
+	err := hc.instance.HAProxyUpdate(timer)
+	if err != nil {
+		hc.logger.Error("error trying to update haproxy id=%d, retrying in 30s: %s err=%s", hc.reloadCount, timer.AsString("total"), err)
+		hc.ingressQueue.AddAfter(nil, 30*time.Second)
+	} else {
+		hc.logger.Info("finish haproxy update id=%d: %s", hc.updateCount, timer.AsString("total"))
+	}
 }
 
 func (hc *HAProxyController) acmeCheck(source string) (int, error) {
@@ -363,6 +368,11 @@ func (hc *HAProxyController) reloadHAProxy(item interface{}) {
 	hc.logger.Info("starting haproxy reload id=%d", hc.reloadCount)
 	timer := utils.NewTimer(hc.metrics.ControllerProcTime)
 
-	hc.instance.Reload(timer)
-	hc.logger.Info("finish haproxy reload id=%d: %s", hc.reloadCount, timer.AsString("total"))
+	err := hc.instance.Reload(timer)
+	if err != nil {
+		hc.logger.Error("error trying to reload haproxy id=%d, retrying in 30s: %s err=%s", hc.reloadCount, timer.AsString("total"), err)
+		hc.reloadQueue.AddAfter(nil, 30*time.Second)
+	} else {
+		hc.logger.Info("finish haproxy reload id=%d: %s", hc.reloadCount, timer.AsString("total"))
+	}
 }
diff --git a/pkg/controller/reconciler/reconciler.go b/pkg/controller/reconciler/reconciler.go
index 43b1b730..6117253d 100644
--- a/pkg/controller/reconciler/reconciler.go
+++ b/pkg/controller/reconciler/reconciler.go
@@ -18,8 +18,10 @@ package reconciler
 
 import (
 	"context"
+	"fmt"
 
 	"github.com/go-logr/logr"
+	k8sworkqueue "k8s.io/client-go/util/workqueue"
 	"k8s.io/utils/ptr"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -37,29 +39,41 @@ type IngressReconciler struct {
 	Config   *config.Config
 	Services *services.Services
 	//
+	log      logr.Logger
 	watchers *watchers
+	queue    k8sworkqueue.TypedRateLimitingInterface[ctrl.Request]
 }
 
 // Reconcile ...
 func (r *IngressReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 	changed := r.watchers.getChangedObjects()
-	r.Services.ReconcileIngress(ctx, changed)
+	err := r.Services.ReconcileIngress(ctx, changed)
+	if err != nil {
+		r.log.Error(err, fmt.Sprintf("error reconciling ingress, retrying in %s", r.Config.ReloadRetry.String()))
+		return ctrl.Result{RequeueAfter: r.Config.ReloadRetry}, nil
+	}
 	return ctrl.Result{}, nil
 }
 
 func (r *IngressReconciler) leaderChanged(ctx context.Context, isLeader bool) {
 	if isLeader && r.watchers.running() {
-		changed := r.watchers.getChangedObjects()
-		changed.NeedFullSync = true
-		r.Services.ReconcileIngress(ctx, changed)
+		r.log.Info("enqueue reconciliation due to leader acquired")
+		r.queue.AddRateLimited(reconcile.Request{})
 	}
 }
 
 // SetupWithManager ...
 func (r *IngressReconciler) SetupWithManager(ctx context.Context, mgr ctrl.Manager) error {
+	r.log = logr.FromContextOrDiscard(ctx).WithName("ingress")
 	r.watchers = createWatchers(ctx, r.Config, r.Services.GetIsValidResource())
 	opt := controller.Options{
-		LogConstructor:     func(*reconcile.Request) logr.Logger { return logr.FromContextOrDiscard(ctx).WithName("reconciler") },
+		LogConstructor: func(*reconcile.Request) logr.Logger { return logr.FromContextOrDiscard(ctx).WithName("reconciler") },
+		NewQueue: func(controllerName string, rateLimiter k8sworkqueue.TypedRateLimiter[reconcile.Request]) k8sworkqueue.TypedRateLimitingInterface[reconcile.Request] {
+			r.queue = k8sworkqueue.NewTypedRateLimitingQueueWithConfig(rateLimiter, k8sworkqueue.TypedRateLimitingQueueConfig[ctrl.Request]{
+				Name: controllerName,
+			})
+			return r.queue
+		},
 		RateLimiter:        workqueue.IngressReconcilerRateLimiter(r.Config.RateLimitUpdate, r.Config.WaitBeforeUpdate),
 		Reconciler:         r,
 		RecoverPanic:       ptr.To(true),
diff --git a/pkg/controller/services/services.go b/pkg/controller/services/services.go
index 4cfb2f94..d1a93a53 100644
--- a/pkg/controller/services/services.go
+++ b/pkg/controller/services/services.go
@@ -256,7 +256,7 @@ func (s *Services) GetIsValidResource() IsValidResource {
 }
 
 // ReconcileIngress ...
-func (s *Services) ReconcileIngress(ctx context.Context, changed *convtypes.ChangedObjects) {
+func (s *Services) ReconcileIngress(ctx context.Context, changed *convtypes.ChangedObjects) error {
 	s.modelMutex.Lock()
 	defer s.modelMutex.Unlock()
 	s.updateCount++
@@ -266,9 +266,15 @@ func (s *Services) ReconcileIngress(ctx context.Context, changed *convtypes.Chan
 	if s.svcleader.isLeader() {
 		s.instance.AcmeUpdate()
 	}
-	s.instance.HAProxyUpdate(timer)
+	err := s.instance.HAProxyUpdate(timer)
 	s.svcstatusing.changed(ctx, changed)
-	s.log.WithValues("id", s.updateCount).WithValues(timer.AsValues("total")...).Info("finish haproxy update")
+	updatelogger := s.log.WithValues("id", s.updateCount).WithValues(timer.AsValues("total")...)
+	if err != nil {
+		updatelogger.Error(err, fmt.Sprintf("error trying to update haproxy, retrying in %s", s.Config.ReloadRetry.String()))
+	} else {
+		updatelogger.Info("finish haproxy update")
+	}
+	return err
 }
 
 func (s *Services) acmeCheck(source string) (count int, err error) {
@@ -297,7 +303,15 @@ func (s *Services) reloadHAProxy(context.Context, any) error {
 	s.reloadCount++
 	s.log.Info("starting haproxy reload", "id", s.reloadCount)
 	timer := utils.NewTimer(s.metrics.ControllerProcTime)
-	s.instance.Reload(timer)
-	s.log.WithValues("id", s.reloadCount).WithValues(timer.AsValues("total")...).Info("finish haproxy reload")
+	err := s.instance.Reload(timer)
+	reloadlogger := s.log.WithValues("id", s.reloadCount).WithValues(timer.AsValues("total")...)
+	if err != nil {
+		reloadlogger.Error(err, fmt.Sprintf("error trying to reload haproxy, retrying in %s", s.Config.ReloadRetry.String()))
+		s.reloadQueue.AddAfter(nil, s.Config.ReloadRetry)
+	} else {
+		reloadlogger.Info("finish haproxy reload")
+	}
+	// return err!=nil adds another event into the queue, we're reloading manually
+	// in the case an error happens in order to override the rate limit of the reload queue.
 	return nil
 }
diff --git a/pkg/controller/services/svcacme.go b/pkg/controller/services/svcacme.go
index f248d351..41c4c319 100644
--- a/pkg/controller/services/svcacme.go
+++ b/pkg/controller/services/svcacme.go
@@ -114,6 +114,13 @@ func (s *svcAcmeClient) Add(item interface{}) {
 	}
 }
 
+// implements utils.QueueFacade
+func (s *svcAcmeClient) AddAfter(item interface{}, duration time.Duration) {
+	if s.leader.isLeader() {
+		s.queue.AddAfter(item, duration)
+	}
+}
+
 // implements utils.QueueFacade
 func (s *svcAcmeClient) Remove(item interface{}) {
 	s.queue.Remove(item)
diff --git a/pkg/haproxy/instance.go b/pkg/haproxy/instance.go
index f775b1b3..83966be7 100644
--- a/pkg/haproxy/instance.go
+++ b/pkg/haproxy/instance.go
@@ -71,8 +71,8 @@ type Instance interface {
 	Config() Config
 	CalcIdleMetric()
 	AcmeUpdate()
-	HAProxyUpdate(timer *utils.Timer)
-	Reload(timer *utils.Timer)
+	HAProxyUpdate(timer *utils.Timer) error
+	Reload(timer *utils.Timer) error
 	Shutdown()
 }
 
@@ -288,10 +288,10 @@ func (i *instance) AcmeUpdate() {
 	}
 }
 
-func (i *instance) HAProxyUpdate(timer *utils.Timer) {
+func (i *instance) HAProxyUpdate(timer *utils.Timer) error {
 	// nil config, just ignore
 	if i.config == nil {
-		return
+		return nil
 	}
 	//
 	// this should be taken into account when refactoring this func:
@@ -303,24 +303,20 @@ func (i *instance) HAProxyUpdate(timer *utils.Timer) {
 	i.config.SyncConfig()
 	i.config.Shrink()
 	if err := i.config.WriteTCPServicesMaps(); err != nil {
-		i.logger.Error("error building tcp services maps: %v", err)
 		i.metrics.IncUpdateNoop()
-		return
+		return fmt.Errorf("error building tcp services maps: %w", err)
 	}
 	if err := i.config.WriteFrontendMaps(); err != nil {
-		i.logger.Error("error building frontend maps: %v", err)
 		i.metrics.IncUpdateNoop()
-		return
+		return fmt.Errorf("error building frontend maps: %w", err)
 	}
 	if err := i.config.WriteBackendMaps(); err != nil {
-		i.logger.Error("error building backend maps: %v", err)
 		i.metrics.IncUpdateNoop()
-		return
+		return fmt.Errorf("error building backend maps: %w", err)
 	}
 	if err := i.writeCrtLists(); err != nil {
-		i.logger.Error("error building certificates lists: %v", err)
 		i.metrics.IncUpdateNoop()
-		return
+		return fmt.Errorf("error building certificates lists: %w", err)
 	}
 	timer.Tick("write_maps")
 	if !i.options.fake {
@@ -344,9 +340,8 @@ func (i *instance) HAProxyUpdate(timer *utils.Timer) {
 		err := i.writeConfig()
 		timer.Tick("write_config")
 		if err != nil {
-			i.logger.Error("error writing configuration: %v", err)
 			i.metrics.IncUpdateNoop()
-			return
+			return fmt.Errorf("error writing configuration: %w", err)
 		}
 	}
 	i.updateCertExpiring()
@@ -371,17 +366,17 @@ func (i *instance) HAProxyUpdate(timer *utils.Timer) {
 			i.logger.Info("old and new configurations match")
 			i.metrics.IncUpdateNoop()
 		}
-		return
+		return nil
 	}
 	if i.options.ReloadQueue != nil {
 		i.options.ReloadQueue.Add(nil)
 		i.logger.InfoV(2, "haproxy reload enqueued")
-	} else {
-		i.Reload(timer)
+		return nil
 	}
+	return i.Reload(timer)
 }
 
-func (i *instance) Reload(timer *utils.Timer) {
+func (i *instance) Reload(timer *utils.Timer) error {
 	i.metrics.IncUpdateFull()
 	if i.options.TrackInstances {
 		timeoutStopDur := i.config.Global().TimeoutStopDuration
@@ -393,12 +388,11 @@ func (i *instance) Reload(timer *utils.Timer) {
 	err := i.reloadHAProxy()
 	timer.Tick("reload_haproxy")
 	if err != nil {
-		i.logger.Error("error reloading server: %v", err)
 		i.updateSuccessful(false)
 		if i.options.TrackInstances {
 			i.conns.ReleaseLastInstance()
 		}
-		return
+		return fmt.Errorf("error reloading server: %w", err)
 	}
 	i.up = true
 	i.updateSuccessful(true)
@@ -414,6 +408,7 @@ func (i *instance) Reload(timer *utils.Timer) {
 		message += "; tracked instance(s): " + strconv.Itoa(i.conns.OldInstancesCount())
 	}
 	i.logger.Info(message)
+	return nil
 }
 
 func (i *instance) Shutdown() {
diff --git a/pkg/main.go b/pkg/main.go
index dd096509..024d7a56 100644
--- a/pkg/main.go
+++ b/pkg/main.go
@@ -14,8 +14,11 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
+//go:debug default=go1.18
 package main
 
+// TODO remove go1.18 compatibility after dropping legacy mode
+
 import (
 	"flag"
 	"log"
diff --git a/pkg/utils/queue.go b/pkg/utils/queue.go
index 1dd9497f..96b4401e 100644
--- a/pkg/utils/queue.go
+++ b/pkg/utils/queue.go
@@ -40,6 +40,7 @@ type Queue interface {
 // QueueFacade ...
 type QueueFacade interface {
 	Add(item interface{})
+	AddAfter(item interface{}, duration time.Duration)
 	Remove(item interface{})
 	Start(context.Context) error
 }
@@ -106,6 +107,13 @@ func (q *queue) Add(item interface{}) {
 	q.workqueue.Add(item)
 }
 
+func (q *queue) AddAfter(item interface{}, duration time.Duration) {
+	q.mutex.Lock()
+	defer q.mutex.Unlock()
+	delete(q.forget, item)
+	q.workqueue.AddAfter(item, duration)
+}
+
 func (q *queue) Notify() {
 	// When using with rateLimiter, `nil` will be deduplicated
 	// and `queue.Get()` will release call to `sync()` just once



Test patch:
diff --git a/pkg/haproxy/instance_test.go b/pkg/haproxy/instance_test.go
index 55ce9980..3042719f 100644
--- a/pkg/haproxy/instance_test.go
+++ b/pkg/haproxy/instance_test.go
@@ -5674,7 +5674,7 @@ INFO haproxy successfully reloaded (embedded daemon)`
 func (c *testConfig) Update() {
 	timer := utils.NewTimer(nil)
 	c.instance.AcmeUpdate()
-	c.instance.HAProxyUpdate(timer)
+	_ = c.instance.HAProxyUpdate(timer)
 }
 
 func (c *testConfig) checkConfig(expected string) {
diff --git a/tests/framework/framework.go b/tests/framework/framework.go
index 485c0a6c..37e02d14 100644
--- a/tests/framework/framework.go
+++ b/tests/framework/framework.go
@@ -836,5 +836,5 @@ func RandomName(prefix string) string {
 var currentPortOffset atomic.Int32
 
 func RandomPort() int32 {
-	return 49152 + currentPortOffset.Add(1)
+	return 16384 + currentPortOffset.Add(1)
 }