Repository: kubernetes-sigs/external-dns

Issue:
external-dns getting wedged
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

-->

**What happened**: external-dns from time to time gets stuck and only gets unwedged when there's an update to endpoints. 
logs:
```
time="2025-06-04T16:04:20Z" level=info msg="All records are already up to date"
time="2025-06-04T16:20:50Z" level=error msg="Failed to do run once: soft error\nrecords retrieval failed: soft error\nfailed to list hosted zones: operation error Route 53: ListHostedZones, https response error StatusCode: 403, RequestID: <redacted>, api error SignatureDoesNotMatch: Signature expired: 20250604T160450Z is now earlier than 20250604T161550Z (20250604T162050Z - 5 min.)"
time="2025-06-04T16:20:50Z" level=info msg="Applying provider record filter for domains: [<redacted>.]"
time="2025-06-04T16:20:50Z" level=info msg="All records are already up to date"
time="2025-06-04T16:21:20Z" level=info msg="Applying provider record filter for domains: [<redacted>.]"
time="2025-06-04T16:21:20Z" level=info msg="All records are already up to date"
```

There's around 20mins between "All records are already up to date" and the next update which is when I deleted a pod from service manually, triggering a reconciliation.

**What you expected to happen**:
The service should run the reconciliation on an interval (set to 30s)

**How to reproduce it (as minimally and precisely as possible)**:
I don't know, it happens randomly. But has been happening few times a day 

**Anything else we need to know?**:

**Environment**:

- External-DNS version (use `external-dns --version`): v0.17.0
- DNS provider: AWS r53
- Others:


Hints from issue comments:
@ivankatliarchuk: What are the actual arguments, not helm? How many records are there?@berkeli: ```
    Args:
      --log-level=info
      --log-format=text
      --interval=30s
      --events
      --source=service
      --policy=sync
      --registry=txt
      --txt-owner-id=<kube cluster name>
      --domain-filter=<redacted>
      --provider=aws
```

There's only 1 record in this cluster, but we run 15 clusters each with it's own controller. They all add record to the same zone, hence why `--txt-owner-id=<kube cluster name>`

oh and the pod getting stuck has been happening in a few clusters, so doesn't seem specific to 1 cluster@berkeli: It's very likely this was introduced in a recent update, as we have last week switched from `v0.14.2` to `v0.17.0` and I don't see metrics reporting this issue before the upgrade

![Image](https://github.com/user-attachments/assets/0e45002f-5584-4185-9fbe-edaa1cd7086f)@ivankatliarchuk: I'm wondering if this involves a single AWS account with multiple clusters. It's possible that one of your External-DNS instances is encountering throttling issues. Route53, for example, has a limit of 5 requests per second. If all instances are operating within a single account, this could easily become a bottleneck, especially with a nuber of zones and records.

External-dns migration to aws-sdk-go-v2 happens in version 0.15 https://github.com/kubernetes-sigs/external-dns/pull/4640, this is a major change, could be related, but difficutl to say.
@ivankatliarchuk: I was trying to find out what the error msg means, found issues
- https://github.com/aws/aws-sdk-go-v2/issues/2567
- https://github.com/cert-manager/cert-manager/issues/668

Are there static AWS credentials or IRSA role?@berkeli: > I'm wondering if this involves a single AWS account with multiple clusters. It's possible that one of your External-DNS instances is encountering throttling issues. Route53, for example, has a limit of 5 requests per second. If all instances are operating within a single account, this could easily become a bottleneck, especially with a nuber of zones and records.
> 
> External-dns migration to aws-sdk-go-v2 happens in version 0.15 [#4640](https://github.com/kubernetes-sigs/external-dns/pull/4640), this is a major change, could be related, but difficutl to say.

Wouldn't throttling errors show up in logs? I've seen a few when setting interval to 10s

> Are there static AWS credentials or IRSA role?

IRSA Role

I was a bit confused by that error too. 

I'm going to enable debug logging to see if that might shed some light and keep an eye. @berkeli: This happened with debug logs enabled, not much in the logs sadly

```

Jun 12, 2025 @ 19:19:26.440 | time="2025-06-12T18:19:26Z" level=error msg="Failed to do run once: soft error\nrecords retrieval failed: soft error\nfailed to list hosted zones: operation error Route 53: ListHostedZones, https response error StatusCode: 403, RequestID: 525146ed-b204-4736-87d9-3977a6aa0bf8, api error SignatureDoesNotMatch: Signature expired: 20250612T180248Z is now earlier than 20250612T181426Z (20250612T181926Z - 5 min.)"
-- | --

  | Jun 12, 2025 @ 19:19:26.440 | time="2025-06-12T18:19:26Z" level=debug msg="Refreshing zones list cache"

  | Jun 12, 2025 @ 19:02:48.508 | time="2025-06-12T18:02:48Z" level=debug msg="Refreshing zones list cache"


```

After `Refreshing zones list cache` it didn't do anything at all for ~20 mins and started again 🤷 . This is the same pod, no restarts or anytihng@ivankatliarchuk: Hard to say. We have no budget and capacity to reproduce this bug. 

If there is `Refreshing zones list cache`, this means that external-dns is actually working, and there are no error, and limited debug logs. It could be that, there are no records, or no zone found as  `--domain-filter` is set.

Is there is a debug messages like
- `Considering zone:`
- `failed to list hosted zones`


AWS provider seems like have limited debug logging. Something for sure needs fixing.@ivankatliarchuk: Maybe an issue in unclear.

> There's around 20mins between "All records are already up to date" and the next update which is when I deleted a pod from service manually, triggering a reconciliation.

This sounds like an expected behaviour. There is nothing to reconcile, and only when pod is deleted and re-created external-dns actually see a difference.

You may need to share kubernetes manifests with services and pod setup. Maybe there is an easy way to reproduce on local cluster. As well as how many approximate nodes, services and pods there?@berkeli: Hmm so I agree that the service works fine and it doesn't seem broken (because manually removing an endpoint from a service unwedges things, or if you just wait for 20mins), but it's an odd behaviour that we can see `All records are already up to date` posted on an interval (30s), but then there's a gap of ~20 mins. 

![Image](https://github.com/user-attachments/assets/407b9482-6398-4b8a-bb60-f8e93a2e4d4a)

And during that 20min gap there are absolutely no logs, debug or otherwise. @berkeli: I'm also inclined to enable debug logs for aws sdk, so I might give that a go with a fork@k8s-triage-robot: The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle stale`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle stale

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@sichenzhao: we have recently updated from 0.14.0 to 0.18.0, seeing similar behavior after updates:

for us, it's throttling error
```
2025-09-15T11:01:46.257976091Z {"level":"info","msg":"All records are already up to date","time":"2025-09-15T11:01:46Z"}
2025-09-15T12:02:06.328077978Z {"level":"error","msg":"Failed to do run once: soft error\nfailed to list resource records sets for zone /hostedzone/<zone-id> using aws profile \"default\": operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: a485c530-235a-4e63-a3d9-c7c4eff34c3d, api error Throttling: Rate exceeded (consecutive soft errors: 1)","time":"2025-09-15T12:02:06Z"}
2025-09-15T13:01:56.903825606Z {"level":"error","msg":"Failed to do run once: soft error\nfailed to list resource records sets for zone /hostedzone/<zone-id> using aws profile \"default\": operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: ff91dc1a-32a2-43e5-be55-912c92dd34bc, api error Throttling: Rate exceeded (consecutive soft errors: 2)","time":"2025-09-15T13:01:56Z"}
2025-09-15T14:02:15.345399710Z {"level":"error","msg":"Failed to do run once: soft error\nfailed to list resource records sets for zone /hostedzone/<zone-id> using aws profile \"default\": operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: a10a73e4-c50f-4991-9ed7-259f3ec0a58c, api error Throttling: Rate exceeded (consecutive soft errors: 3)","time":"2025-09-15T14:02:15Z"}
2025-09-15T14:23:41.112398404Z {"level":"error","msg":"Failed to do run once: soft error\nfailed to list resource records sets for zone /hostedzone/<zone-id> using aws profile \"default\": operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: ad0f114f-490d-4430-915e-98a7ad59a081, api error Throttling: Rate exceeded (consecutive soft errors: 4)","time":"2025-09-15T14:23:41Z"}
2025-09-15T14:24:05.834326192Z {"level":"error","msg":"Failed to do run once: soft error\nfailed to list resource records sets for zone /hostedzone/<zone-id> using aws profile \"default\": operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: 872a46fa-5ed5-4415-a138-6d0c7004db4c, api error Throttling: Rate exceeded (consecutive soft errors: 5)","time":"2025-09-15T14:24:05Z"}
2025-09-15T15:23:56.428485116Z {"level":"error","msg":"Failed to do run once: soft error\nfailed to list resource records sets for zone /hostedzone/<zone-id> using aws profile \"default\": operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: 4ecf6b5f-0127-4225-81cf-7503289512b4, api error Throttling: Rate exceeded (consecutive soft errors: 6)","time":"2025-09-15T15:23:56Z"}
2025-09-15T16:13:46.725690288Z {"level":"error","msg":"Failed to do run once: soft error\nfailed to list resource records sets for zone /hostedzone/<zone-id> using aws profile \"default\": operation error Route 53: ListResourceRecordSets, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: c931f446-4e97-47dc-a810-cb7d28d73d50, api error Throttling: Rate exceeded (consecutive soft errors: 7)","time":"2025-09-15T16:13:46Z"}
```

Many clusters share the same hostedzone for us, but normally, just one cluster hitting this problem and stuck there forever until I manually kick the pod. After restart, there seems to be no throttling errors anymore.

I suspect there is some issues on error handling, or retry logic on AWS route53 client due to that behavior, but not sure exactly.@k8s-triage-robot: The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Mark this issue as fresh with `/remove-lifecycle rotten`
- Close this issue with `/close`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/lifecycle rotten

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@k8s-triage-robot: The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:
- After 90d of inactivity, `lifecycle/stale` is applied
- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed

You can:
- Reopen this issue with `/reopen`
- Mark this issue as fresh with `/remove-lifecycle rotten`
- Offer to help out with [Issue Triage][1]

Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).

/close not-planned

[1]: https://www.kubernetes.dev/docs/guide/issue-triage/@k8s-ci-robot: @k8s-triage-robot: Closing this issue, marking it as "Not Planned".

<details>

In response to [this](https://github.com/kubernetes-sigs/external-dns/issues/5499#issuecomment-3533911857):

>The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
>
>This bot triages issues according to the following rules:
>- After 90d of inactivity, `lifecycle/stale` is applied
>- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied
>- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed
>
>You can:
>- Reopen this issue with `/reopen`
>- Mark this issue as fresh with `/remove-lifecycle rotten`
>- Offer to help out with [Issue Triage][1]
>
>Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community).
>
>/close not-planned
>
>[1]: https://www.kubernetes.dev/docs/guide/issue-triage/


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

Solution patch:
diff --git a/controller/controller.go b/controller/controller.go
index d98fb810..5ee193fa 100644
--- a/controller/controller.go
+++ b/controller/controller.go
@@ -153,6 +153,14 @@ var (
 			Help:      "Number of DNS AAAA-records that exists both in source and registry.",
 		},
 	)
+	consecutiveSoftErrors = metrics.NewGaugeWithOpts(
+		prometheus.GaugeOpts{
+			Namespace: "external_dns",
+			Subsystem: "controller",
+			Name:      "consecutive_soft_errors",
+			Help:      "Number of consecutive soft errors in reconciliation loop.",
+		},
+	)
 )
 
 func init() {
@@ -171,6 +179,7 @@ func init() {
 	metrics.RegisterMetric.MustRegister(sourceAAAARecords)
 	metrics.RegisterMetric.MustRegister(verifiedARecords)
 	metrics.RegisterMetric.MustRegister(verifiedAAAARecords)
+	metrics.RegisterMetric.MustRegister(consecutiveSoftErrors)
 }
 
 // Controller is responsible for orchestrating the different components.
@@ -356,14 +365,23 @@ func (c *Controller) ShouldRunOnce(now time.Time) bool {
 func (c *Controller) Run(ctx context.Context) {
 	ticker := time.NewTicker(time.Second)
 	defer ticker.Stop()
+	var softErrorCount int
 	for {
 		if c.ShouldRunOnce(time.Now()) {
 			if err := c.RunOnce(ctx); err != nil {
 				if errors.Is(err, provider.SoftError) {
-					log.Errorf("Failed to do run once: %v", err)
+					softErrorCount++
+					consecutiveSoftErrors.Gauge.Set(float64(softErrorCount))
+					log.Errorf("Failed to do run once: %v (consecutive soft errors: %d)", err, softErrorCount)
 				} else {
 					log.Fatalf("Failed to do run once: %v", err)
 				}
+			} else {
+				if softErrorCount > 0 {
+					log.Infof("Reconciliation succeeded after %d consecutive soft errors", softErrorCount)
+				}
+				softErrorCount = 0
+				consecutiveSoftErrors.Gauge.Set(0)
 			}
 		}
 		select {
diff --git a/docs/monitoring/metrics.md b/docs/monitoring/metrics.md
index 8d73b202..aa38ae9e 100644
--- a/docs/monitoring/metrics.md
+++ b/docs/monitoring/metrics.md
@@ -20,6 +20,7 @@ curl https://localhost:7979/metrics
 
 | Name                             | Metric Type | Subsystem   |  Help                                                 |
 |:---------------------------------|:------------|:------------|:------------------------------------------------------|
+| consecutive_soft_errors | Gauge | controller | Number of consecutive soft errors in reconciliation loop. |
 | last_reconcile_timestamp_seconds | Gauge | controller | Timestamp of last attempted sync with the DNS provider |
 | last_sync_timestamp_seconds | Gauge | controller | Timestamp of last successful sync with the DNS provider |
 | no_op_runs_total | Counter | controller | Number of reconcile loops ending up with no changes on the DNS provider side. |
@@ -87,5 +88,3 @@ curl https://localhost:7979/metrics
 | process_start_time_seconds |
 | process_virtual_memory_bytes |
 | process_virtual_memory_max_bytes |
-| process_network_receive_bytes_total |
-| process_network_transmit_bytes_total |



Test patch:
diff --git a/controller/controller_test.go b/controller/controller_test.go
index a5fca2c5..e1f8a2b5 100644
--- a/controller/controller_test.go
+++ b/controller/controller_test.go
@@ -22,6 +22,7 @@ import (
 	"math"
 	"reflect"
 	"sort"
+	"sync"
 	"testing"
 	"time"
 
@@ -739,3 +740,68 @@ func TestAAAARecords(t *testing.T) {
 	assert.Equal(t, math.Float64bits(2), valueFromMetric(sourceAAAARecords.Gauge))
 	assert.Equal(t, math.Float64bits(1), valueFromMetric(registryAAAARecords.Gauge))
 }
+
+type toggleRegistry struct {
+	registry.NoopRegistry
+	failCount   int
+	failCountMu sync.Mutex // protects failCount
+}
+
+func (r *toggleRegistry) Records(ctx context.Context) ([]*endpoint.Endpoint, error) {
+	r.failCountMu.Lock()
+	defer r.failCountMu.Unlock()
+	if r.failCount < 3 {
+		r.failCount++
+		return nil, provider.SoftError
+	}
+	return []*endpoint.Endpoint{}, nil
+}
+
+func (r *toggleRegistry) ApplyChanges(ctx context.Context, changes *plan.Changes) error {
+	return nil
+}
+
+func TestToggleRegistry(t *testing.T) {
+	source := getTestSource()
+	cfg := getTestConfig()
+	r := &toggleRegistry{}
+
+	ctrl := &Controller{
+		Source:             source,
+		Registry:           r,
+		Policy:             &plan.SyncPolicy{},
+		ManagedRecordTypes: cfg.ManagedDNSRecordTypes,
+		Interval:           10 * time.Millisecond,
+	}
+	ctrl.nextRunAt = time.Now().Add(-time.Millisecond)
+	ctx, cancel := context.WithCancel(context.Background())
+	stopped := make(chan struct{})
+	go func() {
+		ctrl.Run(ctx)
+		close(stopped)
+	}()
+
+	// Wait up to 2 seconds for failCount to reach at least 3
+	deadline := time.Now().Add(2 * time.Second)
+	for {
+		r.failCountMu.Lock()
+		count := r.failCount
+		r.failCountMu.Unlock()
+		if count >= 3 {
+			break
+		}
+		if time.Now().After(deadline) {
+			break
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	cancel()
+	<-stopped
+
+	r.failCountMu.Lock()
+	finalCount := r.failCount
+	r.failCountMu.Unlock()
+	if finalCount < 3 {
+		t.Fatalf("failCount should be at least 3 after waiting up to 2s, got %d", finalCount)
+	}
+}
diff --git a/internal/gen/docs/metrics/main_test.go b/internal/gen/docs/metrics/main_test.go
index dd9884e9..6f23c434 100644
--- a/internal/gen/docs/metrics/main_test.go
+++ b/internal/gen/docs/metrics/main_test.go
@@ -37,7 +37,7 @@ func TestComputeMetrics(t *testing.T) {
 		t.Errorf("Expected not empty metrics registry, got %d", len(reg.Metrics))
 	}
 
-	assert.Len(t, reg.Metrics, 21)
+	assert.Len(t, reg.Metrics, 22)
 }
 
 func TestGenerateMarkdownTableRenderer(t *testing.T) {