Repository: kubernetes-sigs/external-dns

Issue:
AWS CloudMap (aws-sd) provider fails without aws-zone-type argument
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

-->

**What happened**:

In External DNS versions after 0.15.0, the aws-sd provider fails with the following error message unless the `--aws-zone-type` filter is set:

```txt
Failed to do run once: operation error ServiceDiscovery: ListNamespaces, 2 validation error(s) found.
- missing required field, ListNamespacesInput.Filters[0].Name.
- missing required field, ListNamespacesInput.Filters[0].Values.
```

**What you expected to happen**:

I expect the provider to work without filtering the namespace type, so the same configuration can work with both public and private namespaces.

**How to reproduce it (as minimally and precisely as possible)**:

```terraform
locals {
  external_dns_values = {
    provider = {
      name = "aws-sd"
    }
    registry = "aws-sd"
    env = [
      {
        name = "AWS_DEFAULT_REGION"
        value = data.aws_region.current.id
      }
    ]
    domainFilters = ["my.namespace"]
    rbac = {
      create = true
    }
    serviceAccount = {
      annotations = {
        "eks.amazonaws.com/role-arn" = aws_iam_role.external_dns.arn
      }
    }
    policy = "sync"
    txtOwnerId = "my-cluster-name"
  }
}

resource "helm_release" "external_dns" {
  name = "external-dns"
  namespace = "kube-system"
  repository = "https://kubernetes-sigs.github.io/external-dns/"
  chart = "external-dns"
  version = "1.18.0"

  values = [
    yamlencode(local.external_dns_values),
  ]
}
```

**Anything else we need to know?**:

I fixed the problem and got External DNS to start by adding the following to the chart values:

```terraform
    extraArgs = {
      # starting with version 0.16.?, the aws-sd provider crashes without this argument
      "aws-zone-type" = "private"
    }
```

see also previous report #5167, which was closed because the reporter did not respond

**Environment**:

- External-DNS version: 0.18.0
- DNS provider: aws-sd
- Kubernetes: 1.31
- Terraform: 1.11.2
- Terraform AWS provider: 5.100.0

Discussion:
@ivankatliarchuk: All arguments for external-dns also required, and example manifest files to reproduce it.@ivankatliarchuk: /assign@ivankatliarchuk: Ok from you example missing Terraform or aws cli commands to create exact AWS resources, not Terraform with how to deploy external-dns. As well as exact arguments (not helm) for external-dns.@wjrogers: > All arguments for external-dns also required, and example manifest files to reproduce it.

The Terraform configuration I used to install External DNS is in the issue body. I'll paste the deployment manifest below, with only our actual domain name redacted. Is there anything else you need?

<details>

<summary>deployment.yaml</summary>

```yaml
---
# Source: external-dns/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
  labels:
    helm.sh/chart: external-dns-1.18.0
    app.kubernetes.io/name: external-dns
    app.kubernetes.io/instance: external-dns
    app.kubernetes.io/version: "0.18.0"
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: external-dns
      app.kubernetes.io/instance: external-dns
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: external-dns
        app.kubernetes.io/instance: external-dns
    spec:
      automountServiceAccountToken: true
      serviceAccountName: external-dns
      securityContext:
        fsGroup: 65534
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: external-dns
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsGroup: 65532
            runAsNonRoot: true
            runAsUser: 65532
          image: registry.k8s.io/external-dns/external-dns:v0.18.0
          imagePullPolicy: IfNotPresent
          env:
            - name: AWS_DEFAULT_REGION
              value: us-east-1
          args:
            - --log-level=info
            - --log-format=text
            - --interval=1m
            - --source=service
            - --source=ingress
            - --policy=sync
            - --registry=aws-sd
            - --txt-owner-id=eks-staging
            - --domain-filter=stage.mydomain.com
            - --provider=aws-sd
          ports:
            - name: http
              protocol: TCP
              containerPort: 7979
          livenessProbe:
            failureThreshold: 2
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          readinessProbe:
            failureThreshold: 6
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
```

</details>@ivankatliarchuk: I need to understand how to create actual AWS resources. I just realized I lack AWS access, and LocalStack doesn't support this. However, I might still be able to diagnose the issue if I know how the resources are typically created.@wjrogers: In the provider constructor, `namespaceTypeFilter` is set to `newSdNamespaceFilter(namespaceType)`. When `namespaceType` is neither "public" nor "private", that function returns an empty struct:

https://github.com/kubernetes-sigs/external-dns/blob/20ab886cb665ce9d85c95f678e3e6c62e580af36/provider/awssd/aws_sd.go#L118-L120

Later, in `ListNamespaces`, the provider passes the empty struct to the AWS API:

https://github.com/kubernetes-sigs/external-dns/blob/20ab886cb665ce9d85c95f678e3e6c62e580af36/provider/awssd/aws_sd.go#L356-L358

The `Name` and `Values` fields are required, so the API returns the error "operation error ServiceDiscovery: ListNamespaces, 2 validation error(s) found".

Solution patch:
diff --git a/docs/flags.md b/docs/flags.md
index 68f6307c..59f92888 100644
--- a/docs/flags.md
+++ b/docs/flags.md
@@ -66,7 +66,7 @@
 | `--google-zone-visibility=` | When using the Google provider, filter for zones with this visibility (optional, options: public, private) |
 | `--alibaba-cloud-config-file="/etc/kubernetes/alibaba-cloud.json"` | When using the Alibaba Cloud provider, specify the Alibaba Cloud configuration file (required when --provider=alibabacloud) |
 | `--alibaba-cloud-zone-type=` | When using the Alibaba Cloud provider, filter for zones of this type (optional, options: public, private) |
-| `--aws-zone-type=` | When using the AWS provider, filter for zones of this type (optional, options: public, private) |
+| `--aws-zone-type=` | When using the AWS provider, filter for zones of this type (optional, default: any, options: public, private) |
 | `--aws-zone-tags=` | When using the AWS provider, filter for zones with these tags |
 | `--aws-profile=` | When using the AWS provider, name of the profile to use |
 | `--aws-assume-role=""` | When using the AWS API, assume this IAM role. Useful for hosted zones in another AWS account. Specify the full ARN, e.g. `arn:aws:iam::123455567:role/external-dns` (optional) |
diff --git a/pkg/apis/externaldns/types.go b/pkg/apis/externaldns/types.go
index ea809d3a..c4df4e35 100644
--- a/pkg/apis/externaldns/types.go
+++ b/pkg/apis/externaldns/types.go
@@ -504,7 +504,7 @@ func App(cfg *Config) *kingpin.Application {
 	app.Flag("google-zone-visibility", "When using the Google provider, filter for zones with this visibility (optional, options: public, private)").Default(defaultConfig.GoogleZoneVisibility).EnumVar(&cfg.GoogleZoneVisibility, "", "public", "private")
 	app.Flag("alibaba-cloud-config-file", "When using the Alibaba Cloud provider, specify the Alibaba Cloud configuration file (required when --provider=alibabacloud)").Default(defaultConfig.AlibabaCloudConfigFile).StringVar(&cfg.AlibabaCloudConfigFile)
 	app.Flag("alibaba-cloud-zone-type", "When using the Alibaba Cloud provider, filter for zones of this type (optional, options: public, private)").Default(defaultConfig.AlibabaCloudZoneType).EnumVar(&cfg.AlibabaCloudZoneType, "", "public", "private")
-	app.Flag("aws-zone-type", "When using the AWS provider, filter for zones of this type (optional, options: public, private)").Default(defaultConfig.AWSZoneType).EnumVar(&cfg.AWSZoneType, "", "public", "private")
+	app.Flag("aws-zone-type", "When using the AWS provider, filter for zones of this type (optional, default: any, options: public, private)").Default(defaultConfig.AWSZoneType).EnumVar(&cfg.AWSZoneType, "", "public", "private")
 	app.Flag("aws-zone-tags", "When using the AWS provider, filter for zones with these tags").Default("").StringsVar(&cfg.AWSZoneTagFilter)
 	app.Flag("aws-profile", "When using the AWS provider, name of the profile to use").Default("").StringsVar(&cfg.AWSProfiles)
 	app.Flag("aws-assume-role", "When using the AWS API, assume this IAM role. Useful for hosted zones in another AWS account. Specify the full ARN, e.g. `arn:aws:iam::123455567:role/external-dns` (optional)").Default(defaultConfig.AWSAssumeRole).StringVar(&cfg.AWSAssumeRole)
diff --git a/provider/awssd/aws_sd.go b/provider/awssd/aws_sd.go
index 510fee81..9ee0822a 100644
--- a/provider/awssd/aws_sd.go
+++ b/provider/awssd/aws_sd.go
@@ -78,7 +78,7 @@ type AWSSDProvider struct {
 	// only consider namespaces ending in this suffix
 	namespaceFilter *endpoint.DomainFilter
 	// filter namespace by type (private or public)
-	namespaceTypeFilter sdtypes.NamespaceFilter
+	namespaceTypeFilter []sdtypes.NamespaceFilter
 	// enables service without instances cleanup
 	cleanEmptyService bool
 	// filter services for removal
@@ -102,21 +102,28 @@ func NewAWSSDProvider(domainFilter *endpoint.DomainFilter, namespaceType string,
 	return p, nil
 }
 
-// newSdNamespaceFilter initialized AWS SD Namespace Filter based on given string config
-func newSdNamespaceFilter(namespaceTypeConfig string) sdtypes.NamespaceFilter {
+// newSdNamespaceFilter returns NamespaceFilter based on the given namespace type configuration.
+// If the config is "public", it filters for public namespaces; if "private", for private namespaces.
+// For any other value (including empty), it returns filters for both public and private namespaces.
+// ref: https://docs.aws.amazon.com/cloud-map/latest/api/API_ListNamespaces.html
+func newSdNamespaceFilter(namespaceTypeConfig string) []sdtypes.NamespaceFilter {
 	switch namespaceTypeConfig {
 	case sdNamespaceTypePublic:
-		return sdtypes.NamespaceFilter{
-			Name:   sdtypes.NamespaceFilterNameType,
-			Values: []string{string(sdtypes.NamespaceTypeDnsPublic)},
+		return []sdtypes.NamespaceFilter{
+			{
+				Name:   sdtypes.NamespaceFilterNameType,
+				Values: []string{string(sdtypes.NamespaceTypeDnsPublic)},
+			},
 		}
 	case sdNamespaceTypePrivate:
-		return sdtypes.NamespaceFilter{
-			Name:   sdtypes.NamespaceFilterNameType,
-			Values: []string{string(sdtypes.NamespaceTypeDnsPrivate)},
+		return []sdtypes.NamespaceFilter{
+			{
+				Name:   sdtypes.NamespaceFilterNameType,
+				Values: []string{string(sdtypes.NamespaceTypeDnsPrivate)},
+			},
 		}
 	default:
-		return sdtypes.NamespaceFilter{}
+		return []sdtypes.NamespaceFilter{}
 	}
 }
 
@@ -354,7 +361,7 @@ func (p *AWSSDProvider) ListNamespaces(ctx context.Context) ([]*sdtypes.Namespac
 	namespaces := make([]*sdtypes.NamespaceSummary, 0)
 
 	paginator := sd.NewListNamespacesPaginator(p.client, &sd.ListNamespacesInput{
-		Filters: []sdtypes.NamespaceFilter{p.namespaceTypeFilter},
+		Filters: p.namespaceTypeFilter,
 	})
 	for paginator.HasMorePages() {
 		resp, err := paginator.NextPage(ctx)



Test patch:
diff --git a/provider/awssd/aws_sd_test.go b/provider/awssd/aws_sd_test.go
index 3bf59220..3cb4e758 100644
--- a/provider/awssd/aws_sd_test.go
+++ b/provider/awssd/aws_sd_test.go
@@ -254,14 +254,14 @@ func TestAWSSDProvider_ApplyChanges_Update(t *testing.T) {
 	ctx := context.Background()
 
 	// apply creates
-	provider.ApplyChanges(ctx, &plan.Changes{
+	_ = provider.ApplyChanges(ctx, &plan.Changes{
 		Create: oldEndpoints,
 	})
 
 	ctx = context.Background()
 
 	// apply update
-	provider.ApplyChanges(ctx, &plan.Changes{
+	_ = provider.ApplyChanges(ctx, &plan.Changes{
 		UpdateOld: oldEndpoints,
 		UpdateNew: newEndpoints,
 	})
@@ -306,6 +306,7 @@ func TestAWSSDProvider_ListNamespaces(t *testing.T) {
 	}{
 		{"public filter", endpoint.NewDomainFilter([]string{}), "public", []*sdtypes.NamespaceSummary{namespaceToNamespaceSummary(namespaces["public"])}},
 		{"private filter", endpoint.NewDomainFilter([]string{}), "private", []*sdtypes.NamespaceSummary{namespaceToNamespaceSummary(namespaces["private"])}},
+		{"optional filter", endpoint.NewDomainFilter([]string{}), "", []*sdtypes.NamespaceSummary{namespaceToNamespaceSummary(namespaces["public"]), namespaceToNamespaceSummary(namespaces["private"])}},
 		{"domain filter", endpoint.NewDomainFilter([]string{"public.com"}), "", []*sdtypes.NamespaceSummary{namespaceToNamespaceSummary(namespaces["public"])}},
 		{"non-existing domain", endpoint.NewDomainFilter([]string{"xxx.com"}), "", []*sdtypes.NamespaceSummary{}},
 	} {
@@ -913,7 +914,7 @@ func TestAWSSDProvider_RegisterInstance(t *testing.T) {
 	}
 
 	// AWS NLB instance (ALIAS)
-	provider.RegisterInstance(context.Background(), services["private"]["alias-srv"], &endpoint.Endpoint{
+	_ = provider.RegisterInstance(context.Background(), services["private"]["alias-srv"], &endpoint.Endpoint{
 		RecordType: endpoint.RecordTypeCNAME,
 		DNSName:    "service1.private.com.",
 		RecordTTL:  300,
@@ -927,7 +928,7 @@ func TestAWSSDProvider_RegisterInstance(t *testing.T) {
 	}
 
 	// CNAME instance
-	provider.RegisterInstance(context.Background(), services["private"]["cname-srv"], &endpoint.Endpoint{
+	_ = provider.RegisterInstance(context.Background(), services["private"]["cname-srv"], &endpoint.Endpoint{
 		RecordType: endpoint.RecordTypeCNAME,
 		DNSName:    "service2.private.com.",
 		RecordTTL:  300,
@@ -1001,7 +1002,7 @@ func TestAWSSDProvider_DeregisterInstance(t *testing.T) {
 
 	provider := newTestAWSSDProvider(api, endpoint.NewDomainFilter([]string{}), "", "")
 
-	provider.DeregisterInstance(context.Background(), services["private"]["srv1"], endpoint.NewEndpoint("srv1.private.com.", endpoint.RecordTypeA, "1.2.3.4"))
+	_ = provider.DeregisterInstance(context.Background(), services["private"]["srv1"], endpoint.NewEndpoint("srv1.private.com.", endpoint.RecordTypeA, "1.2.3.4"))
 
 	assert.Empty(t, instances["srv1"])
 }