Repository: kubernetes/kubernetes (language: go)

Issue:
Headless Service not publishing not ready addresses for statefulset
**What happened**:

Deploying a StatefulSet with a headless service and properly configured `spec.serviceName` doesn't publish the underlying pod's on the internal Kube-DNS for at least 30 seconds after pod creation, even if the headless service has explicit `publishNotReadyAddresses: true` and even if a pod has a `readinessProbe` of 1 second which shouldn't matter if the publishNotReadyAddresses is set to true according to [documentation](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields) .

This causes that if the stateful application, which relies on a pod being able to communicate to it's peer for bootstrapping, doesn't start for at least 30 seconds due to this delay.

**What you expected to happen**:

I'm expecting the fully qualified names of the pods under the statefulset to become avaiable in the DNS so the pods in the statefulset can find it's own peers via their dns name (i.e. `pod-0.statefulset-svc` can connect to `pod-1.statefulset-svc`)

**How to reproduce it (as minimally and precisely as possible)**:

I've designed a simple application that since starting it tries to ping it's own peers under the same application, you can clearly see that for the first 30 seconds of the pod existance, it cannot reach it's peers, not even itself, this is because the pod's name is not being announced in the internal KubeDNS.

Using the following statefulset you can observe the behavior:

```
apiVersion: v1
kind: Service
metadata:
  labels:
    app: minio
    controller: message-dist
  name: message-dist
  namespace: default
spec:
  clusterIP: None
  ports:
    - name: minio-port
      port: 8090
      protocol: TCP
      targetPort: 8090
  publishNotReadyAddresses: true
  selector:
    app: minio
    controller: message-dist
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: message-dist
  namespace: default
spec:
  podManagementPolicy: Parallel
  replicas: 4
  selector:
    matchLabels:
      app: minio
      controller: message-dist
  serviceName: message-dist
  template:
    metadata:
      labels:
        app: minio
        controller: message-dist
    spec:
      containers:
        - name: message
          command:
            - /message
            - http://message-dist-{0...3}.message-dist:8090/
          env:
            - name: MESSAGE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          image: dvaldivia/message:dist
#          imagePullPolicy: IfNotPresent
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /
              port: 8090
              scheme: HTTP
            initialDelaySeconds: 1
            periodSeconds: 20
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /
              port: 8090
              scheme: HTTP
            initialDelaySeconds: 1
            periodSeconds: 20
            successThreshold: 1
            timeoutSeconds: 1
          ports:
            - containerPort: 8090
              name: minio-port
              protocol: TCP
          resources: {}
          volumeMounts:
            - mountPath: /data1
              name: data1
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - metadata:
        name: data1
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
```

Or simply apply this file:

```
kubectl apply -f https://raw.githubusercontent.com/dvaldivia/message/master/message-dist.yaml
```

**Anything else we need to know?**:

This problem could be avoided if the Kube-DNS made the service subdomains for the pods available sooner than 30 seconds, I cannot find anywhere in the documentation where it specifies this 30 seconds delay.

**Environment**:
- Kubernetes version (use `kubectl version`):  `client 1.18.4 ` `server 1.18.2`
- Cloud provider or hardware configuration: kind, gke




Discussion / hints:
/sig network
/triage unresolved

Comment `/remove-triage unresolved` when the issue is assessed and confirmed.

🤖 I am a bot run by vllry. 👩‍🔬
Can I have a look at it?
/assign @AkashSarda  
@AkashSarda
If this issue has been triaged, please comment `/remove-triage unresolved`.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the `help-wanted` label.

🤖 I am a bot run by vllry. 👩‍🔬
/unassign @AkashSarda 
/assign
cc  @AkashSarda @mattfenwick  .... definetly something special about this 30 second interval, on a Kind cluster with one node, seeing the same thing.........  need to update docs ? or figure out the exact semantics.

So, weve confirmed that your test definetly is taking almost exactly 30 seconds, in order to publish the logs 

Start time: 

```
2020/07/05 13:19:04 Get http://message-dist-0.message-dist:8090/: dial tcp: lookup message-dist-0.message-dist on 10.96.0.10:53: no such host
2020/07/05 13:19:04 Get http://message-dist-1.message-dist:8090/: dial tcp: lookup message-dist-1.message-dist on 10.96.0.10:53: no such host
```

4 DNS messsages OK about 30 seconds later (31 seconds later)
```
2020/07/05 13:19:35 Get http://message-dist-1.message-dist:8090/: dial tcp: lookup message-dist-1.message-dist on 10.96.0.10:53: no such host
2020/07/05 13:19:35 Total OKs:  3
2020/07/05 13:19:36 Total OKs:  4
2020/07/05 13:19:37 Total OKs:  4
```

We may have to ask about this at sig-network this week.  I'll dig around in the docs as well. 

I can confirm I see this behavior on both Kind and GKE clusters. 

If the DNS resolution could happen sooner, the stateful services that rely on peer communication could start sooner. Regular ClusterIP services register immediately, so it must be possible I guess.
Yeah , i see it in kind for sure,   Curious, why do you think the Total OKs fluctuates up and down  until the 4th container is up (after that its stable) ? 
```
2020/07/05 19:34:42 Total OKs:  0
...
2020/07/05 19:34:52 Total OKs:  0
2020/07/05 19:34:53 Total OKs:  2
2020/07/05 19:34:55 Total OKs:  2
2020/07/05 19:34:56 Total OKs:  1
2020/07/05 19:34:57 Total OKs:  2
2020/07/05 19:34:59 Total OKs:  1
2020/07/05 19:35:00 Total OKs:  1
2020/07/05 19:35:02 Total OKs:  3
2020/07/05 19:35:03 Total OKs:  0
2020/07/05 19:35:04 Total OKs:  2
2020/07/05 19:35:06 Total OKs:  3
2020/07/05 19:35:07 Total OKs:  2
2020/07/05 19:35:08 Total OKs:  3
2020/07/05 19:35:09 Total OKs:  1
2020/07/05 19:35:11 Total OKs:  2
2020/07/05 19:35:12 Total OKs:  3
2020/07/05 19:35:13 Total OKs:  3
2020/07/05 19:35:14 Total OKs:  4
```

I have a deadline of 1 second on the request, perhaps there’s a timeout when contacting one of the peers, I can make it configurable vía environment variable if you’d like 
It’s odd though that all peers go to zero after initially being positive values ? Just thinking out loud. Does it cut out early on failure ?
Ahhhh  i see the issue, After some poking it became clear that
 
- The pods as you show are *clearly* up and running very quickly
- The time interval is religiously 30 seconds.
- To fix , just chance `cache 30` to something like `cache 5` in your coreDNS pods !

so, it turns out that by default, i think most of the time your CoreDNS is running the cache plugin with, you guessed it, *30* seconds for responses.  Thus, since your pod is querying before the other pods are up the first response is that 'i have no clue what pod this is ', and you'll never get positive respose until 30 seconds after the failed request.

The solution was simplee, 
```
- apiVersion: v1
  data:
    Corefile: |
      .:53 {
          errors
          health {
             lameduck 5s
          }
          ready
          kubernetes cluster.local in-addr.arpa ip6.arpa {
             pods insecure
             fallthrough in-addr.arpa ip6.arpa
             ttl 30
          }
          prometheus :9153
          forward . /etc/resolv.conf
          cache 5 # Change this field from cache 30 to cache 5 , and youll see the services come up rapidly 
          loop
          reload
          loadbalance
```
The way you edit this is by running `kubectl edit cm coredns -n kube-system` and then simple re running your test , you'll see an immediate improvement.   
This makes sense, I guess with traditional `ClusterIP` services the same happens, I created a [similar experiment](https://raw.githubusercontent.com/dvaldivia/message/master/message-multi-svc.yaml) using `StatefulSet+ClusterIP Services` which results in the same behavior if the services are created after the pods start asking for the services.

Do you think then delegating a custom domain `message.local` via [DNS customization](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/) is a good strategy if I wished to have this discovery happen sooner?
Hi ! Maybe lets discuss this in sig-network this week, can you come on thursday at 5 (EST) 2 (PST) ? 
and we can close this issue and ill add it to the agenda: ?  Ive pasted the details below... Also I think we should discuss in general the right way to document this..... ?

```
he canonical meeting invite.
Zoom link (client): https://zoom.us/j/361123509
Zoom link (web): https://zoom.us/wc/361123509/
Agenda: https://docs.google.com/document/d/1_w77-zG_Xj0zYvEMfQZTQ-wPP4kXkpGD8smVtW_qqWM/edit#
```
sure, I'll be there!
Looks like the NXDOMAIN ttl was changed to 30s in https://github.com/kubernetes/kubernetes/pull/76238 to be consistent with kube-dns config. However, kube-dns/dnsmasq does not cache NXDOMAIN at all. So, we should probably change this default back to 5s for NXDOMAIN responses? @chrisohaver @Deiken
@prameshj that would improve things a lot, I'd all in for no caching at all for NXDOMAIN
> @prameshj that would improve things a lot, I'd all in for no caching at all for NXDOMAIN

Not caching NXDOMAIN at all will make every single searchpath-expanded and AAAA queries to be processed by kubernetes plugin rather than returned from cache. These 2 are some of the significant sources of DNS traffic. Probably less of an issue for CoreDNS than kube-dns, but I would expect changing that value to result in some performance regression.
i made a note in the docs around this , that might be a reasonable action item for this

@dvaldivia any other ideas we should pursue ? Feel free to schedule a time to discuss again, you can ping me (jayunit100) on slack
> Looks like the NXDOMAIN ttl was changed to 30s in #76238 to be consistent with kube-dns config. However, kube-dns/dnsmasq does not cache NXDOMAIN at all. So, we should probably change this default back to 5s for NXDOMAIN responses? @chrisohaver @Deiken

The TTL control in the kubernetes plugin a single setting for all responses. So, 5s for all records, or 30s for all records.
We could cut the max cache time for negative responses to 5s for _all_ domains, but that would also affect upstream NXDOMAIN responses.

As far as disabling NXDOMAIN cache, negative cache (which includes NXDOMAIN and some other negative responses) can only be disabled globally (i.e. disabling caching of upstream NXDOMAINs).  The performance impact on fruitless search path expansions and AAAA queries would not be terribly significant because the k8s api client keeps all watched objects in a local cache (e.g. services, endpoints). IOW, coredns doesn't need to go back to the API for those requests, it just looks up the objects in a local indexed store.  For that matter - cache misses due to expired cache for kubernetes domain records is also not a big performance hit, for the same reason. So the "performance" reason for moving fro 5s to 30s in the first place is probably not as compelling as it would seem.


Thanks @chrisohaver !
Are you suggesting that moving back to 5s is maybe a better way forward?
> Thanks @chrisohaver !
> Are you suggesting that moving back to 5s is maybe a better way forward?

I am leaning that way. On the other hand, in node local deployments the response cache resides on the local node coredns instances, and the object cache resides in the cluster dns deployment. Which means a node local instance forwards misses to the cluster DNS deployment instead of just handling it internally. i.e. k8s record misses cost more in a node local deployment.

It seems that the primary reason for moving from 5s to 30s was ...

> ...  just use 30s because kube-dns has used it for a long time and seems no big complaint yet.

~... I don't _think_ there was any empirical case for the change.~

**Edit:** Strike that - It was indirectly stated that this caused a [100X](https://github.com/kubernetes/kubernetes/pull/76238#issuecomment-481307933) performance improvement in a case.
> On the other hand, in node local deployments the response cache resides on the local node coredns instances, and the object cache resides in the cluster dns deployment. Which means a node local instance forwards misses to the cluster DNS deployment instead of just handling it internally. i.e. k8s record misses cost more in a node local deployment.

That's correct. In NodeLocalDNS, we keep the NXDOMAIN ttl to 5s. It is a small value and would cause a bunch of searchpath expanded and AAAA records to get forwarded to k8s cluster DNS, but it is somewhat of a balance between caching NXDOMAINs and discarding temporary NXDOMAINs for services whose records are being created.
> That's correct. In NodeLocalDNS, we keep the NXDOMAIN ttl to 5s. It is a small value and would cause a bunch of searchpath expanded and AAAA records to get forwarded to k8s cluster DNS, but it is somewhat of a balance between caching NXDOMAINs and discarding temporary NXDOMAINs for services whose records are being created.

IIRC, the node local Corefile has a separate server block for `cluster.local.` (or whatever the cluster's DNS suffix happens to be).  This allows for a separate cache configuration for cluster records vs records from upstream sources.

Whereas in the Cluster DNS Corefile, there is a single server block for `.` (re: [due to the way CoreDNS handles plugin chains](https://github.com/coredns/deployment/issues/153)). With a single cache, all responses (cluster domain _and_ upstream) are subject to the same cache setting.
I guess at this point, changing the defaults isn't really something that is guaranteed to be a win, but rather, just a decision that will trade one outcome for another.  I suspect by decreasing cache times:
- https://github.com/kubernetes/kubernetes/issues/91236 <- issues like this might get *worse* (large scalability tests overloading coredns) (note originally typo here, I wrote "better" :)) 
- issues for new apps/databases/consensus seeking pods are going to get *better*. 

1) Do we feel like there's enough variability in opinions and history here that its not a black and white solution until we collect more data, and write up an actual proposal / demo for sig-network to decide on the ideal defaults ?    I'm open to pursuing this to its logical endpoint if @dvaldivia wants to collaborate (it would be a decent time investment on both sides, though).

2) If *not* , should we simply close this and update the docs ?

Either way im open to , but just thinking at this point - since there's no *obvious* bug here, we're going to have to dig pretty deep to revert this number again, and that's only possible if we are really sure that we want to put the time/resources into making a data-driven case for it.
I was thinking the other day that the previous `kube-dns` had a nice model to injecting new DNS configurations into the cluster via a a config map, why not grab inspiration from this previous approach (and ingress) and come up with a new Resource to be able to customize the DNS? just throwing this out there for feedback

@jayunit100 I'm down to collaborate in whatever means necessary, either by figuring out healthy defaults, documenting or working on code changes if needed.
> I was thinking the other day that the previous kube-dns had a nice model to injecting new DNS configurations into the cluster via a a config map, why not grab inspiration from this previous approach (and ingress) and come up with a new Resource to be able to customize the DNS? just throwing this out there for feedback

CoreDNS is also configurable via ConfigMap.  If the default configurations do not suit a particular deployment, they can be altered.
True but adding a single stub domain cannot be done so easily, since the Corefile has some Config that needs to be parsed or manually edited.

For applications that need to add something to CoreDNS they could simply include a resource in their yaml that adds a studDomain for example and some internal controller would rebuild the Corefile configuration and apply it. I like this since this resource can be RBACed. 

Right now customers get nervous when I tell them that for my operator to work faster I need them to tweak a ConfigMap on the kube-system namespace :P 

The kube-dns configmap was also in the kube-system namespace. CoreDNS is more configurable than the old kube-dns was; naturally its configuration is more complex.  

There is a CoreDNS Operator project underway that would (eventually) expose these kinds of operations as custom resources, and process them as ConfigMap change. Although for the first version, it is focusing on upgrades and rollouts.


That project sounds exactly like what I have in mind, is there a KEP for it? Or do you know the name of the project?
It's under the Cluster Addons SIG: https://github.com/kubernetes-sigs/cluster-addons

/cc @rajansandeep
iirc core-dns often is still quite often in kube-system namespace... (checked a few clusters to confirm just now)
```
~ » kubectl get pods -n kube-system | grep core                                                                                                                                                                     ubuntu@ubuntu
coredns-dbbffcb66-gzqrj                                         1/1     Running   0          7d19h
coredns-dbbffcb66-qgspv                                         1/1     Running   0          7d19h
```
@dvaldivia so , digging around I guess the defaults are really hardcoded in kubeadm.  Maybe we should  contribute to the cluster-lifecycle KEP for the CoreDNS operator, long term, to reduce this value.

```
~/SOURCE/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns » cat dns.go|grep -B 10 30                                                                                                       ubuntu@ubuntu
				return "", errors.Wrap(err, "invalid format to parse for proxy")
			}
			if len(proxyIP) == 0 {
				continue
			}

			pStanza := map[string]interface{}{}
			pStanza["keys"] = []string{domain + ":53"}
			pStanza["body"] = [][]string{
				{"errors"},
				{"cache", "30"},
```
I guess that would mean reassigning this to sig-cluster-lifecycle to triage it , and decide wether or not such a change to the Kubeadm defaults was valid.  
@dvaldivia ... still interested in this or should we close  ? 
@jayunit100 yes! I'm still interested, sure let's pass it on the sig-cluster-lifecycle to continue the conversation 
/sig cluster-lifecycle

No longer a sig net issue because this involves changing dns defaults and thus intersects into the kubeadm setup side of the house per my comment above 
 should we unassign sig-network now from this ?   its purely in the hands of sig-net, kubeadm configs, and david :) (david i can help if you want reach out on slack)
@jayunit100
If this issue has been triaged, please comment `/remove-triage unresolved`.

If you aren't able to handle this issue, consider unassigning yourself and/or adding the `help-wanted` label.

🤖 I am a bot run by vllry. 👩‍🔬
/remove-triage unresolved

/close
@jayunit100: Closing this issue.

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/92559#issuecomment-670143158):

>/close


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
Im closing this because I think at this point, its been a couple of weeks and we now know that this needs to be taken up with the folks that set the defaults in kubeadm and the kubeadm operator -- its happening at the sig-cluster-lifecycle level and so on, and probably will involve follow up work with the coredns operator stuff, so its not (I think) any longer a bug in any sense from the networking side.

if we need to reopen it to keep the context that's fine, but figured would just close this so we can keep moving along.
For any time travelers from the future, this issue is caused by the CoreDNS component of kube-dns, when they switched from dnsmasq to CoreDNS, they introduce a caching for misses, which lasts 30 seconds, thus the problem described above, solution is to bring your own DNS, or change the CoreDNS settings so the cache of the misses is not as long
This issue hit us in RabbitMQ multiple times:

* https://github.com/rabbitmq/cluster-operator/issues/958
* https://github.com/rabbitmq/cluster-operator/issues/269
* https://github.com/rabbitmq/cluster-operator/issues/752#issuecomment-872118881
* https://github.com/rabbitmq/cluster-operator/issues/298#issuecomment-760153567
* https://github.com/rabbitmq/rabbitmq-server/issues/5322

The [rabbitmq/cluster-operator](https://github.com/rabbitmq/cluster-operator) creates a StatefulSet and headless Service with `publishNotReadyAddresses: true` at the same time and RabbitMQ pods will need to discover each other promptly upon Pod startup (that is they need to resolve each others' hostnames and their own hostname).

I think we should re-open this issue because, after 2 years have passed, there is still no good solution to this CoreDNS bug.
Some proposed solutions / workarounds that were mentioned are:

1. "Query the Kubernetes API directly (for example, using a watch) rather than relying on DNS lookups." as mentioned in the [docs](https://github.com/kubernetes/website/pull/22446/files) is highly inconvenient for some StatefulSet Data Services. They should be able to rely on DNS and shouldn't be aware that they run on K8s. (For example in RabbitMQ, although there is a [dedicated Kubernetes peer discovery plugin](https://www.rabbitmq.com/cluster-formation.html#peer-discovery-k8s), in other Erlang code places during RabbitMQ node startup, we rely on [Erlang Port Mapper Daemon](https://www.erlang.org/doc/man/epmd.html) to resolve correctly via DNS.)
1. "Decrease the time of caching in your Kubernetes DNS provider (typically this means editing the config map for CoreDNS, which currently caches for 30 seconds)." is recommended in the [docs](https://github.com/kubernetes/website/pull/22446/files). (a) Any StatefulSet Data Service relying on discovering Pods promptly will need to document this to their users (e.g. as done [here](https://github.com/rabbitmq/rabbitmq-website/commit/4993b92c6f33446f50da9e021955c3e9aba3f4dc) for RabbitMQ) (b) Changing this global cache value such that 1 StatefulSet Data Service comes up correctly doesn't sound right. (c) I'm not sure whether all users who deploy Data Service Operators are allowed to change DNS settings in the `kube-system` namespace. 
1. Use the [CoreDNS Operator](https://github.com/kubernetes-sigs/cluster-addons/tree/master/coredns#coredns-operator). This operator is in alpha stage, and not recommended for production.

Therefore, the most appropriate current workaround is to wait in the Stateful Data Service until DNS resolution works.

In RabbitMQ this was implemented for example in:
* https://github.com/rabbitmq/rabbitmq-server/pull/4233
* https://github.com/rabbitmq/rabbitmq-server/pull/2722

If DNS resolution does still not work after a few retries and up to 30 seconds, the Data Service start up will fail. This is also a poor workaround because (a) it doesn't allow to differentiate between a real DNS issue and just an outdated cache, and (b) what happens if someone configured their CoreDNS cache TTL to a value considerably higher than the default of 30 second?

Looking at the CoreDNS [code](https://github.com/coredns/coredns/blob/master/plugin/kubernetes/controller.go), it seems that CoreDNS already watches Pods, Services, and Endpoints.

As written in https://github.com/coredns/coredns/issues/2434#issuecomment-451538364:
> If you are creating the headless service at the same time as creating the pods that it selects, it will take some time for the pods to spin up and be ready. But once the pods are ready, coredns is updated almost instantaneously.

My question is:
If you are creating the headless service with `publishNotReadyAddresses: true` at the same time as creating the pods that it selects, why can't CoreDNS (cache) be updated instantaneously when a new Pod gets created? Is there any technical limitation to this solution?
if the CoreDNS cache plugin has a mechanism to invalidate cache for a particular domain which belongs to a service with `publishNotReadyAddresses: true` then we are good to go, we would only need k8s to tell CoreDNS to invalidate any cached value as soon as a pod comes online and gets an internal IP address and there wouldn't be a need to turn off the 30 seconds caching for everyone
> if the CoreDNS cache plugin has a mechanism to invalidate cache for a particular domain which belongs to a service with publishNotReadyAddresses: true then we are good to go

It does not have such a mechanism.  But IMO there is a simpler solution: if we could disable caching of records in the kubernetes zone _without_ disabling caching of upstream responses.  I have recently opened a PR to make this possible coredns/coredns#5540.
@chrisohaver does this mean that for `publishNotReadyAddresses: true` services, Kubernetes would need to add it's domain to a configuration dynamically, correct? 
> @chrisohaver does this mean that for `publishNotReadyAddresses: true` services, Kubernetes would need to add it's domain to a configuration dynamically, correct?

No, I’m suggesting disabling dns cache for all the cluster domain (eg cluster.local), not individual records under it. 

> No, I’m suggesting disabling dns cache for all the cluster domain (eg cluster.local), not individual records under it.

I've brought this up with them before but they didn't though disabling the cache for was a good idea, but perhaps only for misses is the right solution.

this is the `Corefile` k8s installs by default to the internal coredns
```
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
```

so we would disable misses like this

```
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
           cache {
                    disable denial cluster.local in-addr.arpa ip6.arpa
           }

        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
```

is this a good assumption @chrisohaver ?

That's the syntax per https://github.com/coredns/coredns/pull/5540, which is merged, but not yet released.

I was proposing disabling cache for `cluster.local`. Disabling reverse zones like that would prevent caching reverse lookup responses from upstream.

> I've brought this up with them before but they didn't though disabling the cache for was a good idea

Disabling the cache plugin for _upstream_ records is a bad idea.  Disabling the cache plugin for Kubernetes service records is not bad, since the kubernetes plugin will still cache them.
What do you think @jayunit100 is this something we can make part of upstream once https://github.com/coredns/coredns/pull/5540 is released?

Fix patch:
diff --git a/cmd/kubeadm/app/phases/addons/dns/manifests.go b/cmd/kubeadm/app/phases/addons/dns/manifests.go
index 905a2e050e691..1358733b5be52 100644
--- a/cmd/kubeadm/app/phases/addons/dns/manifests.go
+++ b/cmd/kubeadm/app/phases/addons/dns/manifests.go
@@ -178,6 +178,11 @@ data:
            max_concurrent 1000
         }
         cache 30
+        {{- if .DNSDomain }} {
+           disable success {{ .DNSDomain }}
+           disable denial {{ .DNSDomain }}
+        }
+        {{- end }}
         loop
         reload
         loadbalance


Test patch:
diff --git a/cmd/kubeadm/app/phases/addons/dns/dns_test.go b/cmd/kubeadm/app/phases/addons/dns/dns_test.go
index fc7b35137c9cb..5c4925a705b9a 100644
--- a/cmd/kubeadm/app/phases/addons/dns/dns_test.go
+++ b/cmd/kubeadm/app/phases/addons/dns/dns_test.go
@@ -184,7 +184,10 @@ func TestCreateCoreDNSAddon(t *testing.T) {
     forward . /etc/resolv.conf {
        max_concurrent 1000
     }
-    cache 30
+    cache 30 {
+       disable success cluster.local
+       disable denial cluster.local
+    }
     loop
     reload
     loadbalance
@@ -228,7 +231,10 @@ func TestCreateCoreDNSAddon(t *testing.T) {
     forward . /etc/resolv.conf {
        max_concurrent 1000
     }
-    cache 30
+    cache 30 {
+       disable success cluster.local
+       disable denial cluster.local
+    }
     loop
     reload
     loadbalance
@@ -314,7 +320,10 @@ func TestCreateCoreDNSAddon(t *testing.T) {
     forward . /etc/resolv.conf {
        max_concurrent 1000
     }
-    cache 30
+    cache 30 {
+       disable success cluster.local
+       disable denial cluster.local
+    }
     loop
     reload
     loadbalance