Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: does not retaining coredns configmap using v1.19.4 makes no difference What you expected to happen: should follow documentation and retain coredns configmap How to reproduce it (as minimally and precisely as possible): upgrading from v1.18.6 to v1.19.3 Anything else we need to know?: works like a charm Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: OS (e.g: ): CentOS 7 Kernel (e.g. ): 4.20.13- Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
seems to be a regression ?
/assign
Would you check you dns configmap for ? As , federation is removed from coredns. If you have federation enabled in your coredns configmap, the check may fail in
We don't have any parameter in our coredns configmap, or any kube-dns parameters in kubeadm config
Looking closely at the dry run messages above, this may not be a regression. The apparent reason it works in 1.18.12 is that support for CoreDNS 1.7.0 was not until 1.19. i.e. 1.18 sees an unsupported version of CoreDNS and therefore skips the corefile migration. Whereas in 1.19, CoreDNS 1.7.0 is supported, and the migration takes place. What does your complete Corefile look like before the migration?
If it's possible, can you also attach the entire dry run logs?
Sure.
Thanks. I think the issue may be in the detection of the "default" configuration. That is, in the upgrade process, if it detects that you are using the "default" coredns corefile, it simply installs the new default over top of it, rather than migrating the existing one. I think since the default config stanza is in there in its original form, the whole Corefile is incorrectly being detected as a default. If i'm right, the fix could either go in the package (should at least eventually be fixed there), and/or kubeadm could just always migrate the existing config instead of deploying the default over top of the existing one.
I just tested against the package manually - and it correctly identifies the config as "non-default", so that's not it... And I also tested a migration on the Corefile, and the result included the extra stanzas as expected. So, the issue doesn't appear to lie in the migration package.
After testing with the code, I find the issue in , I fixed it in . A workaround for you is to create an empty kube-dns configmap. ———————————————————— edited new comments here I find still fix a problem , when kube-dns configmap is using federations.
Thanks, I'll try it
empty kube-dns configmap doesn't helped.
sorry for my mistake. According to my test, the create or update will trigger is no migrating. Even your corefile will meet some dep that need to be migrated. max_concurrent is new default. However, there is a typo in . Below is a workaround(not good one), add upstream to your corefile in kubernetes. ! It works for me.
if your current coredns is 1.6.7, upstream is deprecated but not removed. In new 1.7.0 coredns, upstream is removed.
thanks for the discussion and PRs this one is mergable for 1.21 as non critical: what do you think about this one and should we backport it as critical (also, to which k8s versions)?
/area kubeadm
It's not obvious to me how this relates to . The kubeadm dry run log appears to be replacing the existing config with the default. It should only be doing this if the existing config is semantically the same as a default Corefile, i.e. if is true.
just explains that the migration is not triggered at all. if is not a fix for this issue, please remove this line from the PR description.
I think issue here is in/around , and how the result is interpreted... The code correctly assumes that if a Corefile is a default Corefile, then no migration is required. However later it incorrectly assumes the reciprocal: If migration is not required, then the corefile must be a default. Hence, it ends up replacing any Corefile that does not require migration with a default… even if it wasn't a default to begin with.
yes. The need be fixed as well, I will learn to fix it later if no one is assigned. I changed releaseNote to None.

Discussion:
// Assume that migration is always possible, rely on migrateCoreDNSCorefile() to fail if not. canMigrateCorefile := true <del> if corefileMigrationRequired { </del> <ins> if corefile == "" || migration.Default("", corefile) { // If the Corefile is empty or default, the latest default Corefile will be applied if err := apiclient.CreateOrUpdateConfigMap(client, coreDNSConfigMap); err != nil { return err } } else if corefileMigrationRequired { // If migration is required, try and migrate the Corefile </ins> if err := migrateCoreDNSCorefile(client, coreDNSConfigMap, corefile, currentInstalledCoreDNSVersion); err != nil { // Errors in Corefile Migration is verified during preflight checks. This part will be executed when a user has chosen // to ignore preflight check errors.

} } } else { <del> if err := apiclient.CreateOrUpdateConfigMap(client, coreDNSConfigMap); err != nil { </del> <ins> // If the Corefile is modified and doesn't require any migration, it'll be retained for the benefit of the user if err := apiclient.CreateOrRetainConfigMap(client, coreDNSConfigMap, kubeadmconstants.CoreDNSConfigMap); err != nil { </ins> return err } }

// isCoreDNSConfigMapMigrationRequired checks if a migration of the CoreDNS ConfigMap is required. func isCoreDNSConfigMapMigrationRequired(corefile, currentInstalledCoreDNSVersion string) (bool, error) { var isMigrationRequired bool <del> if corefile == "" || migration.Default("", corefile) { </del> <ins> // Current installed version is expected to be empty for init if currentInstalledCoreDNSVersion == "" { </ins> return isMigrationRequired, nil } deprecated, err := migration.Deprecated(currentInstalledCoreDNSVersion, kubeadmconstants.CoreDNSVersion, corefile)

} } <del> func TestCreateCoreDNSConfigMap(t *testing.T) { </del> <ins> func TestCreateCoreDNSAddon(t *testing.T) { </ins> tests := []struct { name                 string initialCorefileData  string

coreDNSVersion       string }{ { <ins> name:                "Empty Corefile", initialCorefileData: "", expectedCorefileData: `.:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } `, coreDNSVersion: "1.6.7", }, { name: "Default Corefile", initialCorefileData: `.:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } `, expectedCorefileData: `.:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } `, coreDNSVersion: "1.6.7", }, { name: "Modified Corefile with only newdefaults needed", initialCorefileData: `.:53 { errors log health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } `, expectedCorefileData: `.:53 { errors log health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } `, coreDNSVersion: "1.6.2", }, { name: "Default Corefile with rearranged plugins", initialCorefileData: `.:53 { errors cache 30 prometheus :9153 forward . /etc/resolv.conf loop reload loadbalance kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa ttl 30 } health } `, expectedCorefileData: `.:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance } `, coreDNSVersion: "1.3.1", }, { </ins> name: "Remove Deprecated options", initialCorefileData: `.:53 { errors <ins> logs </ins> health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure

}`, expectedCorefileData: `.:53 { errors <ins> logs </ins> health { lameduck 5s }

`, coreDNSVersion: "1.3.1", }, <ins> { name: "Modified Corefile with no migration required", initialCorefileData: `consul { errors forward . 10.10.96.16:8600 10.10.96.17:8600 10.10.96.18:8600 { max_concurrent 1000 } loadbalance cache 5 reload } domain.int { errors forward . 10.10.0.140 10.10.0.240 10.10.51.40 { max_concurrent 1000 } loadbalance cache 3600 reload } .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp max_concurrent 1000 } cache 30 loop reload loadbalance } `, expectedCorefileData: `consul { errors forward . 10.10.96.16:8600 10.10.96.17:8600 10.10.96.18:8600 { max_concurrent 1000 } loadbalance cache 5 reload } domain.int { errors forward . 10.10.0.140 10.10.0.240 10.10.51.40 { max_concurrent 1000 } loadbalance cache 3600 reload } .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp max_concurrent 1000 } cache 30 loop reload loadbalance } `, coreDNSVersion: "1.6.7", }, </ins> } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { client := createClientAndCoreDNSManifest(t, tc.initialCorefileData, tc.coreDNSVersion) <del> // Get the Corefile and installed CoreDNS version. cm, corefile, currentInstalledCoreDNSVersion, err := GetCoreDNSInfo(client) </del> <ins> configMapBytes, err := kubeadmutil.ParseTemplate(CoreDNSConfigMap, struct{ DNSDomain, UpstreamNameserver, StubDomain string }{ DNSDomain:          "cluster.local", UpstreamNameserver: "/etc/resolv.conf", StubDomain:         "", }) </ins> if err != nil { <del> t.Fatalf("unable to fetch CoreDNS current installed version and ConfigMap.") </del> <ins> t.Errorf("unexpected ParseTemplate failure: %+v", err) </ins> } <del> err = migrateCoreDNSCorefile(client, cm, corefile, currentInstalledCoreDNSVersion) </del> <ins> err = createCoreDNSAddon(nil, nil, configMapBytes, client) </ins> if err != nil { <del> t.Fatalf("error creating the CoreDNS ConfigMap: %v", err) </del> <ins> t.Fatalf("error creating the CoreDNS Addon: %v", err) </ins> } migratedConfigMap, _ := client.CoreV1().ConfigMaps(metav1.NamespaceSystem).Get(context.TODO(), kubeadmconstants.CoreDNSConfigMap, metav1.GetOptions{}) if !strings.EqualFold(migratedConfigMap.Data["Corefile"], tc.expectedCorefileData) {