Change:
What happened: Running kubeadm 1.16.1 on CentOS7 with local etcd, the first kube master () is up and running, try to make the second master () join the cluster and got error: log of etcd on the first kube master shows: Verified the cert used by the 2nd kube master has the right server name (of the IP). What you expected to happen: bring the 2nd kube master into the cluster. How to reproduce it (as minimally and precisely as possible): on master-1 on master-2 Anything else we need to know?: used the same steps to create 1.15.3 without problem is a pure IPv6 env that does not have any IPv4. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Openstack OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug): calico Others: host has IPv6 only, there is no IPv4 interfaces.
/sig cluster-lifecycle
/area kubeadm /priority awaiting-more-evidence are you using the automatic copy of certificates when joining new control plane nodes? ref
is this reproducible consistently?
This is what I was using: Yes, it is, I tried 5 or 6 times with the same result.
as far as i know we already have signal for kubeadm/kind cluster for single control planes clusters using ipv6, but we don't have the same for multi-CP. https://k8s- do you have any plans or reservations for extending the kind config there to be HA? cc
i don't think we have test signal for HA clusters using kubeadm with ipv6, but i can test this locally. and this works consistently? also please mind that 1.16 includes a new version of etcd,
I guess that this error snippet can be a good hint, it seems it's adding some part of the IP address as servername. Will try to reproduce it too There no signal for HA cluster with ipv6 AFAIK
i'm starting to suspect a bug in etcd-3.3.15 that we use from kubeadm / k8s. the kubeadm certificates are signed properly, yet member add using the client API (kubeadm uses that, and not raw etcdctl calls), fails: the suspicion for the time being is that is the result of everything after the is being cut when constructing the the golang TLS client with in it's config.
just tested etcd-3.3.10 and i'm seeing the same problem with it. which possibly indicates the problem is elsewhere. trying k8s 1.15 now.
Probably not, but I wonder if this is related to
i think that problem existed back when we were still using 3.3.10: but i was able to create a good ipv6 / etcd cluster with 3.3.10 from 1.15, so i started bisecting the problem to see when in k/k this regression was introduced.
Yes 1.15.3 always worked fine, fyi, we have 5 or 6 clusters running 1.15.3 (both ubuntu and centos, if it matters), couple of dev/test clusters got redeployed roughly once a week in the past couple of months, there are more personal deployments. I tried to tag from 1.15.3 to the new version and the behavior was the same, I tested only once though.
i think i'm seeing this as a regression in the etcd client we vendor in k/k. after hours of bisecting i ended at: to summarize again: we are seeing: when multiple ipv6 etcd nodes join the cluster. this starts happening only after it seems as the for the golang TLS client config is malformed and everything after the first is trimmed. (location in ) i did try etcdctl 3.3.15 from github and TLS worked fine: do you happen to remember any changes in this area? i did find related changes in the 3.x changelogs, but nothing apparent without checking the code.
might be related
cc
adding api-machinery / apiserver in case this also affects failover
looks very wrong and very related, but was also present in 1.15 () do you know if something in the etcd v3.3.15 bump started making use of that codepath?
oh, nevermind, it's
/assign
that method should switch to using net.SplitHostPort()
/assign
/retitle 1.16: etcd client does not parse IPv6 addresses correctly when members are joining
We will backport the etcd fix to 3.4 and 3.3 once it is merged to master.
etcd backports: 3.4: 3.3:
Seeing the same issue after upgrading to v3.3.17 019-10-24 23:54:17. I | etcdserver: starting server... [version: 3.3.17, cluster version: tobedecided] 2019-10-24 23:54:17. I | etcdserver: as single-node; fast-forwarding 9 ticks (election ticks 10) 2019-10-24 23:54:17. I | etcdserver/membership: member [https://[fd74:ca9b:3a09:868c:10:9:60:218]:2380] to cluster 2019-10-24 23:54:17. I | embed: ClientTLS: cert = , key = , ca = , trusted-ca = , client-cert-auth = true, crl-file = 2019-10-24 23:54:17. I | embed: listening for metrics on http://127.0.0.1:2381 2019-10-24 23:54:18. I | raft: is starting a new election at term 1 2019-10-24 23:54:18. I | raft: became candidate at term 2 2019-10-24 23:54:18. I | raft: received MsgVoteResp from at term 2 2019-10-24 23:54:18. I | raft: became leader at term 2 2019-10-24 23:54:18. I | raft: elected leader at term 2 2019-10-24 23:54:18. I | etcdserver: setting up the initial cluster version to 3.3 2019-10-24 23:54:18. N | etcdserver/membership: set the initial cluster version to 3.3 2019-10-24 23:54:18. I | etcdserver/api: enabled capabilities for version 3.3 2019-10-24 23:54:18. I | etcdserver: published {Name:centos-60-218 ClientURLs:[https://[fd74:ca9b:3a09:868c:10:9:60:218]:2379]} to cluster 2019-10-24 23:54:18. I | embed: ready to serve client requests 2019-10-24 23:54:18. I | embed: ready to serve client requests 2019-10-24 23:54:18. I | embed: serving client requests on [fd74:ca9b:3a09:868c:10:9:60:218]:2379 2019-10-24 23:54:18. I | embed: serving client requests on 127.0.0.1:2379 2019-10-25 00:00:56. I | embed: rejected connection from "[fd74:ca9b:3a09:868c:10:9:60:219]:" (error "remote error: tls: bad certificate", ServerName "[fd74") 2019-10-25 00:00:57. I | embed: rejected connection from "[fd74:ca9b:3a09:868c:10:9:60:219]:" (error "remote error: tls: bad certificate", ServerName "[fd74") 2019-10-25 00:00:59. I | embed: rejected connection from "[fd74:ca9b:3a09:868c:10:9:60:219]:" (error "remote error: tls: bad certificate", ServerName "[fd74") 2019-10-25 00:01:01. I | embed: rejected connection from "[fd74:ca9b:3a09:868c:10:9:60:219]:" (error "remote error: tls: bad certificate", ServerName "[fd74") 2019-10-25 00:01:05. I | embed: rejected connection from "[fd74:ca9b:3a09:868c:10:9:60:219]:" (error "remote error: tls: bad certificate", ServerName "[fd74") 2019-10-25 00:01:12. I | embed: rejected connection from "[fd74:ca9b:3a09:868c:10:9:60:219]:" (error "remote error: tls: bad certificate", ServerName "[fd74") Did anyone verify that it works with this version? K8s Version: 1.16.2 3.3.17
The issue is client-side, and will be fixed in 1.16.3 by
Could you please tell release date of v1.16.3? Thanks.
Target date of 2019-11-13

Discussion:
" # The following tests are known to be flaky, and are thus run only in their own # -flaky- build variants. <del> GCE_FLAKY_TEST_REGEX=" MaxPods.* " </del> <ins> GCE_FLAKY_TEST_REGEX="" </ins> # Tests which are not able to be run in parallel. GCE_PARALLEL_SKIP_TEST_REGEX=" ${GCE_DEFAULT_SKIP_TEST_REGEX}

# TODO: figure out why GCE_FLAKY_TEST_REGEX is not a perfect subset of this list. GCE_PARALLEL_FLAKY_TEST_REGEX=" Elasticsearch <del> |Hostdir.*MOD |Networking.*intra </del> |PD |ServiceAccounts |Servicesendpointsslatency

|Services.*functioningsexternalsloadsbalancer |Services.*identicallysnamed |Services.*release.*loadsbalancer <del> |multiportsendpoints </del> " # Define environment variables based on the Jenkins project name.