Change:
What happened: All the services are asking to be recreated because the service CIDR changed after upgrading from 1.16.3 ro 1.17.0 What you expected to happen: Smooth migration from to , which would not break service CIDR. How to reproduce it (as minimally and precisely as possible): Do an upgrade from version 1.16.3 to 1.17.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
do you mean that the service cidr changed without any intervention after the upgrade? EDIT 96.0.0/12 is the default service cidr when using kubeadm --service-cidr string Default: "10.96.0.0/12" EDIT if not using kubeadm the default is May it be possible you were using another service cidr before the upgrade and you didn't set it during the upgrade, so it fall back to the default?
Can you provide your kube-apiserver invocation?
xref which warnings if no explicit CIDR flag was passed but should not have changed behavior otherwise
Where do you see that? The default is specified here as 10.0.0.0/24 and did not change between 1.16.0 and 1.17.0:
I was assuming it was using kubeadm based on the log message 96.0.0/12 is the default service CIDR for kubeadm
ah, ok. If they are using kubeadm, that default also did not change between 1.16.0 and 1.17.0:
a few questions for you: how did you install your 1.16.x cluster? did you customize the service cidr in that deployment? how did you upgrade your 1.16.x cluster to 1.17.x? did you apply the same service cidr customization in that deployment?
16.x was a straight and normal install no change on service CIRD, and we have never touched this CIDR configuration. And kubeadm is supposed to be used during the upgrade from 1.16.x to 1.17.x. We just followed the describe process: The logs system started showing the shared message.
can you reproduce?
can you paste ? That will show you kubeadm config
created a single node cluster with v1.16.4 and a custom service cidr: applied upgrade to v1.17.0: i do not see a change in the custom service cidr.
if you never touched the service CIDR on the initial install it should have been which is the kubeadm default. your config map shows . how did you adapt the IP to work with the kubeadm default service cidr () if you initially deployed without modifying said cidr? to me this suggests that a change was required at some point.
The only change is the update from 1.16.3 to 1.17.0, done yesterday. Please find below a snapshot from our logs: <img width="1635" alt="Screen Shot 2019-12-21 at 22 22 29" src="https://user-"
my point is that if you did not apply a custom service cidr on "kubeadm init", then this cidr was the default out of kubeadm and after upgrade the service cidr was preserved as your config map shows. what was the elasticsearch/logging service IP before upgrade? EDIT: also do you have the YAML that was used to create this service?
101.249.43 belongs to the service rubnet range at minimum the error message is wrong /kind bug
are those errors happening constantly or only happened during the upgrade? are you able to create new services within that range? let's say, with a service IP of 10.101.249.45 Also, can you paste your current service-cluster-cidr configured in the controller-manager and apiserver? i.e.
this is a bug in core: this call in the repair controller returns for an IP that is actually in the CIDR range: also indicates a missing unit test case. latest commits: this might have regressed with the dual-stack changes? cc /remove-triage support needs-information /sig network /remove-sig cluster-lifecycle
/priority important-longterm
I was able to reproduce the same behavior again, with a fresh install k8s release 1.16.4. Just After installing metrics server, we could notice a shift in the service CIDR, but nothing in the logs.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/cc
so the problem here is that an IPv4 CIDR such as translates to addresses. but this line here clamps the maximum to 2^16 () so for an address like the logic in the ip allocator fails in the function: as the offset ends up bigger than the maximum. it feels like the maximums for IPv4 and IPv6 should be different and larger. otherwise for CIDRs in the notation : can give incorrect results for IPv4 can give incorrect results for IPv6 cc who touched related logic.
I think that was touched in … was this bug preexisting or did that change behavior?
before PR () i do not see a clamp for IPv4: only one for IPv6 at 16bits.
We should: revert add unit tests for the missed cases get that into 1.17.1 reintroduce the reliance on the function carefully
What would be the safest path to recreate all the services whilst waiting a final solution ? Especially for production K8S.
to mitigate the error in 1.17.0 you need to modify the serivce CIDR to be , instead of . modify the ConfigMap CIDR to have modify the kubeadm static pod manifest for the api server () to have edit problematic Service objects and give them an IP in the range if needed.
Thank you for those instructions, I'm affected by this, does this have anything to do with a possibility of ClusterIP's not properly responding or does it only print out alerts?
hello, can you test if the ClusterIP is not responding? i don't think we got a confirmation if this is only an alert. my guess is that it is just that, but there might be side effects.
i see you logged i will comment there.
How do you "recreate" a service? Literally delete and create again? Or is there a command which would cause it to re-create itself without having to be deleted?

Discussion:
deps = [ "//pkg/apis/core:go_default_library", "//pkg/registry/core/service/allocator:go_default_library", <del> "//vendor/k8s.io/utils/integer:go_default_library", "//vendor/k8s.io/utils/net:go_default_library", </del> ], )

import ( "errors" "fmt" <del> "math/big" "net" </del> api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/registry/core/service/allocator" <del> "k8s.io/utils/integer" utilnet "k8s.io/utils/net" </del> <ins> "math/big" "net" </ins> ) // Interface manages the allocation of IP addresses out of a range. Interface

// NewAllocatorCIDRRange creates a Range over a net.IPNet, calling allocatorFactory to construct the backing store. func NewAllocatorCIDRRange(cidr *net.IPNet, allocatorFactory allocator.AllocatorFactory) (*Range, error) { <del> max := integer.Int64Min(utilnet.RangeSize(cidr), 1<<16) base := utilnet.BigForIP(cidr.IP) </del> <ins> max := RangeSize(cidr) base := bigForIP(cidr.IP) </ins> rangeSpec := cidr.String() r := Range{

if !ok { return nil, ErrFull } <del> return utilnet.AddIPOffset(r.base, offset), nil </del> <ins> return addIPOffset(r.base, offset), nil </ins> } // Release releases the IP back to the pool. Releasing an

// ForEach calls the provided function for each allocated IP. func (r *Range) ForEach(fn func(net.IP)) { r.alloc.ForEach(func(offset int) { <del> ip, _ := utilnet.GetIndexedIP(r.net, offset+1) // +1 because Range doesn't store IP 0 </del> <ins> ip, _ := GetIndexedIP(r.net, offset+1) // +1 because Range doesn't store IP 0 </ins> fn(ip) }) }

return true, offset } <ins> // bigForIP creates a big.Int based on the provided net.IP func bigForIP(ip net.IP) *big.Int { b := ip.To4() if b == nil { b = ip.To16() } return big.NewInt(0).SetBytes(b) } // addIPOffset adds the provided integer offset to a base big.Int representing a // net.IP func addIPOffset(base *big.Int, offset int) net.IP { return net.IP(big.NewInt(0).Add(base, big.NewInt(int64(offset))).Bytes()) } </ins> // calculateIPOffset calculates the integer offset of ip from base such that // base + offset = ip. It requires ip >= base. func calculateIPOffset(base *big.Int, ip net.IP) int { <del> return int(big.NewInt(0).Sub(utilnet.BigForIP(ip), base).Int64()) </del> <ins> return int(big.NewInt(0).Sub(bigForIP(ip), base).Int64()) } // RangeSize returns the size of a range in valid addresses. func RangeSize(subnet *net.IPNet) int64 { ones, bits := subnet.Mask.Size() if bits == 32 && (bits-ones) >= 31 || bits == 128 && (bits-ones) >= 127 { return 0 } // For IPv6, the max size will be limited to 65536 // This is due to the allocator keeping track of all the // allocated IP's in a bitmap. This will keep the size of // the bitmap to 64k. if bits == 128 && (bits-ones) >= 16 { return int64(1) << uint(16) } else { return int64(1) << uint(bits-ones) } } // GetIndexedIP returns a net.IP that is subnet.IP + index in the contiguous IP space. func GetIndexedIP(subnet *net.IPNet, index int) (net.IP, error) { ip := addIPOffset(bigForIP(subnet.IP), index) if !subnet.Contains(ip) { return nil, fmt.Errorf("can't generate IP with index %d from subnet. subnet too small. subnet: %q", index, subnet) } return ip, nil </ins> }

t.Fatal(err) } t.Logf("base: %v", r.base.Bytes()) <del> t.Logf("max: %v", r.max) </del> if f := r.Free(); f != tc.free { t.Errorf("Test %s unexpected free %d", tc.name, f) }

t.Logf("allocated: %v", found) } <ins> func TestRangeSize(t *testing.T) { testCases := []struct { name  string cidr  string addrs int64 }{ { name:  "supported IPv4 cidr", cidr:  "192.168.1.0/24", addrs: 256, }, { name:  "supported large IPv4 cidr", cidr:  "10.96.0.0/12", addrs: 1048576, }, { name:  "unsupported IPv4 cidr", cidr:  "192.168.1.0/1", addrs: 0, }, { name:  "supported IPv6 cidr", cidr:  "2001:db8::/48", addrs: 65536, }, { name:  "unsupported IPv6 mask", cidr:  "2001:db8::/1", addrs: 0, }, } for _, tc := range testCases { _, cidr, err := net.ParseCIDR(tc.cidr) if err != nil { t.Errorf("failed to parse cidr for test %s, unexpected error: '%s'", tc.name, err) } if size := RangeSize(cidr); size != tc.addrs { t.Errorf("test %s failed. %s should have a range size of %d, got %d", tc.name, tc.cidr, tc.addrs, size) } } } </ins> func TestForEach(t *testing.T) { _, cidr, err := net.ParseCIDR("192.168.1.0/24") if err != nil {