Change:
OUT_DIR?=_output HYPERKUBE_BIN?=$(shell pwd)/../../../$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.10.1 </del> <ins> BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.10.2 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build

Discussion:
What happened: Kube-proxy ipvs mode CI failed http://k8s- But actually all test are passed in my local-up environment with the newest code. According to ci log, those pr might be related: seems like the most probable, Anyway, looking for help.
Friendly ping , Please take a look, looking forward for your advise.
thanks for the report! /sig network /sig gcp /kind failing-test
/area ipvs
Update: Seems like staging- image do have some problem, command will return err. Not likely to be ipset issue.
Suggestions?
The version is exactly the same in both versions of the image (v10 and v10.1). Where did you find that error message?
List of package differences (given by ) between v10 and v10.1:
Thanks, the error was found in ipvs ci environment kube-proxy log. And I also get this error in my local environment with
Maybe we got some kind of incompatible issue with ipset specific version.
It seems like the removal of is what's causing this failure, which appears to be due to depending on instead of . I'm not sure what caused that to switch.
ok, I think I have an idea of what happened here: was originally built against . At some point was built against what was then , i.e. pulling in the packages from , which included (and ). Later (probably due to security issues), we rebuilt , which by this point was now based on . In this release, they apparently switched to installing by default, which did not include a dependency on . We apparently did not rebuild , though I'm not sure why not. As a result, it was still based on the older debian release. By the time we built , had been released, and so was similarly based on and was thus missing . tl;dr: need to add a dependency on in apparently, and should build/push a (or maybe ). should probably be more careful with these "simple rebases", since they can unexpectedly change what packages are installed. Some automation to do would help here. (cc do you think this is a kubernetes 1.12-blocking issue?
/assign
Thanks a lot. Though I'm not really understand, but glad we found the root cause.

And I think this gonna block 1.12 release, ipvs feature already GA in 1.11, we need make sure this feature can work normally.
for thoughts.
Do you think this is a kubernetes 1.12-blocking issue?
If this causing core IPVS functionality to be broken, which it looks like it is, then I think this should block the 1.12 release Sent from my iPhone
/milestone v1.12 Will treat as release blocking now until we hear otherwise from SIG-Network and SIG-GCP
fix PR for the s is ; we'll need one more PR after that's merged to update the Makefiles to use the new images.
First PR has merged.
Thanks. Working on promoting those images to , though that probably won't happen till Tuesday PT.
... or maybe sooner than that, because is an overachiever.
Second PR:
You're the one up working on promoting images at 9pm :-) I'm just trying to set my dev environment up again and saw this going on.
I can't keep up with the 3 of you!!! - -- An old man
The CI working well again hours ago, thanks all of you, really appreciate it. : )
Awesome. We might want to get better monitoring on this job if IPVS is GA, since it took several weeks from the original PR merged before this issue was opened.
Anyone willing to monitor can PR email alerts per config here: Please consider doing so! It's ~hard~ impossible to track all of testgrid, we really need SIGs to step up on this!
Definitely make sense, I will raise a pr to do this work.
Thank you! Lion-Wei <: