Change:
https://k8s- Multiple broken tests: Failed: [] SchedulerPredicates [Serial] validates that a pod with an invalid podAffinity is rejected because of the LabelSelectorRequirement is invalid {Kubernetes e2e suite} Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to add nodes {Kubernetes e2e suite} Failed: [] Etcd failure [Disruptive] should recover from SIGKILL {Kubernetes e2e suite} Failed: [] Services should work after restarting kube-proxy [Disruptive] {Kubernetes e2e suite} Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite} Failed: [] Namespaces [Serial] should delete fast enough (90 percent of 100 namespaces in 150 seconds) {Kubernetes e2e suite} Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability {Kubernetes e2e suite} Failed: [] Services should work after restarting apiserver [Disruptive] {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite} Failed: [] Namespaces [Serial] should ensure that all services are removed when a namespace is deleted. {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] ReplicaSet Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that taints-tolerations is respected if matching {Kubernetes e2e suite} Failed: [] Etcd failure [Disruptive] should recover from network partition with master {Kubernetes e2e suite} Failed: [] DaemonRestart [Disruptive] Controller Manager should not create/delete replicas across restart {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that required NodeAffinity setting is respected if matching {Kubernetes e2e suite} Failed: [] Nodes [Disruptive] [] Network when a node becomes unreachable All pods on the unreachable node should be marked as NotReady upon the node turn NotReady AND all pods should be mark back to Ready when the node get back to Ready before pod eviction timeout {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] Deployment Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability {Kubernetes e2e suite} Failed: [] Daemon set [Serial] should run and stop complex daemon {Kubernetes e2e suite} Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 0 pods per node {Kubernetes e2e suite} Issues about this test specifically: Failed: [] Nodes [Disruptive] [] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] Deployment Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {Kubernetes e2e suite} Issues about this test specifically: Failed: [] SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow] {Kubernetes e2e suite} Issues about this test specifically: Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 100 pods per node {Kubernetes e2e suite} Issues about this test specifically: Previous issues for this suite:
https://k8s- Multiple broken tests: Failed: [] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover {Kubernetes e2e suite} Issues about this test specifically: Failed: [] SchedulerPredicates [Serial] validates that a pod with an invalid podAffinity is rejected because of the LabelSelectorRequirement is invalid {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] ReplicaSet Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] ReplicationController Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite} Failed: [] DaemonRestart [Disruptive] Controller Manager should not create/delete replicas across restart {Kubernetes e2e suite} Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite} Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to delete nodes {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that NodeSelector is respected if matching [Conformance] {Kubernetes e2e suite} Failed: [] Etcd failure [Disruptive] should recover from network partition with master {Kubernetes e2e suite} Failed: [] Services should work after restarting kube-proxy [Disruptive] {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that NodeAffinity is respected if not matching {Kubernetes e2e suite} Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 0 pods per node {Kubernetes e2e suite} Issues about this test specifically: Failed: [] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching [Conformance] {Kubernetes e2e suite} Failed: [] Nodes [Disruptive] [] Network when a node becomes unreachable All pods on the unreachable node should be marked as NotReady upon the node turn NotReady AND all pods should be mark back to Ready when the node get back to Ready before pod eviction timeout {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance] {Kubernetes e2e suite} Issues about this test specifically: Failed: [] Nodes [Disruptive] [] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] Deployment Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected {Kubernetes e2e suite} Failed: [] DaemonRestart [Disruptive] Scheduler should continue assigning pods to nodes across restart {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow] {Kubernetes e2e suite} Issues about this test specifically: Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] Deployment Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {Kubernetes e2e suite} Issues about this test specifically: Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 100 pods per node {Kubernetes e2e suite} Issues about this test specifically: Failed: [] SchedulerPredicates [Serial] validates that taints-tolerations is respected if not matching {Kubernetes e2e suite} Failed: [] Namespaces [Serial] should ensure that all pods are removed when a namespace is deleted. {Kubernetes e2e suite} Failed: [] Nodes [Disruptive] [] Resize [Slow] should be able to add nodes {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] ReplicaSet Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that taints-tolerations is respected if matching {Kubernetes e2e suite} Failed: [] Etcd failure [Disruptive] should recover from SIGKILL {Kubernetes e2e suite} Failed: [] DaemonRestart [Disruptive] Kubelet should not restart containers across restart {Kubernetes e2e suite} Failed: [] Namespaces [Serial] should delete fast enough (90 percent of 100 namespaces in 150 seconds) {Kubernetes e2e suite} Failed: [] Daemon set [Serial] should run and stop complex daemon {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that required NodeAffinity setting is respected if matching {Kubernetes e2e suite}
https://k8s- Multiple broken tests: Failed: [] Nodes [Disruptive] [] Network when a node becomes unreachable All pods on the unreachable node should be marked as NotReady upon the node turn NotReady AND all pods should be mark back to Ready when the node get back to Ready before pod eviction timeout {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that required NodeAffinity setting is respected if matching {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] ReplicaSet Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {Kubernetes e2e suite} Failed: [] DaemonRestart [Disruptive] Scheduler should continue assigning pods to nodes across restart {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] Deployment Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability {Kubernetes e2e suite} Failed: [] Nodes [Disruptive] [] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] ReplicationController Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that taints-tolerations is respected if matching {Kubernetes e2e suite} Failed: [] Services should work after restarting apiserver [Disruptive] {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite} Failed: [] DaemonRestart [Disruptive] Kubelet should not restart containers across restart {Kubernetes e2e suite}
The suite started failing consistently at build 1456 The etcd on the master node failed the liveness probes repeatedly starting around 08:01, and never recovered. The only PR on that build that seems relevant: /cc
Assigned to since he's started working on it.
Ah..I forgot serial suite became a blocking suite. Thanks for marking this p0.
Can we roll back something? this is blocking the queue.
IIUC, reverting will fix this. how about we revert that first so that you can verify your fix with a test?
Let's revert right now.
+1 for reverting asap
By the way, IIUC, this is a real bug in kubenet that's been triggered in only the serial suite, and is not really related to GCI per se. The master components use host network + ports, and when the daemonset test scheduled a pod onto the master node, all hell broke loose. We don't have any e2e tests for HostNetwork+HostPort, and usually no pods get scheduled to the master node, so this wasn't triggered in any other tests. has more context on the actual cause.
Revert it first.
is reverted and merged.
re Ok, I understood that we decided to switch master node on GCE to GCI image by default, but we also agreed to keep the same set of jenkins tests having master node on old debian-based containerVM image for test coverage during transition time, right? If that is what we agreed, how come we didn't catch the issue earlier?
I guess you referred to did the first step mentioned by that comment to switch the default image. Someone from test-infra team did the second step to add jenkins job to testing ContainerVM master.
https://k8s- Multiple broken tests: Failed: [] SchedulerPredicates [Serial] validates that taints-tolerations is respected if not matching {Kubernetes e2e suite} Failed: [] Namespaces [Serial] should ensure that all services are removed when a namespace is deleted. {Kubernetes e2e suite} Failed: [] Restart [Disruptive] should restart all nodes and ensure all nodes and pods recover {Kubernetes e2e suite} Issues about this test specifically: Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected {Kubernetes e2e suite} Failed: [] Etcd failure [Disruptive] should recover from SIGKILL {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance] {Kubernetes e2e suite} Issues about this test specifically: Failed: [] SchedulerPredicates [Serial] validates that taints-tolerations is respected if matching {Kubernetes e2e suite} Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite} Failed: [] Services should work after restarting apiserver [Disruptive] {Kubernetes e2e suite} Failed: [] Namespaces [Serial] should ensure that all pods are removed when a namespace is deleted. {Kubernetes e2e suite} Failed: [] Services should work after restarting kube-proxy [Disruptive] {Kubernetes e2e suite} Failed: [] DaemonRestart [Disruptive] Controller Manager should not create/delete replicas across restart {Kubernetes e2e suite} Failed: [] Nodes [Disruptive] [] Network when a node becomes unreachable [replication controller] recreates pods scheduled on the unreachable node AND allows scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite} Failed: [] Etcd failure [Disruptive] should recover from network partition with master {Kubernetes e2e suite} Failed: [] DaemonRestart [Disruptive] Kubelet should not restart containers across restart {Kubernetes e2e suite} Failed: [] Daemon set [Serial] should run and stop complex daemon {Kubernetes e2e suite} Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] [Serial] [Slow] Deployment Should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {Kubernetes e2e suite} Issues about this test specifically:
I debugged this issue and also tried the hope the following information is useful for you. The problem in the issue is that after a certain test case runs, etcd pods are done and api-server is down accordingly. Kubelet or etcd log does not tell what is wrong. The symptom in the serial tests is that after a certain test, all following tests fail due to apiserver is down. And "kubectl get" commands never work. Since the serial tests suite is slow, I only verified with running the suite once. In that one run, there were 4 failed test case and other 33 passed. The failed ones are not back-to-back like captured by this issue. Then I ran these 4 tests one by one on the same cluster that just ran the entire serial tests (without rebooting any instance), all of them passed. So I think they were flaky. The important thing is that after running the entire serial suite, "kubectl get" still works and api-server is not down. In summary: from this one run of serial test suites, it looks like fixes the conflict of GCI master + network plugin. Since network plugin currently is not used by GKE, I don't rush to put it back. Instead, I will run the serial tests for more times to further verify and then we can consider put the network plugin support back to GCI config.
There are too many tests in the serial suite. I think it'd be a bit ineffective to run the entire suite. IIUC, daemonset test seems to be the one triggering the issue (i.e., it's the only test that schedules pods on the master node), so perhaps you can try running that one test to see how it goes. The better solution is to add a test for this case specifically, which is included in .
I made a script to repeatedly run the serial tests on GCI cluster with both and . The serial tests were run 10 times, among which 9 times were all passed and only one had a flaky case. From the verification I think really fixed the issue. Are you OK with putting the reverted back?
Let us revert it back and let it run in all CIs.

Discussion:
} // Need the complete api.Pod object pod, ok := plugin.host.GetPodByName(p.Namespace, p.Name) <del> if ok { </del> <ins> // kubenet should not handle hostports for hostnetwork pods if ok && !pod.Spec.SecurityContext.HostNetwork { </ins> for _, container := range pod.Spec.Containers { for _, port := range container.Ports { if port.HostPort != 0 {