Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): None

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2.2+f64c9f2d999ceb", GitCommit:"f64c9f2d999ceb157d5672e9bba6639a4c456f6e", GitTreeState:"clean", BuildDate:"2016-11-29T15:21:56Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2.2+f64c9f2d999ceb", GitCommit:"f64c9f2d999ceb157d5672e9bba6639a4c456f6e", GitTreeState:"clean", BuildDate:"2016-11-29T15:13:51Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GKE

What happened: Upgrades to version 1.5 (from any previous version) change existing addons from ReplicationControllers to Deployments without deleting old ReplicationControllers. This leads to multiple versions of the addons running at the same time. There also seems to be multiple deployments of heapster as well.

$ kubectl get rc --namespace=kube-system
NAME DESIRED CURRENT READY AGE
kube-dns-v17.1 2 2 2 1h
kubernetes-dashboard-v1.1.1 1 1 1 1h
l7-default-backend-v1.0 1 1 1 1h

$ kubectl get deployment --namespace=kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
heapster-v1.1.0 1 1 1 1 1h
heapster-v1.2.0 1 1 1 1 1h
kube-dns 1 1 1 1 1h
kubernetes-dashboard 1 1 1 1 1h
l7-default-backend 1 1 1 1 1h

$ kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-f3wt 1/1 Unknown 0 1h
fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-mgst 1/1 Running 0 1h
fluentd-cloud-logging-gke-jenkins-e2e-default-pool-91ebbcc7-zhm0 1/1 Running 0 1h
heapster-v1.1.0-2096339923-39key 2/2 Running 0 1h
heapster-v1.2.0-2168613315-1bcy3 2/2 Running 0 1h
kube-dns-4101612645-78hx6 4/4 Running 0 1h
kube-dns-v17.1-3pyz0 3/3 Running 0 1h
kube-dns-v17.1-zaskz 3/3 Running 0 1h
kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-f3wt 1/1 Unknown 0 1h
kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-mgst 1/1 Running 0 1h
kube-proxy-gke-jenkins-e2e-default-pool-91ebbcc7-zhm0 1/1 Running 0 1h
kubernetes-dashboard-3697774758-n808h 1/1 Running 0 1h
kubernetes-dashboard-v1.1.1-ljtj5 1/1 Running 0 1h
l7-default-backend-2234341178-vo5z1 1/1 Running 0 1h
l7-default-backend-v1.0-qe9yo 1/1 Running 0 1h

I found this as a counting error in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-cluster/337. At first, I just thought it was incorrectly counting, and I attempted to fix that #36924. That fix is still valid and an improvement, but the underlying problem of multiple versions of addons still running is probably bad.

We need a mechanism to delete the old ReplicationControllers/Deployments after an upgrade.

Issue 2:
There are a number of small pieces of discussion scattered across this and the rkt repository related to how to handle networking setup; I'd like to consolidate this discussion here, talk about all the issues and options I know about, and hopefully figure out what the best option is.

Discussion refs:

rkt/rkt#2249

#26081

#25404 (comment)

#24688

#21923

And quite possibly several more I missed

Introduction

Kubernetes and rkt both make use of CNI to configure container network namespaces. Each has independent means of configuring and executing plugins. Kubernetes also has a number of features that relate somewhat to the network namespace, such as availability of the pod's IP in a number of locations (status, downwards API, etc).

Until recently, rkt's handled network configuration on its own with no regard for Kubernetes' network logic. As of #25062, it was changed such that kubelet handles network setup and configuration for rkt. Unfortunately, both of these solutions have problems.

Below, the problems with each approach are listed. Finally, some paths forwards are suggested.

Issues

The PodIP should be available to the kubelet prior to applications running for a number of features to work

Downwards API (both volume and environment variables) rely on this

/etc/hosts file, created by kubelet, relies on this (alternately, rkt should create this correctly)

The network plugin configuration used by the docker container runtime should work, in general, for rkt with no changes (including filepath) (rkt/rkt#2249)

The solution should work with any rkt stage1 (#26081)

Solutions

Rkt manages the network

Kubelet manages the network

Hybrid of 1 & 2

Rkt manages networking

This option has a few problems:

The PodIP is not available to the kubelet until after the application is finalized (including mounts and environment variables) (rkt/rkt#2223 and rkt/rkt#2375)

The network plugin configuration path is not configurable (rkt/rkt#2312)

Duplication of logic/code between kubelet and rkt (more or less depending on who creates /etc/hosts and so on).

The first issue there is a difficult one to solve and is the main reason that this approach was discarded, whether temporarily or not.

This is probably the only robust long-term solution when considering the large variety of stage1's (and thus network configurations) that are available and, once rkt provides a container-level interface (see rkt/rkt#2375), there should be a natural way to solve problem number one via the separation of sandbox creation and container creation.

Kubelet manages networking

This option has its own problems:

Does not work with kvm flavor stage1 (or any stage1 that does not use Linux network namespaces or does not support --net=host).

Depends on availability of nsenter / ip tools (already depended on elsewhere I believe, mostly a nonissue)

Results in the rkt cli reporting incorrect network information

This allows fulfilling more of kubelet's assumptions about networking, but does not play nicely with rkt's existing abstractions. It does solve the most problems for the case of the default stage1 and existing rkt featureset while also not being a large amount of code. It also shares significant logic with the docker network setup, with the benefits that entails.

Hybrid

The kubelet could decide whether it or rkt should manage the networking based on whether the stage1 suports it. When rkt manages networking, the features dependent on PodIP being available early on will not work.

More code in kubelet to support both methods

rkt does not provide an easy means to determine whether one or the other is appropriate currently

Inconsistent featureset depending on stage1 (😦)

Conclusion

My current preference is to continue with the kubelet-managed network for 1.0 (though given the timeline, I'm not sure there's another choice), and rethink this afterwards. If it's possible to support a hybrid approach by 1.0 with the understanding that it's a temporary solution, that also seems reasonable.

Ideally, the kubelet's abstractions would not interfere with rkt's ability to manage networking itself since, I admit, the above solution is a bit of a hacky one.

It's also possible that the upcoming Container Runtime Interface changes, with a focus on sandboxes, will require further changes in both the kubelet and rkt that changes what issues are present.

I'd like to gather other's thoughts and make sure we're all on the same page and figure out if there are other options or issues I'm not thinking of.

/cc @yifan-gu @jellonek @pskrzyns @jonboulle @dcbw @iaguis @philips @thockin @freehan