Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version):

Environment:

AWS

Xenial

Terraform

What happened:

I am setting up a K8s cluster using Terraform + kubeadm in AWS. Etcd, kube master, and all kube nodes are in auto-scaling groups and behind load balancers, with the idea being that if some or all nodes fail, the cluster will heal itself. Most recently, I have been working to get the aws cloud-provider to work. At one point, I corrected my kubelet systemd file and deleted the kmaster and knode instances. When they came back up, the discovery service failed to start on the master node.

<master/discovery> failed to create "kube-discovery" deployment [deployments.extensions "kube-discovery" already exists]

With the existing deployment, I was surprised that a new discovery pod wasn't scheduled, but it stayed in pending.

root@ip-10-253-128-238:~# kubectl describe pod/kube-discovery-1150918428-aca0v --namespace=kube-system
Name: kube-discovery-1150918428-aca0v
Namespace: kube-system
Node: /
Labels: component=kube-discovery
k8s-app=kube-discovery
kubernetes.io/cluster-service=true
name=kube-discovery
pod-template-hash=1150918428
tier=node
Status: Pending
IP:
Controllers: ReplicaSet/kube-discovery-1150918428
Containers:
kube-discovery:
Image: gcr.io/google_containers/kube-discovery-amd64:1.0
Port: 9898/TCP
Command:
/usr/local/bin/kube-discovery
Volume Mounts:
/tmp/secret from clusterinfo (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-lhygh (ro)
Environment Variables: <none>
Conditions:
Type Status
PodScheduled False
Volumes:
clusterinfo:
Type: Secret (a volume populated by a Secret)
SecretName: clusterinfo
default-token-lhygh:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-lhygh
QoS Class: BestEffort
Tolerations: dedicated=master:NoSchedule
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
59m 20s 147 {default-scheduler } Warning FailedScheduling pod (kube-discovery-1150918428-aca0v) failed to fit in any node
fit failure on node (ip-10-253-128-238.ec2.internal): MatchNodeSelector

What you expected to happen:

Either for kubeadm to delete the existing deployment, or for that deployment to sucessfully schedule new pods on the new master node.

How to reproduce it (as minimally and precisely as possible):

Bring up a cluster using kubeadm and external etcd. After the cluster is up, delete the master node and try to bring it back up (using the same command as the first time).

Issue 2:
From build-log.txt in #21439 (before I retest) (Tried to find a flake dupe but didn't see an obvious one):

I0217 18:27:49.485036 12269 manager.go:1620] Found pod infra container for "phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)"
I0217 18:27:49.485120 12269 manager.go:1633] Pod infra container looks good, keep it "phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)"
I0217 18:27:49.485169 12269 manager.go:1657] pod "phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)" container "c1" exists as /k8s_c1.cd1b02d5_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_46090a9a
I0217 18:27:49.485201 12269 manager.go:1725] Got container changes for pod "phantom.baz_default(3324bd82-d5e7-11e5-bf2a-0242ac110003)": {StartInfraContainer:false InfraChanged:false InfraContainerId:/k8s_POD.6d140183_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_e7f1f119 ContainersToStart:map[] ContainersToKeep:map[/k8s_POD.6d140183_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_e7f1f119:-1 /k8s_c1.cd1b02d5_phantom.baz_default_3324bd82-d5e7-11e5-bf2a-0242ac110003_46090a9a:0]}
I0217 18:27:49.692180 12269 integration.go:958] Scheduler doesn't make phantom pods: test passed.
I0217 18:27:49.692211 12269 integration.go:1068]

Logging high latency metrics from the 10250 kubelet
I0217 18:27:49.716299 12269 server.go:1083] GET /metrics: (23.237036ms) 200 [[Go-http-client/1.1] 127.0.0.1:56949]
�������iI0217 18:27:49.725600 12269 integration.go:1070]

Logging high latency metrics from the 10251 kubelet
Feb 17 18:27:49.725: INFO:
Latency metrics for node localhost:10250
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.99 Latency:6.002896s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_start_latency_microseconds Quantile:0.9 Latency:5.002607s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.99 Latency:5.002121s}
Feb 17 18:27:49.725: INFO: {Operation: Method:pod_worker_start_latency_microseconds Quantile:0.9 Latency:5.002121s}
Feb 17 18:27:49.725: INFO: {Operation:sync Method:pod_worker_latency_microseconds Quantile:0.99 Latency:1.001256s}
Error deleting container: Error response from daemon: Conflict, You cannot remove a running container. Stop the container before attempting removal or use -f
Unrecognized input header
Build step 'Execute shell' marked build as failure
Recording test results