Issue 1:
I have a simple setup with 1 master and 2 nodes running a couple of pods on Kubernetes v0.16.0.

When I simulate a crash on one of them, pods keep being scheduled on it indefinitely.

$ kubectl get nodes

NAME LABELS STATUS
ip-10-0-21-131.eu-west-1.compute.internal <none> Ready
ip-10-0-21-132.eu-west-1.compute.internal <none> NotReady

$ kubectl get pods
web-4a3cq 10.144.71.11 ip-10-0-21-131.eu-west-1.compute.internal/10.0.21.131 app=web Running 23 hours
web-st0du ip-10-0-21-132.eu-west-1.compute.internal/ app=web Pending 2 seconds

$ journalctl -u kube-controller-manager
nodecontroller.go:431] node ip-10-0-21-132.eu-west-1.compute.internal hasn't been updated for 15m30.391884952s. Last ready condition is: {Type:Ready Status:Unknown LastHeartbeatTime:2015-05-01 15:30:47 +0000 UTC LastTransitionTime:2015-05-01 15:31:29 +0000 UTC Reason:Kubelet stopped posting node status. Message:}
nodecontroller.go:504] Evicting pods2: 2015-05-01 15:46:19.911780538 +0000 UTC is later than 2015-05-01 15:46:19.911779449 +0000 UTC + -10s
nodecontroller.go:175] Error syncing cloud: Request has expired. (RequestExpired)nodecontroller.go:431] node ip-10-0-21-132.eu-west-1.compute.internal hasn't been updated for 15m35.402783207s. Last ready condition is: {Type:Ready Status:Unkn
own LastHeartbeatTime:2015-05-01 15:30:47 +0000 UTC LastTransitionTime:2015-05-01 15:31:29 +0000 UTC Reason:Kubelet stopped posting node status. Message:}
nodecontroller.go:504] Evicting pods2: 2015-05-01 15:46:24.924422488 +0000 UTC is later than 2015-05-01 15:46:24.924421484 +0000 UTC + -10snodecontroller.go:584] Delete all pods from ip-10-0-21-132.eu-west-1.compute.internal
nodecontroller.go:595] Delete pod api-seu4e
nodecontroller.go:595] Delete pod web-mbpos
replication_controller.go:253] Too few "api" replicas, creating 1
replication_controller.go:253] Too few "web" replicas, creating 1
endpoints_controller.go:314] Failed to find an IP for pod default/web-hcxtd
endpoints_controller.go:314] Failed to find an IP for pod default/api-rb12s
endpoints_controller.go:314] Failed to find an IP for pod default/web-hcxtd
endpoints_controller.go:314] Failed to find an IP for pod default/api-rb12s

Issue 2:
I am running my cluster on aws. The setups has a etcd cluster with 3 etcd nodes, 1 master and 4 minions.

I am running a simple hello-world nodejs script using docker to test my application. For skydns i am using the steps mentioned here

I shutdown all minions, while my master and etcd were running.

A kubectl get nodes on the master showed that two nodes were not ready and the other two as ready

core@ip-172-31-24-64 ~ $ ./kubectl get nodes
NAME LABELS STATUS
172.31.25.11 Schedulable name=dns NotReady
172.31.25.12 Schedulable name=dns NotReady
172.31.26.95 Schedulable name=minion Ready
172.31.27.92 Schedulable name=minion Ready

Is it expected that the LABEL for a shut-down node is Schedulable?

I now scheduled my pods using replicationControllers and created the associated services. One of the pods was scheduled on a "NotReady" node.

core@ip-172-31-24-64 ~ $ ./kubectl get pods
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED
replicont-0cj5l 172.17.0.4 basicnodejs basicnodejs 172.31.26.95/ name=replicont Running 19 minutes
replicont-gl3wm 172.17.0.4 basicnodejs basicnodejs 172.31.27.92/ name=replicont Running 19 minutes
skydns-1ad74 172.17.0.5 etcd quay.io/coreos/etcd:latest 172.31.27.92/ name=skydns Running 18 minutes
kube2sky kubernetes/kube2sky:1.0
skydns kubernetes/skydns:2014-12-23-001
skydns-oib7w etcd quay.io/coreos/etcd:latest 172.31.25.12/ name=skydns Pending 7 minutes
kube2sky kubernetes/kube2sky:1.0
skydns kubernetes/skydns:2014-12-23-001

I ran a kubectl delete pod on the problematic pod and the new pod was scheduled on another node which is not up.

I waited for some time(may have been 5-10 minutes) and the pod was rescheduled on a live node. So I killed it and a new one was scheduled on a shutdown node.