Issue 1:
Over a year ago, we had the kubernetes-build job down <5m. It's now coming in at 30m. Some of that shot up really recently, too, looking at the build time trend:

It looks like sometime around #10384, we shot up from a fat 20m another 50% to 30m. Can we at least eliminate that jump?

Is there anything else we can do here? Maybe split the non-amd64 builds off to a separate job so that our whole CI infrastructure isn't tied to them?

@kubernetes/test-infra-maintainers @kubernetes/release-maintainers

Issue 2:
Version-Release number of selected component (if applicable):

Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.0.1310+8741217179860e", GitCommit:"8741217179860e9f0ce37997c810f61170a3672a", GitTreeState:"clean", BuildDate:"2016-07-07T06:54:04Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Steps to Reproduce:

1.Install kubernetes with one master and two nodes

2.Create a pod using replicationcontroller

apiVersion: v1

kind: ReplicationController

metadata:

name: chaoyangwildfly-rc

labels:

name: chaoyangwildfly

context: docker-k8s-lab

spec:

replicas: 1

template:

metadata:

labels:

name: chaoyangwildfly

spec:

containers:

- name: chaoyangwildfly-rc-pod

image: jhou/hello-openshift

ports:

- containerPort: 8080

volumeMounts:

- name: html-volume

mountPath: "/usr/share/nginx/html"

volumes:

- name: html-volume

awsElasticBlockStore:

volumeID: aws://us-east-1d/vol-dde44879

fsType: ext4

3.Check pod status

[root@ip-172-18-5-143 ~]# kubectl describe pods chaoyangwildfly-rc-ccx51

Name: chaoyangwildfly-rc-ccx51

Namespace: default

Node: ip-172-18-0-61.ec2.internal/172.18.0.61

Start Time: Fri, 08 Jul 2016 02:56:00 -0400

Labels: name=chaoyangwildfly

Status: Running

IP: 172.16.73.10

Controllers: ReplicationController/chaoyangwildfly-rc

Containers:

chaoyangwildfly-rc-pod:

Container ID: docker://0bb522c3d8305c13decef15c5624637bb3824021be78ae75b086be1560369817

Image: jhou/hello-openshift

Image ID: docker://sha256:3642a95271f490f9d618e29128a089ebaaf58f8f3f4e556c02660b54ebb881fd

Port: 8080/TCP

QoS Tier:

memory: BestEffort

cpu: BestEffort

State: Running

Started: Fri, 08 Jul 2016 02:56:08 -0400

Ready: True

Restart Count: 0

Environment Variables:

Conditions:

Type Status

Initialized True

Ready False

PodScheduled True

Volumes:

html-volume:

Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)

VolumeID: aws://us-east-1d/vol-dde44879

FSType: ext4

Partition: 0

ReadOnly: false

default-token-vs6y1:

Type: Secret (a volume populated by a Secret)

SecretName: default-token-vs6y1

Events:

FirstSeen LastSeen Count From SubobjectPath Type Reason Message

5m 5m 1 {default-scheduler } Normal Scheduled Successfully assigned chaoyangwildfly-rc-ccx51 to ip-172-18-0-61.ec2.internal

5m 5m 1 {kubelet ip-172-18-0-61.ec2.internal} spec.containers{chaoyangwildfly-rc-pod} Normal Pulling pulling image "jhou/hello-openshift"

5m 5m 1 {kubelet ip-172-18-0-61.ec2.internal} spec.containers{chaoyangwildfly-rc-pod} Normal Pulled Successfully pulled image "jhou/hello-openshift"

5m 5m 1 {kubelet ip-172-18-0-61.ec2.internal} spec.containers{chaoyangwildfly-rc-pod} Normal Created Created container with docker id 0bb522c3d830

5m 5m 1 {kubelet ip-172-18-0-61.ec2.internal} spec.containers{chaoyangwildfly-rc-pod} Normal Started Started container with docker id 0bb522c3d830

4..After the pod is running, stop kubelet service on the node ip-172-18-0-61.ec2.internal

5. A new pod will be assigned to the other node

[root@ip-172-18-5-143 ~]# kubectl describe pods chaoyangwildfly-rc-hn9kn

Name: chaoyangwildfly-rc-hn9kn

Namespace: default

Node: ip-172-18-9-229.ec2.internal/172.18.9.229

Start Time: Fri, 08 Jul 2016 03:02:31 -0400

Labels: name=chaoyangwildfly

Status: Pending

IP:

Controllers: ReplicationController/chaoyangwildfly-rc

Containers:

chaoyangwildfly-rc-pod:

Container ID:

Image: jhou/hello-openshift

Image ID:

Port: 8080/TCP

QoS Tier:

cpu: BestEffort

memory: BestEffort

State: Waiting

Reason: ContainerCreating

Ready: False

Restart Count: 0

Environment Variables:

Conditions:

Type Status

Initialized True

Ready False

PodScheduled True

Volumes:

html-volume:

Type: AWSElasticBlockStore (a Persistent Disk resource in AWS)

VolumeID: aws://us-east-1d/vol-dde44879

FSType: ext4

Partition: 0

ReadOnly: false

default-token-vs6y1:

Type: Secret (a volume populated by a Secret)

SecretName: default-token-vs6y1

Events:

FirstSeen LastSeen Count From SubobjectPath Type Reason Message

4m 4m 1 {default-scheduler } Normal Scheduled Successfully assigned chaoyangwildfly-rc-hn9kn to ip-172-18-9-229.ec2.internal

2m 1s 2 {kubelet ip-172-18-9-229.ec2.internal} Warning FailedMount Unable to mount volumes for pod "chaoyangwildfly-rc-hn9kn_default(f059eb80-44d9-11e6-ba51-0ecfeba772c9)": timeout expired waiting for volumes to attach/mount for pod "chaoyangwildfly-rc-hn9kn"/"default". list of unattached/unmounted volumes=[html-volume]

2m 1s 2 {kubelet ip-172-18-9-229.ec2.internal} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "chaoyangwildfly-rc-hn9kn"/"default". list of unattached/unmounted volumes=[html-volume]

From aws web console, I foud volume vol-dde44879 is still attched to node ip-172-18-0-61.ec2.internal