Issue 1:
I setup o 2-node kubernetes cluster using docker containers, as described here: https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/getting-started-guides/docker-multinode.md

Now, I created a secret and a test pod:

(using files from the branch 'release-1.1' and commit 'ac3f023ab28ed4f4c4f88a3fb518e90f4a8d4749')

kubectl create -f docs/user-guide/secrets/secret.yaml
kubectl create -f docs/user-guide/secrets/secret-pod.yaml

Both secret and pod are created:

$ kubectl describe secret test-secret
Name: test-secret
Namespace: default
Labels: <none>
Annotations: <none>

Type: Opaque

Data
====
data-1: 9 bytes
data-2: 11 bytes

$ kubectl describe po/secret-test-pod
Name: secret-test-pod
Namespace: default
Image(s): kubernetes/mounttest:0.1
Node: 127.0.0.1/127.0.0.1
Start Time: Thu, 10 Dec 2015 16:20:49 +0100
Labels: <none>
Status: Running
Reason:
Message:
IP: 10.1.61.8
Replication Controllers: <none>
Containers:
test-container:
Container ID: docker://82409e45ad6fcd73a8100a1b0767692d5e823cc3a14d8154644293d6b4c86854
Image: kubernetes/mounttest:0.1
Image ID: docker://b14127f747a0569e9ab96f0ee2de1b07630b7dbc16f8facf0b0adabd5848d41c
State: Running
Started: Thu, 10 Dec 2015 16:20:51 +0100
Ready: False
Restart Count: 0
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
secret-volume:
Type: Secret (a secret that should populate this volume)
SecretName: test-secret
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
9s 9s 1 {scheduler } Scheduled Successfully assigned secret-test-pod to 127.0.0.1
8s 8s 1 {kubelet 127.0.0.1} implicitly required container POD Pulled Container image "gcr.io/google_containers/pause:0.8.0" already present on machine
8s 8s 1 {kubelet 127.0.0.1} implicitly required container POD Created Created with docker id cb1290a496c8
8s 8s 1 {kubelet 127.0.0.1} implicitly required container POD Started Started with docker id cb1290a496c8
8s 8s 1 {kubelet 127.0.0.1} FailedSync Error syncing pod, skipping: open /var/lib/docker/containers/cb1290a496c8d3721064573b27ddd0530c864462374ec0106b5a2cb25fdee916/resolv.conf: read-only file system
7s 7s 1 {kubelet 127.0.0.1} spec.containers{test-container} Pulled Container image "kubernetes/mounttest:0.1" already present on machine
7s 7s 1 {kubelet 127.0.0.1} spec.containers{test-container} Created Created with docker id 82409e45ad6f
7s 7s 1 {kubelet 127.0.0.1} spec.containers{test-container} Started Started with docker id 82409e45ad6f

but the secret directory inside the container is empty:

$ kubectl logs secret-test-pod
error reading file content for "/etc/secret-volume/data-1": open /etc/secret-volume/data-1: no such file or directory

This is tested on ubuntu 14.04.3 and docker "version 1.9.1, build a34a1d5" with AUFS on ext4.

What does the "FailedSync Error" in the status of a pod mean? The kubelet's log is also full of lines like:

W1210 15:25:08.504876 8522 container_manager_linux.go:278] [ContainerManager] Failed to ensure state of "/docker-daemon": failed to move PID 20479 (in "/user/1000.user/14.session") to "/docker-daemon"
I1210 15:25:09.337270 8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/5819740f39449d681e73c362cbdcef218454bc0e8ba4c442905069d803be0d13 with output du: cannot access '/var/lib/docker/aufs/diff/5819740f39449d681e73c362cbdcef218454bc0e8ba4c442905069d803be0d13': No such file or directory
- exit status 1
I1210 15:25:10.939578 8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/81c12f13a7a06c075b8aa1544c6e070f5c3fbcb0dfc4e75adb290a45804faf16 with output du: cannot access '/var/lib/docker/aufs/diff/81c12f13a7a06c075b8aa1544c6e070f5c3fbcb0dfc4e75adb290a45804faf16': No such file or directory
- exit status 1
I1210 15:25:10.940009 8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/1685f493d119093d3f1ef0a069e0936a8aa7f8708409a22c700faad351f75d08 with output du: cannot access '/var/lib/docker/aufs/diff/1685f493d119093d3f1ef0a069e0936a8aa7f8708409a22c700faad351f75d08': No such file or directory
- exit status 1
I1210 15:25:11.010029 8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/ffef039cd9b3194b33a59b609f8561166b9caea3ca28ed7c2632edfde3322f38 with output du: cannot access '/var/lib/docker/aufs/diff/ffef039cd9b3194b33a59b609f8561166b9caea3ca28ed7c2632edfde3322f38': No such file or directory
- exit status 1
I1210 15:25:13.054248 8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/b47debeea5b8943708b360b55f6152116189c1c7724ad71dfa1596dd2e8f4160 with output du: cannot access '/var/lib/docker/aufs/diff/b47debeea5b8943708b360b55f6152116189c1c7724ad71dfa1596dd2e8f4160': No such file or directory
- exit status 1
I1210 15:25:14.771897 8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/38add271c97d5c427f14433d65a124605a82c88d407f96654acc77dc4016e04b with output du: cannot access '/var/lib/docker/aufs/diff/38add271c97d5c427f14433d65a124605a82c88d407f96654acc77dc4016e04b': No such file or directory
- exit status 1
I1210 15:25:16.292393 8522 fsHandler.go:87] failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/3bce298fa4e90e4122eddcd0f9412b2f663f2dbea41dfd0257f6bcdb9fae4476 with output du: cannot access '/var/lib/docker/aufs/diff/3bce298fa4e90e4122eddcd0f9412b2f663f2dbea41dfd0257f6bcdb9fae4476': No such file or directory
- exit status 1

but none of them is "error"

Issue 2:
When running a containerized kubernetes cluster I cannot spawn pods due to file does not exist errors when trying to mount their secret volumes:

$ kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: busybox-sleep
spec:
containers:
- name: busybox
image: busybox
args:
- sleep
- "1000000"
EOF
pod "busybox-sleep" created

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox-sleep 0/1 ContainerCreating 0 15s
k8s-master-127.0.0.1 3/3 Running 0 8m

$ kubectl describe pod busybox-sleep
Name: busybox-sleep
Namespace: default
Image(s): busybox
Node: 127.0.0.1/127.0.0.1
Start Time: Fri, 04 Dec 2015 22:48:56 +0000
Labels: <none>
Status: Pending
Reason:
Message:
IP:
Replication Controllers: <none>
Containers:
busybox:
Container ID:
Image: busybox
Image ID:
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
default-token-6dwl1:
Type: Secret (a secret that should populate this volume)
SecretName: default-token-6dwl1
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
───────── ──────── ───── ──── ───────────── ────── ───────
32s 32s 1 {scheduler } Scheduled Successfully assigned busybox-sleep to 127.0.0.1
32s 10s 3 {kubelet 127.0.0.1} FailedMount Unable to mount volumes for pod "busybox-sleep_default": IsLikelyNotMountPoint("/var/lib/kubelet/pods/333c85c3-9ad9-11e5-aeed-f6b7b4d3b3d7/volumes/kubernetes.io~secret/default-token-6dwl1"): file does not exist
32s 10s 3 {kubelet 127.0.0.1} FailedSync Error syncing pod, skipping: IsLikelyNotMountPoint("/var/lib/kubelet/pods/333c85c3-9ad9-11e5-aeed-f6b7b4d3b3d7/volumes/kubernetes.io~secret/default-token-6dwl1"): file does not exist

The secrets are there as expected:

$ kubectl get secrets
NAME TYPE DATA AGE
default-token-6dwl1 kubernetes.io/service-account-token 2 8m

Here are the full kubelet logs: https://gist.github.com/2opremio/6fcbae55d83f20bc3c82

The single-node containerized cluster is created using this script, which is based on https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker.md and spawns a cluster based on a recent version of master (https://github.com/kubernetes/kubernetes/tree/81b6ac47557359429e519d889d9ad48b5c8a2411).

This is the docker version I am running:

$ docker info
Containers: 7
Images: 62
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 996.2 MiB
Name: default
ID: 77RC:MDFB:UNPM:62Z6:FYLF:TZTG:43NZ:77DD:IL22:QI6S:LDSV:UN6W
Debug mode (server): true
File Descriptors: 54
Goroutines: 90
System Time: 2015-12-04T23:01:25.308787463Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
provider=virtualbox

$ docker version
Client:
Version: 1.8.1
API version: 1.20
Go version: go1.4.2
Git commit: d12ea79
Built: Thu Aug 13 02:49:29 UTC 2015
OS/Arch: darwin/amd64

Server:
Version: 1.9.1
API version: 1.21
Go version: go1.4.3
Git commit: a34a1d5
Built: Fri Nov 20 17:56:04 UTC 2015
OS/Arch: linux/amd64

The client runs in an OSX laptop and the daemon runs inside a VM provisioned with docker-machine create --driver virtualbox --engine-storage-driver overlay default

Surprisingly, everything works when using a vagrant VM provisioned with https://github.com/weaveworks/weave/blob/vagrant-pin-docker/Vagrantfile, so the problem must somehow be environmental but I haven't managed to triage the difference causing the problem.