Issue 1:
See log here: https://gist.github.com/sebv/914c9b73b48e4542bfeb , the symlink to /mnt/ephemeral/docker/containers is not valid within container.

This is a new issue on 1.1 (it's working on 1.0.7)

On the node:

ubuntu@ip-172-20-0-155:~$ sudo ls -l /var/log/containers
total 80
lrwxrwxrwx 1 root root 171 Nov 23 21:24 elasticsearch-logging-v1-4yjm2_kube-system_elasticsearch-logging-b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041.log -> /mnt/ephemeral/docker/containers/b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041/b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041-json.log

My version:

~ kubectl version
Client Version: version.Info{Major:"1", Minor:"0+", GitVersion:"v1.0.8-beta.3+27719743e6b69e", GitCommit:"27719743e6b69e9526d5133b6d6b71cc6f1024c2", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.1", GitCommit:"92635e23dfafb2ddc828c8ac6c03c7a7205a84d8", GitTreeState:"clean"}

Issue 2:
It seems that there are potentially two separate issues here:

The inconsistent docker root due to a potential race between kubelet and docker.

The misconfiguration (on AWS) of fluentd to mount /var/lib/docker instead of /mnt/ephemeral/docker.

For the first issue, perhaps the kubelet should fail if it is not able to retrieve the docker root from docker info?

An alternative solution would be to mount our volume for AWS to /var/lib/docker instead of /mnt/ephemeral/docker. This would solve both issues. But, it seems that the kubelet is providing the ability to dynamically locate the docker root for some reason.

Details:

I'm having intermittent issues with fluentd pushing logs to elasticsearch. It seems that fluentd is unable to see the log files, even though it is able list the containers directory. This is happening on an AWS cluster.

From inside the fluentd I see the following:

root@fluentd-elasticsearch-ip-172-20-0-194:/# cat /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log
cat: /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log: No such file or directory

On AWS the log file symlinks look like this:

root@fluentd-elasticsearch-ip-172-20-0-194:/# ls -al /varlog/containers/vapor-web-v1-x894w_*
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_POD-99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0.log -> /mnt/ephemeral/docker/containers/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0-json.log
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log -> /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_vapor-unicorn-1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a.log -> /mnt/ephemeral/docker/containers/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a-json.log

Fluentd seems to expect that the symlinks will point to /var/lib/docker/containers but on AWS we use /mnt/ephemeral/docker/containers.

From the node I see:

root@ip-172-20-0-194:/home/ubuntu# ls -al /var/log/containers/vapor-web-v1-x894w_vapor-alpha_*
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log -> /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_POD-99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0.log -> /mnt/ephemeral/docker/containers/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0-json.log
lrwxrwxrwx 1 root root 171 Aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_vapor-unicorn-1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a.log -> /mnt/ephemeral/docker/containers/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a-json.log

root@ip-172-20-0-194:/home/ubuntu# cat /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log | tail -n 5
{"log":"172.20.0.70 - - [28/Aug/2015:17:29:26 +0000] \"GET / HTTP/1.1\" 302 116 \"-\" \"Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)\"\n","stream":"stdout","time":"2015-08-28T17:29:26.242591558Z"}
{"log":"172.20.0.70 - - [28/Aug/2015:17:29:26 +0000] \"GET /users/sign_in HTTP/1.1\" 200 1954 \"-\" \"Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)\"\n","stream":"stdout","time":"2015-08-28T17:29:26.265429648Z"}
{"log":"2015/08/28 17:29:26 [info] 5#5: *58423 client 172.20.0.70 closed keepalive connection\n","stream":"stdout","time":"2015-08-28T17:29:26.271201715Z"}
{"log":"2015/08/28 17:29:38 [info] 5#5: *58426 client closed connection while waiting for request, client: 172.20.0.171, server: 0.0.0.0:80\n","stream":"stdout","time":"2015-08-28T17:29:38.549273837Z"}
{"log":"2015/08/28 17:29:46 [info] 5#5: *58427 client closed connection while waiting for request, client: 172.20.0.33, server: 0.0.0.0:80\n","stream":"stdout","time":"2015-08-28T17:29:46.549263618Z"}