Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): rbd

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:06:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: Baremetal (Supermicro 1028R)

OS (e.g. from /etc/os-release): CoreOS 1068.10.0

Kernel (e.g. uname -a): 4.6.3-coreos

Install tools: custom

Others:

What happened:

I first deployed a new pod (via deployment) for running Grafana, using two Ceph RBD volumes (for /var/lib/grafana and /etc/grafana).

I then found that I'd mounted a volume over the default config with the /etc/grafana mount and decided to just skip that for now and remove that volume. I applied a new version of the deployment with just the RBD for /var/lib/grafana. It got created on a different node and was unable to mount the volume because it was already mounted on the first node.

On investigation, I found that the original node still had the RBD volumes mapped and mounted. The kubelet logs indicate "success" when tearing down the volumes:

Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809860 766 reconciler.go:179] UnmountVolume operation started for volume "kubernetes.io/secret/6e4e5190-7617-11e6-b97e-0015c5edcf5a-default-token-a7f06" (spec.Name: "default-token-a7f06") from pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-001
5c5edcf5a").
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809897 766 reconciler.go:179] UnmountVolume operation started for volume "kubernetes.io/rbd/6e4e5190-7617-
11e6-b97e-0015c5edcf5a-grafana-cfg" (spec.Name: "grafana-cfg") from pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a").
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809915 766 reconciler.go:179] UnmountVolume operation started for volume "kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-data" (spec.Name: "grafana-data") from pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a").
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.840960 766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4e5190-7617-11e6-b97e-0015c5edcf5a-default-token-a7f06" (OuterVolumeSpecName: "default-token-a7f06") pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a"). InnerVolumeSpecName "default-token-a7f06". PluginName "kubernetes.io/secret", VolumeGidValue ""
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.843030 766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume "kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-data" (OuterVolumeSpecName: "grafana-data") pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a"). InnerVolumeSpecName "grafana-data". PluginName "kubernetes.io/rbd", VolumeGidValue ""
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.844789 766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume "kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-cfg" (OuterVolumeSpecName: "grafana-cfg") pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a"). InnerVolumeSpecName "grafana-cfg". PluginName "kubernetes.io/rbd", VolumeGidValue ""

You can see that each volume appears twice in the output of mount (which is true of apparently everything under /var/lib/kubelet:

/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-cfg type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-cfg type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd1 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-data type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd1 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-data type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)

I expect this might be related to recent changes I had to make to how I'm running Docker and Kubelet in order to make secret mounts under Kubernetes 1.3 work. (Bindmounting /var/lib/kubelet onto itself and then marking it shared, per some updated hyperkube instructions... that I can no longer find.) See below for details of how I'm running kubelet.

What you expected to happen:

That the original node would successfully unmount and unmap the rbd volumes.

How to reproduce it (as minimally and precisely as possible):

Delete a pod with an RBD volume running in a kubelet, possibly with my specific options. (See below.)

Anything else do we need to know:

Here's the systemd unit for running the kubelet:

[Unit]
Description=Kubernetes Worker Node
Wants=network-online.target
After=network-online.target

[Service]
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=-/usr/bin/docker kill kubelet
ExecStartPre=-/usr/bin/docker rm kubelet
ExecStartPre=/usr/bin/bash -c "/usr/bin/mountpoint -q /var/lib/kubelet || /usr/bin/mount --bind /var/lib/kubelet /var/lib/kubelet"
ExecStartPre=/usr/bin/mount --make-shared /var/lib/kubelet
ExecStartPre=/usr/bin/docker pull docker.imvu.com/kubernetes/hyperkube:v1.3.6
ExecStart=/usr/bin/docker run \
--name kubelet \
-v /etc/kubernetes:/etc/kubernetes \
-v /etc/ssl:/etc/ssl:ro \
-v /etc/ceph:/etc/ceph \
-v /usr/sbin/modprobe:/sbin/modprobe:ro \
-v /lib/modules:/lib/modules:ro \
-v /dev:/dev \
-v /root/.docker/config.json:/root/.docker/config.json:ro \
-e HOME=/root/ \
--net=host \
--pid=host \
--privileged \
--volume=/:/rootfs:ro \
--volume=/sys:/sys:rw \
--volume=/var/lib/docker/:/var/lib/docker:rw \
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw,shared \
--volume=/var/run:/var/run:rw \
docker.imvu.com/kubernetes/hyperkube:v1.3.6 \
/hyperkube kubelet \
--containerized \
--api-servers=https://10.2.0.24 \
--register-node=true \
--allow-privileged=true \
--config=/etc/kubernetes/manifests \
--cluster-dns=10.96.0.2 \
--cluster-domain=k8s.prod.imvu.com \
--kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
--kube-reserved=cpu=500m,memory=128Mi \
--node-labels=%s \
--tls-cert-file=/etc/ssl/host-cert.pem \
--tls-private-key-file=/etc/ssl/host-key.pem
Restart=always
RestartSec=10

(The %s is replaced with a string at generation time.)

docker.imvu.com/kubernetes/hyperkube:v1.3.6 is a customized image based on the upstream hyperkube Dockerfile, with some extra packages and rebased on top of a version of Ubuntu that we ensure has all security updates. Here's the Dockerfile for it:

FROM docker.imvu.com/base/ubuntu:16.04

ADD docker/kubernetes/hyperkube/ceph-release.asc /tmp/ceph-release.asc
RUN apt-key add /tmp/ceph-release.asc

ADD docker/kubernetes/hyperkube/ceph.list /etc/apt/sources.list.d/ceph.list
RUN apt-get update

RUN apt-get-for-docker.sh \
ca-certificates \
ceph-common \
conntrack \
curl \
ethtool \
file \
git \
iptables \
socat \
nfs-common \
util-linux

RUN cp /usr/bin/nsenter /nsenter

COPY docker/kubernetes/hyperkube/hyperkube /hyperkube
RUN chmod a+rx /hyperkube

RUN ln -s /hyperkube /kubelet

The hyperkube binary there is the one from the 1.3.6 release tarball.

Issue 2:
The containerPort attribute in the pod manifest could be inferred from the image metadata when not supplied by the developer.

That way developers won't have to duplicate the information they already put in EXPOSE in their Dockerfile.

This information looks available through the remote API on /image resources under the PortSpecs attribute:

https://docs.docker.com/reference/api/docker_remote_api_v1.13/#inspect-an-image