Change:
I came upon a cluster that had had the namespace code thoroughly exercised. No namespaces currently existed, but many had been created and deleted in the past. Its etcd had lots of empty directories under each resource that had held items when those namespaces were alive (secrets, serviceaccounts, pods...). Do we need to be doing something to clean these empty directories up? It seems like this etcd was consuming a lot of memory.
or might know the answer to this question.
Right. We should delete them. It might get tricky for concurrent cases like A checks if dir is empty, B creates, then A deletes it. What does it look like currently in creating and deleting namespace?
How much memory does a directory entry take? Does it really sound possible to run out of memory while snapshotting on account of too many empty directories?
-- have you guys encountered this? I think you run with more namespaces than we do. But maybe yours are mostly populated?
Yes, we've seen it. c.f.
Approximate memory use we saw from empty dirs:
cc
The patch we debated carrying is here: Our hope was to just wait to etcd v3 where this would not be a problem.
can you link to your clean-up script here?
I needed another PR to run the pod automatically on master.

Discussion:
<ins> apiVersion: v1 kind: Pod metadata: name: etcd-empty-dir-cleanup namespace: kube-system labels: k8s-app: etcd-empty-dir-cleanup spec: hostNetwork: true dnsPolicy: Default containers: - name: etcd-empty-dir-cleanup image: gcr.io/google_containers/etcd-empty-dir-cleanup:0.0.1 </ins>

<ins> # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. FROM gliderlabs/alpine MAINTAINER Mehdy Bohlool <mehdy@google.com> RUN apk-install bash ADD etcd-empty-dir-cleanup.sh etcd-empty-dir-cleanup.sh ADD etcdctl etcdctl ENV ETCDCTL /etcdctl ENV SLEEP_SECOND 3600 RUN chmod +x etcd-empty-dir-cleanup.sh CMD bash /etcd-empty-dir-cleanup.sh </ins>

<ins> # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. .PHONY:	build push ETCD_VERSION = 2.2.1 IMAGE = gcr.io/google_containers/etcd-empty-dir-cleanup TAG = 0.0.1 clean: rm -rf etcdctl etcd-v$(ETCD_VERSION)-linux-amd64 etcd-v$(ETCD_VERSION)-linux-amd64.tar.gz build: clean curl -L -O https://github.com/coreos/etcd/releases/download/v$(ETCD_VERSION)/etcd-v$(ETCD_VERSION)-linux-amd64.tar.gz tar xzvf etcd-v$(ETCD_VERSION)-linux-amd64.tar.gz cp etcd-v$(ETCD_VERSION)-linux-amd64/etcdctl . docker build -t $(IMAGE):$(TAG) . rm -rf etcdctl etcd-v$(ETCD_VERSION)-linux-amd64 etcd-v$(ETCD_VERSION)-linux-amd64.tar.gz push: build gcloud docker push $(IMAGE):$(TAG) </ins>

<ins> #!/bin/bash # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. echo "Removing empty directories from etcd..." cleanup_empty_dirs () { if [[ $(${ETCDCTL} ls $1) ]]; then for SUBDIR in $(${ETCDCTL} ls -p $1 | grep "/$") do cleanup_empty_dirs ${SUBDIR} done else echo "Removing empty key $1 ..." ${ETCDCTL} rmdir $1 fi } while true do echo "Starting cleanup..." cleanup_empty_dirs "/registry" echo "Done with cleanup." sleep ${SLEEP_SECOND} done </ins>  No newline at end of file