Change:
We seemed to have curbed the kubelet memory issues in , but the runtime issues remain. https://k8s- has a lot of these. This can likely be de-duped to an auto-filed issue when the soak tests are to the munger.
I created a cluster and ran a scrip to repeatedly create/delete pods overnight, but couldn't reproduce this issue. Uploaded PR to enable the debug mode in docker, so that one can run pprof to get more useful information from the soak cluster.
I redeployed the GCE soak tests yesterday to pick up to help you debug faster. I verified that docker is running with this flag on the nodes. So as soon as it re-occurs on https://k8s- , you should be able to debug it. Just out of curiosity, what sort of debugging does that flag enable?
I know it at least enables the /debug/pprof endpoint. EDIT: and thanks for deploying a new cluster.
The new cluster started failing again: Some information I gathered: The goroutines: Will check later(?) to see if the memory uasge keeps increasing
The heap profile shows that is slowly accumulating more memory. This can be reproduced by repeatedly creating and deleting containers for thousands of times using docker v1.11 (without kubernetes involved at all). A potentially related issue is found on I didn't try it on docker v1.12 and am not sure if it suffers from the same symptom. I was able to also reproduce this through unit tests in the package. On such unit test is I modified the test to use UUIDs as keys and change the order to always insert a key and then delete right away. After running the test for 10 million iterations, the test always fails with an increase of memory usage on the heap. I filed an issue to see if the unit test failure is related, or just a red herring:
Is this p0 flake a symptom of this issue?
In , both kubelet and runtime exceed the memory limit. This issue is for the "runtime" memory issue, but not for kubelet. The kubelet memory issue seems to be relatively new. I'll look into that as well.
Update: There is a fix posted in W can't patch docker 1.11 anyway, so the soak test will continue failing. this is the docker memory leak I mentioned yesterday.
I though this was a slow leak that only shows up in soak but apparently integration tests are hitting it too: , or is that something else, or just a confusing error message?
fwiw the logs from this failure are https://k8s-, I haven't trawled through them yet.
Looks to me like the go runtime was out of memory when building/running testmain Not sure how this is related to docker
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
This is still an issue, and would like to get this passing in the kubernetes-soak-continuous-e2e-gce-1.4 test suite for the release. I know we talked about patching docker at least for the GCE images. Is that still an option? If not, could you sync up with about how to proceed? Thanks.
Patching docker for today's GCE images is not an option. I am considering patching docker in GCI image though. cc/
It's already patched in GCI.
given that patching docker in the containervm image is not an option, the only way to pass this test is to disable checking the docker memory usage completely in the test. I'll let determine what's the best course for the release.
Here are the plan I wrote in bot auto-filed flake issue: , : was merged to stop the bleeding. I will leave this open for rest work.
These tests are now also failing in case of kubernetes-e2e-gce-gci-serial-release-1.4.
thanks for checking the last failure. Yes, this is known issue, and shouldn't block 1.4 release since GCI default on node is decoupled from the releases.
If it's not a release blocker, do you suggest that we disable the test in the 1.4 branch? cc
Filed to disable the tests for 1.4.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
To help clean up the list of flakes for 1.6, I am closing this issue as it does not appear to flake anymore with the increased limits.

Discussion:
# Extra docker options for nodes. EXTRA_DOCKER_OPTS="${EXTRA_DOCKER_OPTS:-}" <ins> # Enable the docker debug mode. EXTRA_DOCKER_OPTS="${EXTRA_DOCKER_OPTS} --debug" </ins> SERVICE_CLUSTER_IP_RANGE="10.0.0.0/16"  # formerly PORTAL_NET # When set to true, Docker Cache is enabled by default as part of the cluster bring up.