jiebi/CodeConvo huggingface dataset:
Code change:
<ins> # 3.1 * The pause container gains a signal handler to clean up orphaned zombie processes. ([#36853](https://prs.k8s.io/36853), [@verb](https://github.com/verb)) * `pause -v` will return build information for the pause binary. ([#56762](https://prs.k8s.io/56762), [@verb](https://github.com/verb)) # 3.0 * The pause container was rewritten entirely in C. ([#23009](https://prs.k8s.io/23009), [@uluyol](https://github.com/uluyol)) </ins>

Related discussion:
text: 00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10
Code change:
Spec: api.ServiceSpec{ Ports: []api.ServicePort{ { <del> Name:     "default", Protocol: api.Protocol("UDP"), Port:     14, </del> <ins> Name:       "default", Protocol:   api.Protocol("UDP"), Port:       14, TargetPort: util.NewIntOrStringFromInt(14), </ins> }, }, Selector: map[string]string{"func": "stream"}, Type:     api.ServiceTypeLoadBalancer, }, }, <del> expected: "you will also need to explicitly open a firewall", status:   200, </del> <ins> status: 200, </ins> }, }

Related discussion:
text: This was broken by , and therefore broken in as well. Any fix should be offered as a cherry pick so "make release" is clean there.
I went ahead and worked something up for this this morning, to get Shippable builds back on the rails.
Code change:
func TestSecretForDockerRegistryGenerate(t *testing.T) { username, password, email, server := "test-user", "test-password", "test-user@example.org", "https://index.docker.io/v1/" <del> secretData, err := handleDockercfgContent(username, password, email, server) </del> <ins> secretData, err := handleDockerCfgJsonContent(username, password, email, server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) } <del> secretDataNoEmail, err := handleDockercfgContent(username, password, "", server) </del> <ins> secretDataNoEmail, err := handleDockerCfgJsonContent(username, password, "", server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) }

Related discussion:
text: /kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.
Code change:
func TestSecretForDockerRegistryGenerate(t *testing.T) { username, password, email, server := "test-user", "test-password", "test-user@example.org", "https://index.docker.io/v1/" <del> secretData, err := handleDockercfgContent(username, password, email, server) </del> <ins> secretData, err := handleDockerCfgJsonContent(username, password, email, server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) } <del> secretDataNoEmail, err := handleDockercfgContent(username, password, "", server) </del> <ins> secretDataNoEmail, err := handleDockerCfgJsonContent(username, password, "", server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) }

Related discussion:
text: However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret
Code change:
} require.NoError(t, registerAPIService(ctx, client, groupVersion, service)) <del> defer func() { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) }() </del> <ins> groupVersions = append(groupVersions, groupVersion) </ins> } // Keep repeatedly fetching document from aggregator. // Check to see if it contains our service within a reasonable amount of time require.NoError(t, WaitForGroups(ctx, client, basicTestGroupWithFixup)) <ins> require.NoError(t, WaitForRootPaths(t, ctx, client, sets.New("/apis/"+basicTestGroup.Name), nil)) // Unregister and ensure the group gets dropped from root paths for _, groupVersion := range groupVersions { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) } require.NoError(t, WaitForRootPaths(t, ctx, client, nil, sets.New("/apis/"+basicTestGroup.Name))) </ins> } func runTestCases(t *testing.T, cases []testCase) {

Related discussion:
text: reported that conformance testing with the latest version of sonobuoy against a version 1.29.0-alpha.3 cluster fails on the support the 1.17 Sample API Server using the current Aggregator test: , related to I found the reason why the test is working fine in Kubernetes CI jobs but fails with sonobuoy is because the service account used by sonobuoy "sonobuoy-serviceaccount" doesn't have read permission of the non-resource url "/", which is used by the test to confirm the removal of group path: The permission of "sonobuoy-serviceaccount" is as below: There may be two ways to fix it: the RBAC rule of sonobuoy to include "get" of "/". However, could other test runners encounter this issue? Is there a doc listing the required permissions for running e2e tests? the e2e to skip checking the removal of the group path if the service account doesn't have permission to access the root path. Which way do we usually prefer? Running e2e with sonobuoy should succeed. a v1.29.0-alpha.3 cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
implemented this solution.
/sig api-machinery
thank you for opening this issue and working on a solution. I'm probably not the right person to decide on which solution is preferred. From my perspective, either solution is fine.
Note: Sonobuoy is not owned by any Kubernetes SIG. At most it is between SIG Architecture and the CNCF to choose whether to recommend it for running tests. SIG Testing owns the e2e image that sonobuoy can be used to schedule but not sonobuoy. IMHO this is a bug in running the tests via a pod, which is not how we develop the tests (they're run outside the cluster, by a cluster admin account typically). I would say 1) is the correct answer. xref: /sig architecture /sig testing /triage accepted cc
thanks for your comment. I guess 1) means non of the existing sonobuoy versions can pass K8s conformance test against a cluster 1.29.0-alpha3, as the RBAC rule is hardcoded.
Code change:
} require.NoError(t, registerAPIService(ctx, client, groupVersion, service)) <del> defer func() { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) }() </del> <ins> groupVersions = append(groupVersions, groupVersion) </ins> } // Keep repeatedly fetching document from aggregator. // Check to see if it contains our service within a reasonable amount of time require.NoError(t, WaitForGroups(ctx, client, basicTestGroupWithFixup)) <ins> require.NoError(t, WaitForRootPaths(t, ctx, client, sets.New("/apis/"+basicTestGroup.Name), nil)) // Unregister and ensure the group gets dropped from root paths for _, groupVersion := range groupVersions { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) } require.NoError(t, WaitForRootPaths(t, ctx, client, nil, sets.New("/apis/"+basicTestGroup.Name))) </ins> } func runTestCases(t *testing.T, cases []testCase) {

Related discussion:
text: I will file an issue to sonobuoy first.
question here is what are the set of client permissions required to run the conformance suite? perhaps a level of access or a more granular recommendation should be documented. what is your take on the root path requirement as approver for: EDIT: slack thread:
I agree, we already require e.g. privileged pods for a long time in conformance tests and we should make it clear that we expect a highly privileged client.
+1 for what is saying, that we should expect that the user running the tests has required permissions. These tests are not built for being run unprivileged.
i wanted to PR the admin note in our conformance docs under sig-arch, but then i saw this: so broke this guarantee. sounds like SIG Arch needs to comment.
This statement surprises me. Root discovery is one of the least-privileged permissions you can have on a cluster. For years this was even open for unauthorized users on the internet (if your api endpoint has been exposed). In other words, just add the permission. While thinking about it, it is surprising that the user in question does not have this access already. Don't we allow that for every authorized user?
In contrast, I hope that lots of privileged operations are tested by conformance today. If we don't test things like admission webhooks for example, we have much bigger problems. And admission webhooks are basically equivalent to cluster-admin.
that is outdated, we use hostNetwork pods one some of the tests IIRC
looks like we don't. before this ticket, i was under the impression that the conformance suite already requires admin and that all testers (like Sonobuoy) use admin level credentials. it is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions) looks like we may have to change that sentence completely.
Would have expected it here: The PR in question explicitly fixes the result returned by . If that is part of conformance (IMO it should), every conformance test needs access it obviously. Whether we require admin, or we add it manually, either would work.
Code change:
} require.NoError(t, registerAPIService(ctx, client, groupVersion, service)) <del> defer func() { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) }() </del> <ins> groupVersions = append(groupVersions, groupVersion) </ins> } // Keep repeatedly fetching document from aggregator. // Check to see if it contains our service within a reasonable amount of time require.NoError(t, WaitForGroups(ctx, client, basicTestGroupWithFixup)) <ins> require.NoError(t, WaitForRootPaths(t, ctx, client, sets.New("/apis/"+basicTestGroup.Name), nil)) // Unregister and ensure the group gets dropped from root paths for _, groupVersion := range groupVersions { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) } require.NoError(t, WaitForRootPaths(t, ctx, client, nil, sets.New("/apis/"+basicTestGroup.Name))) </ins> } func runTestCases(t *testing.T, cases []testCase) {

Related discussion:
text: adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.
Code change:
MINION_TAG="${INSTANCE_PREFIX}-minion" MINION_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_MINIONS}})) MINION_IP_RANGES=($(eval echo "10.245.{1..${NUM_MINIONS}}.0/24")) <del> MINION_SCOPES="" </del> <ins> MINION_SCOPES="storage-ro,compute-rw" </ins> # Increase the sleep interval value if concerned about API rate limits. 3, in seconds, is the default. POLL_SLEEP_INTERVAL=3 PORTAL_NET="10.0.0.0/16"

Related discussion:
text: 07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ]
Code change:
MINION_TAG="${INSTANCE_PREFIX}-minion" MINION_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_MINIONS}})) MINION_IP_RANGES=($(eval echo "10.245.{1..${NUM_MINIONS}}.0/24")) <del> MINION_SCOPES="" </del> <ins> MINION_SCOPES="storage-ro,compute-rw" </ins> # Increase the sleep interval value if concerned about API rate limits. 3, in seconds, is the default. POLL_SLEEP_INTERVAL=3 PORTAL_NET="10.0.0.0/16"

Related discussion:
text: Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ]
Code change:
MINION_TAG="${INSTANCE_PREFIX}-minion" MINION_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_MINIONS}})) MINION_IP_RANGES=($(eval echo "10.245.{1..${NUM_MINIONS}}.0/24")) <del> MINION_SCOPES="" </del> <ins> MINION_SCOPES="storage-ro,compute-rw" </ins> # Increase the sleep interval value if concerned about API rate limits. 3, in seconds, is the default. POLL_SLEEP_INTERVAL=3 PORTAL_NET="10.0.0.0/16"

Related discussion:
text: Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]
Code change:
} } <ins> func printStatusAndLogsForNotReadyPods(c *client.Client, oldPods, newPods []*api.Pod) { printFn := func(id, log string, err error, previous bool) { prefix := "Retrieving log for container" if previous { prefix = "Retrieving log for the last terminated container" } if err != nil { Logf("%s %s, err: %v:n%sn", prefix, id, log) } else { Logf("%s %s:n%sn", prefix, id, log) } } for _, oldPod := range oldPods { for _, p := range newPods { if p.Namespace != oldPod.Namespace || p.Name != oldPod.Name { continue } if ok, _ := podRunningReady(p); !ok { Logf("Status for not ready pod %s/%s: %+v", p.Namespace, p.Name, p.Status) // Print the log of the containers if pod is not running and ready. for _, container := range p.Status.ContainerStatuses { cIdentifer := fmt.Sprintf("%s/%s/%s", p.Namespace, p.Name, container.Name) log, err := getPodLogs(c, p.Namespace, p.Name, container.Name) printFn(cIdentifer, log, err, false) // Get log from the previous container. if container.RestartCount > 0 { printFn(cIdentifer, log, err, true) } } } break } } } </ins> // rebootNode takes node name on provider through the following steps using c: //  - ensures the node is ready //  - ensures all pods on the node are running and ready

Related discussion:
text: We are running heapster, dns server, kube-proxy etc. in kube-system namespace. Some e2e tests try to make sure all those kube-system pods are at ready state before firing the real tests, others just simply ingore them. But both cases might cause e2e flakiness when some of those kube-system containers are in crashloop, for example, for heapster crashloop, for influxdb crashloop, etc. When such cases happen, the developer doesn't have much information on why since the nodes might be teared down. We should watch those kube-system pods, and onContainerFailure, we should collect the related logs. One simple way is using kubectl logs <Pod<Container--previous cc/
is this related enough to and that you can bundle it in?
The logs from the pods should be retrieved via the api server. This is different from the kubelet logs. I'll upload a PR for this.
Great job It will be very helpful for debugging.