jiebi/CodeConvo huggingface dataset:
Code change:
rotateCertificates: true runtimeRequestTimeout: "${RUNTIME_REQUEST_TIMEOUT}" staticPodPath: "${POD_MANIFEST_PATH}" <ins> resolvConf: "${KUBELET_RESOLV_CONF}" </ins> EOF { # authentication

Related discussion:
text: changed from "kube-dns" to "core-dns". "core-dns" is failing for me on Debian Buster: works when I use . "core-dns" should work. Perhaps related to this? Kubernetes version (use ): current master () OS (e.g: ): Debian Buster Kernel (e.g. ): 5.5.0-0.bpo.2-amd64 /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
provides a description of the error, causes, and workarounds. Typically this is due to a local dns resolver installed on the host. Solution is to configure kubelet to use a different (more detail in link).
That mirrors what I found through the issue. Needs to be fixed in then.
/sig testing /cc
/cc
I think we have to parameterize this
I think the safer way is to configure kublet to send the correct to all “default” dns policy pods, not just fix in coredns.
Doh, right :)
Code change:
logOptions.SinceSeconds = &sec } <del> if len(o.Selector) > 0 && o.Tail != -1 { </del> <ins> if len(o.Selector) > 0 && o.Tail == -1 { </ins> logOptions.TailLines = &selectorTail } else if o.Tail != -1 { logOptions.TailLines = &o.Tail

Related discussion:
text: What happened: I do . I see the last 10 lines of output of each pod matching . What you expected to happen: I was expecting to see only the last line of output of each pod. How to reproduce it (as minimally and precisely as possible): The image prints the date every second. Normally we should just see two lines of output; instead we see 20. Anything else we need to know?: This seems to be a regression in Kubernetes 1.12, as this worked correctly from 1.8 to 1.11. (I observed this problem starting with k8s 1.12.0 and it's still present in 1.12.2) Environment: Kubernetes version (use ): 1.12.2 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu Kernel (e.g. ): 4.4.0 Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
It looks like the issue was introduced by commit , specifically . If I understand that code correctly, it forces the number of lines to (10) even if is set. Since that seems to be in : /sig cli
/assign
didn't that commit just move the code from ?
Not quite; the old code was: The new code is: I don't know why it was changed, but the new logic reads as "if the user specified a selector and a number of lines different from -1, then set the number of lines to 10".
Code change:
if isHTEnabled() { cpuList = cpuset.MustParse(getCPUSiblingList(0)).ToSlice() cpu1 = cpuList[1] <ins> } else if isMultiNUMA() { cpuList = cpuset.MustParse(getCoreSiblingList(0)).ToSlice() cpu1 = cpuList[1] </ins> } expAllowedCPUsListRegex = fmt.Sprintf("^%dn$", cpu1) err = f.PodClient().MatchContainerOutput(pod.Name, pod.Spec.Containers[0].Name, expAllowedCPUsListRegex)

Related discussion:
text: Which jobs are failing: N/A - AFAICT CI jobs works fine Which test(s) are failing: Since when has it been failing: no historical record on CI Testgrid link: N/A Reason for failure: I believe the tests make too many assumptions around the hardware, and the code is actually OK. to be confirmed. Anything else we need to know:
/sig node /sig testing
also affects the recently topology manager e2e tests. My PR uses a different approach to check resource alignment so is not affected. /cc
I'll work on a PR to fix this issue ASAP
Thanks for opening this issue. We knew about this, just had not opened an issue to track it, so thanks for doing that! Deepthi has been looing at CPU Manager tests, not sure if she has some PR in progress or not. I'll make sure to forward this to her (can find her user id here ....) in case she has something in progress
Code change:
for i := 0; i < AllowedInflightRequestsNo; i++ { // These should hang waiting on block... go func() { <del> expectHTTP(server.URL+"/foo/bar/watch", http.StatusOK, t) </del> <ins> if err := expectHTTP(server.URL+"/foo/bar/watch", http.StatusOK); err != nil { t.Error(err) } responses.Done() </ins> }() } // Check that sever is not saturated by not-accounted calls <del> expectHTTP(server.URL+"/dontwait", http.StatusOK, t) oneAccountedFinished := sync.WaitGroup{} oneAccountedFinished.Add(1) var once sync.Once </del> <ins> if err := expectHTTP(server.URL+"/dontwait", http.StatusOK); err != nil { t.Error(err) } </ins> // These should hang and be accounted, i.e. saturate the server for i := 0; i < AllowedInflightRequestsNo; i++ { // These should hang waiting on block... go func() { <del> expectHTTP(server.URL, http.StatusOK, t) once.Do(oneAccountedFinished.Done) </del> <ins> if err := expectHTTP(server.URL, http.StatusOK); err != nil { t.Error(err) } responses.Done() </ins> }() } // We wait for all calls to be received by the server

Related discussion:
text: Have only seen this once, personally.
This flaked in Jenkins as well. [job/kubernetes-test-go/5351]
can you take a look at this? I think has a lot already :)
Code change:
if [[ -z "${is_push}" ]]; then echo "== kube-up node config starting ==" set-broken-motd <ins> ensure-basic-networking </ins> ensure-install-dir set-kube-env [[ "${KUBERNETES_MASTER}" == "true" ]] && mount-master-pd

Related discussion:
text: Node 2's kubelet logs: controller-manager side: I have a cluster to hand off if someone wants to debug. It's not a transient state: node 2 just failed to ever post a valid status. cc cc (who looked at it briefly with me)
I will take a look soon.
Ok, why kubelet on node 2 talks to ip address of master instead of hostname? root ps -ef | grep kubelet root 1 3 17:35 ? 00:10:45 /usr/local/bin/kubelet --apiservers=https://10.240.49.47 --config=/etc/kubernetes/manifests --allowprivileged=False --v=2 --clusterdns=10.51.240.10 -- It looks like a cluster up issue to me, not a node problem.
It looks like failed to translate the master name into the fqdn (which is done ).
Fascinating. I think we're seeing instance-DNS flakiness, because that name is supposed to get converted ASAP. But what's weirder is ... that's not even the master:
The is definitely right:
Okay, this is messed up. Here's node 1, which is happy: Here's our unhappy node .... that decided to bring itself as a master:
haha... I checked kube-env and did suspect unhappy node treat itself as master node based on those variables. But I didn't check salt state :-)
I ... just can't get there. By the time we get here: , node-2 has set: My best guess is that flaked here: . I think we're seeing a systemic issue where we're racing with GCE network configuration.
I'm going to work up a PR that hopefully improves the reliability of this section to some extent.
<a href=""<img src="" title="made at "/</a
My working theory is that we raced with the configuration done normally by the GCE NetworkManager, leaving us in a bogus state when we made that check.
Why are we relying on inferring the master/node status from the hostname. It should be obvious from the variables set in the kube-env if you are the master or not. Masters don't get passed so why not just check that? Making sure DNS is working is probably also good for reliability...
On GKE, we actually pass the same kube-env on both sides for simplicity.
Code change:
if [[ -z "${is_push}" ]]; then echo "== kube-up node config starting ==" set-broken-motd <ins> ensure-basic-networking </ins> ensure-install-dir set-kube-env [[ "${KUBERNETES_MASTER}" == "true" ]] && mount-master-pd

Related discussion:
text: We can obviously change that, but I was going for a change that didn't involve a deployment change on our side, too.
Code change:
// container.  The container's absolute path refers to its hierarchy in the // cgroup file system. e.g. The root container, which represents the whole // machine, has path "/"; all docker containers have path "/docker/<docker id>" <del> func (kl *Kubelet) statsFromContainerPath(containerPath string, req *info.ContainerInfoRequest) (*info.ContainerInfo, error) { cinfo, err := kl.cadvisorClient.ContainerInfo(containerPath, getCadvisorContainerInfoRequest(req)) </del> <ins> func (kl *Kubelet) statsFromContainerPath(cc CadvisorInterface, containerPath string, req *info.ContainerInfoRequest) (*info.ContainerInfo, error) { cinfo, err := cc.ContainerInfo(containerPath, getCadvisorContainerInfoRequest(req)) </ins> if err != nil { return nil, err }

Related discussion:
text: So today, cadvisor became wedged and just stopped paying attention to its socket. Not kubelet's fault. But when I would launch all I would get would be: It took me a bit of trolling through netstat and lsof to figure out that the problem was cadvisor being completely locked up. kill -9 cadvisor and everything started working well. I wonder if we should emit more messages during startup about what operation we are about to attempt, not just the results of operations, as I was blaming docker for a while....
kubelet shouldn't block on this. I'll take a look.
Any ideas what was locking cAdvisor? and was it just not responding to requests?
I don't have any idea why it was locked. If I find another one locked I'll hang onto it to debug. Journal had: obviously the 'kill' came from me. It would not respond to systemctl cadvisor stop. netstat looking at port 4194 showed me At which point I realized cadvisor wasn't handling its sockets right since so many long dead kubelet sockets hadn't been cleaned up, so I kill -9'd it.
I cut some interesting info from the cadvisor journal log (trying to cut hostnames) You can see the last message about failed to update stats for container ... happened Oct 08 01:06:01. It was then silent until I killed it Oct 08 19:20:13, even though I'm pretty sure dnf was still being launched every hour on the hour. So it was wedged for about 18 hours.
What kernel is this on? We've seen there is a kernel bug fixed in 3.12+ that sometimes hangs us forever on exit. We're not quite sure how it gets triggered...
16.3
Code change:
exit 1 fi <ins> # mark all ./test/integration/* targets as integration # see comment above re: buildozer exit codes buildozer -quiet 'add tags integration' '//test/integration/...:%go_test' && ret=$? || ret=$? if [[ $ret != 0 && $ret != 3 ]]; then exit 1 fi </ins> # we need to set this because gazelle doesn't support pkg-config, which would set this link option # see comment above re: buildozer exit codes buildozer -quiet 'set clinkopts select({"@io_bazel_rules_go//go/platform:linux":["-lseccomp",],"//conditions:default":[],})' //vendor/github.com/seccomp/libseccomp-golang:go_default_library && ret=$? || ret=$?

Related discussion:
text: What happened: when running integration tags were missing on files What you expected to happen: add the needed tags on test/integration files How to reproduce it (as minimally and precisely as possible): add an integration test package under cc
/sig testing /priority important-soon /area build-release
/cc
if we want to assume that all tests in a certain directory are integration tests this is very easy to automate, here's an example that we do now to mark go tests and binaries as
Is the assumption that all tests in should be tagged as integration? I'd be happy to add this to the script, but would prefer to get in without conflicts first
/assign /lifecycle active
Code change:
wantStatus: framework.NewStatus(framework.Unschedulable, getErrReason(v1.ResourceEphemeralStorage)), }, { <ins> pod: newResourceInitPod(newResourcePod(framework.Resource{EphemeralStorage: 25}), framework.Resource{EphemeralStorage: 25}), nodeInfo: framework.NewNodeInfo( newResourcePod(framework.Resource{MilliCPU: 2, Memory: 2})), name: "ephemeral local storage request is ignored due to disabled feature gate", features: map[featuregate.Feature]bool{ features.LocalStorageCapacityIsolation: false, }, }, { </ins> pod: newResourcePod(framework.Resource{EphemeralStorage: 10}), nodeInfo: framework.NewNodeInfo( newResourcePod(framework.Resource{MilliCPU: 2, Memory: 2})),

Related discussion:
text: What happened: In a v1.18 cluster with the LocalStorageCapacityIsolation feature gate disabled setting an ephemeral-storage request or limit causes the pod to be unschedulable: If the ephemeral-storage request is removed the pod is scheduled. This behavior exists in v1.19 and a recent v1.20 build. What you expected to happen: In previous releases such pods are scheduled. I expect that any ephemeral-storage limit is not enforced. How to reproduce it (as minimally and precisely as possible): Disable the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler. Create a pod like this: Anything else we need to know?: The problem was originally seen after moving from IBM Cloud Red Hat OpenShift v4.4 to v4.5. Pods that worked in 4.4 (Kubernetes v1.17) remained in a Pending state in OpenShift v4.5 (Kubernetes 1.18). We discovered that the IBM Cloud OpenShift service has the LocalStorageCapacityIsolation feature-gate disabled while Red Hat has enabled the feature. We then replicated the behavior with a plain Kubernetes cluster. Environment: Kubernetes version (use ): v1.18.10, v1.19.3, v1.20.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I can confirm it's a bug. Will work on it.
Code change:
wantStatus: framework.NewStatus(framework.Unschedulable, getErrReason(v1.ResourceEphemeralStorage)), }, { <ins> pod: newResourceInitPod(newResourcePod(framework.Resource{EphemeralStorage: 25}), framework.Resource{EphemeralStorage: 25}), nodeInfo: framework.NewNodeInfo( newResourcePod(framework.Resource{MilliCPU: 2, Memory: 2})), name: "ephemeral local storage request is ignored due to disabled feature gate", features: map[featuregate.Feature]bool{ features.LocalStorageCapacityIsolation: false, }, }, { </ins> pod: newResourcePod(framework.Resource{EphemeralStorage: 10}), nodeInfo: framework.NewNodeInfo( newResourcePod(framework.Resource{MilliCPU: 2, Memory: 2})),

Related discussion:
text: /assign
Clarifying the recreate steps: I disabled the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler since this behavior seems to be within the scheduler.
Thanks for confirming!
This bug has been fixed in master branch, and will be backported to [1.18] and [1.19] soon. [1.18]: [1.19]:
Code change:
// hoping external CCM or admin can set it. Returning // default values from here will mean, no one can // override them. <del> glog.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) return volumeLimits, nil </del> <ins> return nil, fmt.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) </ins> } instances, ok := az.Instances()

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: What happened: After upgrade of Kubernetes 1.11.3 to 1.12.1 via kubeadm in bare-metal setup I see these errors in kubelet logs: What you expected to happen: Since this is a bare-metal setup I expect no messages concerning Azure cloud How to reproduce it (as minimally and precisely as possible): Do an upgrade from 1.11.3 to 1.12.1 Anything else we need to know?: kubelet-1.12.1- was downloaded from Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Oracle Linux Server 7.4 Kernel (e.g. ): 62 Install tools: Others:
Lines from kubelet log which say that no cloud provider was specified (that is a bare-metal setup): ..
/sig azure
have the same issue related to the azure.. also a bare-metal cluster.. these messages are flooding my centralized logging elasticsearch (though they aren't really problematic more than being an annoyance). i also upgraded from 1.11.3 to 1.12.1 on a 3-master HA set up.
Code change:
// hoping external CCM or admin can set it. Returning // default values from here will mean, no one can // override them. <del> glog.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) return volumeLimits, nil </del> <ins> return nil, fmt.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) </ins> } instances, ok := az.Instances()

Related discussion:
text: i also noted (perhaps worthy of a separate issue) that the two of my three control-planes that are showing the azure error also previously had been having cert authority errors from the api-server container, when having used kubeadm upgrade.. (on the second two control plane systems, with the issue) (and on the first control without the issue) (and i needed to kubeadm reset and then kubeadm init all over again with 1.12.1 on the 2nd two control-plane nodes).. the first control-plane node that didn't need to do a reset/init didn't have the azure errors (nor the cert errors). so i kind of suspect it might not be an kubeadm upgrade issue but a general issue going from 1.11.3 to 1.12.1 ?
Same problems +1
Same problem on brand new bm 1.12.1 cluster built with kubeadm and latest 1alpha3 config based off of customized kubeadm config print-default example
/assign
Same problems +1
Oct 9 11:11:32 localhost kubelet: E1009 11:11:32. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:11:42 localhost kubelet: E1009 11:11:42. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:11:52 localhost kubelet: E1009 11:11:52. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:02 localhost kubelet: E1009 11:12:02. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:12 localhost kubelet: E1009 11:12:12. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:23 localhost kubelet: E1009 11:12:23. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:33 localhost kubelet: E1009 11:12:33.
Code change:
// hoping external CCM or admin can set it. Returning // default values from here will mean, no one can // override them. <del> glog.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) return volumeLimits, nil </del> <ins> return nil, fmt.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) </ins> } instances, ok := az.Instances()

Related discussion:
text: 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:43 localhost kubelet: E1009 11:12:43. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:53 localhost kubelet: E1009 11:12:53. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:13:03 localhost kubelet: E1009 11:13:03. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1
how often did you hit this error log: I am planning to make this log as , is that ok for you? Usually V(2) is the default info(see ) log for k8s on azure I am not sure what the default info log by kubeadmin installation.
It is logged every 10 seconds. I disagree. At least previous incarnation of GetVolumeLimits in version 1.11.3 : would have logged an error "No cloudprovider present". Which I didn't see in kubelet log. In fact, I didn't see any message in kubelet logs with 'azure' substring at all. I think it means that AzureDataDisk plugin was not instantiated at all in 1.11.3. So the question is: why azuredd is instantiated now in 1.12.1 in bare-metal setup (that is without cloud provider)? Do you mind checking this idea? (Besides your method of lowering message priority won't work as I said this is a bare-metal setup where kubelet systemd unit file has no --v option at all )
That's because feature reaches beta in v1.12, in v1.11.3, even the code is there, it won't be invoked since feature is still alpha in v1.11
And in v1.12, aws, gce, and azure disk plugins are all loaded, take aws ebs for example, it will return error directly: And logging as V(4): I suppose V(4) logs won't display in logs. Let me make the same change as aws and gce.
also tries to address this error.
Code change:
// hoping external CCM or admin can set it. Returning // default values from here will mean, no one can // override them. <del> glog.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) return volumeLimits, nil </del> <ins> return nil, fmt.Errorf("failed to get azure cloud in GetVolumeLimits, plugin.host: %s", plugin.host.GetHostName()) </ins> } instances, ok := az.Instances()

Related discussion:
text: The same mistake, I also encountered NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 172.16.16.13 Ready <none5m27s v1.12.1 172.16.16.13 <noneCentOS Linux 7 (Core) 4.18.12- docker://18.6.0 failed to get azure cloud in GetVolumeLimits, 172.16.16.13 failed to get azure cloud in GetVolumeLimits, 172.16.16.13 failed to get azure cloud in GetVolumeLimits, 172.16.16.13
I have cherry picked the fix to v1.12
I understand that the feature Node-specific Volume Limits has now reached beta in 1.12, but for those of us running Kubernetes on-premises, this log error is pretty useless. In fact, I would even want to disable this plugin alltogether, since it will never be used on-prem. Is there a way to stop the plugin from loading, or in the very least make sure the log does not appear? Right now, it is polluting the system log file quite a lot.
I second that. That was the key message of my bug report. That is we must not even instantiate cloud-specific plugins if no cloud provider is configured.
I'm having exactly the same problem, doing an installation from scratch on DigitalOcean
+1
+1
Hi I have the same issue. It would be really nice to know how to disable this error message that floods the logs for no reason.
For those using logstash it was sufficient to just temporarily add this to your syslog filter: if [message] =~ "" { drop { } }
After a fresh installation of Raspbian and kubernetes, same issue is occurring in my 4 node cluster.