jiebi/CodeConvo huggingface dataset:
Code change:
hostNetwork: true containers: - name: fluentd-gcp <del> image: gcr.io/google-containers/fluentd-gcp:2.0.6 </del> <ins> image: gcr.io/google-containers/fluentd-gcp:2.0.5 </ins> # If fluentd consumes its own logs, the following situation may happen: # fluentd fails to send a chunk to the server => writes it to the log => # tries to send this message to the server => fails to send a chunk and so on.

Related discussion:
text: https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Most likely caused by the fluentd update /cc
The gce-gpu tests are failing for the same reason.
Ditto for pull-federation and federation-ci. pull-federation will need a cluster rebuild once the fix is incorporated upstream. /cc csbell
well that's exciting. cc
to revert fluentd to 2.0.5.
I don't understand what's special about these clusters, though. The PR passed presubmit tests a number of times.
none of the presubmit tests run COS images. I think that's the problem. From the error message, it seems like copying libsystemd is not enough, and more libraries need to be copied (e.g., libcap). Will need to build a new fluentd image to fix this.
yeah, I think you're right. I started a cluster myself and the pod is failing on the master (COS) but not on the nodes (CVM). curious that this broke between fluentd 2.0.5 and 2.0.6, which has no code changes and only updates the base image... I guess something materially changed in the base image.
Code change:
} err := clientretry.RetryOnConflict(UpdateNodeSpecBackoff, func() error { <ins> // TODO(wlan0): Move this logic to the route controller using the node taint instead of condition // Since there are node taints, do we still need this? // This condition marks the node as unusable until routes are initialized in the cloud provider if cnc.cloud.ProviderName() == "gce" { if err := nodeutil.SetNodeCondition(cnc.kubeClient, types.NodeName(node.Name), v1.NodeCondition{ Type:               v1.NodeNetworkUnavailable, Status:             v1.ConditionTrue, Reason:             "NoRouteCreated", Message:            "Node created without a route", LastTransitionTime: metav1.Now(), }); err != nil { return err } } </ins> curNode, err := cnc.kubeClient.CoreV1().Nodes().Get(node.Name, metav1.GetOptions{}) if err != nil { return err

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In , the controller add NodeNetworkUnavailable to the node conditions, and use to update it. But node status can not be updated by , it doesn't work because replace the node status with old status. To fix this bug, we should use to update node status. /sig cloud-provider /sig cluster-lifecycle /area cloudprovider What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I agree this is a bug. As a matter of interest though, how are you exercising this bug? It seems specific to cloud controller on gce. However that controller isn't actually enabled on gce. It would be good for me to understand your use case, to make sure we don't break it later.
I built my kubernetes cluster with external cloudprovider, but I found if I stop the route controller, pod will be scheduled on node even if the route is not for that node. I read the code and found that the node condition would not be add when using other cloudproviders. At first, I change "gce" in that line code to the name of my cloudprovider, but the problem also exists. Then I found that didn't take effect actually.
Code change:
REGISTRY?=staging-k8s.gcr.io IMAGE?=debian-hyperkube-base <del> TAG=0.8 </del> <ins> TAG=0.9 </ins> ARCH?=amd64 CACHEBUST?=1

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Upgraded to Kuberenetes 1.9 and now IPVS mode in kube-proxy stopped working due to What you expected to happen: Normal startup as it's been working flawless in 1.8.5 How to reproduce it (as minimally and precisely as possible): Use kube-proxy with IPVS service mode on 1.9.0 hyperkube image from GCR Anything else we need to know?: If i exec into the running kube-proxy pod and do then restart the container kube-proxy starts in ipvs mode fine Environment: Kubernetes version (use ):Cloud provider or hardware configuration: On-prem OS (e.g. from /etc/os-release): CoreOS Container Linux by CoreOS stable (1576.4.0) Kernel (e.g. ): 4.13.16-coreos-r2 Install tools: Own Others:
/sig network
According to jell in #sig-network these images are to be considered experimental and not used in production and might be missing binarys? , could someone shed some light on this
To be more precise: hyperkube binary can hide some options of underlying applications (e.g. of fix for that: ) and whole docker image of hyperkube - as this issue shows, can miss some useful binaries. That's why I named it as a tool to do experiments, while for real deployments probably a better choice would be to use underlying applications directly using own docker image, carefully tuned.
Is this connected to and by chance the changes hasn't propagated down to the public images yet?
image is based on (which is based on ) (and ) has a separate image based on But as has included support, IMO also should be extended by command, after this line.
Seems this issue already got fixed? Can close now?
Where do you get the info it's fixed, all my kube-proxy instances is still failbacking to iptables due to missing ipset that was present in 1.8.5
It was fixed by merging So, as it's in master, the fix will land in next release.
Ok, so it's fixed in the next version. I'll just switch away from Hyperkube then. Thanks
FYI: I've rebuilt hyperkube image for CoreOS with installed. Just for testing purposes. Everything works for me with it.
Code change:
REGISTRY?=staging-k8s.gcr.io IMAGE?=debian-hyperkube-base <del> TAG=0.8 </del> <ins> TAG=0.9 </ins> ARCH?=amd64 CACHEBUST?=1

Related discussion:
text: Waiting for the next release. :relaxed:
So does have ipset. So please use that. However does not have ipset (see below). The commit above was merged well before we shipped v1.9.0, But hyperkube image does not use the debian-iptables, it uses debian-hyperkube-base image if you see Anyone want to log a PR for hyperkube image to have ipset?
Can we re-open this, only fixes this on debian, not amd64!
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
will need to reopen, then :)
Code change:
go_library( name = "go_default_library", <del> srcs = ["validation.go"], </del> <ins> srcs = [ "validation.go", ] + select({ "@io_bazel_rules_go//go/platform:android": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:darwin": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:dragonfly": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:freebsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:linux": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:nacl": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:netbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:openbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:plan9": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:solaris": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:windows": [ "validation_windows.go", ], "//conditions:default": [], }), </ins> importpath = "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/validation", deps = [ "//pkg/features:go_default_library",

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.
Code change:
go_library( name = "go_default_library", <del> srcs = ["validation.go"], </del> <ins> srcs = [ "validation.go", ] + select({ "@io_bazel_rules_go//go/platform:android": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:darwin": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:dragonfly": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:freebsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:linux": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:nacl": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:netbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:openbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:plan9": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:solaris": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:windows": [ "validation_windows.go", ], "//conditions:default": [], }), </ins> importpath = "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/validation", deps = [ "//pkg/features:go_default_library",

Related discussion:
text: Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!
Code change:
return storage.NewInternalErrorf("unable to transform key %q: %v", kv.Key, err) } <ins> // Check if the request has already timed out before decode object select { case <-ctx.Done(): // parent context is canceled or timed out, no point in continuing return storage.NewTimeoutError(string(kv.Key), "request did not complete within requested timeout") default: } </ins> if err := appendListItem(v, data, uint64(kv.ModRevision), opts.Predicate, s.codec, s.versioner, newItemFunc); err != nil { recordDecodeError(s.groupResourceString, string(kv.Key)) return err

Related discussion:
text: It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.
Code change:
return storage.NewInternalErrorf("unable to transform key %q: %v", kv.Key, err) } <ins> // Check if the request has already timed out before decode object select { case <-ctx.Done(): // parent context is canceled or timed out, no point in continuing return storage.NewTimeoutError(string(kv.Key), "request did not complete within requested timeout") default: } </ins> if err := appendListItem(v, data, uint64(kv.ModRevision), opts.Predicate, s.codec, s.versioner, newItemFunc); err != nil { recordDecodeError(s.groupResourceString, string(kv.Key)) return err

Related discussion:
text: We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v2": {}, </del> <ins> "/apis/apiservicegroup/v1/path2": {}, </ins> }, }, },

Related discussion:
text: When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.
Code change:
SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v2": {}, </del> <ins> "/apis/apiservicegroup/v1/path2": {}, </ins> }, }, },

Related discussion:
text: Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.