jiebi/CodeConvo huggingface dataset:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pod was deleted successfully but there is still a lock on the ceph image. I find two logs,one for that lock not removed and another for that lock removed. I find that there are little difference between them. the log that lock not removed: and the log that lock removed: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.6.7 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): centos7 Kernel (e.g. ): 3.10.0-327.el7.x86_64 Install tools: Others:
/sig storage
Error 16 is EBUSY. similar to
were you running kubelet with cadvisor on?
/assign
yes, every kubelet run with cadvisor,and i know what code 16 represent for,but i still don't know the reason, is this a bug? How can i avoid this kind of situation from happening,thank you
It is probably related to
We see some other case may cause , kubelet run with nodeexporter and nodeexporter is deployed by k8s daemonset with -v /:/rootfs:ro (mount kubelet host rootfs into nodeexporter container). rbd device is mounted before nodeexporter started, because k8s 1.6.7 not support mount propagation, when the rbd plugin unmount /dev/rbd0 on the host namespace, the mount info will still leave over in node_exporter container namespace, may cause . hope this would help
thanks, does the problem persist if node_exporter stops collecting filesystem metrics?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .
Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I'm experiencing this issue. From what I can read above, if the rbd mount happens before the node_exporter starts; we'll always experience status 16?
We're getting this issue. This is because the mounted volume is mounted on Cadvisor container too. So the /dev/rbdX file is still in use. But still didn't find any solution to fix.
rbd volume will stuck when failed as will always returen
My solution: container use a volumeMount. make a script file to be executed every a minute:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, IPVS proxier does not run any e2e test cases. What you expected to happen: I think it's important to create test suite for IPVS in upstream CI which will help a lot in moving IPVS existing alpha. Anything else we need to know?: Let's work together. /sig network
Self /assign
There are to ways to do this work: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.
When the master dials nodes, it is mutating the tlsConfig held by the master kubelet client, to add a server name. That means if two different nodes are dialed from a master started with , x509 verification will fail for the second one. Introduced in
this is a regression from 1.3 and breaks pod logs/exec/attach and node proxying on installations that use
We probably need to review that PR and look to see if any other deliberate copies were flagged incorrectly by govet.
Installations that depend on secure trust from externally provided kubelet server certs are the highest security rings. If we don't fix this we need to tell anyone using this flag not to upgrade until we release 1.4.1.
the rest of the changes in the govet cleanup PR looked fine to me. fix for this issue is in
Since we are telling folks they have to upgrade to kubectl 1.4.0 for rollingupdate to work it would definitely be a confusing message to say other folks should not upgrade to 1.4.0 at all.
If the fix is merged into master by this afternoon, lets discuss whether or not we should cherrypick it - e.g. risks involved, how we can manually qa to verify nothing else was broken, etc.
Risk involved: If you are using a secure cluster (defined as ensuring no one can MITM your nodes and masters) and you upgrade to 1.4.0 and it does not include this fix - you are broken.
What is the likelihood of interactions with other code? Is it possible this fix could introduce a breaking change to something else? Has the fix been manually verified? Are we confident that this will fully fixes the issue (vs other edge cases exist for secure clusters that this doesn't catch). Ideally I would like to see this cherrypicked into 1.4 this evening and then cut a new beta that includes it.
None. The single line that is currently broken was changed. The rest is a new function (with no side effects), and tests. No, it restores the original intent of the TLS config copy, and tests cover the cases we are using this with (secure, insecure, custom dialer) Yes. I swept for the same issue elsewhere and didn't find any issues. We test with a fully secured master-kubelet setup in Openshift and got a clean test run with this fix applied.
SGTM. Thanks for fixing this.
What happened: has a typo, (should be ) This also shows up in the rendered documentation. What you expected to happen: Documentation has correct path for readyz endpoint. How to reproduce it (as minimally and precisely as possible): Easiest to see this wrong in the source code, I think. Anything else we need to know?: Environment: Kubernetes version (use ): v1.16 Cloud provider or hardware configuration: not applicable OS (e.g: ): not applicable Kernel (e.g. ): not applicable Install tools: Network plugin and version (if this is a network-related bug): Others: /sig docs /kind bug /priority backlog
pointed out to me that a CEL x-kubernetes-validations rule in a CRD was hitting the estimated cost limit when performing a simple "string has prefix" check on an en enum field. Putting a limit in the OpenAPI of the enum string field fixed the problem, but having to add a limit like this is counter-intuitive and unnecessary, since the allowed enum values already make the max allowed length obvious. The apiserver automatically computes the max length of an enum as the length of the longest allowed enum value. I believe we already refuse evaluate CEL rules on values that fail OpenAPI value validation, since we rely in the value validations to establish all length limits. I haven't tested it locally, but something like this should work (given enough nested arrays): As a workaround, one can manually set on the string enum, e.g.: 25+ All No response No response No response No response
/sig api-machinery /triage accepted /assign
/assign
I wasn't sure if we special-cased the maxLength / maxItems / maxProperties failures, or if all openapi validation had to pass before we started messing with CEL evaluation
it looks like we currently special-case specific openapi errors that short-circuit CEL validation, I'm not sure an enum validation failure would currently prevent evaluation:
The enum err is converted to () error which is not short-circuit here. But for defaulting case, all error would be short-circuit. I would suggest to add error into the blockingErr list to short-circuit CEL validation. Would this make sense?
yeah, that sounds good if we're going to rely on it to catch things that are longer than the maximum length enum value
/kind bug /sig node /area hw-accelerators What happened: In devices are allocated from a function called . This function inspects only container resource limits, ignoring container resource requests. This is a usability problem because it is valid to omit limits for extended resources. If limit is omitted for a device resource, then the scheduler properly places the pod with respect to available node resources. However, the device manager never dispatches to the device plugin for that resource. The result is that the user application cannot access the requested device from inside the container. What you expected to happen: Following the documented behavior of extended resources, I expected that specifying a container resource request for a device would cause the device subsystem to properly configure access for my workload. How to reproduce it (as minimally and precisely as possible): Launch an alpha GKE cluster with GPU support enabled. Attempt to run any program that accesses the GPU from a container where the resource is specified in requests but omitted from limits. Environment: Kubernetes version (use ): v1.8+ Cloud provider or hardware configuration: device plugins enabled, devices available.
Related issue:
cc
/assign
FYI. The said issue has been fixed in . There are several valid cases when requesting extended resources: The user can specify 'limits' only. The API server will automatically set 'requests' to be equal to 'limits'. The user specify 'limits' and 'requests', and they are equal. The user specify 'requests' only. We can either reject the request saying that 'limits' must be set, or implicitly treat 'limits' as equal to 'requests'.
Hi went with to begin since it was ready to go and smaller (easier to revert if necessary.) Hopefully it's easy to rebase your work on top of that. Sorry for the inconvenience.
I feel the documented behavior is incorrect. The original intention was to always require for extended resources until we have better APIs to express bursting scenarios. I vote for fixing the documentation and fixing the validation logic and having kubelet use limits only.
/reopen
Recently I was experimenting with changing for a Golang pre-purposed faux editor when editing resources when I noticed that it couldn't be marshaled. Upon inspecting the output JSON I noticed it had the same comments that the default yaml version has. Valid JSON cannot have comments. Since the comment also gives warning / error information upon saving it must be decided where to put that in the JSON output. Perhaps a new JSON property?
Maybe just not save the comments as part of the json to get rid of the warning, but still should them?
Is this a 1.2 thing? I do not see any errors when running the two commands. I made an edit in one run, and did not make an edit in a separate run. Maybe I don't understand the problem that is seen.
there isn't a kubernetes error that is shown, the issue is that the output JSON that kubernetes generates is actually invalid JSON due to the comments at the top. This caused errors when I was trying to unmarshal that JSON in another project.
Okay. Got it. The editor is being very strict about json having no comments.
It's not only about editor, it's simply part of language specification. Explanation for this: IMO if we want to use format, we should comply with it's standard. Also currently used sign for comments is unusual in javascript world, from which json evolved. In this world comments are marked in "C++ like" style (what in early days of JSON was also supported), i mean using and . So IMO in time of converting into JSON - we should strip such comments (I would prefer that), or they should be in some documented and specific to kubernetes way into JSON object.
Checked in: Not checked in: Can anyone explain why?
ones published outside k/k that are in libraries/modules expected to be usable without running custom code generation steps are checked in
example issue: there were similar issues with other instances
/triage accepted
I guess I am asking why not check in
oh, I'd be in favor of that :)
downstream users very often end up realizing that we warn folks not to use ( ) because they go looking for why this file is not checked in. This paper cut makes people realize that they are doing something that we don't want them to do. (not arguing for or against, just stating where this was effective before)
I think the v0.0.0 local replace directives in accomplish similar things w.r.t. making people go digging about friction using k/k as a library
While running one of my soak tests I noticed the error message: I don't see why I should get this error message. Why should it matter if the namespace already exists? It was created as part of the service creation call just beforehand. And the second attempt to create this pod in exactly the same namespace works. Is this a bug?
Running a second time did not produce this error.
I agree it should not matter. The error is produced there, and I suspect there were multiple concurrent calls being made that try to create a resource in a namespace that had not yet existed. We need to update the code in question to swallow the error if the namespace already exists.
FYI - longer term, I would like to eliminate the use of autoprovision of namespace and require you to first create a namespace manually. what is there now is more to preserve existing behavior prior to namespace as object was introduced.
We allow CRD authors to write . Currently we don't restrict writing validation for , which means people can have . For example, people can have additive name constraints (require CR to be ) with the following schema: This brings us two questions: do we if "metadata" is included in the schema: simplest option is to publish standard metav1.ObjectMeta schema info, and use any CRD-specified "metadata" properties as additive validation (just like in-tree types do, publishing standard ObjectMeta schema info, and implementing type-specific name validation internally) alternatively, we can try merging user-specified metadata schema with the standard one. Technically people should only add extra validation to existing metadata properties (adding new fields to metadata won't work since they will be pruned by built-in validation first), but it's unclear to me how arbitrary people may want the metadata validation to be term question is whether to allow people specifying metadata validation at all, or just allow a subset (like name validation) in CRD v1 /sig api-machinery /area custom-resources cc
What is the difference between these two? I mean the end result is the same right? The only thing that can be is validation. Nothing else works unless you are considering extensions?
The former means we drop/hide user-specified metadata schema when we publish CRD OpenAPI. We still validate user-specified constraints on server side, but client-side validation won't see them. The latter enables client-side user-specified metadata validation.
If it's possible to merge with standard metadata schema info, or at the cost of not publishing the standard schema. My inclination would be to publish the complete standard schema for metadata/apiVersion/kind, which we know is complete and accurate, and use and user-specified schema for those fields as addition validations server side
Need to finish so the client allows you to select an init container, and verify the kubelet works correctly.
(this is a nice intersection of new in 1.3, node and usability, if anyone wants to give it a shot. Pretty sure Clayton is busier than I am and I don't think I'll get to it)
Will be happy to review any changes - should be fairly straightforward.
Yes, init containers are "run to completion" always. , harryz wrote:
IIUC when the init containers of pod have not complete, the pod should be in status, which means will fail, if init containers run to completion, they will exit, still fail. I don't know what you mean exec should work on init containers?
the init container is still up and running even when the pod is in pending, we should be able to exec and debug why it's hung, for example.
add a straightforward exec implementation ptal.
[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The arg large-cluster-size-threshold for kube-controller-manager works as zone size threshold when determining node eviction rate. If there are multiple zones, the description for arg stating "cluster" threshold would be missleading. ARG: Matched arg description Multiple zones cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated [Will update shortly] links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: links to appreciatedlinks to specific failures in spyglass appreciated /sig api-machinary Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/assign /triage accepted
Flaked twice ~2 weeks ago but now looking good: Since it's been relatively quiet, I'm going to go ahead and close this for now. /close
Closing this issue. <detailsIn response to : Flaked twice ~2 weeks ago but now looking good: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign I meet this test failure in And I can reproduce it with local stress testing. The reason is the same with
Reopened this issue. <detailsIn response to : /reopen /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: test-cmd runkubectlrun_tests Since when has it been failing: According to triage, the first failure was 2019-02-21 12:00 UTC Testgrid link: Reason for failure: Unknown Anything else we need to know: This may be a flake? I can't tell yet. The testgrid link is not a continuous failure, but it could be that not enough PR's have whatever change is causing the failure Triage link: It looks like it may be happening in CI as well
/sig cli /priority critical-urgent /milestone v1.14
/remove-kind failing-test /kind flake I can't see it consistently happening in CI, so it must be a flake?
What I got when running locally...
And I see the same thing in the build- for
Trying to track down the first occurrence. I think there was an XML parsing error that was obscuring this that was fixed recently. Backtracking through pull-kubernetes-integration failures and visually inspecting them. - runkubectlruntests - 2019-02-22 15:45 PST - runkubectlruntests - 2019-02-21 01:13 PST - runkubectlrun_tests - 2019-02-20 16:50 PST ... and then I don't see this particular test failing during 02-19 or 02-18
So going by that, let's cast a net in commits that landed just prior to the earliest PR failure, and N hours prior
Suspects - ~only because it touches nope, rolled back to right after this landed, no repro after 30 attempts - ~enables server side apply <-- this is my guess~ nope, rolled back to right after this landed, no repro after 30 attempts - reduce lock contention in watchcache
the test might be racy in the first place, maybe enabling the feature changes the time profile of the test.
Welp, now that I have what I think is a local repro loop for this, and I'm bisecting, it's looking like the whole commit range I suspected is fine Looking closer, this particular test case is overly optimistic, it deletes a job and immediately gets pods expecting there to be none.