jiebi/CodeConvo huggingface dataset:
/kind bug Using , if a block device is symlinked to the discovery directory e.g. , then MapVolume will fail with since the containerized kubelet cannot see the path on host. is mounted by containerized kubelet, so if before we try to map the device we set devicePath to the evaluation of symlink and end up with a devicePath of , containerized kubelet will see the device and successfully map the device. Are there any issues with this solution? This is on Kubernetes 1.10
/sig storage
ah yes containerized kubelet has many issues when dealing with symlinks. Check out Mounter.GetMode(), it calls a nsenter EvalSymlinks which should be able to do what you need. But actually, one thing to consider, is do you need to do the losetup on the host, or does inside the container work?
Yes it appears to work from inside the container, on the host I run which disappears after I delete the pod using the device. My understanding of running losetup in MapVolume is not that great but I suppose that as long as that /dev/loop0 is created we are fine. Thanks for the quick response, I'll create a PR using that function and tag you
Does it create it on the host or in the containerized kubelet? What happens when containerized kubelet is restarted? The purpose of losetup is to do "refcounting" on the block device to prevent possible naming collisions when devices are attached/detached under unhealthy conditions. Ie, if a device showed up as /dev/sdb and you gave that to a pod, and then for some reason it become unhealthy and disappeared (while the pod is still running), another device (maybe even another volume type) could get attached to the node and reuse /dev/sdb. Now the pod has access to a device it's not supposed to. With filesystem volumes, the refcounting is done with mount points, but for block device, loopback is the equivalent.
It is created on the host (and also visible in the container), and stays around through restart. I understand now, thank you.
Actually after I restart it becomes /vdb instead of /dev/vdb in the losetup output, idk what that is about.
ci-kubernetes-e2e-gce-scale-correctness Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands kubetest.Test Jul 10 14:01:51 UTC+0200 Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands: more . Test /sig node
/cc
/triage accepted
Uhhh, this may help here
/milestone v1.22
/sig scalability FYI
I got annoyed that I couldn't click through to see the other logs, so now you can, ref: (spyglass doesn't render build- because it's 500MB, and the limit for rendering is 100MB)
/assign SIG Scalability chairs/TLs, please find someone to address or call this a non-release-blocking failure
Assigning to current scalability oncall (): /assign
This is sig-node issue - should be addres by SIG-node folks. /sig node
is being discussed as a fix for this and I'm not sure it's correct
is it sig scalability's intent that 5k node clusters should support SSH?
The SSH test in question passes in the "default" job: So the fact that this failure is unique to a scalability job means to me that scalability needs to own at least some part of this
I did a little bit of digging and ci-kubernetes-e2e-gce-scale-correctness uses option in kubetest, which according to seems to be setting KUBESSHBASTION env. So I believe this is main difference between our scalability job and "default".
Debugging the test in and providing a fix there. The crictl host PR () is doing something different.
What happened: When the duration is exactly 2 years or 3 years, human readable duration is returned as or . But when the duration is exactly 4 years, 5 years, etc, the human readable duration is returned as , , etc. What you expected to happen: The format of an exact year should be consistent. Remove when duration is an exact year: should be should be Check other years... 4, 5, 6, 7... not sure if they also have the problem. How to reproduce it (as minimally and precisely as possible): Because this is time-dependent, it can be difficult to reproduce via the CLI, but you can see the problem illustrated in which shows the current behavior: The and test cases should be changed to want and , and the code should be changed to make that pass. Anything else we need to know?: Note, this is not a problem when the duration is exactly 1 year because the human readable duration is still shown in days. So nothing needs to change to handle the case of exactly 1 year. Environment: Kubernetes version (use ): 1.18 Cloud provider or hardware configuration: n/a OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a
/sig api-machinery
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could I pick this up as my first issue?
Absolutely. Just use to assign it to yourself.
Also I just updated the issue description. Check the other exact years 4, 5, 6, 7 because they might also have the problem, I can't remember. I know 8 and beyond don't have the problem because it just stops showing days after 8 years.
/assign bitscuit
might recall a related PR?
(also, I have to leave the pedantic note: a year isn't always 365 days, so it's surprising to me to see the abbreviation exists at all)
I can't find it. Maybe I was remembering
Yeah it was probably that one. At first I had the tests and the fix. But it sat for a while. I changed it to just the tests of existing behavior so it could be approved more easily. Then I made this issue for the fix but decided it would probably be a good first time issue for someone instead of me doing it.
so should leap years be considered by the code?
There's no way to know how long a year is without knowing the date, so it's not a good unit--in the abstract, there's no way to know if it's a leap year or not. The go language deliberately omits it IIRC. Henry Li <:
What's the intention of HumanDuration? From what I've seen, it is just an approximation of a timestamp solely for humans to understand, so assuming 365 days per year sounds about right to me. This issue is about consistency though, so it could be solved the other way. Instead of abbreviating when it's an exact year (2y, 3y, etc.), exact years could just all have 0d (2y0d, 3y0d, etc.)
I think it is just used for a ballpark duration. It loses accuracy as the duration increases anyway because it changes the units. I guess my intention for opening the issue was to make it consistent with itself. For example it doesn't say 2d0h or 2h0m, so it should not say 2y0d.
(FTR I agree that Xy0d looks worse than Xy.) Brian Pursley <:
Hi I have made the required changes. Shall I raise the PR if you haven't started working in it ?
already assigned it, so I think it would be good to let him submit the PR, unless he unassigns or does not submit a PR in a reasonable amount of time.
Thanks I was planning on submitting the PR over the weekend
Hello, I am a first timer in open source contribution community. I think I made the required changes for that issue. However, I don't know how assignment rules work.
/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug
release-master-informing: capz-windows-containerd-masterKubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Both the tests have been failing during most of the runs since 20th October Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 1m6s -- | -- {Oct 31 18:42:02.425: Timed out after 30.001s. Expected <errors.errorString | 0xc0015283b0: { s: "failed to match regexp "GET /echo?msg=prestop" in output "I1031 18:41:06. 8988 ] Started HTTP server on port 9090nI1031 18:41:06. 8988 ] Started UDP server on port 9091n"", } to be nil failed (0xc000f1d000) +0x469 () +0x335} Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 5m11s -- | -- {Oct 31 19:05:50.205: timed out while waiting for pod container-lifecycle-hook-6892/pod-with-poststart-https-hook to be running and ready failed (PodClient).CreateSync(0xc000eb5920, 0x22?) +0x94 (0xc0045e8c00) +0x73 () +0x325} No response_ /sig windows cc
/triage accepted /cc
These failures happen also with Kubernetes master + containerd master with Flannel CNI. Testgrid link:
I figured out why these new failures appear.
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? controller /etc/ssl/certs kubeadm /etc/ssl/certs Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 5 custom built master I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ... Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): CentOS 7.2 Kernel (e.g. ): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x8664 SMP Mon Oct 24 16:09:20 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: deployed with Azure RM template and ansible What happened: controller is not able to do https, including calls to the Azure RM API What you expected to happen: https should work as usual How to reproduce it (as minimally and precisely as possible): Use kubeadm to install kubernetes on CentOS exec into the controller and try "curl " Anything else do we need to know: It looks like caused this issue. On CentOS, I see these files on the host OS: I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image. I also tried to verify this with It seems to not be based on Debian and thus gives a different error: I see 2 possible solutions: of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image.
This is already done for hyperkube and would work out of the box when the host mount would be omitted. could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required. I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.
Same problem seems to apply to the apiserver. Not sure however if it actually does https calls.
implements solution 2.
This issue was moved to kubernetes/kubeadm
When a --resolv-conf (DNS resolver config) is passed to kubelet on a Windows node, the config content is used as part of sandbox initialization (through getHostDNSConfig()). Now with the recent changes (), it seems now to fail with an error like: E0315 10:22:16. 4764 ] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "img-pull-(-b40a-459d-9033-)" with CreatePodSandboxError: "Failed to generate sandbox config for pod "img-pull-(-b40a-459d-9033-)": Unexpected resolver config value: "C:etckubernetescni". Expected "" or "Host"."" pod="img-puller-3842/img-pull-" podUID=-b40a-459d-9033- This is showing on all the runs of GCE testgrid (latest): Reading/using the passed resolv config on Windows Run a Windows pool with DNS resolv config being set as part of kubelet params. No response_ 27 alpha.3 1.27 beta.0 master <detailsGCE </details<details</details<details</details<details</details<details</details
/triage accepted /milestone v1.27 /sig node /sig network
/sig windows /cc
/priority criticial-urgent as this is likely a release blocking
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority criticial-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
Hmm, so you're setting the [1]. This PR introduced the capability to read the DNS config options from the Windows host registries while removing the capability to read from a file, introducing a regression. I've sent a PR addressing this issue: [1]
Is the file you are pointing to a UNIX-style or something else?
Was the file changing the behavior of the node?
Yeah, what used on GCE/GKE, seems to be a Linux-like conf, eg: nameserver 10.128.0.1 search c.<project_name.internal google.internal ` We haven't tried to remove from kubelet to see the behavior.
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager
I opened up a revert PR in case this PR does not merge by code freeze.
Thanks for the resolution! both 2019 and 2022 grids seems to be back to green now!
Thanks for confirming! I opened to ensure gets updated too
New tests for subpath consistently fail in ci-kubernetes-e2e-gce-multizone: https://k8s- /kind bug /sig storage
Is this a blocker for 1.10?
I would expect so, yes. /milestone v1.10
I'm on it /assign
The problem is likely due to inline GCE PD volumes not having enough zone information. Considering changing volumes to PVs instead.
fix was in progress in to change to a PV
/unassign
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
What would you like to be : Currently Mutex lock is acquired in for accessing different maps. Some functions only read map entry: We can utilize RWMutex where the above function acquires read lock. Why is this needed: For concurrent calls to IsFitPredicateRegistered / IsPriorityFunctionRegistered, read lock would lower contention.
/sig scheduling
functions getFitPredicateFunctions, getPriorityFunctionConfigs, ListRegisteredFitPredicates, ListRegisteredPriorityFunctions and GetAlgorithmProvider can also use read lock.
I happened to raise earlier today. Anyone can finish resting ones.
Good to know,
Thanks for raising this ! My PR covered and . Feel free to raise another PR to cover others.
Once is merged, I will send my PR thanks
it's merged now.
is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .
/lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority important-soon /sig network
Ref:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
my fu points me to this commit [] Fix multiple conversion tests that belongs to this PR , that seems related can you confirm if that PR is causing this test to fail? (should we gate on this test?)
Yes, that PR is related to the failure. The test utility is trying to convert extensions/v1beta1 data read from various files directly to a go type. This was never a valid thing to do, and happened to silently succeed because the two structs had an identical structure. This is exactly the type of bug was designed to flush out
the correct fix is to update the test/e2e/testing- files to
/assign
It was my PR that broke this - I can also fix that.
Thanks, o it was the commit that was good but the test that was wrong :upsidedownface: , At least I got right the commit :sweat_smile:
is out for review (talked with offline, he was aware that I'll send that out)
pointed out that for scheduling and kubelet admission, we should consider resources of terminated-but-not-yet-deleted pods to still be in use, because it may take an arbitrary amount of time for the system to deallocate resources. We didn't check to see if this is currently how it is implemented or not. cc/ /sig scheduling
cc/
For terminated-but-not-yet-deleted & phase is case, the Pod is still in scheduler's cache, so it's handled well in scheduler; the Pod will be removed from cache when it's deleted. But in kubelet, it filtered terminated-but-not-yet-deleted out () when using scheduler.predicates. The option in my mind is to ~pass all Pods (including terminated-but-not-yet-deleted) to each admin, and let each admin decide whether filter terminated-but-not-yet-deleted out~ only filter out finished (Failed or Succeed) pods for Admits.
In -, it already consider by . So both scheduler & kubelet handles terminated-but-not-yet-deleted pods well.
Great, thanks for looking into it!
should this issue be closed then?
I think so.
Just to clarify: Kubelet correctly accounts the resources used by terminated-but-not-yet-deleted pods, i.e, kubelet does not count the resource usage of those pods towards total usage on the node. Scheduler, however, does not do the same. Scheduler waits for a pod to be deleted, and then it deducts its resource request from the associated node total usage. Upon deletion of a pod, scheduler is notified and acts: So, scheduler and kubelet behaviors are not exactly the same. I am not sure if scheduler can receive an event when a pod's container status changes. If it could, we could make the logic the same as that of Kubelet.
I guess the current behavior is OK because it is conservative (scheduler considers the resources to be in use even though kubelet considers them free). And actually it seems that scheduler is more "correct" than kubelet, since the resources really aren't usable until the pod is deleted, right? (IIUC that was Vish's original point)
I agree. The current behavior is correct, but a bit conservative. Basically, the scheduler side is more conservative and waits until a pod is deleted from the cache before considering its resources released.