jiebi/CodeConvo huggingface dataset:
What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it.
[1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting
There are use cases where users want to expand the local volume (may be managed by LVM or something similar), but even though they can edit underlying PV (and expand the underlying device) - the reported size in pvc.Status keeps reporting older value because in-tree local volumes are non-expandable. We could make this easier for users by - implementing interface for local volumes. This would allow following steps for local volume expansion: /admin edits the pvc - requesting new size. expands the underlying device. edits the PV and sets new size. will expand the filesystem and update This would at least get the user from impossible territory to possible. cc /sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hi & any news on this issue ? The related PR seems stalled.
merged. So this is fixed now.
Oh right ! I didn't looked close enough and thought it was only for directory based volumes. Thank you.
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: A Pod is run with local volume mounted in block mode. When that pod is deleted, the tear down process fails because device unmap fails. The device path appears to be empty, causing the unmap to fail. What you expected to happen: The teardown should have succeeded How to reproduce it (as minimally and precisely as possible): Create a pod with a PV for local volume in block mode. Delete the pod. Look at kubelet log files on the minion node to observe the error messages. Anything else we need to know?: Log messages: Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: E0218 09:21:43. 1510 ] Operation for """ ("-148c-11e8-b315-")" failed. No retries permitted until 2018-02-18 09:21:44. +0000 UTC m=+. (durationBeforeRetry 500ms). Error: "UnmapVolume.UnmapDevice on global map path failed for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") : Failed to unmap device from map path. mapPath is empty" Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: I0218 09:21:43. 1510 ] operationExecutor.UnmountVolume started for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") Environment: Kubernetes version (use ): 1.10+ Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): "Container-Optimized OS from Google" Kernel (e.g. ): Linux 4.4.86+ Install tools: Others:
/sig storage
This error is returned in this . It same that we set the global mount path to only if the volume plugin is attachable(see ).
However, local volume plugin is not attachable, so will be empty and the operation will fail. I think we should extract the operation from the block, and execute this operation just before we try to map the device to global map path, then this global map path can be cached in for both attachable and non attachable volume plugins. Any comments? If not, I can help to fix this.
Thank you for the investigation. We've discussed this problem by offline yesterday. I proposed same fix which you mentioned above, then tested the fix works correctly with local volume plugin. I'll push a PR soon.
The device path appears to be empty, causing the unmap to fail. Please note, which was stored in is empty. device path isn't related to this problem.
Sorry about that, this is a typo, what I mean is global mount path.
No, no. My above comment is just note for the section. Your investigation is correct.
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In a windows cluster, containers can get scheduled on Windows nodes where they will never work. Warning Failed 54m (x3 over 54m) kubelet, 35934k8s9000 Failed to pull image "coredns/coredns:1.2.2": rpc error: code = Unknown desc = no matching manifest for windows/amd64 in the manifest list entries What you expected to happen: CoreDNS starts and runs on Linux master nodes. How to reproduce it (as minimally and precisely as possible): Build a Windows Cluster. Note: I first encountered this issue when deploying with ACS-Engine: but it occurs to me that it should be mirrored here, so that the relevant node selectors can be included in the canonical add-on directory. /sig Windows
I suppose this is fine if there are always linux nodes in the cluster. I don't know how Kubernetes or docker manifests work for windows, but CoreDNS does build and run on windows: That is not included in the manifest, is it possible to get that working? I guess you would probably also have to change the way the mounted config file is referred to in windows? Or is it this change good enough?
For now, IMO yes. what do you think? I'm assuming, perhaps incorrectly, that not all core k8s components support windows yet. do you know the current state of Windows support for core k8s components? Is CoreDNS the only core component whose image doesnt support windows?
Have not looked into it in detail yet, but one complication would be configuration. AFAIK Windows systems don't have an ... and the default CoreDNS configuration makes specific reference to the file (to get upstream servers).
Yeah, good point. I was thinking about the -conf /etc/coredns/Corefile which I think we pass to coredns IIRC. But is a bigger issue.
While it is possible to run Windows coreDNS pods, this is still undocumented and not officially supported yet. Officially, only kubelet & kube-proxy are in beta on Windows and scheduled to GA with 1.13 (assuming it meets the criteria and community votes of course). I think for the time being it should be fine to limit scheduling to Linux only to avoid pods being stuck but we should design in a way to keep the option open of undoing this in the future. what do you think?
treats any error returned by the condition as terminal, but silently ignores any errors that aren't returned along with . We should change WaitForPodCondition to behave like . /sig testing /area e2e-test-framework
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Was there any reason why you instead of doing this right away? I fully agree with this issue, we should abort early. I just found the TODO because I have a failing test that ran longer than expected after a final pod failure and that made me check out why it didn't abort quicker :grin:
There should be a way to select the iptables proxy-mode.
Are you interested in modifying ubuntu deploy scripts to use a user provided configure file?
This is fixed by and it will be merged in soon.
This fix breaks cluster/kube- while kubernetes provider is rather than ubuntu, as the following variables are only defined in ubuntu folder PROXYSETTING=${PROXYSETTING:-""} DEBUG=${DEBUG:-"false"} When I run KUBERNETES_PROVIDER=vagrant and run cluster/kube- It will fail with error "==master: [ERROR ] {'pid': 5871, 'retcode': 1, 'stderr': ' line 21: DEBUG: unbound variable', 'stdout': ''}" I will open a separated issue to discuss the solution.
I do not understand. I have not tried KUBERNETESPROVIDER=vagrant and run cluster/kube-, can you please clarify what is going wrong? This PR () affected only files in cluster/ubuntu/; are you saying that some of them are used when KUBERNETESPROVIDER=vagrant? This PR did not introduce PROXY_SETTING or DEBUG.
I tried running kube- with the Vagrant provider, but ran into a different (and much prior) problem; see --- any help would be appreciated.
OK, I see you opened issue for this and PR fixed it. Thanks. I will respond to the larger question under .
What happened: does not handle "holes" in interface device-number correctly, dropping later interfaces. Create instance with eth0, eth1, eth2 ENIs as usual. Remove (eg: by selectively deleting pods) eth1. Only the addresses on eth0 are reported by NodeAddresses. What you expected to happen: Addresses on eth0 and eth2 should be reported.
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry. ‍
Currently, (best-effort) watch bookmarks are sent just right before the given watch is supposed to timeout: In we're adjusting how the size of watch cache is determined and make it dynamic and more duration-oriented (rather than based on some magic constants) - we're targeting to always keep 5 minutes of history in watch cache. With that change, we should get back to our bookmark policy: change it to (additionally to right before timeout) also send bookmarks every N seconds (setting N to slightly less than expected watch cache history period, so slightly less than X minutes we will use there) scale-test this change (to see if it doesn't introduce non-negligible overhead) if so permanently introduce this change /sig scalability api-machinery
/cc
/cc
/cc
I'm reverting as it broke bookmarks. I'm going to fix and re-submit for review hopefully later this week.
/kind bug What happened: The wording of kubectl cluster-info is misleading. When no cluster is running, it will state (in green) that the kubernetes master is running. What you expected to happen: I would expect a different message, like Or a How to reproduce it (as minimally and precisely as possible): Execute kubectl with no config files, and no local kubernetes cluster running on 8080
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Right now, we have e2e test for that executes only CSI mock driver with artificial limit of 2 volumes per node. We need a test with real CSI driver(s). Proposal: Allow individual test storage drivers to specify if they support CSI limits. Add a test that picks a random node and creates as many pods with a PV that the volume can handle (128 on GCE!). Check that the node can't handle any other volume. Make this test part of . Add similar test(s) for in-tree volume plugins. Make sure the tests work with in-tree volume plugins migrated to CSI.
/sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
This sounds important /remove-lifecycle rotten /lifecycle frozen
/priority backlog
To add to this I believe we should test that an in-tree plugin and its "replacement" CSI Driver share the same attach limits (both max and current usage)
/cc
I see this test has been here: But I want to add a more basic/faster version of the test that simply checks if a driver has then has been set in CSINode, similar to wdyt, would it be useful?
This is a really old issue :-D. I think the required test has been implemented, however, we do not run it in CI, because it takes too much time to provision 128 PDs that a GCP node can handle. /unassign That sounds useful!
/assign opened to close this for good :)
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If secondard CIDR are disassociated and associated again, AWS LoadBalancer creation fails. It's due to the VPC state is like so CIDRBlock is twice to securityGroup rules, which result in error What you expected to happen: only associated CIDR are considered when calculate VPC CIDR How to reproduce it (as minimally and precisely as possible): and then associated a secondary CIDR AWS CLB Service Anything else we need to know?: Environment: Kubernetes version (use ): <=1.18.0 Cloud provider or hardware configuration: aws OS (e.g: ): linux Kernel (e.g. ): linux Install tools: kops/eks Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry. ‍
When you setup TLS assets, you need to make sure each master API server is using the same ones, unless you were running a single master. But this is a super common mistake, and it often breaks things. Because the masters validating tokens using that key, the controller-manager also has this service-account key, and it’s responsible for creating tokens for service accounts. So, depending on which apiserver the pod is talking to, that service-account token may, or may not work. This most commonly surfaces in kube2sky (the dns addon). Result: Tokens are invalidated regularly (~10 minutes) Expected: Warning about system having different certs (or some alert). Would this ever be a valid configuration?
Is this specifically for an HA setup (since you mention multiple apiservers)? Or do you mean that each master component should have the same cert?
Correct - it's the HA deployment.
I don't think that the apiservers should necessarily cross-check each other (and all other system components like the controller manager) and alert. If you are running an HA deployment, you should have something else making sure that your deployment is correct. On cluster creation, the startup scripts should do it.
I guess what I'm saying is that we know this is going to be a problem. There's almost no way to make this NOT a problem, yet it is a very easy bug to get into. So... with that in mind, what's the path for doing this? When you add a second master to your HA config, how does it establish trust with the first HA? Could we check at that point?
Yup, we do this on provision time with Ansible.
pull-kubernetes-unit Probably since it was merged in The test checks output from the E2E framework and sometimes some extra log output shows up that isn't expected: No response /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --N/A Triage instead: Data race. links to appreciated links to specific failures in spyglass appreciated specific failure <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
previously:
The data race is for this time . fixed only by using a copy.
/assign
GitHub didn't allow me to assign the following users: Haleygo. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Seems noone work here, try to reproduce and fix. /assign
kubernetes# ./stress -p 20 - "TestRunPositive_VolumeMountControllerAttachEnabledRace" 15m25s: runs so far, 0 failures :( I cannot reproduce it, does anyone have better suggestions?
? I have no better idea.
-p 200 ,-p 300, I reproduced this fault (and another DATA RACE). I continued to test after adding deepCopy and found that the stress did not seem to recognize my modification, because the DATA RACE error line was the same as the original, I was a little bit confused....
confused: Use to compile stress, Add Info log to prove that the code has been modified: Call and fail the increased log is not printed (one timeout). As a result, I cannot verify whether I have fixed this problem!
I forgot to update , now the test continues. :-)
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.