jiebi/CodeConvo huggingface dataset:
NodeDisruptionExclusion - Should be moved to GA, already defaulted to true. I'll open another issue for this I'm afraid of this having some sort of direct impact in cloud providers, do you want me to take this to the sig-cloudproviders meet? Otherwise can we move the above Feature Gates to GA, and schedule their removal for v1.21, wdyt?
Per deprecation lifecycle it seems that a feature needs 6 months or 2 releases (the longer one) in Beta before going to GA. So this actually seems to be the missing part before moving to GA. As the 3 FeatureGates are directed related, probably this issue should be: Move all of them to GA in v1.21 Default LegacyNodeRoleBehavior to false Sounds a good approach? I can keep this opened until we reach v1.21
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
/remove-triage unresolved
/retitle Move Feature gate ServiceNodeExclusion to GA and LockToDefault
/triage accepted as this FG is scheduled to be moved to GA in v1.21, do you want to tackle on this?
fixed in /close
Closing this issue. <detailsIn response to : fixed in /close </details
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: (as documented ) reports this: But certificates are actually stored in a Secret (not ConfigMap) by that name and in that namespace. The message printed by kubeadm should be changed accordingly. /sig cluster-lifecycle Environment: Kubernetes version (use ): 1.14.1 Cloud provider or hardware configuration: n/a OS (e.g: ): Debian Install tools: kubeadm
changed from "kube-dns" to "core-dns". "core-dns" is failing for me on Debian Buster: works when I use . "core-dns" should work. Perhaps related to this? Kubernetes version (use ): current master () OS (e.g: ): Debian Buster Kernel (e.g. ): 5.5.0-0.bpo.2-amd64 /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
provides a description of the error, causes, and workarounds. Typically this is due to a local dns resolver installed on the host. Solution is to configure kubelet to use a different (more detail in link).
That mirrors what I found through the issue. Needs to be fixed in then.
/sig testing /cc
/cc
I think we have to parameterize this
I think the safer way is to configure kublet to send the correct to all “default” dns policy pods, not just fix in coredns.
Doh, right :)
I'm running Kubernetes under Docker (following guide). I tried to set up an RC that would run a registry:2 container and attach a gitRepo volume with the SSL certs I want to use. When I run it though, I get this error: If k8s supports git repositories as volumes though, it feels like the executable should be included in the container.
k8s supports git repositories as volumes but it's an optional feature that not everyone wants, so it would not be appropriate to require/include the git executable. This is similar to how k8s supports nfs volumes, but we don't require you to run the kubelet on an os with nfs kernel support, because not everyone wants it.
Hmm. Okay, that makes sense. Would it be feasible to create tags for those containers that have git?
I think I misunderstood the request. I think it's actually reasonable to add git to hyperkube, since afaict, it's only used for single node docker. The Dockerfile is , feel free to send a pr.
When running the scheduler at or greater, the processed component config may be logged as a single line with newline characters inserted, like so: config should always be readable and formatted to an easily copied/parsed format, like: Not sure how often this happens, need to try reproducing ref and Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem has been reproduced. Here are the steps : Steps to Reproduce: kubeadm a pod network runtime, ( I used calico in my case). to reporter's kube-scheduler config yaml, here is another one. apiVersion: clientConnection: acceptContentTypes: "" burst: 100 contentType: application/vnd.kubernetes.protobuf kubeconfig: qps: 50 enableContentionProfiling: true enableProfiling: true healthzBindAddress: "" kind: KubeSchedulerConfiguration leaderElection: leaderElect: true leaseDuration: 15s renewDeadline: 10s resourceLock: leases resourceName: kube-scheduler resourceNamespace: kube-system retryPeriod: 2s metricsBindAddress: "" parallelism: 16 percentageOfNodesoScore: 0 podInitialBackoffSeconds: 1 podMaxBackoffSeconds: 10 profiles: pluginConfig: args: apiVersion: kind: NodeAffinityArgs name: NodeAffinity args: run the kube-scheduler command with an unassigned ip address that your control-plane can bind to: -scheduler --config scheduler- --bind-address "172.18.56.64" -v 5 Here, v can be anything greater than 3. the config file will have n characters in it, whereas it should have been formatted correctly. W1012 19:21:54. ] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
I1012 19:21:54. ] "Using component config" config="apiVersion: nclientConnection:n acceptContentTypes: ""n burst: 100n contentType: application/vnd.kubernetes.protobufn kubeconfig: n qps: 50nenableContentionProfiling: truenenableProfiling: truenhealthzBindAddress: ""nkind: KubeSchedulerConfigurationnleaderElection:n leaderElect: truen leaseDuration: 15sn renewDeadline: 10sn resourceLock: leasesn resourceName: kube-schedulern resourceNamespace: kube-systemn retryPeriod: 2snmetricsBindAddress: ""nparallelism: 16npercentageOfNodesToScore: 0npodInitialBackoffSeconds: 1npodMaxBackoffSeconds: 10nprofiles:n- pluginConfig:n - args:n apiVersion: n kind: DefaultPreemptionArgsn minCandidateNodesAbsolute: 100n minCandidateNodesPercentage: 10n name: DefaultPreemptionn - args:n apiVersion: n hardPodAffinityWeight: 1n kind: InterPodAffinityArgsn name: InterPodAffinityn - args:n apiVersion: n kind: NodeAffinityArgsn name: NodeAffinityn - args:n apiVersion: n kind: NodeResourcesBalancedAllocationArgsn resources:n - name: cpun weight: 1n - name: memoryn weight: 1n name: NodeResourcesBalancedAllocationn - args:n apiVersion: n kind: NodeResourcesFitArgsn scoringStrategy:n resources:n - name: cpun weight: 1n - name: memoryn weight: 1n type: LeastAllocatedn name: NodeResourcesFitn - args:n apiVersion: n defaultingType: Systemn kind: PodTopologySpreadArgsn name: PodTopologySpreadn - args:n apiVersion: n bindTimeoutSeconds: 600n kind: VolumeBindingArgsn name: VolumeBindingn plugins:n bind:n enabled:n - name: DefaultBindern weight: 0n filter:n enabled:n - name: NodeUnschedulablen weight: 0n - name: NodeNamen weight: 0n - name: TaintTolerationn weight: 0n
- name: NodeAffinityn weight: 0n - name: NodePortsn weight: 0n - name: NodeResourcesFitn weight: 0n - name: VolumeRestrictionsn weight: 0n - name: EBSLimitsn weight: 0n - name: GCEPDLimitsn weight: 0n - name: NodeVolumeLimitsn weight: 0n - name: AzureDiskLimitsn weight: 0n - name: VolumeBindingn weight: 0n - name: VolumeZonen weight: 0n - name: PodTopologySpreadn weight: 0n - name: InterPodAffinityn weight: 0n permit: {}n postBind: {}n postFilter:n enabled:n - name: DefaultPreemptionn weight: 0n preBind:n enabled:n - name: VolumeBindingn weight: 0n preFilter:n enabled:n - name: NodeResourcesFitn weight: 0n - name: NodePortsn weight: 0n - name: VolumeRestrictionsn weight: 0n - name: PodTopologySpreadn weight: 0n - name: InterPodAffinityn weight: 0n - name: VolumeBindingn weight: 0n - name: NodeAffinityn weight: 0n preScore:n enabled:n - name: InterPodAffinityn weight: 0n - name: PodTopologySpreadn weight: 0n - name: TaintTolerationn weight: 0n - name: NodeAffinityn weight: 0n queueSort:n enabled:n - name: PrioritySortn weight: 0n reserve:n enabled:n - name: VolumeBindingn weight: 0n score:n enabled:n - name: NodeResourcesBalancedAllocationn weight: 1n - name: ImageLocalityn weight: 1n - name: InterPodAffinityn weight: 1n - name: NodeResourcesFitn weight: 1n - name: NodeAffinityn weight: 1n - name: PodTopologySpreadn weight: 2n - name: TaintTolerationn weight: 1n schedulerName: default-schedulern" Will be trying to write a unit-test for the same.
Awesome, thank you If you want to open a WIP PR to address the changes, we can start reviewing those while you work on the unit test in the same PR.
Hi Sorry for the super late reply! Was working on other stuff. I can open a PR to fix the issue. i have fixed the actual issue and now the logs format correctly. I am just super confused with how to write a test case to verify this. I went through the unit test framework, and couldn't figure out to write a unit test cases using the go libraries that kube provides. I will write now set up the WIP PR.
thanks! a test case might not be totally necessary, since this is just a change to logging, but if you open up your PR we can see how you fixed it and that might help show how we can add a test around your fix
This is the logging output after the fix: be-system, so request-header client certificate authentication won't work. W1108 05:15:16. 8731 ] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work. I1108 05:15:16. 8731 ] Using component configconfigapiVersion: clientConnection: acceptContentTypes: "" burst: 100 contentType: application/vnd.kubernetes.protobuf kubeconfig: qps: 50 enableContentionProfiling: true enableProfiling: true healthzBindAddress: "" kind: KubeSchedulerConfiguration leaderElection: leaderElect: true leaseDuration: 15s renewDeadline: 10s resourceLock: leases resourceName: kube-scheduler resourceNamespace: kube-system retryPeriod: 2s metricsBindAddress: "" parallelism: 16 percentageOfNodesToScore: 0 podInitialBackoffSeconds: 1 podMaxBackoffSeconds: 10 profiles: pluginConfig: args: apiVersion: kind: DefaultPreemptionArgs minCandidateNodesAbsolute: 100 minCandidateNodesPercentage: 10 name: DefaultPreemption args: apiVersion: hardPodAffinityWeight: 1 kind: InterPodAffinityArgs name: InterPodAffinity args: apiVersion: kind: NodeAffinityArgs name: NodeAffinity args: apiVersion: kind: NodeResourcesBalancedAllocationArgs resources: name: cpu weight: 1 name: memory weight: 1 name: NodeResourcesBalancedAllocation args:
apiVersion: kind: NodeResourcesFitArgs scoringStrategy: resources: name: cpu weight: 1 name: memory weight: 1 type: LeastAllocated name: NodeResourcesFit args: apiVersion: defaultingType: System kind: PodTopologySpreadArgs name: PodTopologySpread args: apiVersion: bindTimeoutSeconds: 600 kind: VolumeBindingArgs name: VolumeBinding plugins: bind: enabled: name: DefaultBinder weight: 0 filter: enabled: name: NodeUnschedulable weight: 0 name: NodeName weight: 0 name: TaintToleration weight: 0 name: NodeAffinity weight: 0 name: NodePorts weight: 0 name: NodeResourcesFit weight: 0 name: VolumeRestrictions weight: 0 name: EBSLimits weight: 0 name: GCEPDLimits weight: 0 name: NodeVolumeLimits weight: 0 name: AzureDiskLimits weight: 0 name: VolumeBinding weight: 0 name: VolumeZone weight: 0 name: PodTopologySpread weight: 0 name: InterPodAffinity weight: 0 permit: {} postBind: {} postFilter: enabled: name: DefaultPreemption weight: 0 preBind: enabled: name: VolumeBinding weight: 0 preFilter: enabled: name: NodeResourcesFit weight: 0 name: NodePorts weight: 0 name: VolumeRestrictions weight: 0 name: PodTopologySpread weight: 0 name: InterPodAffinity weight: 0 name: VolumeBinding weight: 0 name: NodeAffinity weight: 0 preScore: enabled: name: InterPodAffinity weight: 0 name: PodTopologySpread weight: 0 name: TaintToleration weight: 0 name: NodeAffinity weight: 0 queueSort: enabled: name: PrioritySort weight: 0 reserve: enabled: name: VolumeBinding weight: 0 score: enabled: name: NodeResourcesBalancedAllocation weight: 1 name: ImageLocality weight: 1 name: InterPodAffinity weight: 1 name: NodeResourcesFit weight: 1 name: NodeAffinity weight: 1 name: PodTopologySpread weight: 2 name: TaintToleration weight: 1 Here, there are no more escape sequences.
cool, if you have time please feel free to open a PR with your changes
Problem showed, that despite e2e tests for container's logs, we still can encounter regressions for system logs. We need to establish an e2e test for ingesting system components's logs: kubelet docker CC
broke ingesting logs from master and we didn't notice it at once Test should check that logs are present from all nodes, including master (when possible)
Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.
/sig instrumentation do you want this in v1.7? (since you're listed as sig-instrumentation leads at and I can't find a team to ping)
No, thanks for noticing!
Fixed by
apiserverlongrunninggauge and apiserverregisterwatchers are kinda redundant . We discussed this in our SIG instrumentation meeting and while it's clear that apiserverlongrunninggauge, the name does not conform to instrumentation guidelines (it was created before those guidelines existed). What we want to do is deprecate both metrics and create a new one which actually does conform to our naming conventions probably something like . has volunteered to undertake this. /cc /sig instrumentation
/assign
GitHub didn't allow me to assign the following users: yan-lgtm. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
Hello! Have you still working on this issue? I am new to the community, and can I try it?
Hey I the deprecate in If you want to add the new metric, feel free to do so.
Can I add the new metric? If I can, can you give me some advice about the new metric? Thank you~
/assign
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Testing API Priority and Fairness, I issued this command in a rather idle cluster: and I got this result: That is, a list of one request with no information of any sort. That comes from , which invents one phantom request for each priority level that is exempt. We do not want or need a phantom request; is for listing requests in queues. What you expected to happen: I expected to get only queued requests. How to reproduce it (as minimally and precisely as possible): As shown above, with API Priority and Fairness enabled. Anything else we need to know?: Environment: Kubernetes version (use ): this is in Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc /cc /cc
We observed in an e2e that if attach is performed on a volume that was detached before from same node, sometimes the mount of the volume (for 2nd pod) could be stuck waiting on invalid devicePath in call. The problem is - when attach for second pod failed with this caused an dangling volume error from AWS cloudprovider(implemented by us). The dangling volume error is supposed to auto-correct any volumes which are attached to a node that it shouldn't be. But in this case - the dangling volume error was triggered by the fact that, volume was not actually attached to the node but because call for 2nd attachment returned stale data (because of AWS eventual consistency). So, dangling volume exception caused volume to be to actual state of world (so as it can be detached later), but this caused volume to be reported to node's Status field. And that caused second pod to assume volume as attached to invalid device path and hence 2nd pod was stuck for 10 minutes waiting for volume to attach on a device path (/dev/xvdbl) which does not exist.
/sig storage /sig aws cc
Here is k8s logs that confirm that, did return stale data for volume: See detach succeeds at 19.30.25: and yet: so second DescribeVolume API call is actually returning stale data.
There are couple of ways of fixing this: error is used by AWS and Openstack cloudproviders to fix incorrectly attached volumes. We could fix mechanism of dangling volumes, so as when it adds volumes to actual state of the world - it does so without reporting the volume to node's Status. can make a local fix inside AWS so that when we return error, before returning that error we also verify volume attachment by querying the instance. We already have instance information in that call. Verifying volume attachment by and should reduce likelyhood of returning incorrect errors. There is still a chance though that, both API calls would land on a AWS backend which does not yet have updated information.
expanding on fix - I am thinking when we add a volume to actual state of the world via DanglingVolume error, then we should add the volume as "uncertain". We will update the code to not report "uncertain" volumes in node's status and hence in this case volume state will be fixed correctly.
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: cloud-provider-openstack jobs Which test(s) are failing: hack/local-up- itself fails. Since when has it been failing: Recent change , checks for node ready by doing a grep of KUBELETHOST. But when HOSTNAMEOVERRIDE is set , its unable to find the ip and exits. Testgrid link: Reason for failure: Anything else we need to know:
Repalce KUBELETHOST with HOSTNAMEOVERRIDE shoud fix this issue, I have created a pull request
Thanks. But what if HOSTNAME_OVERRIDE is not set , then again it would fail? I guess its optional parameter
cc
If not set, the default value is 127.0.0.1, won't fail
Hi ramineni, nodpad is correct. HOSTNAME_OVERRIDE do have default value which is 127.0.0.1.
Thanks. Then grep HOSTNAME_OVERRIDE is ideal solution. That would work for me. Hopefully PR gets merged soon
Two cases of Reboot e2e are flaky on Jenkins: unclean reboot (echo b sudo tee /proc/sysrq-trigger). They reboot only one node, while reboot of the second doesn't happen and causes test timeout:
CC
Master tracking bug
Re-opening. Current status as follows: each node by dropping all outbound packages for a while and ensure they function afterwards (Failed 8 times in the last 30 runs. Stability: 73 %) each node by dropping all inbound packages for a while and ensure they function afterwards (Failed 5 times in the last 30 runs. Stability: 83 %) each node by ordering unclean reboot and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %) each node by triggering kernel panic and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %) each node by switching off the network interface and ensure they function upon switch on (Failed 3 times in the last 30 runs. Stability: 90 %) each node by ordering clean reboot and ensure they function upon restart (Failed 3 times in the last 30 runs. Stability: 90 %)
Any progress on this ? cc in case Jerzey is OOTO.
Sorry for replying so late, but I was on holidays. The flakiness was caused by liveness probes of some pods (e.g.: kube-dns).
Also - implemented a fix for it - see
For some reason seems to have fixed most but not all reboot tests. For example: Reboot each node by dropping all outbound packets for a while and ensure they function afterwards: kubernetes-e2e-gce-reboot (Failed 8 times in the last 30 runs. Stability: 73 %) Need to investigate and fix.
perhaps you could take a look in Poland while the U.S. is celebrating July 2nd, 3rd, 4th? :-)
There was a problem with monit restarting kubelet because of blocking localhost communication on firewall. Fixed by .
- since is now merged, can we close this issue? Or is there anything remaining here?
Yes, I think that we can close this now.
All reboot tests are pretty stable (1 failure in 180 runs, in total): each node by dropping all inbound packets for a while and ensure they function afterwards (No known failures. Stability 100 %) 4 min 5 sec Passed each node by dropping all outbound packets for a while and ensure they function afterwards (No known failures. Stability 100 %) 4 min 6 sec Passed each node by ordering clean reboot and ensure they function upon restart (Failed 1 times in the last 30 runs. Stability: 96 %) 1 min 41 sec Passed each node by ordering unclean reboot and ensure they function upon restart (No known 1. failures. Stability 100 %) 1 min 25 sec Passed each node by switching off the network interface and ensure they function upon switch on (No known failures. Stability 100 %) 4 min 0 sec Passed each node by triggering kernel panic and ensure they function upon restart (No known failures. Stability 100 %)
This is for manual upgrade testing: I am running the script on the release-1.5 branch and the script has been running for a long time now (40mins) with no sign of completion. Has anyone experienced a similar issue or has been successfully able to run the upgrade script ? Command I ran: -M v1.5.0-beta.1 $ -M v1.5.0-beta.1 == Pre-Upgrade Node OS and Kubelet Versions == name: "kubernetes-master", osImage: "Google Container-VM Image", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-7593", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-bigx", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-q5uf", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" Your active configuration is: [default] Project: rkouj-test1 Zone: us-central1-b INSTANCEGROUPS=kubernetes-minion-group NODENAMES=kubernetes-minion-group-7593 kubernetes-minion-group-bigx kubernetes-minion-group-q5uf == Upgrading master to ''. Do not interrupt, deleting master instance. == Trying to find master named 'kubernetes-master' Looking for address 'kubernetes-master-ip' Using master: kubernetes-master (external IP: 104.197.149.96) Deleted []. WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: Created [].
Once that finished, my master upgrade succeeded after being in a waiting state for a long time: The bad news is that is baked into the k8s release tar, so we'll need to cut a new beta with a fix before we can resume manual upgrade testing.
Thanks for the investigation, and ! I LGTMd the PR sent with the fix.
FYI this won't actually be fixed until this PR is in the 1.5 release branch and we cut a beta release. At that point we will update the instructions in to point to and should be working properly again.
Agreed; I'll take care of doing that once I see the anago announcement of the new beta.