jiebi/CodeConvo huggingface dataset:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e tests which use RBD server image fail with What you expected to happen: These tests succeed. How to reproduce it (as minimally and precisely as possible): Run e2e tests which use RBD server image. Anything else we need to know?: /sig storage This issue is introduced by below commit: It change the base image from fedora 26 to fedora 28. In fedora 26, hostname command is installed as a dependency for  However, in fedora 28, it is not installed, as a result, this issue happens due to lack of hostname command. Environment: Kubernetes version (use ): k8s 1.12 alpha
Please see the difference of the behaviour between fedora 26 and 28. [Fedora26] [Fedora28]
Could you help solving this issue? I assume the fix for the code would be as below, but I'm not familiar with the build and push process for the test image. ( in doesn't work.) So, I need your help to make further progress.
(or) Both need to be run from directory
Thank you for letting me know it. It should be s/all-containers/all-container/, but it seems to work. I will test with above patch and above build commands, then make a PR, once I confirm that it works well.
sorry typing from memory :)
I need to ask you two more things: Do we need to increment the version for this image, like 1.0.1, even for this small change? (I guess that and test/images/volume/rbd/VERSION will need to be updated in this case.) How the image push process will be? Does it pushed by CI, if there are a change in the code or docker file, or do someone who have permission will push it manually?
creates below three docker images: Is my understanding right that to make each test access to each arch image, needs to be modified like below? (Without doing so, each test seems to keep trying to access to , without arch.)
For fedora28 image, just installing hostname doesn't seem to work well. Please see the log below and attached . I guess that this is due to the difference of ceph version, which will be beyond my beyond my expertise.
So, could you help debugging this issue? Meanwhile, I will try testing with the image by using fedora26 for workaround.
I confirmed that below commit, which change the base image back to fedora 26, fix this issue. Please see the log . After I checked the image-, I come to think that by pushing the image, each image will be able to be pulled without -arch, so I guess the last parameter for ImageConfig should be false. (I couldn't test it in my environment.) I will fix the commit as so and make a PR once I confirm it. Also, if it will easy to fix with Fedora 28, I will wait for the actual fix and quit making a PR for the workaround.
thanks yes, a quick fix is welcome, looking forward
Thank you for your comment. I will continue with the workaround. I saw a commit for deleting multi arch consideration. I rebased the patch to the commit and made a PR. Please review it.
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: VMAS nodes are reporting error in vmss cluster (vmType set to vmss). What you expected to happen: Node should register correctly in the kubernetes cluster. How to reproduce it (as minimally and precisely as possible): Create a VMAS nodes inside vmss cluster. Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
What would you like to be : There is a tab in the k8s perf dashaboard for the scheduler perf benchmarks, but it is showing nothing, we should fix that: http://perf- What this dashboard should be showing is multiple runs per day of We can run it every hour, and summarize the 24 runs of each day to show 50%, 90% and 99% percentile. Why is this needed: Having daily runs of the benchmarks is important to track performance regressions. /sig scheduling /assign /priority important-soon
Ref
/cc
The ci job's config is at And one of failed logs could be found at I tried to reproduce it on my local machine(because I do not have Gloud credentials, I run the task directly in the container): However it works well on my local machine. It is so weird. It will be very appreciated if anyone could help reproduce it with Gloud credentials.
can you try and reach out to the test-infra people? ping them on the slack channel perhaps.
yes, I have tried to get help from , he is busy now. I'll try more.
While working on reproducing this in order to file a bug against test-infra folks I think I found a solution. Will send PRs shortly. /assign
/retitle sig-scalability-benchmarks#scheduler fails consistently due to build issues
/sig scalability Testgrid link: https://k8s- Error logs:
A word about the solution, the job uses the deprecated script which is not supported anymore by test-infra oncall / folks. I've rewritten the job definition to use pod utilities and it started working (will require to get merged first).
Thanks
Thanks !
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I have tried FlexVolume on a Windows node, it does not work. I am now working on this feature now. Below is the FlexVolume initialization error What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.7.7 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows /assign
Detailed steps about how to use flexvolume on Windows with this PR:
cc:
I am tracking a changed behavior in networking with kubernetes 1.26. If I create a service of type externalTrafficPolicy: Local, and there are no local endpoints in a node, kube-proxy does not create hns loadbalancers for the clusterIP of that service. With kubernetes 1.25, it creates a loadbalancer for clusterIP (but not externalIP). This is the relevant log message: Looks like this comes with a result from a new function isAllEndpointsTerminating introduced in kubernetes 1.26. In Linux, we can still access clusterIP even if there are no local endpoints for the service. Thoughts? The end result is that in Windows, clients cannot access the clusterIP of such a service with there are no local endpoints.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
cc
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right?
, Also let us know, do you have any InternalTrafficPolicy configured ?
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right? That is correct , Also let us know, do you have any InternalTrafficPolicy configured ? We do not have that configured
needs a test to ensure that clauses are implemented correctly. Specifically, that they are not implemented as "deny" rules. eg, given a client pod with IP , if you add a NetworkPolicy with: spec: ingress: from: ipBlock: cidr: 0.0.0.0/0 except: A.B.C.0/24 then it should block ingress from that pod. But if you then add a second policy with: spec: ingress: from: ipBlock: cidr: A.B.C.D/32 then it should reallow ingress from that pod. Then if you delete the first policy, ingress should still be allowed, and if you create the first policy again, ingress should still be allowed. (ie, the result does not depend on which order the two policies were created in). /sig network /priority important-longterm
I can work on this as my first issue related to sig-network.
just noticed this issue while working on a similar effort. since there was no activity for the past 2 months I pushed a PR for this scenario
After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.
But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran
We need to fix this potentially flaky test and add it back to conformance suite. Please see discussion in Also waiting on:
cc /sig testing
/area conformance
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
can we convert it to conformance now??
Once 1.15 opens up
Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestPreemptionRaces Testgrid link: /sig scheduling /help
/reopen I just saw this again
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Seems to be green: https://k8s-
It's still flaky on both master and PR testing: master: https://k8s- () PR testing: (). /reopen
Reopened this issue. <detailsIn response to : It's still flaky on both master and PR testing: PR testing: (). </details
From the above two logs, I'm feeling that there is a "phantom" event (very likely of type Update by enforced resync) triggering the pod in the completed N<subth</subtest, to be still alive in the N+1<subth</subtest.
/triage accepted working on a fix.
It would be great if we assign a different name to the Nth round test's preemptor pod. Currently, whatever this is Nth test, the preemptor pod is always called 'preemptor-pod'.
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When deleting/recreating pods backing a UDP service, traffic from external senders stops flowing. What you expected to happen: Traffic would continue to flow to new pods. How to reproduce it (as minimally and precisely as possible): Based on reading of similar issues and commit messages it seems like this would be most noticable on high-volume UDP applications, statsd and syslog are good examples. a deployment and service for a UDP app. In my test case, this would be the container. an external node, create a small test script utilizing a statsd library such as pystatsd. The most important part is that the sender uses the same source-port for every packet sent. Ex: the sender. Install tcpdump into the pod and watch for incoming packets, example from the test case: the pod tcpdump in the newly created pod. Observe that no new matching packets are flowing. existing conntrack rules on all nodes in the cluster. You should see traffic start flowing to the pod again. example command for clearing conntrack rules on the nodes: Anything else we need to know?: I have tested this only on GKE-managed clusters that utilize google forwarding-rules for external LBs, but I believe the issue may be present on any platform where iptables mode for kube-proxy is used. Tested and verified on 1.8.6-gke.0 and 1.7.11-gke.1. Also tested with both Google Container-optimized OS and the Ubuntu-based node images. I have only test this on GKE It appears this was a known issue and should have been cleared up by: From my reading of , it seems as if the func should exec() to clear up UDP entries on service endpoint change, but the only place I see that func called is from syncProxyRules(). That func looks complex and possibly like it is only run occasionally and not on service/endpoint changes. Environment: Kubernetes version (use ): (Also tested on 1.7.11-gke.1) Cloud provider or hardware configuration: GKE-managed clusters, google cloud. OS (e.g.
from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.
I've my script . Note that it's not tested much so don't use it in a production environment. Also it cleans up all conntrack entries for a deleted pod, not only the UDP related ones.
We are also seeing this issue with type loadbalancer. We are on 1.11. Our use-case is the following: multiple devices are sending to an aggreagator machine. This machine forwards the UDP packages to a loadBalancer in gcp. When debugging the service, I noticed the following: The traffic is assigned to one pod, and the packages are not loadbalanced (client and server are the same cfr use-case) When redeploying a new cluster, Conntrack is cleared on termination signal of the old pod. (client receives connection refused) red/black and highlander deployment strategies have the same effect. How would I expect this to be working then? New cluster is . Conntrack is cleared Data is routed to NEW pods old pods are terminated Best Regards, Tim
I am seeing this exact same issue on our cluster which has k8s v1.14.5 . We have endpoint nginx pods backing a UDP service with nodeport. Suddenly one day we see traffic not hitting into nginx pod but reaching the node hosting the nginx pod. We could simulate the problem by restarting the nginx pods. I am assuming with this issue marked as closed; and having made its way into v1.14; we should not be seeing this. Would also like to point out; the one thing, in addition we have in the service is ExternalTrafficPolicy set to Local. Can anyone please guide/help on why we are seeing this in v.1.14.5 of our cluster.
Seeing this with k8s v1.16.5
The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: flag is not supported on Azure Stack, should switch to use original way to remove a disk from data disk list, refer to for more details What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure
only check that the output includes the correct keys - they don't confirm that it has appropriate values. Doing full validation on the version data, or at least enough to catch null-string values, would guard against failures like the bad functionality seen in OpenSUSE's binaries, which have been shipping with many of the fields left completely blank (as I discovered ).
/sig testing
/remove-sig testing /sig cli Edit: testing owns the framework, binaries, infrastructure etc. for testing. SIG CLI is nominally responsible for their test cases.
If SIGs are responsible for their own testcases, then this touches test cases for api-machinery as well (I'm not sure what the non-CLI version of this is, but there's definitely a schema adherence test that should fail at the API level). /sig api-machinery
/triage accepted /assign
GitHub didn't allow me to assign the following users: SaiHarshaK. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /triage accepted /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign