jiebi/CodeConvo huggingface dataset:
Blocking merges, P0
Looks like this e2e code hasn't changed since .
Ugh. What changed is that we moved it off teh flaky list. Perhaps wrongly so. , Stephen Milner wrote:
Ah, I see. removed a number of e2e service items from variables. Relevant here:
the test is actually still running " Services should serve identically named services in different namespaces on different load-balancers" is running green.. Saad's fix where we correctly wait for GCE operations to complete, is checked in. Closing this issue.
Looks like this test is failing forever in and
Somehow I lost this one. Will look at it ASAP , Janet Kuo wrote:
The problem is that this test depends on the api proxy which doesn't do UDP, as far as I can tell, so it could never have worked when UDP support was . I will see what I can do to rectify.
Fixed test is - it's a doozy.
I've asked oncall to move this to the head of the queue. It's hard to figure out existing flakes when the test driving them has been totally changed. I want to re-examine all flakes after tghis
is in. I am going to close a few issues and wait for reoccurence, which is to be handled P(-1)
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Seems to be impacting a few KIND jobs on the master branch the most, but also assorted other jobs see the triage link above Reported by Sample failure: /sig node <!-- Please see the deflaking doc () for more guidance! --
xref probing tests
/assign ref /triage accepted
Thanks Elana, still trying today... Does anyone know if we can use the kubelet logs in e2e tests?
Not very portably in cluster e2e tests at least. Maybe in node_e2e instead.
Have you considered that the issue is not with the test but with the underlying code in kubelet like detailed in this issue Like detailed in the issue I can reliably reproduce (on k8s 1.18) the startup probe passing within a few seconds, but the container is only marked as ready after about a minute.
there is already a fix in the 1.21 release for that (part of a bigger PR): And actually this test was to verify that part of the code, but unfortunately sometimes it doesn't respond as quick as it should (but never on my PC, only in ci/cd tests)...
Some recent flakes:
one of the most common flakes in kind jobs currently failures like: Mar 25 17:03:42.508: Pod became ready in 42.007286524s, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.
Maybe we could speed up the review on
Thank you I will take a look. I'm making a pass through flakes tonight trying to identify if anything is on kind's end
I don't think it's kind... since running on my PC with kind it works 100%, I think the issue is either the resources... or contention in the queue as mentioned to in the comment
/milestone v1.22
commented on PR with the suggested fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.
</details
Yeah...
Thanks for your great work in keeping things in check
FYI
I have (finally) understood what's happening, and I can even explain why the deflake made things worse... Please have a look at the sleep before the worker starts to process probes: So, you have to remember that we have one worker per pod and per probe, and since we have these probe settings for the test: We have (PeriodSeconds=10) starting sometimes immediately (if the Kubelet has started since more than 10s) or after a quick sleep of rand10s. On the contrary, will likely sleep most of the time for rand60s. This causes a delay in the processing of notifications: If we are lucky, this delay is less than 5s and the test passes: Now you can understand why the deflaked settings were actually worse:
The reason I didn't spot these was that I usually start my cluster once for all and iterate on the tests... so the kubelet had plenty of time to start and I never ended in that sleep.
For the solution, I suggest we either remove the sleep completely, or we find something that would make all 3 probe workers start at the same time to avoid sync issues... For the record, I was able to spot this issue thanks to the following log error (because the channel was overflowing): What do you think ?
I have found another issue in that code: If has already ran (and failed), we go to L245 and don't probe via and therefore wait for the next tick, making the test fail.
This seems consistent with what I found when I was adding e2es for the new probe-level termination grace period and why I did my timing math the way I did.
/reopen I don't think this is fixed, I just hit it again and gubernator is still showing plenty of flakes:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
how to check the exact commit used for each test?
For instance, I am not sure (from the name) that should include my latest fix...
In your PR the fix wasn't there:
you may send cherry-pick to 1.21 for which is using kubelet 1.21, not 1.22.
is using kubelet. Cherry-pick to 1.20 may be needed as well.
I have found this one that failed even with the fix: corresponding to:
I don't know what could explain this atm... but it seems we still have delay propagating container statuses.
In the job artifacts there are multiple places with this info including the clone records/logs, and In testgrid it's a row.
yeah afaict since the tests ran on my PR after your fix was merged into master, it showed the flake was still present :( Prow will always use the latest master when running CI.
Sorry for doubting what you said... I was probably in deny and hoped this annoyance was fixed once for all :(
Plan: we'll see if Clayton's pod lifecycle refactor and/or the jitter refactor for probes affects this.
Clayton's PR jitter refactor
since isn't (likely) coming to 1.22 we could cut off 90% of flakes by increasing to 15s delay:
it seems it fails even after merging so let's continue our discussion on slack
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
/milestone v1.23
In Kubelet maybe, but in the flakiness of the e2e tests it was. Unfortunately we can't easily explore the current state :/
In the release blocking jobs the frequency is not horrible now, so unblocking is probably fine, it is not as bad as it was. Outside of the release cycle this is still the flakiest test in some jobs, see e.g. where it's the only flake source we have on record currently.
if we end up needing it, we have a timing/period adjustment PR we can take that should help with the flakiness with
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No. Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST in we have that perform unnecessary shallow copy on pods () : we get the pods from the cache i think we do not mutate the pods (i.e only reading values) so we can avoid this deepCopy i think.
/assign
/remove-kind enhancement /kind technical-debt
That's actually a shallow copy
- I'm sorry i was thinking about another issue while filing this one.. I will update the issue, Thanks.
/sig network
1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.
edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?
We've seen this a few times on GCE now, and I saw it flake on now as well (cc
cc Victor integrated cadvisor into the kubelet. I wonder if a fake version of cadvisor gets integrated into kubelet in the case of builds.
I noticed this first on GCE. (FWIW, GKE uses the same binaries as GCE.)
Jenkins shows the following four changes for the GCE build where this test started consistently failing: Commit by mattmoor Enable usage of a "json key" for authenticating with Commit by dbsmith Add a system modeler to scheduler Commit by dawnchen Convert RestartPolicy to string for v1beta3. Commit by abshah Build statically linked binaryies. With the change to go 1.4, we probably were generating dynamically linked binaries accidentally.
If I had to guess, I would take a stab and say it was , . cc
Feel free to revert by 5pm if no fix.
I believe and are correct, makes the build not use cgo which places a fake cAdvisor in the binary and that fails. Looking into the fix.
is out, it is a revert of the old one along with a plan for not needing static binaries.
First, the formatting. Second, "init-ctr-demo_demos" is an internal artifact, not fit for human-facing errors. Third, the container DOES exist, it's just in a terminating state.
The formatting issue should be fixed by . Regarding to the pod name, I think is better. As for the pod name, the error message is from the HTTP response of kubelet server. I've looked at the code, functions defined in interface are using ($podName_podNamespace) to identify a pod. Without spiting the parameter into two separated parameters, it could be tricky to change the output format of an error message. Personally I would prefer to change the signatures, but I don't know if we are planing to do it. In order to improve the readability, I a client-side check for pod status. See .
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Discovery should validate PreferredVersion for each APIGroup Testgrid link: Reason for failure: The test does not handle API groups which are or removed dynamically during the run of the test. Just because an API group is present in at the beginning of the test does not mean or will be present by the end of the test. This is particularly true for the dynamic/custom-resource API groups registered/unregistered during the e2e test run /sig api-machinery /assign
mitigation opened at
Saw this today in It seems to happen relatively often Likely due to the CRD created in: Should the ValidatingAdmissionPolicy test CRD have its group name template changed to match the existing test exclusion for groups ending in /cc
[x] Original approval issue: Start 9 June 2020 3 Flakes No Flakes for 2 weeks According to this APIsnoop query, there are still some remaining ReplicationController endpoints which are untested. a ReplicationController with a static label the ReplicationController with a new Label and updated data the ReplicationController's status /Scale the ReplicationController's Replica count to 2 Replicas the ReplicationController the ReplicationController's Status all ReplicationControllers in all Namespaces find the ReplicationControllers(1) ensure that the ReplicationController is found and is patched Namespaced ReplicationControllers(1) via a Collection with a LabelSelector Discover useragents: See all endpoints hit from this test Display test coverage change If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
Can you please add in PR
/reopen Merged. Promotion to be created
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?
Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm
What would you like to be : Deprecate the flag which enables exposing logs of apiserver. Why is this needed: As per , in order to disable accessing apiserver logs in the future, we need to deprecate the current default and wait for the requisite time period.
/sig api-machinery
/assign
I'm in favor of deprecating this. It's potentially disruptive, I think an email blast to kubernetes-dev would be a good idea.
Thanks , I have posted in kubernetes-dev mailing list.
I'm also in favor of removing this functionality from the apiserver. Since this is GA behavior, the deprecation period would be 12 months.
cc
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today there are many obstacles preventing users from setting up FlexVolume on a GCE COS image easily. The default FlexVolume driver directory is not writable. Custom mount utilities can't be installed. Solutions exist for these problems but they are time-consuming to execute manually. On top of these, there are also the common FlexVolume pain points such as manual driver and mount utilities installation, and manually restarting kubelet. Ideally all these tasks should be automated, making it a quick and simple experience to have custom out-of-tree volume plugins. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig storage
/assign
The path to the server tarball resolves to: is hardcoded, and does not exist in release builds (v1.3.0-beta.2) The file in question is located at: Correcting the path is a workaround for release builds
cc
Hey, sorry about that. I think I got a fix, but I need your help to test it. I'm traveling and my cellular upload speeds are not so great. Please take a look at this and let me know if it solves the issue:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using IPVS and not natting (no cluster-cidr specified) requests from the host to service IPs are send with the service ip as the source ip. Let be the kubernetes API IP. Kubeproxy will set up with . This means that traffic from the host (which sees the ) will originate from (best matching ip/network). What you expected to happen: The source ip should be natted to the host ip. How to reproduce it (as minimally and precisely as possible): a ipvs based cluster any cluster-cidr from kube-proxy traffic to a service ip (e.g. http to the kubernetes api server) from the node / host network pod the outgoing interface looking for the service ip () Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): Kernel (e.g. ): (based on the Ubuntu 18.04 kernel) Install tools: custom repackaging of kubernetes release tarballs Others: /sig network
We have a similar issue in kube-router FWIW resolution is included in the issue
Ah nice, I reproduced the source IP selection on my local Ubuntu 18.04 without any kubernetes installed (use ip link add / ip addr add and then ping the new dummy IP / tcpdump lo). Might be related to the kernel version (4.15) as you couldn't reproduce it.... Changing the routing table would avoid an extra NAT rule. That sounds interesting but not doable without changing the kubernetes sources Murali Reddy < am Fr., 27. Apr. 2018, 14:06:
let me try with to reproduce. but its no brainer, this issue must be fixed.
getting closer... -CLUSTER-IP is only populated if masquerade-all or cluster-cidr is used: is no OUTPUT masquerading for service IPs if cluster-cidr is not specified (no branch matches): With cluster-cidr enabled I've saved the ipset.... I've disabled cluster-cidr and rebooted (to reset all iptables/ipset/...).
I think the code should: Always populate Set a masquerade rule for service IPs (src == service IP should not happen on OUTPUT)
How to test this with a local cluster.... the node (install , & modprobe the modules , , , ). a new shell: define a service with manual external endpoints tcpdump on the outgoing interface of the node the service ip the tcpdump, it will show outgoing traffic from_ the service IP:
Can you please take a look at this?
/assign
I assume this issue is related to some specific kernel versions? I just neither can't reproduce it nor can't see the failures in our test grid.
I was aware of there is a PR() in flight, my team will evaluate the effects of that PR. But, please rebase the PR first, thanks!