Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /cc /sig network /priority important-soon /milestone v1.20
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
xref
/assign
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
This has been green for a while now. Should we close this one?
I would ideally like to see a statement that this was triaged, had a root cause analysis done, had a fix for the root cause applied, and that the fix was monitored and deemed successful. Hey can you take a look at let us know if you are happy to that this flake has been figured out and fixed? is correct, this looks green. but this does not
Flaked today: And triage also shows that the version of this test has been having a bad time for a long time on :
I think there was a CI change that caused this to go green. Will follow up.
I am looking at this test grid. It is relatively green. Am I looking at the right one?
Yes, that's the one where this one is. But there is also a one that Rob and I are referring to which is here: and has not passed for a while.
Can we remove this from SIG-Net in favor of GCE or GKE cloudprovider? If this is mostly to due with NEG, it likely only applies to the GKE specific test.
What do we do with this one?
this continues to fail hundreds of times a day - if the thing it is testing is important, should this be prioritized? if not, should the test be removed?
we have several of these, and despite they are tagged as sig-network most of them seems to be GCE only tests. I'm tagging you because you'll have better visibility. Maybe we should define clearly the ownership, is this sig-network, gce or cloud-provider? and then the owner decide if we want to run them or fix them, leaving tests failing in this limbo is not useful as Jordan correctly pointed out
+100 for not staying in that state - we should either prioritize or remove. Re ownership - I don't have strong opinion here. But for sure the name is misleading if the tests are GCE-specific (I would expect "conformance" stuff to not be gce-specific. What are the arguments against 'sig-network' owning it? It's just that you don't want to debug GCE-specific stuff? Or some more fundamental ones (like ability to iterate etc.)?
I didn't dig too much into the tests because I always assumed that was testing GCE only things, but it is possible that I'm wrong. I personally don't have problem on debugging GCE stuff, is the ability to iterate and the codebase that is being tested that worries me: is any person able to debug the test? are there parts of the test that need "internal GCE" access to debug it? is the test testing kubernetes network or gce cloud-provider, gce ingress, ....?
I think that everything should be accessible - to confirm. Technically, many networking (and other) tests are relying on cloud-provider (e.g. all tests that exercise LoadBalanced service). So that isn't unique (do we have complaints about these tests too)? So what make ingress tests different is a separate out-of-tree controller, I guess. [To be clear - I'm not trying to push sig-network to own that, i'm just trying to understand the arguments better.]
Let me share my thoughts, I'm personally trying to make all the dashboards in sig-network green, and this week I was able to sort out most of the problems, the last flake with TCP CLOSE_WAIT seems solved, and everything is acceptable except in GCE and Azure jobs. The dual-stack azure jobs are pending of a new image version bump, but I know there is an Azure person that is taking responsibility and will be working on fixing it: For GCE jobs I use to ping or and you can see that the gce dashboard is in a good state However, there are GCE jobs that are in very bad state, failing continuously, I'm practically doing this by myself, that is a personal decision, spending some time every day to review the dashboards and addressing the "generic" failures, but my time is limited, so if this has to go under sig-network umbrella I just want to be sure that these are really needed and, if they are GCE only features, asking for some help from GCE folks.
I agree we're relying to much on heroics of individual people that isn't sustainable. I think those are valid questions, but I would first like to hear from network folks with Google too.
is taking a look
This might just be a issue with 10m not being quite enough for 100% of the time to pass.
Strike my comment -- looks like there is test related issue.
The tests are failing because glbc needs new rbac permissions for IngressClass and GCPIngressParams. adds those which should cause the the failures to stop.
/assign Thanks guys, we are close to be completely green in sig-network :smile:
to fix the
/reopen It seems only 2 tests were fixed, there are still 7 failing tests
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
maybe this error failure gives you a hint
seem to show another rbac issue:
patch permission to services/status has to be granted separately
adds the permission
Is this resolved yet? I just saw another failure today -
The most recent failure does not look to be because of rbac permissions. But I realize this issue is tracking multiple errors and this error looks more similar to the original problem described. The rbac permissions in has not been merged in yet.
/milestone v1.21 (marking in the current milestone until we get confirmation whether it is relevant to the release or not)
Slack Discussion Thread:
I have been looking at the test failure: This failure actually is a different test than the one this issue was opened with. Looking at the logs, there does not seem to be a clear indication of the root cause. The error is because the loadbalancer comes up within the timeout but the path has not be configured leading to the 404. Looking at the logs, the error seems to be outside of the Kubernetes release and therefore this flake should not be considered as release blocking.
/milestone clear (since this has been deemed to be non release-blocking)
Current logs are not enough to determine the issue. For now, opened to update the glbc image which should give us more instrumentation. The timeout is currently 10 min. One theory is that this sometimes is not enough for the loadbalancer to be ready. The LB comes up, but the path is not available yet. However since the the timeout is currently shared amongst many different tests, I propose we wait until we can determine that is the root cause before increasing the timeout.
this is much better now, only one failures in the last 10 days and the failures is as you said:
Can we update the timeout to 2x (20 min) -- I think this will drop failures significantly.
The timeout is set in a helper method: and uses: Are we okay changing it for all the tests that use that, or do we only want to target this one?
Let's target it for this test -- as I think this is the only one that will require such a long timeout.
:upsidedownface: if you want to be on the safe side :) :)
This still needs more work, while working past week on the ingress v1beta1 problems I found that sometimes the client that polls the LB hangs, we should at least have a better logging to understand which part is broken on the test. Is the client? the LB? something between both?
-- any code reference? Improving the client logging sounds like an easy PR
It failed twice the last week, but now is failing in another point can you PTAL, what is that certificate error?
/milestone v1.22
it only failed once, last week, however, the job is running every 30 min that seems excessive. I'll update the job to run with a larger interval and we'll keep monitoring it
Decreasing period sounds good to me.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
# GLBC ensures that the `cloud.google.com/backendconfigs` and `networking.gke.io/servicenetworkendpointgroups` CRD exists in a desired state: # https://github.com/kubernetes/ingress-gce/blob/5c3fcb5845e74b92ea8bd52929b15fc5c9fa7970/cmd/glbc/main.go#L108 # https://github.com/kubernetes/ingress-gce/blob/5c3fcb5845e74b92ea8bd52929b15fc5c9fa7970/cmd/glbc/main.go#L133 <ins> # GLBC creates and updates `networking.gke.io/GCPIngressParams` # https://github.com/kubernetes/ingress-gce/blob/7f0928629c85e7a54c6af9e6e490ac89d057461a/cmd/glbc/main.go#L151-L162 </ins> # TODO(rramkumar1): https://github.com/kubernetes/ingress-gce/issues/744 - apiGroups: ["apiextensions.k8s.io"] resources: ["customresourcedefinitions"]

resources: ["backendconfigs"] verbs: ["get", "list", "watch", "update", "create", "patch"] - apiGroups: ["networking.gke.io"] <del> resources: ["servicenetworkendpointgroups"] </del> <ins> resources: ["servicenetworkendpointgroups","gcpingressparams"] verbs: ["get", "list", "watch", "update", "create", "patch"] # GLBC creates and updates `networking.k8s.io/IngressClass` - apiGroups: ["networking.k8s.io"] resources: ["ingressclasses"] </ins> verbs: ["get", "list", "watch", "update", "create", "patch"]