Change:
We upgraded iptables on one of our hosts, and kube-proxy began logging errors about being unable to acquire the . However, we didn't notice this, because kube-proxy's health check () was still passing (and that's generally what we look at as our sign that something is amiss.) As a result, kube-proxy ran for days without being able to update iptables rules, eventually causing an outage for one of our services. We expected that kube-proxy's health check would fail if it was not able to interface with iptables. Our system was running Debian stretch, running kube-proxy in iptables mode with systemd, with iptables version 1.6.0. Given that starting configuration, the repro case is: iptables to 1.6.2. that kube-proxy begins emitting logs saying that it cannot update iptables because another process has the . that kube-proxy's health check endpoint () does not return an error. Example health check output: <details</details<details</details<detailsNone </details<details</details<detailsThe hard way </details<detailsDocker version 19.03.8, build </details<detailsCalico v3.18.4 </details
/sig network
you upgraded iptables just one node ? what is the version of iptables on other nodes ?
does it means that is not able to never get the lock or is just that sometimes is able to get it?
Yeah we upgraded on just one node at first. Like my original bug report says, we started at 1.6.0, so all the other nodes had 1.6.0. it never gets the lock. You have to restart kube-proxy so it can go through the iptables version detection routine again and change its behavior to the 1.6.2 logic branch. That's fine, it's whatever, but the health check doesn't fail even though kube-proxy can't do its job, which is a bug in my opinion. Also yes, we checked to see if some other process had the lock. It didn't. It's something to do with changes in how iptables actually works after the upgrade to 1.6.2, specifically. I believe this launchpad issue is relevant:
the logic of the healthcheck should fail if is not able to process the updates, right?
I this to the description as well, but here is an example response during this failure mode: And here are what the logs look like: Running gives no output.
This is interesting the and values are the same. This is the healthz logic, it will be unhealthy if the difference between both is greater than the health timeout. is it possible to share the entire kube-proxy log ? I want to discard something
to save me some time, could you please tell me what sort of messages you are looking for? I would rather go search splunk for them than to set up a repro case again.
just the first 1000 lines since kube-proxy starts should be enough
The JSON is kind of meaningless these days; if the proxier thinks it's healthy, it will just write out the current time as both and . It's only when it thinks its unhealthy that it writes out the real time. See I'm not sure how this could happen...
I'm afraid I'm not at liberty to give you 1000 lines of logs without some more targeted context. Company policy and whatnot. What I will tell you is I put the proxy up to and reproduced the issue and pulled those logs, and if you tell me the kind of messages you are looking for, I can pull the (redacted) context around them. What I can give you now, even without being directed, is the more verbose version of what I posted before, i.e., what happens around the time that the error gets logged: <detailsFollowed by the whole list of iptables rules. Then </detailsI can also give you the startup arguments: <details</detailsAnd finally, I can give you these lines from the iptables code, the health check getting registered, and a couple other startup items: <details</detailsAny more than this, though, and I'll have to ask you to tell me something to grep for.
First thing to check is that the healthcheck actually works EVER. Given the same timestamp (we should fix that?) it thinks it is healthy.
/assign
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
A situation: when there is a service or endpoint is constantly updated (which will update the ): so there is a high probability that the difference between the and the is less than the , so this will be considered healthy If you reduce ( iptables-sync-period * 2, the defaults value of flag is 30s), you may see response code is 503, but in fact, it is not easy to estimate how much it should be reduced. so As for case 3, I think there are something that can be optimized.
Oh! That's wrong. We don't care about the time of the most-recently queued update, we care about the time of the oldest queued update that was after the last . So rather than having and updating it every time gets called, we should have a variable something like , and can set it to 0, and should only set it to if it was previously 0. In fact, we don't even need . Kube-proxy would be healthy if or .
Thanks I see, I'd fix it, can I take this one?
sure /assign

Discussion:
} var _ ProxierHealthUpdater = &proxierHealthServer{} <ins> var zeroTime = time.Time{} </ins> // proxierHealthServer returns 200 "OK" by default. It verifies that the delay between // QueuedUpdate() calls and Updated() calls never exceeds healthTimeout.

recorder      events.EventRecorder nodeRef       *v1.ObjectReference <del> lastUpdated atomic.Value lastQueued  atomic.Value </del> <ins> lastUpdated         atomic.Value oldestPendingQueued atomic.Value </ins> } // NewProxierHealthServer returns a proxier health http server.

// Updated updates the lastUpdated timestamp. func (hs *proxierHealthServer) Updated() { <ins> hs.oldestPendingQueued.Store(zeroTime) </ins> hs.lastUpdated.Store(hs.clock.Now()) } // QueuedUpdate updates the lastQueued timestamp. func (hs *proxierHealthServer) QueuedUpdate() { <del> hs.lastQueued.Store(hs.clock.Now()) </del> <ins> // Set oldestPendingQueued only if it's currently zero hs.oldestPendingQueued.CompareAndSwap(zeroTime, hs.clock.Now()) </ins> } // Run starts the healthz HTTP server and blocks until it exits.

} func (h healthzHandler) ServeHTTP(resp http.ResponseWriter, req *http.Request) { <del> var lastQueued, lastUpdated time.Time if val := h.hs.lastQueued.Load(); val != nil { lastQueued = val.(time.Time) </del> <ins> var oldestPendingQueued, lastUpdated time.Time if val := h.hs.oldestPendingQueued.Load(); val != nil { oldestPendingQueued = val.(time.Time) </ins> } if val := h.hs.lastUpdated.Load(); val != nil { lastUpdated = val.(time.Time)

healthy := false switch { <del> case lastUpdated.IsZero(): </del> <ins> case oldestPendingQueued.IsZero(): </ins> // The proxy is healthy while it's starting up <del> // TODO: this makes it useless as a readinessProbe. Consider changing // to only become healthy after the proxy is fully synced. </del> <ins> // or the proxy is fully synced. </ins> healthy = true <del> case lastUpdated.After(lastQueued): // We've processed all updates healthy = true case currentTime.Sub(lastQueued) < h.hs.healthTimeout: </del> <ins> case currentTime.Sub(oldestPendingQueued) < h.hs.healthTimeout: </ins> // There's an unprocessed update queued, but it's not late yet healthy = true }

resp.WriteHeader(http.StatusServiceUnavailable) } else { resp.WriteHeader(http.StatusOK) // In older releases, the returned "lastUpdated" time indicated the last // time the proxier sync loop ran, even if nothing had changed. To // preserve compatibility, we use the same semantics: the returned // lastUpdated value is "recent" if the server is healthy. The kube-proxy // metrics provide more detailed information. lastUpdated = currentTime } fmt.Fprintf(resp, `{"lastUpdated": %q,"currentTime": %q}`, lastUpdated, currentTime) }