Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: has failed to create namespace with error: Related kube-apiserver log says: What I find interesting is "0" status code. What you expected to happen: Nonempty response returned. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master version, this attempt ran at v1.14.0-alpha.0.793+dc9261bc3b6b81Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): cos Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /cc
/assign
This happened again in:
/cc
- from what I understand, we also know that (at least in some cases) the namespace was actually created. Is that true? [I'm asking, because it significantly reduces the number of places where something could go wrong in those cases]
Yes, at least in the namespace was created despite the 0-length response. All occurrences I saw so far was for creating object of some kind (namespace and replicationcontroller)
It seems we started seeing that quite regularly recently. There was at least one more yesterday (and as in all other cases, it was again on POST operation).
I'm wondering if it may be related to this: In theory, status.Status should only be either StatusSuccess or StatusFailure, but if the data is somewhat corrupted, we would return status.Code = 0.
Though, I went through the code and didn't find anything that could have caused that TBH.
we are seeing this cause many integration test failures, across completely different tests we also saw an odd error that appears to indicate truncated input:
/kind failing-test
Attempting to isolate this with
failed in
- tracking down the earliest time when the "0-length" showed up seems to be at of the CI jobs that shows as failed, the earliest seems to be 2018-11-16 09:24 EST which is a job ( logs are at ). So picking the SHA from that and the one before that ( ) shows the following: seems to include the apiserver panic propagation PR too. may be you can spot something in this list of changes when you get a chance
prior to the panic propagation PR, if the call in was hit, the kube-apiserver process would exit, so edit on 1/12: root-caused it to this PR. see I'm also not seeing panic output in the logs of those test failures most of the other changes in that range were actually reverted in
ack, if i go back a few ci jobs and pick , i see the timeout thing from sttts -
(creat%7Cupdat%7Cdeploy).*0-length%20response I see this in the kube-apiserver logs from , with no other info/error/panic logs around the request line:
so fake success?
yes, but notably, not a (which was what sttts and I had touched shortly before) also, the time between stageTimestamp and requestReceivedTimestamp is not long enough to have hit the timeout case
i expanded the search range and ended up with this: can you please peek when you get a chance?
the only other thing i can think of is somehow the grpc bump is causing issues
the grpc bump from 1.7.5 to 1.13.0 before the the version of the etcd client we use is still tested with grpc 1.7.5 - are you aware of any grpc changes between 1.7.5 and 1.13.0 that could explain response truncation like this?
In all cases I've seen, the request finished successfully in etcd (i.e. I've seen a couple cases where the request was creation of an object (POST xxx), and the object was then readable from etcd (so it successed). Not sure what etcd returned though..
if grpc is suspected I could add a grpc interceptor whcih would log grpc packets and if the issue is triggered then it would be easier to confirm where truncation occurs in grpc library or higher in the stack. Please let me know if you think it is worth to try.
seen in
that seems to rule out issues on the client-go side as uses directly
agreed
Here's an interesting run from CI about an hour or so ago: Seems to be happening in release-1.13 branch too!
that's not surprising, given the first occurrence was well before code freeze lifted
This seems to be yet another failure mode - most of previous ones were status code 0 - this one has 200.
I believe that also is affected by this. It has failed with and associated kube- is: The kube- entry has 0 status code and the error suggests that response was incomplete.
xref
the logged status of 0 on the server side, and no status stack even with merged indicates that neither Write nor WriteHeader is called. the default go server handles that as a 0-length response with a 200 status code, which is why the server logs a status of 0 and clients see a 0-length 200 response. I tried creating all sorts of errors (metav1.Status errors with and without valid status and code fields, non-status errors, etc), and could not reproduce a case where an error passed to would result in a zero-length response and no log. If the switch statement referenced in was hit with a status that was neither success/failure, and a zero code, the server would attempt to write it and would panic and log in the server log.
with the unit test output fix merged, we're getting better visibility to the occurrences of this:
opened to try to capture more details if any code paths are calling WriteObjectNegotiated and returning without calling Write or WriteHeader
strangely, that PR just , but the check for having called Write/WriteHeader passed and did not log any other details: that might seem to indicate that one of the response writers (we wrap several times, for metrics, auditing, etc) might not be propagating header/body data to the original ResponseWriter, which is super strange
seeing more and more of these failures... for example, 3 out of the last 6 runs of failed on this error
tracked it down and deserves all praise. Fix opened in
and for anyone else in case this comes up again, is at least one underlying cause for the client side 200 when the server responded with a 0. Filed issue to track it on the go-restful side.
Thanks

Discussion:
buf := make([]byte, size) buf = buf[:goruntime.Stack(buf, false)] panicReason = strings.TrimSuffix(fmt.Sprintf("%vn%s", panicReason, string(buf)), "n") <ins> // Propagate to parent goroutine panicCh <- panicReason </ins> } <del> // Propagate to parent goroutine panicCh <- panicReason </del> }() if result, err := fn(); err != nil {