Change:
From kube-, Kube-apiserver returns 200 as Success code upon some requests, otherwise return 0 as Success. I1102 20:11:33. 5 ] GET /api/v1/nodes/kubernetes-minion-3io1: (2.683448ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.5:] I1102 20:11:33. 5 ] PUT /api/v1/nodes/kubernetes-minion-3io1/status: (5.474235ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.5:] I1102 20:11:34. 5 ] GET /api/v1/nodes: (2.139174ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:37. 5 ] GET /api/v1/nodes/kubernetes-minion-jmxr: (1.452673ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.3:] I1102 20:11:37. 5 ] PUT /api/v1/nodes/kubernetes-minion-jmxr/status: (4.833358ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.3:] I1102 20:11:38. 5 ] GET /healthz: (104.272µs) 0 [[Go 1.1 package http] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumes?resourceVersion=1687: (9.998412938s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumeclaims?resourceVersion=1687: (10.000876825s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumes?resourceVersion=1687: (9.998155747s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/namespaces?resourceVersion=1667: (59.977954152s) 0 [[heapster/v1.0.3 (linux/amd64) kubernetes/$Format] 10.240.0.5:] I1102 20:11:39. 5 ] GET /api/v1/nodes: (1.471292ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumes: (884.225µs) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumeclaims: (942.355µs) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumes: (1.065416ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/namespaces: (1.441822ms) 200 [[heapster/v1.0.3 (linux/amd64) kubernetes/$Format] 10.240.0.5:] I1102 20:11:40. 5 ] GET /api/v1/nodes: (1.958739ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:]
If it were only the watches I would understand, but it's pretty weird for /healthz.
Looking into this, this is likely a false alarm due to an error in pkg/httplog. It assumes that everything calls does. In this case, the status is defaulted to 200. From the So when you just write, the httplog package doesn't see the status and assumes it's 0. I'm going to work on reproducing, confirming, then fixing this.
I confirmed with curl that this is just a logging error at least for healthz. Status code actually returns was 200. This is also pretty obvious since someone writing a 0 status code would result in a stacktrace in the log.
Watches are a different issue than healthz. I think watches are hijacked, and that kinda disables all status recording ability by this package. Do we care to fix that?
websockets hijack-- I thought watches did ordinary HTTP chunked encoding? , krousey wrote:
Probably, but the way it's currently implemented, hijacking completely bypasses this middle-ware logging layer we have.
Maybe we detect hijacking and log that instead of false data? e.g. would become
works for me! , krousey wrote:
Ok out for review in

Discussion:
// the http.ResponseWriter. We can recover panics from go-restful, and // the logging value is questionable. type respLogger struct { <del> status      int statusStack string addedInfo   string startTime   time.Time </del> <ins> hijacked       bool statusRecorded bool status         int statusStack    string addedInfo      string startTime      time.Time </ins> req *http.Request w   http.ResponseWriter

func (rl *respLogger) Log() { latency := time.Since(rl.startTime) if glog.V(2) { <del> glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) %v%v%v [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.status, rl.statusStack, rl.addedInfo, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) </del> <ins> if !rl.hijacked { glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) %v%v%v [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.status, rl.statusStack, rl.addedInfo, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) } else { glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) hijacked [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) } </ins> } }

// Write implements http.ResponseWriter. func (rl *respLogger) Write(b []byte) (int, error) { <ins> if !rl.statusRecorded { rl.recordStatus(http.StatusOK) // Default if WriteHeader hasn't been called } </ins> return rl.w.Write(b) }

// WriteHeader implements http.ResponseWriter. func (rl *respLogger) WriteHeader(status int) { <ins> rl.recordStatus(status) rl.w.WriteHeader(status) } // Hijack implements http.Hijacker. func (rl *respLogger) Hijack() (net.Conn, *bufio.ReadWriter, error) { rl.hijacked = true return rl.w.(http.Hijacker).Hijack() } func (rl *respLogger) recordStatus(status int) { </ins> rl.status = status <ins> rl.statusRecorded = true </ins> if rl.logStacktracePred(status) { // Only log stacks for errors stack := make([]byte, 2048)

} else { rl.statusStack = "" } <del> rl.w.WriteHeader(status) } // Hijack implements http.Hijacker. func (rl *respLogger) Hijack() (net.Conn, *bufio.ReadWriter, error) { return rl.w.(http.Hijacker).Hijack() </del> }

handler(w, req) } } <ins> type testResponseWriter struct{} func (*testResponseWriter) Header() http.Header       { return nil } func (*testResponseWriter) Write([]byte) (int, error) { return 0, nil } func (*testResponseWriter) WriteHeader(int)           {} func TestLoggedStatus(t *testing.T) { req, err := http.NewRequest("GET", "http://example.com", nil) if err != nil { t.Errorf("unexpected error: %v", err) } var tw http.ResponseWriter = new(testResponseWriter) logger := NewLogged(req, &tw) logger.Write(nil) if logger.status != http.StatusOK { t.Errorf("expected status after write to be %v, got %v", http.StatusOK, logger.status) } tw = new(testResponseWriter) logger = NewLogged(req, &tw) logger.WriteHeader(http.StatusForbidden) logger.Write(nil) if logger.status != http.StatusForbidden { t.Errorf("expected status after write to remain %v, got %v", http.StatusForbidden, logger.status) } } </ins>