Change:
func (rl *respLogger) Log() { latency := time.Since(rl.startTime) if glog.V(2) { <del> glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) %v%v%v [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.status, rl.statusStack, rl.addedInfo, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) </del> <ins> if !rl.hijacked { glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) %v%v%v [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.status, rl.statusStack, rl.addedInfo, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) } else { glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) hijacked [%s %s]", rl.req.Method, rl.req.RequestURI, latency, rl.req.Header["User-Agent"], rl.req.RemoteAddr)) } </ins> } }

Discussion:
From kube-, Kube-apiserver returns 200 as Success code upon some requests, otherwise return 0 as Success. I1102 20:11:33. 5 ] GET /api/v1/nodes/kubernetes-minion-3io1: (2.683448ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.5:] I1102 20:11:33. 5 ] PUT /api/v1/nodes/kubernetes-minion-3io1/status: (5.474235ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.5:] I1102 20:11:34. 5 ] GET /api/v1/nodes: (2.139174ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:37. 5 ] GET /api/v1/nodes/kubernetes-minion-jmxr: (1.452673ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.3:] I1102 20:11:37. 5 ] PUT /api/v1/nodes/kubernetes-minion-jmxr/status: (4.833358ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.3:] I1102 20:11:38. 5 ] GET /healthz: (104.272µs) 0 [[Go 1.1 package http] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumes?resourceVersion=1687: (9.998412938s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39.

5 ] GET /api/v1/watch/persistentvolumeclaims?resourceVersion=1687: (10.000876825s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumes?resourceVersion=1687: (9.998155747s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/namespaces?resourceVersion=1667: (59.977954152s) 0 [[heapster/v1.0.3 (linux/amd64) kubernetes/$Format] 10.240.0.5:] I1102 20:11:39. 5 ] GET /api/v1/nodes: (1.471292ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumes: (884.225µs) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumeclaims: (942.355µs) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumes: (1.065416ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/namespaces: (1.441822ms) 200 [[heapster/v1.0.3 (linux/amd64) kubernetes/$Format] 10.240.0.5:] I1102 20:11:40.

5 ] GET /api/v1/nodes: (1.958739ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:]
If it were only the watches I would understand, but it's pretty weird for /healthz.
Looking into this, this is likely a false alarm due to an error in pkg/httplog. It assumes that everything calls does. In this case, the status is defaulted to 200. From the So when you just write, the httplog package doesn't see the status and assumes it's 0. I'm going to work on reproducing, confirming, then fixing this.
I confirmed with curl that this is just a logging error at least for healthz. Status code actually returns was 200. This is also pretty obvious since someone writing a 0 status code would result in a stacktrace in the log.
Watches are a different issue than healthz. I think watches are hijacked, and that kinda disables all status recording ability by this package. Do we care to fix that?
websockets hijack-- I thought watches did ordinary HTTP chunked encoding? , krousey wrote:
Probably, but the way it's currently implemented, hijacking completely bypasses this middle-ware logging layer we have.
Maybe we detect hijacking and log that instead of false data? e.g. would become
works for me! , krousey wrote:
Ok out for review in