Change:
I noticed that if I'm following a pod log that is being actively written to it will timeout after 5 minutes. However if the pod is not writing logs the request will not timeout. In that case the request will remain open seemingly indefinitely. I can change the behavior by modifying this line of code: To make it happen faster you can set the value lower. I've been testing with 1 minute. Steps to reproduce: cluster the nginx pod in another terminal generate log output by hitting the pod: (you'll need to change the ip). the logs: In this case whenever the WriteTimeout is reached curl will receive the zero sized chunk and the connection will be closed. If you leave out step 3 curl will continue seemingly indefinitely. This behavior definitely seems odd. Is the usage of WriteTimeout correct here? A quick search shows that others have been confused by the intention of WriteTimeout which is why I ask.
is this same as ?
From discussions on the OpenShift side it sounds somewhat related however what we're talking about is a reasonable (but arbitrary) timeout set. Users of are subjected to this timeout with no good way to pick up where they left off to avoid redownloading potentially large quantities of logs.
To me, it would make sense if the api endpoint had argument which would fetch logs from line (it's easy to track on client on which line we're at), exactly what does: I'm assuming that kubelet gets logs using docker api, unfortunately there is no such thing in docker's api:
I think users that want to do this should get the logs from elastic search instead of from kubectl.
I can't say I disagree about needing something like elastic search in a real world deployment. I guess part of the weirdness is that clients of the logging API rarely know how much is going to be pulled down before they make the request. I guess the answer now if you're following the logs is "5 minutes worth". I know this isn't how it's implemented today but in my mind following the logs should behave more like or . Meaning, when you state you want to follow you don't fetch the entire log file you just get wherever it happens to be (and maybe a little more for context). If you want the whole file you don't pass in .
Any update? This is really annoying and I'm thinking of patching my k8s.
I think the default API timeout has been lengthened to 60 minutes Defined: Used for API server: It looks like the kubelet timeout is still 5 minutes, so that would gate log streaming to 5 minutes:
More nuances in Looks like the API http server times out after 60 minutes: Watches time out between 5 and 10 minutes: Kubelet http server still times out after 5 minutes
Which means that getting logs timeouts after 5 minutes?
yeah
did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
nm, just merged the pr. did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
we could easily bump up the kubelet timeout. The handlers on the kubelet don't have their own built in timeouts (like I did for watch), so a potential downside is long hanging /stats requests could hog connections, and depending on why they're hanging, continue consuming kubelet/cpu/cadvisor time. had a suggestion previously about using elasticsearch, are you running with a cluster that doesn't have the addons (, if you set it up with kube-up it should)?
IMO there's no reason not to make kubelet server's timeout really long, at least for these requests.
I'm trying to put in a fix to increase the kubelet timeouts to 1 hour to match the master. Unfortunately adding proper timeout handling for these cases may not make 1.0, but hopefully this can make it less annoying.

Discussion:
} longRunningRE := regexp.MustCompile(s.LongRunningRequestRE) <ins> longRunningTimeout := func(req *http.Request) (<-chan time.Time, string) { // TODO unify this with apiserver.MaxInFlightLimit if longRunningRE.MatchString(req.URL.Path) || req.URL.Query().Get("watch") == "true" { return nil, "" } return time.After(time.Minute), "" } </ins> if secureLocation != "" { <ins> handler := apiserver.TimeoutHandler(m.Handler, longRunningTimeout) </ins> secureServer := &http.Server{ Addr:           secureLocation, <del> Handler:        apiserver.MaxInFlightLimit(sem, longRunningRE, apiserver.RecoverPanics(m.Handler)), ReadTimeout:    ReadWriteTimeout, WriteTimeout:   ReadWriteTimeout, </del> <ins> Handler:        apiserver.MaxInFlightLimit(sem, longRunningRE, apiserver.RecoverPanics(handler)), </ins> MaxHeaderBytes: 1 << 20, TLSConfig: &tls.Config{ // Change default from SSLv3 to TLSv1.0 (because of POODLE vulnerability)

} }() } <ins> handler := apiserver.TimeoutHandler(m.InsecureHandler, longRunningTimeout) </ins> http := &http.Server{ Addr:           insecureLocation, <del> Handler:        apiserver.RecoverPanics(m.InsecureHandler), ReadTimeout:    ReadWriteTimeout, WriteTimeout:   ReadWriteTimeout, </del> <ins> Handler:        apiserver.RecoverPanics(handler), </ins> MaxHeaderBytes: 1 << 20, } if secureLocation == "" {

package apiserver import ( <ins> "bufio" "encoding/json" </ins> "fmt" <ins> "net" </ins> "net/http" "regexp" "runtime/debug" "strings" <ins> "sync" "time" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors"

}) } <ins> // TimeoutHandler returns an http.Handler that runs h with a timeout // determined by timeoutFunc. The new http.Handler calls h.ServeHTTP to handle // each request, but if a call runs for longer than its time limit, the // handler responds with a 503 Service Unavailable error and the message // provided. (If msg is empty, a suitable default message with be sent.) After // the handler times out, writes by h to its http.ResponseWriter will return // http.ErrHandlerTimeout. If timeoutFunc returns a nil timeout channel, no // timeout will be enforced. func TimeoutHandler(h http.Handler, timeoutFunc func(*http.Request) (timeout <-chan time.Time, msg string)) http.Handler { return &timeoutHandler{h, timeoutFunc} } type timeoutHandler struct { handler http.Handler timeout func(*http.Request) (<-chan time.Time, string) } func (t *timeoutHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { after, msg := t.timeout(r) if after == nil { t.handler.ServeHTTP(w, r) return } done := make(chan struct{}, 1) tw := newTimeoutWriter(w) go func() { t.handler.ServeHTTP(tw, r) done <- struct{}{} }() select { case <-done: return case <-after: tw.timeout(msg) } } type timeoutWriter interface { http.ResponseWriter timeout(string) } func newTimeoutWriter(w http.ResponseWriter) timeoutWriter { base := &baseTimeoutWriter{w: w} _, notifiable := w.(http.CloseNotifier) _, hijackable := w.(http.Hijacker) switch { case notifiable && hijackable: return &closeHijackTimeoutWriter{base} case notifiable: return &closeTimeoutWriter{base} case hijackable: return &hijackTimeoutWriter{base} default: return base } } type baseTimeoutWriter struct { w http.ResponseWriter mu          sync.Mutex timedOut    bool wroteHeader bool hijacked    bool } func (tw *baseTimeoutWriter) Header() http.Header { return tw.w.Header() } func (tw *baseTimeoutWriter) Write(p []byte) (int, error) { tw.mu.Lock() defer tw.mu.Unlock() tw.wroteHeader = true if tw.hijacked { return 0, http.ErrHijacked } if tw.timedOut { return 0, http.ErrHandlerTimeout } return tw.w.Write(p) } func (tw *baseTimeoutWriter) WriteHeader(code int) { tw.mu.Lock() defer tw.mu.Unlock() if tw.timedOut || tw.wroteHeader || tw.hijacked { return } tw.wroteHeader = true tw.w.WriteHeader(code) } func (tw *baseTimeoutWriter) timeout(msg string) { tw.mu.Lock() defer tw.mu.Unlock() if !tw.wroteHeader && !tw.hijacked { tw.w.WriteHeader(http.StatusGatewayTimeout) if msg != "" { tw.w.Write([]byte(msg)) } else { enc := json.NewEncoder(tw.w) enc.Encode(errors.NewServerTimeout("", "", 0)) } } tw.timedOut = true } func (tw *baseTimeoutWriter) closeNotify() <-chan bool { return tw.w.(http.CloseNotifier).CloseNotify() } func (tw *baseTimeoutWriter) hijack() (net.Conn, *bufio.ReadWriter, error) { tw.mu.Lock() defer tw.mu.Unlock() if tw.timedOut { return nil, nil, http.ErrHandlerTimeout } conn, rw, err := tw.w.(http.Hijacker).Hijack() if err == nil { tw.hijacked = true } return conn, rw, err } type closeTimeoutWriter struct { *baseTimeoutWriter } func (tw *closeTimeoutWriter) CloseNotify() <-chan bool { return tw.closeNotify() } type hijackTimeoutWriter struct { *baseTimeoutWriter } func (tw *hijackTimeoutWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { return tw.hijack() } type closeHijackTimeoutWriter struct { *baseTimeoutWriter } func (tw *closeHijackTimeoutWriter) CloseNotify() <-chan bool { return tw.closeNotify() } func (tw *closeHijackTimeoutWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) { return tw.hijack() } </ins> // TODO: use restful.CrossOriginResourceSharing // Simple CORS implementation that wraps an http Handler // For a more detailed implementation use https://github.com/martini-contrib/cors

package apiserver import ( <ins> "io/ioutil" </ins> "net/http" "net/http/httptest" "reflect"

} } <ins> func TestTimeout(t *testing.T) { sendResponse := make(chan struct{}, 1) writeErrors := make(chan error, 1) timeout := make(chan time.Time, 1) resp := "test response" timeoutResp := "test timeout" ts := httptest.NewServer(TimeoutHandler(http.HandlerFunc( func(w http.ResponseWriter, r *http.Request) { <-sendResponse _, err := w.Write([]byte(resp)) writeErrors <- err }), func(*http.Request) (<-chan time.Time, string) { return timeout, timeoutResp })) defer ts.Close() // No timeouts sendResponse <- struct{}{} res, err := http.Get(ts.URL) if err != nil { t.Error(err) } if res.StatusCode != http.StatusOK { t.Errorf("got res.StatusCode %d; expected %d", res.StatusCode, http.StatusOK) } body, _ := ioutil.ReadAll(res.Body) if string(body) != resp { t.Errorf("got body %q; expected %q", string(body), resp) } if err := <-writeErrors; err != nil { t.Errorf("got unexpected Write error on first request: %v", err) } // Times out timeout <- time.Time{} res, err = http.Get(ts.URL) if err != nil { t.Error(err) } if res.StatusCode != http.StatusGatewayTimeout { t.Errorf("got res.StatusCode %d; expected %d", res.StatusCode, http.StatusServiceUnavailable) } body, _ = ioutil.ReadAll(res.Body) if string(body) != timeoutResp { t.Errorf("got body %q; expected %q", string(body), timeoutResp) } // Now try to send a response sendResponse <- struct{}{} if err := <-writeErrors; err != http.ErrHandlerTimeout { t.Errorf("got Write error of %v; expected %v", err, http.ErrHandlerTimeout) } } </ins> func TestGetAPIRequestInfo(t *testing.T) { successCases := []struct { method              string

s := &http.Server{ Addr:           net.JoinHostPort(address.String(), strconv.FormatUint(uint64(port), 10)), Handler:        &handler, <del> ReadTimeout:    5 * time.Minute, WriteTimeout:   5 * time.Minute, </del> MaxHeaderBytes: 1 << 20, } if tlsOptions != nil {

server := &http.Server{ Addr:           net.JoinHostPort(address.String(), strconv.FormatUint(uint64(port), 10)), Handler:        &s, <del> ReadTimeout:    5 * time.Minute, WriteTimeout:   5 * time.Minute, </del> MaxHeaderBytes: 1 << 20, } glog.Fatal(server.ListenAndServe())