Change:
We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...

Discussion:
} } <ins> var ( err                                                  error errorStream, remoteStdin, remoteStdout, remoteStderr httpstream.Stream ) // set up all the streams first </ins> headers := http.Header{} headers.Set(api.StreamType, api.StreamTypeError) <del> errorStream, err := conn.CreateStream(headers) </del> <ins> errorStream, err = conn.CreateStream(headers) </ins> if err != nil { return err } <ins> defer errorStream.Reset() // Create all the streams first, then start the copy goroutines. The server doesn't start its copy // goroutines until it's received all of the streams. If the client creates the stdin stream and // immediately begins copying stdin data to the server, it's possible to overwhelm and wedge the // spdy frame handler in the server so that it is full of unprocessed frames. The frames aren't // getting processed because the server hasn't started its copying, and it won't do that until it // gets all the streams. By creating all the streams first, we ensure that the server is ready to // process data before the client starts sending any. See https://issues.k8s.io/16373 for more info. if e.stdin != nil { headers.Set(api.StreamType, api.StreamTypeStdin) remoteStdin, err = conn.CreateStream(headers) if err != nil { return err } defer remoteStdin.Reset() } if e.stdout != nil { headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err = conn.CreateStream(headers) if err != nil { return err } defer remoteStdout.Reset() } if e.stderr != nil && !e.tty { headers.Set(api.StreamType, api.StreamTypeStderr) remoteStderr, err = conn.CreateStream(headers) if err != nil { return err } defer remoteStderr.Reset() } // now that all the streams have been created, proceed with reading & copying // always read from errorStream </ins> go func() { message, err := ioutil.ReadAll(errorStream) if err != nil && err != io.EOF {

return } }() <del> defer errorStream.Reset() </del> if e.stdin != nil { <del> headers.Set(api.StreamType, api.StreamTypeStdin) remoteStdin, err := conn.CreateStream(headers) if err != nil { return err } defer remoteStdin.Reset() </del> // TODO this goroutine will never exit cleanly (the io.Copy never unblocks) // because stdin is not closed until the process exits. If we try to call // stdin.Close(), it returns no error but doesn't unblock the copy. It will

if e.stdout != nil { waitCount++ <del> headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err := conn.CreateStream(headers) if err != nil { return err } defer remoteStdout.Reset() </del> go cp(api.StreamTypeStdout, e.stdout, remoteStdout) } if e.stderr != nil && !e.tty { waitCount++ <del> headers.Set(api.StreamType, api.StreamTypeStderr) remoteStderr, err := conn.CreateStream(headers) if err != nil { return err } defer remoteStderr.Reset() </del> go cp(api.StreamTypeStderr, e.stderr, remoteStderr) }

var _ streamProtocolHandler = &streamProtocolV2{} func (e *streamProtocolV2) stream(conn httpstream.Connection) error { <ins> var ( err                                                  error errorStream, remoteStdin, remoteStdout, remoteStderr httpstream.Stream ) </ins> headers := http.Header{} <ins> // set up all the streams first </ins> // set up error stream errorChan := make(chan error) headers.Set(api.StreamType, api.StreamTypeError) <del> errorStream, err := conn.CreateStream(headers) </del> <ins> errorStream, err = conn.CreateStream(headers) </ins> if err != nil { return err } <ins> // set up stdin stream if e.stdin != nil { headers.Set(api.StreamType, api.StreamTypeStdin) remoteStdin, err = conn.CreateStream(headers) if err != nil { return err } } // set up stdout stream if e.stdout != nil { headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err = conn.CreateStream(headers) if err != nil { return err } } // set up stderr stream if e.stderr != nil && !e.tty { headers.Set(api.StreamType, api.StreamTypeStderr) remoteStderr, err = conn.CreateStream(headers) if err != nil { return err } } // now that all the streams have been created, proceed with reading & copying // always read from errorStream </ins> go func() { message, err := ioutil.ReadAll(errorStream) switch {

var wg sync.WaitGroup var once sync.Once <del> // set up stdin stream </del> if e.stdin != nil { <del> headers.Set(api.StreamType, api.StreamTypeStdin) remoteStdin, err := conn.CreateStream(headers) if err != nil { return err } </del> // copy from client's stdin to container's stdin go func() { // if e.stdin is noninteractive, e.g. `echo abc | kubectl exec -i <pod> -- cat`, make sure

}() } <del> // set up stdout stream </del> if e.stdout != nil { <del> headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err := conn.CreateStream(headers) if err != nil { return err } </del> wg.Add(1) go func() { defer wg.Done()

}() } <del> // set up stderr stream </del> if e.stderr != nil && !e.tty { <del> headers.Set(api.StreamType, api.StreamTypeStderr) remoteStderr, err := conn.CreateStream(headers) if err != nil { return err } </del> wg.Add(1) go func() { defer wg.Done()