Repository: pkg/sftp (Go)

Issue:
Remote disconnect not detected
Errors from recvPacket are returned to callers directly from Client methods, including the io.EOF and io.ErrUnexpectedEOF from two calls to io.ReadFull. This means that some errors are not detected at all: in TestServerRoughDisconnect,

https://github.com/pkg/sftp/blob/3c22ebff33f246c81ec85ef7fedb04c2bb2da018/client_integration_test.go#L1822-L1836

the io.Copy in fact returns nil, because it interprets io.EOF as "whole file copied", while actually the server was killed. The same occurs when an SSH child process dies. In the latter case, callers probably Wait on the SSH process and will get a "signal: terminated" error, but they can't know if the last operation on the pipe was successfully completed before SSH exited.

I see two ways of dealing with this. Both need a custom error, say errDisconnect.

**Option 1**: Return errDisconnect instead of EOF (both kinds) when a ReadFull fails in recvPacket. In the client, that signals an actual unexpected behavior, because it's normally the client that disconnects. To keep current Server and RequestServer behavior, their Serve methods then need to translate that error back to io.EOF.

@puellanivis suggested using io.ErrUnexpectedEOF when I first raised this issue in #389, but then a client doing io.ReadFull on an sftp.File would have no way to distinguish an actual premature EOF in a remote file from a disconnect.

**Option 2**: keep recvPacket largely as is, but let Client.recvVersion and clientConn.recv translate any EOFs to errDisconnect. This is probably cleaner.

The only desirable change to recvPacket would, I think, be to translate an io.EOF from the second ReadFull,

https://github.com/pkg/sftp/blob/3c22ebff33f246c81ec85ef7fedb04c2bb2da018/packet.go#L160

into an ErrUnexpectedEOF. This EOF occurs in the middle of a packet, which is a protocol error. Only the Servers would notice, though.


Discussion:
Thank you for writing up this issue. My philosophy on error handling is not to inspect the type or value of the error, only to check if the error occurred or not. This was the idea I had in my head when I wrote that code years ago, and sadly my impression of that code has not changed since (although I’m sure the maintainers have improved this package considerably). Would you be able to illustrate the problem with a small program or test case that demonstrates the problem. 

Also, and I apologise if I’m reading too much to this, but this is the third issue that you’ve raised for recvPacket. I don’t doubt that improvements can be made, but are we missing the bigger problem by focusing on symptoms in isolation?
The existing TestServerRoughDisconnect demonstrates the problem when the error from io.Copy is checked:

```go
_, err = io.Copy(ioutil.Discard, f)
if err == nil {
 	t.Errorf("no error from killed server")
}
```

This fails.

> I don’t doubt that improvements can be made, but are we missing the bigger problem by focusing on symptoms in isolation?

I think the code is mostly fine, there's just some corner cases that need handling.
Thank you for explaining. Could you write up the problem as a runnable go program, or a test case?
I would suggest adding error checks to all the RoughDisconnect tests: they simulate exactly the right scenario. An additional standalone test could be

```go
// Uses the sink type from #389.

// Test behavior when the remote (server or SSH client subprocess)
// quits before a connection is properly established.
// See also TestServerRoughDisconnect*.
func TestClientRemoteDisconnects(t *testing.T) {
	for _, packet := range [][]byte{
		nil,
		{0, 0, 0, 10},
		{0, 0, 0, 10, sshFxpVersion},
		{0, 0, 0, 10, sshFxpVersion, 0, 0, 0},
	} {
		r := bytes.NewReader(packet)
		c, err := NewClientPipe(r, &sink{})
		if err == nil || err == io.EOF || err == io.ErrUnexpectedEOF {
			t.Error("wrong error:", err)
		}
		if c != nil {
			c.Close()
		}
	}
}
```

Perhaps this could be merged with the test from #389 (from which I derived it).

I have implemented both fixes I outlined. I can set up PRs when #389 is merged, since they will depend on it (or conflict with it).
Thank you for your reply. I think you are asking that the set of errors from NewClientPipe be specified. As I have outlined above, that is not the error handling philosophy I follow so I am not sure how much I can add to the conversation. I will leave this discussion for the maintainers of the package. Thank you. 
I think we have a misunderstanding. I respect the error handling design of this package, but I'm saying you have to make an exception for it to abide by stdlib interface contracts (io.Reader, ReaderAt).

NewClientPipe is not the essential part of this issue, it's the I/O methods on sftp.File. If those return io.EOF or io.ErrUnexpectedEOF, as they currently can, they're essentially saying no error occurred, but the end of a file was reached. Client code, including stdlib utilities in io, ioutil and bufio interpret EOF this way. The set of errors needs to be specified in so far as it does not include those two.

I suppose the problem can also be fixed in Read and ReadAt, but I believe fixing it one layer down in the call chain is preferable. It's easier to get right, potentially catches cases that I hadn't thought of yet, and has the side benefit that other methods give errors that are easy to interpret.

I suggested the NewClientPipe test because it's an easy proxy for a test on recvPacket. The real test is TestServerRoughDisconnect with the addition I proposed.
So, as I understand the situation, in some cases, if I establish a client connection to a server, and there is an EOF or UnexpectedEOF from the underlying protocol, this will propagate that error into a individual file transfer.

So, if there is an unexpected disconnect, while I am doing a `io.Copy` of a file itself, I may receive the situation where I am receiving an `io.EOF` from the underlying client connection, not from the file itself. This could potentially lead to situations where `io.Copy` (which considers `io.EOF` specifically) will misinterpret the cause of the error, and then swallow the error. Resulting in no error, when there should be an error.

Now that I clearly understand the issue I think it can be more clearly stated: we should never broadcast an `io.EOF` or `io.UnexpectedEOF` from the underlying `conn.clientConn` into the file operations themselves. In fact, I’m starting to wonder if we should broadcast any error to the clients other than “unexpected disconnect”. We can still record the actual error into the `clientConn` so that `clientConn.Wait()` can return the actual error that happened, but the individual file streams shouldn’t be getting anything but a generic disconnect error.
Exactly.

I just pushed branches [disconnect1](https://github.com/greatroar/sftp/tree/disconnect1) and [disconnect2](https://github.com/greatroar/sftp/tree/disconnect2) with proposed fixes. They correspond to Option 1 and 2 above. The former puts the fix inside recvPacket, but then has to translate errDisconnect back to EOF in the server to preserve its behavior (almost, because the server can actually return ErrUnexpectedEOF when it occurs in the middle of a packet). The latter puts the fix in the clientConn, making it less intrusive.
I think both of your proposals do far more than really necessary. https://github.com/pkg/sftp/tree/proposal/broadcast-only-disconnect-to-client-streams

We just don’t broadcast server errors into client file streams. That should be sufficient.
You obviously know this code way better than I do :)

That also passes the extended RoughDisconnect tests.
:laughing: only because you got me looking at it already with the optimizations.
Shall I turn your patch and my test into a PR?
Sure, sounds like a good start.



Patch:
diff --git a/conn.go b/conn.go
index 0d8de601..fe07f433 100644
--- a/conn.go
+++ b/conn.go
@@ -138,8 +138,9 @@ func (c *clientConn) broadcastErr(err error) {
 		listeners = append(listeners, ch)
 	}
 	c.Unlock()
+	bcastRes := result{err: errors.New("unexpected server disconnect")}
 	for _, ch := range listeners {
-		ch <- result{err: err}
+		ch <- bcastRes
 	}
 	c.err = err
 	close(c.closed)


Test patch:
diff --git a/client_integration_test.go b/client_integration_test.go
index 081f975d..eda5d329 100644
--- a/client_integration_test.go
+++ b/client_integration_test.go
@@ -1872,7 +1872,8 @@ func TestServerRoughDisconnect(t *testing.T) {
 		cmd.Process.Kill()
 	}()
 
-	io.Copy(ioutil.Discard, f)
+	_, err = io.Copy(ioutil.Discard, f)
+	assert.Error(t, err)
 }
 
 // sftp/issue/181, abrupt server hangup would result in client hangs.
@@ -1930,7 +1931,8 @@ func TestServerRoughDisconnect3(t *testing.T) {
 		cmd.Process.Kill()
 	}()
 
-	io.Copy(rf, lf)
+	_, err = io.Copy(rf, lf)
+	assert.Error(t, err)
 }
 
 // sftp/issue/234 - also affected Write
@@ -1962,11 +1964,39 @@ func TestServerRoughDisconnect4(t *testing.T) {
 	for {
 		_, err = rf.Write(b)
 		if err != nil {
+			assert.NotEqual(t, io.EOF, err)
 			break
 		}
 	}
 
-	io.Copy(rf, lf)
+	_, err = io.Copy(rf, lf)
+	assert.Error(t, err)
+}
+
+// sftp/issue/390 - server disconnect should not cause io.EOF or
+// io.ErrUnexpectedEOF in sftp.File.Read, because those confuse io.ReadFull.
+func TestServerRoughDisconnectEOF(t *testing.T) {
+	skipIfWindows(t)
+	if *testServerImpl {
+		t.Skipf("skipping with -testserver")
+	}
+	sftp, cmd := testClient(t, READONLY, NODELAY)
+	defer cmd.Wait()
+	defer sftp.Close()
+
+	f, err := sftp.Open("/dev/null")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer f.Close()
+	go func() {
+		time.Sleep(100 * time.Millisecond)
+		cmd.Process.Kill()
+	}()
+
+	_, err = io.ReadFull(f, make([]byte, 10))
+	assert.Error(t, err)
+	assert.NotEqual(t, io.ErrUnexpectedEOF, err)
 }
 
 // sftp/issue/26 writing to a read only file caused client to loop.
@@ -1974,7 +2004,10 @@ func TestClientWriteToROFile(t *testing.T) {
 	skipIfWindows(t)
 	sftp, cmd := testClient(t, READWRITE, NODELAY)
 	defer cmd.Wait()
-	defer sftp.Close()
+	defer func() {
+		err := sftp.Close()
+		assert.NoError(t, err)
+	}()
 
 	f, err := sftp.Open("/dev/zero")
 	if err != nil {