Repository: minio/minio (language: go)

Issue:
sftp issue with executing commands with TLS
## NOTE
If this case is urgent, please subscribe to [Subnet](https://min.io/pricing) so that our 24/7 support team may help you faster.

<!--- Provide a general summary of the issue in the Title above -->

## Expected Behavior
sftp>ls  should display list of files in sftp bucket 

## Current Behavior
sftp> ls
remote readdir("/"): Failure
- getting same error message if running any of the sftp commands for example `mkdir` 

## Possible Solution
not sure

## Steps to Reproduce (for bugs)
my understanding debugs are not available for sftp
1. login using sftp client with admin user 
2. issue ls command
3. getting the error remote readdir("/"): Failure

## Context
sftp service is un-useable for us 
## Regression
No
## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used (`minio --version`): (`minio_new version RELEASE.2023-06-23T20-26-00Z (commit-id=216069d0da24d8a8bfc885344c221a8d91fde614)
Runtime: go1.19.10 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2023 MinIO, Inc.`)

* Server setup and configuration:
* Operating System and version (`uname -a`):
(`Linux emlog01 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Tue Mar 8 12:56:54 EST 2022 x86_64 x86_64 x86_64 GNU/Linux`)




Discussion / hints:
@merewind you can do `mc admin trace --call ftp` and share the results 
how did you start the sftp service? @merewind 
(`MINIO_VOLUMES="/var/minio_volume/data"
MINIO_OPTS="--certs-dir /.minio/certs --address :9100 --console-address :9101 --sftp  address=:8022 --sftp ssh-private-key=/home/minio/.ssh/id_rsa"
MINIO_PROMETHEUS_URL="http://xx.0.203.100:6060"
MINIO_ROOT_USER=admin
MINIO_ROOT_PASSWORD=xxx
`)
You need to have a bucket first @merewind we do not allow creating buckets via `sftp` - first create the bucket and then you can perform `mkdir` all you want. 
> @merewind you can do `mc admin trace --call ftp` and share the results

ok I have buckets:
 (`mc ls new
[2023-05-18 19:54:45 EDT]     0B ftp/
[2023-06-01 15:01:46 EDT]     0B loki/
[2023-05-19 11:22:28 EDT]     0B sftp/`)

(`mc admin trace --call ftp new
mc: <ERROR> Unable to start tracing: unknown call name: `ftp`.
`)

> You need to have a bucket first @merewind we do not allow creating buckets via `sftp` - first create the bucket and then you can perform `mkdir` all you want.

yep, I did created buckets first. please see mc ls command above
Upgrade your `mc` also @merewind to trace `ftp` calls. 

```
sftp -P 8022 minioadmin@localhost
minioadmin@localhost's password: 
Connected to localhost.
sftp> ls
testbucket  
sftp> cd testbucket/
sftp> ls
sftp> mkdir test
sftp> ls
test  
sftp> 
```

I don't see any error @merewind 
> Upgrade your `mc` also @merewind to trace `ftp` calls.
> 
> ```
> sftp -P 8022 minioadmin@localhost
> minioadmin@localhost's password: 
> Connected to localhost.
> sftp> ls
> testbucket  
> sftp> cd testbucket/
> sftp> ls
> sftp> mkdir test
> sftp> ls
> test  
> sftp> 
> ```
> 
> I don't see any error @merewind

OK, let me do that and I will post the results, thanks
> (`mc ls new [2023-05-18 19:54:45 EDT] 0B ftp/ [2023-06-01 15:01:46 EDT] 0B loki/ [2023-05-19 11:22:28 EDT] 0B sftp/`)

You reported first `ls` says read failure, now you are saying it returns actual results. then you said you had `mkdir` failure which I can clearly it's not. 

So looks like you are doing something locally and not saying clearly what you are doing in this issue. 

I am going to close this since this is a non-issue. 
 mc admin trace --call ftp new
2023-06-28T13:27:15.907 [FTP] sftp USER=admin COMMAND=List PARAM=/, Source=[sftp-server-driver.go:338:(*sftpDriver).Filelist()] 127.0.0.1:9100 / 170.961492ms
2023-06-28T13:27:49.356 [FTP] sftp USER=admin COMMAND=List PARAM=/, Source=[sftp-server-driver.go:338:(*sftpDriver).Filelist()] 127.0.0.1:9100 / 63.934169ms

@harshavardhana  please see the results for the traces
> > (`mc ls new [2023-05-18 19:54:45 EDT] 0B ftp/ [2023-06-01 15:01:46 EDT] 0B loki/ [2023-05-19 11:22:28 EDT] 0B sftp/`)
> 
> You reported first `ls` says read failure, now you are saying it returns actual results. then you said you had `mkdir` failure which I can clearly it's not.
> 
> So looks like you are doing something locally and not saying clearly what you are doing in this issue.
> 
> I am going to close this since this is a non-issue.

@harshavardhana  can you please have a look at the trace resutls?
I see it working perfectly fine. I am simply assuming you have a setup issue. 

```
sftp -P 8022 dillon@localhost
dillon@localhost's password: 
Connected to localhost.
sftp> ls
testbucket  
sftp> 
```

?
> I see it working perfectly fine. I am simply assuming you have a setup issue.
> 
> ```
> sftp -P 8022 dillon@localhost
> dillon@localhost's password: 
> Connected to localhost.
> sftp> ls
> testbucket  
> sftp> 
> ```
> 
> ?

can you please share your startup config for this? if possible..
```
minio server /tmp/disk{1...4} --ftp="address=:8021" --ftp="passive-port-range=30000-40000" --sftp="address=:8022" --sftp="ssh-private-key=/home/harsha/.ssh/id_rsa" &
```
so the issue seems to be TLS trust in my case it's enabled on the server; disabling TLS makes SFTP functional  

Fix patch:
diff --git a/cmd/ftp-server-driver.go b/cmd/ftp-server-driver.go
index 01f20fce4530a..35ea43c5061a7 100644
--- a/cmd/ftp-server-driver.go
+++ b/cmd/ftp-server-driver.go
@@ -322,7 +322,7 @@ func (driver *ftpDriver) getMinIOClient(ctx *ftp.Context) (*minio.Client, error)
 		return minio.New(driver.endpoint, &minio.Options{
 			Creds:     credentials.NewStaticV4(cred.AccessKey, cred.SecretKey, cred.SessionToken),
 			Secure:    globalIsTLS,
-			Transport: globalRemoteTargetTransport,
+			Transport: globalRemoteFTPClientTransport,
 		})
 	}
 
@@ -336,7 +336,7 @@ func (driver *ftpDriver) getMinIOClient(ctx *ftp.Context) (*minio.Client, error)
 	return minio.New(driver.endpoint, &minio.Options{
 		Creds:     credentials.NewStaticV4(ui.Credentials.AccessKey, ui.Credentials.SecretKey, ""),
 		Secure:    globalIsTLS,
-		Transport: globalRemoteTargetTransport,
+		Transport: globalRemoteFTPClientTransport,
 	})
 }
 
diff --git a/cmd/ftp-server.go b/cmd/ftp-server.go
index 9605184e98e6e..0ae1fddcacea8 100644
--- a/cmd/ftp-server.go
+++ b/cmd/ftp-server.go
@@ -34,6 +34,8 @@ import (
 	"golang.org/x/crypto/ssh"
 )
 
+var globalRemoteFTPClientTransport = NewRemoteTargetHTTPTransport(true)()
+
 // minioLogger use an instance of this to log in a standard format
 type minioLogger struct{}
 
diff --git a/cmd/server-main.go b/cmd/server-main.go
index 39aadde32dd71..b1920fc0c6827 100644
--- a/cmd/server-main.go
+++ b/cmd/server-main.go
@@ -256,7 +256,7 @@ func serverHandleCmdArgs(ctx *cli.Context) {
 	globalProxyTransport = NewCustomHTTPProxyTransport()()
 	globalProxyEndpoints = GetProxyEndpoints(globalEndpoints)
 	globalInternodeTransport = NewInternodeHTTPTransport()()
-	globalRemoteTargetTransport = NewRemoteTargetHTTPTransport()()
+	globalRemoteTargetTransport = NewRemoteTargetHTTPTransport(false)()
 
 	globalForwarder = handlers.NewForwarder(&handlers.Forwarder{
 		PassHost:     true,
diff --git a/cmd/sftp-server-driver.go b/cmd/sftp-server-driver.go
index 6d28e9fe854c5..f87bdc217b798 100644
--- a/cmd/sftp-server-driver.go
+++ b/cmd/sftp-server-driver.go
@@ -143,7 +143,7 @@ func (f *sftpDriver) getMinIOClient() (*minio.Client, error) {
 		return minio.New(f.endpoint, &minio.Options{
 			Creds:     credentials.NewStaticV4(cred.AccessKey, cred.SecretKey, cred.SessionToken),
 			Secure:    globalIsTLS,
-			Transport: globalRemoteTargetTransport,
+			Transport: globalRemoteFTPClientTransport,
 		})
 	}
 
@@ -157,7 +157,7 @@ func (f *sftpDriver) getMinIOClient() (*minio.Client, error) {
 	return minio.New(f.endpoint, &minio.Options{
 		Creds:     credentials.NewStaticV4(ui.Credentials.AccessKey, ui.Credentials.SecretKey, ""),
 		Secure:    globalIsTLS,
-		Transport: globalRemoteTargetTransport,
+		Transport: globalRemoteFTPClientTransport,
 	})
 }
 
diff --git a/cmd/utils.go b/cmd/utils.go
index 8f23c789e68c1..9e1511c14ddf2 100644
--- a/cmd/utils.go
+++ b/cmd/utils.go
@@ -664,14 +664,14 @@ func newCustomDialContext() dialContext {
 
 // NewRemoteTargetHTTPTransport returns a new http configuration
 // used while communicating with the remote replication targets.
-func NewRemoteTargetHTTPTransport() func() *http.Transport {
+func NewRemoteTargetHTTPTransport(insecure bool) func() *http.Transport {
 	return xhttp.ConnSettings{
 		DialContext: newCustomDialContext(),
 		DNSCache:    globalDNSCache,
 		RootCAs:     globalRootCAs,
 		TCPOptions:  globalTCPOptions,
 		EnableHTTP2: false,
-	}.NewRemoteTargetHTTPTransport()
+	}.NewRemoteTargetHTTPTransport(insecure)
 }
 
 // Load the json (typically from disk file).
diff --git a/internal/http/transports.go b/internal/http/transports.go
index fe898fbcadc31..e8c4d31bef7d3 100644
--- a/internal/http/transports.go
+++ b/internal/http/transports.go
@@ -78,7 +78,6 @@ func (s ConnSettings) getDefaultTransport() *http.Transport {
 		IdleConnTimeout:       15 * time.Second,
 		ResponseHeaderTimeout: 15 * time.Minute, // Conservative timeout is the default (for MinIO internode)
 		TLSHandshakeTimeout:   10 * time.Second,
-		ExpectContinueTimeout: 10 * time.Second,
 		TLSClientConfig:       &tlsClientConfig,
 		ForceAttemptHTTP2:     s.EnableHTTP2,
 		// Go net/http automatically unzip if content-type is
@@ -117,7 +116,6 @@ func (s ConnSettings) NewInternodeHTTPTransport() func() http.RoundTripper {
 
 	// Settings specific to internode requests.
 	tr.TLSHandshakeTimeout = 15 * time.Second
-	tr.ExpectContinueTimeout = 15 * time.Second
 
 	return func() http.RoundTripper {
 		return tr
@@ -167,12 +165,12 @@ func (s ConnSettings) NewHTTPTransportWithClientCerts(ctx context.Context, clien
 
 // NewRemoteTargetHTTPTransport returns a new http configuration
 // used while communicating with the remote replication targets.
-func (s ConnSettings) NewRemoteTargetHTTPTransport() func() *http.Transport {
+func (s ConnSettings) NewRemoteTargetHTTPTransport(insecure bool) func() *http.Transport {
 	tr := s.getDefaultTransport()
 
-	tr.TLSHandshakeTimeout = 5 * time.Second
-	tr.ExpectContinueTimeout = 5 * time.Second
+	tr.TLSHandshakeTimeout = 10 * time.Second
 	tr.ResponseHeaderTimeout = 0
+	tr.TLSClientConfig.InsecureSkipVerify = insecure
 
 	return func() *http.Transport {
 		return tr


Test patch: