Repository: peak/s5cmd (go)

Issue:
s5cmd Doesn't respect proxy environment variables
The AWS CLI + Boto3 support setting environment variables to pass requests through a proxy. In my case, we require this functionality because we use a proxy to route requests from our dev network to our VPC.

The environment variables that are set are:
`https_proxy`
`http_proxy`
`HTTPS_PROXY`
`HTTP_PROXY`

Using the AWS CLI, the following command succeeds:
```
aws --region us-east-2 --no-verify-ssl s3 ls s3://path/to/prefix
```

However, using `s5cmd` (v1.4.0), the analagous command fails:
```
s5cmd --log debug --no-verify-ssl ls s3://path/to/prefix
```

with the following output
```
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.80.130:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.80.130:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.80.130:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.80.130:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.80.130:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.80.130:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.80.130:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.142.89:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.106.241:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.99.57:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: dial tcp 52.219.88.226:443: connect: connection refused
ERROR session: fetching region failed: RequestError: send request failed
caused by: dial tcp 52.219.88.226:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.84.232:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.84.232:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.84.232:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.84.232:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.84.232:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.84.232:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.84.232:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.99.34:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.105.122:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.110.170:443: connect: connection refused
DEBUG retryable error: RequestError: send request failed
caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.109.122:443: connect: connection refused
ERROR "ls s3://path/to/prefix": RequestError: send request failed caused by: Get "https://path.s3.us-east-2.amazonaws.com/?delimiter=%2F&list-type=2&prefix=to%2Fprefix": dial tcp 52.219.109.122:443: connect: connection refused

```

Is there some way to get s5cmd to use the proxy variables? Or is this unsupported?


Pull request description:
storage/s3: Fix proxy with no-verify-ssl flag
This change fixes the problem where proxy is ignored when no-verify-ssl flag is used.

- Test environment is created to test proxy support: proxy_fake.go is added.

Fixes #445

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2fad068..3ee761f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,15 +3,18 @@
 ## not released yet
 
 #### Features
+- Added `--content-type` and `--content-encoding` flags to `cp` command. ([#264](https://github.com/peak/s5cmd/issues/264))
 - Added `--profile` flag to allow users to specify a [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). ([#353](https://github.com/peak/s5cmd/issues/353))
 - Added `--credentials-file` flag to allow users to specify path for the AWS credentials file instead of using the [default location](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where). 
 
+#### Improvements
+- Disable AWS SDK logger if log level is not "trace"
+
 #### Bugfixes
 - Fixed a bug where (`--stat`) prints unnecessarily when used with help and version commands ([#452](https://github.com/peak/s5cmd/issues/452))
 - Changed cp error message to be more precise. "given object not found" error message now will also include absolute path of the file. ([#463](https://github.com/peak/s5cmd/pull/463))
+- Fixed a bug where proxy is not being used when `--no-verify-ssl` flag is used. ([#445](https://github.com/peak/s5cmd/issues/445))
 
-#### Improvements
-- Disable AWS SDK logger if log level is not "trace"
 
 ## v2.0.0 - 4 Jul 2022
 
diff --git a/command/cp.go b/command/cp.go
index 9072367..cba3c42 100644
--- a/command/cp.go
+++ b/command/cp.go
@@ -98,6 +98,9 @@ Examples:
 
 	20. Download an S3 object from a requester pays bucket
 		 > s5cmd --request-payer=requester {{.HelpName}} s3://bucket/prefix/object.gz .
+
+	21. Upload a file to S3 with a content-type and content-encoding header
+		 > s5cmd --content-type "text/css" --content-encoding "br" myfile.css.br s3://bucket/
 `
 
 func NewSharedFlags() []cli.Flag {
@@ -166,6 +169,14 @@ func NewSharedFlags() []cli.Flag {
 			Name:  "raw",
 			Usage: "disable the wildcard operations, useful with filenames that contains glob characters",
 		},
+		&cli.StringFlag{
+			Name:  "content-type",
+			Usage: "set content type for target: defines content type header for object, e.g. --content-type text/plain",
+		},
+		&cli.StringFlag{
+			Name:  "content-encoding",
+			Usage: "set content encoding for target: defines content encoding header for object, e.g. --content-encoding gzip",
+		},
 	}
 }
 
@@ -244,6 +255,8 @@ type Copy struct {
 	raw                   bool
 	cacheControl          string
 	expires               string
+	contentType           string
+	contentEncoding       string
 
 	// region settings
 	srcRegion string
@@ -281,6 +294,8 @@ func NewCopy(c *cli.Context, deleteSource bool) Copy {
 		raw:                   c.Bool("raw"),
 		cacheControl:          c.String("cache-control"),
 		expires:               c.String("expires"),
+		contentType:           c.String("content-type"),
+		contentEncoding:       c.String("content-encoding"),
 		// region settings
 		srcRegion: c.String("source-region"),
 		dstRegion: c.String("destination-region"),
@@ -550,7 +565,6 @@ func (c Copy) doUpload(ctx context.Context, srcurl *url.URL, dsturl *url.URL) er
 	}
 
 	metadata := storage.NewMetadata().
-		SetContentType(guessContentType(file)).
 		SetStorageClass(string(c.storageClass)).
 		SetSSE(c.encryptionMethod).
 		SetSSEKeyID(c.encryptionKeyID).
@@ -558,6 +572,16 @@ func (c Copy) doUpload(ctx context.Context, srcurl *url.URL, dsturl *url.URL) er
 		SetCacheControl(c.cacheControl).
 		SetExpires(c.expires)
 
+	if c.contentType != "" {
+		metadata.SetContentType(c.contentType)
+	} else {
+		metadata.SetContentType(guessContentType(file))
+	}
+
+	if c.contentEncoding != "" {
+		metadata.SetContentEncoding(c.contentEncoding)
+	}
+
 	err = dstClient.Put(ctx, file, dsturl, metadata, c.concurrency, c.partSize)
 	if err != nil {
 		return err
@@ -606,6 +630,13 @@ func (c Copy) doCopy(ctx context.Context, srcurl, dsturl *url.URL) error {
 		SetCacheControl(c.cacheControl).
 		SetExpires(c.expires)
 
+	if c.contentType != "" {
+		metadata.SetContentType(c.contentType)
+	}
+	if c.contentEncoding != "" {
+		metadata.SetContentEncoding(c.contentEncoding)
+	}
+
 	err = c.shouldOverride(ctx, srcurl, dsturl)
 	if err != nil {
 		if errorpkg.IsWarning(err) {
diff --git a/storage/s3.go b/storage/s3.go
index 3877fd0..7f9fbdd 100644
--- a/storage/s3.go
+++ b/storage/s3.go
@@ -549,6 +549,11 @@ func (s *S3) Put(
 		}
 	}
 
+	contentEncoding := metadata.ContentEncoding()
+	if contentEncoding != "" {
+		input.ContentEncoding = aws.String(contentEncoding)
+	}
+
 	_, err := s.uploader.UploadWithContext(ctx, input, func(u *s3manager.Uploader) {
 		u.PartSize = partSize
 		u.Concurrency = concurrency
@@ -934,6 +939,7 @@ func (c *customRetryer) ShouldRetry(req *request.Request) bool {
 var insecureHTTPClient = &http.Client{
 	Transport: &http.Transport{
 		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
+		Proxy:           http.ProxyFromEnvironment,
 	},
 }
 
diff --git a/storage/storage.go b/storage/storage.go
index 77d0631..83a7391 100644
--- a/storage/storage.go
+++ b/storage/storage.go
@@ -279,3 +279,12 @@ func (m Metadata) SetSSEKeyID(kid string) Metadata {
 	m["EncryptionKeyID"] = kid
 	return m
 }
+
+func (m Metadata) ContentEncoding() string {
+	return m["ContentEncoding"]
+}
+
+func (m Metadata) SetContentEncoding(contentEncoding string) Metadata {
+	m["ContentEncoding"] = contentEncoding
+	return m
+}


Test patch:
diff --git a/e2e/app_test.go b/e2e/app_test.go
index 46e76fd..a4eb32d 100644
--- a/e2e/app_test.go
+++ b/e2e/app_test.go
@@ -2,6 +2,7 @@ package e2e
 
 import (
 	"fmt"
+	"os"
 	"strconv"
 	"strings"
 	"testing"
@@ -132,6 +133,50 @@ func TestAppDashStat(t *testing.T) {
 	}
 }
 
+func TestAppProxy(t *testing.T) {
+
+	testcases := []struct {
+		name string
+		flag string
+	}{
+		{
+			name: "without no-verify-ssl flag",
+			flag: "",
+		},
+		{
+			name: "with no-verify-ssl flag",
+			flag: "--no-verify-ssl",
+		},
+	}
+	for _, tc := range testcases {
+		tc := tc
+		t.Run(tc.name, func(t *testing.T) {
+			const expectedReqs = 1
+
+			proxy := httpProxy{}
+			pxyUrl, cleanup := setupProxy(&proxy)
+			defer cleanup()
+
+			os.Setenv("http_proxy", pxyUrl)
+
+			_, s5cmd, cleanup := setup(t, withProxy())
+			defer cleanup()
+
+			var cmd icmd.Cmd
+			if tc.flag != "" {
+				cmd = s5cmd(tc.flag, "ls")
+			} else {
+				cmd = s5cmd("ls")
+			}
+
+			result := icmd.RunCmd(cmd)
+
+			result.Assert(t, icmd.Success)
+			assert.Assert(t, proxy.isSuccessful(expectedReqs))
+		})
+	}
+}
+
 func TestAppUnknownCommand(t *testing.T) {
 	t.Parallel()
 
diff --git a/e2e/proxy.go b/e2e/proxy.go
new file mode 100644
index 0000000..7fce994
--- /dev/null
+++ b/e2e/proxy.go
@@ -0,0 +1,85 @@
+package e2e
+
+import (
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"sync/atomic"
+)
+
+// Hop-by-hop headers. These are removed when sent to the backend.
+// http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html
+var hopHeaders = []string{
+	"Connection",
+	"Keep-Alive",
+	"Proxy-Authenticate",
+	"Proxy-Authorization",
+	"Te", // canonicalized version of "TE"
+	"Trailers",
+	"Transfer-Encoding",
+	"Upgrade",
+}
+
+func copyHeader(dst, src http.Header) {
+	for k, vv := range src {
+		for _, v := range vv {
+			dst.Add(k, v)
+		}
+	}
+}
+
+func delHopHeaders(header http.Header) {
+	for _, h := range hopHeaders {
+		header.Del(h)
+	}
+}
+
+type httpProxy struct {
+	successReqs int64
+	errorReqs   int64
+}
+
+func (p *httpProxy) ServeHTTP(wr http.ResponseWriter, req *http.Request) {
+	if req.URL.Scheme != "http" && req.URL.Scheme != "https" {
+		msg := "unsupported protocol scheme " + req.URL.Scheme
+		http.Error(wr, msg, http.StatusBadRequest)
+		return
+	}
+	// We need to explicitly set New Transport with no Proxy to make
+	// sure ServeHttp only receives the requests once. Otherwise, it
+	// causes httpProxy to call itself infinitely over and over again.
+	client := &http.Client{
+		Transport: &http.Transport{Proxy: nil},
+	}
+	// http: Request.RequestURI can't be set in client requests.
+	// http://golang.org/src/pkg/net/http/client.go
+	req.RequestURI = ""
+
+	delHopHeaders(req.Header)
+
+	resp, err := client.Do(req)
+	if err != nil {
+		http.Error(wr, "Server Error", http.StatusInternalServerError)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusOK {
+		atomic.AddInt64(&p.successReqs, 1)
+	} else {
+		atomic.AddInt64(&p.errorReqs, 1)
+	}
+	delHopHeaders(resp.Header)
+
+	copyHeader(wr.Header(), resp.Header)
+	wr.WriteHeader(resp.StatusCode)
+	io.Copy(wr, resp.Body)
+}
+
+func (p *httpProxy) isSuccessful(totalReqs int64) bool {
+	return totalReqs == atomic.LoadInt64(&p.successReqs) && atomic.LoadInt64(&p.errorReqs) == 0
+}
+
+func setupProxy(p *httpProxy) (string, func()) {
+	proxysrv := httptest.NewServer(p)
+	return proxysrv.URL, proxysrv.Close
+}
diff --git a/e2e/s3_fake.go b/e2e/s3_fake.go
index 466cd48..639d10c 100644
--- a/e2e/s3_fake.go
+++ b/e2e/s3_fake.go
@@ -2,6 +2,7 @@ package e2e
 
 import (
 	"net/http/httptest"
+	"net/url"
 	"strings"
 	"testing"
 
@@ -11,7 +12,7 @@ import (
 	"gotest.tools/v3/fs"
 )
 
-func s3ServerEndpoint(t *testing.T, testdir *fs.Dir, loglvl, backend string, timeSource gofakes3.TimeSource) (string, func()) {
+func s3ServerEndpoint(t *testing.T, testdir *fs.Dir, loglvl, backend string, timeSource gofakes3.TimeSource, enableProxy bool) (string, func()) {
 	var s3backend gofakes3.Backend
 	switch backend {
 	case "mem":
@@ -58,5 +59,13 @@ func s3ServerEndpoint(t *testing.T, testdir *fs.Dir, loglvl, backend string, tim
 		// after each test.
 	}
 
+	if enableProxy {
+		parsedUrl, err := url.Parse(s3srv.URL)
+		if err != nil {
+			t.Fatal(err)
+		}
+		proxyEnabledURL := "http://localhost.:" + parsedUrl.Port()
+		return proxyEnabledURL, cleanup
+	}
 	return s3srv.URL, cleanup
 }
diff --git a/e2e/util_test.go b/e2e/util_test.go
index 38efbc8..cfede70 100644
--- a/e2e/util_test.go
+++ b/e2e/util_test.go
@@ -60,6 +60,7 @@ type setupOpts struct {
 	s3backend   string
 	endpointURL string
 	timeSource  gofakes3.TimeSource
+	enableProxy bool
 }
 
 type option func(*setupOpts)
@@ -82,6 +83,12 @@ func withTimeSource(timeSource gofakes3.TimeSource) option {
 	}
 }
 
+func withProxy() option {
+	return func(opts *setupOpts) {
+		opts.enableProxy = true
+	}
+}
+
 func setup(t *testing.T, options ...option) (*s3.S3, func(...string) icmd.Cmd, func()) {
 	t.Helper()
 
@@ -124,7 +131,7 @@ func server(t *testing.T, opts *setupOpts) (string, string, func()) {
 		s3LogLevel = "info" // aws has no level other than 'debug'
 	}
 
-	endpoint, dbcleanup := s3ServerEndpoint(t, testdir, s3LogLevel, opts.s3backend, opts.timeSource)
+	endpoint, dbcleanup := s3ServerEndpoint(t, testdir, s3LogLevel, opts.s3backend, opts.timeSource, opts.enableProxy)
 	if opts.endpointURL != "" {
 		endpoint = opts.endpointURL
 	}