Repository: projectdiscovery/httpx (language: go)

Issue:
`-irrb` Sometimes returns corrupted base64 body
<!-- 
1. Please search to see if an issue already exists for the bug you encountered.
2. For support requests, FAQs or "How to" questions, please use the GitHub Discussions section instead - https://github.com/projectdiscovery/httpx/discussions or
3. Join our discord server at https://discord.gg/projectdiscovery and post the question on the #httpx channel.
-->

<!-- ISSUES MISSING IMPORTANT INFORMATION MAY BE CLOSED WITHOUT INVESTIGATION. -->

### httpx version:
<!-- You can find current version of httpx with "httpx -version" -->
<!-- We only accept issues that are reproducible on the latest version of httpx. -->
<!-- You can find the latest version of project at https://github.com/projectdiscovery/httpx/releases/ -->
1.2.6

### Current Behavior:
<!-- A concise description of what you're experiencing. -->
Sometimes, binary body is altered, and decoding base64 body returns unexpected data (who looks to have been pre-encoded as utf-8)

Here is a POC:

$ `echo 'https://zc-console.taobao.com/favicon.ico' | httpx -silent -json -irrb | jq .body -j | base64 -d | wc -c`
> 2246

$ `curl -s 'https://zc-console.taobao.com/favicon.ico' | wc -c`
> 1863

Note that in this specific case, `Content-Type` header is set to `image/x-icon;charset=GBK`, so underlying http library might be automatically trying to encode data, given that `charset` is specified.  

### Expected Behavior:
<!-- A concise description of what you expected to happen. -->
`-irrb` should always returns a base64 version of the raw, original body's binary content, with no previous encoding 



### Anything else:
<!-- Links? References? Screnshots? Anything that will give us more context about the issue that you are encountering! -->

related to https://github.com/projectdiscovery/httpx/issues/813, https://github.com/projectdiscovery/httpx/issues/814


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 0c4718702..9dd2503fe 100644
--- a/README.md
+++ b/README.md
@@ -186,6 +186,7 @@ CONFIGURATIONS:
    -sd, -skip-dedupe             disable dedupe input items (only used with stream mode)
    -ldp, -leave-default-ports    leave default http/https ports in host header (eg. http://host:80 - https//host:443
    -ztls                         use ztls library with autofallback to standard one for tls13
+   -no-decode                    avoid decoding body
 
 DEBUG:
    -health-check, -hc        run diagnostic check up
diff --git a/common/httpx/httpx.go b/common/httpx/httpx.go
index 56224d950..12794e6d3 100644
--- a/common/httpx/httpx.go
+++ b/common/httpx/httpx.go
@@ -218,6 +218,10 @@ get_response:
 		return nil, closeErr
 	}
 
+	// Todo: replace with https://github.com/projectdiscovery/utils/issues/110
+	resp.RawData = make([]byte, len(respbody))
+	copy(resp.RawData, respbody)
+
 	respbody, err = DecodeData(respbody, httpresp.Header)
 	if err != nil && !shouldIgnoreBodyErrors {
 		return nil, closeErr
diff --git a/common/httpx/response.go b/common/httpx/response.go
index bf0e15f66..816551f53 100644
--- a/common/httpx/response.go
+++ b/common/httpx/response.go
@@ -12,7 +12,8 @@ import (
 type Response struct {
 	StatusCode    int
 	Headers       map[string][]string
-	Data          []byte
+	RawData       []byte // undecoded data
+	Data          []byte // decoded data
 	ContentLength int
 	Raw           string
 	RawHeaders    string
diff --git a/runner/options.go b/runner/options.go
index d231c5dc7..4e46e573b 100644
--- a/runner/options.go
+++ b/runner/options.go
@@ -262,6 +262,7 @@ type Options struct {
 	OutputMatchCondition      string
 	OnResult                  OnResultCallback
 	DisableUpdateCheck        bool
+	NoDecode                  bool
 }
 
 // ParseOptions parses the command line options for application
@@ -388,6 +389,7 @@ func ParseOptions() *Options {
 		flagSet.BoolVarP(&options.SkipDedupe, "skip-dedupe", "sd", false, "disable dedupe input items (only used with stream mode)"),
 		flagSet.BoolVarP(&options.LeaveDefaultPorts, "leave-default-ports", "ldp", false, "leave default http/https ports in host header (eg. http://host:80 - https//host:443"),
 		flagSet.BoolVar(&options.ZTLS, "ztls", false, "use ztls library with autofallback to standard one for tls13"),
+		flagSet.BoolVar(&options.NoDecode, "no-decode", false, "avoid decoding body"),
 	)
 
 	flagSet.CreateGroup("debug", "Debug",
diff --git a/runner/runner.go b/runner/runner.go
index 0736cd6b1..e536d50fb 100644
--- a/runner/runner.go
+++ b/runner/runner.go
@@ -1301,17 +1301,25 @@ retry:
 		builder.WriteString(fmt.Sprintf(" [%s]", serverHeader))
 	}
 
-	var serverResponseRaw string
-	var request string
-	var rawResponseHeader string
-	var responseHeader map[string]interface{}
+	var (
+		serverResponseRaw string
+		request           string
+		rawResponseHeader string
+		responseHeader    map[string]interface{}
+	)
+
+	respData := string(resp.Data)
+	if r.options.NoDecode {
+		respData = string(resp.RawData)
+	}
+
 	if scanopts.ResponseInStdout {
-		serverResponseRaw = string(resp.Data)
+		serverResponseRaw = string(respData)
 		request = string(requestDump)
 		responseHeader = normalizeHeaders(resp.Headers)
 		rawResponseHeader = resp.RawHeaders
 	} else if scanopts.Base64ResponseInStdout {
-		serverResponseRaw = stringz.Base64(resp.Data)
+		serverResponseRaw = stringz.Base64([]byte(respData))
 		request = stringz.Base64(requestDump)
 		responseHeader = normalizeHeaders(resp.Headers)
 		rawResponseHeader = stringz.Base64([]byte(resp.RawHeaders))


Test patch: