Repository: opensearch-project/opensearch-go
Pull request #620 (2024-09-21 23:21:49)

Fix signer receiving a drained body on retries
### Description
When retrying, the body can be drained, and there is already code that resets the request body on retries. The issue was that we were sending the drained request to the signer before reseting the request body, so signature would be based on an empty body

### Issues Resolved
#202 

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/opensearch-project/opensearch-go/pull/620?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 15.37%. Comparing base [(`06a6dc8`)](https://app.codecov.io/gh/opensearch-project/opensearch-go/commit/06a6dc8cbaeb8d1e3657487661edd4900e4f31df?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) to head [(`e485819`)](https://app.codecov.io/gh/opensearch-project/opensearch-go/commit/e485819f6155992b5f5212dc68df1a71e5c1b5ba?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project).
> Report is 68 commits behind head on main.

> :exclamation:  There is a different number of reports uploaded between BASE (06a6dc8) and HEAD (e485819). Click for more details.
> 
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (06a6dc8) | HEAD (e485819) |
>|------|------|------|
>|integration|1|0|
></details>

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##             main     #620       +/-   ##
===========================================
- Coverage   57.29%   15.37%   -41.92%     
===========================================
  Files         315      316        +1     
  Lines        9823     7751     -2072     
===========================================
- Hits         5628     1192     -4436     
- Misses       2902     6476     +3574     
+ Partials     1293       83     -1210     
```

| [Flag](https://app.codecov.io/gh/opensearch-project/opensearch-go/pull/620/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/opensearch-project/opensearch-go/pull/620/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `?` | |
| [unit](https://app.codecov.io/gh/opensearch-project/opensearch-go/pull/620/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | `15.37% <100.00%> (+2.53%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Files with missing lines](https://app.codecov.io/gh/opensearch-project/opensearch-go/pull/620?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project) | Coverage Δ | |
|---|---|---|
| [opensearchtransport/opensearchtransport.go](https://app.codecov.io/gh/opensearch-project/opensearch-go/pull/620?src=pr&el=tree&filepath=opensearchtransport%2Fopensearchtransport.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project#diff-b3BlbnNlYXJjaHRyYW5zcG9ydC9vcGVuc2VhcmNodHJhbnNwb3J0Lmdv) | `87.16% <100.00%> (+2.00%)` | :arrow_up: |

... and [312 files with indirect coverage changes](https://app.codecov.io/gh/opensearch-project/opensearch-go/pull/620/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=opensearch-project)

</details>
aouji: I'm not sure why integration tests are failing. Doesn't seem related to my change. 

BTW sorry for the mess I seem to have created in this PR. I can't re-trigger DCO and code coverage checks. Please trigger on your side or let me know if I need to close this and open a new PR
dblock: Thank you for fixing this! Let's deal with the integ test separately.

```
--- FAIL: TestNodes (1.48s)
    --- FAIL: TestNodes/Stats (0.06s)
        --- FAIL: TestNodes/Stats/without_request (0.05s)
```

You will need to fix DCO, `git commit --amend -s` and force push your branch please?

aouji: DCO is now fixed
Jakob3xD: > Any ideas why the integ test started failing?
> 
> ```
> === RUN   TestNodes/Stats/without_request
>     api_nodes_test.go:182: 
>         	Error Trace:	/home/runner/work/opensearch-go/opensearch-go/internal/test/helper.go:147
>         	            				/home/runner/work/opensearch-go/opensearch-go/opensearchapi/api_nodes_test.go:182
>         	Error:      	Should be empty, but was [{"op":"remove","path":"/nodes/tFKzdl01Ss2oOLoYJtcXcg/indices/search/request/took"}]
>         	Test:       	TestNodes/Stats/without_request
> ```

The Integration tests runs against latest which was updated 7 days ago to 2.17.  I assume the version adds the new field which is currently not present in our spec. We need to add it. Also fails in the dependency updates.
dblock: > > Any ideas why the integ test started failing?
> > ```
> > === RUN   TestNodes/Stats/without_request
> >     api_nodes_test.go:182: 
> >         	Error Trace:	/home/runner/work/opensearch-go/opensearch-go/internal/test/helper.go:147
> >         	            				/home/runner/work/opensearch-go/opensearch-go/opensearchapi/api_nodes_test.go:182
> >         	Error:      	Should be empty, but was [{"op":"remove","path":"/nodes/tFKzdl01Ss2oOLoYJtcXcg/indices/search/request/took"}]
> >         	Test:       	TestNodes/Stats/without_request
> > ```
> 
> The Integration tests runs against latest which was updated 7 days ago to 2.17. I assume the version adds the new field which is currently not present in our spec. We need to add it. Also fails in the dependency updates.

Looks right. Should we lock it against a specific version (probably)? Want to take this up?
dblock: @Jakob3xD thanks for taking care of the integ tests in #623 

@aouji rebase?

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8c7357d..db39175 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Fix ISM Transition to omitempty Conditions field ([#609](https://github.com/opensearch-project/opensearch-go/pull/609))
 - Fix ISM Allocation field types ([#609](https://github.com/opensearch-project/opensearch-go/pull/609))
 - Fix ISM Error Notification types ([#612](https://github.com/opensearch-project/opensearch-go/pull/612))
+- Fix signer receiving drained body on retries ([#620](https://github.com/opensearch-project/opensearch-go/pull/620))
 
 ### Security
 
diff --git a/opensearchtransport/opensearchtransport.go b/opensearchtransport/opensearchtransport.go
index 8e49259..8257143 100644
--- a/opensearchtransport/opensearchtransport.go
+++ b/opensearchtransport/opensearchtransport.go
@@ -292,10 +292,6 @@ func (c *Client) Perform(req *http.Request) (*http.Response, error) {
 		c.setReqURL(conn.URL, req)
 		c.setReqAuth(conn.URL, req)
 
-		if err = c.signRequest(req); err != nil {
-			return nil, fmt.Errorf("failed to sign request: %w", err)
-		}
-
 		if !c.disableRetry && i > 0 && req.Body != nil && req.Body != http.NoBody {
 			body, err := req.GetBody()
 			if err != nil {
@@ -304,6 +300,10 @@ func (c *Client) Perform(req *http.Request) (*http.Response, error) {
 			req.Body = body
 		}
 
+		if err = c.signRequest(req); err != nil {
+			return nil, fmt.Errorf("failed to sign request: %w", err)
+		}
+
 		// Set up time measures and execute the request
 		start := time.Now().UTC()
 		res, err = c.transport.RoundTrip(req)


Test patch:
diff --git a/opensearchtransport/opensearchtransport_internal_test.go b/opensearchtransport/opensearchtransport_internal_test.go
index e04a0ab..9e32531 100644
--- a/opensearchtransport/opensearchtransport_internal_test.go
+++ b/opensearchtransport/opensearchtransport_internal_test.go
@@ -64,9 +64,13 @@ type mockSigner struct {
 	SampleKey   string
 	SampleValue string
 	ReturnError bool
+	testHook    func(*http.Request)
 }
 
 func (m *mockSigner) SignRequest(req *http.Request) error {
+	if m.testHook != nil {
+		m.testHook(req)
+	}
 	if m.ReturnError {
 		return fmt.Errorf("invalid data")
 	}
@@ -732,6 +736,47 @@ func TestTransportPerformRetries(t *testing.T) {
 		}
 	})
 
+	t.Run("Signer can sign correctly during retry", func(t *testing.T) {
+		u, _ := url.Parse("https://foo.com/bar")
+		signer := mockSigner{}
+		callsToSigner := 0
+		expectedBody := "FOOBAR"
+
+		signer.testHook = func(req *http.Request) {
+			callsToSigner++
+			body, err := io.ReadAll(req.Body)
+			if err != nil {
+				panic(err)
+			}
+			if string(body) != expectedBody {
+				t.Fatalf("request %d body: expected %q, got %q", callsToSigner, expectedBody, body)
+			}
+		}
+
+		tp, _ := New(
+			Config{
+				URLs:   []*url.URL{u},
+				Signer: &signer,
+				Transport: &mockTransp{
+					RoundTripFunc: func(req *http.Request) (*http.Response, error) {
+						return &http.Response{Status: "MOCK", StatusCode: http.StatusBadGateway}, nil
+					},
+				},
+			},
+		)
+
+		req, _ := http.NewRequest(http.MethodPost, "/abc", strings.NewReader(expectedBody))
+		//nolint:bodyclose // Mock response does not have a body to close
+		_, err := tp.Perform(req)
+		if err != nil {
+			t.Fatalf("Unexpected error: %s", err)
+		}
+
+		if callsToSigner != 4 {
+			t.Fatalf("expected 4 requests, got %d", callsToSigner)
+		}
+	})
+
 	t.Run("Don't retry request on regular error", func(t *testing.T) {
 		var i int