Repository: cloudwego/hertz
Pull request #239 (2022-09-13 13:30:30)

feat: add retry extention for client
#### What type of PR is this?
feat

#### Check the PR title.
<!--
The description of the title will be attached in Release Notes, 
so please describe it from user-oriented, what this PR does / why we need it.
Please check your PR title with the below requirements:
-->
- [x] This PR title match the format: \<type\>(optional scope): \<description\>
- [x] The description of this PR title is user-oriented and clear enough for others to understand.


#### (Optional) Translate the PR title into Chinese.
给 client 扩展重试能力


#### (Optional) More detail description for this PR(en: English/zh: Chinese).
<!--
Provide more detailed info for review. If it is a perf type PR, perf data is suggested to give.
-->
en: Various delay backoff strategies are introduced, and the user-defined retry conditions.

Through client SetRetryConfig (RetryConfig) injects retry config and fails to pass the client WithRetryConfig (), otherwise the circular reference seems to be unable to be solved. For the initialization of RetryConfig, use the option mode of withXXX().
Users can customize the RetryIf function through req, resp and err to realize customized retry conditions.
Introduce four delay strategies: default, random, fixed and exponential. Users can customize the combination
The default value of retryconfig remains to be discussed
The default retry condition feels to be optimized. pkg/common/retry/retry.go::DefaultRetryIf()

zh:引入了各种延迟退避策略，用户自定义重试发生条件。
1.用户通过client.SetRetryConfig(RetryConfig)注入重试配置，没有通过client.withRetryConfig()的方式，否则的话循环引用好像不能被解决。对于retryConfig的初始化使用withXXX()的选项模式。
2.用户可以通过req、resp和err来自定义RetryIf函数，实现自定义重试条件。
3.引入 默认、随机、固定、指数 四种延迟策略，用户可以自定义组合
4.retryConfig的默认值还有待商榷
5.默认重试条件感觉还有待优化。pkg/common/retry/retry.go::DefaultRetryIf()

ps：相关文档 https://bytedancecampus1.feishu.cn/docx/doxcnfQGEZeVgZFkJf4nfcazGbg ，完善中

#### Which issue(s) this PR fixes:
closes: #63 

codecov[bot]: # [Codecov](https://codecov.io/gh/cloudwego/hertz/pull/239?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego) Report
Base: **59.14**% // Head: **59.22**% // Increases project coverage by **`+0.08%`** :tada:
> Coverage data is based on head [(`1d9fa5e`)](https://codecov.io/gh/cloudwego/hertz/pull/239?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego) compared to base [(`f7e03d0`)](https://codecov.io/gh/cloudwego/hertz/commit/f7e03d0603ef1ba33c3d1012381778ffe72fd770?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego).
> Patch coverage: 57.77% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           develop     #239      +/-   ##
===========================================
+ Coverage    59.14%   59.22%   +0.08%     
===========================================
  Files           79       79              
  Lines         8111     8125      +14     
===========================================
+ Hits          4797     4812      +15     
+ Misses        2960     2958       -2     
- Partials       354      355       +1     
```


| [Impacted Files](https://codecov.io/gh/cloudwego/hertz/pull/239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego) | Coverage Δ | |
|---|---|---|
| [pkg/common/bytebufferpool/bytebuffer.go](https://codecov.io/gh/cloudwego/hertz/pull/239/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego#diff-cGtnL2NvbW1vbi9ieXRlYnVmZmVycG9vbC9ieXRlYnVmZmVyLmdv) | `68.75% <ø> (ø)` | |
| [pkg/common/config/client\_option.go](https://codecov.io/gh/cloudwego/hertz/pull/239/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego#diff-cGtnL2NvbW1vbi9jb25maWcvY2xpZW50X29wdGlvbi5nbw==) | `0.00% <0.00%> (ø)` | |
| [pkg/protocol/header.go](https://codecov.io/gh/cloudwego/hertz/pull/239/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego#diff-cGtnL3Byb3RvY29sL2hlYWRlci5nbw==) | `55.29% <0.00%> (-0.15%)` | :arrow_down: |
| [pkg/protocol/http1/client.go](https://codecov.io/gh/cloudwego/hertz/pull/239/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego#diff-cGtnL3Byb3RvY29sL2h0dHAxL2NsaWVudC5nbw==) | `35.86% <45.83%> (+0.31%)` | :arrow_up: |
| [pkg/app/client/client.go](https://codecov.io/gh/cloudwego/hertz/pull/239/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego#diff-cGtnL2FwcC9jbGllbnQvY2xpZW50Lmdv) | `73.61% <100.00%> (+1.38%)` | :arrow_up: |
| [pkg/app/client/option.go](https://codecov.io/gh/cloudwego/hertz/pull/239/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego#diff-cGtnL2FwcC9jbGllbnQvb3B0aW9uLmdv) | `100.00% <100.00%> (ø)` | |
| [pkg/network/standard/buffer.go](https://codecov.io/gh/cloudwego/hertz/pull/239/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego#diff-cGtnL25ldHdvcmsvc3RhbmRhcmQvYnVmZmVyLmdv) | `81.08% <0.00%> (-0.50%)` | :arrow_down: |
| [pkg/common/compress/compress.go](https://codecov.io/gh/cloudwego/hertz/pull/239/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego#diff-cGtnL2NvbW1vbi9jb21wcmVzcy9jb21wcmVzcy5nbw==) | `83.80% <0.00%> (+2.85%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/cloudwego/hertz/pull/239?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cloudwego).


Patch:
diff --git a/pkg/app/client/client.go b/pkg/app/client/client.go
index 92514ae..37dc29b 100644
--- a/pkg/app/client/client.go
+++ b/pkg/app/client/client.go
@@ -243,10 +243,8 @@ type Client struct {
 	// If Proxy is nil or returns a nil *URL, no proxy is used.
 	Proxy protocol.Proxy
 
-	// RetryIf controls whether a retry should be attempted after an error.
-	//
-	// By default will use isIdempotent function
-	RetryIf func(request *protocol.Request) bool
+	// RetryIfFunc sets the retry decision function. If nil, the client.DefaultRetryIf will be applied.
+	RetryIfFunc client.RetryIfFunc
 
 	clientFactory suite.ClientFactory
 
@@ -260,6 +258,10 @@ func (c *Client) GetOptions() *config.ClientOptions {
 	return c.options
 }
 
+func (c *Client) SetRetryIfFunc(retryIf client.RetryIfFunc) {
+	c.RetryIfFunc = retryIf
+}
+
 // SetProxy is used to set client proxy.
 //
 // Don't SetProxy twice for a client.
@@ -268,11 +270,6 @@ func (c *Client) SetProxy(p protocol.Proxy) {
 	c.Proxy = p
 }
 
-// SetRetryIf is used to set RetryIf func.
-func (c *Client) SetRetryIf(fn func(request *protocol.Request) bool) {
-	c.RetryIf = fn
-}
-
 // Get returns the status code and body of url.
 //
 // The contents of dst will be replaced by the body and returned, if the dst
@@ -594,14 +591,14 @@ func newHttp1OptionFromClient(c *Client) *http1.ClientOptions {
 		MaxConns:                      c.options.MaxConnsPerHost,
 		MaxConnDuration:               c.options.MaxConnDuration,
 		MaxIdleConnDuration:           c.options.MaxIdleConnDuration,
-		MaxIdempotentCallAttempts:     c.options.MaxIdempotentCallAttempts,
 		ReadTimeout:                   c.options.ReadTimeout,
 		WriteTimeout:                  c.options.WriteTimeout,
 		MaxResponseBodySize:           c.options.MaxResponseBodySize,
 		DisableHeaderNamesNormalizing: c.options.DisableHeaderNamesNormalizing,
 		DisablePathNormalizing:        c.options.DisablePathNormalizing,
 		MaxConnWaitTimeout:            c.options.MaxConnWaitTimeout,
-		RetryIf:                       c.RetryIf,
 		ResponseBodyStream:            c.options.ResponseBodyStream,
+		RetryConfig:                   c.options.RetryConfig,
+		RetryIfFunc:                   c.RetryIfFunc,
 	}
 }
diff --git a/pkg/app/client/option.go b/pkg/app/client/option.go
index 80a74cc..343ce62 100644
--- a/pkg/app/client/option.go
+++ b/pkg/app/client/option.go
@@ -20,9 +20,11 @@ import (
 	"crypto/tls"
 	"time"
 
+	"github.com/cloudwego/hertz/pkg/app/client/retry"
 	"github.com/cloudwego/hertz/pkg/common/config"
 	"github.com/cloudwego/hertz/pkg/network"
 	"github.com/cloudwego/hertz/pkg/network/standard"
+	"github.com/cloudwego/hertz/pkg/protocol/consts"
 )
 
 // WithDialTimeout sets dial timeout.
@@ -67,13 +69,6 @@ func WithKeepAlive(b bool) config.ClientOption {
 	}}
 }
 
-// WithMaxIdempotentCallAttempts sets maximum number of attempts for idempotent calls.
-func WithMaxIdempotentCallAttempts(n int) config.ClientOption {
-	return config.ClientOption{F: func(o *config.ClientOptions) {
-		o.MaxIdempotentCallAttempts = n
-	}}
-}
-
 // WithClientReadTimeout sets maximum duration for full response reading (including body).
 func WithClientReadTimeout(t time.Duration) config.ClientOption {
 	return config.ClientOption{F: func(o *config.ClientOptions) {
@@ -130,3 +125,18 @@ func WithDisablePathNormalizing(isDisablePathNormalizing bool) config.ClientOpti
 		o.DisablePathNormalizing = isDisablePathNormalizing
 	}}
 }
+
+func WithRetryConfig(opts ...retry.Option) config.ClientOption {
+	retryCfg := &retry.Config{
+		MaxAttemptTimes: consts.DefaultMaxRetryTimes,
+		Delay:           1 * time.Millisecond,
+		MaxDelay:        100 * time.Millisecond,
+		MaxJitter:       20 * time.Millisecond,
+		DelayPolicy:     retry.CombineDelay(retry.DefaultDelayPolicy),
+	}
+	retryCfg.Apply(opts)
+
+	return config.ClientOption{F: func(o *config.ClientOptions) {
+		o.RetryConfig = retryCfg
+	}}
+}
diff --git a/pkg/app/client/retry/option.go b/pkg/app/client/retry/option.go
new file mode 100644
index 0000000..bfaa9da
--- /dev/null
+++ b/pkg/app/client/retry/option.go
@@ -0,0 +1,59 @@
+/*
+ * Copyright 2022 CloudWeGo Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package retry
+
+import "time"
+
+// Option is the only struct that can be used to set Retry Config.
+type Option struct {
+	F func(o *Config)
+}
+
+// WithMaxAttemptTimes set WithMaxAttemptTimes , including the first call.
+func WithMaxAttemptTimes(maxAttemptTimes uint) Option {
+	return Option{F: func(o *Config) {
+		o.MaxAttemptTimes = maxAttemptTimes
+	}}
+}
+
+// WithInitDelay set init Delay.
+func WithInitDelay(delay time.Duration) Option {
+	return Option{F: func(o *Config) {
+		o.Delay = delay
+	}}
+}
+
+// WithMaxDelay set MaxDelay.
+func WithMaxDelay(maxDelay time.Duration) Option {
+	return Option{F: func(o *Config) {
+		o.MaxDelay = maxDelay
+	}}
+}
+
+// WithDelayPolicy set DelayPolicy.
+func WithDelayPolicy(delayPolicy DelayPolicyFunc) Option {
+	return Option{F: func(o *Config) {
+		o.DelayPolicy = delayPolicy
+	}}
+}
+
+// WithMaxJitter set MaxJitter.
+func WithMaxJitter(maxJitter time.Duration) Option {
+	return Option{F: func(o *Config) {
+		o.MaxJitter = maxJitter
+	}}
+}
diff --git a/pkg/app/client/retry/retry.go b/pkg/app/client/retry/retry.go
new file mode 100644
index 0000000..f866d05
--- /dev/null
+++ b/pkg/app/client/retry/retry.go
@@ -0,0 +1,104 @@
+/*
+ * Copyright 2022 CloudWeGo Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package retry
+
+import (
+	"math"
+	"math/rand"
+	"time"
+)
+
+// Config All configurations related to retry
+type Config struct {
+	// The maximum number of call attempt times, including the initial call
+	MaxAttemptTimes uint
+
+	// Initial retry delay time
+	Delay time.Duration
+
+	// Maximum retry delay time. When the retry time increases beyond this time,
+	// this configuration will limit the upper limit of waiting time
+	MaxDelay time.Duration
+
+	// The maximum jitter time, which takes effect when the delay policy is configured as RandomDelay
+	MaxJitter time.Duration
+
+	// Delay strategy, which can combine multiple delay strategies. such as CombineDelay(BackOffDelayPolicy, RandomDelayPolicy) or BackOffDelayPolicy,etc
+	DelayPolicy DelayPolicyFunc
+}
+
+func (o *Config) Apply(opts []Option) {
+	for _, op := range opts {
+		op.F(o)
+	}
+}
+
+// DelayPolicyFunc signature of delay policy function
+// is called to return the delay of retry
+type DelayPolicyFunc func(attempts uint, err error, retryConfig *Config) time.Duration
+
+// DefaultDelayPolicy is a DelayPolicyFunc which keep 0 delay in all iterations
+func DefaultDelayPolicy(_ uint, _ error, _ *Config) time.Duration {
+	return 0 * time.Millisecond
+}
+
+// FixedDelayPolicy is a DelayPolicyFunc which keeps delay the same through all iterations
+func FixedDelayPolicy(_ uint, _ error, retryConfig *Config) time.Duration {
+	return retryConfig.Delay
+}
+
+// RandomDelayPolicy is a DelayPolicyFunc which picks a random delay up toRetryConfig.MaxJitter
+func RandomDelayPolicy(_ uint, _ error, retryConfig *Config) time.Duration {
+	return time.Duration(rand.Int63n(int64(retryConfig.MaxJitter)))
+}
+
+// BackOffDelayPolicy is a DelayPolicyFunc which exponentially increases delay between consecutive retries
+func BackOffDelayPolicy(attempts uint, _ error, retryConfig *Config) time.Duration {
+	// 1 << 63 would overflow signed int64 (time.Duration), thus 62.
+	const max uint = 62
+	if attempts > max {
+		attempts = max
+	}
+
+	return retryConfig.Delay << attempts
+}
+
+// CombineDelay return DelayPolicyFunc, which combines the optional DelayPolicyFunc into a new DelayPolicyFunc
+func CombineDelay(delays ...DelayPolicyFunc) DelayPolicyFunc {
+	const maxInt64 = uint64(math.MaxInt64)
+
+	return func(attempts uint, err error, config *Config) time.Duration {
+		var total uint64
+		for _, delay := range delays {
+			total += uint64(delay(attempts, err, config))
+			if total > maxInt64 {
+				total = maxInt64
+			}
+		}
+
+		return time.Duration(total)
+	}
+}
+
+// Delay generate the delay time required for the current retry
+func Delay(attempts uint, err error, retryConfig *Config) time.Duration {
+	delayTime := retryConfig.DelayPolicy(attempts, err, retryConfig)
+	if retryConfig.MaxDelay > 0 && delayTime > retryConfig.MaxDelay {
+		delayTime = retryConfig.MaxDelay
+	}
+	return delayTime
+}
diff --git a/pkg/common/bytebufferpool/bytebuffer.go b/pkg/common/bytebufferpool/bytebuffer.go
index 8783aa4..6561746 100644
--- a/pkg/common/bytebufferpool/bytebuffer.go
+++ b/pkg/common/bytebufferpool/bytebuffer.go
@@ -51,7 +51,6 @@ import "io"
 //
 // Use Get for obtaining an empty byte buffer.
 type ByteBuffer struct {
-
 	// B is a byte buffer to use in append-like workloads.
 	// See example code for details.
 	B []byte
diff --git a/pkg/common/config/client_option.go b/pkg/common/config/client_option.go
index ec6f5e6..8b44d24 100644
--- a/pkg/common/config/client_option.go
+++ b/pkg/common/config/client_option.go
@@ -20,6 +20,7 @@ import (
 	"crypto/tls"
 	"time"
 
+	"github.com/cloudwego/hertz/pkg/app/client/retry"
 	"github.com/cloudwego/hertz/pkg/network"
 	"github.com/cloudwego/hertz/pkg/protocol/consts"
 )
@@ -35,14 +36,13 @@ type ClientOptions struct {
 	// The max connection nums for each host
 	MaxConnsPerHost int
 
-	MaxIdleConnDuration       time.Duration
-	MaxConnDuration           time.Duration
-	MaxConnWaitTimeout        time.Duration
-	MaxIdempotentCallAttempts int
-	KeepAlive                 bool
-	ReadTimeout               time.Duration
-	TLSConfig                 *tls.Config
-	ResponseBodyStream        bool
+	MaxIdleConnDuration time.Duration
+	MaxConnDuration     time.Duration
+	MaxConnWaitTimeout  time.Duration
+	KeepAlive           bool
+	ReadTimeout         time.Duration
+	TLSConfig           *tls.Config
+	ResponseBodyStream  bool
 
 	// Client name. Used in User-Agent request header.
 	//
@@ -103,15 +103,17 @@ type ClientOptions struct {
 	// By default path values are normalized, i.e.
 	// extra slashes are removed, special characters are encoded.
 	DisablePathNormalizing bool
+
+	// all configurations related to retry
+	RetryConfig *retry.Config
 }
 
 func NewClientOptions(opts []ClientOption) *ClientOptions {
 	options := &ClientOptions{
-		DialTimeout:               consts.DefaultDialTimeout,
-		MaxConnsPerHost:           consts.DefaultMaxConnsPerHost,
-		MaxIdleConnDuration:       consts.DefaultMaxIdleConnDuration,
-		MaxIdempotentCallAttempts: consts.DefaultMaxIdempotentCallAttempts,
-		KeepAlive:                 true,
+		DialTimeout:         consts.DefaultDialTimeout,
+		MaxConnsPerHost:     consts.DefaultMaxConnsPerHost,
+		MaxIdleConnDuration: consts.DefaultMaxIdleConnDuration,
+		KeepAlive:           true,
 	}
 	options.Apply(opts)
 
diff --git a/pkg/protocol/client/client.go b/pkg/protocol/client/client.go
index e5e6cf1..33d7c39 100644
--- a/pkg/protocol/client/client.go
+++ b/pkg/protocol/client/client.go
@@ -43,6 +43,7 @@ package client
 
 import (
 	"context"
+	"io"
 	"sync"
 	"time"
 
@@ -78,6 +79,40 @@ type Doer interface {
 	Do(ctx context.Context, req *protocol.Request, resp *protocol.Response) error
 }
 
+// DefaultRetryIf Default retry condition, mainly used for idempotent requests.
+// If this cannot be satisfied, you can implement your own retry condition.
+func DefaultRetryIf(req *protocol.Request, resp *protocol.Response, err error) bool {
+	// cannot retry if the request body is not rewindable
+	if req.IsBodyStream() {
+		return false
+	}
+
+	if isIdempotent(req, resp, err) {
+		return true
+	}
+	// Retry non-idempotent requests if the server closes
+	// the connection before sending the response.
+	//
+	// This case is possible if the server closes the idle
+	// keep-alive connection on timeout.
+	//
+	// Apache and nginx usually do this.
+	if err == io.EOF {
+		return true
+	}
+
+	return false
+}
+
+func isIdempotent(req *protocol.Request, resp *protocol.Response, err error) bool {
+	return req.Header.IsGet() ||
+		req.Header.IsHead() ||
+		req.Header.IsPut() ||
+		req.Header.IsDelete() ||
+		req.Header.IsOptions() ||
+		req.Header.IsTrace()
+}
+
 // DynamicConfig is config set which will be confirmed when starts a request.
 type DynamicConfig struct {
 	Addr     string
@@ -86,9 +121,8 @@ type DynamicConfig struct {
 }
 
 // RetryIfFunc signature of retry if function
-//
-// Request argument passed to RetryIfFunc, if there are any request errors.
-type RetryIfFunc func(request *protocol.Request) bool
+// Judge whether to retry by request,response or error , return true is retry
+type RetryIfFunc func(req *protocol.Request, resp *protocol.Response, err error) bool
 
 type clientURLResponse struct {
 	statusCode int
diff --git a/pkg/protocol/consts/default.go b/pkg/protocol/consts/default.go
index ae8fbc2..ae76074 100644
--- a/pkg/protocol/consts/default.go
+++ b/pkg/protocol/consts/default.go
@@ -67,4 +67,7 @@ const (
 
 	// DefaultMaxIdempotentCallAttempts is the default idempotent calls attempts count.
 	DefaultMaxIdempotentCallAttempts = 1
+
+	// DefaultMaxRetryTimes is the default call times of retry
+	DefaultMaxRetryTimes = 1
 )
diff --git a/pkg/protocol/header.go b/pkg/protocol/header.go
index bb2e033..9291d5f 100644
--- a/pkg/protocol/header.go
+++ b/pkg/protocol/header.go
@@ -424,6 +424,11 @@ func (h *RequestHeader) IsPost() bool {
 	return bytes.Equal(h.Method(), bytestr.StrPost)
 }
 
+// IsDelete returns true if request method is DELETE.
+func (h *RequestHeader) IsDelete() bool {
+	return bytes.Equal(h.Method(), bytestr.StrDelete)
+}
+
 // IsConnect returns true if request method is CONNECT.
 func (h *RequestHeader) IsConnect() bool {
 	return bytes.Equal(h.Method(), bytestr.StrConnect)
@@ -1063,6 +1068,16 @@ func (h *RequestHeader) IsGet() bool {
 	return bytes.Equal(h.Method(), bytestr.StrGet)
 }
 
+// IsOptions returns true if request method is Options.
+func (h *RequestHeader) IsOptions() bool {
+	return bytes.Equal(h.Method(), bytestr.StrOptions)
+}
+
+// IsTrace returns true if request method is Trace.
+func (h *RequestHeader) IsTrace() bool {
+	return bytes.Equal(h.Method(), bytestr.StrTrace)
+}
+
 // SetHostBytes sets Host header value.
 func (h *RequestHeader) SetHostBytes(host []byte) {
 	h.host = append(h.host[:0], host...)
diff --git a/pkg/protocol/http1/client.go b/pkg/protocol/http1/client.go
index 4be7be1..f310ee6 100644
--- a/pkg/protocol/http1/client.go
+++ b/pkg/protocol/http1/client.go
@@ -56,6 +56,7 @@ import (
 	"github.com/cloudwego/hertz/internal/bytesconv"
 	"github.com/cloudwego/hertz/internal/bytestr"
 	"github.com/cloudwego/hertz/internal/nocopy"
+	"github.com/cloudwego/hertz/pkg/app/client/retry"
 	errs "github.com/cloudwego/hertz/pkg/common/errors"
 	"github.com/cloudwego/hertz/pkg/common/hlog"
 	"github.com/cloudwego/hertz/pkg/common/timer"
@@ -325,45 +326,54 @@ func (c *HostClient) DoRedirects(ctx context.Context, req *protocol.Request, res
 // It is recommended obtaining req and resp via AcquireRequest
 // and AcquireResponse in performance-critical code.
 func (c *HostClient) Do(ctx context.Context, req *protocol.Request, resp *protocol.Response) error {
-	var err error
-	var retry bool
+	var (
+		err                error
+		canIdempotentRetry bool
+		isDefaultRetryFunc                    = true
+		attempts           uint               = 0
+		maxAttempts        uint               = 1
+		isRequestRetryable client.RetryIfFunc = client.DefaultRetryIf
+	)
+	retryCfg := c.ClientOptions.RetryConfig
+	if retryCfg != nil {
+		maxAttempts = retryCfg.MaxAttemptTimes
+	}
 
-	maxAttempts := c.MaxIdempotentCallAttempts
-	isRequestRetryable := isIdempotent
-	if c.RetryIf != nil {
-		isRequestRetryable = c.RetryIf
+	if c.ClientOptions.RetryIfFunc != nil {
+		isRequestRetryable = c.ClientOptions.RetryIfFunc
+		// if the user has provided a custom retry function, the canIdempotentRetry has no meaning anymore.
+		// User will have full control over the retry logic through the custom retry function.
+		isDefaultRetryFunc = false
 	}
-	attempts := 0
 
 	atomic.AddInt32(&c.pendingRequests, 1)
+
 	for {
-		retry, err = c.do(req, resp)
-		if err == nil || !retry {
+		canIdempotentRetry, err = c.do(req, resp)
+		if err == nil {
 			break
 		}
 
+		if isDefaultRetryFunc {
+			// canIdempotentRetry only makes sense if the user hasn't provided a custom retry function.
+			if !canIdempotentRetry {
+				break
+			}
+		}
+
 		attempts++
 		if attempts >= maxAttempts {
 			break
 		}
 
-		if req.IsBodyStream() {
+		// Check whether this request should be retried
+		if !isRequestRetryable(req, resp, err) {
 			break
 		}
 
-		if !isRequestRetryable(req) {
-			// Retry non-idempotent requests if the server closes
-			// the connection before sending the response.
-			//
-			// This case is possible if the server closes the idle
-			// keep-alive connection on timeout.
-			//
-			// Apache and nginx usually do this.
-			if err != io.EOF {
-				break
-			}
-		}
-
+		wait := retry.Delay(attempts, err, retryCfg)
+		// Retry after wait time
+		time.Sleep(wait)
 	}
 	atomic.AddInt32(&c.pendingRequests, -1)
 
@@ -382,10 +392,6 @@ func (c *HostClient) PendingRequests() int {
 	return int(atomic.LoadInt32(&c.pendingRequests))
 }
 
-func isIdempotent(req *protocol.Request) bool {
-	return req.Header.IsGet() || req.Header.IsHead() || req.Header.IsPut()
-}
-
 func (c *HostClient) do(req *protocol.Request, resp *protocol.Response) (bool, error) {
 	nilResp := false
 	if resp == nil {
@@ -393,13 +399,13 @@ func (c *HostClient) do(req *protocol.Request, resp *protocol.Response) (bool, e
 		resp = protocol.AcquireResponse()
 	}
 
-	ok, err := c.doNonNilReqResp(req, resp)
+	canIdempotentRetry, err := c.doNonNilReqResp(req, resp)
 
 	if nilResp {
 		protocol.ReleaseResponse(resp)
 	}
 
-	return ok, err
+	return canIdempotentRetry, err
 }
 
 func (c *HostClient) doNonNilReqResp(req *protocol.Request, resp *protocol.Response) (bool, error) {
@@ -424,6 +430,7 @@ func (c *HostClient) doNonNilReqResp(req *protocol.Request, resp *protocol.Respo
 		req.URI().DisablePathNormalizing = true
 	}
 	cc, err := c.acquireConn()
+	// if getting connection error, fast fail
 	if err != nil {
 		return false, err
 	}
@@ -444,6 +451,7 @@ func (c *HostClient) doNonNilReqResp(req *protocol.Request, resp *protocol.Respo
 		currentTime := time.Now()
 		if err = conn.SetWriteDeadline(currentTime.Add(c.WriteTimeout)); err != nil {
 			c.closeConn(cc)
+			// try another connection if retry is enabled
 			return true, err
 		}
 	}
@@ -472,6 +480,7 @@ func (c *HostClient) doNonNilReqResp(req *protocol.Request, resp *protocol.Respo
 	if err == nil {
 		err = zw.Flush()
 	}
+	// error happened when writing request, close the connection, and try another connection if retry is enabled
 	if err != nil {
 		c.closeConn(cc)
 		return true, err
@@ -482,6 +491,7 @@ func (c *HostClient) doNonNilReqResp(req *protocol.Request, resp *protocol.Respo
 		// See https://github.com/golang/go/issues/15133#issuecomment-271571395 for details
 		if err = conn.SetReadTimeout(c.ReadTimeout); err != nil {
 			c.closeConn(cc)
+			// try another connection if retry is enabled
 			return true, err
 		}
 	}
@@ -1108,11 +1118,6 @@ type ClientOptions struct {
 	// after DefaultMaxIdleConnDuration.
 	MaxIdleConnDuration time.Duration
 
-	// Maximum number of attempts for idempotent calls
-	//
-	// DefaultMaxIdempotentCallAttempts is used if not set.
-	MaxIdempotentCallAttempts int
-
 	// Maximum duration for full response reading (including body).
 	//
 	// By default response read timeout is unlimited.
@@ -1161,11 +1166,11 @@ type ClientOptions struct {
 	// By default will not wait, return errNoFreeConns immediately
 	MaxConnWaitTimeout time.Duration
 
-	// RetryIf controls whether a retry should be attempted after an error.
-	//
-	// By default will use isIdempotent function
-	RetryIf client.RetryIfFunc
-
 	// ResponseBodyStream enables response body streaming
 	ResponseBodyStream bool
+
+	// All configurations related to retry
+	RetryConfig *retry.Config
+
+	RetryIfFunc client.RetryIfFunc
 }


Test patch:
diff --git a/pkg/app/client/client_test.go b/pkg/app/client/client_test.go
index bd5464c..8c86aff 100644
--- a/pkg/app/client/client_test.go
+++ b/pkg/app/client/client_test.go
@@ -65,6 +65,7 @@ import (
 
 	"github.com/cloudwego/hertz/internal/bytestr"
 	"github.com/cloudwego/hertz/pkg/app"
+	"github.com/cloudwego/hertz/pkg/app/client/retry"
 	"github.com/cloudwego/hertz/pkg/common/config"
 	errs "github.com/cloudwego/hertz/pkg/common/errors"
 	"github.com/cloudwego/hertz/pkg/network"
@@ -389,9 +390,9 @@ func TestClientReadTimeout(t *testing.T) {
 
 	c := &http1.HostClient{
 		ClientOptions: &http1.ClientOptions{
-			ReadTimeout:               time.Second * 4,
-			MaxIdempotentCallAttempts: 1,
-			Dialer:                    standard.NewDialer(),
+			ReadTimeout: time.Second * 4,
+			RetryConfig: &retry.Config{MaxAttemptTimes: 1},
+			Dialer:      standard.NewDialer(),
 		},
 		Addr: opt.Addr,
 	}
@@ -434,7 +435,7 @@ func TestClientReadTimeout(t *testing.T) {
 	case <-done:
 		// It is abnormal when waiting time exceeds the value of readTimeout times the number of retries.
 		// Give it extra 2 seconds just to be sure.
-	case <-time.After(c.ReadTimeout*time.Duration(c.MaxIdempotentCallAttempts) + time.Second*2):
+	case <-time.After(c.ReadTimeout*time.Duration(c.RetryConfig.MaxAttemptTimes) + time.Second*2):
 		t.Fatal("Client.ReadTimeout didn't work")
 	}
 }
@@ -1880,6 +1881,128 @@ func newMockDialerWithCustomFunc(network, address string, timeout time.Duration,
 	}
 }
 
+func TestClientRetry(t *testing.T) {
+	t.Parallel()
+	client, err := NewClient(
+		WithDialTimeout(2*time.Second),
+		WithRetryConfig(
+			retry.WithMaxAttemptTimes(3),
+			retry.WithInitDelay(100*time.Millisecond),
+			retry.WithMaxDelay(10*time.Second),
+			retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy)),
+		),
+	)
+	client.SetRetryIfFunc(func(req *protocol.Request, resp *protocol.Response, err error) bool {
+		return err != nil
+	})
+	if err != nil {
+		t.Fatal(err)
+		return
+	}
+	startTime := time.Now().UnixNano()
+	_, resp, err := client.Get(context.Background(), nil, "http://127.0.0.1:1234/ping")
+	if err != nil {
+		// first delay 100+200ms , second delay 100+400ms
+		if time.Duration(time.Now().UnixNano()-startTime) > 800*time.Millisecond && time.Duration(time.Now().UnixNano()-startTime) < 2*time.Second {
+			t.Logf("Retry triggered : delay=%dms\tresp=%v\terr=%v\n", time.Duration(time.Now().UnixNano()-startTime)/(1*time.Millisecond), string(resp), fmt.Sprintln(err))
+		} else if time.Duration(time.Now().UnixNano()-startTime) < 1*time.Second { // Compatible without triggering retry
+			t.Logf("Retry not triggered : delay=%dms\tresp=%v\terr=%v\n", time.Duration(time.Now().UnixNano()-startTime)/(1*time.Millisecond), string(resp), fmt.Sprintln(err))
+		} else {
+			t.Fatal(err)
+		}
+	}
+
+	client2, err := NewClient(
+		WithDialTimeout(2*time.Second),
+		WithRetryConfig(
+			retry.WithMaxAttemptTimes(2),
+			retry.WithInitDelay(500*time.Millisecond),
+			retry.WithMaxJitter(1*time.Second),
+			retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy)),
+		),
+	)
+	if err != nil {
+		t.Fatal(err)
+		return
+	}
+	client2.SetRetryIfFunc(func(req *protocol.Request, resp *protocol.Response, err error) bool {
+		return err != nil
+	})
+	startTime = time.Now().UnixNano()
+	_, resp, err = client2.Get(context.Background(), nil, "http://127.0.0.1:1234/ping")
+	if err != nil {
+		// delay max{500ms+rand([0,1))s,100ms}. Because if the MaxDelay is not set, we will use the default MaxDelay of 100ms
+		if time.Duration(time.Now().UnixNano()-startTime) > 100*time.Millisecond && time.Duration(time.Now().UnixNano()-startTime) < 1100*time.Millisecond {
+			t.Logf("Retry triggered : delay=%dms\tresp=%v\terr=%v\n", time.Duration(time.Now().UnixNano()-startTime)/(1*time.Millisecond), string(resp), fmt.Sprintln(err))
+		} else if time.Duration(time.Now().UnixNano()-startTime) < 1*time.Second { // Compatible without triggering retry
+			t.Logf("Retry not triggered : delay=%dms\tresp=%v\terr=%v\n", time.Duration(time.Now().UnixNano()-startTime)/(1*time.Millisecond), string(resp), fmt.Sprintln(err))
+		} else {
+			t.Fatal(err)
+		}
+	}
+
+	client3, err := NewClient(
+		WithDialTimeout(2*time.Second),
+		WithRetryConfig(
+			retry.WithMaxAttemptTimes(2),
+			retry.WithInitDelay(100*time.Millisecond),
+			retry.WithMaxDelay(5*time.Second),
+			retry.WithMaxJitter(1*time.Second),
+			retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy, retry.RandomDelayPolicy)),
+		),
+	)
+	if err != nil {
+		t.Fatal(err)
+		return
+	}
+	client3.SetRetryIfFunc(func(req *protocol.Request, resp *protocol.Response, err error) bool {
+		return err != nil
+	})
+	startTime = time.Now().UnixNano()
+	_, resp, err = client3.Get(context.Background(), nil, "http://127.0.0.1:1234/ping")
+	if err != nil {
+		// delay 100ms+200ms+rand([0,1))s
+		if time.Duration(time.Now().UnixNano()-startTime) > 300*time.Millisecond && time.Duration(time.Now().UnixNano()-startTime) < 2300*time.Millisecond {
+			t.Logf("Retry triggered : delay=%dms\tresp=%v\terr=%v\n", time.Duration(time.Now().UnixNano()-startTime)/(1*time.Millisecond), string(resp), fmt.Sprintln(err))
+		} else if time.Duration(time.Now().UnixNano()-startTime) < 1*time.Second { // Compatible without triggering retry
+			t.Logf("Retry not triggered : delay=%dms\tresp=%v\terr=%v\n", time.Duration(time.Now().UnixNano()-startTime)/(1*time.Millisecond), string(resp), fmt.Sprintln(err))
+		} else {
+			t.Fatal(err)
+		}
+	}
+
+	client4, err := NewClient(
+		WithDialTimeout(2*time.Second),
+		WithRetryConfig(
+			retry.WithMaxAttemptTimes(2),
+			retry.WithInitDelay(1*time.Second),
+			retry.WithMaxDelay(10*time.Second),
+			retry.WithMaxJitter(5*time.Second),
+			retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy, retry.RandomDelayPolicy)),
+		),
+	)
+	if err != nil {
+		t.Fatal(err)
+		return
+	}
+	/* If the retryIfFunc is not set , idempotent logic is used by default */
+	//client4.SetRetryIfFunc(func(req *protocol.Request, resp *protocol.Response, err error) bool {
+	//	return err != nil
+	//})
+	startTime = time.Now().UnixNano()
+	_, resp, err = client4.Get(context.Background(), nil, "http://127.0.0.1:1234/ping")
+	if err != nil {
+		if time.Duration(time.Now().UnixNano()-startTime) > 1*time.Second && time.Duration(time.Now().UnixNano()-startTime) < 9*time.Second {
+			t.Logf("Retry triggered : delay=%dms\tresp=%v\terr=%v\n", time.Duration(time.Now().UnixNano()-startTime)/(1*time.Millisecond), string(resp), fmt.Sprintln(err))
+		} else if time.Duration(time.Now().UnixNano()-startTime) < 1*time.Second { // Compatible without triggering retry
+			t.Logf("Retry not triggered : delay=%dms\tresp=%v\terr=%v\n", time.Duration(time.Now().UnixNano()-startTime)/(1*time.Millisecond), string(resp), fmt.Sprintln(err))
+		} else {
+			t.Fatal(err)
+		}
+		return
+	}
+}
+
 func TestClientDialerName(t *testing.T) {
 	client, _ := NewClient()
 	dName, err := client.GetDialerName()
diff --git a/pkg/app/client/option_test.go b/pkg/app/client/option_test.go
index ff82147..149d6a7 100644
--- a/pkg/app/client/option_test.go
+++ b/pkg/app/client/option_test.go
@@ -17,9 +17,11 @@
 package client
 
 import (
+	"fmt"
 	"testing"
 	"time"
 
+	"github.com/cloudwego/hertz/pkg/app/client/retry"
 	"github.com/cloudwego/hertz/pkg/common/config"
 	"github.com/cloudwego/hertz/pkg/common/test/assert"
 )
@@ -31,18 +33,28 @@ func TestClientOptions(t *testing.T) {
 		WithMaxIdleConnDuration(5 * time.Second),
 		WithMaxConnDuration(10 * time.Second),
 		WithMaxConnWaitTimeout(5 * time.Second),
-		WithMaxIdempotentCallAttempts(10),
 		WithKeepAlive(false),
 		WithClientReadTimeout(1 * time.Second),
 		WithResponseBodyStream(true),
+		WithRetryConfig(
+			retry.WithMaxAttemptTimes(2),
+			retry.WithInitDelay(100*time.Millisecond),
+			retry.WithMaxDelay(5*time.Second),
+			retry.WithMaxJitter(1*time.Second),
+			retry.WithDelayPolicy(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy, retry.RandomDelayPolicy)),
+		),
 	})
 	assert.DeepEqual(t, 100*time.Millisecond, opt.DialTimeout)
 	assert.DeepEqual(t, 128, opt.MaxConnsPerHost)
 	assert.DeepEqual(t, 5*time.Second, opt.MaxIdleConnDuration)
 	assert.DeepEqual(t, 10*time.Second, opt.MaxConnDuration)
 	assert.DeepEqual(t, 5*time.Second, opt.MaxConnWaitTimeout)
-	assert.DeepEqual(t, 10, opt.MaxIdempotentCallAttempts)
 	assert.DeepEqual(t, false, opt.KeepAlive)
 	assert.DeepEqual(t, 1*time.Second, opt.ReadTimeout)
 	assert.DeepEqual(t, true, opt.ResponseBodyStream)
+	assert.DeepEqual(t, uint(2), opt.RetryConfig.MaxAttemptTimes)
+	assert.DeepEqual(t, 100*time.Millisecond, opt.RetryConfig.Delay)
+	assert.DeepEqual(t, 5*time.Second, opt.RetryConfig.MaxDelay)
+	assert.DeepEqual(t, 1*time.Second, opt.RetryConfig.MaxJitter)
+	assert.DeepEqual(t, fmt.Sprint(retry.CombineDelay(retry.FixedDelayPolicy, retry.BackOffDelayPolicy, retry.RandomDelayPolicy)), fmt.Sprint(opt.RetryConfig.DelayPolicy))
 }
diff --git a/pkg/common/config/client_option_test.go b/pkg/common/config/client_option_test.go
index 803807e..8e9f015 100644
--- a/pkg/common/config/client_option_test.go
+++ b/pkg/common/config/client_option_test.go
@@ -31,7 +31,6 @@ func TestDefaultClientOptions(t *testing.T) {
 	assert.DeepEqual(t, consts.DefaultDialTimeout, options.DialTimeout)
 	assert.DeepEqual(t, consts.DefaultMaxConnsPerHost, options.MaxConnsPerHost)
 	assert.DeepEqual(t, consts.DefaultMaxIdleConnDuration, options.MaxIdleConnDuration)
-	assert.DeepEqual(t, consts.DefaultMaxIdempotentCallAttempts, options.MaxIdempotentCallAttempts)
 	assert.DeepEqual(t, true, options.KeepAlive)
 }
 
diff --git a/pkg/protocol/http1/req/request_test.go b/pkg/protocol/http1/req/request_test.go
index 2edee37..fc2fe71 100644
--- a/pkg/protocol/http1/req/request_test.go
+++ b/pkg/protocol/http1/req/request_test.go
@@ -1120,7 +1120,8 @@ func testContinueReadBodyStream(t *testing.T, header, body string, maxBodySize,
 }
 
 func verifyRequestHeader(t *testing.T, h *protocol.RequestHeader, expectedContentLength int,
-	expectedRequestURI, expectedHost, expectedReferer, expectedContentType string) {
+	expectedRequestURI, expectedHost, expectedReferer, expectedContentType string,
+) {
 	if h.ContentLength() != expectedContentLength {
 		t.Fatalf("Unexpected Content-Length %d. Expected %d", h.ContentLength(), expectedContentLength)
 	}
diff --git a/pkg/protocol/http1/resp/response_test.go b/pkg/protocol/http1/resp/response_test.go
index 69f42f2..072f670 100644
--- a/pkg/protocol/http1/resp/response_test.go
+++ b/pkg/protocol/http1/resp/response_test.go
@@ -159,7 +159,8 @@ func testResponseReadError(t *testing.T, resp *protocol.Response, response strin
 }
 
 func testResponseReadSuccess(t *testing.T, resp *protocol.Response, response string, expectedStatusCode, expectedContentLength int,
-	expectedContentType, expectedBody, expectedTrailer string) {
+	expectedContentType, expectedBody, expectedTrailer string,
+) {
 	zr := mock.NewZeroCopyReader(response)
 	err := Read(resp, zr)
 	if err != nil {
@@ -436,7 +437,8 @@ func verifyResponseHeader(t *testing.T, h *protocol.ResponseHeader, expectedStat
 }
 
 func testResponseSuccess(t *testing.T, statusCode int, contentType, serverName, body string,
-	expectedStatusCode int, expectedContentType, expectedServerName string) {
+	expectedStatusCode int, expectedContentType, expectedServerName string,
+) {
 	var resp protocol.Response
 	resp.SetStatusCode(statusCode)
 	resp.Header.Set("Content-Type", contentType)
@@ -479,7 +481,8 @@ func testResponseSuccess(t *testing.T, statusCode int, contentType, serverName,
 }
 
 func testResponseReadWithoutBody(t *testing.T, resp *protocol.Response, s string, skipBody bool,
-	expectedStatusCode, expectedContentLength int, expectedContentType, expectedTrailer string) {
+	expectedStatusCode, expectedContentLength int, expectedContentType, expectedTrailer string,
+) {
 	zr := mock.NewZeroCopyReader(s)
 	resp.SkipBody = skipBody
 	err := Read(resp, zr)