Repository: go-kratos/kratos
Pull request #3122 (2023-12-16 07:37:02)

feat(middleware): adding sentry as errortracker
<!--
🎉 Thanks for sending a pull request to Kratos! Here are some tips for you:

1. If this is your first time contributing to Kratos, please read our contribution guide: https://go-kratos.dev/en/docs/community/contribution/
2. Ensure you have added or ran the appropriate tests and lint for your PR, please use `make lint` and `make test` before filing your PR, use `make clean` to tidy your go mod.
3. If the PR is unfinished, you may need to mark it as a WIP(Work In Progress) PR or Draft PR
4. Please use a semantic commits format title, such as `<type>[optional scope]: <description>`, see: https://go-kratos.dev/docs/community/contribution#type
5. at the same time, please note that similar work should be submitted in one PR as far as possible to reduce the workload of reviewers. Do not split a work into multiple PR unless it should.
-->

<!--
🎉 感谢您向 Kratos 发送 PR！以下是一些提示：
如果这是你第一次为 Kratos 贡献，请阅读我们的贡献指南：https://go-kratos.dev/en/docs/community/contribution/
2、确保您已经为您的 PR 添加或运行了适当的测试和lint，请在提交PR之前使用“make lint”和“make test”，使用“make clean”整理您的 go.mod。
3、如果 PR 未完成，您可能需要将其标记为 WIP（Work In Progress）PR 或 Draft PR
4、请使用语义提交格式标题，如“<类型>[可选范围]：<说明>`，请参阅：https://go-kratos.dev/docs/community/contribution#type
5. 同时请注意，同类的工作请尽量在一个PR中提交，以减轻 review 者的工作负担，不要把一项工作拆分成很多个PR，除非它应该这样做。
-->


#### Description (what this PR does / why we need it):
<!--
* The description should include the motivation for this PR or contrast this with previous behavior
-->
add sentry middleware for error tracking

#### Which issue(s) this PR fixes (resolves / be part of):
<!--
* Automatically closes linked issue when PR is merged.
* If your PR is not fully resolved the issue, please use `part of #<issue number>` instead.

Usage: `fixes/resolves #<issue number>`, or `fixes/resolves (paste link of issue)`.
-->
1. move https://github.com/go-kratos/sentry into the main repo
2. bump the go-sentry version
3. refactoring attributes naming
4. add tags options

#### Other special notes for the reviewers:
<!--
* Somethings that need extra attention for the reviewers
* Some additional notes, TODO list, etc.
-->
The documents should be add to go-kratos.dev and the origin repo https://github.com/go-kratos/sentry should be set to archived

codecov-commenter: ## [Codecov](https://app.codecov.io/gh/go-kratos/kratos/pull/3122?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=go-kratos) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`08300d8`)](https://app.codecov.io/gh/go-kratos/kratos/commit/08300d8a4178b4e0ff7b226887d66c0764f0d172?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=go-kratos) 84.53% compared to head [(`5010987`)](https://app.codecov.io/gh/go-kratos/kratos/pull/3122?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=go-kratos) 84.38%.

> :exclamation: Current head 5010987 differs from pull request most recent head 367f0f5. Consider uploading reports for the commit 367f0f5 to get more accurate results


:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #3122      +/-   ##
==========================================
- Coverage   84.53%   84.38%   -0.16%     
==========================================
  Files          88       88              
  Lines        3983     3983              
==========================================
- Hits         3367     3361       -6     
- Misses        441      445       +4     
- Partials      175      177       +2     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/go-kratos/kratos/pull/3122?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=go-kratos).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=go-kratos).

shenqidebaozi: Are you ready to merge?
shenqidebaozi: @Windfarer I have archived https://github.com/go-kratos/sentry
Windfarer: @shenqidebaozi ready for review

Patch:
diff --git a/app.go b/app.go
index 2fa52ec9..1fa23132 100644
--- a/app.go
+++ b/app.go
@@ -139,6 +139,7 @@ func (a *App) Run() error {
 	if err = eg.Wait(); err != nil && !errors.Is(err, context.Canceled) {
 		return err
 	}
+	err = nil
 	for _, fn := range a.opts.afterStop {
 		err = fn(sctx)
 	}
diff --git a/contrib/errortracker/sentry/README.md b/contrib/errortracker/sentry/README.md
new file mode 100644
index 00000000..edab5f57
--- /dev/null
+++ b/contrib/errortracker/sentry/README.md
@@ -0,0 +1,54 @@
+# Sentry middleware for Kratos
+This middleware helps you to catch panics and report them to [sentry](https://sentry.io/)
+
+## Quick Start
+You could check the full demo in example folder.
+```go
+// Step 1: 
+// init sentry in the entry of your application
+import "github.com/getsentry/sentry-go"
+
+sentry.Init(sentry.ClientOptions{
+		Dsn: "<your dsn>",
+		AttachStacktrace: true, // recommended
+})
+
+
+// Step 2: 
+// set middleware
+import 	ksentry "github.com/go-kratos/kratos/contrib/errortracker/sentry/v2"
+
+// for HTTP server, new HTTP server with sentry middleware options
+var opts = []http.ServerOption{
+	http.Middleware(
+		recovery.Recovery(),
+		tracing.Server(),
+		ksentry.Server(ksentry.WithTags(map[string]interface{}{
+			"tag": "some-custom-constant-tag",
+			"trace_id": tracing.TraceID(), // If you want to use the TraceID valuer, you need to place it after the A middleware.
+		})), // must after Recovery middleware, because of the exiting order will be reversed
+		
+		logging.Server(logger), 
+	),
+}
+
+// for gRPC server, new gRPC server with sentry middleware options
+var opts = []grpc.ServerOption{
+     grpc.Middleware(
+		recovery.Recovery(),
+		tracing.Server(),
+		ksentry.Server(ksentry.WithTags(map[string]interface{}{
+			"tag": "some-custom-constant-tag",
+			"trace_id": tracing.TraceID(), // If you want to use the TraceID valuer, you need to place it after the A middleware.
+		})), // must after Recovery middleware, because of the exiting order will be reversed
+		logging.Server(logger),
+     ),
+ }
+
+
+// Then, the framework will report events to Sentry when your trigger panics.
+// Or your can push events to Sentry manually
+```
+
+## Reference
+* [https://docs.sentry.io/platforms/go/](https://docs.sentry.io/platforms/go/)
diff --git a/contrib/errortracker/sentry/go.mod b/contrib/errortracker/sentry/go.mod
new file mode 100644
index 00000000..201597d8
--- /dev/null
+++ b/contrib/errortracker/sentry/go.mod
@@ -0,0 +1,28 @@
+module github.com/go-kratos/kratos/contrib/errortracker/sentry/v2
+
+go 1.19
+
+require (
+	github.com/getsentry/sentry-go v0.25.0
+	github.com/go-kratos/kratos/v2 v2.0.0-00010101000000-000000000000
+)
+
+require (
+	github.com/go-kratos/aegis v0.2.0 // indirect
+	github.com/go-playground/form/v4 v4.2.0 // indirect
+	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/gorilla/mux v1.8.1 // indirect
+	github.com/kr/text v0.2.0 // indirect
+	golang.org/x/net v0.17.0 // indirect
+	golang.org/x/sys v0.13.0 // indirect
+	golang.org/x/text v0.13.0 // indirect
+	google.golang.org/genproto v0.0.0-20230629202037-9506855d4529 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20230629202037-9506855d4529 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20230629202037-9506855d4529 // indirect
+	google.golang.org/grpc v1.56.3 // indirect
+	google.golang.org/protobuf v1.31.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)
+
+replace github.com/go-kratos/kratos/v2 => ../../../
diff --git a/contrib/errortracker/sentry/go.sum b/contrib/errortracker/sentry/go.sum
new file mode 100644
index 00000000..1d510f24
--- /dev/null
+++ b/contrib/errortracker/sentry/go.sum
@@ -0,0 +1,55 @@
+github.com/census-instrumentation/opencensus-proto v0.4.1 h1:iKLQ0xPNFxR/2hzXZMrBo8f1j86j5WHzznCCQxV/b8g=
+github.com/cncf/xds/go v0.0.0-20230607035331-e9ce68804cb4 h1:/inchEIKaYC1Akx+H+gqO04wryn5h75LSazbRlnya1k=
+github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/envoyproxy/go-control-plane v0.11.2-0.20230627204322-7d0032219fcb h1:kxNVXsNro/lpR5WD+P1FI/yUHn2G03Glber3k8cQL2Y=
+github.com/envoyproxy/protoc-gen-validate v0.10.1 h1:c0g45+xCJhdgFGw7a5QAfdS4byAbud7miNWJ1WwEVf8=
+github.com/getsentry/sentry-go v0.25.0 h1:q6Eo+hS+yoJlTO3uu/azhQadsD8V+jQn2D8VvX1eOyI=
+github.com/getsentry/sentry-go v0.25.0/go.mod h1:lc76E2QywIyW8WuBnwl8Lc4bkmQH4+w1gwTf25trprY=
+github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
+github.com/go-kratos/aegis v0.2.0 h1:dObzCDWn3XVjUkgxyBp6ZeWtx/do0DPZ7LY3yNSJLUQ=
+github.com/go-kratos/aegis v0.2.0/go.mod h1:v0R2m73WgEEYB3XYu6aE2WcMwsZkJ/Rzuf5eVccm7bI=
+github.com/go-playground/assert/v2 v2.0.1 h1:MsBgLAaY856+nPRTKrp3/OZK38U/wa0CcBYNjji3q3A=
+github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
+github.com/go-playground/form/v4 v4.2.0 h1:N1wh+Goz61e6w66vo8vJkQt+uwZSoLz50kZPJWR8eic=
+github.com/go-playground/form/v4 v4.2.0/go.mod h1:q1a2BY+AQUUzhl6xA/6hBetay6dEIhMHjgvJiGo6K7U=
+github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
+github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
+github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
+github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
+github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
+github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
+github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
+github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
+github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8=
+golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM=
+golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
+golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
+golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k=
+golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/genproto v0.0.0-20230629202037-9506855d4529 h1:9JucMWR7sPvCxUFd6UsOUNmA5kCcWOfORaT3tpAsKQs=
+google.golang.org/genproto v0.0.0-20230629202037-9506855d4529/go.mod h1:xZnkP7mREFX5MORlOPEzLMr+90PPZQ2QWzrVTWfAq64=
+google.golang.org/genproto/googleapis/api v0.0.0-20230629202037-9506855d4529 h1:s5YSX+ZH5b5vS9rnpGymvIyMpLRJizowqDlOuyjXnTk=
+google.golang.org/genproto/googleapis/api v0.0.0-20230629202037-9506855d4529/go.mod h1:vHYtlOoi6TsQ3Uk2yxR7NI5z8uoV+3pZtR4jmHIkRig=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20230629202037-9506855d4529 h1:DEH99RbiLZhMxrpEJCZ0A+wdTe0EOgou/poSLx9vWf4=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20230629202037-9506855d4529/go.mod h1:66JfowdXAEgad5O9NnYcsNPLCPZJD++2L9X0PCMODrA=
+google.golang.org/grpc v1.56.3 h1:8I4C0Yq1EjstUzUJzpcRVbuYA2mODtEmpWiQoN/b2nc=
+google.golang.org/grpc v1.56.3/go.mod h1:I9bI3vqKfayGqPUAwGdOSu7kt6oIJLixfffKrpXqQ9s=
+google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
+google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
+google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8=
+google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
diff --git a/contrib/errortracker/sentry/sentry.go b/contrib/errortracker/sentry/sentry.go
new file mode 100644
index 00000000..2aaba196
--- /dev/null
+++ b/contrib/errortracker/sentry/sentry.go
@@ -0,0 +1,145 @@
+package sentry
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/getsentry/sentry-go"
+
+	"github.com/go-kratos/kratos/v2/log"
+	"github.com/go-kratos/kratos/v2/middleware"
+	"github.com/go-kratos/kratos/v2/transport"
+	"github.com/go-kratos/kratos/v2/transport/grpc"
+	"github.com/go-kratos/kratos/v2/transport/http"
+)
+
+type ctxKey struct{}
+
+type Option func(*options)
+
+type options struct {
+	repanic         bool
+	waitForDelivery bool
+	timeout         time.Duration
+	tags            map[string]interface{}
+}
+
+// Repanic configures whether Sentry should repanic after recovery, in most cases it should be set to true.
+func WithRepanic(repanic bool) Option {
+	return func(opts *options) {
+		opts.repanic = repanic
+	}
+}
+
+// WaitForDelivery configures whether you want to block the request before moving forward with the response.
+func WithWaitForDelivery(waitForDelivery bool) Option {
+	return func(opts *options) {
+		opts.waitForDelivery = waitForDelivery
+	}
+}
+
+// Timeout for the event delivery requests.
+func WithTimeout(timeout time.Duration) Option {
+	return func(opts *options) {
+		opts.timeout = timeout
+	}
+}
+
+// Global tags injection, the value type must be string or log.Valuer
+func WithTags(kvs map[string]interface{}) Option {
+	return func(opts *options) {
+		opts.tags = kvs
+	}
+}
+
+// Server returns a new server middleware for Sentry.
+func Server(opts ...Option) middleware.Middleware {
+	conf := options{repanic: true}
+	for _, o := range opts {
+		o(&conf)
+	}
+	if conf.timeout == 0 {
+		conf.timeout = 2 * time.Second
+	}
+	return func(handler middleware.Handler) middleware.Handler {
+		return func(ctx context.Context, req interface{}) (reply interface{}, err error) {
+			hub := GetHubFromContext(ctx)
+			scope := hub.Scope()
+
+			for k, v := range conf.tags {
+				switch val := v.(type) {
+				case string:
+					scope.SetTag(k, val)
+				case log.Valuer:
+					scope.SetTag(k, fmt.Sprintf("%v", val(ctx)))
+				}
+			}
+
+			if tr, ok := transport.FromServerContext(ctx); ok {
+				switch tr.Kind() {
+				case transport.KindGRPC:
+					gtr := tr.(*grpc.Transport)
+					scope.SetContext("gRPC", map[string]interface{}{
+						"endpoint":  gtr.Endpoint(),
+						"operation": gtr.Operation(),
+					})
+					headers := make(map[string]interface{})
+					for _, k := range gtr.RequestHeader().Keys() {
+						headers[k] = gtr.RequestHeader().Get(k)
+					}
+					scope.SetContext("Headers", headers)
+				case transport.KindHTTP:
+					htr := tr.(*http.Transport)
+					r := htr.Request()
+					scope.SetRequest(r)
+				}
+			}
+
+			ctx = context.WithValue(ctx, ctxKey{}, hub)
+			defer recoverWithSentry(ctx, conf, hub, req)
+			return handler(ctx, req)
+		}
+	}
+}
+
+func recoverWithSentry(ctx context.Context, opts options, hub *sentry.Hub, req interface{}) {
+	if err := recover(); err != nil {
+		if !isBrokenPipeError(err) {
+			eventID := hub.RecoverWithContext(
+				context.WithValue(ctx, sentry.RequestContextKey, req),
+				err,
+			)
+			if eventID != nil && opts.waitForDelivery {
+				hub.Flush(opts.timeout)
+			}
+		}
+		if opts.repanic {
+			panic(err)
+		}
+	}
+}
+
+func isBrokenPipeError(err interface{}) bool {
+	if netErr, ok := err.(*net.OpError); ok {
+		if sysErr, ok := netErr.Err.(*os.SyscallError); ok {
+			if strings.Contains(strings.ToLower(sysErr.Error()), "broken pipe") ||
+				strings.Contains(strings.ToLower(sysErr.Error()), "connection reset by peer") {
+				return true
+			}
+		}
+	}
+	return false
+}
+
+// GetHubFromContext retrieves attached *sentry.Hub instance from context or sentry.
+// You can use this hub for extra information reporting
+func GetHubFromContext(ctx context.Context) *sentry.Hub {
+	if hub, ok := ctx.Value(ctxKey{}).(*sentry.Hub); ok {
+		return hub
+	}
+	return sentry.CurrentHub().Clone()
+}
diff --git a/contrib/registry/consul/client.go b/contrib/registry/consul/client.go
index ea5fe846..ab1551cd 100644
--- a/contrib/registry/consul/client.go
+++ b/contrib/registry/consul/client.go
@@ -220,7 +220,11 @@ func (c *Client) Register(_ context.Context, svc *registry.ServiceInstance, enab
 						_ = c.cli.Agent().ServiceDeregister(svc.ID)
 						return
 					}
-					err = c.cli.Agent().UpdateTTL("service:"+svc.ID, "pass", "pass")
+					err = c.cli.Agent().UpdateTTLOpts("service:"+svc.ID, "pass", "pass", new(api.QueryOptions).WithContext(c.ctx))
+					if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
+						_ = c.cli.Agent().ServiceDeregister(svc.ID)
+						return
+					}
 					if err != nil {
 						log.Errorf("[Consul] update ttl heartbeat to consul failed! err=%v", err)
 						// when the previous report fails, try to re register the service
diff --git a/selector/node/ewma/node.go b/selector/node/ewma/node.go
index c718f252..540a0bb3 100644
--- a/selector/node/ewma/node.go
+++ b/selector/node/ewma/node.go
@@ -1,11 +1,9 @@
 package ewma
 
 import (
-	"container/list"
 	"context"
 	"math"
 	"net"
-	"sync"
 	"sync/atomic"
 	"time"
 
@@ -17,7 +15,7 @@ const (
 	// The mean lifetime of `cost`, it reaches its half-life after Tau*ln(2).
 	tau = int64(time.Millisecond * 600)
 	// if statistic not collected,we add a big lag penalty to endpoint
-	penalty = uint64(time.Second * 10)
+	penalty = uint64(time.Microsecond * 100)
 )
 
 var (
@@ -33,18 +31,21 @@ type Node struct {
 	lag       int64
 	success   uint64
 	inflight  int64
-	inflights *list.List
+	inflights [200]int64
 	// last collected timestamp
-	stamp     int64
-	predictTs int64
-	predict   int64
+	stamp int64
 	// request number in a period time
 	reqs int64
 	// last lastPick timestamp
 	lastPick int64
 
-	errHandler func(err error) (isErr bool)
-	lk         sync.RWMutex
+	errHandler   func(err error) (isErr bool)
+	cachedWeight *atomic.Value
+}
+
+type nodeWeight struct {
+	value    float64
+	updateAt int64
 }
 
 // Builder is ewma node builder.
@@ -55,12 +56,12 @@ type Builder struct {
 // Build create a weighted node.
 func (b *Builder) Build(n selector.Node) selector.WeightedNode {
 	s := &Node{
-		Node:       n,
-		lag:        0,
-		success:    1000,
-		inflight:   1,
-		inflights:  list.New(),
-		errHandler: b.ErrHandler,
+		Node:         n,
+		lag:          0,
+		success:      1000,
+		inflight:     1,
+		errHandler:   b.ErrHandler,
+		cachedWeight: &atomic.Value{},
 	}
 	return s
 }
@@ -72,64 +73,58 @@ func (n *Node) health() uint64 {
 func (n *Node) load() (load uint64) {
 	now := time.Now().UnixNano()
 	avgLag := atomic.LoadInt64(&n.lag)
-	lastPredictTs := atomic.LoadInt64(&n.predictTs)
-	predictInterval := avgLag / 5
-	if predictInterval < int64(time.Millisecond*5) {
-		predictInterval = int64(time.Millisecond * 5)
-	}
-	if predictInterval > int64(time.Millisecond*200) {
-		predictInterval = int64(time.Millisecond * 200)
-	}
-	if now-lastPredictTs > predictInterval && atomic.CompareAndSwapInt64(&n.predictTs, lastPredictTs, now) {
-		var (
-			total   int64
-			count   int
-			predict int64
-		)
-		n.lk.RLock()
-		first := n.inflights.Front()
-		for first != nil {
-			lag := now - first.Value.(int64)
-			if lag > avgLag {
-				count++
-				total += lag
-			}
-			first = first.Next()
-		}
-		if count > (n.inflights.Len()/2 + 1) {
-			predict = total / int64(count)
-		}
-		n.lk.RUnlock()
-		atomic.StoreInt64(&n.predict, predict)
-	}
+	predict := n.predict(avgLag, now)
 
 	if avgLag == 0 {
 		// penalty is the penalty value when there is no data when the node is just started.
-		// The default value is 1e9 * 10
 		load = penalty * uint64(atomic.LoadInt64(&n.inflight))
 		return
 	}
-	predict := atomic.LoadInt64(&n.predict)
 	if predict > avgLag {
 		avgLag = predict
 	}
+	// add 5ms to eliminate the latency gap between different zones
+	avgLag += int64(time.Millisecond * 5)
+	avgLag = int64(math.Sqrt(float64(avgLag)))
 	load = uint64(avgLag) * uint64(atomic.LoadInt64(&n.inflight))
+	return load
+}
+
+func (n *Node) predict(avgLag int64, now int64) (predict int64) {
+	var (
+		total    int64
+		slowNum  int
+		totalNum int
+	)
+	for i := range n.inflights {
+		start := atomic.LoadInt64(&n.inflights[i])
+		if start != 0 {
+			totalNum++
+			lag := now - start
+			if lag > avgLag {
+				slowNum++
+				total += lag
+			}
+		}
+	}
+	if slowNum >= (totalNum/2 + 1) {
+		predict = total / int64(slowNum)
+	}
 	return
 }
 
 // Pick pick a node.
 func (n *Node) Pick() selector.DoneFunc {
-	now := time.Now().UnixNano()
-	atomic.StoreInt64(&n.lastPick, now)
+	start := time.Now().UnixNano()
+	atomic.StoreInt64(&n.lastPick, start)
 	atomic.AddInt64(&n.inflight, 1)
-	atomic.AddInt64(&n.reqs, 1)
-	n.lk.Lock()
-	e := n.inflights.PushBack(now)
-	n.lk.Unlock()
+	reqs := atomic.AddInt64(&n.reqs, 1)
+	slot := reqs % 200
+	swapped := atomic.CompareAndSwapInt64(&n.inflights[slot], 0, start)
 	return func(ctx context.Context, di selector.DoneInfo) {
-		n.lk.Lock()
-		n.inflights.Remove(e)
-		n.lk.Unlock()
+		if swapped {
+			atomic.CompareAndSwapInt64(&n.inflights[slot], start, 0)
+		}
 		atomic.AddInt64(&n.inflight, -1)
 
 		now := time.Now().UnixNano()
@@ -141,7 +136,6 @@ func (n *Node) Pick() selector.DoneFunc {
 		}
 		w := math.Exp(float64(-td) / float64(tau))
 
-		start := e.Value.(int64)
 		lag := now - start
 		if lag < 0 {
 			lag = 0
@@ -172,7 +166,19 @@ func (n *Node) Pick() selector.DoneFunc {
 
 // Weight is node effective weight.
 func (n *Node) Weight() (weight float64) {
-	weight = float64(n.health()*uint64(time.Second)) / float64(n.load())
+	w, ok := n.cachedWeight.Load().(*nodeWeight)
+	now := time.Now().UnixNano()
+	if !ok || time.Duration(now-w.updateAt) > (time.Millisecond*5) {
+		health := n.health()
+		load := n.load()
+		weight = float64(health*uint64(time.Microsecond)*10) / float64(load)
+		n.cachedWeight.Store(&nodeWeight{
+			value:    weight,
+			updateAt: now,
+		})
+	} else {
+		weight = w.value
+	}
 	return
 }
 
diff --git a/transport/http/server.go b/transport/http/server.go
index c967e4a1..9c5550b2 100644
--- a/transport/http/server.go
+++ b/transport/http/server.go
@@ -141,6 +141,18 @@ func PathPrefix(prefix string) ServerOption {
 	}
 }
 
+func NotFoundHandler(handler http.Handler) ServerOption {
+	return func(s *Server) {
+		s.router.NotFoundHandler = handler
+	}
+}
+
+func MethodNotAllowedHandler(handler http.Handler) ServerOption {
+	return func(s *Server) {
+		s.router.MethodNotAllowedHandler = handler
+	}
+}
+
 // Server is an HTTP server wrapper.
 type Server struct {
 	*http.Server
@@ -177,12 +189,12 @@ func NewServer(opts ...ServerOption) *Server {
 		strictSlash: true,
 		router:      mux.NewRouter(),
 	}
+	srv.router.NotFoundHandler = http.DefaultServeMux
+	srv.router.MethodNotAllowedHandler = http.DefaultServeMux
 	for _, o := range opts {
 		o(srv)
 	}
 	srv.router.StrictSlash(srv.strictSlash)
-	srv.router.NotFoundHandler = http.DefaultServeMux
-	srv.router.MethodNotAllowedHandler = http.DefaultServeMux
 	srv.router.Use(srv.filter())
 	srv.Server = &http.Server{
 		Handler:   FilterChain(srv.filters...)(srv.router),


Test patch:
diff --git a/contrib/errortracker/sentry/sentry_test.go b/contrib/errortracker/sentry/sentry_test.go
new file mode 100644
index 00000000..210c1762
--- /dev/null
+++ b/contrib/errortracker/sentry/sentry_test.go
@@ -0,0 +1,49 @@
+package sentry
+
+import (
+	"testing"
+	"time"
+)
+
+func TestWithTags(t *testing.T) {
+	opts := new(options)
+	strval := "bar"
+	kvs := map[string]interface{}{
+		"foo": strval,
+	}
+	funcTags := WithTags(kvs)
+	funcTags(opts)
+	if opts.tags["foo"].(string) != strval {
+		t.Errorf("TestWithTags() = %v, want %v", opts.tags["foo"].(string), strval)
+	}
+}
+
+func TestWithRepanic(t *testing.T) {
+	opts := new(options)
+	val := true
+	f := WithRepanic(val)
+	f(opts)
+	if opts.repanic != val {
+		t.Errorf("TestWithRepanic() = %v, want %v", opts.repanic, val)
+	}
+}
+
+func TestWithWaitForDelivery(t *testing.T) {
+	opts := new(options)
+	val := true
+	f := WithWaitForDelivery(val)
+	f(opts)
+	if opts.waitForDelivery != val {
+		t.Errorf("TestWithWaitForDelivery() = %v, want %v", opts.waitForDelivery, val)
+	}
+}
+
+func TestWithTimeout(t *testing.T) {
+	opts := new(options)
+	val := time.Second * 10
+	f := WithTimeout(val)
+	f(opts)
+	if opts.timeout != val {
+		t.Errorf("TestWithTimeout() = %v, want %v", opts.timeout, val)
+	}
+}
diff --git a/selector/node/ewma/node_test.go b/selector/node/ewma/node_test.go
index d930495c..7f4d59b8 100644
--- a/selector/node/ewma/node_test.go
+++ b/selector/node/ewma/node_test.go
@@ -36,16 +36,16 @@ func TestDirect(t *testing.T) {
 		t.Errorf("done2 is equal to nil")
 	}
 
-	time.Sleep(time.Millisecond * 10)
+	time.Sleep(time.Millisecond * 15)
 	done(context.Background(), selector.DoneInfo{})
-	if float64(30000) >= wn.Weight() {
+	if float64(70) >= wn.Weight() {
 		t.Errorf("float64(30000) >= wn.Weight()(%v)", wn.Weight())
 	}
-	if float64(60000) <= wn.Weight() {
-		t.Errorf("float64(60000) <= wn.Weight()(%v)", wn.Weight())
+	if float64(1200) <= wn.Weight() {
+		t.Errorf("float64(1000) <= wn.Weight()(%v)", wn.Weight())
 	}
-	if time.Millisecond*15 <= wn.PickElapsed() {
-		t.Errorf("time.Millisecond*15 <= wn.PickElapsed()(%v)", wn.PickElapsed())
+	if time.Millisecond*30 <= wn.PickElapsed() {
+		t.Errorf("time.Millisecond*30 <= wn.PickElapsed()(%v)", wn.PickElapsed())
 	}
 	if time.Millisecond*5 >= wn.PickElapsed() {
 		t.Errorf("time.Millisecond*5 >= wn.PickElapsed()(%v)", wn.PickElapsed())
@@ -77,11 +77,11 @@ func TestDirectError(t *testing.T) {
 		time.Sleep(time.Millisecond * 20)
 		done(context.Background(), selector.DoneInfo{Err: err})
 	}
-	if float64(30000) >= wn.Weight() {
-		t.Errorf("float64(30000) >= wn.Weight()(%v)", wn.Weight())
+	if float64(1000) >= wn.Weight() {
+		t.Errorf("float64(1000) >= wn.Weight()(%v)", wn.Weight())
 	}
-	if float64(60000) <= wn.Weight() {
-		t.Errorf("float64(60000) <= wn.Weight()(%v)", wn.Weight())
+	if float64(2000) <= wn.Weight() {
+		t.Errorf("float64(2000) <= wn.Weight()(%v)", wn.Weight())
 	}
 }
 
@@ -118,10 +118,32 @@ func TestDirectErrorHandler(t *testing.T) {
 		time.Sleep(time.Millisecond * 20)
 		done(context.Background(), selector.DoneInfo{Err: err})
 	}
-	if float64(30000) >= wn.Weight() {
-		t.Errorf("float64(30000) >= wn.Weight()(%v)", wn.Weight())
+	if float64(1000) >= wn.Weight() {
+		t.Errorf("float64(100) >= wn.Weight()(%v)", wn.Weight())
 	}
-	if float64(60000) <= wn.Weight() {
-		t.Errorf("float64(60000) <= wn.Weight()(%v)", wn.Weight())
+	if float64(2000) <= wn.Weight() {
+		t.Errorf("float64(200) <= wn.Weight()(%v)", wn.Weight())
 	}
 }
+
+func BenchmarkPickAndWeight(b *testing.B) {
+	bu := &Builder{}
+	node := bu.Build(selector.NewNode(
+		"http",
+		"127.0.0.1:9090",
+		&registry.ServiceInstance{
+			ID:        "127.0.0.1:9090",
+			Name:      "helloworld",
+			Version:   "v1.0.0",
+			Endpoints: []string{"http://127.0.0.1:9090"},
+			Metadata:  map[string]string{"weight": "10"},
+		}))
+	di := selector.DoneInfo{}
+	b.RunParallel(func(pb *testing.PB) {
+		for pb.Next() {
+			done := node.Pick()
+			node.Weight()
+			done(context.Background(), di)
+		}
+	})
+}
diff --git a/transport/http/server_test.go b/transport/http/server_test.go
index 781c06a7..44477504 100644
--- a/transport/http/server_test.go
+++ b/transport/http/server_test.go
@@ -371,3 +371,19 @@ func TestListener(t *testing.T) {
 		t.Errorf("expected not empty")
 	}
 }
+
+func TestNotFoundHandler(t *testing.T) {
+	mux := http.NewServeMux()
+	srv := NewServer(NotFoundHandler(mux))
+	if !reflect.DeepEqual(srv.router.NotFoundHandler, mux) {
+		t.Errorf("expected %v got %v", mux, srv.router.NotFoundHandler)
+	}
+}
+
+func TestMethodNotAllowedHandler(t *testing.T) {
+	mux := http.NewServeMux()
+	srv := NewServer(MethodNotAllowedHandler(mux))
+	if !reflect.DeepEqual(srv.router.MethodNotAllowedHandler, mux) {
+		t.Errorf("expected %v got %v", mux, srv.router.MethodNotAllowedHandler)
+	}
+}