Repository: jaegertracing/jaeger

Issue:
Jaeger-Collector: too many open files
**Jaeger-Collector: too many open files**

We have a set of collectors in use. In the console, in the log output of the collector (which is created in JSON format), we suddenly see a large number of messages that always have this structure (and are not JSON);

`2021/07/21 13:24:45 http: Accept error: accept tcp [::]:14269: accept4: too many open files; retrying in 1s`

Since these messages have no information about their severity, it is unclear whether an error event is logged here without it being just a debug information.


**To Reproduce**
Steps to reproduce the behavior:
1.  Start Jaeger-Collector
2. Create traffic

**Expected behavior**
Thiese message should be visible


**Version (please complete the following information):**
 - OS:  Amazon Linux 2
 - Jaeger version: 1.24 (same behaviour with 1.22)
 - Deployment: bare metal

**What troubleshooting steps did you try?**
We modified ulimit for the Collector process - no effect



Hints from issue comments:
@jpkrohling: Interesting, that's the admin port, the same we serve the health checks. Are you able to tell what's on the other side of the connection? Perhaps your client isn't properly closing the HTTP connections?

Your instructions to reproduce the problem could also be improved. Are you able to reproduce it by running Jaeger with, say, `podman run jaegertracing/all-in-one:1.24.0` on the server side and `tracegen` as the client?@yurishkuro: It appears you have too many clients that keep opening new connections to the backend instead of reusing an existing connection. You should also check what file descriptor limit you have.

The log seems appropriate, it says that accept of an incoming connection was unsuccessful. What more information do you expect to see?@pbdger: > The log seems appropriate, it says that accept of an incoming connection was unsuccessful. What more information do you expect to see?

Unfortunately, the log entry is not created in json format like all other log events of the collector. Also the specification of an event type like error or warning etc. is missing.

On our side we process the log events with Fluentbit. Therefore we always expect a Json string from the Jaegercollector for logging. But this message unfortunately breaks the Json pattern. Therefore we did not notice this error for months, because it was not identified as an error on our systems.


As this has nothing to do with the problem "Too many open files". Maybe I should post a separate ticket because of the incorrect log output ? @yurishkuro: Looks like the log is coming from https://github.com/golang/go/blob/d77f4c0c5c966c37960cd691656fba184ae770ff/src/net/http/server.go#L3017. We're not defining `ErrorLog *log.Logger` on the Server, so it uses std logger, which is not JSON-formatted.

Also, since this is a health check port, we should double-check that we're properly closing the req/resp objects.@jpkrohling: I just tried hitting the admin port with `ab`, and couldn't reproduce the problem with 10 concurrent clients and 1 million requests:

```
$ ab -c 100 -n 1000000 http://localhost:14269/
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            14269

Document Path:          /
Document Length:        102 bytes

Concurrency Level:      100
Time taken for tests:   57.174 seconds
Complete requests:      1000000
Failed requests:        718434
   (Connect: 0, Receive: 0, Length: 718434, Exceptions: 0)
Total transferred:      212137980 bytes
HTML transferred:       103138252 bytes
Requests per second:    17490.39 [#/sec] (mean)
Time per request:       5.717 [ms] (mean)
Time per request:       0.057 [ms] (mean, across all concurrent requests)
Transfer rate:          3623.41 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3   0.8      3       7
Processing:     0    3   0.8      3       9
Waiting:        0    2   0.8      2       7
Total:          2    6   0.5      6      14

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      6
  80%      6
  90%      6
  95%      7
  98%      7
  99%      7
 100%     14 (longest request)
```

Once `ab` finishes, the state seems sane to me:

```
$ lsof -p 1297434
COMMAND       PID        USER   FD      TYPE  DEVICE SIZE/OFF    NODE NAME
collector 1297434 jpkroehling  cwd       DIR    0,43      820   58547 /home/jpkroehling/Projects/src/github.com/jaegertracing/jaeger
collector 1297434 jpkroehling  rtd       DIR    0,35      176     256 /
collector 1297434 jpkroehling  txt       REG    0,39 31259320  114714 /tmp/go-build4254053393/b001/exe/collector
collector 1297434 jpkroehling  mem       REG    0,34          4327015 /var/lib/sss/mc/passwd (path dev=0,35)
collector 1297434 jpkroehling  mem       REG    0,34          4173794 /usr/lib64/libnss_sss.so.2 (path dev=0,35)
collector 1297434 jpkroehling  mem       REG    0,34          4308749 /usr/lib64/libc-2.33.so (path dev=0,35)
collector 1297434 jpkroehling  mem       REG    0,34          4308763 /usr/lib64/libpthread-2.33.so (path dev=0,35)
collector 1297434 jpkroehling  mem       REG    0,34          4308742 /usr/lib64/ld-2.33.so (path dev=0,35)
collector 1297434 jpkroehling    0u      CHR   136,0      0t0       3 /dev/pts/0
collector 1297434 jpkroehling    1u      CHR   136,0      0t0       3 /dev/pts/0
collector 1297434 jpkroehling    2u      CHR   136,0      0t0       3 /dev/pts/0
collector 1297434 jpkroehling    3r      REG    0,35  9253600 4327015 /var/lib/sss/mc/passwd
collector 1297434 jpkroehling    4u  a_inode    0,14        0   11556 [eventpoll:5,7,8,9]
collector 1297434 jpkroehling    5r     FIFO    0,13      0t0 5921185 pipe
collector 1297434 jpkroehling    6w     FIFO    0,13      0t0 5921185 pipe
collector 1297434 jpkroehling    7u     IPv6 5921192      0t0     TCP *:14269 (LISTEN)
collector 1297434 jpkroehling    8u     IPv6 5920365      0t0     TCP *:cps (LISTEN)
collector 1297434 jpkroehling    9u     IPv6 5920366      0t0     TCP *:14268 (LISTEN)
```

I could really use more information on how to reproduce this.@pbdger: > I just tried hitting the admin port with `ab`, and couldn't reproduce the problem with 10 concurrent clients and 1 million requests:
> I could really use more information on how to reproduce this.


I did the same and I can confirm; the root cause seems to be in one of our clients. As mentioned a connection were not closed correctly.


Then why does the health check port appear so often in the list of Connections ?

The display is correct, but the interpretation is probably a bit more complicated; all HTTP requests end up in a loop and can only be processed when a connection becomes free. Since the health check is also executed every n-seconds, it also ends up in the loop and thus in the log output. It does not block a connection but waits for a connection.

I'm still testing a new version of the client today. I expect that the misbehavior will disappear. Then I will close the issue.

Thank you very much for your great support. And even if it was due to the client, we got a better logging in place as a spin-off :-)@pbdger: What were wrong;

We work on collectorClient and forgot to close the connection...
```
collectorClient := api_v2.NewCollectorServiceClient(connection)
response, err = collectorClient.PostSpans(ctx, r)
```
Just add: 
```
connection.Close()
```

Everything looks fine, but you get some new logs entries in jaeger-collector;
```
{"level":"info","ts":1626955277.52554,"caller":"grpclog/component.go:71","msg":"[transport]transport: loopyWriter.run returning. connection error: desc = \"transport is closing\"","system":"grpc","grpc_log":true}
```

transport.controlbuf.run() cause this info. Comment in code marks this a fine.

@ksmulpuri: @yurishkuro we are hitting the same issue described above
2021/07/21 13:24:45 http: Accept error: accept tcp [::]:14269: accept4: too many open files; retrying in 1s

it's seems like jaeger-collector scaling issue on our end as we see high traffic when the issue occurs. 

To fix: we are considering increasing file descriptor limit - will that alone fix the issue? are there any other parameters that we can tweak to help fix the issue?


env: jaeger version v1.37@yurishkuro: @mulpk thanks for asking on the existing issue.

* Are you monitoring the number of open file descriptors in the collector process? What are the levels and do they match your understanding of the number of clients? As mentioned earlier, clients are supposed to use persistent connections, but you're saying this issue is related to "high traffic" - is high traffic a result of more clients (e.g. maybe you're spawning off many lambda functions), or just more QPS? The latter should not affect FDs.
* IIRC we used to run with about 120k FDs per Jaeger collector at Uber
* FDs alone may not help to scale, you have to look at other health metrics like memory usage, CPU utilization.@ksmulpuri: @yurishkuro thanks for the response

setup:
we have a load balancer which distributes load to multiple hosts and each one runs jaeger collector as system process. I was looking at LB stats and see high inflow of span data when the **too many files error** occurs (that's what I was referring to high traffic before)

- how do I monitor open file descriptors in the collector process? I don't see it in the metrics I have - do I need to enable some flag on the collector process to monitor open file descriptors?

- do you have recommendation on below flags as well? I see they can help scale (the current value of **jaeger_collector_in_queue_latency_bucket** is **10291739** - it should be lower value right? guess I need to increase the workers)
collector.queue-size: 6000
collector.num-workers: 200

we have FDs at 1024 before and bumped it to 16k which did not help. I will try high FD number like 65k (current CPU/memory stats on the hosts show that it's using less than 30% of system resources)




@yurishkuro: > how do I monitor open file descriptors in the collector process?

Jaeger itself does not expose these, usually metrics like that are surfaced by your orchestration system for all workloads.@yurishkuro: > jaeger_collector_in_queue_latency_bucket

increasing queue size will not help with this number. Increasing number of workers might help, but only to a certain point - the bottleneck is usually the storage.@ksmulpuri: right, makes sense.. I see high numbers for **jaeger_collector_save_latency_bucket** : **10467444**

right now the main problem is - the jaeger-collector process is getting stuck when **too many files error** occurs. is there a way to discard the extra spans and make sure jaeger collector is operational? @yurishkuro: We do not have that capability right now, but it's a good idea. I don't know if we have too much control around FDs, but it's feasible for collector to monitor this number, as well as the internal queue length, and start responding with "I AM BUSY" and closing incoming connections@ksmulpuri: @yurishkuro we increased the FDs to 65535 and it's using under 50 FDs for the entire week except the peak that you see in the graph. we anticipated it might be connection close issue and made sure connections are handled properly but still see the issue. 

lsof on the jaeger host returns the 65535 TCP connections hanging around

jaeger-co 1098899 build   15u     sock                0,8      0t0 1050943012 protocol: TCP
jaeger-co 1098899 build   16u     sock                0,8      0t0 1050898799 protocol: TCP

I'm not sure if it's the load that we are not able to handle or any other issue.. any pointers would be helpful?

<img width="845" alt="Screenshot 2024-08-26 at 1 28 16 PM" src="https://github.com/user-attachments/assets/42495334-bedf-41bc-82a0-34c5fc55a923">
@yurishkuro: How many clients do you have connecting to collector? Are those incoming connections?@ksmulpuri: i'm using ThreadPoolExecutor from the java code to send parallel span requests to jaeger-collector. i'm batching 1000 requests and each request to jaeger-collector consists of 100 spans.

here is the http client connection configuration I'm using


```
PoolingHttpClientConnectionManager poolingHttpClientConnectionManager = new PoolingHttpClientConnectionManager();
poolingHttpClientConnectionManager.setDefaultMaxPerRoute(60);
poolingHttpClientConnectionManager.setMaxTotal(240);
RequestConfig rc = RequestConfig.custom().setConnectTimeout((int)Duration.ofSeconds(5L).toMillis()).setSocketTimeout((int)Duration.ofSeconds(20L).toMillis()).build();
CloseableHttpClient client = HttpClients.custom().setConnectionManager(connectionManager).setDefaultRequestConfig(rc).build();
```


these requests sent to jaeger-collector load balancer that I created and the load is split between 3 instances running jaeger-collector.

each jaeger-collector instance uses below configuration
  collector.queue-size: 50000
  collector.num-workers: 4000@yurishkuro: This doesn't answer my question, but it looks like you are forcing a synthetic load that creates lots of new client connections thus causing an issue. You either need to size Jaeger collector fleet to handle your peak concurrent traffic or use better connection management on the client, such as
  * multiple OTLP requests from a single client could be sent through a single gRPC connection
  * clients should maintain long lived connections, not open/close them per request
  * clients should not leave many idle connections not closed@ksmulpuri: Thanks for the response. I'm more inclined toward sizing Jaeger collector fleet to handle your peak concurrent traffic - currently using /api/v2/spans endpoint for submitting spans in Zipkin JSON v2. As mentioned above, we are sending like 100 spans in each request - do you have any recommendation here?

when you say size jaeger-collector does that mean horizontal or vertical scaling?
@yurishkuro: I meant horizontal scaling, to distribute connections across instances. You will have other bottlenecks if you are creating huge traffic spikes - your storage may not be able to handle the load.

Solution patch:
diff --git a/cmd/agent/app/builder.go b/cmd/agent/app/builder.go
index b09137f3..cb4c884e 100644
--- a/cmd/agent/app/builder.go
+++ b/cmd/agent/app/builder.go
@@ -111,7 +111,7 @@ func (b *Builder) CreateAgent(primaryProxy CollectorProxy, logger *zap.Logger, m
 	if err != nil {
 		return nil, fmt.Errorf("cannot create processors: %w", err)
 	}
-	server := b.HTTPServer.getHTTPServer(primaryProxy.GetManager(), mFactory)
+	server := b.HTTPServer.getHTTPServer(primaryProxy.GetManager(), mFactory, logger)
 	b.publishOpts(mFactory)
 
 	return NewAgent(processors, server, logger), nil
@@ -170,11 +170,11 @@ func (b *Builder) getProcessors(rep reporter.Reporter, mFactory metrics.Factory,
 }
 
 // GetHTTPServer creates an HTTP server that provides sampling strategies and baggage restrictions to client libraries.
-func (c HTTPServerConfiguration) getHTTPServer(manager configmanager.ClientConfigManager, mFactory metrics.Factory) *http.Server {
+func (c HTTPServerConfiguration) getHTTPServer(manager configmanager.ClientConfigManager, mFactory metrics.Factory, logger *zap.Logger) *http.Server {
 	if c.HostPort == "" {
 		c.HostPort = defaultHTTPServerHostPort
 	}
-	return httpserver.NewHTTPServer(c.HostPort, manager, mFactory)
+	return httpserver.NewHTTPServer(c.HostPort, manager, mFactory, logger)
 }
 
 // GetThriftProcessor gets a TBufferedServer backed Processor using the collector configuration
diff --git a/cmd/agent/app/httpserver/srv.go b/cmd/agent/app/httpserver/srv.go
index 37a4dce1..227be0bb 100644
--- a/cmd/agent/app/httpserver/srv.go
+++ b/cmd/agent/app/httpserver/srv.go
@@ -20,6 +20,8 @@ import (
 
 	"github.com/gorilla/mux"
 	"github.com/uber/jaeger-lib/metrics"
+	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
 
 	"github.com/jaegertracing/jaeger/cmd/agent/app/configmanager"
 	"github.com/jaegertracing/jaeger/pkg/clientcfg/clientcfghttp"
@@ -27,7 +29,7 @@ import (
 
 // NewHTTPServer creates a new server that hosts an HTTP/JSON endpoint for clients
 // to query for sampling strategies and baggage restrictions.
-func NewHTTPServer(hostPort string, manager configmanager.ClientConfigManager, mFactory metrics.Factory) *http.Server {
+func NewHTTPServer(hostPort string, manager configmanager.ClientConfigManager, mFactory metrics.Factory, logger *zap.Logger) *http.Server {
 	handler := clientcfghttp.NewHTTPHandler(clientcfghttp.HTTPHandlerParams{
 		ConfigManager:          manager,
 		MetricsFactory:         mFactory,
@@ -35,5 +37,10 @@ func NewHTTPServer(hostPort string, manager configmanager.ClientConfigManager, m
 	})
 	r := mux.NewRouter()
 	handler.RegisterRoutes(r)
-	return &http.Server{Addr: hostPort, Handler: r}
+	errorLog, _ := zap.NewStdLogAt(logger, zapcore.ErrorLevel)
+	return &http.Server{
+		Addr:     hostPort,
+		Handler:  r,
+		ErrorLog: errorLog,
+	}
 }
diff --git a/cmd/collector/app/server/http.go b/cmd/collector/app/server/http.go
index 7860caf0..09edb742 100644
--- a/cmd/collector/app/server/http.go
+++ b/cmd/collector/app/server/http.go
@@ -21,6 +21,7 @@ import (
 	"github.com/gorilla/mux"
 	"github.com/uber/jaeger-lib/metrics"
 	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
 
 	"github.com/jaegertracing/jaeger/cmd/collector/app/handler"
 	"github.com/jaegertracing/jaeger/cmd/collector/app/sampling/strategystore"
@@ -46,7 +47,11 @@ type HTTPServerParams struct {
 func StartHTTPServer(params *HTTPServerParams) (*http.Server, error) {
 	params.Logger.Info("Starting jaeger-collector HTTP server", zap.String("http host-port", params.HostPort))
 
-	server := &http.Server{Addr: params.HostPort}
+	errorLog, _ := zap.NewStdLogAt(params.Logger, zapcore.ErrorLevel)
+	server := &http.Server{
+		Addr:     params.HostPort,
+		ErrorLog: errorLog,
+	}
 	if params.TLSConfig.Enabled {
 		tlsCfg, err := params.TLSConfig.Config(params.Logger) // This checks if the certificates are correctly provided
 		if err != nil {
diff --git a/cmd/collector/app/server/zipkin.go b/cmd/collector/app/server/zipkin.go
index a537b5fe..64ebc3c3 100644
--- a/cmd/collector/app/server/zipkin.go
+++ b/cmd/collector/app/server/zipkin.go
@@ -23,6 +23,7 @@ import (
 	"github.com/rs/cors"
 	"github.com/uber/jaeger-lib/metrics"
 	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
 
 	"github.com/jaegertracing/jaeger/cmd/collector/app/handler"
 	"github.com/jaegertracing/jaeger/cmd/collector/app/zipkin"
@@ -56,7 +57,11 @@ func StartZipkinServer(params *ZipkinServerParams) (*http.Server, error) {
 		return nil, err
 	}
 
-	server := &http.Server{Addr: params.HostPort}
+	errorLog, _ := zap.NewStdLogAt(params.Logger, zapcore.ErrorLevel)
+	server := &http.Server{
+		Addr:     params.HostPort,
+		ErrorLog: errorLog,
+	}
 	serveZipkin(server, listener, params)
 
 	return server, nil
diff --git a/cmd/flags/admin.go b/cmd/flags/admin.go
index 43f54e82..734b9978 100644
--- a/cmd/flags/admin.go
+++ b/cmd/flags/admin.go
@@ -24,6 +24,7 @@ import (
 
 	"github.com/spf13/viper"
 	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
 
 	"github.com/jaegertracing/jaeger/pkg/healthcheck"
 	"github.com/jaegertracing/jaeger/pkg/recoveryhandler"
@@ -107,7 +108,12 @@ func (s *AdminServer) serveWithListener(l net.Listener) {
 	version.RegisterHandler(s.mux, s.logger)
 	s.registerPprofHandlers()
 	recoveryHandler := recoveryhandler.NewRecoveryHandler(s.logger, true)
-	s.server = &http.Server{Handler: recoveryHandler(s.mux)}
+	errorLog, _ := zap.NewStdLogAt(s.logger, zapcore.ErrorLevel)
+	s.server = &http.Server{
+		Handler:  recoveryHandler(s.mux),
+		ErrorLog: errorLog,
+	}
+
 	s.logger.Info("Starting admin HTTP server", zap.String("http-addr", s.adminHostPort))
 	go func() {
 		switch err := s.server.Serve(l); err {
diff --git a/cmd/query/app/server.go b/cmd/query/app/server.go
index 4a460cae..3c410294 100644
--- a/cmd/query/app/server.go
+++ b/cmd/query/app/server.go
@@ -26,6 +26,7 @@ import (
 	"github.com/opentracing/opentracing-go"
 	"github.com/soheilhy/cmux"
 	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/credentials"
 
@@ -50,6 +51,7 @@ type Server struct {
 	conn               net.Listener
 	grpcConn           net.Listener
 	httpConn           net.Listener
+	cmuxServer         cmux.CMux
 	grpcServer         *grpc.Server
 	httpServer         *http.Server
 	separatePorts      bool
@@ -78,7 +80,7 @@ func NewServer(logger *zap.Logger, querySvc *querysvc.QueryService, metricsQuery
 		return nil, err
 	}
 
-	httpServer, cancelFunc, err := createHTTPServer(querySvc, metricsQuerySvc, options, tracer, logger)
+	httpServer, closeGRPCGateway, err := createHTTPServer(querySvc, metricsQuerySvc, options, tracer, logger)
 	if err != nil {
 		return nil, err
 	}
@@ -92,7 +94,7 @@ func NewServer(logger *zap.Logger, querySvc *querysvc.QueryService, metricsQuery
 		httpServer:         httpServer,
 		separatePorts:      grpcPort != httpPort,
 		unavailableChannel: make(chan healthcheck.Status),
-		grpcGatewayCancel:  cancelFunc,
+		grpcGatewayCancel:  closeGRPCGateway,
 	}, nil
 }
 
@@ -145,9 +147,9 @@ func createHTTPServer(querySvc *querysvc.QueryService, metricsQuerySvc querysvc.
 		r = r.PathPrefix(queryOpts.BasePath).Subrouter()
 	}
 
-	ctx, cancelFunc := context.WithCancel(context.Background())
+	ctx, closeGRPCGateway := context.WithCancel(context.Background())
 	if err := apiv3.RegisterGRPCGateway(ctx, logger, r, queryOpts.BasePath, queryOpts.GRPCHostPort, queryOpts.TLSGRPC); err != nil {
-		cancelFunc() // make go vet happy
+		closeGRPCGateway()
 		return nil, nil, err
 	}
 
@@ -161,20 +163,22 @@ func createHTTPServer(querySvc *querysvc.QueryService, metricsQuerySvc querysvc.
 	handler = handlers.CompressHandler(handler)
 	recoveryHandler := recoveryhandler.NewRecoveryHandler(logger, true)
 
+	errorLog, _ := zap.NewStdLogAt(logger, zapcore.ErrorLevel)
 	server := &http.Server{
-		Handler: recoveryHandler(handler),
+		Handler:  recoveryHandler(handler),
+		ErrorLog: errorLog,
 	}
 
 	if queryOpts.TLSHTTP.Enabled {
 		tlsCfg, err := queryOpts.TLSHTTP.Config(logger) // This checks if the certificates are correctly provided
 		if err != nil {
-			cancelFunc()
+			closeGRPCGateway()
 			return nil, nil, err
 		}
 		server.TLSConfig = tlsCfg
 
 	}
-	return server, cancelFunc, nil
+	return server, closeGRPCGateway, nil
 }
 
 // initListener initialises listeners of the server
@@ -222,7 +226,6 @@ func (s *Server) initListener() (cmux.CMux, error) {
 	s.httpConn = cmuxServer.Match(cmux.Any())
 
 	return cmuxServer, nil
-
 }
 
 // Start http, GRPC and cmux servers concurrently
@@ -231,6 +234,7 @@ func (s *Server) Start() error {
 	if err != nil {
 		return err
 	}
+	s.cmuxServer = cmuxServer
 
 	var tcpPort int
 	if !s.separatePorts {
@@ -283,7 +287,7 @@ func (s *Server) Start() error {
 			s.logger.Info("Starting CMUX server", zap.Int("port", tcpPort), zap.String("addr", s.queryOptions.HTTPHostPort))
 
 			err := cmuxServer.Serve()
-			// TODO: Remove string comparison when https://github.com/soheilhy/cmux/pull/69 is merged
+			// TODO: find a way to avoid string comparison. Even though cmux has ErrServerClosed, it's not returned here.
 			if err != nil && !strings.Contains(err.Error(), "use of closed network connection") {
 				s.logger.Error("Could not start multiplexed server", zap.Error(err))
 			}
@@ -305,6 +309,7 @@ func (s *Server) Close() error {
 		s.httpConn.Close()
 		s.grpcConn.Close()
 	} else {
+		s.cmuxServer.Close()
 		s.conn.Close()
 	}
 	return nil



Test patch:
diff --git a/cmd/agent/app/httpserver/srv_test.go b/cmd/agent/app/httpserver/srv_test.go
index e14427ff..4edda60a 100644
--- a/cmd/agent/app/httpserver/srv_test.go
+++ b/cmd/agent/app/httpserver/srv_test.go
@@ -19,9 +19,10 @@ import (
 	"testing"
 
 	"github.com/stretchr/testify/assert"
+	"go.uber.org/zap"
 )
 
 func TestHTTPServer(t *testing.T) {
-	s := NewHTTPServer(":1", nil, nil)
+	s := NewHTTPServer(":1", nil, nil, zap.NewNop())
 	assert.NotNil(t, s)
 }