Repository: pinpoint-apm/pinpoint-c-agent (language: python)

Issue:
panic: send on closed channel
**Describe the bug**


```
2024.08.01T02:57:12.796panic: send on closed channel
2024.08.01T02:57:12.796goroutine 38 [running]:
2024.08.01T02:57:12.796github.com/pinpoint-apm/pinpoint-c-agent/collector-agent/agent.(*SpanSender).Interceptor(0xc0002bc1e8, 0xc0005a4180?)
2024.08.01T02:57:12.796	/apps/agent/SpanSender.go:344 +0x165
2024.08.01T02:57:12.796github.com/pinpoint-apm/pinpoint-c-agent/collector-agent/agent.(*GrpcAgent).consumeJsonSpan(0xc0002bc180)
2024.08.01T02:57:12.796	/apps/agent/GrpcAgent.go:458 +0x1c9
2024.08.01T02:57:12.796created by github.com/pinpoint-apm/pinpoint-c-agent/collector-agent/agent.(*GrpcAgent).Start
2024.08.01T02:57:12.796	/apps/agent/GrpcAgent.go:311 +0x56

```

panic: send on closed channel
**Describe the bug**


```
2024.08.01T02:57:12.796panic: send on closed channel
2024.08.01T02:57:12.796goroutine 38 [running]:
2024.08.01T02:57:12.796github.com/pinpoint-apm/pinpoint-c-agent/collector-agent/agent.(*SpanSender).Interceptor(0xc0002bc1e8, 0xc0005a4180?)
2024.08.01T02:57:12.796	/apps/agent/SpanSender.go:344 +0x165
2024.08.01T02:57:12.796github.com/pinpoint-apm/pinpoint-c-agent/collector-agent/agent.(*GrpcAgent).consumeJsonSpan(0xc0002bc180)
2024.08.01T02:57:12.796	/apps/agent/GrpcAgent.go:458 +0x1c9
2024.08.01T02:57:12.796created by github.com/pinpoint-apm/pinpoint-c-agent/collector-agent/agent.(*GrpcAgent).Start
2024.08.01T02:57:12.796	/apps/agent/GrpcAgent.go:311 +0x56

```



Discussion / hints:

Fix patch:
diff --git a/Changes-CN.md b/Changes-CN.md
deleted file mode 100644
index e7cbfa3f..00000000
--- a/Changes-CN.md
+++ /dev/null
@@ -1,31 +0,0 @@
-# 变化
-
-## v0.4.0-beta  异步监控API
-* [PHP] 添加一 ID 用来跟踪异步调用
-* [python] 同上
-
-## v0.3.2
-* [Python] 支持python2.7 module 
-* [PHP] thinkphp6,Codeigniter4,Symfony framework demo
-* [PHP] 支持Guzzle, MongoClient 
-
-## v0.3.0-beta
-
-- 支持 Python
-  - demo for django
-  - demo for flask
-  - demo for tornado
-  - demo for BaseHTTPRequestHandler
-- 使用更加方便
-- 提供中文使用说明
-
-## v0.2.3 
-
-- 支持 Yii 框架类加载器 [(如何在Yii中使用pinpoint-php agent?)](Example/PHP/demo/yii-demo/Readme-CN.md)
-- 为 Flraum 添加一些开箱即用的插件 [ (如何在Flarum中使用pinpoint-php agent?)](Example/PHP/demo/Flarum-demo/Readme-CN.md)
-- 修正了引用参数的bug
-- 支持返回类型为object
-
-## v0.2.2
-
-- 支持 GRPC
\ No newline at end of file
diff --git a/DOC/PHP/Readme-CN.md b/DOC/PHP/Readme-CN.md
index da07820a..0d95af3e 100644
--- a/DOC/PHP/Readme-CN.md
+++ b/DOC/PHP/Readme-CN.md
@@ -1,3 +1,5 @@
+[English](Readme.md) | [中文](Readme-CN.md) | [한국어](Readme-KR.md)
+
 ## 安装参照
 
 ### 依赖组件
diff --git a/DOC/PHP/Readme-KR.md b/DOC/PHP/Readme-KR.md
index eb2a2a15..6c130914 100644
--- a/DOC/PHP/Readme-KR.md
+++ b/DOC/PHP/Readme-KR.md
@@ -1,3 +1,5 @@
+[English](Readme.md) | [中文](Readme-CN.md) | [한국어](Readme-KR.md)
+
 ## 시작 가이드 
 
 ### 요구
diff --git a/DOC/PHP/Readme.md b/DOC/PHP/Readme.md
index da67418b..23c50ec8 100644
--- a/DOC/PHP/Readme.md
+++ b/DOC/PHP/Readme.md
@@ -1,3 +1,5 @@
+[English](Readme.md) | [中文](Readme-CN.md) | [한국어](Readme-KR.md)
+
 ## Supported
 
 [pinpoint-apm/pinpoint-php-aop#supported-librariesframework ](https://github.com/pinpoint-apm/pinpoint-php-aop?tab=readme-ov-file#supported-librariesframework)
diff --git a/DOC/PY/Readme-CN.md b/DOC/PY/Readme-CN.md
index adbf6fb8..a1ea8a1e 100644
--- a/DOC/PY/Readme-CN.md
+++ b/DOC/PY/Readme-CN.md
@@ -1,3 +1,5 @@
+[English](Readme.md) | [中文](Readme-CN.md) | [한국어](Readme-KR.md)
+
 # pinpoint-py agent 使用说明
 
 ## 入门指南
diff --git a/DOC/PY/Readme-KR.md b/DOC/PY/Readme-KR.md
index 66461331..3647cc29 100644
--- a/DOC/PY/Readme-KR.md
+++ b/DOC/PY/Readme-KR.md
@@ -1,3 +1,5 @@
+[English](Readme.md) | [中文](Readme-CN.md) | [한국어](Readme-KR.md)
+
 # pinpoint-py agent 가이드
 
 ## 시작 가이드
diff --git a/DOC/PY/Readme.md b/DOC/PY/Readme.md
index a36c4c6d..64159db1 100644
--- a/DOC/PY/Readme.md
+++ b/DOC/PY/Readme.md
@@ -1,4 +1,5 @@
-﻿# Tutorial for pinpoint-py agent
+﻿[English](Readme.md) | [中文](Readme-CN.md) | [한국어](Readme-KR.md)
+
 
 ## Getting Started
 
@@ -15,7 +16,7 @@
 $ pip install pinpointPy
 ```
 
-### How to Use
+### Choose your framework
 
 
 #### 1.1 Flask
diff --git a/README.md b/README.md
index 8f80a275..a1ab5553 100644
--- a/README.md
+++ b/README.md
@@ -1,47 +1,78 @@
 ![pinpoint](images/logo.png)
 
-[![CI](https://github.com/pinpoint-apm/pinpoint-c-agent/actions/workflows/main.yml/badge.svg)](https://github.com/pinpoint-apm/pinpoint-c-agent/actions/workflows/main.yml) [![Gitter](https://badges.gitter.im/naver/pinpoint-c-agent.svg)](https://gitter.im/naver/pinpoint-c-agent?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) [![codecov](https://codecov.io/gh/pinpoint-apm/pinpoint-c-agent/branch/master/graph/badge.svg?token=KswbmFvWp3)](https://codecov.io/gh/pinpoint-apm/pinpoint-c-agent) [![License](https://img.shields.io/github/license/pinpoint-apm/pinpoint-c-agent)](LICENSE)
-
+[![CI](https://github.com/pinpoint-apm/pinpoint-c-agent/actions/workflows/main.yml/badge.svg)](https://github.com/pinpoint-apm/pinpoint-c-agent/actions/workflows/main.yml) [![Gitter](https://badges.gitter.im/naver/pinpoint-c-agent.svg)](https://gitter.im/naver/pinpoint-c-agent?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) [![codecov](https://codecov.io/gh/pinpoint-apm/pinpoint-c-agent/branch/master/graph/badge.svg?token=KswbmFvWp3)](https://codecov.io/gh/pinpoint-apm/pinpoint-c-agent) [![License](https://img.shields.io/github/license/pinpoint-apm/pinpoint-c-agent)](LICENSE) \
+`pinpoint_php_ext:` [![pinpoint_php](https://img.shields.io/badge/php-7.1~8.3-8892BF)](https://pecl.php.net/package/pinpoint_php)\
+`pinpoint-php-aop:` ![Packagist License](https://img.shields.io/packagist/l/pinpoint-apm/pinpoint-php-aop)
+ [![Total Downloads](https://img.shields.io/packagist/dt/pinpoint-apm/pinpoint-php-aop.svg?style=flat-square)](https://packagist.org/packages/pinpoint-apm/pinpoint-php-aop)\
+`pinpointpy:` ![PyPI - License](https://img.shields.io/pypi/l/pinpointpy)
+ [![pypi](https://badge.fury.io/py/pinpointpy.svg)](https://pypi.org/project/pinpointpy/)
+![PyPI - Downloads](https://img.shields.io/pypi/dm/pinpointpy)
 
 
 **Visit [our official website](http://pinpoint-apm.github.io/pinpoint/) for more information and [the Latest updates on Pinpoint](https://pinpoint-apm.github.io/pinpoint/news.html)**  
 
 
-The current stable version is [the Latest](https://github.com/pinpoint-apm/pinpoint-c-agent/releases).
+# Pinpoint `C` Agent
+
+> What's `C`
+
+1. Use `C/Cpp` common API
+2. `C`ross-platform: windows/*nux/macOS 
+
+Pinpoint C Agent helps your monitor your PHP/PYTHON applications into [pinpoint-apm](https://github.com/pinpoint-apm/pinpoint).\
+Our advantage:
+1. Continuous maintenance and optimization. (Since Jul 31,2018)
+2. Auto-injection, that means less aggression against your code.
+    - python: leverage [PEP 318 Decorator](https://peps.python.org/pep-0318/) 
+    - php: leverage `CG(*_table)` in php kernel and AST parser([nikic/PHP-Parser](https://github.com/nikic/PHP-Parser))
+3. Cross-platform: windows/*nux/macOS. It helps address the problems even when you developing.
+
+## Python Agent
+
+> py 3.8+
+
+`pip install pinpointPy`
+
+[Full guide for python](DOC/PY/Readme.md) 
 
-# Pinpoint Agent
+## PHP Agent
 
-It is an agent written by C++, PHP and Python languages. And we hope to support other languages by this agent. Until now, it supports **_PHP_**, **_C/CPP_** and **_PYTHON_**.
+> php 7.1+
 
-## Overview Pinpoint Agent
+1. Install extension
 
-### How does it work
+` pecl install pinpoint_php `
 
-![How does it work](images/pinpoint_v0.5.x.png)
+2. Import pinpoint aspect plugin
 
-### Pinpoint Agent
+`composer require pinpoint-apm/pinpoint-php-aop`
 
-language| tutorial
----|---
-php|[EN](DOC/PHP/Readme.md) [CN](DOC/PHP/Readme-CN.md) [KR](DOC/PHP/Readme-KR.md)
-python3|[EN](DOC/PY/Readme.md) [CN](DOC/PY/Readme-CN.md) [KR](DOC/PY/Readme-KR.md)
-c/cpp|[English](DOC/C-CPP/Readme.md)
+[Full guide for PHP](DOC/PHP/Readme.md) 
 
-> [Try playground](/testapps/readme.md)
+## C/CPP built in API
+
+[Full guide for C](DOC/C-CPP/Readme.md)
+
+
+## Our agent playground
+
+`cd testapps && docker compose build --build-arg PHP_VERSION=7.4 && docker compose up`
+
+[Tested project](/testapps/readme.md)
 
 ### Blogs 
 
-- Intro pinpoint php aop : https://github.com/pinpoint-apm/pinpoint-php-aop/wiki/Intro-pinpoint-php-aop
-- pinpoint php aop 内部原理: https://github.com/pinpoint-apm/pinpoint-php-aop/wiki/pinpoint-php-aop-%E5%86%85%E9%83%A8%E5%8E%9F%E7%90%86
-- pinpoint php aop 내부 원리: https://github.com/pinpoint-apm/pinpoint-php-aop/wiki/pinpoint-php-aop-%EB%82%B4%EB%B6%80-%EC%9B%90%EB%A6%AC
+- [Intro pinpoint php aop](https://github.com/pinpoint-apm/pinpoint-php-aop/wiki/Intro-pinpoint-php-aop)
+- [pinpoint php aop 内部原理](https://github.com/pinpoint-apm/pinpoint-php-aop/wiki/pinpoint-php-aop-%E5%86%85%E9%83%A8%E5%8E%9F%E7%90%86)
+- [pinpoint php aop 내부 원리](https://github.com/pinpoint-apm/pinpoint-php-aop/wiki/pinpoint-php-aop-%EB%82%B4%EB%B6%80-%EC%9B%90%EB%A6%AC)
 
 
 ## Contact Us
 
-* Submit an [issue](https://github.com/pinpoint-apm/pinpoint-c-agent/issues)
+* Submit an [issue](https://github.com/pinpoint-apm/pinpoint-c-agent/issues) 👍👍
+* [Email Us](mailto:dl_cd_pinpoint@navercorp.com)   👍👍
 * [Gitter char room](https://gitter.im/naver/pinpoint-c-agent)
-* dl_cd_pinpoint@navercorp.com
-* Chinese Community (QQ Group: 882020485)
+* QQ us (QQ Group: 882020485)
     <details>
     <summary> show more 
     </summary>
@@ -49,11 +80,12 @@ c/cpp|[English](DOC/C-CPP/Readme.md)
     QQ Group1: 897594820 | QQ Group2: 812507584 | QQ Group3: 882020485| DING Group : 21981598
     :----------------: |:----------------: | :-----------: | :-----------: 
     ![QQ Group1](images/NAVERPinpoint.png) | ![QQ Group2](images/NAVERPinpoint2.png)| ![QQ Group3](images/NAVERPinpoint3.png)| ![DING Group](images/NaverPinpoint交流群-DING.jpg)
-
 </details>
 
 ## Contributing
 
+Thanks all [![GitHub contributors](https://img.shields.io/github/contributors/pinpoint-apm/pinpoint-c-agent)](https://github.com/pinpoint-apm/pinpoint-c-agent/graphs/contributors)
+
 We are looking forward to your contributions via pull requests.
 
 ## License
diff --git a/collector-agent/CHANGES.md b/collector-agent/CHANGES.md
index e2d2a9ab..ffe90cb2 100644
--- a/collector-agent/CHANGES.md
+++ b/collector-agent/CHANGES.md
@@ -1,4 +1,8 @@
+## v0.6.4
+- fix: panic: send on closed channel #658
+
 ## v0.5.3 2024.05.31
 - support go install
+
 ## [v0.5.0] start time in ms
 - return current time in ms
\ No newline at end of file
diff --git a/collector-agent/agent/AgentRouter.go b/collector-agent/agent/AgentRouter.go
index 5b471063..00a53051 100644
--- a/collector-agent/agent/AgentRouter.go
+++ b/collector-agent/agent/AgentRouter.go
@@ -189,9 +189,10 @@ func (manager *AgentRouter) Clean() {
 			manager.rwMutex.RUnlock()
 			manager.rwMutex.Lock()
 			delete(manager.AgentMap, id)
-			agent.Stop()
 			manager.rwMutex.Unlock()
 			manager.rwMutex.RLock()
+			//shrink rwMutex scope
+			agent.Stop()
 		}
 	}
 	manager.rwMutex.RUnlock()
diff --git a/collector-agent/agent/ErrorAnalysis.go b/collector-agent/agent/ErrorAnalysis.go
index 2f3e46fa..159000d4 100644
--- a/collector-agent/agent/ErrorAnalysis.go
+++ b/collector-agent/agent/ErrorAnalysis.go
@@ -32,7 +32,7 @@ func (e *ErrorAnalysisFilter) sendExpMetaData(meta *v1.PExceptionMetaData) {
 	defer conn.Close()
 	client := v1.NewMetadataClient(conn)
 
-	ctx, cancel := common.BuildPinpointCtx(config.MetaDataTimeWaitSec, e.md)
+	ctx, cancel := common.BuildPinpointCtx(config.MetaDataTimeWait, e.md)
 	defer cancel()
 	result, err := client.RequestExceptionMetaData(ctx, meta)
 	if err != nil {
diff --git a/collector-agent/agent/GrpcAgent.go b/collector-agent/agent/GrpcAgent.go
index 42e1f697..ac68bdaa 100644
--- a/collector-agent/agent/GrpcAgent.go
+++ b/collector-agent/agent/GrpcAgent.go
@@ -31,7 +31,7 @@ type GrpcAgent struct {
 	requestCounter      RequestProfiler
 	utReport            *UrlTemplateReport
 	tasksGroup          sync.WaitGroup
-	tSpanCh             chan *TSpan
+	tSpanBufCh          chan *TSpan
 	ExitCh              chan bool
 	log                 *log.Entry
 	errorAnalysisFilter *ErrorAnalysisFilter
@@ -52,7 +52,7 @@ func (agent *GrpcAgent) SendSpan(span *TSpan) {
 			log.Warnf("sendSpan met:%s", r)
 		}
 	}()
-	agent.tSpanCh <- span
+	agent.tSpanBufCh <- span
 }
 
 func (agent *GrpcAgent) GetLastBusyTime() int64 {
@@ -61,7 +61,6 @@ func (agent *GrpcAgent) GetLastBusyTime() int64 {
 
 func (agent *GrpcAgent) Stop() {
 	agent.log.Warn("I'm exiting")
-	close(agent.tSpanCh)
 	close(agent.ExitCh)
 	agent.tasksGroup.Wait()
 	agent.log.Warn("I'm exit")
@@ -85,7 +84,7 @@ func (agent *GrpcAgent) String() string {
 	return fmt.Sprintf("id:%s name:%s type:%d startTime:%s", agent.AgentId, agent.agentName, agent.agentType, agent.StartTime)
 }
 
-func (agent *GrpcAgent) agentOnline() error {
+func (agent *GrpcAgent) handleRegisterAgent() error {
 
 	commandTask := sync.WaitGroup{}
 	defer commandTask.Wait()
@@ -107,7 +106,7 @@ func (agent *GrpcAgent) agentOnline() error {
 	}()
 
 	client := v1.NewAgentClient(conn)
-	ctx, cancel := common.BuildPinpointCtx(config.GrpcConTextTimeOutSec, agent.BaseMD)
+	ctx, cancel := common.BuildPinpointCtx(config.GrpcConTextTimeOut, agent.BaseMD)
 	defer cancel()
 	pbAgentInfo := common.GetPBAgentInfo(agent.agentType)
 	agent.log.Debugf("RequestAgentInfo pbAgentInfo:%v", pbAgentInfo)
@@ -136,6 +135,7 @@ func (agent *GrpcAgent) agentOnline() error {
 	// handle command
 	go agent.handleCommand(conn, &commandTask)
 
+	agent.log.Info("agent online ")
 	agent.AgentOnLine = true
 
 	defer func() { agent.AgentOnLine = false }()
@@ -161,20 +161,18 @@ func (agent *GrpcAgent) agentOnline() error {
 }
 
 func (agent *GrpcAgent) keepAgentOnline() {
-	agent.tasksGroup.Add(1)
 	defer agent.tasksGroup.Done()
 
 	for {
-		if err := agent.agentOnline(); err != nil {
+		if err := agent.handleRegisterAgent(); err != nil {
 			agent.log.Infof("agent online exit:%s ", err)
 		}
 
 		config := common.GetConfig()
-		if common.WaitChannelEvent(agent.ExitCh, config.PingInterval) == common.E_AGENT_STOPPING {
+		if common.WaitChannelEvent(agent.ExitCh, config.AgentReTryTimeout) == common.E_AGENT_STOPPING {
 			break
 		}
 	}
-	agent.spanSender.Stop()
 }
 
 func (agent *GrpcAgent) registerFilter() {
@@ -235,7 +233,7 @@ func (agent *GrpcAgent) sendStat() {
 		for {
 			msg := CollectPStateMessage(agent.requestCounter.GetMaxAvg, agent.requestCounter.GetReqTimeProfiler)
 
-			agent.log.Debugf("%v", msg)
+			agent.log.Debugf("PStatMessage: %v", msg)
 			if err := stream.Send(msg); err != nil {
 				agent.log.Warn(err)
 				break
@@ -270,7 +268,7 @@ func (agent *GrpcAgent) sendStat() {
 }
 
 func (agent *GrpcAgent) uploadStatInfo() {
-	agent.tasksGroup.Add(1)
+
 	defer agent.tasksGroup.Done()
 
 	for {
@@ -309,22 +307,25 @@ func (agent *GrpcAgent) Init(id, _name string, _type int32, StartTime string) {
 
 	config := common.GetConfig()
 
-	agent.tSpanCh = make(chan *TSpan, config.AgentChannelSize)
+	agent.tSpanBufCh = make(chan *TSpan, config.AgentChannelSize)
 	agent.ExitCh = make(chan bool)
-	agent.spanSender = createSpanSender(agent.BaseMD, agent.ExitCh)
+	agent.spanSender = createSpanSender(agent.BaseMD, agent.ExitCh, &agent.tasksGroup, agent.log)
 	agent.requestCounter.CTime = time.Now().Unix()
 
 	agent.errorAnalysisFilter = createErrorAnalysisFilter(agent.BaseMD)
 
 	agent.registerFilter()
+	agent.tasksGroup.Add(1)
 	// start agentOnline
 	go agent.keepAgentOnline()
 	// send stat
+	agent.tasksGroup.Add(1)
 	go agent.uploadStatInfo()
 
 }
 
 func (agent *GrpcAgent) Start() {
+	agent.tasksGroup.Add(1)
 	go agent.consumeJsonSpan()
 }
 
@@ -463,18 +464,17 @@ func (agent *GrpcAgent) handleCommand(conn *grpc.ClientConn, wg *sync.WaitGroup)
 
 func (agent *GrpcAgent) consumeJsonSpan() {
 	defer agent.tasksGroup.Done()
-	agent.tasksGroup.Add(1)
-
-	for span := range agent.tSpanCh {
-		if span == nil {
-			agent.log.Infof("agent:%v get EOF", agent)
-			return
-		}
-
-		for _, filter := range agent.spanFilters {
-			if !filter.Interceptor(span) {
-				break
+	for {
+		select {
+		case span := <-agent.tSpanBufCh:
+			for _, filter := range agent.spanFilters {
+				if !filter.Interceptor(span) {
+					break
+				}
 			}
+		case <-agent.spanSender.exitCh:
+			agent.log.Warn("consumeJsonSpan task done, as agent exit")
+			return
 		}
 	}
 }
diff --git a/collector-agent/agent/SpanSender.go b/collector-agent/agent/SpanSender.go
index 11020095..f5efb4db 100644
--- a/collector-agent/agent/SpanSender.go
+++ b/collector-agent/agent/SpanSender.go
@@ -7,7 +7,6 @@ import (
 	"strconv"
 	"strings"
 	"sync"
-	"time"
 
 	"github.com/golang/protobuf/ptypes/wrappers"
 	"github.com/pinpoint-apm/pinpoint-c-agent/collector-agent/common"
@@ -24,37 +23,33 @@ type ApiIdMap map[string]interface{}
 var unique_id_count = int32(1)
 
 type SpanSender struct {
-	sequenceId    int32
-	idMap         ApiIdMap
-	Md            metadata.MD
-	exitCh        chan bool
-	spanMessageCh chan *v1.PSpanMessage
-	spanRespCh    chan int32
-	wg            sync.WaitGroup
+	sequenceId          int32
+	idMap               ApiIdMap
+	Md                  metadata.MD
+	exitCh              chan bool
+	spanMessageBufferCh chan *v1.PSpanMessage
+	sendStreamRespCh    chan int32
+	wg                  *sync.WaitGroup
+	log                 *log.Entry
 }
 
-func createSpanSender(base metadata.MD, exitCh chan bool) *SpanSender {
+func createSpanSender(base metadata.MD, exitCh chan bool, agent_wg *sync.WaitGroup, log *log.Entry) *SpanSender {
 	sender := &SpanSender{
-		Md: base, exitCh: exitCh,
-		idMap: make(ApiIdMap),
+		Md:     base,
+		exitCh: exitCh,
+		idMap:  make(ApiIdMap),
+		wg:     agent_wg,
+		log:    log,
 	}
 	sender.Init()
 	return sender
 }
 
-func (spanSender *SpanSender) Stop() {
-	log.Warn("Try to close spanSend goroutine")
-	close(spanSender.spanMessageCh)
-	close(spanSender.spanRespCh)
-	spanSender.wg.Wait()
-	log.Warn("Span sender goroutine exit")
-}
-
-func (spanSender *SpanSender) senderMain() {
+func (spanSender *SpanSender) sendSpan() {
 	config := common.GetConfig()
 	conn, err := common.CreateGrpcConnection(config.SpanAddress)
 	if err != nil {
-		log.Warnf("connect:%s failed. %s", config.SpanAddress, err)
+		spanSender.log.Warnf("connect:%s failed. %s", config.SpanAddress, err)
 		return
 	}
 	defer conn.Close()
@@ -64,35 +59,52 @@ func (spanSender *SpanSender) senderMain() {
 
 	stream, err := client.SendSpan(ctx)
 	if err != nil {
-		log.Warnf("create stream failed. %s", err)
+		spanSender.log.Warnf("create stream failed. %s", err)
 		return
 	}
 	defer stream.CloseSend()
 
-	for span := range spanSender.spanMessageCh {
-		log.Debugf("send %v", span)
-
-		if err := stream.Send(span); err != nil {
-			log.Warnf("send span failed with:%s", err)
-			// response the stream is not available
-			spanSender.spanRespCh <- 500
+	// for span := range spanSender.spanMessageBufferCh {
+	// 	spanSender.log.Debugf("send %v", span)
+
+	// 	if err := stream.Send(span); err != nil {
+	// 		spanSender.log.Warnf("send span failed with:%s", err)
+	// 		// response the stream is not available
+	// 		spanSender.sendStreamRespCh <- 500
+	// 		return
+	// 	}
+	// }
+	// watch exitCh
+	for {
+		select {
+		case span := <-spanSender.spanMessageBufferCh:
+			spanSender.log.Debugf("send %v", span)
+
+			if err := stream.Send(span); err != nil {
+				spanSender.log.Warnf("send span failed with:%s", err)
+				// response the stream is not available
+				spanSender.sendStreamRespCh <- 500
+				return
+			}
+		case <-spanSender.exitCh:
+			spanSender.log.Warn("sendSpan failed with agent exiting")
 			return
 		}
 	}
+
 }
 
 func (spanSender *SpanSender) sendThread() {
 	defer spanSender.wg.Done()
 
-	spanSender.wg.Add(1)
 	for {
-		spanSender.senderMain()
+		spanSender.sendSpan()
 		config := common.GetConfig()
-		if common.WaitChannelEvent(spanSender.exitCh, config.SpanTimeWaitSec) == common.E_AGENT_STOPPING {
+		if common.WaitChannelEvent(spanSender.exitCh, config.SpanTimeWait) == common.E_AGENT_STOPPING {
 			break
 		}
 	}
-	log.Info("sendThread exit")
+	spanSender.log.Info("sendThread exit")
 }
 
 func (spanSender *SpanSender) Init() {
@@ -100,17 +112,18 @@ func (spanSender *SpanSender) Init() {
 	// spanSender.apiMeta = MetaData{MetaDataType: common.META_API, IDMap: make(PARAMS_TYPE), Sender: spanSender}
 	// spanSender.stringMeta = MetaData{MetaDataType: common.META_STRING, IDMap: make(PARAMS_TYPE), Sender: spanSender}
 
-	spanSender.spanMessageCh = make(chan *v1.PSpanMessage, common.GetConfig().AgentChannelSize)
-	spanSender.spanRespCh = make(chan int32, 1)
-	log.Debug("SpanSender::Init span spanSender thread start")
+	spanSender.spanMessageBufferCh = make(chan *v1.PSpanMessage, common.GetConfig().AgentChannelSize)
+	spanSender.sendStreamRespCh = make(chan int32, 1)
+	spanSender.log.Debug("SpanSender::Init span spanSender thread start")
 	for i := int32(0); i < common.GetConfig().SpanStreamParallelismSize; i++ {
+		spanSender.wg.Add(1)
 		go spanSender.sendThread()
 	}
-	log.Debug("SpanSender::Init done")
+	spanSender.log.Debug("SpanSender::Init done")
 }
 
 func (spanSender *SpanSender) cleanAllMetaData() {
-	log.Info("Clean all metaData")
+	spanSender.log.Info("Clean all metaData")
 	spanSender.idMap = make(ApiIdMap)
 }
 
@@ -365,24 +378,23 @@ func (spanSender *SpanSender) makeSpan(span *TSpan) (*v1.PSpan, error) {
 }
 
 func (spanSender *SpanSender) Interceptor(span *TSpan) bool {
-	log.Debug("span spanSender interceptor")
+	spanSender.log.Debug("span spanSender interceptor")
 	if pbSpan, err := spanSender.makeSpan(span); err == nil {
-		// send channel
-		spanSender.spanMessageCh <- &v1.PSpanMessage{
+		// recv the channel status
+		select {
+		case spanSender.spanMessageBufferCh <- &v1.PSpanMessage{
 			Field: &v1.PSpanMessage_Span{
 				Span: pbSpan,
 			},
-		}
-		// recv the channel status
-		select {
-		case statusCode := <-spanSender.spanRespCh:
-			log.Warnf("span send stream is offline statusCode:%d, clear all string/sql/api meta data", statusCode)
+		}:
+		case statusCode := <-spanSender.sendStreamRespCh:
+			spanSender.log.Warnf("span send stream is offline statusCode:%d, clear all string/sql/api meta data", statusCode)
 			spanSender.cleanAllMetaData()
-		case <-time.After(0 * time.Second):
-			// do nothing, just go on
+		default:
+			spanSender.log.Warn("current span dropped, due to spanStream slow or disconnected and spanMessageBufferCh is full")
 		}
 	} else {
-		log.Warnf("SpanSender::Interceptor return err:%s", err)
+		spanSender.log.Warnf("SpanSender::Interceptor return err:%s", err)
 	}
 	return true
 }
@@ -391,14 +403,14 @@ func (spanSender *SpanSender) SenderGrpcMetaData(name string, metaType int32) er
 	config := common.GetConfig()
 	conn, err := common.CreateGrpcConnection(config.AgentAddress)
 	if err != nil {
-		log.Warnf("connect:%s failed. %s", config.AgentAddress, err)
+		spanSender.log.Warnf("connect:%s failed. %s", config.AgentAddress, err)
 		return errors.New("SenderGrpcMetaData: connect failed")
 	}
 
 	defer conn.Close()
 	client := v1.NewMetadataClient(conn)
 
-	ctx, cancel := common.BuildPinpointCtx(config.MetaDataTimeWaitSec, spanSender.Md)
+	ctx, cancel := common.BuildPinpointCtx(config.MetaDataTimeWait, spanSender.Md)
 	defer cancel()
 
 	switch metaType {
@@ -408,7 +420,7 @@ func (spanSender *SpanSender) SenderGrpcMetaData(name string, metaType int32) er
 			apiMeta := v1.PApiMetaData{ApiId: id, ApiInfo: name, Type: common.API_DEFAULT}
 
 			if _, err = client.RequestApiMetaData(ctx, &apiMeta); err != nil {
-				log.Warnf("agentOnline api meta failed %s", err)
+				spanSender.log.Warnf("agentOnline api meta failed %s", err)
 				return errors.New("SenderGrpcMetaData: PApiMetaData failed")
 			}
 		}
@@ -419,7 +431,7 @@ func (spanSender *SpanSender) SenderGrpcMetaData(name string, metaType int32) er
 			apiMeta := v1.PApiMetaData{ApiId: id, ApiInfo: name, Type: common.API_WEB_REQUEST}
 
 			if _, err = client.RequestApiMetaData(ctx, &apiMeta); err != nil {
-				log.Warnf("agentOnline api meta failed %s", err)
+				spanSender.log.Warnf("agentOnline api meta failed %s", err)
 				return errors.New("SenderGrpcMetaData: PApiMetaData failed")
 			}
 		}
@@ -432,7 +444,7 @@ func (spanSender *SpanSender) SenderGrpcMetaData(name string, metaType int32) er
 			}
 
 			if _, err = client.RequestStringMetaData(ctx, &metaMeta); err != nil {
-				log.Warnf("agentOnline api meta failed %s", err)
+				spanSender.log.Warnf("agentOnline api meta failed %s", err)
 				return errors.New("SenderGrpcMetaData: RequestStringMetaData failed")
 			}
 		}
@@ -445,14 +457,14 @@ func (spanSender *SpanSender) SenderGrpcMetaData(name string, metaType int32) er
 				Sql:    name,
 			}
 			if _, err = client.RequestSqlUidMetaData(ctx, &sqlUidMeta); err != nil {
-				log.Warnf("agentOnline api meta failed %s", err)
+				spanSender.log.Warnf("agentOnline api meta failed %s", err)
 				return errors.New("SenderGrpcMetaData: RequestSqlUidMetaData failed")
 			}
 		}
 	default:
-		log.Warnf("SenderGrpcMetaData: No such Type:%d", metaType)
+		spanSender.log.Warnf("SenderGrpcMetaData: No such Type:%d", metaType)
 	}
 
-	log.Debugf("send metaData %s", name)
+	spanSender.log.Debugf("send metaData %s", name)
 	return nil
 }
diff --git a/collector-agent/common/Utils.go b/collector-agent/common/Utils.go
index 772ce77c..1788699c 100644
--- a/collector-agent/common/Utils.go
+++ b/collector-agent/common/Utils.go
@@ -65,7 +65,7 @@ func BuildPinpointCtx(sec time.Duration, md metadata.MD) (ctx context.Context, c
 }
 
 func CreateGrpcConnection(address string) (*grpc.ClientConn, error) {
-	ctx, cancel := context.WithTimeout(context.Background(), time.Second*config.GrpcConTextTimeOutSec)
+	ctx, cancel := context.WithTimeout(context.Background(), time.Second*config.GrpcConTextTimeOut)
 	defer cancel()
 	conn, err := grpc.DialContext(ctx, address, GetDialOption()...)
 	if err != nil {
diff --git a/collector-agent/common/config.go b/collector-agent/common/config.go
index 41c7b260..8dca7988 100644
--- a/collector-agent/common/config.go
+++ b/collector-agent/common/config.go
@@ -24,12 +24,12 @@ type Config struct {
 	AgentChannelSize          uint64
 	SpanStreamParallelismSize int32
 
-	AgentReTryTimeout     time.Duration
-	PingInterval          time.Duration
-	StatInterval          time.Duration
-	SpanTimeWaitSec       time.Duration
-	MetaDataTimeWaitSec   time.Duration
-	GrpcConTextTimeOutSec time.Duration
+	AgentReTryTimeout  time.Duration
+	PingInterval       time.Duration
+	StatInterval       time.Duration
+	SpanTimeWait       time.Duration
+	MetaDataTimeWait   time.Duration
+	GrpcConTextTimeOut time.Duration
 	// agent auto offline after xxxx
 	AgentFreeOnlineSurvivalTimeSec time.Duration
 	// agentInfo
@@ -48,9 +48,9 @@ var config = &Config{
 	AgentReTryTimeout:              10,
 	PingInterval:                   300,
 	StatInterval:                   5,
-	SpanTimeWaitSec:                10,
-	MetaDataTimeWaitSec:            10,
-	GrpcConTextTimeOutSec:          5,
+	SpanTimeWait:                   10,
+	MetaDataTimeWait:               10,
+	GrpcConTextTimeOut:             5,
 	AgentFreeOnlineSurvivalTimeSec: 1 * 60 * 60, // 1 hour
 	ServerType:                     1700,        // PHP
 	LoggerLevel:                    "DEBUG",
@@ -129,5 +129,5 @@ func GetPBAgentInfo(serverType int32) *v1.PAgentInfo {
 }
 
 func (self *Config) String() string {
-	return fmt.Sprintf("SocketType:%s Address:%s  StatAddress:%s SpanAddress:%s AgentAddress%s WebPorts:%d  LoggerLevel:%s LoggerDir:%s AgentChannelSize:%d SpanStreamParallelismSize:%d AgentReTryTimeout:%s PingInterval:%d StatInterval:%d SpanTimeWaitSec:%d MetaDataTimeWaitSec:%d Pid:%d StartTime:%d ServerType:%d Container:%t", self.SocketType, self.Address, self.StatAddress, self.SpanAddress, self.AgentAddress, self.WebPorts, self.LoggerLevel, self.LoggerDir, self.AgentChannelSize, self.SpanStreamParallelismSize, self.AgentReTryTimeout, self.PingInterval, self.StatInterval, self.SpanTimeWaitSec, self.MetaDataTimeWaitSec, self.Pid, self.StartTime, self.ServerType, self.Container)
+	return fmt.Sprintf("SocketType:%s Address:%s  StatAddress:%s SpanAddress:%s AgentAddress%s WebPorts:%d  LoggerLevel:%s LoggerDir:%s AgentChannelSize:%d SpanStreamParallelismSize:%d AgentReTryTimeout:%s PingInterval:%d StatInterval:%d SpanTimeWaitSec:%d MetaDataTimeWaitSec:%d Pid:%d StartTime:%d ServerType:%d Container:%t", self.SocketType, self.Address, self.StatAddress, self.SpanAddress, self.AgentAddress, self.WebPorts, self.LoggerLevel, self.LoggerDir, self.AgentChannelSize, self.SpanStreamParallelismSize, self.AgentReTryTimeout, self.PingInterval, self.StatInterval, self.SpanTimeWait, self.MetaDataTimeWait, self.Pid, self.StartTime, self.ServerType, self.Container)
 }
diff --git a/collector-agent/makefile b/collector-agent/makefile
index 1fa7749e..e5c20438 100644
--- a/collector-agent/makefile
+++ b/collector-agent/makefile
@@ -15,5 +15,8 @@ clean:
 	rm -rf pinpoint-grpc-idl/proto/v1/*.go
 	rm -rf collector-agent
 
+test: server
+	go test ./... -v	
+
 docker:
 	docker build  -f Dockerfile -t collector-agent:${TAG}  . 
\ No newline at end of file
diff --git a/collector-agent/server.go b/collector-agent/server.go
index e657b890..b33555e4 100644
--- a/collector-agent/server.go
+++ b/collector-agent/server.go
@@ -1,7 +1,9 @@
 package main
 
 import (
-	// _ "net/http/pprof"
+	"net/http"
+	_ "net/http/pprof"
+	"runtime"
 
 	"flag"
 
@@ -11,11 +13,11 @@ import (
 )
 
 var (
-	server_recv_buf=flag.Int("RecvBufSize", server.RECV_BUF_SIZE_DEF, "Set recv buf; eg: -RecvBufSize=409600")
+	server_recv_buf = flag.Int("RecvBufSize", server.RECV_BUF_SIZE_DEF, "Set recv buf; eg: -RecvBufSize=409600")
 )
 
-func initServerSetting(){
-	if *server_recv_buf > server.RECV_BUF_SIZE_DEF{
+func initServerSetting() {
+	if *server_recv_buf > server.RECV_BUF_SIZE_DEF {
 		server.Setting.RecvBufSize = *server_recv_buf
 	}
 }
@@ -24,14 +26,14 @@ func main() {
 	flag.Parse()
 
 	initServerSetting()
-	
+
 	server.InitServerConfig()
 	spanServer := server.SpanServer{}
 	// disable performance profile
-	// go func() {
-	// 	log.Println(http.ListenAndServe("0.0.0.0:8081", nil))
-	// 	runtime.SetBlockProfileRate(1)
-	// }()
+	go func() {
+		log.Println(http.ListenAndServe("0.0.0.0:8081", nil))
+		runtime.SetBlockProfileRate(1)
+	}()
 
 	if _, err := spanServer.Run(); err != nil {
 		log.Warn("SpanServer is exit ....")


Test patch:
diff --git a/collector-agent/agent/SpanSender_test.go b/collector-agent/agent/SpanSender_test.go
index a65ac1c2..fdbe9051 100644
--- a/collector-agent/agent/SpanSender_test.go
+++ b/collector-agent/agent/SpanSender_test.go
@@ -1,13 +1,17 @@
 package agent
 
 import (
+	"sync"
 	"testing"
+
+	log "github.com/sirupsen/logrus"
 )
 
 func TestSqlUidFormat(t *testing.T) {
 	// test  chan
 	exitCh := make(chan bool) // chan bool
-	sender := createSpanSender(nil, exitCh)
+	var wg sync.WaitGroup
+	sender := createSpanSender(nil, exitCh, &wg, log.WithField("test", nil))
 
 	id := sender.getSqlUidMetaApiId("INSERT INTO chengji_m VALUES (%s, %s, %s)")
 	t.Logf("%v", string(id))
diff --git a/collector-agent/server/server_test.go b/collector-agent/server/server_test.go
index 68767a60..1032ebe6 100644
--- a/collector-agent/server/server_test.go
+++ b/collector-agent/server/server_test.go
@@ -31,9 +31,9 @@ func launchJsonServer() *SpanServer {
 	config.AgentReTryTimeout = 10
 	config.PingInterval = 5
 	config.StatInterval = 5
-	config.SpanTimeWaitSec = 10
-	config.MetaDataTimeWaitSec = 10
-	config.GrpcConTextTimeOutSec = 5
+	config.SpanTimeWait = 10
+	config.MetaDataTimeWait = 10
+	config.GrpcConTextTimeOut = 5
 	config.HostName = "dev-pinpoint"
 	config.HostIp = "10.34.135.214"
 	config.Pid = 5689
diff --git a/testapps/SimplePHP/simplephp.dockerfile b/testapps/SimplePHP/simplephp.dockerfile
index bb843ed8..044b4673 100644
--- a/testapps/SimplePHP/simplephp.dockerfile
+++ b/testapps/SimplePHP/simplephp.dockerfile
@@ -10,7 +10,7 @@ COPY testapps/SimplePHP/composer-real.json /workspace/composer.json
 COPY testapps/SimplePHP/run.php /workspace/run.php
 RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
 RUN git clone --recurse-submodules --depth 1 --branch 1.18.1 https://github.com/mongodb/mongo-php-driver.git && cd mongo-php-driver && phpize && ./configure && make -j && make install 
-RUN cd /tmp/ && git clone https://github.com/phpredis/phpredis.git && cd phpredis && phpize && ./configure &&  make install
+RUN cd /tmp/ && git clone https://github.com/phpredis/phpredis.git && cd phpredis && git checkout 6.0.2 && phpize && ./configure &&  make install
 RUN cd /tmp/ && git clone https://github.com/php-memcached-dev/php-memcached.git && cd php-memcached && phpize && ./configure &&  make install
 RUN cd /tmp/ && git clone https://github.com/krakjoe/apcu.git && cd apcu && phpize && ./configure &&  make install
 COPY config.m4 /pinpoint-c-agent/config.m4 
diff --git a/testapps/php_compatible.dockerfile b/testapps/php_compatible.dockerfile
index b949356e..0ac44324 100644
--- a/testapps/php_compatible.dockerfile
+++ b/testapps/php_compatible.dockerfile
@@ -10,7 +10,7 @@ COPY tests /pinpoint-c-agent/tests
 ENV NO_INTERACTION=1
 RUN cd /pinpoint-c-agent/ && phpize && ./configure && make && make install
 ## install redis
-RUN cd /tmp/ && git clone https://github.com/phpredis/phpredis.git && cd phpredis && phpize && ./configure &&  make install
+RUN cd /tmp/ && git clone https://github.com/phpredis/phpredis.git && cd phpredis && git checkout 6.0.2 && phpize && ./configure &&  make install
 RUN docker-php-ext-install mysqli pdo pdo_mysql && docker-php-ext-enable pdo_mysql
 
 COPY testapps/shell.sh shell.sh