Repository: dragonflyoss/Dragonfly2
Pull request #980 (2022-01-12 02:50:37)

test: dump goroutine in e2e
Signed-off-by: Jim Ma <majinjing3@gmail.com>

<!--- Provide a general summary of your changes in the Title above -->

## Description

<!--- Describe your changes in detail -->

## Related Issue

<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- Please link to the issue here: -->

## Motivation and Context

<!--- Why is this change required? What problem does it solve? -->

## Screenshots (if appropriate):

## Types of changes

<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Documentation Update (if none of the other choices apply)

## Checklist:

<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->

- [ ] Code compiles correctly.
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.


codecov-commenter: # [Codecov](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) Report
> Merging [#980](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) (3b80176) into [main](https://codecov.io/gh/dragonflyoss/Dragonfly2/commit/4c94a268d3e6c5a5f08e17021d4485a183b4495b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) (4c94a26) will **increase** coverage by `0.05%`.
> The diff coverage is `12.50%`.

> :exclamation: Current head 3b80176 differs from pull request most recent head 3ccb6cb. Consider uploading reports for the commit 3ccb6cb to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980/graphs/tree.svg?width=650&height=150&src=pr&token=PUICJYSAT7&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)

```diff
@@            Coverage Diff             @@
##             main     #980      +/-   ##
==========================================
+ Coverage   43.66%   43.71%   +0.05%     
==========================================
  Files         115      115              
  Lines        8542     8548       +6     
==========================================
+ Hits         3730     3737       +7     
- Misses       4349     4350       +1     
+ Partials      463      461       -2     
```

| Flag | Coverage Δ | |
|---|---|---|
| Object-compatibility-e2etests | `?` | |
| e2etests | `∅ <ø> (∅)` | |
| unittests | `43.71% <12.50%> (+0.05%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) | Coverage Δ | |
|---|---|---|
| [client/daemon/proxy/proxy.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wcm94eS9wcm94eS5nbw==) | `9.80% <0.00%> (-0.13%)` | :arrow_down: |
| [client/daemon/peer/peertask\_manager.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BlZXJ0YXNrX21hbmFnZXIuZ28=) | `34.84% <33.33%> (-1.09%)` | :arrow_down: |
| [client/daemon/peer/peertask\_conductor.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BlZXJ0YXNrX2NvbmR1Y3Rvci5nbw==) | `53.41% <0.00%> (+0.33%)` | :arrow_up: |
| [pkg/sync/krwmutex.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-cGtnL3N5bmMva3J3bXV0ZXguZ28=) | `90.90% <0.00%> (+9.09%)` | :arrow_up: |
| [pkg/sync/kmutex.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/980/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-cGtnL3N5bmMva211dGV4Lmdv) | `100.00% <0.00%> (+16.66%)` | :arrow_up: |


Patch:
diff --git a/client/daemon/peer/peertask_conductor.go b/client/daemon/peer/peertask_conductor.go
index 26569368..5a66d6de 100644
--- a/client/daemon/peer/peertask_conductor.go
+++ b/client/daemon/peer/peertask_conductor.go
@@ -702,7 +702,7 @@ loop:
 		}
 
 		// update content length
-		if piecePacket.ContentLength > 0 {
+		if piecePacket.ContentLength > -1 {
 			pt.SetContentLength(piecePacket.ContentLength)
 			_ = pt.UpdateStorage(false)
 			pt.Debugf("update content length: %d", pt.GetContentLength())
@@ -731,7 +731,7 @@ loop:
 
 func (pt *peerTaskConductor) init(piecePacket *base.PiecePacket, pieceBufferSize uint32) (chan *DownloadPieceRequest, bool) {
 	pt.contentLength.Store(piecePacket.ContentLength)
-	if piecePacket.ContentLength > 0 {
+	if piecePacket.ContentLength > -1 {
 		pt.span.SetAttributes(config.AttributeTaskContentLength.Int64(piecePacket.ContentLength))
 	}
 	if err := pt.InitStorage(); err != nil {
diff --git a/client/daemon/peer/peertask_manager.go b/client/daemon/peer/peertask_manager.go
index 934ab0e1..f70647aa 100644
--- a/client/daemon/peer/peertask_manager.go
+++ b/client/daemon/peer/peertask_manager.go
@@ -169,6 +169,8 @@ func (ptm *peerTaskManager) getOrCreatePeerTaskConductor(ctx context.Context, ta
 	// double check
 	if p, ok := ptm.findPeerTaskConductor(taskID); ok {
 		ptm.conductorLock.Unlock()
+		logger.Debugf("same peer task found: %s/%s, cancel created peer task %s/%s",
+			p.taskID, p.peerID, ptc.taskID, ptc.peerID)
 		// cancel duplicate peer task
 		ptc.cancel(base.Code_ClientContextCanceled, reasonContextCanceled)
 		return p, nil
@@ -231,7 +233,7 @@ func (ptm *peerTaskManager) Stop(ctx context.Context) error {
 }
 
 func (ptm *peerTaskManager) PeerTaskDone(taskID string) {
-	logger.Debugf("delete task %s in running tasks", taskID)
+	logger.Debugf("delete done task %s in running tasks", taskID)
 	ptm.runningPeerTasks.Delete(taskID)
 }
 
diff --git a/client/daemon/proxy/proxy.go b/client/daemon/proxy/proxy.go
index 7a5c6de9..07518afb 100644
--- a/client/daemon/proxy/proxy.go
+++ b/client/daemon/proxy/proxy.go
@@ -343,7 +343,12 @@ func (proxy *Proxy) handleHTTP(span trace.Span, w http.ResponseWriter, req *http
 	w.WriteHeader(resp.StatusCode)
 	span.SetAttributes(semconv.HTTPStatusCodeKey.Int(resp.StatusCode))
 	if n, err := io.Copy(w, resp.Body); err != nil && err != io.EOF {
-		logger.Errorf("failed to write http body: %v", err)
+		if peerID := resp.Header.Get(config.HeaderDragonflyPeer); peerID != "" {
+			logger.Errorf("failed to write http body: %v, peer: %s, task: %s",
+				err, peerID, resp.Header.Get(config.HeaderDragonflyTask))
+		} else {
+			logger.Errorf("failed to write http body: %v", err)
+		}
 		span.RecordError(err)
 	} else {
 		span.SetAttributes(semconv.HTTPResponseContentLengthKey.Int64(n))


Test patch:
diff --git a/test/e2e/constants.go b/test/e2e/constants.go
index 8366880a..7c8710c1 100644
--- a/test/e2e/constants.go
+++ b/test/e2e/constants.go
@@ -40,6 +40,7 @@ type server struct {
 	namespace  string
 	logDirName string
 	replicas   int
+	pprofPort  int
 }
 
 var servers = map[string]server{
@@ -66,11 +67,13 @@ var servers = map[string]server{
 		namespace:  dragonflyNamespace,
 		logDirName: "daemon",
 		replicas:   1,
+		pprofPort:  9999,
 	},
 	proxyServerName: {
 		name:       proxyServerName,
 		namespace:  dragonflyE2ENamespace,
 		logDirName: "daemon",
 		replicas:   3,
+		pprofPort:  9999,
 	},
 }
diff --git a/test/e2e/dfget_test.go b/test/e2e/dfget_test.go
index 0ff57ca4..6ff78836 100644
--- a/test/e2e/dfget_test.go
+++ b/test/e2e/dfget_test.go
@@ -59,6 +59,7 @@ func singleDfgetTest(name, ns, label, podNamePrefix, container string) {
 		}
 
 		for url, path := range urls {
+			fmt.Printf("--------------------------------------------------------------------------------\n\n")
 			fmt.Println("download url: " + url)
 			// get original file digest
 			out, err = e2eutil.DockerCommand("sha256sum", path).CombinedOutput()
diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go
index 5b826973..9bc23222 100644
--- a/test/e2e/e2e_test.go
+++ b/test/e2e/e2e_test.go
@@ -59,7 +59,7 @@ var _ = AfterSuite(func() {
 			out, err = pod.Command("sh", "-c", fmt.Sprintf(`
               set -x
               cp -r /var/log/dragonfly/%s /tmp/artifact/%s-%d
-							find /tmp/artifact -type d -exec chmod 777 {} \;
+              find /tmp/artifact -type d -exec chmod 777 {} \;
               `, server.logDirName, server.name, i)).CombinedOutput()
 			if err != nil {
 				fmt.Printf("copy log output: %s, error: %s\n", string(out), err)
@@ -75,6 +75,13 @@ var _ = AfterSuite(func() {
 				fmt.Printf("upload pod %s artifact prev stdout file error: %v\n", podName, err)
 			}
 
+			if server.pprofPort > 0 {
+				if out, err := e2eutil.UploadArtifactPProf(server.namespace, podName,
+					fmt.Sprintf("%s-%d", server.name, i), server.name, server.pprofPort); err != nil {
+					fmt.Printf("upload pod %s artifact pprof error: %v, output: %s\n", podName, err, out)
+				}
+			}
+
 		}
 	}
 })
diff --git a/test/e2e/e2eutil/artifact.go b/test/e2e/e2eutil/artifact.go
index 0c4d17c7..70205f94 100644
--- a/test/e2e/e2eutil/artifact.go
+++ b/test/e2e/e2eutil/artifact.go
@@ -66,3 +66,17 @@ func UploadArtifactPrevStdout(namespace, podName, logDirName, logPrefix string)
 
 	return nil
 }
+
+func UploadArtifactPProf(namespace, podName, logDirName, logPrefix string, pprofPort int) (string, error) {
+	logDirname := fmt.Sprintf("/tmp/artifact/%s/", logDirName)
+	out, err := KubeCtlCommand("-n", namespace, "exec", podName, "--", "sh", "-c", fmt.Sprintf(`
+      set -x
+      port=%d
+      dir=%s
+      prefix=%s
+      ip=$(hostname -i)
+      wget $ip:$port/debug/pprof/"goroutine?debug=1" -O $dir/$prefix-pprof-goroutine-1.log
+      wget $ip:$port/debug/pprof/"goroutine?debug=2" -O $dir/$prefix-pprof-goroutine-2.log
+      `, pprofPort, logDirname, logPrefix)).CombinedOutput()
+	return string(out), err
+}
diff --git a/test/testdata/charts/config.yaml b/test/testdata/charts/config.yaml
index e5034ce6..d2521155 100644
--- a/test/testdata/charts/config.yaml
+++ b/test/testdata/charts/config.yaml
@@ -33,6 +33,8 @@ dfdaemon:
     hostPath:
       path: /tmp/artifact
   config:
+    verbose: true
+    pprofPort: 9999
     scheduler:
       disableAutoBackSource: true
     proxy:
diff --git a/test/testdata/k8s/proxy.yaml b/test/testdata/k8s/proxy.yaml
index 7cd404fa..77e7dc63 100644
--- a/test/testdata/k8s/proxy.yaml
+++ b/test/testdata/k8s/proxy.yaml
@@ -7,9 +7,9 @@ data:
   dfget.yaml: |-
     aliveTime: 0s
     gcInterval: 1m0s
-    keepStorage: false
+    keepStorage: true
     verbose: true
-    pprof-port: 0
+    pprof-port: 9999
     scheduler:
       manager:
         enable: true