Repository: pingcap/chaos-mesh
Pull request #353 (2020-03-20 23:09:34)

netm: Add Netem as new action for applying delay, loss etc. at same time
UCP #345 

### What problem does this PR solve? <!--add and issue link with summary if exists-->

Fix #345 

### What is changed and how does it work?

Add a new action called `netem` for networkchaos, when provided, the controller will merge all individual specs (delay, loss etc.) into one `Netem` RPC request.

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

 - [ ] Unit test, seem we don't have any ... #263 
 - [ ] E2E test, it's WIP #322 
 - [x] Manual test (add detailed scripts or steps below) blocked by #352 

Code changes

 - Has Go code change

Side effects

 - [ ] Breaking backward compatibility (depends on how we want impl it ...)

Related changes

 - [ ] Need to update the documentation

### Does this PR introduce a user-facing change?:
<!--
If no, just leave the release note block below as is.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->

```release-note
Add netem as new network chaos action. It supports applying delay, loss etc. at same time.
```


codecov-io: # [Codecov](https://codecov.io/gh/pingcap/chaos-mesh/pull/353?src=pr&el=h1) Report
> Merging [#353](https://codecov.io/gh/pingcap/chaos-mesh/pull/353?src=pr&el=desc) into [master](https://codecov.io/gh/pingcap/chaos-mesh/commit/1c6a077335d19b9c412b4f6cda0236283882e0a2&el=desc) will **decrease** coverage by `3.03%`.
> The diff coverage is `5.79%`.

[![Impacted file tree graph](https://codecov.io/gh/pingcap/chaos-mesh/pull/353/graphs/tree.svg?width=650&height=150&src=pr&token=a3XIzuy5Dk)](https://codecov.io/gh/pingcap/chaos-mesh/pull/353?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #353      +/-   ##
==========================================
- Coverage   61.00%   57.97%   -3.04%     
==========================================
  Files          56       57       +1     
  Lines        3347     3529     +182     
==========================================
+ Hits         2042     2046       +4     
- Misses       1145     1319     +174     
- Partials      160      164       +4     
```


| [Impacted Files](https://codecov.io/gh/pingcap/chaos-mesh/pull/353?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [api/v1alpha1/networkchaos\_types.go](https://codecov.io/gh/pingcap/chaos-mesh/pull/353/diff?src=pr&el=tree#diff-YXBpL3YxYWxwaGExL25ldHdvcmtjaGFvc190eXBlcy5nbw==) | `13.08% <0.00%> (ø)` | |
| [controllers/twophase/types.go](https://codecov.io/gh/pingcap/chaos-mesh/pull/353/diff?src=pr&el=tree#diff-Y29udHJvbGxlcnMvdHdvcGhhc2UvdHlwZXMuZ28=) | `63.09% <ø> (ø)` | |
| [pkg/chaosdaemon/util.go](https://codecov.io/gh/pingcap/chaos-mesh/pull/353/diff?src=pr&el=tree#diff-cGtnL2NoYW9zZGFlbW9uL3V0aWwuZ28=) | `80.32% <ø> (ø)` | |
| [pkg/utils/chaosdaemon.go](https://codecov.io/gh/pingcap/chaos-mesh/pull/353/diff?src=pr&el=tree#diff-cGtnL3V0aWxzL2NoYW9zZGFlbW9uLmdv) | `0.00% <0.00%> (ø)` | |
| [controllers/networkchaos/netem/types.go](https://codecov.io/gh/pingcap/chaos-mesh/pull/353/diff?src=pr&el=tree#diff-Y29udHJvbGxlcnMvbmV0d29ya2NoYW9zL25ldGVtL3R5cGVzLmdv) | `2.58% <11.76%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pingcap/chaos-mesh/pull/353?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pingcap/chaos-mesh/pull/353?src=pr&el=footer). Last update [1c6a077...bf2f78a](https://codecov.io/gh/pingcap/chaos-mesh/pull/353?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

at15: I tested manually, deployed nginx, then applied the example (labels are changed). I saw ping would go high because of the injected delay. However, loss is added at same time but I didn't see it happening ...

I didn't touch the webhook, it seems a lot of validation logic need to be added there, now most time invalid spec result in warning from `kubectl get events` like `nextRecover shouldn't be later than nextStart`

```
64 bytes from 10.244.1.4: icmp_seq=55 ttl=64 time=0.060 ms
64 bytes from 10.244.1.4: icmp_seq=56 ttl=64 time=0.185 ms
64 bytes from 10.244.1.4: icmp_seq=57 ttl=64 time=0.045 ms
```

```yaml
apiVersion: pingcap.com/v1alpha1
kind: NetworkChaos
metadata:
  name: network-netem-example
  namespace: chaos-testing
spec:
  action: netem
  mode: one
  selector:
    labelSelectors:
      "app": "nginx"
  delay:
    latency: "90ms"
    correlation: "25"
    jitter: "90ms"
  loss:
    loss: "25"
    correlation: "25"
  duration: "10s"
  scheduler:
    cron: "@every 15s"
```

PTAL @cwen0 
YangKeao: @cwen0 PTAL
cwen0: /merge
sre-bot: /run-all-tests

Patch:
diff --git a/api/v1alpha1/networkchaos_types.go b/api/v1alpha1/networkchaos_types.go
index ea005c1b..043a8bf6 100644
--- a/api/v1alpha1/networkchaos_types.go
+++ b/api/v1alpha1/networkchaos_types.go
@@ -17,9 +17,9 @@ import (
 	"strconv"
 	"time"
 
-	chaosdaemon "github.com/pingcap/chaos-mesh/pkg/chaosdaemon/pb"
-
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	chaosdaemonpb "github.com/pingcap/chaos-mesh/pkg/chaosdaemon/pb"
 )
 
 // EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
@@ -29,6 +29,10 @@ import (
 type NetworkChaosAction string
 
 const (
+	// NetemAction is a combination of several chaos actions i.e. delay, loss, duplicate, corrupt.
+	// When using this action multiple specs are merged into one Netem RPC and sends to chaos daemon.
+	NetemAction NetworkChaosAction = "netem"
+
 	// DelayAction represents the chaos action of adding delay on pods.
 	DelayAction NetworkChaosAction = "delay"
 
@@ -92,7 +96,7 @@ func (t *PartitionTarget) GetValue() string {
 // NetworkChaosSpec defines the desired state of NetworkChaos
 type NetworkChaosSpec struct {
 	// Action defines the specific network chaos action.
-	// Supported action: delay
+	// Supported action: partition, netem, delay, loss, duplicate, corrupt
 	// Default action: delay
 	Action NetworkChaosAction `json:"action"`
 
@@ -252,7 +256,8 @@ type DelaySpec struct {
 	Reorder     *ReorderSpec `json:"reorder,omitempty"`
 }
 
-func (delay *DelaySpec) ToNetem() (*chaosdaemon.Netem, error) {
+// ToNetem implements Netem interface.
+func (delay *DelaySpec) ToNetem() (*chaosdaemonpb.Netem, error) {
 	delayTime, err := time.ParseDuration(delay.Latency)
 	if err != nil {
 		return nil, err
@@ -267,7 +272,7 @@ func (delay *DelaySpec) ToNetem() (*chaosdaemon.Netem, error) {
 		return nil, err
 	}
 
-	netem := &chaosdaemon.Netem{
+	netem := &chaosdaemonpb.Netem{
 		Time:      uint32(delayTime.Nanoseconds() / 1e3),
 		DelayCorr: float32(corr),
 		Jitter:    uint32(jitter.Nanoseconds() / 1e3),
@@ -298,7 +303,8 @@ type LossSpec struct {
 	Correlation string `json:"correlation"`
 }
 
-func (loss *LossSpec) ToNetem() (*chaosdaemon.Netem, error) {
+// ToNetem implements Netem interface.
+func (loss *LossSpec) ToNetem() (*chaosdaemonpb.Netem, error) {
 	lossPercentage, err := strconv.ParseFloat(loss.Loss, 32)
 	if err != nil {
 		return nil, err
@@ -309,7 +315,7 @@ func (loss *LossSpec) ToNetem() (*chaosdaemon.Netem, error) {
 		return nil, err
 	}
 
-	return &chaosdaemon.Netem{
+	return &chaosdaemonpb.Netem{
 		Loss:     float32(lossPercentage),
 		LossCorr: float32(corr),
 	}, nil
@@ -321,7 +327,8 @@ type DuplicateSpec struct {
 	Correlation string `json:"correlation"`
 }
 
-func (duplicate *DuplicateSpec) ToNetem() (*chaosdaemon.Netem, error) {
+// ToNetem implements Netem interface.
+func (duplicate *DuplicateSpec) ToNetem() (*chaosdaemonpb.Netem, error) {
 	duplicatePercentage, err := strconv.ParseFloat(duplicate.Duplicate, 32)
 	if err != nil {
 		return nil, err
@@ -332,7 +339,7 @@ func (duplicate *DuplicateSpec) ToNetem() (*chaosdaemon.Netem, error) {
 		return nil, err
 	}
 
-	return &chaosdaemon.Netem{
+	return &chaosdaemonpb.Netem{
 		Duplicate:     float32(duplicatePercentage),
 		DuplicateCorr: float32(corr),
 	}, nil
@@ -344,7 +351,8 @@ type CorruptSpec struct {
 	Correlation string `json:"correlation"`
 }
 
-func (corrupt *CorruptSpec) ToNetem() (*chaosdaemon.Netem, error) {
+// ToNetem implements Netem interface.
+func (corrupt *CorruptSpec) ToNetem() (*chaosdaemonpb.Netem, error) {
 	corruptPercentage, err := strconv.ParseFloat(corrupt.Corrupt, 32)
 	if err != nil {
 		return nil, err
@@ -355,7 +363,7 @@ func (corrupt *CorruptSpec) ToNetem() (*chaosdaemon.Netem, error) {
 		return nil, err
 	}
 
-	return &chaosdaemon.Netem{
+	return &chaosdaemonpb.Netem{
 		Corrupt:     float32(corruptPercentage),
 		CorruptCorr: float32(corr),
 	}, nil
diff --git a/config/crd/bases/pingcap.com_networkchaos.yaml b/config/crd/bases/pingcap.com_networkchaos.yaml
index 7c076de5..561d9b38 100644
--- a/config/crd/bases/pingcap.com_networkchaos.yaml
+++ b/config/crd/bases/pingcap.com_networkchaos.yaml
@@ -36,7 +36,8 @@ spec:
           properties:
             action:
               description: 'Action defines the specific network chaos action. Supported
-                action: delay Default action: delay'
+                action: partition, netem, delay, loss, duplicate, corrupt Default
+                action: delay'
               type: string
             corrupt:
               description: Corrupt represents the detail about corrupt action
diff --git a/controllers/networkchaos/netem/types.go b/controllers/networkchaos/netem/types.go
index e4e9968e..13703f8f 100644
--- a/controllers/networkchaos/netem/types.go
+++ b/controllers/networkchaos/netem/types.go
@@ -41,6 +41,7 @@ import (
 
 const (
 	networkNetemActionMsg = "network netem action duration %s"
+	invalidNetemSpecMsg   = "invalid spec for netem action, at least one is required from delay, loss, duplicate, corrupt"
 )
 
 // NetemSpec defines the interface to convert to a Netem protobuf
@@ -234,12 +235,23 @@ func (r *Reconciler) applyAllPods(ctx context.Context, pods []v1.Pod, networkcha
 func (r *Reconciler) applyPod(ctx context.Context, pod *v1.Pod, networkchaos *v1alpha1.NetworkChaos) error {
 	r.Log.Info("Try to apply netem on pod", "namespace", pod.Namespace, "name", pod.Name)
 
-	action := string(networkchaos.Spec.Action)
-	action = strings.Title(action)
-
-	spec, ok := reflect.Indirect(reflect.ValueOf(networkchaos.Spec)).FieldByName(action).Interface().(NetemSpec)
-	if !ok {
-		return fmt.Errorf("spec %s is not a NetemSpec", action)
+	var (
+		netem *pb.Netem
+		err   error
+	)
+	switch networkchaos.Spec.Action {
+	case v1alpha1.NetemAction:
+		netem, err = mergeNetem(networkchaos.Spec)
+	default:
+		action := strings.Title(string(networkchaos.Spec.Action))
+		spec, ok := reflect.Indirect(reflect.ValueOf(networkchaos.Spec)).FieldByName(action).Interface().(NetemSpec)
+		if !ok {
+			return fmt.Errorf("spec %s is not a NetemSpec", action)
+		}
+		netem, err = spec.ToNetem()
+	}
+	if err != nil {
+		return err
 	}
 
 	pbClient, err := utils.NewChaosDaemonClient(ctx, r.Client, pod, os.Getenv("CHAOS_DAEMON_PORT"))
@@ -254,11 +266,6 @@ func (r *Reconciler) applyPod(ctx context.Context, pod *v1.Pod, networkchaos *v1
 
 	containerID := pod.Status.ContainerStatuses[0].ContainerID
 
-	netem, err := spec.ToNetem()
-	if err != nil {
-		return err
-	}
-
 	_, err = pbClient.SetNetem(ctx, &pb.NetemRequest{
 		ContainerId: containerID,
 		Netem:       netem,
@@ -266,3 +273,40 @@ func (r *Reconciler) applyPod(ctx context.Context, pod *v1.Pod, networkchaos *v1
 
 	return err
 }
+
+// mergeNetem calls ToNetem on all non nil network emulation specs and merges them into one request.
+func mergeNetem(spec v1alpha1.NetworkChaosSpec) (*pb.Netem, error) {
+	// NOTE: a cleaner way like
+	// emSpecs = []NetemSpec{spec.Delay, spec.Loss} won't work.
+	// Because in the for _, spec := range emSpecs loop,
+	// spec != nil would always be true.
+	// See https://stackoverflow.com/questions/13476349/check-for-nil-and-nil-interface-in-go
+	// And https://groups.google.com/forum/#!topic/golang-nuts/wnH302gBa4I/discussion
+	// > In short: If you never store (*T)(nil) in an interface, then you can reliably use comparison against nil
+	var emSpecs []NetemSpec
+	if spec.Delay != nil {
+		emSpecs = append(emSpecs, spec.Delay)
+	}
+	if spec.Loss != nil {
+		emSpecs = append(emSpecs, spec.Loss)
+	}
+	if spec.Duplicate != nil {
+		emSpecs = append(emSpecs, spec.Duplicate)
+	}
+	if spec.Corrupt != nil {
+		emSpecs = append(emSpecs, spec.Corrupt)
+	}
+	if len(emSpecs) == 0 {
+		return nil, errors.New(invalidNetemSpecMsg)
+	}
+
+	merged := &pb.Netem{}
+	for _, spec := range emSpecs {
+		em, err := spec.ToNetem()
+		if err != nil {
+			return nil, err
+		}
+		merged = utils.MergeNetem(merged, em)
+	}
+	return merged, nil
+}
diff --git a/controllers/networkchaos/types.go b/controllers/networkchaos/types.go
index 55807115..9aceacf5 100644
--- a/controllers/networkchaos/types.go
+++ b/controllers/networkchaos/types.go
@@ -60,7 +60,7 @@ func (r *Reconciler) Reconcile(req ctrl.Request, chaos *v1alpha1.NetworkChaos) (
 func (r *Reconciler) commonNetworkChaos(networkchaos *v1alpha1.NetworkChaos, req ctrl.Request) (ctrl.Result, error) {
 	var cr *common.Reconciler
 	switch networkchaos.Spec.Action {
-	case v1alpha1.DelayAction, v1alpha1.DuplicateAction, v1alpha1.CorruptAction, v1alpha1.LossAction:
+	case v1alpha1.NetemAction, v1alpha1.DelayAction, v1alpha1.DuplicateAction, v1alpha1.CorruptAction, v1alpha1.LossAction:
 		cr = netem.NewCommonReconciler(r.Client, r.Log.WithValues("action", "netem"),
 			req, r.EventRecorder)
 	case v1alpha1.PartitionAction:
@@ -75,7 +75,7 @@ func (r *Reconciler) commonNetworkChaos(networkchaos *v1alpha1.NetworkChaos, req
 func (r *Reconciler) scheduleNetworkChaos(networkchaos *v1alpha1.NetworkChaos, req ctrl.Request) (ctrl.Result, error) {
 	var sr *twophase.Reconciler
 	switch networkchaos.Spec.Action {
-	case v1alpha1.DelayAction, v1alpha1.DuplicateAction, v1alpha1.CorruptAction, v1alpha1.LossAction:
+	case v1alpha1.NetemAction, v1alpha1.DelayAction, v1alpha1.DuplicateAction, v1alpha1.CorruptAction, v1alpha1.LossAction:
 		sr = netem.NewTwoPhaseReconciler(r.Client, r.Log.WithValues("action", "netem"),
 			req, r.EventRecorder)
 	case v1alpha1.PartitionAction:
diff --git a/controllers/twophase/types.go b/controllers/twophase/types.go
index 2bc696af..fb959636 100644
--- a/controllers/twophase/types.go
+++ b/controllers/twophase/types.go
@@ -128,6 +128,12 @@ func (r *Reconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
 		}
 
 		nextRecover := now.Add(*duration)
+		// TODO(at15): this should be validated in webhook instead of throwing error at runtime.
+		// User can give a cron with interval shorter than duration.
+		// Example:
+		//  duration: "10s"
+		//  scheduler:
+		//    cron: "@every 5s
 		if nextStart.Before(nextRecover) {
 			err := fmt.Errorf("nextRecover shouldn't be later than nextStart")
 			r.Log.Error(err, "nextRecover is later than nextStart. Then recover can never be reached",
diff --git a/examples/network-netem-example.yaml b/examples/network-netem-example.yaml
new file mode 100644
index 00000000..648a79c3
--- /dev/null
+++ b/examples/network-netem-example.yaml
@@ -0,0 +1,21 @@
+apiVersion: pingcap.com/v1alpha1
+kind: NetworkChaos
+metadata:
+  name: network-netem-example
+  namespace: chaos-testing
+spec:
+  action: netem
+  mode: one
+  selector:
+    labelSelectors:
+      "app.kubernetes.io/component": "tikv"
+  delay:
+    latency: "90ms"
+    correlation: "25"
+    jitter: "90ms"
+  loss:
+    loss: "25"
+    correlation: "25"
+  duration: "10s"
+  scheduler:
+    cron: "@every 15s"
\ No newline at end of file
diff --git a/manifests/crd.yaml b/manifests/crd.yaml
index cde9c2b5..fa64de93 100644
--- a/manifests/crd.yaml
+++ b/manifests/crd.yaml
@@ -548,7 +548,8 @@ spec:
           properties:
             action:
               description: 'Action defines the specific network chaos action. Supported
-                action: delay Default action: delay'
+                action: partition, netem, delay, loss, duplicate, corrupt Default
+                action: delay'
               type: string
             corrupt:
               description: Corrupt represents the detail about corrupt action
diff --git a/pkg/chaosdaemon/util.go b/pkg/chaosdaemon/util.go
index 59b3bb98..70f093e7 100755
--- a/pkg/chaosdaemon/util.go
+++ b/pkg/chaosdaemon/util.go
@@ -24,13 +24,12 @@ import (
 	"sync"
 	"syscall"
 
-	"github.com/pingcap/chaos-mesh/pkg/utils"
-
 	"github.com/containerd/containerd"
 	"github.com/docker/docker/api/types"
 	dockerclient "github.com/docker/docker/client"
 
 	"github.com/pingcap/chaos-mesh/pkg/mock"
+	"github.com/pingcap/chaos-mesh/pkg/utils"
 )
 
 const (
diff --git a/pkg/utils/chaosdaemon.go b/pkg/utils/chaosdaemon.go
index f10be33d..91f9629e 100644
--- a/pkg/utils/chaosdaemon.go
+++ b/pkg/utils/chaosdaemon.go
@@ -2,6 +2,7 @@ package utils
 
 import (
 	"context"
+	"math"
 
 	"google.golang.org/grpc"
 	v1 "k8s.io/api/core/v1"
@@ -45,3 +46,49 @@ func NewChaosDaemonClient(ctx context.Context, c client.Client, pod *v1.Pod, por
 		conn:              cc,
 	}, nil
 }
+
+// MergeNetem merges two Netem protos into a new one.
+// REMEMBER to assign the return value, i.e. merged = utils.MergeNetm(merged, em)
+// For each field it takes the bigger value of the two.
+// Its main use case is merging netem of different types, e.g. delay and loss.
+// It returns nil if both inputs are nil.
+// Otherwise it returns a new Netem with merged values.
+func MergeNetem(a, b *chaosdaemon.Netem) *chaosdaemon.Netem {
+	if a == nil && b == nil {
+		return nil
+	}
+	// NOTE: because proto getters check nil, we are good here even if one of them is nil.
+	// But we just assign empty value to make IDE and linters happy.
+	if a == nil {
+		a = &chaosdaemon.Netem{}
+	}
+	if b == nil {
+		b = &chaosdaemon.Netem{}
+	}
+	return &chaosdaemon.Netem{
+		Time:          maxu32(a.GetTime(), b.GetTime()),
+		Jitter:        maxu32(a.GetJitter(), b.GetJitter()),
+		DelayCorr:     maxf32(a.GetDelayCorr(), b.GetDelayCorr()),
+		Limit:         maxu32(a.GetLimit(), b.GetLimit()),
+		Loss:          maxf32(a.GetLoss(), b.GetLoss()),
+		LossCorr:      maxf32(a.GetLossCorr(), b.GetLossCorr()),
+		Gap:           maxu32(a.GetGap(), b.GetGap()),
+		Duplicate:     maxf32(a.GetDuplicate(), b.GetDuplicate()),
+		DuplicateCorr: maxf32(a.GetDuplicateCorr(), b.GetDuplicateCorr()),
+		Reorder:       maxf32(a.GetReorder(), b.GetReorder()),
+		ReorderCorr:   maxf32(a.GetReorderCorr(), b.GetReorderCorr()),
+		Corrupt:       maxf32(a.GetCorrupt(), b.GetCorrupt()),
+		CorruptCorr:   maxf32(a.GetCorruptCorr(), b.GetCorruptCorr()),
+	}
+}
+
+func maxu32(a, b uint32) uint32 {
+	if a > b {
+		return a
+	}
+	return b
+}
+
+func maxf32(a, b float32) float32 {
+	return float32(math.Max(float64(a), float64(b)))
+}


Test patch:
diff --git a/controllers/networkchaos/netem/types_test.go b/controllers/networkchaos/netem/types_test.go
new file mode 100644
index 00000000..542441c2
--- /dev/null
+++ b/controllers/networkchaos/netem/types_test.go
@@ -0,0 +1,54 @@
+package netem
+
+import (
+	"testing"
+
+	. "github.com/onsi/gomega"
+
+	"github.com/pingcap/chaos-mesh/api/v1alpha1"
+
+	chaosdaemonpb "github.com/pingcap/chaos-mesh/pkg/chaosdaemon/pb"
+)
+
+func TestMergenetem(t *testing.T) {
+
+	t.Run("empty", func(t *testing.T) {
+		spec := v1alpha1.NetworkChaosSpec{
+			Action: "netem",
+		}
+		_, err := mergeNetem(spec)
+		if err == nil {
+			t.Errorf("expect invalid spec failed with message %s but got nil", invalidNetemSpecMsg)
+		}
+		if err != nil && err.Error() != invalidNetemSpecMsg {
+			t.Errorf("expect merge failed with message %s but got %v", invalidNetemSpecMsg, err)
+		}
+	})
+
+	t.Run("delay loss", func(t *testing.T) {
+		g := NewGomegaWithT(t)
+
+		spec := v1alpha1.NetworkChaosSpec{
+			Action: "netem",
+			Delay: &v1alpha1.DelaySpec{
+				Latency:     "90ms",
+				Correlation: "25",
+				Jitter:      "90ms",
+			},
+			Loss: &v1alpha1.LossSpec{
+				Loss:        "25",
+				Correlation: "25",
+			},
+		}
+		m, err := mergeNetem(spec)
+		g.Expect(err).ShouldNot(HaveOccurred())
+		em := &chaosdaemonpb.Netem{
+			Time:      90000,
+			Jitter:    90000,
+			DelayCorr: 25,
+			Loss:      25,
+			LossCorr:  25,
+		}
+		g.Expect(m).Should(Equal(em))
+	})
+}
diff --git a/controllers/networkchaos_controller_test.go b/controllers/networkchaos_controller_test.go
index 1a59df7a..c42da85e 100644
--- a/controllers/networkchaos_controller_test.go
+++ b/controllers/networkchaos_controller_test.go
@@ -28,6 +28,7 @@ import (
 
 var _ = Describe("NetworkChaos Controller", func() {
 	supportingModes := []v1alpha1.NetworkChaosAction{
+		v1alpha1.NetemAction,
 		v1alpha1.DelayAction,
 		v1alpha1.LossAction,
 		v1alpha1.DuplicateAction,
diff --git a/pkg/utils/chaosdaemon_test.go b/pkg/utils/chaosdaemon_test.go
new file mode 100644
index 00000000..1bb048c3
--- /dev/null
+++ b/pkg/utils/chaosdaemon_test.go
@@ -0,0 +1,38 @@
+package utils
+
+import (
+	"testing"
+
+	. "github.com/onsi/gomega"
+
+	chaosdaemonpb "github.com/pingcap/chaos-mesh/pkg/chaosdaemon/pb"
+)
+
+func TestMergeNetem(t *testing.T) {
+	g := NewGomegaWithT(t)
+
+	cases := []struct {
+		a      *chaosdaemonpb.Netem
+		b      *chaosdaemonpb.Netem
+		merged *chaosdaemonpb.Netem
+	}{
+		{nil, nil, nil}, // nil, nil -> nil
+		{
+			// no conflict
+			&chaosdaemonpb.Netem{Loss: 25},
+			&chaosdaemonpb.Netem{DelayCorr: 90},
+			&chaosdaemonpb.Netem{Loss: 25, DelayCorr: 90},
+		},
+		{
+			// pick the max
+			&chaosdaemonpb.Netem{Loss: 25, DelayCorr: 100.2},
+			&chaosdaemonpb.Netem{DelayCorr: 90},
+			&chaosdaemonpb.Netem{Loss: 25, DelayCorr: 100.2},
+		},
+	}
+
+	for _, tc := range cases {
+		m := MergeNetem(tc.a, tc.b)
+		g.Expect(tc.merged).Should(Equal(m))
+	}
+}