Repository: vanus-labs/vanus
Pull request #467 (2023-02-28 01:51:45)

test: Unit test fixing of TestEventlogManager_UpdateSegmentReplicas
<!--

Thank you for contributing to Vanus!

PR Title Format: 

feat/fix/refactor/docs/...: what's changed

Note: see https://github.com/linkall-labs/vanus/blob/main/CONTRIBUTING.md to know details about title format
-->

### What problem does this PR solve?
<!--

Please create an issue first to describe the problem

There MUST be one line starting with "Issue Number:  ".

-->

Issue Number: close #xxx

### Problem Summary
Unit test fixing of TestEventlogManager_UpdateSegmentReplicas
### What is changed and how does it work?

### Check List

<!-- At least one of them must be included. -->

#### Tests

- [x] Unit test
- [ ] Integration test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No code


codecov[bot]: # [Codecov](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs) Report
> Merging [#467](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs) (3e24de2) into [main](https://codecov.io/gh/vanus-labs/vanus/commit/2a78cb5c98e21ede2d084739bfadd4151ac48054?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs) (2a78cb5) will **decrease** coverage by `0.58%`.
> The diff coverage is `41.57%`.

<details><summary>Additional details and impacted files</summary>


[![Impacted file tree graph](https://codecov.io/gh/vanus-labs/vanus/pull/467/graphs/tree.svg?width=650&height=150&src=pr&token=RSXSIMEY4V&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs)](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs)

```diff
@@            Coverage Diff             @@
##             main     #467      +/-   ##
==========================================
- Coverage   57.71%   57.13%   -0.58%     
==========================================
  Files         180      185       +5     
  Lines       14614    15051     +437     
==========================================
+ Hits         8434     8599     +165     
- Misses       5552     5819     +267     
- Partials      628      633       +5     
```


| [Impacted Files](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs) | Coverage Δ | |
|---|---|---|
| [...ernal/controller/trigger/validation/subscripton.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvY29udHJvbGxlci90cmlnZ2VyL3ZhbGlkYXRpb24vc3Vic2NyaXB0b24uZ28=) | `69.29% <ø> (+0.85%)` | :arrow_up: |
| [internal/gateway/proxy/deadletter.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvZ2F0ZXdheS9wcm94eS9kZWFkbGV0dGVyLmdv) | `0.00% <0.00%> (ø)` | |
| [internal/gateway/proxy/direct.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvZ2F0ZXdheS9wcm94eS9kaXJlY3QuZ28=) | `73.33% <0.00%> (-5.24%)` | :arrow_down: |
| [internal/store/segment/server.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvc3RvcmUvc2VnbWVudC9zZXJ2ZXIuZ28=) | `14.10% <0.00%> (-0.03%)` | :arrow_down: |
| [internal/trigger/client/gcloud\_functions.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvdHJpZ2dlci9jbGllbnQvZ2Nsb3VkX2Z1bmN0aW9ucy5nbw==) | `0.00% <0.00%> (ø)` | |
| [internal/trigger/client/grpc.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvdHJpZ2dlci9jbGllbnQvZ3JwYy5nbw==) | `0.00% <0.00%> (ø)` | |
| [internal/trigger/client/http.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvdHJpZ2dlci9jbGllbnQvaHR0cC5nbw==) | `0.00% <0.00%> (ø)` | |
| [internal/trigger/client/interface.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvdHJpZ2dlci9jbGllbnQvaW50ZXJmYWNlLmdv) | `0.00% <0.00%> (-19.24%)` | :arrow_down: |
| [internal/trigger/client/lambda.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvdHJpZ2dlci9jbGllbnQvbGFtYmRhLmdv) | `0.00% <0.00%> (ø)` | |
| [internal/gateway/proxy/proxy.go](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs#diff-aW50ZXJuYWwvZ2F0ZXdheS9wcm94eS9wcm94eS5nbw==) | `23.08% <6.52%> (-2.01%)` | :arrow_down: |
| ... and [31 more](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs). Last update [e75c679...3e24de2](https://codecov.io/gh/vanus-labs/vanus/pull/467?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=vanus-labs).

</details>

Patch:
diff --git a/internal/primitive/transform/action/array/unfold_array.go b/internal/primitive/transform/action/array/unfold_array.go
new file mode 100644
index 0000000..7ff09db
--- /dev/null
+++ b/internal/primitive/transform/action/array/unfold_array.go
@@ -0,0 +1,75 @@
+// Copyright 2023 Linkall Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package array
+
+import (
+	"errors"
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/linkall-labs/vanus/internal/primitive/transform/action"
+	"github.com/linkall-labs/vanus/internal/primitive/transform/arg"
+	"github.com/linkall-labs/vanus/internal/primitive/transform/common"
+	"github.com/linkall-labs/vanus/internal/primitive/transform/context"
+)
+
+// ["unfold_array","sourceJsonPath", "targetPathPrefix"].
+type unfoldArrayAction struct {
+	action.CommonAction
+	targetPathPrefix string
+}
+
+func NewUnfoldArrayAction() action.Action {
+	a := &unfoldArrayAction{}
+	a.CommonAction = action.CommonAction{
+		ActionName: "UNFOLD_ARRAY",
+		FixedArgs:  []arg.TypeList{[]arg.Type{arg.EventData}, []arg.Type{arg.EventData}},
+	}
+	return a
+}
+
+func (a *unfoldArrayAction) Init(args []arg.Arg) error {
+	a.targetPathPrefix = args[1].Original()
+	a.Args = args[:1]
+	a.ArgTypes = []common.Type{common.Array}
+	return nil
+}
+
+func (a *unfoldArrayAction) Execute(ceCtx *context.EventContext) error {
+	args, err := a.RunArgs(ceCtx)
+	if err != nil {
+		return err
+	}
+	arrayValue, _ := args[0].([]interface{})
+	errs := make([]string, 0, len(arrayValue))
+	for i, v := range arrayValue {
+		argument := a.targetPathPrefix + "-" + strconv.Itoa(i)
+		targetArg, err := arg.NewArg(argument)
+		if err != nil {
+			errs = append(errs, fmt.Sprintf("failed to NewArg, arg: %v, err: %v", argument, err))
+			continue
+		}
+		err = targetArg.SetValue(ceCtx, v)
+		if err != nil {
+			errs = append(errs, fmt.Sprintf("failed to SetValue, v: %v, err: %v", v, err))
+			continue
+		}
+	}
+	if len(errs) > 0 {
+		return errors.New(strings.Join(errs, ";"))
+	}
+	return nil
+}
diff --git a/internal/primitive/transform/runtime/init.go b/internal/primitive/transform/runtime/init.go
index 390cd85..d9d775f 100644
--- a/internal/primitive/transform/runtime/init.go
+++ b/internal/primitive/transform/runtime/init.go
@@ -60,6 +60,7 @@ func init() {
 		// array
 		array.NewRenderArrayAction,
 		array.NewArrayForeachAction,
+		array.NewUnfoldArrayAction,
 		// common
 		common.NewLengthAction,
 		// source


Test patch:
diff --git a/internal/controller/eventbus/eventlog/eventlog_test.go b/internal/controller/eventbus/eventlog/eventlog_test.go
index 3d740d0..96d766e 100644
--- a/internal/controller/eventbus/eventlog/eventlog_test.go
+++ b/internal/controller/eventbus/eventlog/eventlog_test.go
@@ -287,8 +287,11 @@ func TestEventlogManager_ScaleSegmentTask(t *testing.T) {
 		So(util.MapLen(&utMgr.globalBlockMap), ShouldEqual, (defaultAppendableSegmentNumber*2+1)*3)
 
 		utMgr.stop()
+		// avoid data race during UT
 		head = el2.head()
+		el2.lock()
 		head.State = StateFrozen
+		el2.unlock()
 		So(err, ShouldBeNil)
 		time.Sleep(50 * time.Millisecond)
 		So(el.size(), ShouldEqual, defaultAppendableSegmentNumber+1)
@@ -783,21 +786,30 @@ func TestEventlogManager_UpdateSegmentReplicas(t *testing.T) {
 		kvCli := kv.NewMockClient(ctrl)
 		utMgr.kvClient = kvCli
 
+		ctx := stdCtx.Background()
+		el, err := newEventlog(ctx, &metadata.Eventlog{
+			ID:           vanus.NewTestID(),
+			EventbusID:   vanus.NewTestID(),
+			EventbusName: "ut",
+		}, kvCli, false)
+		So(err, ShouldBeNil)
+		utMgr.eventLogMap.Store(el.md.ID.Key(), el)
+
 		blk := &metadata.Block{
-			ID:        vanus.NewTestID(),
-			Capacity:  64 * 1024 * 1024,
-			SegmentID: 0,
+			ID:         vanus.NewTestID(),
+			Capacity:   64 * 1024 * 1024,
+			EventlogID: el.md.ID,
 		}
 		seg := createTestSegment(vanus.NewTestID())
+		seg.EventLogID = el.md.ID
 		seg.Replicas.Term = 3
 		utMgr.globalSegmentMap.Store(seg.ID.Key(), seg)
 		utMgr.globalBlockMap.Store(blk.ID.Key(), blk)
 
-		ctx := stdCtx.Background()
 		kvCli.EXPECT().Set(ctx, filepath.Join(metadata.SegmentKeyPrefixInKVStore, seg.ID.String()),
 			gomock.Any()).Times(1).Return(nil)
 
-		err := utMgr.UpdateSegmentReplicas(ctx, vanus.NewTestID(), 3)
+		err = utMgr.UpdateSegmentReplicas(ctx, vanus.NewTestID(), 3)
 		So(err, ShouldEqual, errors.ErrBlockNotFound)
 
 		err = utMgr.UpdateSegmentReplicas(ctx, blk.ID, 3)
diff --git a/internal/primitive/transform/action/array/unfold_array_test.go b/internal/primitive/transform/action/array/unfold_array_test.go
new file mode 100644
index 0000000..5d73963
--- /dev/null
+++ b/internal/primitive/transform/action/array/unfold_array_test.go
@@ -0,0 +1,63 @@
+// Copyright 2023 Linkall Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package array_test
+
+import (
+	"testing"
+
+	cetest "github.com/cloudevents/sdk-go/v2/test"
+	"github.com/linkall-labs/vanus/internal/primitive/transform/action/array"
+	"github.com/linkall-labs/vanus/internal/primitive/transform/context"
+	"github.com/linkall-labs/vanus/internal/primitive/transform/runtime"
+	. "github.com/smartystreets/goconvey/convey"
+)
+
+func TestUnfoldArrayAction(t *testing.T) {
+	funcName := array.NewUnfoldArrayAction().Name()
+	Convey("test unfold array valid", t, func() {
+		a, err := runtime.NewAction([]interface{}{funcName, "$.data.array", "$.data.target"})
+		So(err, ShouldBeNil)
+
+		e := cetest.MinEvent()
+		data := map[string]interface{}{
+			"array": []interface{}{"one", 2, []string{"three"}},
+		}
+		err = a.Execute(&context.EventContext{
+			Event: &e,
+			Data:  data,
+		})
+		So(err, ShouldBeNil)
+		So(data, ShouldHaveLength, 4)
+		So(data["target-0"], ShouldEqual, "one")
+		So(data["target-1"], ShouldEqual, 2)
+		So(data["target-2"], ShouldResemble, []interface{}{"three"})
+	})
+	Convey("test unfold array string input", t, func() {
+		a, err := runtime.NewAction([]interface{}{funcName, "$.data.array", "$.data.target"})
+		So(err, ShouldBeNil)
+
+		e := cetest.MinEvent()
+		data := map[string]interface{}{
+			"array": "one",
+		}
+		err = a.Execute(&context.EventContext{
+			Event: &e,
+			Data:  data,
+		})
+		So(err, ShouldBeNil)
+		So(data, ShouldHaveLength, 2)
+		So(data["target-0"], ShouldEqual, "one")
+	})
+}
diff --git a/internal/primitive/transform/runtime/action_bench_test.go b/internal/primitive/transform/runtime/action_bench_test.go
index 0ba903b..ae8c9f5 100644
--- a/internal/primitive/transform/runtime/action_bench_test.go
+++ b/internal/primitive/transform/runtime/action_bench_test.go
@@ -80,4 +80,5 @@ func BenchmarkAction(b *testing.B) {
 	b.Run("capitalize_sentence", actionBenchmark([]interface{}{"capitalize_sentence", "$.data.str"}))
 	b.Run("check_custom_values", actionBenchmark([]interface{}{"check_custom_values", "$.data.str", "value", "$.data.target", "true", "false"}))
 	b.Run("split_with_delimiter", actionBenchmark([]interface{}{"split_with_delimiter", "$.data.str", "a", "$.data.target"}))
+	b.Run("unfold_array", actionBenchmark([]interface{}{"unfold_array", "$.data.str", "$.data.target"}))
 }