Repository: milvus-io/milvus-sdk-go
Pull request #621 (2023-11-08 18:11:30)

Support row based Array & fix varchar array type
See also #609 
/kind bug

sre-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/milvus-io/milvus-sdk-go/pull/621#" title="Author self-approved">congqixia</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=milvus-io%2Fmilvus-sdk-go).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/milvus-io/milvus-sdk-go/blob/master/OWNERS)~~ [congqixia]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) Report
> Merging [#621](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) (763dfac) into [master](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/commit/40a4e1eea35250b87aa87607acd53dd997a1704d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) (40a4e1e) will **decrease** coverage by `0.56%`.
> The diff coverage is `2.56%`.

[![Impacted file tree graph](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621/graphs/tree.svg?width=650&height=150&src=pr&token=WJK43PV390&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io)](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io)

```diff
@@            Coverage Diff             @@
##           master     #621      +/-   ##
==========================================
- Coverage   75.15%   74.60%   -0.56%     
==========================================
  Files          41       42       +1     
  Lines        4033     4063      +30     
==========================================
  Hits         3031     3031              
- Misses        890      920      +30     
  Partials      112      112              
```


| [Files](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) | Coverage Δ | |
|---|---|---|
| [entity/columns\_array\_gen.go](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io#diff-ZW50aXR5L2NvbHVtbnNfYXJyYXlfZ2VuLmdv) | `0.00% <ø> (ø)` | |
| [entity/schema.go](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io#diff-ZW50aXR5L3NjaGVtYS5nbw==) | `91.25% <100.00%> (ø)` | |
| [entity/columns.go](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io#diff-ZW50aXR5L2NvbHVtbnMuZ28=) | `50.38% <0.00%> (-1.20%)` | :arrow_down: |
| [entity/rows.go](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io#diff-ZW50aXR5L3Jvd3MuZ28=) | `77.11% <0.00%> (-7.12%)` | :arrow_down: |
| [entity/columns\_array.go](https://app.codecov.io/gh/milvus-io/milvus-sdk-go/pull/621?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io#diff-ZW50aXR5L2NvbHVtbnNfYXJyYXkuZ28=) | `0.00% <0.00%> (ø)` | |

sre-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/milvus-io/milvus-sdk-go/pull/621#" title="Author self-approved">congqixia</a>*, *<a href="https://github.com/milvus-io/milvus-sdk-go/pull/621#pullrequestreview-1721007866" title="LGTM">yah01</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=milvus-io%2Fmilvus-sdk-go).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/milvus-io/milvus-sdk-go/blob/master/OWNERS)~~ [congqixia,yah01]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Patch:
diff --git a/entity/columns.go b/entity/columns.go
index 59192e6..af35632 100644
--- a/entity/columns.go
+++ b/entity/columns.go
@@ -244,7 +244,14 @@ func FieldDataColumn(fd *schema.FieldData, begin, end int) (Column, error) {
 		if data == nil {
 			return nil, errFieldDataTypeNotMatch
 		}
-		return parseArrayData(fd.GetFieldName(), data)
+		var arrayData []*schema.ScalarField
+		if end < 0 {
+			arrayData = data.GetData()[begin:]
+		} else {
+			arrayData = data.GetData()[begin:end]
+		}
+
+		return parseArrayData(fd.GetFieldName(), data.GetElementType(), arrayData)
 
 	case schema.DataType_JSON:
 		data, ok := fd.GetScalars().GetData().(*schema.ScalarField_JsonData)
@@ -304,9 +311,7 @@ func FieldDataColumn(fd *schema.FieldData, begin, end int) (Column, error) {
 	}
 }
 
-func parseArrayData(fieldName string, array *schema.ArrayArray) (Column, error) {
-	fieldDataList := array.Data
-	elementType := array.ElementType
+func parseArrayData(fieldName string, elementType schema.DataType, fieldDataList []*schema.ScalarField) (Column, error) {
 
 	switch elementType {
 	case schema.DataType_Bool:
@@ -368,11 +373,17 @@ func parseArrayData(fieldName string, array *schema.ArrayArray) (Column, error)
 		}
 		return NewColumnDoubleArray(fieldName, data), nil
 
-	case schema.DataType_VarChar:
+	case schema.DataType_VarChar, schema.DataType_String:
 		var data [][][]byte
 		for _, fd := range fieldDataList {
-			data = append(data, fd.GetBytesData().GetData())
+			strs := fd.GetStringData().GetData()
+			bytesData := make([][]byte, 0, len(strs))
+			for _, str := range strs {
+				bytesData = append(bytesData, []byte(str))
+			}
+			data = append(data, bytesData)
 		}
+
 		return NewColumnVarCharArray(fieldName, data), nil
 
 	default:
diff --git a/entity/columns_array.go b/entity/columns_array.go
new file mode 100644
index 0000000..15a2c4f
--- /dev/null
+++ b/entity/columns_array.go
@@ -0,0 +1,107 @@
+package entity
+
+import (
+	"fmt"
+
+	"github.com/cockroachdb/errors"
+	"github.com/milvus-io/milvus-proto/go-api/v2/schemapb"
+)
+
+// ColumnVarCharArray generated columns type for VarChar
+type ColumnVarCharArray struct {
+	ColumnBase
+	name   string
+	values [][][]byte
+}
+
+// Name returns column name
+func (c *ColumnVarCharArray) Name() string {
+	return c.name
+}
+
+// Type returns column FieldType
+func (c *ColumnVarCharArray) Type() FieldType {
+	return FieldTypeArray
+}
+
+// Len returns column values length
+func (c *ColumnVarCharArray) Len() int {
+	return len(c.values)
+}
+
+// Get returns value at index as interface{}.
+func (c *ColumnVarCharArray) Get(idx int) (interface{}, error) {
+	var r []string // use default value
+	if idx < 0 || idx >= c.Len() {
+		return r, errors.New("index out of range")
+	}
+	return c.values[idx], nil
+}
+
+// FieldData return column data mapped to schemapb.FieldData
+func (c *ColumnVarCharArray) FieldData() *schemapb.FieldData {
+	fd := &schemapb.FieldData{
+		Type:      schemapb.DataType_Array,
+		FieldName: c.name,
+	}
+
+	data := make([]*schemapb.ScalarField, 0, c.Len())
+	for _, arr := range c.values {
+		converted := make([]string, 0, c.Len())
+		for i := 0; i < len(arr); i++ {
+			converted = append(converted, string(arr[i]))
+		}
+		data = append(data, &schemapb.ScalarField{
+			Data: &schemapb.ScalarField_StringData{
+				StringData: &schemapb.StringArray{
+					Data: converted,
+				},
+			},
+		})
+	}
+	fd.Field = &schemapb.FieldData_Scalars{
+		Scalars: &schemapb.ScalarField{
+			Data: &schemapb.ScalarField_ArrayData{
+				ArrayData: &schemapb.ArrayArray{
+					Data:        data,
+					ElementType: schemapb.DataType_VarChar,
+				},
+			},
+		},
+	}
+	return fd
+}
+
+// ValueByIdx returns value of the provided index
+// error occurs when index out of range
+func (c *ColumnVarCharArray) ValueByIdx(idx int) ([][]byte, error) {
+	var r [][]byte // use default value
+	if idx < 0 || idx >= c.Len() {
+		return r, errors.New("index out of range")
+	}
+	return c.values[idx], nil
+}
+
+// AppendValue append value into column
+func (c *ColumnVarCharArray) AppendValue(i interface{}) error {
+	v, ok := i.([][]byte)
+	if !ok {
+		return fmt.Errorf("invalid type, expected []string, got %T", i)
+	}
+	c.values = append(c.values, v)
+
+	return nil
+}
+
+// Data returns column data
+func (c *ColumnVarCharArray) Data() [][][]byte {
+	return c.values
+}
+
+// NewColumnVarChar auto generated constructor
+func NewColumnVarCharArray(name string, values [][][]byte) *ColumnVarCharArray {
+	return &ColumnVarCharArray{
+		name:   name,
+		values: values,
+	}
+}
diff --git a/entity/columns_array_gen.go b/entity/columns_array_gen.go
index 60eb25e..3951b8c 100755
--- a/entity/columns_array_gen.go
+++ b/entity/columns_array_gen.go
@@ -704,102 +704,4 @@ func NewColumnDoubleArray(name string, values [][]float64) *ColumnDoubleArray {
 	}
 }
 
-// ColumnVarCharArray generated columns type for VarChar
-type ColumnVarCharArray struct {
-	ColumnBase
-	name   string
-	values [][][]byte
-}
-
-// Name returns column name
-func (c *ColumnVarCharArray) Name() string {
-	return c.name
-}
-
-// Type returns column FieldType
-func (c *ColumnVarCharArray) Type() FieldType {
-	return FieldTypeArray
-}
-
-// Len returns column values length
-func (c *ColumnVarCharArray) Len() int {
-	return len(c.values)
-}
-
-// Get returns value at index as interface{}.
-func (c *ColumnVarCharArray) Get(idx int) (interface{}, error) {
-	var r [][]byte // use default value
-	if idx < 0 || idx >= c.Len() {
-		return r, errors.New("index out of range")
-	}
-	return c.values[idx], nil
-}
-
-// FieldData return column data mapped to schemapb.FieldData
-func (c *ColumnVarCharArray) FieldData() *schemapb.FieldData {
-	fd := &schemapb.FieldData{
-		Type:      schemapb.DataType_Array,
-		FieldName: c.name,
-	}
-
-	data := make([]*schemapb.ScalarField, 0, c.Len())
-	for _, arr := range c.values {
-		converted := make([][]byte, 0, c.Len())
-		for i := 0; i < len(arr); i++ {
-			converted = append(converted, []byte(arr[i]))
-		}
-		data = append(data, &schemapb.ScalarField{
-			Data: &schemapb.ScalarField_BytesData{
-				BytesData: &schemapb.BytesArray{
-					Data: converted,
-				},
-			},
-		})
-	}
-	fd.Field = &schemapb.FieldData_Scalars{
-		Scalars: &schemapb.ScalarField{
-			Data: &schemapb.ScalarField_ArrayData{
-				ArrayData: &schemapb.ArrayArray{
-					Data:        data,
-					ElementType: schemapb.DataType_VarChar,
-				},
-			},
-		},
-	}
-	return fd
-}
-
-// ValueByIdx returns value of the provided index
-// error occurs when index out of range
-func (c *ColumnVarCharArray) ValueByIdx(idx int) ([][]byte, error) {
-	var r [][]byte // use default value
-	if idx < 0 || idx >= c.Len() {
-		return r, errors.New("index out of range")
-	}
-	return c.values[idx], nil
-}
-
-// AppendValue append value into column
-func(c *ColumnVarCharArray) AppendValue(i interface{}) error {
-	v, ok := i.([][]byte)
-	if !ok {
-		return fmt.Errorf("invalid type, expected [][]byte, got %T", i)
-	}
-	c.values = append(c.values, v)
-
-	return nil
-}
-
-// Data returns column data
-func (c *ColumnVarCharArray) Data() [][][]byte {
-	return c.values
-}
-
-// NewColumnVarChar auto generated constructor
-func NewColumnVarCharArray(name string, values [][][]byte) *ColumnVarCharArray {
-	return &ColumnVarCharArray {
-		name: name,
-		values: values,
-	}
-}
 
diff --git a/entity/genarray/gen_array.go b/entity/genarray/gen_array.go
index bd011ff..63c951b 100644
--- a/entity/genarray/gen_array.go
+++ b/entity/genarray/gen_array.go
@@ -131,7 +131,7 @@ func main() {
 		entity.FieldTypeInt64,
 		entity.FieldTypeFloat,
 		entity.FieldTypeDouble,
-		entity.FieldTypeVarChar,
+		// entity.FieldTypeVarChar, change to hand written
 	}
 
 	pf := func(ft entity.FieldType) interface{} {
diff --git a/entity/rows.go b/entity/rows.go
index 788346b..4c473cb 100644
--- a/entity/rows.go
+++ b/entity/rows.go
@@ -13,12 +13,13 @@ package entity
 
 import (
 	"encoding/json"
-	"errors"
 	"fmt"
 	"go/ast"
 	"reflect"
 	"strconv"
 	"strings"
+
+	"github.com/cockroachdb/errors"
 )
 
 const (
@@ -324,6 +325,12 @@ func AnyToColumns(rows []interface{}, schemas ...*Schema) ([]Column, error) {
 			data := make([][]byte, 0, rowsLen)
 			col := NewColumnJSONBytes(field.Name, data)
 			nameColumns[field.Name] = col
+		case FieldTypeArray:
+			col := NewArrayColumn(field)
+			if col == nil {
+				return nil, errors.Errorf("unsupported element type %s for Array", field.ElementType.String())
+			}
+			nameColumns[field.Name] = col
 		case FieldTypeFloatVector:
 			data := make([][]float32, 0, rowsLen)
 			dimStr, has := field.TypeParams[TypeParamDim]
@@ -415,6 +422,37 @@ func AnyToColumns(rows []interface{}, schemas ...*Schema) ([]Column, error) {
 	return columns, nil
 }
 
+func NewArrayColumn(f *Field) Column {
+	switch f.ElementType {
+	case FieldTypeBool:
+		return NewColumnBoolArray(f.Name, nil)
+
+	case FieldTypeInt8:
+		return NewColumnInt8Array(f.Name, nil)
+
+	case FieldTypeInt16:
+		return NewColumnInt16Array(f.Name, nil)
+
+	case FieldTypeInt32:
+		return NewColumnInt32Array(f.Name, nil)
+
+	case FieldTypeInt64:
+		return NewColumnInt64Array(f.Name, nil)
+
+	case FieldTypeFloat:
+		return NewColumnFloatArray(f.Name, nil)
+
+	case FieldTypeDouble:
+		return NewColumnDoubleArray(f.Name, nil)
+
+	case FieldTypeVarChar:
+		return NewColumnVarCharArray(f.Name, nil)
+
+	default:
+		return nil
+	}
+}
+
 // RowsToColumns rows to columns
 func RowsToColumns(rows []Row, schemas ...*Schema) ([]Column, error) {
 	anys := make([]interface{}, 0, len(rows))
diff --git a/entity/schema.go b/entity/schema.go
index f9dfd85..ee7179a 100644
--- a/entity/schema.go
+++ b/entity/schema.go
@@ -391,7 +391,7 @@ func (t FieldType) String() string {
 	case FieldTypeString:
 		return "string"
 	case FieldTypeVarChar:
-		return "[]byte"
+		return "string"
 	case FieldTypeArray:
 		return "Array"
 	case FieldTypeJSON:
@@ -425,7 +425,7 @@ func (t FieldType) PbFieldType() (string, string) {
 	case FieldTypeString:
 		return "String", "string"
 	case FieldTypeVarChar:
-		return "Bytes", "[]byte"
+		return "VarChar", "string"
 	case FieldTypeJSON:
 		return "JSON", "JSON"
 	case FieldTypeBinaryVector:


Test patch:
diff --git a/entity/columns_varchar_test.go b/entity/columns_varchar_test.go
index 4913554..46a8835 100644
--- a/entity/columns_varchar_test.go
+++ b/entity/columns_varchar_test.go
@@ -21,10 +21,10 @@ func TestColumnVarChar(t *testing.T) {
 	t.Run("test meta", func(t *testing.T) {
 		ft := FieldTypeVarChar
 		assert.Equal(t, "VarChar", ft.Name())
-		assert.Equal(t, "[]byte", ft.String())
+		assert.Equal(t, "string", ft.String())
 		pbName, pbType := ft.PbFieldType()
-		assert.Equal(t, "Bytes", pbName)
-		assert.Equal(t, "[]byte", pbType)
+		assert.Equal(t, "VarChar", pbName)
+		assert.Equal(t, "string", pbType)
 	})
 
 	t.Run("test column attribute", func(t *testing.T) {