Project: SeaTunnel (App Definition and Development / Streaming & Messaging)
File: Assert.md

Assert sink connector A flink sink plugin which can assert illegal data by user defined rules | Name | Type | Required | Default | ||-|-|| | rules | ConfigMap | yes | - | | rules.field_rules | string | yes | - | | rules.fieldrules.fieldname | string\|ConfigMap | yes | - | | rules.fieldrules.fieldtype | string | no | - | | rules.fieldrules.fieldvalue | ConfigList | no | - | | rules.fieldrules.fieldvalue.rule_type | string | no | - | | rules.fieldrules.fieldvalue.rule_value | numeric | no | - | | rules.fieldrules.fieldvalue.equals_to | boolean\|numeric\|string\|ConfigList\|ConfigMap | no | - | | rules.row_rules | string | yes | - | | rules.rowrules.ruletype | string | no | - | | rules.rowrules.rulevalue | string | no | - | | rules.catalogtablerule | ConfigMap | no | - | | rules.catalogtablerule.primarykeyrule | ConfigMap | no | - | | rules.catalogtablerule.primarykeyrule.primarykeyname | string | no | - | | rules.catalogtablerule.primarykeyrule.primarykeycolumns | ConfigList | no | - | | rules.catalogtablerule.constraintkeyrule | ConfigList | no | - | | rules.catalogtablerule.constraintkeyrule.constraintkeyname | string | no | - | | rules.catalogtablerule.constraintkeyrule.constraintkeytype | string | no | - | | rules.catalogtablerule.constraintkeyrule.constraintkeycolumns | ConfigList | no | - | | rules.catalogtablerule.constraintkeyrule.constraintkeycolumns.constraintkeycolumn_name | string | no | - | | rules.catalogtablerule.constraintkeyrule.constraintkeycolumns.constraintkeysort_type | string | no | - | | rules.catalogtablerule.column_rule | ConfigList | no | - | | rules.catalogtablerule.column_rule.name | string | no | - | | rules.catalogtablerule.column_rule.type | string | no | - | | rules.catalogtablerule.columnrule.columnlength | int | no | - | | rules.catalogtablerule.column_rule.nullable | boolean | no | - | | rules.catalogtablerule.columnrule.defaultvalue | string | no | - | | rules.catalogtablerule.column_rule.comment | comment | no | - | | rules.table-names | ConfigList | no | - | | common-options | | no | - | Rule definition of user's available data. Each rule represents one field validation or row num validation. field rules for field validation field namestring Field type declarations should adhere to this . A list value rule define the data value validation The following rules are supported for now NOT_NULL `value can't be null` NULL `value can be null` MIN `define the minimum value of data` MAX `define the maximum value of data` MIN_LENGTH `define the minimum string length of a string data` MAX_LENGTH `define the maximum string length of a string data` MIN_ROW `define the minimun number of rows` MAX_ROW `define the maximum number of rows` The value related to rule type. When the `ruletype` is `MIN`, `MAX`, `MINLENGTH`, `MAXLENGTH`, `MINROW` or `MAXROW`, users need to assign a value to the `rulevalue`. `equalsto` is used to compare whether the field value is equal to the configured expected value. You can assign values of all types to `equalsto`. These types are detailed

For instance, if one field is a row with three fields, and the declaration of row type is `{a = array<string>, b = map<string, decimal(30, 2)>, c={c0 = int, b = string}}`, users can assign the value `[["a", "b"], { k0 = 9999.99, k1 = 111.11 }, [123, "abcd"]]` to `equalsto`. The way of defining field values is consistent with . `equalsto` cannot be applied to `null` type fields. However, users can use the rule type `NULL` for verification, such as `{ruletype = NULL}`. Used to assert the catalog table is same with the user defined table. Used to assert the table should be in the data. Sink plugin common parameters, please refer to for details the whole config obey with `hocon` style ```hocon Assert { rules = { row_rules = [ { ruletype = MAXROW rule_value = 10 }, { ruletype = MINROW rule_value = 5 } ], field_rules = [{ field_name = name field_type = string field_value = [ { ruletype = NOTNULL }, { ruletype = MINLENGTH rule_value = 5 }, { ruletype = MAXLENGTH rule_value = 10 } ] }, { field_name = age field_type = int field_value = [ { ruletype = NOTNULL equals_to = 23 }, { rule_type = MIN rule_value = 32767 }, { rule_type = MAX rule_value = 2147483647 } ] } ] catalogtablerule { primarykeyrule = { primarykeyname = "primary key" primarykeycolumns = ["id"] } constraintkeyrule = [ { constraintkeyname = "unique_name" constraintkeytype = UNIQUE_KEY constraintkeycolumns = [ { constraintkeycolumn_name = "id" constraintkeysort_type = ASC } ] } ] column_rule = [ { name = "id" type = bigint }, { name = "name" type = string }, { name = "age" type = int } ] } } } ``` Here is a more complex example about `equals_to`. The example involves FakeSource. You may want to learn it, please read this . ```hocon source { FakeSource { row.num = 1 schema = { fields { c_null = "null" c_string = string c_boolean = boolean c_tinyint = tinyint c_smallint = smallint c_int = int c_bigint = bigint c_float = float c_double = double c_decimal = "decimal(30, 8)" c_date = date c_timestamp = timestamp c_time = time c_bytes = bytes c_array = "array<int>" c_map = "map<time, string>" cmapnest = "map<string, {cint = int, cstring = string}>" c_row = { c_null = "null" c_string = string c_boolean = boolean c_tinyint = tinyint c_smallint = smallint c_int = int c_bigint = bigint c_float = float c_double = double c_decimal = "decimal(30, 8)" c_date = date c_timestamp = timestamp c_time = time c_bytes = bytes c_array = "array<int>" c_map = "map<string, string>" } } } rows = [ { kind = INSERT fields = [ null, "AAA", false, 1, 1, 333, 323232, 3.1, 9.33333, 99999.99999999, "2012-12-21", "2012-12-21T12:34:56", "12:34:56", "bWlJWmo=", [0, 1, 2], "{ 12:01:26 = v0 }", { k1 = [123, "BBB-BB"]}, [ null, "AAA", false, 1, 1, 333, 323232, 3.1, 9.33333,

"2012-12-21", "2012-12-21T12:34:56", "12:34:56", "bWlJWmo=", [0, 1, 2], { k0 = v0 } ] ] } ] resulttablename = "fake" } } sink{ Assert { sourcetablename = "fake" rules = { row_rules = [ { ruletype = MAXROW rule_value = 1 }, { ruletype = MINROW rule_value = 1 } ], field_rules = [ { fieldname = cnull field_type = "null" field_value = [ { rule_type = NULL } ] }, { fieldname = cstring field_type = string field_value = [ { ruletype = NOTNULL equals_to = "AAA" } ] }, { fieldname = cboolean field_type = boolean field_value = [ { ruletype = NOTNULL equals_to = false } ] }, { fieldname = ctinyint field_type = tinyint field_value = [ { ruletype = NOTNULL equals_to = 1 } ] }, { fieldname = csmallint field_type = smallint field_value = [ { ruletype = NOTNULL equals_to = 1 } ] }, { fieldname = cint field_type = int field_value = [ { ruletype = NOTNULL equals_to = 333 } ] }, { fieldname = cbigint field_type = bigint field_value = [ { ruletype = NOTNULL equals_to = 323232 } ] }, { fieldname = cfloat field_type = float field_value = [ { ruletype = NOTNULL equals_to = 3.1 } ] }, { fieldname = cdouble field_type = double field_value = [ { ruletype = NOTNULL equals_to = 9.33333 } ] }, { fieldname = cdecimal field_type = "decimal(30, 8)" field_value = [ { ruletype = NOTNULL equals_to = 99999.99999999 } ] }, { fieldname = cdate field_type = date field_value = [ { ruletype = NOTNULL equals_to = "2012-12-21" } ] }, { fieldname = ctimestamp field_type = timestamp field_value = [ { ruletype = NOTNULL equals_to = "2012-12-21T12:34:56" } ] }, { fieldname = ctime field_type = time field_value = [ { ruletype = NOTNULL equals_to = "12:34:56" } ] }, { fieldname = cbytes field_type = bytes field_value = [ { ruletype = NOTNULL equals_to = "bWlJWmo=" } ] }, { fieldname = carray field_type = "array<int>" field_value = [ { ruletype = NOTNULL equals_to = [0, 1, 2] } ] }, { fieldname = cmap field_type = "map<time, string>" field_value = [ { ruletype = NOTNULL equals_to = "{ 12:01:26 = v0 }" } ] }, { fieldname = cmap_nest fieldtype = "map<string, {cint = int, c_string = string}>" field_value = [ { ruletype = NOTNULL equals_to = { k1 = [123, "BBB-BB"] } } ] }, { fieldname = crow field_type = { c_null = "null" c_string = string c_boolean = boolean c_tinyint = tinyint c_smallint = smallint c_int = int c_bigint = bigint c_float = float c_double = double c_decimal = "decimal(30, 8)" c_date = date c_timestamp = timestamp c_time = time c_bytes = bytes c_array = "array<int>" c_map = "map<string, string>" } field_value = [ { ruletype = NOTNULL equals_to = [ null, "AAA", false, 1, 1, 333, 323232, 3.1, 9.33333, 99999.99999999, "2012-12-21", "2012-12-21T12:34:56", "12:34:56", "bWlJWmo=", [0, 1, 2], { k0 = v0 } ] } ] } ] } } } ``` Add Assert Sink Connector ) (): check rows not empty check minimum number of rows check maximum number of rows ) () ) ()