Repository: risingwavelabs/risingwave
Pull request #20822: feat(stream): support not null in stream
Author: login: KeXiangWang
Labels: name: type/feature
description: Type: New feature.
name: user-facing-changes
description: Contains changes that are visible to users
Created: 2025-03-11T01:54:11Z | Merged: 2025-03-12T15:14:19Z

I hereby agree to the terms of the [RisingWave Labs, Inc. Contributor License Agreement](https://raw.githubusercontent.com/risingwavelabs/risingwave/17af8a747593ebdbfa826691daf75bdab7d14fa0/.github/contributor-license-agreement.txt).

## What's changed and what's your intention?

Followup of https://github.com/risingwavelabs/risingwave/pull/20611. For streaming input, for example, kafka input, or Sink into input, we drop the row violating NOT NULL constraint. And we ban NOT NULL in cdc tables.

## Checklist

- [ ] I have written necessary rustdoc comments.
- [ ] <!-- OPTIONAL --> I have added necessary unit tests and integration tests.
- [ ] <!-- OPTIONAL --> I have added test labels as necessary. <!-- See https://github.com/risingwavelabs/risingwave/blob/main/docs/developer-guide.md#ci-labels-guide) -->
- [ ] <!-- OPTIONAL --> I have added fuzzing tests or opened an issue to track them. <!-- Recommended for new SQL features, see #7934 -->
- [ ] <!-- OPTIONAL --> My PR contains breaking changes. <!-- If it deprecates some features, please create a tracking issue to remove them in the future -->
- [ ] <!-- OPTIONAL --> My PR changes performance-critical code, so I will run (micro) benchmarks and present the results. <!-- To manually trigger a benchmark, please check out [Notion](https://www.notion.so/risingwave-labs/Manually-trigger-nexmark-performance-dashboard-test-b784f1eae1cf48889b2645d020b6b7d3). -->
- [ ] <!-- OPTIONAL --> My PR contains critical fixes that are necessary to be merged into the latest release. <!-- Please check out the [details](https://github.com/risingwavelabs/risingwave/blob/main/CONTRIBUTING.md) -->

## Documentation

- [x] <!-- OPTIONAL --> My PR needs documentation updates. <!-- Please use the **Release note** section below to summarize the impact on users -->

<details>
<summary><b>Release note</b></summary>

Support NOT NULL in table schema.
For BatchInsert/Update, RW will throw an error for rows with NULL.
For Streaming, RW will ignore the rows with NULL silently.



Diff:
diff --git a/e2e_test/ddl/not_null.slt b/e2e_test/ddl/not_null.slt
index 7ddbea9d54141..e247e2132ae02 100644
--- a/e2e_test/ddl/not_null.slt
+++ b/e2e_test/ddl/not_null.slt
@@ -91,9 +91,27 @@ DROP TABLE test_array;
 
 ### Test stream input
 
+# TODO(Kexiang): enforce DEFAULT when altering table with NOT NULL, e.g., ALTER TABLE test_person ADD weight FLOAT DEFAULT 5 NOT NULL
 statement ok
 CREATE TABLE test_person (id INT PRIMARY KEY NOT NULL, name VARCHAR NULL, age INT NOT NULL, height FLOAT NOT NULL);
 
+query TT
+select column_name, is_nullable from information_schema.columns where table_name = 'test_person' order by column_name;
+----
+age    NO
+height NO
+id     NO
+name   YES
+
+query TT
+select rw_catalog.rw_columns.name, is_nullable from rw_catalog.rw_columns JOIN rw_catalog.rw_tables ON rw_catalog.rw_columns.relation_id = w_catalog.rw_tables.id WHERE rw_catalog.rw_tables.name = 'test_person' order by rw_catalog.rw_columns.name;
+----
+_rw_timestamp t
+age           f
+height        f
+id            f
+name          t
+
 statement ok
 INSERT INTO test_person (id, name, age, height) VALUES (1, 'Alice', 30, 5.5);
 
@@ -140,9 +158,7 @@ SELECT * FROM test_person ORDER BY id;
   1 Alice    30    5.5
   2 NULL     25      6
 101 John     28    5.9
-102 Sarah    NULL  5.5
 103 NULL     40    6.1
-104 Michael  35    NULL
 105 Emma     29    5.4
 
 statement ok
diff --git a/e2e_test/source_legacy/cdc/cdc.share_stream.slt b/e2e_test/source_legacy/cdc/cdc.share_stream.slt
index 45eccd5a6143e..03d0fd70f3b2f 100644
--- a/e2e_test/source_legacy/cdc/cdc.share_stream.slt
+++ b/e2e_test/source_legacy/cdc/cdc.share_stream.slt
@@ -173,6 +173,19 @@ create table orders_test (
    PRIMARY KEY (order_id)
 ) from mysql_mytest table 'mytest.orders';
 
+# invalid not null
+statement error CDC table with NOT NULL constraint is not supported
+create table orders_test (
+   order_id int,
+   order_date timestamp,
+   customer_name string,
+   price decimal,
+   product_id int NOT NULL,
+   order_status smallint NOT NULL,
+   next_order_id int as order_id + 1,
+   PRIMARY KEY (order_id)
+) from mysql_mytest table 'mytest.orders';
+
 statement ok
 create table orders_test (
    order_id int,
diff --git a/e2e_test/source_legacy/cdc/cdc.validate.postgres.slt b/e2e_test/source_legacy/cdc/cdc.validate.postgres.slt
index 30df0d298ee71..5102144209c6b 100644
--- a/e2e_test/source_legacy/cdc/cdc.validate.postgres.slt
+++ b/e2e_test/source_legacy/cdc/cdc.validate.postgres.slt
@@ -21,6 +21,25 @@ create table shipments (
  slot.name = 'shipments'
 );
 
+# invalid not null
+statement error CDC table with NOT NULL constraint is not supported
+create table shipments (
+  shipment_id INTEGER,
+  order_id INTEGER,
+  origin STRING,
+  destination STRING NOT NULL,
+  is_arrived boolean NOT NULL,
+ PRIMARY KEY (shipment_id)
+) with (
+ connector = 'postgres-cdc',
+ hostname = '${PGHOST:localhost}',
+ port = '${PGPORT:5432}',
+ username = '${PGUSER:$USER}',
+ password = '${PGPASSWORD:}',
+ database.name = '${PGDATABASE:postgres}',
+ table.name = 'shipments',
+ slot.name = 'shipments'
+);
 
 # invalid password
 statement error
diff --git a/src/common/src/catalog/column.rs b/src/common/src/catalog/column.rs
index dcc3c5a298307..22c8fda209a40 100644
--- a/src/common/src/catalog/column.rs
+++ b/src/common/src/catalog/column.rs
@@ -123,7 +123,9 @@ pub struct ColumnDesc {
     /// Currently the system column is used for `_rw_timestamp` only and is generated at runtime,
     /// so this field is not persisted.
     pub system_column: Option<SystemColumn>,
-    /// Whether the column is nullable. Only applies to BatchInsert/BatchUpdate operations into tables.
+    /// Whether the column is nullable.
+    /// If a column is not nullable, BatchInsert/BatchUpdate operations will throw an error when NULL is inserted/updated into.
+    /// The row contains NULL for this column will be ignored when streaming data into the table.
     pub nullable: bool,
 }
 
diff --git a/src/frontend/src/catalog/system_catalog/information_schema/columns.rs b/src/frontend/src/catalog/system_catalog/information_schema/columns.rs
index b48be438e6f04..b2fbb851fbe65 100644
--- a/src/frontend/src/catalog/system_catalog/information_schema/columns.rs
+++ b/src/frontend/src/catalog/system_catalog/information_schema/columns.rs
@@ -35,7 +35,10 @@ use risingwave_frontend_macro::system_catalog;
         NULL::integer AS numeric_scale,
         NULL::integer AS datetime_precision,
         c.position AS ordinal_position,
-        'YES' AS is_nullable,
+        CASE
+            WHEN c.is_nullable THEN 'YES'
+            ELSE 'NO'
+        END AS is_nullable,
         CASE
             WHEN c.data_type = 'varchar' THEN 'character varying'
             ELSE c.data_type
diff --git a/src/frontend/src/catalog/system_catalog/rw_catalog/rw_columns.rs b/src/frontend/src/catalog/system_catalog/rw_catalog/rw_columns.rs
index 01eed6d323479..8ec7228c2f84a 100644
--- a/src/frontend/src/catalog/system_catalog/rw_catalog/rw_columns.rs
+++ b/src/frontend/src/catalog/system_catalog/rw_catalog/rw_columns.rs
@@ -34,6 +34,7 @@ struct RwColumn {
     is_primary_key: bool,
     is_distribution_key: bool,
     is_generated: bool,
+    is_nullable: bool,
     generation_expression: Option<String>,
     data_type: String,
     type_oid: i32,
@@ -68,6 +69,7 @@ fn read_rw_columns_in_schema(current_user: &UserCatalog, schema: &SchemaCatalog)
                 is_primary_key: false,
                 is_distribution_key: false,
                 is_generated: false,
+                is_nullable: false,
                 generation_expression: None,
                 data_type: column.data_type().to_string(),
                 type_oid: column.data_type().to_oid(),
@@ -88,6 +90,7 @@ fn read_rw_columns_in_schema(current_user: &UserCatalog, schema: &SchemaCatalog)
                 is_primary_key: sink.downstream_pk.contains(&index),
                 is_distribution_key: sink.distribution_key.contains(&index),
                 is_generated: false,
+                is_nullable: column.nullable(),
                 generation_expression: None,
                 data_type: column.data_type().to_string(),
                 type_oid: column.data_type().to_oid(),
@@ -109,6 +112,7 @@ fn read_rw_columns_in_schema(current_user: &UserCatalog, schema: &SchemaCatalog)
                 is_primary_key: table.pk.contains(&index),
                 is_distribution_key: false,
                 is_generated: false,
+                is_nullable: column.nullable(),
                 generation_expression: None,
                 data_type: column.data_type().to_string(),
                 type_oid: column.data_type().to_oid(),
@@ -133,6 +137,7 @@ fn read_rw_columns_in_schema(current_user: &UserCatalog, schema: &SchemaCatalog)
                     is_primary_key: table.pk().iter().any(|idx| idx.column_index == index),
                     is_distribution_key: table.distribution_key.contains(&index),
                     is_generated: column.is_generated(),
+                    is_nullable: column.nullable(),
                     generation_expression: column.generated_expr().map(|expr_node| {
                         let expr = ExprImpl::from_expr_proto(expr_node).unwrap();
                         let expr_display = ExprDisplay {
@@ -163,6 +168,7 @@ fn read_rw_columns_in_schema(current_user: &UserCatalog, schema: &SchemaCatalog)
                     is_primary_key: source.pk_col_ids.contains(&column.column_id()),
                     is_distribution_key: false,
                     is_generated: false,
+                    is_nullable: column.nullable(),
                     generation_expression: None,
                     data_type: column.data_type().to_string(),
                     type_oid: column.data_type().to_oid(),
diff --git a/src/frontend/src/handler/create_table.rs b/src/frontend/src/handler/create_table.rs
index 0d3576625a31b..abb2cac3d6b47 100644
--- a/src/frontend/src/handler/create_table.rs
+++ b/src/frontend/src/handler/create_table.rs
@@ -470,6 +470,7 @@ pub(crate) async fn gen_create_table_plan_with_source(
     let with_properties = bind_connector_props(&handler_args, &format_encode, false)?;
     if with_properties.is_shareable_cdc_connector() {
         generated_columns_check_for_cdc_table(&column_defs)?;
+        not_null_check_for_cdc_table(&wildcard_idx, &column_defs)?;
     }
 
     let db_name: &str = &session.database();
@@ -1047,7 +1048,7 @@ pub(super) async fn handle_create_table_plan(
         }
 
         (None, Some(cdc_table)) => {
-            sanity_check_for_cdc_table(
+            sanity_check_for_table_on_cdc_source(
                 append_only,
                 &column_defs,
                 &wildcard_idx,
@@ -1055,6 +1056,9 @@ pub(super) async fn handle_create_table_plan(
                 &source_watermarks,
             )?;
 
+            generated_columns_check_for_cdc_table(&column_defs)?;
+            not_null_check_for_cdc_table(&wildcard_idx, &column_defs)?;
+
             let session = &handler_args.session;
             let db_name = &session.database();
             let user_name = &session.user_name();
@@ -1068,8 +1072,6 @@ pub(super) async fn handle_create_table_plan(
             let (format_encode, source_name) =
                 Binder::resolve_schema_qualified_name(db_name, cdc_table.source_name.clone())?;
 
-            generated_columns_check_for_cdc_table(&column_defs)?;
-
             let source = {
                 let catalog_reader = session.env().catalog_reader().read_guard();
                 let schema_path =
@@ -1152,6 +1154,7 @@ pub(super) async fn handle_create_table_plan(
     Ok((plan, source, table, job_type))
 }
 
+// For both table from cdc source and table with cdc connector
 fn generated_columns_check_for_cdc_table(columns: &Vec<ColumnDef>) -> Result<()> {
     let mut found_generated_column = false;
     for column in columns {
@@ -1178,7 +1181,29 @@ fn generated_columns_check_for_cdc_table(columns: &Vec<ColumnDef>) -> Result<()>
     Ok(())
 }
 
-fn sanity_check_for_cdc_table(
+// For both table from cdc source and table with cdc connector
+fn not_null_check_for_cdc_table(
+    wildcard_idx: &Option<usize>,
+    column_defs: &Vec<ColumnDef>,
+) -> Result<()> {
+    if !wildcard_idx.is_some()
+        && column_defs.iter().any(|col| {
+            col.options
+                .iter()
+                .any(|opt| matches!(opt.option, ColumnOption::NotNull))
+        })
+    {
+        return Err(ErrorCode::NotSupported(
+            "CDC table with NOT NULL constraint is not supported".to_owned(),
+            "Please remove the NOT NULL constraint for columns".to_owned(),
+        )
+        .into());
+    }
+    Ok(())
+}
+
+// Only for table from cdc source
+fn sanity_check_for_table_on_cdc_source(
     append_only: bool,
     column_defs: &Vec<ColumnDef>,
     wildcard_idx: &Option<usize>,
@@ -1217,6 +1242,7 @@ fn sanity_check_for_cdc_table(
         )
         .into());
     }
+
     if append_only {
         return Err(ErrorCode::NotSupported(
             "append only modifier on the table created from a CDC source".into(),
diff --git a/src/frontend/src/optimizer/mod.rs b/src/frontend/src/optimizer/mod.rs
index 3930d07867dfd..8441b83dbaf64 100644
--- a/src/frontend/src/optimizer/mod.rs
+++ b/src/frontend/src/optimizer/mod.rs
@@ -17,6 +17,7 @@ use std::sync::Arc;
 
 pub mod plan_node;
 
+use plan_node::StreamFilter;
 pub use plan_node::{Explain, PlanRef};
 
 pub mod property;
@@ -779,12 +780,19 @@ impl PlanRoot {
             PrimaryKeyKind::UserDefinedPrimaryKey
         };
 
-        let column_descs = columns
+        let column_descs: Vec<ColumnDesc> = columns
             .iter()
             .filter(|&c| c.can_dml())
             .map(|c| c.column_desc.clone())
             .collect();
 
+        let mut not_null_idxs = vec![];
+        for (idx, column) in column_descs.iter().enumerate() {
+            if !column.nullable {
+                not_null_idxs.push(idx);
+            }
+        }
+
         let version_column_index = if let Some(version_column) = with_version_column {
             find_version_column_index(&columns, version_column)?
         } else {
@@ -909,7 +917,12 @@ impl PlanRoot {
             RequiredDist::ShardByKey(bitset)
         };
 
-        let stream_plan = inline_session_timezone_in_exprs(context, stream_plan)?;
+        let mut stream_plan = inline_session_timezone_in_exprs(context, stream_plan)?;
+
+        if !not_null_idxs.is_empty() {
+            stream_plan =
+                StreamFilter::filter_out_any_null_rows(stream_plan.clone(), &not_null_idxs);
+        }
 
         StreamMaterialize::create_for_table(
             stream_plan,
diff --git a/src/frontend/src/optimizer/plan_node/stream_filter.rs b/src/frontend/src/optimizer/plan_node/stream_filter.rs
index a08883149b403..c554c6198b373 100644
--- a/src/frontend/src/optimizer/plan_node/stream_filter.rs
+++ b/src/frontend/src/optimizer/plan_node/stream_filter.rs
@@ -12,13 +12,14 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use risingwave_common::types::DataType;
 use risingwave_pb::stream_plan::FilterNode;
 use risingwave_pb::stream_plan::stream_node::PbNodeBody;
 
 use super::stream::prelude::*;
 use super::utils::impl_distill_by_unit;
 use super::{ExprRewritable, PlanRef, PlanTreeNodeUnary, StreamNode, generic};
-use crate::expr::{Expr, ExprImpl, ExprRewriter, ExprVisitor};
+use crate::expr::{Expr, ExprImpl, ExprRewriter, ExprType, ExprVisitor, FunctionCall, InputRef};
 use crate::optimizer::plan_node::PlanBase;
 use crate::optimizer::plan_node::expr_visitable::ExprVisitable;
 use crate::stream_fragmenter::BuildFragmentGraphState;
@@ -50,6 +51,22 @@ impl StreamFilter {
     pub fn predicate(&self) -> &Condition {
         &self.core.predicate
     }
+
+    /// Create a `StreamFilter` to filter out rows where null values violate NOT NULL constraints.
+    pub fn filter_out_any_null_rows(input: PlanRef, not_null_idxs: &[usize]) -> PlanRef {
+        let schema = input.schema();
+        let cond = ExprImpl::and(not_null_idxs.iter().map(|&i| {
+            FunctionCall::new_unchecked(
+                ExprType::IsNotNull,
+                vec![InputRef::new(i, schema.fields()[i].data_type.clone()).into()],
+                DataType::Boolean,
+            )
+            .into()
+        }));
+        let predicate = Condition::with_expr(cond);
+        let logical_filter = generic::Filter::new(predicate, input);
+        StreamFilter::new(logical_filter).into()
+    }
 }
 
 impl PlanTreeNodeUnary for StreamFilter {


Reviews:
user: login: st1page
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T04:01:33Z
user: login: BugenZhao
body: LGTM
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T05:27:26Z
user: login: KeXiangWang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T15:51:28Z

Review comments:
diff_hunk: @@ -909,7 +917,12 @@ impl PlanRoot {
             RequiredDist::ShardByKey(bitset)
         };
 
-        let stream_plan = inline_session_timezone_in_exprs(context, stream_plan)?;
+        let mut stream_plan = inline_session_timezone_in_exprs(context, stream_plan)?;
+
+        if !not_null_keys.is_empty() {
path: src/frontend/src/optimizer/mod.rs
user: login: BugenZhao
body: Why does it called `keys`? IIUC the `NOT NULL` constraint can be applied to any columns. 🤔
created_at: 2025-03-11T05:27:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -909,7 +917,12 @@ impl PlanRoot {
             RequiredDist::ShardByKey(bitset)
         };
 
-        let stream_plan = inline_session_timezone_in_exprs(context, stream_plan)?;
+        let mut stream_plan = inline_session_timezone_in_exprs(context, stream_plan)?;
+
+        if !not_null_keys.is_empty() {
path: src/frontend/src/optimizer/mod.rs
user: login: KeXiangWang
body: Updated. Thx!
created_at: 2025-03-11T15:51:28Z
author_association: CONTRIBUTOR

PR comments:
user: login: BugenZhao
created_at: 2025-03-11T04:51:35Z
body: Could you please allow some time for the other reviewers to take a look? Thanks. cc @st1page 
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-11T05:24:52Z
body: Hi, there.

:memo: **Telemetry Reminder**:
If you're implementing this feature, please consider adding telemetry metrics to track its usage. This helps us understand how the feature is being used and improve it further.
You can find the function `report_event` of telemetry reporting in the following files. Feel free to ask questions if you need any guidance!
* `src/frontend/src/telemetry.rs`
* `src/meta/src/telemetry.rs`
* `src/stream/src/telemetry.rs`
* `src/storage/compactor/src/telemetry.rs`
Or calling `report_event_common` (`src/common/telemetry_event/src/lib.rs`) as if finding it hard to implement.
:sparkles: Thank you for your contribution to RisingWave! :sparkles:


---

> This is an automated comment created by the [peaceiris/actions-label-commenter]. Responding to the bot or mentioning it won't have any effect.

[peaceiris/actions-label-commenter]: https://github.com/peaceiris/actions-label-commenter
author_association: CONTRIBUTOR
user: login: st1page
created_at: 2025-03-12T03:58:55Z
body: > Could you please allow some time for the other reviewers to take a look? Thanks. cc @st1page

Sorry for that, I thought it was not a big change.
author_association: CONTRIBUTOR