Repository: ClickHouse/ClickHouse
Pull request #80037: Fix bug where materialized views with UNIONs don't work on new replicas
Author: login: samay-sharma
Labels: name: pr-bugfix
description: Pull request with bugfix, not backported by default
name: pr-must-backport
description: Pull request should be backported intentionally. Use this label with great care!
name: pr-backports-created
description: Backport PRs are successfully created, it won't be processed by CI script anymore
name: pr-backports-created-cloud
description: deprecated label, NOOP
name: pr-synced-to-cloud
description: The PR is synced to the cloud repo
name: pr-must-backport-synced
description: The `*-must-backport` labels are synced into the cloud Sync PR
Created: 2025-05-10T05:16:43Z | Merged: 2025-05-11T12:32:43Z

Should fix https://github.com/ClickHouse/ClickHouse/issues/77569

<!---
A technical comment, you are free to remove or leave it as it is when PR is created
The following categories are used in the next scripts, update them accordingly
utils/changelog/changelog.py
tests/ci/cancel_and_rerun_workflow_lambda/app.py
-->
### Changelog category (leave one):
- Bug Fix (user-visible misbehavior in an official stable release)

### Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Allow materialized views with UNIONs to work correctly on new replicas.

### Documentation entry for user-facing changes

- [ ] Documentation is written (mandatory for new features)

<!---
Directly edit documentation source files in the "docs" folder with the same pull-request as code changes

or

Add a user-readable short description of the changes that should be added to docs.clickhouse.com below.

At a minimum, the following information should be added (but add more as needed).
- Motivation: Why is this function, table engine, etc. useful to ClickHouse users?

- Parameters: If the feature being added takes arguments, options or is influenced by settings, please list them below with a brief explanation.

- Example use: A query or command.
-->



Diff:
diff --git a/src/Databases/DatabaseReplicated.cpp b/src/Databases/DatabaseReplicated.cpp
index bb580af01a78..da6764b0879a 100644
--- a/src/Databases/DatabaseReplicated.cpp
+++ b/src/Databases/DatabaseReplicated.cpp
@@ -24,6 +24,7 @@
 #include <Interpreters/DatabaseCatalog.h>
 #include <Interpreters/InterpreterCreateQuery.h>
 #include <Interpreters/InterpreterSetQuery.h>
+#include <Interpreters/NormalizeSelectWithUnionQueryVisitor.h>
 #include <Interpreters/ReplicatedDatabaseQueryStatusSource.h>
 #include <Interpreters/evaluateConstantExpression.h>
 #include <Interpreters/executeDDLQueryOnCluster.h>
@@ -66,6 +67,7 @@ namespace Setting
     extern const SettingsDistributedDDLOutputMode distributed_ddl_output_mode;
     extern const SettingsInt64 distributed_ddl_task_timeout;
     extern const SettingsBool throw_on_unsupported_query_inside_transaction;
+    extern const SettingsSetOperationMode union_default_mode;
 }
 
 namespace ServerSetting
@@ -1463,8 +1465,11 @@ void DatabaseReplicated::recoverLostReplica(const ZooKeeperPtr & current_zookeep
                 }
 
                 auto query_ast = parseQueryFromMetadataInZooKeeper(table_name, create_query_string);
-
                 auto create_query_context = make_query_context();
+
+                NormalizeSelectWithUnionQueryVisitor::Data data{create_query_context->getSettingsRef()[Setting::union_default_mode]};
+                NormalizeSelectWithUnionQueryVisitor{data}.visit(query_ast);
+
                 /// Check larger comment in DatabaseOnDisk::createTableFromAST
                 /// TL;DR applySettingsFromQuery will move the settings from engine to query level
                 /// making it possible to overcome a backward incompatible change.
diff --git a/tests/integration/test_matview_union_replicated/__init__.py b/tests/integration/test_matview_union_replicated/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tests/integration/test_matview_union_replicated/test.py b/tests/integration/test_matview_union_replicated/test.py
new file mode 100644
index 000000000000..bdefd65a5f31
--- /dev/null
+++ b/tests/integration/test_matview_union_replicated/test.py
@@ -0,0 +1,86 @@
+import pytest
+from helpers.cluster import ClickHouseCluster
+from helpers.test_tools import assert_eq_with_retry
+
+cluster = ClickHouseCluster(__file__)
+
+# Create two nodes - one main node and one replica
+node1 = cluster.add_instance("node1", with_zookeeper=True)
+node2 = cluster.add_instance("node2", with_zookeeper=True)
+
+@pytest.fixture(scope="module")
+def started_cluster():
+    try:
+        cluster.start()
+        yield cluster
+    finally:
+        cluster.shutdown()
+
+def test_matview_union_replicated(started_cluster):
+
+    # Create replicated database, source and target tables and matview
+    node1.query("DROP DATABASE IF EXISTS union_test_replicated SYNC")
+    node1.query("CREATE DATABASE union_test_replicated ENGINE=Replicated('/test/union_replica' , 'shard1', 'replica' || '1');")
+
+    node1.query("""
+    CREATE TABLE union_test_replicated.source_1
+    (
+        timestamp DateTime,
+        value Float64
+    )
+    ENGINE = ReplicatedMergeTree
+    ORDER BY timestamp
+    """)
+
+    node1.query("""
+    CREATE TABLE union_test_replicated.source_2
+    (
+        timestamp DateTime,
+        value Float64
+    )
+    ENGINE = ReplicatedMergeTree
+    ORDER BY timestamp
+    """)
+
+    node1.query("""
+    CREATE TABLE union_test_replicated.target
+    (
+        timestamp DateTime,
+        value Float64
+    )
+    ENGINE = ReplicatedMergeTree
+    ORDER BY timestamp
+    """)
+    
+    node1.query("""
+    CREATE MATERIALIZED VIEW union_test_replicated.mv_test TO union_test_replicated.target AS
+    WITH source_data AS
+    (
+        SELECT timestamp, value FROM union_test_replicated.source_1
+        UNION ALL
+        SELECT timestamp, value FROM union_test_replicated.source_2
+    )
+    SELECT timestamp, value FROM source_data
+    """)
+
+    # Verify INSERT works on Node #1
+    node1.query("INSERT INTO union_test_replicated.source_1 VALUES (now(), 1)")
+
+    # Attach replica on second node
+    node2.query("DROP DATABASE IF EXISTS union_test_replicated SYNC")
+    node2.query("CREATE DATABASE union_test_replicated ENGINE=Replicated('/test/union_replica' , 'shard1', 'replica' || '2');")
+    node2.query("SYSTEM SYNC DATABASE REPLICA union_test_replicated")
+
+    # Verify the table structure on replica
+    assert_eq_with_retry(
+        node2,
+        "DESCRIBE TABLE union_test_replicated.source_1",
+        "timestamp\tDateTime\t\t\t\t\nvalue\tFloat64\t\t\t\t\n"
+    )
+
+    # Run INSERT on replica
+    node2.query("INSERT INTO union_test_replicated.source_1 VALUES (now(), 1)")
+
+    # Clean up
+    node1.query("DROP DATABASE IF EXISTS union_test_replicated SYNC")
+    node2.query("DROP DATABASE IF EXISTS union_test_replicated SYNC")


Reviews:
user: login: mstetsyuk
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-10T07:48:36Z
user: login: nikitamikhaylov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-11T12:14:04Z
user: login: nikitamikhaylov
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-11T12:14:48Z

Review comments:
diff_hunk: @@ -1463,8 +1465,11 @@ void DatabaseReplicated::recoverLostReplica(const ZooKeeperPtr & current_zookeep
                 }
 
                 auto query_ast = parseQueryFromMetadataInZooKeeper(table_name, create_query_string);
-
                 auto create_query_context = make_query_context();
+
+                NormalizeSelectWithUnionQueryVisitor::Data data{create_query_context->getSettingsRef()[Setting::union_default_mode]};
path: src/Databases/DatabaseReplicated.cpp
user: login: mstetsyuk
body: nit: this probably belongs in `InterpreterCreateQuery`
created_at: 2025-05-10T07:48:32Z
author_association: MEMBER
diff_hunk: @@ -1463,8 +1465,11 @@ void DatabaseReplicated::recoverLostReplica(const ZooKeeperPtr & current_zookeep
                 }
 
                 auto query_ast = parseQueryFromMetadataInZooKeeper(table_name, create_query_string);
-
                 auto create_query_context = make_query_context();
+
+                NormalizeSelectWithUnionQueryVisitor::Data data{create_query_context->getSettingsRef()[Setting::union_default_mode]};
path: src/Databases/DatabaseReplicated.cpp
user: login: nikitamikhaylov
body: Interestingly, DatabaseBackup and DatabaseOrdinary have the same code. Maybe indeed it is worth to move it to the InterpreterCreateQuery instead. 
created_at: 2025-05-11T12:14:04Z
author_association: MEMBER

PR comments:
user: login: clickhouse-gh[bot]
created_at: 2025-05-10T05:17:09Z
body: Workflow [[PR](https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=80037&sha=latest&name_0=PR)], commit [b1d8df96]
author_association: CONTRIBUTOR