Repository: apache/doris-flink-connector
Pull request #573: [improve] auto create Doris table for new MongoDB collections in Data…
Author: login: kwonder0926
Created: 2025-03-11T12:05:20Z | Merged: 2025-03-27T10:57:19Z


# Proposed changes

Issue Number: close #570 

## Problem Summary:

During full database synchronization, if `single-sink=true`, when a new collection is created in MongoDB, Doris will also create a corresponding table synchronously and start synchronizing the data of that collection.

## Checklist(Required)

1. Does it affect the original behavior: No
2. Has unit tests been added: No Need
3. Has document been added or modified: No
4. Does it need to update dependencies: No
5. Are there any changes that cannot be rolled back: No


Linked issue #570: [Enhancement] Database Sync: Doris-Flink Connector Fails to Auto-Create Table When MongoDB New Collection Inserted
### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.


### Version

- Doris Version: 2.1.8
- Flink Version: 1.17.2
- Connector Version:  flink-doris-connector-1.17-25.0.0.jar
- MongoDB Version: 4.4.29

### What's Wrong?

Here is the command I used for synchronization. I'm performing a full-table synchronization:

```bash
./bin/flink run \
    -Dexecution.checkpointing.interval=10s \
    -Dparallelism.default=2 \
    -Dpipeline.operator-chaining=false \
    -c org.apache.doris.flink.tools.cdc.CdcTools \
    ./lib/flink-doris-connector-1.17-25.0.0.jar \
    mongodb-sync-database \
    --job-name mongo2Doris \
    --database xlsms_test \
    --table-prefix db_ \
    --schema-change-mode debezium_structure \
    --mongodb-conf hosts=192.168.10.102:27017,192.168.10.103:27017,192.168.10.104:27017 \
    --mongodb-conf username=stuser \
    --mongodb-conf password=stpw \
    --mongodb-conf database=xlsms \
    --mongodb-conf scan.startup.mode=initial \
    --mongodb-conf schema.sample-percent=0.2 \
    --including-tables ".*" \
    --sink-conf fenodes=192.168.10.102:8030 \
    --sink-conf username=root \
    --sink-conf password=123456 \
    --sink-conf jdbc-url=jdbc:mysql://192.168.10.102:9030 \
    --sink-conf sink.label-prefix=sms \
    --sink-conf sink.enable-2pc=false \
    --sink-conf sink.properties.format=json \
    --sink-conf sink.properties.read_json_by_line=true \
    --single-sink true \
    --table-conf replication_num=1
```

The synchronization process was normal. It successfully synchronized all the existing tables and data from my MongoDB database to Doris. However, when I created a new collection in MongoDB and inserted data into it, no new table appeared in Doris. My command for creating the collection and inserting data is as follows:

```javascript
db["test"].insertMany([
    {
        "name": "John Doe",
        "age": 30,
        "city": "New York"
    },
    {
        "name": "Jane Smith",
        "age": 25,
        "city": "Los Angeles"
    }
]);
```

### What You Expected?

I expect that when I create a new collection with data in MongoDB, Doris can also synchronously create a corresponding table and synchronize the data. However, if I create an empty collection in MongoDB, Doris doesn't need to take any action.

I've gone through the source code of the Doris connector on GitHub. Although I'm not entirely familiar with all the code logic, I suspect that the problem might be in the part where the connector listens for metadata changes in MongoDB. It may not be handling the CREATE COLLECTION events correctly, or there might be an issue with the logic that maps the new collection's schema to create a table in Doris.
I'm looking forward to getting help on how to solve this problem. Is there a way to modify the source code to achieve the functionality I need? Or is there a configuration option that I've missed? Any advice or guidance would be greatly appreciated.

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Linked issue comments: user: login: kwonder0926
created_at: 2025-03-10T11:56:51Z
body: After reviewing the source code, I found that this isn't a bug but rather an unimplemented feature. I'm currently exploring solutions for this and hope to submit a pull request if my efforts prove successful.
author_association: CONTRIBUTOR

Diff:
diff --git a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
index 06e49e24d..326b4e879 100644
--- a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
+++ b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
@@ -28,14 +28,12 @@
 import org.apache.flink.util.Preconditions;
 import org.apache.flink.util.StringUtils;
 
-import org.apache.doris.flink.catalog.doris.DorisSchemaFactory;
 import org.apache.doris.flink.catalog.doris.DorisSystem;
 import org.apache.doris.flink.catalog.doris.TableSchema;
 import org.apache.doris.flink.cfg.DorisConnectionOptions;
 import org.apache.doris.flink.cfg.DorisExecutionOptions;
 import org.apache.doris.flink.cfg.DorisOptions;
 import org.apache.doris.flink.cfg.DorisReadOptions;
-import org.apache.doris.flink.exception.DorisSystemException;
 import org.apache.doris.flink.sink.DorisSink;
 import org.apache.doris.flink.sink.schema.SchemaChangeMode;
 import org.apache.doris.flink.sink.writer.WriteMode;
@@ -44,12 +42,12 @@
 import org.apache.doris.flink.table.DorisConfigOptions;
 import org.apache.doris.flink.tools.cdc.converter.TableNameConverter;
 import org.apache.doris.flink.tools.cdc.oracle.OracleDatabaseSync;
+import org.apache.doris.flink.tools.cdc.utils.DorisTableUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.sql.Connection;
 import java.sql.SQLException;
-import java.sql.SQLSyntaxErrorException;
 import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -150,7 +148,13 @@ public boolean build() throws Exception {
             // Calculate the mapping relationship between upstream and downstream tables
             tableMapping.put(
                     schema.getTableIdentifier(), String.format("%s.%s", targetDb, dorisTable));
-            tryCreateTableIfAbsent(dorisSystem, targetDb, dorisTable, schema);
+            DorisTableUtil.tryCreateTableIfAbsent(
+                    dorisSystem,
+                    targetDb,
+                    dorisTable,
+                    schema,
+                    dorisTableConfig,
+                    ignoreIncompatible);
 
             if (!dorisTables.contains(Tuple2.of(targetDb, dorisTable))) {
                 dorisTables.add(Tuple2.of(targetDb, dorisTable));
@@ -470,40 +474,6 @@ public void setTableSchemaBuckets(
         }
     }
 
-    private void tryCreateTableIfAbsent(
-            DorisSystem dorisSystem, String targetDb, String dorisTable, SourceSchema schema) {
-        if (!dorisSystem.tableExists(targetDb, dorisTable)) {
-            if (dorisTableConfig.isConvertUniqToPk()
-                    && CollectionUtil.isNullOrEmpty(schema.primaryKeys)
-                    && !CollectionUtil.isNullOrEmpty(schema.uniqueIndexs)) {
-                schema.primaryKeys = new ArrayList<>(schema.uniqueIndexs);
-            }
-            TableSchema dorisSchema =
-                    DorisSchemaFactory.createTableSchema(
-                            targetDb,
-                            dorisTable,
-                            schema.getFields(),
-                            schema.getPrimaryKeys(),
-                            dorisTableConfig,
-                            schema.getTableComment());
-            try {
-                dorisSystem.createTable(dorisSchema);
-            } catch (Exception ex) {
-                handleTableCreationFailure(ex);
-            }
-        }
-    }
-
-    private void handleTableCreationFailure(Exception ex) throws DorisSystemException {
-        if (ignoreIncompatible && ex.getCause() instanceof SQLSyntaxErrorException) {
-            LOG.warn(
-                    "Doris schema and source table schema are not compatible. Error: {} ",
-                    ex.getCause().toString());
-        } else {
-            throw new DorisSystemException("Failed to create table due to: ", ex);
-        }
-    }
-
     protected Properties getJdbcProperties() {
         Properties jdbcProps = new Properties();
         for (Map.Entry<String, String> entry : config.toMap().entrySet()) {
diff --git a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBDatabaseSync.java b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBDatabaseSync.java
index ca034ac34..b77ec0e3e 100644
--- a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBDatabaseSync.java
+++ b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBDatabaseSync.java
@@ -240,6 +240,9 @@ public DorisRecordSerializer<String> buildSchemaSerializer(
                 .setTableMapping(tableMapping)
                 .setTableConf(dorisTableConfig)
                 .setTargetDatabase(database)
+                .setTargetTablePrefix(tablePrefix)
+                .setTargetTableSuffix(tableSuffix)
+                .setTableNameConverter(converter)
                 .build();
     }
 
diff --git a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBSchema.java b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBSchema.java
index 984419bcc..2e2788de0 100644
--- a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBSchema.java
+++ b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBSchema.java
@@ -20,6 +20,7 @@
 import org.apache.flink.annotation.VisibleForTesting;
 import org.apache.flink.api.java.tuple.Tuple2;
 
+import com.fasterxml.jackson.databind.JsonNode;
 import org.apache.doris.flink.catalog.doris.DorisType;
 import org.apache.doris.flink.catalog.doris.FieldSchema;
 import org.apache.doris.flink.tools.cdc.SourceSchema;
@@ -64,6 +65,28 @@ public MongoDBSchema(
         primaryKeys.add("_id");
     }
 
+    public MongoDBSchema(
+            JsonNode jsonData, String databaseName, String tableName, String tableComment)
+            throws Exception {
+        super(databaseName, null, tableName, tableComment);
+        fields = new LinkedHashMap<>();
+        processSampleData(jsonData);
+
+        primaryKeys = new ArrayList<>();
+        primaryKeys.add("_id");
+    }
+
+    @VisibleForTesting
+    protected void processSampleData(JsonNode data) {
+        data.fieldNames()
+                .forEachRemaining(
+                        fieldName -> {
+                            JsonNode value = data.get(fieldName);
+                            String dorisType = MongoDBType.jsonNodeToDorisType(value);
+                            fields.put(fieldName, new FieldSchema(fieldName, dorisType, null));
+                        });
+    }
+
     @VisibleForTesting
     protected void processSampleData(Document sampleData) {
         for (Map.Entry<String, Object> entry : sampleData.entrySet()) {
diff --git a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoDBJsonDebeziumSchemaSerializer.java b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoDBJsonDebeziumSchemaSerializer.java
index 296a37277..dec06e91b 100644
--- a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoDBJsonDebeziumSchemaSerializer.java
+++ b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoDBJsonDebeziumSchemaSerializer.java
@@ -29,6 +29,7 @@
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.CdcSchemaChange;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeContext;
 import org.apache.doris.flink.tools.cdc.DorisTableConfig;
+import org.apache.doris.flink.tools.cdc.converter.TableNameConverter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -62,6 +63,7 @@ public class MongoDBJsonDebeziumSchemaSerializer implements DorisRecordSerialize
 
     private String targetTablePrefix;
     private String targetTableSuffix;
+    private TableNameConverter tableNameConverter;
 
     public MongoDBJsonDebeziumSchemaSerializer(
             DorisOptions dorisOptions,
@@ -72,7 +74,8 @@ public MongoDBJsonDebeziumSchemaSerializer(
             DorisTableConfig dorisTableConfig,
             String targetDatabase,
             String targetTablePrefix,
-            String targetTableSuffix) {
+            String targetTableSuffix,
+            TableNameConverter tableNameConverter) {
         this.dorisOptions = dorisOptions;
         this.pattern = pattern;
         this.sourceTableName = sourceTableName;
@@ -85,6 +88,7 @@ public MongoDBJsonDebeziumSchemaSerializer(
         this.targetDatabase = targetDatabase;
         this.targetTablePrefix = targetTablePrefix;
         this.targetTableSuffix = targetTableSuffix;
+        this.tableNameConverter = tableNameConverter;
         if (executionOptions != null) {
             this.lineDelimiter =
                     executionOptions
@@ -111,6 +115,7 @@ private void init() {
                         targetTablePrefix,
                         targetTableSuffix,
                         enableDelete);
+        changeContext.setTableNameConverter(tableNameConverter);
         this.dataChange = new MongoJsonDebeziumDataChange(changeContext);
         this.schemaChange = new MongoJsonDebeziumSchemaChange(changeContext);
     }
@@ -143,6 +148,7 @@ public static class Builder {
         private String targetDatabase;
         private String targetTablePrefix = "";
         private String targetTableSuffix = "";
+        private TableNameConverter tableNameConverter;
 
         public MongoDBJsonDebeziumSchemaSerializer.Builder setDorisOptions(
                 DorisOptions dorisOptions) {
@@ -192,6 +198,24 @@ public MongoDBJsonDebeziumSchemaSerializer.Builder setTargetDatabase(
             return this;
         }
 
+        public MongoDBJsonDebeziumSchemaSerializer.Builder setTargetTablePrefix(
+                String targetTablePrefix) {
+            this.targetTablePrefix = targetTablePrefix;
+            return this;
+        }
+
+        public MongoDBJsonDebeziumSchemaSerializer.Builder setTargetTableSuffix(
+                String targetTableSuffix) {
+            this.targetTableSuffix = targetTableSuffix;
+            return this;
+        }
+
+        public MongoDBJsonDebeziumSchemaSerializer.Builder setTableNameConverter(
+                TableNameConverter converter) {
+            this.tableNameConverter = converter;
+            return this;
+        }
+
         public MongoDBJsonDebeziumSchemaSerializer build() {
             return new MongoDBJsonDebeziumSchemaSerializer(
                     dorisOptions,
@@ -202,7 +226,8 @@ public MongoDBJsonDebeziumSchemaSerializer build() {
                     dorisTableConfig,
                     targetDatabase,
                     targetTablePrefix,
-                    targetTableSuffix);
+                    targetTableSuffix,
+                    tableNameConverter);
         }
     }
 }
diff --git a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumDataChange.java b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumDataChange.java
index 9dbe7ffe1..12fa8581b 100644
--- a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumDataChange.java
+++ b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumDataChange.java
@@ -20,7 +20,6 @@
 import org.apache.flink.util.StringUtils;
 
 import com.fasterxml.jackson.core.JsonProcessingException;
-import com.fasterxml.jackson.core.type.TypeReference;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.node.NullNode;
@@ -30,6 +29,7 @@
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.CdcDataChange;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeContext;
 import org.apache.doris.flink.tools.cdc.SourceSchema;
+import org.apache.doris.flink.tools.cdc.utils.JsonNodeExtractUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -127,16 +127,7 @@ public Map<String, Object> extractBeforeRow(JsonNode record) {
     @Override
     public Map<String, Object> extractAfterRow(JsonNode recordRoot) {
         JsonNode dataNode = recordRoot.get(FIELD_DATA);
-        Map<String, Object> rowMap = extractRow(dataNode);
-        String objectId;
-        // if user specifies the `_id` field manually, the $oid field may not exist
-        if (rowMap.get(ID_FIELD) instanceof Map<?, ?>) {
-            objectId = ((Map<?, ?>) rowMap.get(ID_FIELD)).get(OID_FIELD).toString();
-        } else {
-            objectId = rowMap.get(ID_FIELD).toString();
-        }
-        rowMap.put(ID_FIELD, objectId);
-        return rowMap;
+        return JsonNodeExtractUtil.extractAfterRow(dataNode, objectMapper);
     }
 
     private Map<String, Object> extractDeleteRow(JsonNode recordRoot)
@@ -154,10 +145,4 @@ private Map<String, Object> extractDeleteRow(JsonNode recordRoot)
         row.put(ID_FIELD, objectId);
         return row;
     }
-
-    private Map<String, Object> extractRow(JsonNode recordRow) {
-        Map<String, Object> recordMap =
-                objectMapper.convertValue(recordRow, new TypeReference<Map<String, Object>>() {});
-        return recordMap != null ? recordMap : new HashMap<>();
-    }
 }
diff --git a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java
index c67e856e9..cfcf53f21 100644
--- a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java
+++ b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java
@@ -17,10 +17,14 @@
 
 package org.apache.doris.flink.tools.cdc.mongodb.serializer;
 
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.util.StringUtils;
+
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.node.NullNode;
 import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.doris.flink.catalog.doris.DataModel;
 import org.apache.doris.flink.catalog.doris.DorisSystem;
 import org.apache.doris.flink.catalog.doris.FieldSchema;
 import org.apache.doris.flink.cfg.DorisOptions;
@@ -30,8 +34,11 @@
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.CdcSchemaChange;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeContext;
 import org.apache.doris.flink.tools.cdc.SourceSchema;
+import org.apache.doris.flink.tools.cdc.mongodb.MongoDBSchema;
 import org.apache.doris.flink.tools.cdc.mongodb.MongoDBType;
 import org.apache.doris.flink.tools.cdc.mongodb.MongoDateConverter;
+import org.apache.doris.flink.tools.cdc.utils.DorisTableUtil;
+import org.apache.doris.flink.tools.cdc.utils.JsonNodeExtractUtil;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -60,17 +67,19 @@ public class MongoJsonDebeziumSchemaChange extends CdcSchemaChange {
 
     private final Map<String, Map<String, String>> tableFields;
 
-    private final SchemaChangeManager schemaChangeManager;
+    private SchemaChangeManager schemaChangeManager;
 
-    private final DorisSystem dorisSystem;
+    private DorisSystem dorisSystem;
 
     public Map<String, String> tableMapping;
     private final DorisOptions dorisOptions;
+    public JsonDebeziumChangeContext changeContext;
 
     private final Set<String> specialFields =
             new HashSet<>(Arrays.asList(DATE_FIELD, TIMESTAMP_FIELD, DECIMAL_FIELD, LONG_FIELD));
 
     public MongoJsonDebeziumSchemaChange(JsonDebeziumChangeContext changeContext) {
+        this.changeContext = changeContext;
         this.objectMapper = changeContext.getObjectMapper();
         this.dorisOptions = changeContext.getDorisOptions();
         this.tableFields = new HashMap<>();
@@ -96,6 +105,35 @@ public boolean schemaChange(JsonNode recordRoot) {
             String cdcTableIdentifier = getCdcTableIdentifier(recordRoot);
             String dorisTableIdentifier =
                     getDorisTableIdentifier(cdcTableIdentifier, dorisOptions, tableMapping);
+
+            // if table dorisTableIdentifier is null, create table
+            if (StringUtils.isNullOrWhitespaceOnly(dorisTableIdentifier)) {
+                String[] split = cdcTableIdentifier.split("\\.");
+                String targetDb = changeContext.getTargetDatabase();
+                String sourceTable = split[1];
+                String dorisTable = changeContext.getTableNameConverter().convert(sourceTable);
+                LOG.info(
+                        "The table [{}.{}] does not exist. Attempting to create a new table named: {}.{}",
+                        targetDb,
+                        sourceTable,
+                        targetDb,
+                        dorisTable);
+                tableMapping.put(cdcTableIdentifier, String.format("%s.%s", targetDb, dorisTable));
+                dorisTableIdentifier = tableMapping.get(cdcTableIdentifier);
+                Map<String, Object> stringObjectMap = extractAfterRow(logData);
+                JsonNode jsonNode = objectMapper.valueToTree(stringObjectMap);
+
+                MongoDBSchema mongoSchema = new MongoDBSchema(jsonNode, targetDb, dorisTable, "");
+
+                mongoSchema.setModel(DataModel.UNIQUE);
+                DorisTableUtil.tryCreateTableIfAbsent(
+                        dorisSystem,
+                        targetDb,
+                        dorisTable,
+                        mongoSchema,
+                        changeContext.getDorisTableConf());
+            }
+
             String[] tableInfo = dorisTableIdentifier.split("\\.");
             if (tableInfo.length != 2) {
                 throw new DorisRuntimeException();
@@ -163,6 +201,10 @@ private JsonNode getFullDocument(JsonNode recordRoot) {
         }
     }
 
+    public Map<String, Object> extractAfterRow(JsonNode recordRoot) {
+        return JsonNodeExtractUtil.extractAfterRow(recordRoot, objectMapper);
+    }
+
     private void checkAndUpdateSchemaChange(
             JsonNode logData, String dorisTableIdentifier, String database, String table) {
         Map<String, String> tableFieldMap = tableFields.get(dorisTableIdentifier);
@@ -207,4 +249,14 @@ public String getCdcTableIdentifier(JsonNode record) {
         String db = nameSpace.get(FIELD_DATABASE).asText();
         return SourceSchema.getString(db, null, table);
     }
+
+    @VisibleForTesting
+    public void setDorisSystem(DorisSystem dorisSystem) {
+        this.dorisSystem = dorisSystem;
+    }
+
+    @VisibleForTesting
+    public void setSchemaChangeManager(SchemaChangeManager schemaChangeManager) {
+        this.schemaChangeManager = schemaChangeManager;
+    }
 }
diff --git a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/utils/DorisTableUtil.java b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/utils/DorisTableUtil.java
new file mode 100644
index 000000000..114bf151d
--- /dev/null
+++ b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/utils/DorisTableUtil.java
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.tools.cdc.utils;
+
+import org.apache.flink.util.CollectionUtil;
+
+import org.apache.doris.flink.catalog.doris.DorisSchemaFactory;
+import org.apache.doris.flink.catalog.doris.DorisSystem;
+import org.apache.doris.flink.catalog.doris.TableSchema;
+import org.apache.doris.flink.exception.DorisSystemException;
+import org.apache.doris.flink.tools.cdc.DorisTableConfig;
+import org.apache.doris.flink.tools.cdc.SourceSchema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLSyntaxErrorException;
+import java.util.ArrayList;
+
+/** Utility class for Doris table operations. */
+public class DorisTableUtil {
+    private static final Logger LOG = LoggerFactory.getLogger(DorisTableUtil.class);
+
+    /**
+     * Try to create a table in doris if it doesn't exist.
+     *
+     * @param dorisSystem Doris system instance
+     * @param targetDb Doris database name
+     * @param dorisTable Doris table name
+     * @param schema Doris table schema
+     * @param tableConfig Table configuration
+     * @param ignoreIncompatible Whether to ignore incompatible schema errors
+     * @throws DorisSystemException if table creation fails
+     */
+    public static void tryCreateTableIfAbsent(
+            DorisSystem dorisSystem,
+            String targetDb,
+            String dorisTable,
+            SourceSchema schema,
+            DorisTableConfig tableConfig,
+            boolean ignoreIncompatible)
+            throws DorisSystemException {
+
+        if (!dorisSystem.tableExists(targetDb, dorisTable)) {
+            if (tableConfig.isConvertUniqToPk()
+                    && CollectionUtil.isNullOrEmpty(schema.primaryKeys)
+                    && !CollectionUtil.isNullOrEmpty(schema.uniqueIndexs)) {
+                schema.primaryKeys = new ArrayList<>(schema.uniqueIndexs);
+            }
+
+            TableSchema dorisSchema =
+                    DorisSchemaFactory.createTableSchema(
+                            targetDb,
+                            dorisTable,
+                            schema.getFields(),
+                            schema.getPrimaryKeys(),
+                            tableConfig,
+                            schema.getTableComment());
+            try {
+                dorisSystem.createTable(dorisSchema);
+            } catch (Exception ex) {
+                handleTableCreationFailure(ex, ignoreIncompatible);
+            }
+        }
+    }
+
+    /** Overloaded method without ignoreIncompatible parameter. */
+    public static void tryCreateTableIfAbsent(
+            DorisSystem dorisSystem,
+            String targetDb,
+            String dorisTable,
+            SourceSchema schema,
+            DorisTableConfig tableConfig)
+            throws DorisSystemException {
+        tryCreateTableIfAbsent(dorisSystem, targetDb, dorisTable, schema, tableConfig, false);
+    }
+
+    /**
+     * Handle table creation failure.
+     *
+     * @param ex Exception that occurred during table creation
+     * @param ignoreIncompatible Whether to ignore incompatible schema errors
+     * @throws DorisSystemException if table creation fails and errors should not be ignored
+     */
+    private static void handleTableCreationFailure(Exception ex, boolean ignoreIncompatible)
+            throws DorisSystemException {
+        if (ignoreIncompatible && ex.getCause() instanceof SQLSyntaxErrorException) {
+            LOG.warn(
+                    "Doris schema and source table schema are not compatible. Error: {} ",
+                    ex.getCause().toString());
+        } else {
+            throw new DorisSystemException("Failed to create table due to: ", ex);
+        }
+    }
+}
diff --git a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/utils/JsonNodeExtractUtil.java b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/utils/JsonNodeExtractUtil.java
new file mode 100644
index 000000000..93aeca0b7
--- /dev/null
+++ b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/utils/JsonNodeExtractUtil.java
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.tools.cdc.utils;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.ID_FIELD;
+import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.OID_FIELD;
+
+/** Utility class for extracting data from JSON nodes */
+public class JsonNodeExtractUtil {
+    private static final Logger LOG = LoggerFactory.getLogger(JsonNodeExtractUtil.class);
+
+    /**
+     * Extract row data from JsonNode and convert to Map
+     *
+     * @param recordRow JsonNode containing row data
+     * @param objectMapper ObjectMapper instance for JSON conversion
+     * @return Map containing extracted row data
+     */
+    public static Map<String, Object> extractRow(JsonNode recordRow, ObjectMapper objectMapper) {
+        Map<String, Object> recordMap =
+                objectMapper.convertValue(recordRow, new TypeReference<Map<String, Object>>() {});
+        return recordMap != null ? recordMap : new HashMap<>();
+    }
+
+    public static Map<String, Object> extractAfterRow(
+            JsonNode recordRoot, ObjectMapper objectMapper) {
+        Map<String, Object> rowMap = JsonNodeExtractUtil.extractRow(recordRoot, objectMapper);
+        String objectId;
+        // if user specifies the `_id` field manually, the $oid field may not exist
+        if (rowMap.get(ID_FIELD) instanceof Map<?, ?>) {
+            objectId = ((Map<?, ?>) rowMap.get(ID_FIELD)).get(OID_FIELD).toString();
+        } else {
+            objectId = rowMap.get(ID_FIELD).toString();
+        }
+        rowMap.put(ID_FIELD, objectId);
+        return rowMap;
+    }
+}
diff --git a/flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBCreateTableTest.java b/flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBCreateTableTest.java
new file mode 100644
index 000000000..b395d3a72
--- /dev/null
+++ b/flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBCreateTableTest.java
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.tools.cdc.mongodb;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.doris.flink.catalog.doris.DorisSystem;
+import org.apache.doris.flink.cfg.DorisOptions;
+import org.apache.doris.flink.sink.schema.SchemaChangeManager;
+import org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeContext;
+import org.apache.doris.flink.sink.writer.serializer.jsondebezium.TestJsonDebeziumChangeBase;
+import org.apache.doris.flink.tools.cdc.DorisTableConfig;
+import org.apache.doris.flink.tools.cdc.converter.TableNameConverter;
+import org.apache.doris.flink.tools.cdc.mongodb.serializer.MongoJsonDebeziumSchemaChange;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.mockito.Mock;
+import org.mockito.junit.MockitoJUnit;
+import org.mockito.junit.MockitoRule;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+public class MongoDBCreateTableTest extends TestJsonDebeziumChangeBase {
+
+    private MongoJsonDebeziumSchemaChange schemaChange;
+
+    @Rule public MockitoRule mockitoRule = MockitoJUnit.rule();
+
+    @Mock private DorisSystem mockDorisSystem;
+
+    @Mock private DorisOptions mockDorisOptions;
+
+    @Mock private SchemaChangeManager mockSchemaManager;
+
+    private final String dbName = "test_db";
+    private final String prefix = "ods_";
+    private final String suffix = "_dt";
+
+    @Before
+    public void setUp() {
+        super.setUp();
+
+        Map<String, String> tableConfig = new HashMap<>();
+        tableConfig.put(DorisTableConfig.REPLICATION_NUM, "1");
+        tableConfig.put(DorisTableConfig.TABLE_BUCKETS, ".*:1");
+
+        JsonDebeziumChangeContext changeContext =
+                new JsonDebeziumChangeContext(
+                        mockDorisOptions,
+                        tableMapping,
+                        null,
+                        dbName,
+                        new DorisTableConfig(tableConfig),
+                        objectMapper,
+                        null,
+                        lineDelimiter,
+                        ignoreUpdateBefore,
+                        prefix,
+                        suffix,
+                        true,
+                        new TableNameConverter(prefix, suffix));
+        schemaChange = new MongoJsonDebeziumSchemaChange(changeContext);
+    }
+
+    @Test
+    public void testAutoCreateTable() throws IOException {
+        String newTableName = "test_table";
+        String record =
+                "{"
+                        + "\"_id\":\"{\\\"_id\\\": {\\\"_id\\\": {\\\"$oid\\\": \\\"67d2d13807fe0c4336070cfd\\\"}}}\","
+                        + "\"operationType\":\"insert\","
+                        + "\"fullDocument\":\"{\\\"_id\\\": {\\\"$oid\\\": \\\"67d2d13807fe0c4336070cfd\\\"}, \\\"name\\\": \\\"John Doe\\\", \\\"age\\\": 30, \\\"city\\\": \\\"New York\\\"}\","
+                        + "\"fullDocumentBeforeChange\":null,"
+                        + "\"source\":{\"ts_ms\":1741869368000,\"snapshot\":\"false\"},"
+                        + "\"ts_ms\":1741869368365,"
+                        + "\"ns\":{\"db\":\"testDB\",\"coll\":\""
+                        + newTableName
+                        + "\"},"
+                        + "\"to\":null,"
+                        + "\"documentKey\":\"{\\\"_id\\\": {\\\"$oid\\\": \\\"67d2d13807fe0c4336070cfd\\\"}}\","
+                        + "\"updateDescription\":null,"
+                        + "\"clusterTime\":\"{\\\"$timestamp\\\": {\\\"t\\\": 1741869368, \\\"i\\\": 2}}\","
+                        + "\"txnNumber\":null,"
+                        + "\"lsid\":null"
+                        + "}";
+        schemaChange.setSchemaChangeManager(mockSchemaManager);
+        schemaChange.setDorisSystem(mockDorisSystem);
+
+        JsonNode recordRoot = objectMapper.readTree(record);
+        boolean result = schemaChange.schemaChange(recordRoot);
+        Assert.assertTrue(
+                tableMapping.containsValue(
+                        dbName
+                                + "."
+                                + new TableNameConverter(prefix, suffix).convert(newTableName)));
+        Assert.assertTrue(result);
+    }
+}


Reviews:
user: login: JNSimba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T02:17:32Z
user: login: JNSimba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T02:17:51Z
user: login: kwonder0926
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T02:21:35Z
user: login: JNSimba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T06:18:06Z
user: login: JNSimba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-12T06:37:29Z
user: login: kwonder0926
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T06:45:42Z
user: login: JNSimba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T12:26:37Z
user: login: kwonder0926
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T01:13:19Z
user: login: JNSimba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T10:08:56Z
user: login: JNSimba
body: LGTM
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-27T09:45:50Z

Review comments:
diff_hunk: @@ -17,40 +17,36 @@
 
 package org.apache.doris.flink.tools.cdc.mongodb.serializer;
 
+import org.apache.flink.util.CollectionUtil;
+import org.apache.flink.util.StringUtils;
+
+import com.fasterxml.jackson.core.type.TypeReference;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.node.NullNode;
 import com.fasterxml.jackson.databind.node.ObjectNode;
-import org.apache.doris.flink.catalog.doris.DorisSystem;
-import org.apache.doris.flink.catalog.doris.FieldSchema;
+import org.apache.doris.flink.catalog.doris.*;
 import org.apache.doris.flink.cfg.DorisOptions;
 import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.apache.doris.flink.exception.DorisSystemException;
 import org.apache.doris.flink.exception.IllegalArgumentException;
 import org.apache.doris.flink.sink.schema.SchemaChangeManager;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.CdcSchemaChange;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeContext;
 import org.apache.doris.flink.tools.cdc.SourceSchema;
+import org.apache.doris.flink.tools.cdc.mongodb.MongoDBSchema;
 import org.apache.doris.flink.tools.cdc.mongodb.MongoDBType;
 import org.apache.doris.flink.tools.cdc.mongodb.MongoDateConverter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
+import java.sql.SQLSyntaxErrorException;
+import java.util.*;
path: flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java
user: login: JNSimba
body: not import *
created_at: 2025-03-12T02:17:32Z
author_association: MEMBER
diff_hunk: @@ -17,40 +17,36 @@
 
 package org.apache.doris.flink.tools.cdc.mongodb.serializer;
 
+import org.apache.flink.util.CollectionUtil;
+import org.apache.flink.util.StringUtils;
+
+import com.fasterxml.jackson.core.type.TypeReference;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.node.NullNode;
 import com.fasterxml.jackson.databind.node.ObjectNode;
-import org.apache.doris.flink.catalog.doris.DorisSystem;
-import org.apache.doris.flink.catalog.doris.FieldSchema;
+import org.apache.doris.flink.catalog.doris.*;
 import org.apache.doris.flink.cfg.DorisOptions;
 import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.apache.doris.flink.exception.DorisSystemException;
 import org.apache.doris.flink.exception.IllegalArgumentException;
 import org.apache.doris.flink.sink.schema.SchemaChangeManager;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.CdcSchemaChange;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeContext;
 import org.apache.doris.flink.tools.cdc.SourceSchema;
+import org.apache.doris.flink.tools.cdc.mongodb.MongoDBSchema;
 import org.apache.doris.flink.tools.cdc.mongodb.MongoDBType;
 import org.apache.doris.flink.tools.cdc.mongodb.MongoDateConverter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
+import java.sql.SQLSyntaxErrorException;
+import java.util.*;
 
 import static org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeUtils.getDorisTableIdentifier;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.DATE_FIELD;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.DECIMAL_FIELD;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.FIELD_DATA;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.FIELD_DATABASE;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.FIELD_NAMESPACE;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.FIELD_TABLE;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.LONG_FIELD;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.TIMESTAMP_FIELD;
+import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.*;
path: flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java
user: login: JNSimba
body: same as above
created_at: 2025-03-12T02:17:51Z
author_association: MEMBER
diff_hunk: @@ -17,40 +17,36 @@
 
 package org.apache.doris.flink.tools.cdc.mongodb.serializer;
 
+import org.apache.flink.util.CollectionUtil;
+import org.apache.flink.util.StringUtils;
+
+import com.fasterxml.jackson.core.type.TypeReference;
 import com.fasterxml.jackson.databind.JsonNode;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.node.NullNode;
 import com.fasterxml.jackson.databind.node.ObjectNode;
-import org.apache.doris.flink.catalog.doris.DorisSystem;
-import org.apache.doris.flink.catalog.doris.FieldSchema;
+import org.apache.doris.flink.catalog.doris.*;
 import org.apache.doris.flink.cfg.DorisOptions;
 import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.apache.doris.flink.exception.DorisSystemException;
 import org.apache.doris.flink.exception.IllegalArgumentException;
 import org.apache.doris.flink.sink.schema.SchemaChangeManager;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.CdcSchemaChange;
 import org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeContext;
 import org.apache.doris.flink.tools.cdc.SourceSchema;
+import org.apache.doris.flink.tools.cdc.mongodb.MongoDBSchema;
 import org.apache.doris.flink.tools.cdc.mongodb.MongoDBType;
 import org.apache.doris.flink.tools.cdc.mongodb.MongoDateConverter;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
-import java.util.Set;
+import java.sql.SQLSyntaxErrorException;
+import java.util.*;
 
 import static org.apache.doris.flink.sink.writer.serializer.jsondebezium.JsonDebeziumChangeUtils.getDorisTableIdentifier;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.DATE_FIELD;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.DECIMAL_FIELD;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.FIELD_DATA;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.FIELD_DATABASE;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.FIELD_NAMESPACE;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.FIELD_TABLE;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.LONG_FIELD;
-import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.TIMESTAMP_FIELD;
+import static org.apache.doris.flink.tools.cdc.mongodb.ChangeStreamConstant.*;
path: flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java
user: login: kwonder0926
body: Understood, I will revise the code and resubmit it promptly
created_at: 2025-03-12T02:21:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -64,6 +65,28 @@ public MongoDBSchema(
         primaryKeys.add("_id");
     }
 
+    public MongoDBSchema(
+            JsonNode jsonData, String databaseName, String tableName, String tableComment)
+            throws Exception {
+        super(databaseName, null, tableName, tableComment);
+        fields = new LinkedHashMap<>();
+        processSampleData(jsonData);
+
+        primaryKeys = new ArrayList<>();
+        primaryKeys.add("_id");
+    }
+
+    @VisibleForTesting
+    protected void processSampleData(JsonNode data) {
+        data.fieldNames()
+                .forEachRemaining(
+                        fieldName -> {
+                            JsonNode value = data.get(fieldName);
+                            String dorisType = MongoDBType.jsonNodeToDorisType(value);
+                            fields.put(fieldName, new FieldSchema(fieldName, dorisType, null));
+                        });
+    }
+
path: flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBSchema.java
user: login: JNSimba
body: maybe we can add ut for this like MongoDBSchemaTest
created_at: 2025-03-12T06:18:06Z
author_association: MEMBER
diff_hunk: @@ -118,6 +159,45 @@ public boolean schemaChange(JsonNode recordRoot) {
         }
     }
 
+    public static <K, V> V getRandomValue(Map<K, V> map) {
+        Collection<V> values = map.values();
+        return values.stream().findAny().orElse(null);
+    }
+
+    private void tryCreateTableIfAbsent(
+            DorisSystem dorisSystem, String targetDb, String dorisTable, SourceSchema schema) {
+        if (!dorisSystem.tableExists(targetDb, dorisTable)) {
+            if (changeContext.getDorisTableConf().isConvertUniqToPk()
+                    && CollectionUtil.isNullOrEmpty(schema.primaryKeys)
+                    && !CollectionUtil.isNullOrEmpty(schema.uniqueIndexs)) {
+                schema.primaryKeys = new ArrayList<>(schema.uniqueIndexs);
+            }
+            TableSchema dorisSchema =
+                    DorisSchemaFactory.createTableSchema(
+                            targetDb,
+                            dorisTable,
+                            schema.getFields(),
+                            schema.getPrimaryKeys(),
+                            changeContext.getDorisTableConf(),
+                            schema.getTableComment());
+            try {
+                dorisSystem.createTable(dorisSchema);
+            } catch (Exception ex) {
+                handleTableCreationFailure(ex);
+            }
+        }
+    }
+
+    private void handleTableCreationFailure(Exception ex) throws DorisSystemException {
+        if (ex.getCause() instanceof SQLSyntaxErrorException) {
+            LOG.warn(
+                    "Doris schema and source table schema are not compatible. Error: {} ",
+                    ex.getCause().toString());
+        } else {
+            throw new DorisSystemException("Failed to create table due to: ", ex);
+        }
+    }
+
path: flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java
user: login: JNSimba
body: This looks like duplicate code, maybe we can probably pull it out.
created_at: 2025-03-12T06:37:29Z
author_association: MEMBER
diff_hunk: @@ -118,6 +159,45 @@ public boolean schemaChange(JsonNode recordRoot) {
         }
     }
 
+    public static <K, V> V getRandomValue(Map<K, V> map) {
+        Collection<V> values = map.values();
+        return values.stream().findAny().orElse(null);
+    }
+
+    private void tryCreateTableIfAbsent(
+            DorisSystem dorisSystem, String targetDb, String dorisTable, SourceSchema schema) {
+        if (!dorisSystem.tableExists(targetDb, dorisTable)) {
+            if (changeContext.getDorisTableConf().isConvertUniqToPk()
+                    && CollectionUtil.isNullOrEmpty(schema.primaryKeys)
+                    && !CollectionUtil.isNullOrEmpty(schema.uniqueIndexs)) {
+                schema.primaryKeys = new ArrayList<>(schema.uniqueIndexs);
+            }
+            TableSchema dorisSchema =
+                    DorisSchemaFactory.createTableSchema(
+                            targetDb,
+                            dorisTable,
+                            schema.getFields(),
+                            schema.getPrimaryKeys(),
+                            changeContext.getDorisTableConf(),
+                            schema.getTableComment());
+            try {
+                dorisSystem.createTable(dorisSchema);
+            } catch (Exception ex) {
+                handleTableCreationFailure(ex);
+            }
+        }
+    }
+
+    private void handleTableCreationFailure(Exception ex) throws DorisSystemException {
+        if (ex.getCause() instanceof SQLSyntaxErrorException) {
+            LOG.warn(
+                    "Doris schema and source table schema are not compatible. Error: {} ",
+                    ex.getCause().toString());
+        } else {
+            throw new DorisSystemException("Failed to create table due to: ", ex);
+        }
+    }
+
path: flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mongodb/serializer/MongoJsonDebeziumSchemaChange.java
user: login: kwonder0926
body: My apologies, I assumed both methods were overloads of existing ones and test cases could be omitted, even though I've run the synchronization process locally multiple times. I'll add unit tests shortly.

 This is my first time submitting code, so I hesitated to modify existing code  to avoid disrupting functionality. I'll try to refactor it into a reusable  component later to prevent code duplication.
created_at: 2025-03-12T06:45:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,7 +131,8 @@ public JsonDebeziumSchemaSerializer(
             String targetTablePrefix,
             String targetTableSuffix,
             SchemaChangeMode schemaChangeMode,
-            TableNameConverter tableNameConverter) {
+            TableNameConverter tableNameConverter,
+            boolean ignoreIncompatible) {
path: flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/JsonDebeziumSchemaSerializer.java
user: login: JNSimba
body: When we dynamically create a table, we don't need to perceive ignoreIncompatible in theory.
created_at: 2025-03-17T12:26:37Z
author_association: MEMBER
diff_hunk: @@ -130,7 +131,8 @@ public JsonDebeziumSchemaSerializer(
             String targetTablePrefix,
             String targetTableSuffix,
             SchemaChangeMode schemaChangeMode,
-            TableNameConverter tableNameConverter) {
+            TableNameConverter tableNameConverter,
+            boolean ignoreIncompatible) {
path: flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/JsonDebeziumSchemaSerializer.java
user: login: kwonder0926
body: This is considered because the extracted common method uses this parameter when creating the table during the first synchronization. I will introduce a method overload next, so that compatibility is no longer considered when dynamically creating the table.

created_at: 2025-03-18T01:13:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,99 @@
+package org.apache.doris.flink.tools.cdc.mongodb;
path: flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/mongodb/MongoDBCreateTableTest.java
user: login: JNSimba
body: add license header
created_at: 2025-03-18T10:08:56Z
author_association: MEMBER

PR comments:
user: login: JNSimba
created_at: 2025-03-17T12:28:23Z
body: Thanks @kwonder0926 , maybe you can refer to here https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/JsonDebeziumSchemaChangeImplV2.java#L119-L132 

https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/serializer/jsondebezium/SQLParserSchemaChange.java#L78-L92
author_association: MEMBER
user: login: JNSimba
created_at: 2025-03-24T01:52:50Z
body: ci build fail...
author_association: MEMBER
user: login: kwonder0926
created_at: 2025-03-24T02:54:05Z
body: > ci build fail...

Thank you very much for your patient guidance. I'm not quite sure how to make its build successful. Since the test cases need to connect to Doris to proceed, I don't know how to connect to a Doris that comes with the build system in the test cases (nor am I sure if there is a built-in one).
author_association: CONTRIBUTOR
user: login: kwonder0926
created_at: 2025-03-24T02:55:02Z
body: I will continue to attempt to make it successful later when I have some free time.
author_association: CONTRIBUTOR
user: login: JNSimba
created_at: 2025-03-24T10:10:27Z
body: build ci will run the commands in https://github.com/apache/doris-flink-connector/blob/master/.github/workflows/build-connector.yml . You can try to run it manually locally.
author_association: MEMBER