Project: SeaTunnel (App Definition and Development / Streaming & Messaging)
File: Iceberg.md

Apache Iceberg source connector 1.4.2 Spark<br/> Flink<br/> SeaTunnel Zeta<br/> [x] data format [x] parquet [x] orc [x] avro [x] iceberg catalog [x] hadoop(2.7.1 , 2.7.5 , 3.1.3) [x] hive(2.3.9 , 3.1.2) Source connector for Apache Iceberg. It can support batch and stream mode. | Datasource | Dependent | Maven | ||--|| | Iceberg | hive-exec | | | Iceberg | libfb303 | | In order to be compatible with different versions of Hadoop and Hive, the scope of hive-exec in the project pom file are provided, so if you use the Flink engine, first you may need to add the following Jar packages to <FLINK_HOME>/lib directory, if you are using the Spark engine and integrated with Hadoop, then you do not need to add the following Jar packages. ``` hive-exec-xxx.jar libfb303-xxx.jar ``` Some versions of the hive-exec package do not have libfb303-xxx.jar, so you also need to manually import the Jar package. | Iceberg Data type | SeaTunnel Data type | |-|| | BOOLEAN | BOOLEAN | | INTEGER | INT | | LONG | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DATE | DATE | | TIME | TIME | | TIMESTAMP | TIMESTAMP | | STRING | STRING | | FIXED<br/>BINARY | BYTES | | DECIMAL | DECIMAL | | STRUCT | ROW | | LIST | ARRAY | | MAP | MAP | | Name | Type | Required | Default | Description | |--||-|-|| | catalog_name | string | yes | - | User-specified catalog name. | | namespace | string | yes | - | The iceberg database name in the backend catalog. | | table | string | yes | - | The iceberg table name in the backend catalog. | | iceberg.catalog.config | map | yes | - | Specify the properties for initializing the Iceberg catalog, which can be referenced in this file:"https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/CatalogProperties.java" | | hadoop.config | map | no | - | Properties passed through to the Hadoop configuration | | iceberg.hadoop-conf-path | string | no | - | The specified loading paths for the 'core-site.xml', 'hdfs-site.xml', 'hive-site.xml' files. | | schema | config | no | - | Use projection to select data columns and columns order. | | case_sensitive | boolean | no | false | If data columns where selected via schema [config], controls whether the match to the schema will be done with case

| | startsnapshottimestamp | long | no | - | Instructs this scan to look for changes starting from the most recent snapshot for the table as of the timestamp. <br/>timestamp the timestamp in millis since the Unix epoch | | startsnapshotid | long | no | - | Instructs this scan to look for changes starting from a particular snapshot (exclusive). | | endsnapshotid | long | no | - | Instructs this scan to look for changes up to a particular snapshot (inclusive). | | usesnapshotid | long | no | - | Instructs this scan to look for use the given snapshot ID. | | usesnapshottimestamp | long | no | - | Instructs this scan to look for use the most recent snapshot as of the given time in milliseconds. timestamp the timestamp in millis since the Unix epoch | | streamscanstrategy | enum | no | FROMLATESTSNAPSHOT | Starting strategy for stream mode execution, Default to use `FROMLATESTSNAPSHOT` if dont specify any value,The optional values are:<br/>TABLESCANTHENINCREMENTAL: Do a regular table scan then switch to the incremental mode.<br/>FROMLATESTSNAPSHOT: Start incremental mode from the latest snapshot inclusive.<br/>FROMEARLIESTSNAPSHOT: Start incremental mode from the earliest snapshot inclusive.<br/>FROMSNAPSHOTID: Start incremental mode from a snapshot with a specific id inclusive.<br/>FROMSNAPSHOT_TIMESTAMP: Start incremental mode from a snapshot with a specific timestamp inclusive. | | common-options | | no | - | Source plugin common parameters, please refer to for details. | ```hocon env { parallelism = 2 job.mode = "BATCH" } source { Iceberg { schema { fields { f2 = "boolean" f1 = "bigint" f3 = "int" f4 = "bigint" f5 = "float" f6 = "double" f7 = "date" f9 = "timestamp" f10 = "timestamp" f11 = "string" f12 = "bytes" f13 = "bytes" f14 = "decimal(19,9)" f15 = "array<int>" f16 = "map<string, int>" } } catalog_name = "seatunnel" iceberg.catalog.config={ type = "hadoop" warehouse = "file:///tmp/seatunnel/iceberg/hadoop/" } namespace = "database1" table = "source" resulttablename = "iceberg" } } transform { } sink { Console { sourcetablename = "iceberg" } } ``` ```hocon source { Iceberg { catalog_name = "seatunnel" iceberg.catalog.config={ type = "hive" uri = "thrift://localhost:9083" warehouse = "hdfs://your_cluster//tmp/seatunnel/iceberg/" } catalog_type = "hive" namespace = "youricebergdatabase" table = "youricebergtable" } } ``` ```hocon source { Iceberg { catalog_name = "seatunnel" iceberg.catalog.config={ type = "hadoop" warehouse = "hdfs://your_cluster/tmp/seatunnel/iceberg/" } namespace = "youricebergdatabase" table = "youricebergtable" schema { fields { f2 = "boolean" f1 = "bigint" f3 = "int" f4 = "bigint" } } } } ``` Add Iceberg Source Connector ) )