Project: SeaTunnel (App Definition and Development / Streaming & Messaging)
File: S3File.md

S3 File Source Connector Spark<br/> Flink<br/> SeaTunnel Zeta<br/> Read all the data in a split in a pollNext call. What splits are read will be saved in snapshot. [x] file format type [x] text [x] csv [x] parquet [x] orc [x] json [x] excel [x] xml Read data from aws s3 file system. | Datasource | Supported versions | ||--| | S3 | current | If you use spark/flink, In order to use this connector, You must ensure your spark/flink cluster already integrated hadoop. The tested hadoop version is 2.x.<br/> If you use SeaTunnel Zeta, It automatically integrated the hadoop jar when you download and install SeaTunnel Zeta. You can check the jar package under ${SEATUNNEL_HOME}/lib to confirm this.<br/> To use this connector you need put hadoop-aws-3.1.4.jar and aws-java-sdk-bundle-1.12.692.jar in ${SEATUNNEL_HOME}/lib dir. Data type mapping is related to the type of file being read, We supported as the following file types: `text` `csv` `parquet` `orc` `json` `excel` `xml` If you assign file type to `json`, you should also assign schema option to tell connector how to parse data to the row you want. For example: upstream data is the following: ```json {"code": 200, "data": "get success", "success": true} ``` You can also save multiple pieces of data in one file and split them by newline: ```json lines {"code": 200, "data": "get success", "success": true} {"code": 300, "data": "get failed", "success": false} ``` you should assign schema as the following: ```hocon schema { fields { code = int data = string success = boolean } } ``` connector will generate data as the following: | code | data | success | ||-|| | 200 | get success | true | If you assign file type to `text` `csv`, you can choose to specify the schema information or not. For example, upstream data is the following: ```text tyrantlucifer#26#male ``` If you do not assign data schema connector will treat the upstream data as the following: | content | |--| | tyrantlucifer#26#male | If you assign data schema, you should also assign the option `field_delimiter` too except CSV file type you should assign schema and delimiter as the following: ```hocon field_delimiter = "#" schema { fields { name = string age = int gender = string } } ``` connector will generate data as the following: | name | age | gender | ||--|--| | tyrantlucifer | 26 | male | If you assign file type to `parquet` `orc`, schema option not required, connector can find the schema of upstream data

| Orc Data type | SeaTunnel Data type | |-|-| | BOOLEAN | BOOLEAN | | INT | INT | | BYTE | BYTE | | SHORT | SHORT | | LONG | LONG | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | BINARY | BINARY | | STRING<br/>VARCHAR<br/>CHAR<br/> | STRING | | DATE | LOCALDATETYPE | | TIMESTAMP | LOCALDATETIME_TYPE | | DECIMAL | DECIMAL | | LIST(STRING) | STRINGARRAYTYPE | | LIST(BOOLEAN) | BOOLEANARRAYTYPE | | LIST(TINYINT) | BYTEARRAYTYPE | | LIST(SMALLINT) | SHORTARRAYTYPE | | LIST(INT) | INTARRAYTYPE | | LIST(BIGINT) | LONGARRAYTYPE | | LIST(FLOAT) | FLOATARRAYTYPE | | LIST(DOUBLE) | DOUBLEARRAYTYPE | | Map<K,V> | MapType, This type of K and V will transform to SeaTunnel type | | STRUCT | SeaTunnelRowType | If you assign file type to `parquet` `orc`, schema option not required, connector can find the schema of upstream data automatically. | Orc Data type | SeaTunnel Data type | |-|-| | INT_8 | BYTE | | INT_16 | SHORT | | DATE | DATE | | TIMESTAMP_MILLIS | TIMESTAMP | | INT64 | LONG | | INT96 | TIMESTAMP | | BINARY | BYTES | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | BOOLEAN | BOOLEAN | | FIXEDLENBYTE_ARRAY | TIMESTAMP<br/> DECIMAL | | DECIMAL | DECIMAL | | LIST(STRING) | STRINGARRAYTYPE | | LIST(BOOLEAN) | BOOLEANARRAYTYPE | | LIST(TINYINT) | BYTEARRAYTYPE | | LIST(SMALLINT) | SHORTARRAYTYPE | | LIST(INT) | INTARRAYTYPE | | LIST(BIGINT) | LONGARRAYTYPE | | LIST(FLOAT) | FLOATARRAYTYPE | | LIST(DOUBLE) | DOUBLEARRAYTYPE | | Map<K,V> | MapType, This type of K and V will transform to SeaTunnel type | | STRUCT | SeaTunnelRowType | | name | type | required | default value | Description | |||-|-|| | path | string | yes | - | The s3 path that needs to be read can have sub paths, but the sub paths need to meet certain format requirements. Specific requirements can be referred to "parsepartitionfrom_path" option | | fileformattype | string | yes | - | File type, supported as the following file types: `text` `csv` `parquet` `orc` `json` `excel` `xml` | | bucket | string | yes | - | The bucket address of s3 file system, for example: `s3n://seatunnel-test`, if you use `s3a` protocol, this parameter should be `s3a://seatunnel-test`. | | fs.s3a.endpoint | string | yes | - | fs s3a endpoint | | fs.s3a.aws.credentials.provider | string | yes | com.amazonaws.auth.InstanceProfileCredentialsProvider | The way to authenticate s3a. We only support `org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` and `com.amazonaws.auth.InstanceProfileCredentialsProvider` now. More information about the credential provider you can see | | read_columns | list | no | - | The read column list of the data source, user can use it to implement field projection. The file type supported column projection as the following shown: `text` `csv` `parquet` `orc` `json` `excel` `xml` . If the user wants to use this feature when reading `text` `json` `csv` files, the "schema" option must be configured. | | access_key | string | no | - | Only used when `fs.s3a.aws.credentials.provider = org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider ` | | access_secret | string | no | - | Only used when `fs.s3a.aws.credentials.provider = org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider ` | | hadoops3properties | map | no | - | If you need to add other option, you could add it here and refer to this | | delimiter/field_delimiter | string | no | \001 | Field delimiter, used to tell connector how to slice and dice fields when reading text files. Default `\001`, the same as hive's default delimiter. | | parsepartitionfrom_path | boolean | no | true | Control whether parse the partition keys and values from file path. For example if you read a file from path `s3n://hadoop-cluster/tmp/seatunnel/parquet/name=tyrantlucifer/age=26`. Every record data from file will be added these two fields: name="tyrantlucifer", age=16 | | date_format | string | no | yyyy-MM-dd | Date type format, used to tell connector how to convert string to date, supported as the following formats:`yyyy-MM-dd` `yyyy.MM.dd`

default `yyyy-MM-dd` | | datetime_format | string | no | yyyy-MM-dd HH:mm:ss | Datetime type format, used to tell connector how to convert string to datetime, supported as the following formats:`yyyy-MM-dd HH:mm:ss` `yyyy.MM.dd HH:mm:ss` `yyyy/MM/dd HH:mm:ss` `yyyyMMddHHmmss` | | time_format | string | no | HH:mm:ss | Time type format, used to tell connector how to convert string to time, supported as the following formats:`HH:mm:ss` `HH:mm:ss.SSS` | | skipheaderrownumber | long | no | 0 | Skip the first few lines, but only for the txt and csv. For example, set like following:`skipheaderrownumber = 2`. Then SeaTunnel will skip the first 2 lines from source files | | schema | config | no | - | The schema of upstream data. | | sheetname | string | no | - | Reader the sheet of the workbook,Only used when fileformat is excel. | | xmlrowtag | string | no | - | Specifies the tag name of the data rows within the XML file, only valid for XML files. | | xmluseattr_format | boolean | no | - | Specifies whether to process data using the tag attribute format, only valid for XML files. | | compress_codec | string | no | none | | encoding | string | no | UTF-8 | | common-options | | no | - | Source plugin common parameters, please refer to for details. | delimiter parameter will deprecate after version 2.3.5, please use field_delimiter instead. The compress codec of files and the details that supported as the following shown: txt: `lzo` `none` json: `lzo` `none` csv: `lzo` `none` orc/parquet: automatically recognizes the compression type, no additional settings required. Only used when fileformattype is json,text,csv,xml. The encoding of the file to read. This param will be parsed by `Charset.forName(encoding)`. In this example, We read data from s3 path `s3a://seatunnel-test/seatunnel/text` and the file type is orc in this path. We use `org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` to authentication so `accesskey` and `secretkey` is required. All columns in the file will be read and send to sink. ``` env { parallelism = 1 job.mode = "BATCH" } source { S3File { path = "/seatunnel/text" fs.s3a.endpoint="s3.cn-north-1.amazonaws.com.cn" fs.s3a.aws.credentials.provider = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" access_key = "xxxxxxxxxxxxxxxxx" secret_key = "xxxxxxxxxxxxxxxxx" bucket = "s3a://seatunnel-test" fileformattype = "orc" } } transform { } sink { Console {} } ``` Use `InstanceProfileCredentialsProvider` to authentication The file type in S3 is json, so need config schema option. ```hocon S3File { path = "/seatunnel/json" bucket = "s3a://seatunnel-test" fs.s3a.endpoint="s3.cn-north-1.amazonaws.com.cn" fs.s3a.aws.credentials.provider="com.amazonaws.auth.InstanceProfileCredentialsProvider" fileformattype = "json" schema { fields { id = int name = string } } } ``` Use `InstanceProfileCredentialsProvider` to authentication The file type in S3 is json and has five fields (`id`, `name`, `age`, `sex`, `type`), so need config schema option. In this job, we only need send `id` and `name` column to mysql. ``` env { parallelism = 1 job.mode = "BATCH" } source { S3File { path = "/seatunnel/json" bucket = "s3a://seatunnel-test" fs.s3a.endpoint="s3.cn-north-1.amazonaws.com.cn" fs.s3a.aws.credentials.provider="com.amazonaws.auth.InstanceProfileCredentialsProvider" fileformattype = "json" read_columns = ["id", "name"] schema { fields { id = int name = string age = int sex = int type = string } } } } transform { } sink { Console {} } ``` Add S3File Source Connector ) Allow user to add additional hadoop-s3 parameters Allow the use of the s3a protocol Decouple hadoop-aws dependencies )