Project: SeaTunnel (App Definition and Development / Streaming & Messaging)
File: Http.md

Http source connector Spark<br/> Flink<br/> SeaTunnel Zeta<br/> Used to read data from Http. Supported DataSource Info In order to use the Http connector, the following dependencies are required. They can be downloaded via install-plugin.sh or from the Maven central repository. | Datasource | Supported Versions | Dependency | ||--|| | Http | universal | | | Name | Type | Required | Default | Description | |--||-||--| | url | String | Yes | - | Http request url. | | schema | Config | No | - | Http and seatunnel data structure mapping | | schema.fields | Config | No | - | The schema fields of upstream data | | json_field | Config | No | - | This parameter helps you configure the schema,so this parameter must be used with schema. | | pageing | Config | No | - | This parameter is used for paging queries | | pageing.page_field | String | No | - | This parameter is used to specify the page field name in the request parameter | | pageing.totalpagesize | Int | No | - | This parameter is used to control the total number of pages | | pageing.batch_size | Int | No | - | The batch size returned per request is used to determine whether to continue when the total number of pages is unknown | | contentjson | String | No | - | This parameter can get some json data.If you only need the data in the 'book' section, configure `contentfield = "$.store.book.*"`. | | format | String | No | text | The format of upstream data, now only support `json` `text`, default `text`. | | method | String | No | get | Http request method, only supports GET, POST method. | | headers | Map | No | - | Http headers. | | params | Map | No | - | Http params,the program will automatically add http header application/x-www-form-urlencoded. | | body | String | No | - | Http body,the program will automatically add http header application/json,body is jsonbody. | | pollintervalmillis | Int | No | - | Request http api interval(millis) in stream mode. | | retry | Int | No | - | The max retry times if request http return to `IOException`. | | retrybackoffmultiplier_ms | Int | No | 100 | The retry-backoff times(millis) multiplier if request http failed. | | retrybackoffmax_ms | Int | No | 10000 | The maximum retry-backoff times(millis) if request http failed | | enablemultilines | Boolean | No | false | | | connecttimeoutms | Int | No | 12000 | Connection timeout setting, default 12s. | | sockettimeoutms | Int | No | 60000 | Socket timeout setting, default 60s. | | common-options | | No | - | Source plugin common parameters, please refer to for details | ```hocon env { parallelism = 1

= "BATCH" } source { Http { resulttablename = "http" url = "http://mockserver:1080/example/http" method = "GET" format = "json" schema = { fields { c_map = "map<string, string>" c_array = "array<int>" c_string = string c_boolean = boolean c_tinyint = tinyint c_smallint = smallint c_int = int c_bigint = bigint c_float = float c_double = double c_bytes = bytes c_date = date c_decimal = "decimal(38, 18)" c_timestamp = timestamp c_row = { C_MAP = "map<string, string>" C_ARRAY = "array<int>" C_STRING = string C_BOOLEAN = boolean C_TINYINT = tinyint C_SMALLINT = smallint C_INT = int C_BIGINT = bigint C_FLOAT = float C_DOUBLE = double C_BYTES = bytes C_DATE = date C_DECIMAL = "decimal(38, 18)" C_TIMESTAMP = timestamp } } } } } sink { Console { parallelism = 1 } } ``` when you assign format is `json`, you should also assign schema option, for example: upstream data is the following: ```json { "code": 200, "data": "get success", "success": true } ``` you should assign schema as the following: ```hocon schema { fields { code = int data = string success = boolean } } ``` connector will generate data as the following: | code | data | success | ||-|| | 200 | get success | true | when you assign format is `text`, connector will do nothing for upstream data, for example: upstream data is the following: ```json { "code": 200, "data": "get success", "success": true } ``` connector will generate data as the following: | content | |-| | {"code": 200, "data": "get success", "success": true} | This parameter can get some json data.If you only need the data in the 'book' section, configure `content_field = "$.store.book.*"`. If your return data looks something like this. ```json { "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99 } ], "bicycle": { "color": "red", "price": 19.95 } }, "expensive": 10 } ``` You can configure `content_field = "$.store.book.*"` and the result returned looks like this: ```json [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99 } ] ``` Then you can get the desired result with a simpler schema,like ```hocon Http { url = "http://mockserver:1080/contentjson/mock" method = "GET" format = "json" content_field = "$.store.book.*" schema = { fields { category = string author = string title = string price = string } } } ``` Here is an example: Test data can be found at this link See this link for task configuration . This parameter helps you configure the schema,so this parameter must be used with schema. If your data looks something like this: ```json { "store": { "book": [ { "category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95 }, { "category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99 } ], "bicycle": { "color": "red", "price": 19.95 } }, "expensive": 10 } ``` You can get the contents of 'book' by configuring the task as follows: ```hocon source { Http { url = "http://mockserver:1080/jsonpath/mock" method = "GET" format = "json" json_field = { category = "$.store.book[*].category" author = "$.store.book[*].author" title = "$.store.book[*].title" price = "$.store.book[*].price" } schema = { fields { category = string author = string title = string price = string } } } } ``` Test data can be found at this link See this link for task configuration . ```hocon source { Http { url = "http://localhost:8080/mock/queryData" method = "GET"