Project: Flink (App Definition and Development / Streaming & Messaging)
File: intro_to_table_api.md

title: "Intro to the Python Table API" weight: 26 type: docs aliases: /dev/python/table-api-users-guide/intrototable_api.html <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> This document is a short introduction to the PyFlink Table API, which is used to help novice users quickly understand the basic usage of PyFlink Table API. For advanced usage, please refer to other documents in this user guide. Common Structure of Python Table API Program -- All Table API and SQL programs, both batch and streaming, follow the same pattern. The following code example shows the common structure of Table API and SQL programs. ```python from pyflink.table import EnvironmentSettings, TableEnvironment from pyflink.table.expressions import col envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) tableenv.executesql(""" CREATE TABLE datagen ( id INT, data STRING ) WITH ( 'connector' = 'datagen', 'fields.id.kind' = 'sequence', 'fields.id.start' = '1', 'fields.id.end' = '10' ) """) tableenv.executesql(""" CREATE TABLE print ( id INT, data STRING ) WITH ( 'connector' = 'print' ) """) sourcetable = tableenv.from_path("datagen") resulttable = sourcetable.select(col("id") + 1, col("data")) resulttable.executeinsert("print").wait() ``` {{< top >}} Create a TableEnvironment `TableEnvironment` is a central concept of the Table API and SQL integration. The following code example shows how to create a TableEnvironment: ```python from pyflink.table import EnvironmentSettings, TableEnvironment envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) envsettings = EnvironmentSettings.inbatch_mode() tableenv = TableEnvironment.create(envsettings) ``` For more details about the different ways to create a `TableEnvironment`, please refer to the . `TableEnvironment` is responsible for: `Table` management: , listing Tables, , etc. User-defined function management: User-defined function registration, dropping, listing, etc. See and for more details about Python user-defined functions. Executing queries: See for more details. Job configuration: See for more details. Python dependency management: See for more details. Job submission: See for more details. {{< top >}} Create Tables `Table` is a core component of the Python Table API. A `Table` object describes a pipeline of data transformations. It does not contain the data itself in any way. Instead, it describes how to read data from a table source, and how to eventually write data to a table sink. The declared pipeline can be printed, optimized, and eventually executed in a cluster. The pipeline can work with bounded or unbounded streams which enables both streaming and batch scenarios. A `Table` is always bound to a specific `TableEnvironment`. It is not possible to combine tables from different TableEnvironments in same query, e.g., to join or union them. You can create a Table from a list object, this is usually used when writing examples or unit tests. ```python from pyflink.table import EnvironmentSettings, TableEnvironment envsettings = EnvironmentSettings.inbatch_mode() tableenv = TableEnvironment.create(envsettings) table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')])

``` The results are as following: ```text +-+--+ | 1 | 2 | +-+--+ | 1 | Hi | | 2 | Hello | +-+--+ ``` You can also create a Table with specified column names: ```python table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) table.execute().print() ``` The results are as following: ```text +-+--+ | id | data | +-+--+ | 1 | Hi | | 2 | Hello | +-+--+ ``` By default, the table schema is extracted from the data automatically. If the automatically generated table schema isn't as expected, you can also specify it manually: ```python table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) print('By default the type of the "id" column is %s.' % table.getschema().getfielddatatype("id")) from pyflink.table import DataTypes table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], DataTypes.ROW([DataTypes.FIELD("id", DataTypes.TINYINT()), DataTypes.FIELD("data", DataTypes.STRING())])) print('Now the type of the "id" column is %s.' % table.getschema().getfielddatatype("id")) ``` The results are as following: ```text By default the type of the "id" column is BIGINT. Now the type of the "id" column is TINYINT. ``` You can also create a Table using SQL DDL statements. It represents a Table which reads data from the specified external storage. ```python from pyflink.table import EnvironmentSettings, TableEnvironment envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) tableenv.executesql(""" CREATE TABLE random_source ( id BIGINT, data TINYINT ) WITH ( 'connector' = 'datagen', 'fields.id.kind'='sequence', 'fields.id.start'='1', 'fields.id.end'='3', 'fields.data.kind'='sequence', 'fields.data.start'='4', 'fields.data.end'='6' ) """) table = tableenv.frompath("random_source") table.execute().print() ``` The results are as following: ```text +-+-+--+ | op | id | data | +-+-+--+ | +I | 1 | 4 | | +I | 2 | 5 | | +I | 3 | 6 | +-+-+--+ ``` TableDescriptor is another way to define a Table. It's equivalent to SQL DDL statements. ```python from pyflink.table import EnvironmentSettings, TableEnvironment, TableDescriptor, Schema, DataTypes envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) tableenv.createtemporary_table( 'random_source', TableDescriptor.for_connector('datagen') .schema(Schema.new_builder() .column('id', DataTypes.BIGINT()) .column('data', DataTypes.TINYINT()) .build()) .option('fields.id.kind', 'sequence') .option('fields.id.start', '1') .option('fields.id.end', '3') .option('fields.data.kind', 'sequence') .option('fields.data.start', '4') .option('fields.data.end', '6') .build()) table = tableenv.frompath("random_source") table.execute().print() ``` The results are as following: ```text +-+-+--+ | op | id | data | +-+-+--+ | +I | 1 | 4 | | +I | 2 | 5 | | +I | 3 | 6 | +-+-+--+ ``` `TableEnvironment` maintains a map of catalogs of tables which are created with an identifier. The tables in a catalog may either be temporary, and tied to the lifecycle of a single Flink session, or permanent, and visible across multiple Flink sessions. The tables and views created via SQL DDL, e.g. "create table ..." and "create view ..." are also stored in a catalog. You can directly access the tables in a catalog via SQL. If you want to use tables from a catalog with the Table API, you can use the "from_path" method to create the Table API objects: ```python table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) tableenv.createtemporaryview('sourcetable', table) newtable = tableenv.frompath('sourcetable') new_table.execute().print() ``` The results are as following: ```text +-+-+--+ | op | id | data | +-+-+--+ | +I | 1 | Hi | | +I | 2 | Hello | +-+-+--+ ``` {{< top >}} Write Queries The `Table` object offers many methods for applying relational operations. These methods return new `Table` objects representing the result of applying the relational operations on the input `Table`. These relational operations may be composed of multiple method calls, such as

The documentation describes all Table API operations that are supported on streaming and batch tables. The following example shows a simple Table API aggregation query: ```python from pyflink.table import EnvironmentSettings, TableEnvironment from pyflink.table.expressions import call, col envsettings = EnvironmentSettings.inbatch_mode() tableenv = TableEnvironment.create(envsettings) orders = tableenv.fromelements([('Jack', 'FRANCE', 10), ('Rose', 'ENGLAND', 30), ('Jack', 'FRANCE', 20)], ['name', 'country', 'revenue']) revenue = orders \ .select(col("name"), col("country"), col("revenue")) \ .where(col("country") == 'FRANCE') \ .group_by(col("name")) \ .select(col("name"), call("sum", col("revenue")).alias('rev_sum')) revenue.execute().print() ``` The results are as following: ```text +--+-+ | name | rev_sum | +--+-+ | Jack | 30 | +--+-+ ``` The are also supported in Python Table API, which include , , and . The following example shows a simple row-based operation query: ```python from pyflink.table import EnvironmentSettings, TableEnvironment from pyflink.table import DataTypes from pyflink.table.udf import udf import pandas as pd envsettings = EnvironmentSettings.inbatch_mode() tableenv = TableEnvironment.create(envsettings) orders = tableenv.fromelements([('Jack', 'FRANCE', 10), ('Rose', 'ENGLAND', 30), ('Jack', 'FRANCE', 20)], ['name', 'country', 'revenue']) map_function = udf(lambda x: pd.concat([x.name, x.revenue * 10], axis=1), result_type=DataTypes.ROW( [DataTypes.FIELD("name", DataTypes.STRING()), DataTypes.FIELD("revenue", DataTypes.BIGINT())]), func_type="pandas") orders.map(map_function).execute().print() ``` The results are as following: ```text +--+-+ | name | revenue | +--+-+ | Jack | 100 | | Rose | 300 | | Jack | 200 | +--+-+ ``` Flink's SQL integration is based on , which implements the SQL standard. SQL queries are specified as Strings. The documentation describes Flink's SQL support for streaming and batch tables. The following example shows a simple SQL aggregation query: ```python from pyflink.table import EnvironmentSettings, TableEnvironment envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) tableenv.executesql(""" CREATE TABLE random_source ( id BIGINT, data TINYINT ) WITH ( 'connector' = 'datagen', 'fields.id.kind'='sequence', 'fields.id.start'='1', 'fields.id.end'='8', 'fields.data.kind'='sequence', 'fields.data.start'='4', 'fields.data.end'='11' ) """) tableenv.executesql(""" CREATE TABLE print_sink ( id BIGINT, data_sum TINYINT ) WITH ( 'connector' = 'print' ) """) tableenv.executesql(""" INSERT INTO print_sink SELECT id, sum(data) as data_sum FROM (SELECT id / 2 as id, data FROM random_source) WHERE id > 1 GROUP BY id """).wait() ``` The results are as following: ```text 2> +I(4,11) 6> +I(2,8) 8> +I(3,10) 6> -U(2,8) 8> -U(3,10) 6> +U(2,15) 8> +U(3,19) ``` In fact, this shows the change logs received by the print sink. The output format of a change log is: ```text {subtask id}> {message type}{string format of the value} ``` For example, "2> +I(4,11)" means this message comes from the 2nd subtask, and "+I" means it is an insert message. "(4, 11)" is the content of the message. In addition, "-U" means a retract record (i.e. update-before), which means this message should be deleted or retracted from the sink. "+U" means this is an update record (i.e. update-after), which means this message should be updated or inserted by the sink. So, we get this result from the change logs above: ```text (4, 11) (2, 15) (3, 19) ``` The `Table` objects used in Table API and the tables used in SQL can be freely converted to each other. The following example shows how to use a `Table` object in SQL: ```python tableenv.executesql(""" CREATE TABLE table_sink ( id BIGINT, data VARCHAR ) WITH ( 'connector' = 'print' ) """) table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) tableenv.createtemporaryview('tableapi_table', table) tableenv.executesql("INSERT INTO tablesink SELECT * FROM tableapi_table").wait() ``` The results are as following: ```text 6> +I(1,Hi) 6> +I(2,Hello) ``` And the following example shows how to use SQL tables in the Table API: ```python

CREATE TABLE sql_source ( id BIGINT, data TINYINT ) WITH ( 'connector' = 'datagen', 'fields.id.kind'='sequence', 'fields.id.start'='1', 'fields.id.end'='4', 'fields.data.kind'='sequence', 'fields.data.start'='4', 'fields.data.end'='7' ) """) table = tableenv.frompath("sql_source") table.execute().print() ``` The results are as following: ```text +-+-+--+ | op | id | data | +-+-+--+ | +I | 1 | 4 | | +I | 2 | 5 | | +I | 3 | 6 | | +I | 4 | 7 | +-+-+--+ ``` {{< top >}} Emit Results You can call the `TableResult.print` method to print the content of the Table to console. This is usually used when you want to preview the table. ```python source = tableenv.fromelements([(1, "Hi", "Hello"), (2, "Hello", "Hello")], ["a", "b", "c"]) tableresult = tableenv.execute_sql("select a + 1, b, c from %s" % source) table_result.print() ``` The results are as following: ```text +-+-+--+--+ | op | EXPR$0 | b | c | +-+-+--+--+ | +I | 2 | Hi | Hello | | +I | 3 | Hello | Hello | +-+-+--+--+ ``` <span class="label label-info">Note</span> It will trigger the materialization of the table and collect table content to the memory of the client, it's a good practice to limit the number of rows collected via {{< pythondoc file="pyflink.table.html#pyflink.table.Table.limit" name="Table.limit">}}. You can call the `TableResult.collect` method to collect results of a table to client. The type of the results is an auto closeable iterator. The following code shows how to use the `TableResult.collect()` method ```python source = tableenv.fromelements([(1, "Hi", "Hello"), (2, "Hello", "Hello")], ["a", "b", "c"]) tableresult = tableenv.execute_sql("select a + 1, b, c from %s" % source) with table_result.collect() as results: for result in results: print(result) ``` The results are as following ```text <Row(2, 'Hi', 'Hello')> <Row(3, 'Hello', 'Hello')> ``` <span class="label label-info">Note</span> It will trigger the materialization of the table and collect table content to the memory of the client, it's a good practice to limit the number of rows collected via {{< pythondoc file="pyflink.table.html#pyflink.table.Table.limit" name="Table.limit">}}. You can call the "to_pandas" method to : ```python table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) print(table.to_pandas()) ``` The results are as following: ```text id data 0 1 Hi 1 2 Hello ``` <span class="label label-info">Note</span> It will trigger the materialization of the table and collect table content to the memory of the client, it's a good practice to limit the number of rows collected via {{< pythondoc file="pyflink.table.html#pyflink.table.Table.limit" name="Table.limit">}}. <span class="label label-info">Note</span> Not all the data types are supported. You can call the "execute_insert" method to emit the data in a `Table` object to a sink table: ```python tableenv.executesql(""" CREATE TABLE sink_table ( id BIGINT, data VARCHAR ) WITH ( 'connector' = 'print' ) """) table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) table.executeinsert("sinktable").wait() ``` The results are as following: ```text 6> +I(1,Hi) 6> +I(2,Hello) ``` This could also be done using SQL: ```python tableenv.createtemporaryview("tablesource", table) tableenv.executesql("INSERT INTO sinktable SELECT * FROM tablesource").wait() ``` You can use a `StatementSet` to emit the `Table`s to multiple sink tables in one job: ```python table = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) tableenv.createtemporaryview("simplesource", table) tableenv.executesql(""" CREATE TABLE firstsinktable ( id BIGINT, data VARCHAR ) WITH ( 'connector' = 'print' ) """) tableenv.executesql(""" CREATE TABLE secondsinktable ( id BIGINT, data VARCHAR ) WITH ( 'connector' = 'print' ) """) statementset = tableenv.createstatementset() statementset.addinsert("firstsinktable", table) statementset.addinsertsql("INSERT INTO secondsinktable SELECT * FROM simplesource")

``` The results are as following: ```text 7> +I(1,Hi) 7> +I(1,Hi) 7> +I(2,Hello) 7> +I(2,Hello) ``` Explain Tables -- The Table API provides a mechanism to explain the logical and optimized query plans used to compute a `Table`. This is done through the `Table.explain()` or `StatementSet.explain()` methods. `Table.explain()` returns the plan of a `Table`. `StatementSet.explain()` is used to get the plan for a job which contains multiple sinks. These methods return a string describing three things: the Abstract Syntax Tree of the relational query, i.e., the unoptimized logical query plan, the optimized logical query plan, and the physical execution plan. `TableEnvironment.explainsql()` and `TableEnvironment.executesql()` support executing an `EXPLAIN` statement to get the plans. Please refer to the page for more details. The following code shows how to use the `Table.explain()` method: ```python from pyflink.table import EnvironmentSettings, TableEnvironment from pyflink.table.expressions import col envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) table1 = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) table2 = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) table = table1 \ .where(col("data").like('H%')) \ .union_all(table2) print(table.explain()) ``` The results are as following: ```text == Abstract Syntax Tree == LogicalUnion(all=[true]) :- LogicalFilter(condition=[LIKE($1, _UTF-16LE'H%')]) : +- LogicalTableScan(table=[[defaultcatalog, defaultdatabase, UnregisteredTableSource201907291, source: [PythonInputFormatTableSource(id, data)]]]) +- LogicalTableScan(table=[[defaultcatalog, defaultdatabase, UnregisteredTableSource1709623525, source: [PythonInputFormatTableSource(id, data)]]]) == Optimized Logical Plan == Union(all=[true], union=[id, data]) :- Calc(select=[id, data], where=[LIKE(data, _UTF-16LE'H%')]) : +- LegacyTableSourceScan(table=[[defaultcatalog, defaultdatabase, UnregisteredTableSource201907291, source: [PythonInputFormatTableSource(id, data)]]], fields=[id, data]) +- LegacyTableSourceScan(table=[[defaultcatalog, defaultdatabase, UnregisteredTableSource1709623525, source: [PythonInputFormatTableSource(id, data)]]], fields=[id, data]) == Physical Execution Plan == Stage 133 : Data Source content : Source: PythonInputFormatTableSource(id, data) Stage 134 : Operator content : SourceConversion(table=[defaultcatalog.defaultdatabase.UnregisteredTableSource201907291, source: [PythonInputFormatTableSource(id, data)]], fields=[id, data]) ship_strategy : FORWARD Stage 135 : Operator content : Calc(select=[id, data], where=[(data LIKE _UTF-16LE'H%')]) ship_strategy : FORWARD Stage 136 : Data Source content : Source: PythonInputFormatTableSource(id, data) Stage 137 : Operator content : SourceConversion(table=[defaultcatalog.defaultdatabase.UnregisteredTableSource1709623525, source: [PythonInputFormatTableSource(id, data)]], fields=[id, data]) ship_strategy : FORWARD ``` The following code shows how to use the `StatementSet.explain()` method: ```python from pyflink.table import EnvironmentSettings, TableEnvironment from pyflink.table.expressions import col envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) table1 = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) table2 = tableenv.fromelements([(1, 'Hi'), (2, 'Hello')], ['id', 'data']) tableenv.executesql(""" CREATE TABLE printsinktable ( id BIGINT, data VARCHAR ) WITH ( 'connector' = 'print' ) """) tableenv.executesql(""" CREATE TABLE blackholesink_table ( id BIGINT, data VARCHAR ) WITH ( 'connector' = 'blackhole' ) """) statementset = tableenv.createstatementset() statementset.addinsert("printsinktable", table1.where(col("data").like('H%'))) statementset.addinsert("blackholesink_table", table2) print(statement_set.explain()) ``` The results are as following: ```text == Abstract Syntax Tree == LogicalSink(table=[defaultcatalog.defaultdatabase.printsinktable], fields=[id, data]) +- LogicalFilter(condition=[LIKE($1, _UTF-16LE'H%')]) +- LogicalTableScan(table=[[defaultcatalog, defaultdatabase, UnregisteredTableSource541737614, source: [PythonInputFormatTableSource(id, data)]]]) LogicalSink(table=[defaultcatalog.defaultdatabase.blackholesink_table], fields=[id, data]) +- LogicalTableScan(table=[[defaultcatalog, defaultdatabase, UnregisteredTableSource1437429083, source: [PythonInputFormatTableSource(id, data)]]]) == Optimized Logical Plan == Sink(table=[defaultcatalog.defaultdatabase.printsinktable], fields=[id, data]) +- Calc(select=[id, data], where=[LIKE(data, _UTF-16LE'H%')]) +- LegacyTableSourceScan(table=[[defaultcatalog, defaultdatabase, UnregisteredTableSource541737614, source: [PythonInputFormatTableSource(id, data)]]], fields=[id, data]) Sink(table=[defaultcatalog.defaultdatabase.blackholesink_table], fields=[id, data]) +- LegacyTableSourceScan(table=[[defaultcatalog, defaultdatabase, UnregisteredTableSource1437429083, source: [PythonInputFormatTableSource(id, data)]]], fields=[id, data]) == Physical Execution Plan == Stage 139 : Data Source content : Source: PythonInputFormatTableSource(id, data) Stage 140 : Operator content : SourceConversion(table=[defaultcatalog.defaultdatabase.UnregisteredTableSource541737614, source: [PythonInputFormatTableSource(id, data)]], fields=[id, data]) ship_strategy : FORWARD Stage 141 : Operator content : Calc(select=[id, data], where=[(data LIKE _UTF-16LE'H%')]) ship_strategy : FORWARD Stage 143 : Data Source content : Source: PythonInputFormatTableSource(id, data) Stage 144 : Operator content : SourceConversion(table=[defaultcatalog.defaultdatabase.UnregisteredTableSource1437429083, source: [PythonInputFormatTableSource(id, data)]], fields=[id, data]) ship_strategy : FORWARD Stage 142 : Data Sink content : Sink: Sink(table=[defaultcatalog.defaultdatabase.printsinktable], fields=[id, data]) ship_strategy : FORWARD Stage 145 : Data Sink content : Sink: Sink(table=[defaultcatalog.defaultdatabase.blackholesink_table], fields=[id, data]) ship_strategy : FORWARD ```