Project: Flink (App Definition and Development / Streaming & Messaging)
File: intro_to_datastream_api.md

title: "Intro to the Python DataStream API" weight: 1 type: docs <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., filtering, updating state, defining windows, aggregating). The data streams are initially created from various sources (e.g., message queues, socket streams, files). Results are returned via sinks, which may for example write the data to files, or to standard output (for example the command line terminal). Python DataStream API is a Python version of DataStream API which allows Python users could write Python DatStream API jobs. Common Structure of Python DataStream API Programs -- The following code example shows the common structure of Python DataStream API programs. ```python from pyflink.common import WatermarkStrategy, Row from pyflink.common.serialization import Encoder from pyflink.common.typeinfo import Types from pyflink.datastream import StreamExecutionEnvironment from pyflink.datastream.connectors.file_system import FileSink, OutputFileConfig from pyflink.datastream.connectors.number_seq import NumberSequenceSource from pyflink.datastream.functions import RuntimeContext, MapFunction from pyflink.datastream.state import ValueStateDescriptor class MyMapFunction(MapFunction): def open(self, runtime_context: RuntimeContext): statedesc = ValueStateDescriptor('cnt', Types.PICKLEDBYTE_ARRAY()) self.cntstate = runtimecontext.getstate(statedesc) def map(self, value): cnt = self.cnt_state.value() if cnt is None or cnt < 2: self.cnt_state.update(1 if cnt is None else cnt + 1) return value[0], value[1] + 1 else: return value[0], value[1] def stateaccessdemo(): env = StreamExecutionEnvironment.getexecutionenvironment() seqnumsource = NumberSequenceSource(1, 10000) ds = env.from_source( source=seqnumsource, watermarkstrategy=WatermarkStrategy.formonotonous_timestamps(), sourcename='seqnum_source', type_info=Types.LONG()) ds = ds.map(lambda a: Row(a % 4, 1), output_type=Types.ROW([Types.LONG(), Types.LONG()])) \ .key_by(lambda a: a[0]) \ .map(MyMapFunction(), output_type=Types.TUPLE([Types.LONG(), Types.LONG()])) output_path = '/opt/output/' file_sink = FileSink \ .forrowformat(outputpath, Encoder.simplestring_encoder()) \ .withoutputfileconfig(OutputFileConfig.builder().withpartprefix('pre').withpart_suffix('suf').build()) \ .build() ds.sinkto(filesink) env.execute('stateaccessdemo') if name == 'main': stateaccessdemo() ``` {{< top >}} Create a StreamExecutionEnvironment The `StreamExecutionEnvironment` is a central concept of the DataStream API program. The following code example shows how to create a `StreamExecutionEnvironment`: ```python from pyflink.datastream import StreamExecutionEnvironment env = StreamExecutionEnvironment.getexecutionenvironment() ``` {{< top >}} Create a DataStream The DataStream API gets its name from the special `DataStream` class that is used to represent a collection of data in a Flink program. You can think of them as immutable collections of data that can contain duplicates. This data can either be finite or unbounded, the API that you use to work on them is the same. A `DataStream` is similar to a regular Python `Collection` in terms of usage but is quite different in some key

They are immutable, meaning that once they are created you cannot add or remove elements. You can also not simply inspect the elements inside but only work on them using the `DataStream` API operations, which are also called transformations. You can create an initial `DataStream` by adding a source in a Flink program. Then you can derive new streams from this and combine them by using API methods such as `map`, `filter`, and so on. You can create a `DataStream` from a list object: ```python from pyflink.common.typeinfo import Types from pyflink.datastream import StreamExecutionEnvironment env = StreamExecutionEnvironment.getexecutionenvironment() ds = env.from_collection( collection=[(1, 'aaa|bb'), (2, 'bb|a'), (3, 'aaa|a')], type_info=Types.ROW([Types.INT(), Types.STRING()])) ``` The parameter `type_info` is optional, if not specified, the output type of the returned `DataStream` will be `Types.PICKLEDBYTEARRAY()`. You can also create a `DataStream` using DataStream connectors with method `add_source` as following: ```python from pyflink.common.typeinfo import Types from pyflink.datastream import StreamExecutionEnvironment from pyflink.datastream.connectors.kafka import FlinkKafkaConsumer from pyflink.datastream.formats.json import JsonRowDeserializationSchema env = StreamExecutionEnvironment.getexecutionenvironment() env.add_jars("file:///path/to/flink-sql-connector-kafka.jar") deserialization_schema = JsonRowDeserializationSchema.builder() \ .typeinfo(typeinfo=Types.ROW([Types.INT(), Types.STRING()])).build() kafka_consumer = FlinkKafkaConsumer( topics='testsourcetopic', deserializationschema=deserializationschema, properties={'bootstrap.servers': 'localhost:9092', 'group.id': 'test_group'}) ds = env.addsource(kafkaconsumer) ``` <span class="label label-info">Note</span> It currently only supports `FlinkKafkaConsumer` to be used as DataStream source connectors with method `add_source`. <span class="label label-info">Note</span> The `DataStream` created using `add_source` could only be executed in `streaming` executing mode. You could also call the `from_source` method to create a `DataStream` using unified DataStream source connectors: ```python from pyflink.common.typeinfo import Types from pyflink.common.watermark_strategy import WatermarkStrategy from pyflink.datastream import StreamExecutionEnvironment from pyflink.datastream.connectors.number_seq import NumberSequenceSource env = StreamExecutionEnvironment.getexecutionenvironment() seqnumsource = NumberSequenceSource(1, 1000) ds = env.from_source( source=seqnumsource, watermarkstrategy=WatermarkStrategy.formonotonous_timestamps(), sourcename='seqnum_source', type_info=Types.LONG()) ``` <span class="label label-info">Note</span> Currently, it only supports `NumberSequenceSource` and `FileSource` as unified DataStream source connectors. <span class="label label-info">Note</span> The `DataStream` created using `from_source` could be executed in both `batch` and `streaming` executing mode. Table & SQL connectors could also be used to create a `DataStream`. You could firstly create a `Table` using Table & SQL connectors and then convert it to a `DataStream`. ```python from pyflink.common.typeinfo import Types from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import StreamTableEnvironment env = StreamExecutionEnvironment.getexecutionenvironment() tenv = StreamTableEnvironment.create(streamexecution_environment=env) tenv.executesql(""" CREATE TABLE my_source ( a INT, b VARCHAR ) WITH ( 'connector' = 'datagen', 'number-of-rows' = '10' ) """) ds = tenv.toappend_stream( tenv.frompath('my_source'), Types.ROW([Types.INT(), Types.STRING()])) ``` <span class="label label-info">Note</span> The StreamExecutionEnvironment `env` should be specified when creating the TableEnvironment `t_env`. <span class="label label-info">Note</span> As all the Java Table & SQL connectors could be used in PyFlink Table API, this means that all of them could also be used in PyFlink DataStream API. {{< top >}} DataStream Transformations Operators transform one or more `DataStream` into a new `DataStream`. Programs can combine multiple transformations into sophisticated dataflow topologies. The following example shows a simple example about how to convert a `DataStream` into another `DataStream` using `map` transformation: ```python ds = ds.map(lambda a: a + 1) ``` Please see for an overview of the available DataStream

Conversion between DataStream and Table It also supports to convert a `DataStream` to a `Table` and vice verse. ```python table = tenv.fromdata_stream(ds, 'a, b, c') ds = tenv.toappend_stream(table, Types.ROW([Types.INT(), Types.STRING()])) ds = tenv.toretract_stream(table, Types.ROW([Types.INT(), Types.STRING()])) ``` {{< top >}} Emit Results You can call the `print` method to print the data of a `DataStream` to the standard output: ```python ds.print() ``` You can call the `executeandcollect` method to collect the data of a `DataStream` to client: ```python with ds.executeandcollect() as results: for result in results: print(result) ``` <span class="label label-info">Note</span> The method `executeandcollect` will collect the data of the `DataStream` to the memory of the client and so it's a good practice to limit the number of rows collected. You can call the `add_sink` method to emit the data of a `DataStream` to a DataStream sink connector: ```python from pyflink.common.typeinfo import Types from pyflink.datastream.connectors.kafka import FlinkKafkaProducer from pyflink.datastream.formats.json import JsonRowSerializationSchema serializationschema = JsonRowSerializationSchema.builder().withtype_info( type_info=Types.ROW([Types.INT(), Types.STRING()])).build() kafka_producer = FlinkKafkaProducer( topic='testsinktopic', serializationschema=serializationschema, producerconfig={'bootstrap.servers': 'localhost:9092', 'group.id': 'testgroup'}) ds.addsink(kafkaproducer) ``` <span class="label label-info">Note</span> It currently only supports FlinkKafkaProducer and JdbcSink to be used as DataStream sink connectors with method `add_sink`. <span class="label label-info">Note</span> The method `add_sink` could only be used in `streaming` executing mode. You could also call the `sink_to` method to emit the data of a `DataStream` to a unified DataStream sink connector: ```python from pyflink.datastream.connectors.file_system import FileSink, OutputFileConfig from pyflink.common.serialization import Encoder output_path = '/opt/output/' file_sink = FileSink \ .forrowformat(outputpath, Encoder.simplestring_encoder()) \ .withoutputfileconfig(OutputFileConfig.builder().withpartprefix('pre').withpart_suffix('suf').build()) \ .build() ds.sinkto(filesink) ``` <span class="label label-info">Note</span> It currently only supports `FileSink` as unified DataStream sink connectors. <span class="label label-info">Note</span> The method `sink_to` could be used in both `batch` and `streaming` executing mode. Table & SQL connectors could also be used to write out a `DataStream`. You need firstly convert a `DataStream` to a `Table` and then write it to a Table & SQL sink connector. ```python from pyflink.common import Row from pyflink.common.typeinfo import Types from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import StreamTableEnvironment env = StreamExecutionEnvironment.getexecutionenvironment() tenv = StreamTableEnvironment.create(streamexecution_environment=env) def split(s): splits = s[1].split("|") for sp in splits: yield Row(s[0], sp) ds = ds.map(lambda i: (i[0] + 1, i[1])) \ .flat_map(split, Types.ROW([Types.INT(), Types.STRING()])) \ .key_by(lambda i: i[1]) \ .reduce(lambda i, j: Row(i[0] + j[0], i[1])) def split(s): splits = s[1].split("|") for sp in splits: yield s[0], sp ds = ds.map(lambda i: (i[0] + 1, i[1])) \ .flat_map(split, Types.TUPLE([Types.INT(), Types.STRING()])) \ .key_by(lambda i: i[1]) \ .reduce(lambda i, j: (i[0] + j[0], i[1])) tenv.executesql(""" CREATE TABLE my_sink ( a INT, b VARCHAR ) WITH ( 'connector' = 'print' ) """) table = tenv.fromdata_stream(ds) tableresult = table.executeinsert("my_sink") ``` <span class="label label-info">Note</span> The output type of DataStream `ds` must be composite type. Submit Job Finally, you should call the `StreamExecutionEnvironment.execute` method to submit the DataStream API job for execution: ```python env.execute() ``` If you convert the `DataStream` to a `Table` and then write it to a Table API & SQL sink connector, it may happen that you need to submit the job using `TableEnvironment.execute` method. ```python t_env.execute() ```