Project: Beam (App Definition and Development / Streaming & Messaging)
File: python-streaming.md

type: languages title: "Apache Beam Python Streaming Pipelines" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> Python streaming pipeline execution became available (with some ) starting with Beam SDK version 2.5.0. Beam creates an unbounded PCollection if your pipeline reads from a streaming or continuously-updating data source (such as Cloud Pub/Sub). A runner must process an unbounded PCollection using a streaming job that runs continuously, as the entire collection is never available for processing at any one time. has more information about bounded and unbounded collections. To modify a batch pipeline to support streaming, you must make the following code changes: Use an I/O connector that supports reading from an unbounded source. Use an I/O connector that supports writing to an unbounded source. Choose a . The Beam SDK for Python includes two I/O connectors that support unbounded PCollections: Google Cloud Pub/Sub (reading and writing) and Google BigQuery (writing). The following snippets show the necessary code changes to modify the batch WordCount example to support streaming: These batch WordCount snippets are from . This code uses the TextIO I/O connector to read from and write to a bounded collection. ``` lines = p | 'read' >> ReadFromText(known_args.input) ... counts = (lines | 'split' >> (beam.ParDo(WordExtractingDoFn()) .withoutputtypes(six.text_type)) | 'pairwithone' >> beam.Map(lambda x: (x, 1)) | 'group' >> beam.GroupByKey() | 'count' >> beam.Map(count_ones)) ... output = counts | 'format' >> beam.Map(format_result) output | 'write' >>

``` These streaming WordCount snippets are from . This code uses an I/O connector that reads from and writes to an unbounded source (Cloud Pub/Sub) and specifies a fixed windowing strategy. ``` lines = p | beam.io.ReadFromPubSub(topic=knownargs.inputtopic) ... counts = (lines | 'split' >> (beam.ParDo(WordExtractingDoFn()) .withoutputtypes(six.text_type)) | 'pairwithone' >> beam.Map(lambda x: (x, 1)) | beam.WindowInto(window.FixedWindows(15, 0)) | 'group' >> beam.GroupByKey() | 'count' >> beam.Map(count_ones)) ... output = counts | 'format' >> beam.Map(format_result) output | beam.io.WriteStringsToPubSub(knownargs.outputtopic) ``` To run the example streaming WordCount pipeline, you must have a Cloud Pub/Sub input topic and output topic. To create, subscribe to, and pull from a topic for testing purposes, you can use the commands in the . The following simple bash script feeds lines of an input text file to your input topic: ``` cat <YOURLOCALTEXTFILE> | while read line; do gcloud pubsub topics publish <YOURINPUTTOPICNAME> --message "$line"; done ``` Alternately, you can read from a publicly available Cloud Pub/Sub stream, such as `projects/pubsub-public-data/topics/taxirides-realtime`. However, you must create your own output topic to test writes. The following commands run the example streaming pipeline. Specify your Cloud Pub/Sub project and input topic (`--inputtopic`), output Cloud Pub/Sub project and topic (`--outputtopic`). {{< runner direct >}} python -m apachebeam.examples.streamingwordcount \ --inputtopic "projects/YOURPUBSUBPROJECTNAME/topics/YOURINPUTTOPIC" \ --outputtopic "projects/YOURPUBSUBPROJECTNAME/topics/YOUROUTPUTTOPIC" \ --streaming {{< /runner >}} {{< runner flink >}} See /documentation/runners/flink/ for more information. {{< /runner >}} {{< runner spark >}} See /documentation/runners/spark/ for more information. {{< /runner >}} {{< runner dataflow >}} pip install apache-beam[gcp] python -m apachebeam.examples.streamingwordcount \ --runner DataflowRunner \ --project YOURGCPPROJECT \ --region YOURGCPREGION \ --templocation gs://YOURGCS_BUCKET/tmp/ \ --inputtopic "projects/YOURPUBSUBPROJECTNAME/topics/YOURINPUTTOPIC" \ --outputtopic "projects/YOURPUBSUBPROJECTNAME/topics/YOUROUTPUTTOPIC" \ --streaming {{< /runner >}} Check your runner's documentation for any additional runner-specific information about executing streaming pipelines: Python streaming execution does not currently support the following features: Custom source API User-defined custom merging `WindowFn` (with fnapi) For portable runners, see .