Project: Beam (App Definition and Development / Streaming & Messaging)
File: learning-resources.md

title: "Learning Resources" aliases: /get-started/resources/learning-resources/ /documentation/resources/learning-resources/ <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> Welcome to our learning resources. This page contains a collection of resources that will help you to get started and use Apache Beam. If youre just starting, you can view this as a guided tour, otherwise you can jump straight to any section of your interest. If you have additional material that you would like to see here, please let us know at ! {{< toc >}} - How to set up and run a WordCount pipeline on the Java SDK. - How to set up and run a WordCount pipeline on the Python SDK. - How to set up and run a WordCount pipeline on the Go SDK. - Setting up a Java development environment for Apache Beam using IntelliJ and Maven. - Setting up a Python development environment for Apache Beam using PyCharm. - Walks you through the code of a simple WordCount pipeline. This is a very basic pipeline intended to show the most basic concepts of data processing. WordCount is the "Hello World" for data processing. - Introduces how to consider time while processing data, user defined transforms, windowing, filtering data, streaming pipelines, triggers, and session analysis. This is a great place to start once you get the hang of WordCount. - The Programming Guide contains more in-depth information on most topics in the Apache Beam SDK. These include descriptions on how everything works as well as code snippets to see how to use every part. This can be used as a reference guidebook. - Covers some basic background information, terminology, time domains, batch processing, and streaming. - Tour of the unified batch and streaming programming model in Beam, alongside with an example to explain many of the concepts. - Explanation on how runners execute an Apache Beam pipeline. This includes why serialization is important, and how a runner might distribute the work in parallel to multiple machines. - Common patterns such as writing data to multiple storage locations, slowly-changing lookup cache, calling external services, dealing with bad data, and starting jobs through a REST endpoint. - Common patterns such as GroupBy using multiple data properties, joining two PCollections on a common key, streaming large lookup tables, merging two streams with different window lengths, and threshold detection with time-series data. - Adding a retry policy to a `DoFn`. - Using multiple data sources, many common design patterns, and sentiment analysis to get insights into different news articles for TensorFlow and Dataflow. - IoT sensors are continuously streaming data to the

Learn how to handle the sensor data which can be useful for real-time monitoring, alerts, long-term data storage for analysis, performance improvement, and model training. - Migrate data from an into using . - Migrate data from a table into without thinking of its schema. - Migrate data from a in-memory database into . - Use Apache Beam with the RunInference API to use machine learning (ML) models to do local and remote inference with batch and streaming pipelines. Follow the to do image classification, image segmentation, language modeling, and MNIST digit classification. See examples of . - Predict the molecular energy from data stored in the (SDF) format. Train a model with for preprocessing in Python. This also shows how to create batch and streaming prediction pipelines in Apache Beam. - Find the optimal parameter settings for simulated physical machines like a bottle filler or cookie machine. The goal of each simulated machine is to have the same input/output of the actual machine, making it a "digital twin". This uses for preprocessing. - Use a Dataflow template to launch a pipeline from Google AppEngine, and how to run the pipeline periodically via a cron job. - Learn how to access a persistent mutable state while processing input elements, this allows for side effects in a `DoFn`. This can be used for arbitrary-but-consistent index assignment, if you want to assign a unique incrementing index to each incoming element where order doesn't matter. - An example on how to do batched RPC calls. The call requests are stored in a mutable state as they are received. Once there are either enough requests or a certain time has passed, the batch of requests is triggered to be sent. - Call an external library written in a language that does not have a native SDK in Apache Beam such as C++. - Five part video series for understanding basic to advanced concepts. See more -- Free live and recorded lessons for learning Beam and data processing. - Course specialized for Dataflow runner. by Jan Lukavsk, Packt. (January 2022). A general description of the Apache Beam model including gradually built examples that help create solid understanding of the subject. In the first part the book explains concepts using Java SDK, then SQL DSL and Portability layer with focus on Python SDK. The last part of the book is dedicated to more advanced topics like IO connectors using Splittable DoFn and description of how a typical runner executes Pipeline. by Tyler Akidau, Slava Chernyak, Reuven Lax. (August 2018). Expanded from Tyler Akidaus popular blog posts "Streaming 101" and "Streaming 102", this book takes you from an introductory level to a nuanced understanding of the what, where, when, and how of processing real-time data streams. This quest includes four labs that teach you how to write and test Apache Beam pipelines. Three of the labs use Java and one uses Python. Each lab takes about 1.5 hours to complete. When you complete the quest, you're granted a badge that you can use to show your Beam expertise. (40m) - Run a word count pipeline on the Dataflow runner. (45m) - Create a real-time flight delay prediction service using historical data on internal flights in the United

(60m) - Process real-time streaming data from a real-time real world historical data set, store the results in BigQuery, and visualize the geospatial data on Data Studio. (90m) - Implement time-windowed aggregation to augment the raw data in order to produce a consistent training and test datasets for a machine learning model. (30m) - Run a word count pipeline on the Dataflow runner. (60m) - Simulate real-time historic internal flights in the United States and store the resulting simulated data in BigQuery. Beam Katas are interactive Beam coding exercises (i.e. ) that can help you to learn Apache Beam concepts and programming model hands-on. Built based on , Beam Katas objective is to provide a series of structured hands-on learning experiences for learners to understand about Apache Beam and its SDKs by solving exercises with gradually increasing complexity. Beam Katas are available for both Java and Python SDKs. Download Upon opening the IDE, expand the "Learn and Teach" menu, then select "Browse Courses" Search for "Beam Katas - Java" Expand the "Advanced Settings" and modify the "Location" and "Jdk" appropriately Click "Join" about how to use the Education product Download Upon opening the IDE, expand the "Learn and Teach" menu, then select "Browse Courses" Search for "Beam Katas - Python" Expand the "Advanced Settings" and modify the "Location" and "Interpreter" appropriately Click "Join" about how to use the Education product The includes examples in Java, Python, and Scala (via Scio), provides ready-to-launch and self-contained Beam pipelines. - Commonly-used data analysis patterns such as how to use , a CombinePerKey transform, remove duplicate lines in files, filtering, joining PCollections, getting the maximum value of a PCollection, etc. - Additional examples on common tasks such as configuring , , writing one file per window, etc. - End-to-end example pipelines such as an auto complete, a streaming word extract, calculating the Term Frequency-Inverse Document Frequency (), getting the top Wikipedia sessions, traffic max lane flow, traffic routes, etc. - A complete example demonstrates using Apache Beam on Dataflow to convert JSON encoded Pub/Sub subscription message strings into structured data and write that data to a BigQuery table. - Commonly-used data analysis patterns such as how to use , , coders, combiners, filters, custom PTransforms, etc. - End-to-end example pipelines such as an auto complete, getting mobile gaming statistics, calculating the , solving distributing optimization tasks, estimating PI, calculating the Term Frequency-Inverse Document Frequency (), getting the top Wikipedia sessions, etc. is an interactive environment to try out Beam transforms and examples without having to install Apache Beam in your environment. You can try the available Apache Beam examples at . Learn more about how to add an Apache Beam example/test/kata into Beam Playground catalog . - Official API Reference for the Java SDK. - Official API Reference for the Python SDK. - Official API Reference for the Go SDK. We are open for feedback and suggestions, you can find different ways to reach out to the community in the page. If you have a bug report or want to suggest a new feature, you can let us know by . We welcome contributions from everyone! To learn more on how to contribute, check our .