Project: Beam (App Definition and Development / Streaming & Messaging)
File: load_your_code.md

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> Adding a new example, code snippet, or Tour of Beam learning unit into the Playground is a three-step process: Prepare a code snippet. Add the code snippet to Apache Beam and/or Playground. Create a link to view the code snippet in Playground or to embed in a website page. Playground sources and output presentation formats: This guide will walk through all steps. * * * + + * * * + + + * + * * * Playground runs example code snippets using Apache Beam Direct Runner and requires that a code snippet is a complete runnable code. Code snippets can use data sources to demonstrate transforms and concepts. Playground restricts code access to Internet for security reasons. Following are the recommend ways for code snippet's data sources and dependecies: | Source/Dependency | Notes | |-|--| | File | Store code snippet's data file in a GCS bucket in `apache-beam-testing` project. | | BigQuery | Create a BigQuery dataset/table in `apache-beam-testing` project. | | Python package | Python packages accessible by Playground are located in a and in . Add required packages to . Please submit pull request with changes to the container or contact | | GitHub repo | If your example clones or dependes on files in a GitHub repo, copy required files to a GCS bucket in `apache-beam-testing` project and use the GCS files. | Playground provides multiple features to help focus users on certain parts of the code. Playground automatically applies the following to all snippets: Folds a comment if a snippet starts with one. Folds imports. Playground supports Named Sections to tag code blocks and provide the following view options: Fold all blocks except tagged code blocks. This can be useful to help user focus on specific code blocks and features presented in a snippet. Hide all code except tagged code blocks. This can be useful to create runnable snippets illustrating specific concepts or transforms, and hide all non-essential code blocks. Such snippet can be embedded on a website to make examples in documentation and tutorials runnable. Make certain code parts read-only. This feature can be useful to create learning units where user modifications are desired only in certain parts of the

Please see section for details how different view options can be used. If you do not need any of those view options, skip to the . Named Sections are defined with the following syntax: ``` // [START section_name] void method() { ... } // [END section_name] ``` Create a named section for each part of your code that you want the above features for. To learn more details about the syntax please see the that Playground uses. There are several types of code snippets in the Playground: Example a code snippet displayed in the Playground Examples Catalog. See . Unlisted Example the same as an example, but is not listed in the example dropdown and can only be accessed by direct linking. These are typically embedded on a website. See . learning unit. See . User-shared code snippets do not require a PR and should be used for code not displayed on Beam resources. See . GitHub or other HTTPS URL sources. See . See the how artifacts map to these sources. Playground Examples Catalog helps users discover example snippets and is the recommended way to add examples. Playground automatically scans, verifies and deploys example snippets from the directories listed below. Note: SCIO examples are stored in a separate repository. To add support for a new SCIO example, please refer to . Playground Java, Python, and Go examples are automatically picked from these predefined directories by the `playgroundexamplesci.yml` GitHub workflow after a PR is merged to Beam repo: `/examples` `/learning/katas` `/sdks`. Adding Scala example snippets automatically is not supported, and Scala example snippets can be added to the catalog manually. Playground relies on metadata comments block to identify and place an example into the database, which is required for an example to show in the Examples Catalog. See for an example. Playground automatically removes metadata comments block before storing the example in database, so the metadata is not visible to end users. The block is in the format of a YAML map: ```yaml beam-playground: name: "" description: "" pipeline_options: "--name1 value1 --name2 value2" context_line: 1 categories: "Combiners" "Core Transforms" tags: "numbers" "count" complexity: BASIC default_example: true url_notebook: "https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb" multifile: true always_run: true datasets: CountWords: location: local format: avro emulators: type: kafka topic: id: dataset source_dataset: "CountWords" ``` For metadata reference, see the fields in the `Tag` class . The list of supported categories for an example is . To add a new category, submit a PR that adds a category to the . When it is merged, the new category can be used in an example. Each SDK must have a single default example. If there is none, the user will see an error in the app and a blank editor. If there are more than one, it is not defined which one will be selected. Examples which require Kafka server emulator need to include the `emulators` tag and provide `dataset` in the example's tag. You can refer to an example

Add your dataset in either JSON or Avro format into the `playground/backend/datasets` path. Add the following elements to the example's metadata tag: ```YAML emulators: type: kafka topic: id: dataset sourcedataset: <datasetname> datasets: <dataset_name>: location: local format: json # or 'avro' ``` replace `<dataset_name>` with the name of your dataset file without the file name extension. Use the exact string `"kafka_server:9092"` as the server name in your code snippet. This string will be replaced by the actual host name and port automatically before the compilation step by Playground. Kafka emulator limitations: Playground Kafka emulator currently supports only Beam Java SDK. The exact string `"kafka_server:9092"` should be present in the code snippet; any other variation like `"kafa_server" + ":9092"` will not work. Create and submit a PR with the code snippet into following the . Verify that all pre-commit tests are passing. Playground CI will verify and deploy the example to Playground Example Catalog when the PR is merged. The snippet will be assigned an ID. You can find it in the address bar of the browser when you select it in the dropdown. For example, in this URL: ``` https://play.beam.apache.org/?path=SDKJAVAMinimalWordCount&sdk=java ``` the ID is: `SDKJAVAMinimalWordCount`. You will need the snippet ID to embed the Playground with the snippet into a website page. Not all examples must be visible in the example dropdown. Some examples are best in the context of Apache Beam documentation. To embed them into the documentation, use unlisted examples. They work and are checked and cached the same way as the examples displayed in the Playground catalog. Proceed the same way as with except: Use the directory `/learning/beamdoc` Do not use the following attributes: `categories` `default_example` `tags` The ID of the snippet is a function of the SDK and the `name` attribute from its metadata: | SDK | ID | |--|--| | Go | SDKGOname | | Java | SDKJAVAname | | Python | SDKPYTHONname | "Tour of Beam" is a separate project that combines learning materials with runnable snippets and allows students to track their learning progress. It uses the Playground engine, and so its content is added in a similar way. A Tour of Beam unit consists of learning materials and an optional runnable snippet. See on how to add units and link snippets to them. Tour of Beam snippets are checked and cached the same way as Playground examples. Proceed the same way as with except: Use the directory `/learning/tour-of-beam/learning-content`. It is recommended to follow the directory hierarchy as described in . Do not use the following attributes: `categories` `default_example` `tags` The ID of the snippet is a function of the SDK and the `name` attribute from its metadata: | SDK | ID | |--|--| | Go | TBEXAMPLESSDKGOname | | Java | TBEXAMPLESSDKJAVAname | | Python | TBEXAMPLESSDKPYTHONname | For instance, for the Go the example `CSV` it is `TBEXAMPLESSDKGOCSV`. A code snippet can be saved to the Playground using "Share my code" button in the Playground: This is easy and

It does not require any interaction with the Beam team. Share my code considerations: A user-shared snippet is immutable. If you edit the code and re-share, a new snippet and a new link will be generated. Playground automatically applies a 3-month retention policy to shared snippets that are not used. To request a deletion of a snippet, please send an email to with subject: [Playground] Delete a snippet. Playground does not cache output or graph for user-shared snippets. Playground does not verify user-shared snippets. Playground can load a snippet stored on an HTTPS server using the provided URL, including GitHub direct links to raw file content. This is as easy and fast as using Share my code button, but also allows you to modify a snippet after it is published without changing a link. Loading snippet from HTTPS URL considerations: Playground does not cache output or graph for HTTPS URL snippets. Playground does not verify HTTPS URL snippets. For Playground to be able to load the snippet over HTTPS, the HTTPS server needs to allow the access by sending the following header: ``` Access-Control-Allow-Origin: * ``` at least when requested with `*.beam.apache.org` as . This is related to Cross-Origin Resource Sharing (CORS), to read more about CORS please see . Many prefer to host code snippets in their GitHub repositories. GitHub is known to allow cross-origin access on direct links to raw file content. An example of loading a GitHub snippet: ``` https://play.beam.apache.org/?sdk=go&url=https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go ``` The snippet can now be shown in the Playground. Choose any of the following ways. Open your snippet in the dropdown menu. Without changing it, click "Share my code". Copy the link. The link contains the `path` to your snippet in the database. It is in the following format: ``` https://play.beam.apache.org/?path=SDKJAVAMinimalWordCount&sdk=java ``` A special case is the default snippet for an SDK. It can be loaded by the following link: ``` https://play.beam.apache.org/?sdk=python&default=true ``` This way if another snippet is ever made default, the links you shared will lead to the new snippet. Link to an unlisted example can be constructed by providing your snippet ID and SDK in the following URL: ``` https://play.beam.apache.org/?path=<ID>&sdk=<SDK> ``` The ID of the snippet is a function of the SDK and the `name` attribute from its metadata: | SDK | ID | |--|--| | Go | SDKGOname | | Java | SDKJAVAname | | Python | SDKPYTHONname | Link to a snippet can be constructed by providing your snippet ID and SDK in the following URL: ``` https://play.beam.apache.org/?path=<ID>&sdk=<SDK> ``` The ID of the snippet is a function of the SDK and the `name` attribute from its metadata: | SDK | ID | |--|--| | Go | TBEXAMPLESSDKGOname | | Java | TBEXAMPLESSDKJAVAname | | Python | TBEXAMPLESSDKPYTHONname | For instance, for the Go the example `CSV` it is `TBEXAMPLESSDKGOCSV`, and the link is ``` https://play.beam.apache.org/?path=TBEXAMPLESSDKGOCSV&sdk=go ``` You get the link when you click "Share my code" button. It is in the following format: ```

``` Add the URL to the `url` parameter, for example: ``` https://play.beam.apache.org/?sdk=go&url=https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go ``` You can link to an empty editor to make your users start their snippets from scratch: ``` https://play.beam.apache.org/?sdk=go&empty=true ``` The above URLs load snippets that you want. But what happens if the user switches SDK? Normally this will be shown: The catalog default example for the new SDK. The empty editor for the new SDK if the Playground is embedded. This can be changed by linking to multiple examples, up to one per SDK. For this purpose, make a JSON array with any combination of parameters that are allowed for loading single examples, for instance: ```json [ { "sdk": "java", "path": "SDKJAVAAggregationMax" }, { "sdk": "go", "url": "https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go" } ] ``` Then pass it in`examples` query parameter like this: `https://play.beam.apache.org/?sdk=go&examples=[{"sdk":"java","path":"SDKJAVAAggregationMax"},{"sdk":"go","url":"https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go"}]` This starts with the Go example loaded from the URL. If SDK is then switched to Java, the `AggregationMax` catalog example is loaded for it. If SDK is switched to any other one, the default example for that SDK is loaded, because no override was provided. Embedded Playground is a simplified interface of the Playground web app designed to be embedded into an `<iframe>` in web pages. It supports most of the Playground web app features. The embedded Playground URLs start with `https://play.beam.apache.org/embedded` and use the same query string parameters as the Playground web app. Additionally, the Embedded playground supports `editable=0` parameter to make the editor read-only. Open your snippet in the dropdown menu. Without changing it, click "Share my code". Go to "Embed" tab. Copy the HTML code and add to your web page. Open your code by the link that you got when you shared it. Again click "Share my code". Go to "Embed" tab. Copy the HTML code and add to your web page. Follow the instructions to to your code. Optionally make the link to the Embedded Playground by replacing `play.beam.apache.org/?...` with `play.beam.apache.org/embedded?...` because the embedded interface is simpler. Insert this link into an `<iframe>` HTML element as follows: ```html <iframe src="https://play.beam.apache.org/embedded?sdk=go&url=https://raw.githubusercontent.com/apache/beam-starter-go/main/main.go" width="90%" height="600px" allow="clipboard-write" /> ``` Apache Beam website uses . Custom Hugo shortcodes were added to Apache Beam website to embed Playground snippets. Use the custom shortcodes to embed Playground into the Apache Beam website: `playground` shortcode, see for a complete example. `playground_snippet` shortcode, see for all supported options. These shortcodes generate an `iframe` with the URLs described above. If your code contains named sections as described in the , you can apply view options to those sections. Otherwise skip this. Add `readonly` parameter with comma-separated section names: `https://play.beam.apache.org/?sdk=go&url=...&readonly=section_name` Add `unfold` parameter with comma-separated section names: `https://play.beam.apache.org/?sdk=go&url=...&unfold=section_name` This folds all foldable blocks that do not overlap with any of the given sections. Add `show` parameter with a single section name: `https://play.beam.apache.org/?sdk=go&url=...&show=section_name` It is still the whole snippet that is sent for execution, although only the given section is visible. This also makes the editor read-only so the user cannot add code that conflicts with the hidden text.