Project: Beam (App Definition and Development / Streaming & Messaging)
File: environments.md

title: "Container environments" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> The Beam SDK runtime environment can be with to isolate it from other runtime systems. To learn more about the container environment, read the Beam . Prebuilt SDK container images are released per supported language during Beam releases and pushed to . You may want to customize container images for many reasons, including: Pre-installing additional dependencies Launching third-party software in the worker environment Further customizing the execution environment This guide describes how to create and use customized containers for the Beam SDKs. This guide requires building images using Docker. . Some CI/CD platforms like also provide the ability to build images using Docker. For remote execution engines/runners, have a container registry to host your custom container image. Options include or a "self-hosted" repository, including cloud-specific container registries like (GCR) or (ECR). Make sure your registry can be accessed by your execution engine or runner. NOTE: On Nov 20, 2020, Docker Hub put into effect for anonymous and free authenticated use, which may impact larger pipelines that pull containers several times. For optimal user experience, we also recommend you use the latest released version of Beam. Beam are built from Dockerfiles checked into the repository and published to Docker Hub for every release. You can build customized containers in one of three ways: Dockerfile based on a released container image. This is sufficient for simple additions to the image, such as adding artifacts or environment variables. a source Dockerfile in . This method requires building from Beam source but allows for greater customization of the container (including replacement of artifacts or base OS/language versions). an existing container image to make it compatible with Apache Beam Runners. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners. Create a new Dockerfile that designates a base image using the . ``` FROM apache/beampython3.7sdk:2.25.0 ENV FOO=bar COPY /src/path/to/file /dest/path/to/file/ ``` This `Dockerfile` uses the prebuilt Python 3.7 SDK container image tagged at (SDK version) `2.25.0`, and adds an additional environment variable and file to the image. and the image using Docker. ``` export BASEIMAGE="apache/beampython3.7_sdk:2.25.0" export IMAGE_NAME="myremoterepo/mybeamsdk" export TAG="mybeamsdk-versioned-tag" docker pull "${BASE_IMAGE}" docker build -f Dockerfile -t "${IMAGE_NAME}:${TAG}" . ``` If your runner is running remotely, retag and the image to the appropriate repository. ``` docker push "${IMAGE_NAME}:${TAG}" ``` After pushing a container image, verify the remote image ID and digest matches the local image ID and digest, output from `docker build` or `docker images`. This method requires building image artifacts from Beam source. For additional instructions on setting up your development environment, see the . NOTE: It is recommended that you start from a stable release branch (`release-X.XX.X`) corresponding to the same version of the SDK to run your

Differences in SDK version may result in unexpected errors. Clone the `beam` repository. ``` export BEAMSDKVERSION="2.26.0" git clone https://github.com/apache/beam.git cd beam export BEAM_WORKDIR=$PWD git checkout origin/release-$BEAMSDKVERSION ``` Customize the `Dockerfile` for a given language, typically `sdks/<language>/container/Dockerfile` directory (e.g. the . Return to the root Beam directory and run the Gradle `docker` target for your image. ``` cd $BEAM_WORKDIR ./gradlew :sdks:java:container:java8:docker ./gradlew :sdks:java:container:java11:docker ./gradlew :sdks:java:container:java17:docker ./gradlew :sdks:go:container:docker ./gradlew :sdks:python:container:py38:docker ./gradlew :sdks:python:container:py39:docker ./gradlew :sdks:python:container:py310:docker ./gradlew :sdks:python:container:py311:docker ./gradlew :sdks:python:container:buildAll ``` Verify the images you built were created by running `docker images`. ``` $> docker images --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE apache/beamjava8sdk latest sha256:... ... 1 min ago ... apache/beamjava11sdk latest sha256:... ... 1 min ago ... apache/beamjava17sdk latest sha256:... ... 1 min ago ... apache/beampython3.6sdk latest sha256:... ... 1 min ago ... apache/beampython3.7sdk latest sha256:... ... 1 min ago ... apache/beampython3.8sdk latest sha256:... ... 1 min ago ... apache/beampython3.9sdk latest sha256:... ... 1 min ago ... apache/beampython3.10sdk latest sha256:... ... 1 min ago ... apache/beamgosdk latest sha256:... ... 1 min ago ... ``` If your runner is running remotely, retag the image and the image to your repository. You can skip this step if you provide a custom repo/tag as . ``` export BEAMSDKVERSION="2.26.0" export IMAGENAME="gcr.io/my-gcp-project/beampython3.7_sdk" export TAG="${BEAMSDKVERSION}-custom" docker tag apache/beampython3.7sdk "${IMAGE_NAME}:${TAG}" docker push "${IMAGE_NAME}:${TAG}" ``` After pushing a container image, verify the remote image ID and digest matches the local image ID and digest output from `docker_images --digests`. The docker Gradle task defines a default image repository and is the SDK version defined at . The default repository is the Docker Hub `apache` namespace, and the default tag is the defined at gradle.properties. You can specify a different repository or tag for built images by providing parameters to the build task. For example: ``` ./gradlew :sdks:python:container:py36:docker -Pdocker-repository-root="example-repo" -Pdocker-tag="2.26.0-custom" ``` builds the Python 3.6 container and tags it as `example-repo/beampython3.6sdk:2.26.0-custom`. From Beam 2.21.0 and later, a `docker-pull-licenses` flag was introduced to add licenses/notices for third party dependencies to the docker images. For example: ``` ./gradlew :sdks:java:container:java8:docker -Pdocker-pull-licenses ``` creates a Java 8 SDK image with appropriate licenses in `/opt/apache/beam/thirdpartylicenses/`. By default, no licenses/notices are added to the docker images. Beam offers a way to provide your own custom container image. The easiest way to build a new custom image that is compatible with Apache Beam Runners is to use a process. This copies over the necessary artifacts from a default Apache Beam base image to build your custom container image. Copy necessary artifacts from Apache Beam base image to your image. ``` FROM python:3.8-bookworm RUN pip install --no-cache-dir apache-beam[gcp]==2.52.0 COPY --from=apache/beampython3.8sdk:2.52.0 /opt/apache/beam /opt/apache/beam ENTRYPOINT ["/opt/apache/beam/boot"] ``` NOTE: This example assumes necessary dependencies (in this case, Python 3.8 and pip) have been installed on the existing base image. Installing the Apache Beam SDK into the image will ensure that the image has the necessary SDK dependencies and reduce the worker startup time. The version specified in the `RUN` instruction must match the version used to launch the pipeline.<br> Make sure that the Python or Java runtime version specified in the base image is the same as the version used to run the pipeline. NOTE: Any additional Python dependenices should be installed in the global Python environment in the custom image. and the image using Docker. ``` export

export IMAGE_NAME="myremoterepo/mybeamsdk" export TAG="latest" docker pull "${BASE_IMAGE}" docker build -f Dockerfile -t "${IMAGE_NAME}:${TAG}" . ``` If your runner is running remotely, retag the image and the image to your repository. ``` docker push "${IMAGE_NAME}:${TAG}" ``` From the 2.55.0 release, the Beam Go SDK has moved to using as a base. These images have a reduced security attack surface by not including common tools and utilities. This may cause difficulties customizing the image with using one of the above approaches. As a fallback, it's possible to build a custom image from scratch, by building a matching boot loader, and setting that as the container's entry point. For example, if it's preferable to use alpine as the container OS your multi-stage docker file might look like the following: ``` FROM golang:latest-alpine AS build_base WORKDIR /tmp/beam RUN GOBIN=`pwd` go install github.com/apache/beam/sdks/v2/go/container@v2.53.0 FROM alpine:3.9 RUN apk add ca-certificates COPY --from=build_base /tmp/beam/container /opt/apache/beam/boot ENTRYPOINT ["/opt/apache/beam/boot"] ``` Build and push the new image as when above. NOTE: Java and Python require additional dependencies, such as their runtimes, and SDK packages for a valid container image. The bootloader isn't sufficient for creating a custom container for these SDKs. The common method for providing a container image requires using the PortableRunner flag `--environment_config` as supported by the Portable Runner or by runners supported PortableRunner flags. Other runners, such as Dataflow, support specifying containers with different flags. {{< runner direct >}} export IMAGE="my-repo/beampythonsdk_custom" export TAG="X.Y.Z" export IMAGE_URL="${IMAGE}:${TAG}" python -m apache_beam.examples.wordcount \ --input=/path/to/inputfile \ --output /path/to/write/counts \ --runner=PortableRunner \ --job_endpoint=embed \ --environment_type="DOCKER" \ --environmentconfig="${IMAGEURL}" {{< /runner >}} {{< runner flink >}} export IMAGE="my-repo/beampythonsdk_custom" export TAG="X.Y.Z" export IMAGE_URL = "${IMAGE}:${TAG}" python -m apache_beam.examples.wordcount \ --input=/path/to/inputfile \ --output=path/to/write/counts \ --runner=FlinkRunner \ --environmentcachemillis=10000 \ --environment_type="DOCKER" \ --environmentconfig="${IMAGEURL}" {{< /runner >}} {{< runner spark >}} export IMAGE="my-repo/beampythonsdk_custom" export TAG="X.Y.Z" export IMAGE_URL = "${IMAGE}:${TAG}" python -m apache_beam.examples.wordcount \ --input=/path/to/inputfile \ --output=path/to/write/counts \ --runner=SparkRunner \ --environmentcachemillis=10000 \ --environment_type="DOCKER" \ --environmentconfig="${IMAGEURL}" {{< /runner >}} {{< runner dataflow >}} export GCS_PATH="gs://my-gcs-bucket" export GCP_PROJECT="my-gcp-project" export REGION="us-central1" export IMAGE="my-repo/beampythonsdk_custom" export TAG="X.Y.Z" export IMAGE_URL = "${IMAGE}:${TAG}" python -m apache_beam.examples.wordcount \ --input gs://dataflow-samples/shakespeare/kinglear.txt \ --output "${GCS_PATH}/counts" \ --runner DataflowRunner \ --project $GCP_PROJECT \ --region $REGION \ --templocation "${GCSPATH}/tmp/" \ --experiment=userunnerv2 \ --sdkcontainerimage=$IMAGE_URL {{< /runner >}} Avoid using the tag `:latest` with your custom images. Tag your builds with a date or a unique identifier. If something goes wrong, using this type of tag might make it possible to revert the pipeline execution to a previously known working configuration and allow for an inspection of changes. The following section describes some common issues to consider when you encounter unexpected errors running Beam pipelines with custom containers. Differences in language and SDK version between the container SDK and pipeline SDK may result in unexpected errors due to incompatibility. For best results, make sure to use the same stable SDK version for your base container and when running your pipeline. If you are running into unexpected errors when using remote containers, make sure that your container exists in the remote repository and can be accessed by any third-party service, if needed. Local runners attempt to pull remote images and default to local images. If an image cannot be pulled locally (by the docker daemon), you may see an log message like: ``` Error response from daemon: manifest for remote.repo/beampython3.7sdk:2.25.0-custom not found: manifest unknown: ... INFO:apachebeam.runners.portability.fnapirunner.workerhandlers:Unable to pull image... ```