Project: Beam (App Definition and Development / Streaming & Messaging)
File: TASKS.md

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> - - - - - - - - - ``` cd beam ./gradlew playgroundPrecommit ``` ``` cd beam ./gradlew playground:tasks ``` ``` cd beam ./gradlew playground:generateProto ``` Start: ```bash cd beam ./gradlew playground:backend:containers:router:dockerComposeLocalUp ``` Stop: ```bash cd beam ./gradlew playground:backend:containers:router:dockerComposeLocalDown ``` Edit `/playground/frontend/playgroundcomponents/lib/src/constants/backendurls.dart` to override backend URLs with yours found in `/playground/docker-compose.local.yaml`. To start, run: ```bash cd beam ./gradlew playground:dockerComposeLocalUp ``` To stop, run: ```bash cd beam ./gradlew playground:dockerComposeLocalDown ``` This way of running may not work in all environments because it is not maintained. It is used occasionally by the Frontend team to test complex tasks against a not-yet-deployed backend. The full start may take ~30 minutes and is demanding, so you should likely enable only one backend runner for the SDK you need. If you do not need particular runners, comment out: Dependencies on them in `/playground/build.gradle.kts` in `dockerComposeLocalUp` task. Their Docker image configurations in `/playground/docker-compose.local.yaml`. See also in the Frontend. Run the method to remove unused code snippets from the Cloud Datastore. Unused snippets are snippets that are out of date. If the last visited date property less or equals than the current date minus dayDiff parameter then a snippet is out of date ``` cd beam ./gradlew playground:backend:removeUnusedSnippet -DdayDiff={int} -DprojectId={string} -Dnamespace={datastore namespace} ``` Run the method to remove a specific code snippet from the Cloud Datastore. ``` cd beam ./gradlew playground:backend:removeSnippet -DsnippetId={string} -DprojectId={string} -Dnamespace={datastore namespace} ``` ``` cd beam ./gradlew playground:backend:testWithoutCache ``` Update default `BEAM_VERSION` values in Java contianer Update `defaultbeamversion` in Update `defaultbeamversion` in Update SDK version in CI GitHub workflow: Update `-Psdk-tag=` in Update `BEAMVERSION` in - Update Go version in Run `go mod download` and `go mod tidy` in directory Update `BASE_IMAGE` parameter in and Update `GOBASEIMAGE` parameter in and Update `GOBASEIMAGE` parameter in and Update `GOBASEIMAGE` parameter in and Update `BASE_IMAGE` for `buildArgs` argument in Update `BASE_IMAGE` in Update base images version in `FROM

as dep` and `FROM apache/beamjava11sdk:$BEAM_VERSION` lines in Update `BASE_IMAGE` in and Update the version in the following locations: , the line with `flutter: '>=x.x.x'`. , the line with `flutter: '>=x.x.x'`. , the line with `flutter: '>=x.x.x'`. In Tour of Beam: , the line with `flutter: '>=x.x.x'`. Run `dart pub get` in all of these directories for these files to be updated. Verify and merge the changes. Playground: , the line with `flutter: ">=x.x.x"`. Tour of Beam: , the line with `flutter: ">=x.x.x"`. , the line with `ARG FLUTTER_VERSION=x.x.x`. , the line with `FLUTTER_VERSION: x.x.x`. , the line with `FLUTTER_VERSION: x.x.x`. , the line with `FLUTTER_VERSION: x.x.x`. See for details on how to build and deploy the application and its dependent infrastructure. SCIO examples are based on examples found in the upstream SCIO . To obtain the examples, use script: ```shell python fetchscalaexamples.py --outputdir <outputdir> ``` this will download all supported SCIO examples into `<output_dir>`. To add a new example, add it to the list of examples in the script . You will need to provide the path to the example in the `scio` git repository, the name of the example, description, run options, categories and tags. The following requirements are needed for deploying examples manually: GCP project with deployed Playground backend Python (3.9.x) Login into GCP (gcloud default login or using service account key) Example deployment scripts uses following environment variables: GOOGLECLOUDPROJECT - GCP project id where Playground backend is deployed BEAMROOTDIR - root folder to search for playground examples SDK_CONFIG - location of sdk and default example configuration file BEAMEXAMPLECATEGORIES - location of example category configuration file BEAMUSEWEBGRPC - use grpc-Web instead of grpc (default) GRPC_TIMEOUT - timeout for grpc calls (defaults to 10 sec) BEAM_CONCURRENCY - number of eaxmples to run in parallel (defaults to 10) SERVER_ADDRESS - address of the backend runnner service for a particular SDK ``` usage: cicd.py [-h] --step {CI,CD} [--namespace NAMESPACE] --datastore-project DATASTOREPROJECT --sdk {SDKJAVA,SDKGO,SDKPYTHON,SDKSCIO} --origin {PGEXAMPLES,TBEXAMPLES} --subdirs SUBDIRS [SUBDIRS ...] CI/CD Steps for Playground objects optional arguments: -h, --help show this help message and exit --step {CI,CD} CI step to verify all beam examples/tests/katas. CD step to save all beam examples/tests/katas and their outputs on the GCD --namespace NAMESPACE Datastore namespace to use when saving data (default: Playground) --datastore-project DATASTORE_PROJECT Datastore project to use when saving data --sdk {SDKJAVA,SDKGO,SDKPYTHON,SDKSCIO} Supported SDKs --origin {PGEXAMPLES,TBEXAMPLES} ORIGIN field of pgexamples/pgsnippets --subdirs SUBDIRS [SUBDIRS ...] limit sub directories to walk through, relative to BEAMROOTDIR ``` Helper script to deploy examples for all supported sdk's: ```shell cd playground/infrastructure export BEAMROOTDIR="../../" export SDK_CONFIG="../../playground/sdks.yaml" export BEAMEXAMPLECATEGORIES="../categories.yaml" export BEAMUSEWEBGRPC=yes export BEAM_CONCURRENCY=4 export PLAYGROUNDDNSNAME="your registered dns name for Playground" for sdk in go java python scio; do export SDK=$sdk && export SERVERADDRESS=https://${SDK}.$PLAYGROUNDDNS_NAME && python3 cicd.py --datastore-project $GOOGLECLOUD_PROJECT \ --step CD --sdk SDK_${SDK^^} \ --origin PG_EXAMPLES \ --subdirs ./learning/katas ./examples ./sdks done ```