Package: google-dataproc-templates
Dataproc Templates (Python - PySpark)AzureBlobStorageToBigQueryBigQueryToGCS(blogpostlink)CassandraToBigqueryCassandraToGCS(blogpostlink)GCSToBigQuery(blogpostlink)GCSToBigTable(blogpostlink)GCSToGCS(blogpostlink)GCSToJDBC(blogpostlink)GCSToMongo(blogpostlink)HbaseToGCSHiveToBigQuery(blogpostlink)HiveToGCS(blogpostlink)JDBCToBigQuery(blogpostlink)JDBCToGCS(blogpostlink)JDBCToJDBC(blogpostlink)KafkaToGCSKafkaToBigQueryMongoToGCS(blogpostlink)PubSubLiteToBigtableRedshiftToGCS(blogpostlink)S3ToBigQuerySnowflakeToGCS(blogpostlink)TextToBigQueryDataproc Templates (Python - PySpark) submit jobs to Dataproc Serverless usingbatches submit pyspark.Run using PyPi packageIn this README, you see instructions on how to submit Dataproc Serverless template jobs.Currently, 3 options are described:Using bin/start.shUsing gcloud CLIUsing Vertex AIThose 3 options require you to clone this repo and start running the templates.TheDataproc Templates PyPi packageis a4th optionto run templates from a PySpark environment directly (Dataproc or local/another).Example:!pip3 install --user google-dataproc-templates==0.0.3

from dataproc_templates.bigquery.bigquery_to_gcs import BigQueryToGCSTemplate
from pyspark.sql import SparkSession

args = dict()
args["bigquery.gcs.input.table"] = "<bq_dataset>.<bq_table>"
args["bigquery.gcs.input.location"] = "<location>"
args["bigquery.gcs.output.format"] = "<format>"
args["bigquery.gcs.output.mode"] = "<mode>"
args["bigquery.gcs.output.location"] = "gs://<bucket_name/path>"

spark = SparkSession.builder \
        .appName("BIGQUERYTOGCS") \
        .enableHiveSupport() \
        .getOrCreate()

template = BigQueryToGCSTemplate()
template.run(spark, args)Pro Tip:Start a Dataproc Serverless Spark sessionsin a Vertex AI managed notebook, and leverage a serverless Spark session, in which your job will run using Dataproc Serverless, instead of your local PySpark environment.While this provides an easy way to get started, remember that the bin/start.sh already provides an easy way for you to, for example, specify required .jar dependencies. Using the PyPi package, you need to configure your PySpark sessions in accordance with the requirements of your specific template. You would need to, for example, specify the spark.driver.extraClassPath configuration:spark = SparkSession.builder \
        ... \
        .config('spark.driver.extraClassPath', '<template_required_dependency>.jar')
        ... \
        .getOrCreate()Setting up the local environmentIt is recommended to use avirtual environmentwhen setting up the local environment. This setup is not required for submitting templates, only for running and developing locally.# Create a virtual environment, activate it and install requirementsmkdirvenv
python-mvenvvenv/sourcevenv/bin/activate
pipinstall-rrequirements.txtRunning unit testsUnit tests are developed usingpytest.To run all unit tests, simply run pytest:pytestTo generate a coverage report, run the tests using coveragecoveragerun\--source=dataproc_templates\--modulepytest\--verbose\testcoveragereport--show-missingSubmitting templates to Dataproc ServerlessA shell script is provided to:Build the python packageSet Dataproc parameters based on environment variablesSubmit the desired template to Dataproc with the provided template parametersWhen submitting, there are 3 types of properties/parameters for the user to provide.Spark properties: Refer to thisdocumentationto see the available spark properties.Each template's specific parameters: refer to each template's README.Common arguments: --template_name and --log_levelThe--log_levelparameter is optional, it defaults to INFO.Possible choices are the Spark log levels: ["ALL", "DEBUG", "ERROR", "FATAL", "INFO", "OFF", "TRACE", "WARN"].bin/start.sh usage:# Set required environment variables
export GCP_PROJECT=<project_id>
export REGION=<region>
export GCS_STAGING_LOCATION=<gs://path>

# Set optional environment variables
export SUBNET=<subnet>
export JARS="gs://additional/dependency.jar"
export HISTORY_SERVER_CLUSTER=projects/{projectId}/regions/{regionId}/clusters/{clusterId}
export METASTORE_SERVICE=projects/{projectId}/locations/{regionId}/services/{serviceId}

# Submit to Dataproc passing template parameters
./bin/start.sh [--properties=<spark.something.key>=<value>] \
               -- --template=TEMPLATENAME \
                  --log_level=INFO \
                  --my.property="<value>" \
                  --my.other.property="<value>"
                  (etc...)gcloud CLI usage:It is also possible to submit jobs using thegcloudCLI directly. That can be achieved by:Building thedataproc_templatespackage into an.eggPACKAGE_EGG_FILE=dist/dataproc_templates_distribution.egg
pythonsetup.pybdist_egg--output=${PACKAGE_EGG_FILE}Submitting the jobThemain.pyfile should be the main python scriptThe.eggfile for the package must be bundled using the--py-filesflaggcloud dataproc batches submit pyspark \
      --region=<region> \
      --project=<project_id> \
      --jars="<required_jar_dependencies>" \
      --deps-bucket=<gs://path> \
      --subnet=<subnet> \
      --py-files=${PACKAGE_EGG_FILE} \
      [--properties=<spark.something.key>=<value>] \
      main.py \
      -- --template=TEMPLATENAME \
         --log_level=INFO \
         --<my.property>="<value>" \
         --<my.other.property>="<value>"
         (etc...)Vertex AI usage:FollowDataproc Templates (Jupyter Notebooks) READMEto submit Dataproc Templates from a Vertex AI notebook.