Package: ox-bqpipeline
BigQuery PipelineUtility class for building data pipelines in BigQuery.Provides methods for query, copy table, delete table and export to GCS.Supports Jinja2 templated SQL.Getting StartedCheck out the codelab!UsageCreate an instance of BQPipeline. By settingquery_project,default_projectanddefault_dataset, you can omit project and dataset from table references in your SQL statements.default_projectis the project used when a tablespec does not specify a project.default_datasetis the dataset used when a tablespec does not specify project or dataset.Place files containing a single BigQuery Standard SQL statement per file.Note that if you reference a project with a '-' in the name, you must backtick the tablespec in your SQL:`my-project.dataset_id.table_id`Writing scripts to be easily portable between environmentsUse{{ project }}in all your sql queriesIn your replacements dictionary, set'project'to the value ofBQPipeline.infer_project()this will infer the project from the credentials. This means in your local shell it will useGOOGLE_APPLICATION_DEFAULTand on the cron box it will use project of the CronBox's Service Account.#!/usr/bin/env python
# -*- coding: utf-8 -*-
from ox_bqpipeline.bqpipeline import BQPipeline

bq = BQPipeline(job_name='myjob',
                default_dataset='mydataset',
                json_credentials_path='credentials.json')

replacements = {
    'project': bq.infer_project(),
    'dataset': 'mydataset'
}

bq.copy_table('source_table', 'dest_table')

bq.run_queries(['../sql/q1.sql', ('../sql/q2.sql', 'tmp_table_1')], **replacements)

bq.export_csv_to_gcs('tmp_table_2', 'gs://my-bucket/path/to/tmp_table_2-*.csv')

bq.delete_tables(['tmp_table_1', 'tmp_table_2'])Note, that therun_queriesmethod provided this utility can alternatively take a list of tuples where the first entry is the sql path, and the second is a destination table. You can see an example of this inexample_pipeline.py.For detailed documentation about the methods provided by this utility class seedocs.md.Writing scripts with parameterized queriesBigquery standard sql provides support forparameterized queries.To run parameterized queries using the bqpipeline from commandline, use the following syntax:python3ox_bqpipeline/bqpipeline.py--query_filequery.sql--gcs_destinationgs://bucket_path--query_params'{"int_param": 1, "str_param": "one"}'In order to invoke the BQPipelines.run_queries method from within your python
module, use the following pattern.bqp=bqpipeline.BQPipeline(job_name='testjob',default_project='project_name',default_dataset='dataset_name')qj_list=bqp.run_queries([(<query1_path>,<table_or_gcs_dest>,{'query1_params key,val'}),(<query2_path>,<table_or_gcs_dest>,{'query2_params key,val'}),],batch=False,overwrite=False,dry_run=False)Creating Service Account JSON CredentialsVisit theService Account ConsoleSelect a service accountSelect "Create Key"Select "JSON"Click "Create" to download the fileInstallationOptional: Install in virtualenvpython3 -m virtualenv venv
source venv/bin/activateInstall with pippipenvinstall--python3Install with pipenvpython3-mpipinstall-rrequirements.txtorpip3install-rrequirements.txtRun test suitepython3-munittestdiscoverRun test suite using pipenvpipenvrunpython-munittestdiscoverRequirementsYou'll need todownload Python 3.4 or laterGoogle Cloud Python ClientDisclaimerThis is not an official Google project.ReferencesPython Example Codegoogle-cloud-bigqueryJinja2ox_bqpipeline Reference