Package: target-bigquery-partition
target-bigqueryANELEN's implementation of target-bigquery.This is aSingertarget that loads JSON-formatted data
following theSinger specto Google BigQuery.InstallationStep 0: Acknowledge LICENSE and TERMSPlease especially note that the author(s) of target-bigquery is not responsible
for the cost, including but not limited to BigQuery cost) incurred by running
this program.Step 1: Activate the Google BigQuery API(originally found in theGoogle API docs)Usethis wizardto create or select a project in the Google Developers Console and activate the BigQuery API. Click Continue, then Go to credentials.On theAdd credentials to your projectpage, click theCancelbutton.At the top of the page, select theOAuth consent screentab. Select anEmail address, enter aProduct nameif not already set, and click theSavebutton.Select theCredentialstab, click theCreate credentialsbutton and selectOAuth client ID.Select the application typeOther, enter the name "Singer BigQuery Tap", and click theCreatebutton.ClickOKto dismiss the resulting dialog.Click the Download button to the right of the client ID.Move this file to your working directory and rename itclient_secrets.json.Export the location of the secret file:export GOOGLE_APPLICATION_CREDENTIALS="./client_secret.json"For other authentication method, please see Authentication section.Step 2: InstallFirst, make sure Python 3 is installed on your system or follow these
installation instructions for Mac or Ubuntu.pip install -U target-bigquery-partitionOr you can install the lastest development version from GitHub:pip install --no-cache-dir https://github.com/anelendata/target-bigquery/archive/master.tar.gz#egg=target-bigqueryRunStep 1: ConfigureCreate a file called target_config.json in your working directory, following
config.sample.json:{
    "project_id": "your-gcp-project-id",
    "dataset_id": "your-bigquery-dataset",
    "table_prefix": "optional_table_prefix",
    "table_ext": "optional_table_ext",
    "partition_by": "optional_column_name",
    "partition_type": "day",
    "partition_exp_ms": null,
    "stream": false,
}Notes:The table name is set as stream name from the tap. You can add prefix and ext to the name.Optionally, you can set partition_by to create a partitioned table. Many production quailty taps implements a ingestion timestamp and it is recommended to use the column here to partition the table. It will increase the query performance and lower the BigQuery costs. partition_type can be hour, day, month, or year and the default is day. partition_exp_ms sets the partition expiration in millisecond. Default is null (never expire).stream: Make this true to run the streaming updates to BigQuery. Note that performance of batch update is better when keeping this optionfalse.Step 2: Runtarget-bigquery can be run with any Singer Target. As example, let usetap-exchangeratesapi.pip install tap-exchangeratesapiRun:tap-exchangeratesapi | target-bigquery -c target_config.jsonAuthenticationIt is recommended to usetarget-bigquerywith a service account.Download the client_secrets.json file for your service account, and place it
on the machine wheretarget-bigquerywill be executed.Set aGOOGLE_APPLICATION_CREDENTIALSenvironment variable on the machine,
where the value is the fully qualified path to client_secrets.jsonIn the testing environment, you can also manually authenticate before runnig
the tap. In this case you do not needGOOGLE_APPLICATION_CREDENTIALSdefined:gcloud auth application-default loginYou may also have to set the project:gcloud config set project <project-id>Though not tested, it should also be possible to use the OAuth flow to
authenticate to GCP as well:target-bigquerywill attempt to open a new window or tab in your default
browser. If this fails, copy the URL from the console and manually open it
in your browser.If you are not already logged into your Google account, you will be prompted
to log in.If you are logged into multiple Google accounts, you will be asked to select
one account to use for the authorization.Click theAcceptbutton to allowtarget-bigqueryto access your Google BigQuery
table.You can close the tab after the signup flow is complete.Original repohttps://github.com/anelendata/target-bigqueryAbout this projectThis project is developed by
ANELEN and friends. Please check out the ANELEN'sopen innovation philosophy and other projectsCopyright © 2020~ Anelen Co., LLC