Package: tap-bigquery
tap-bigqueryExtract data from BigQuery tables.This is aSingertap that produces JSON-formatted data
following theSinger spec.This tap:Pulls data from Google BigQuery tables/views with datetime field.Infers the schema for each resource and produce catalog file.Incrementally pulls data based on the input state.InstallationStep 0: Acknowledge LICENSE and TERMSPlease especially note that the author(s) of tap-bigquery is not responsible
for the cost, including but not limited to BigQuery cost) incurred by running
this program.Step 1: Activate the Google BigQuery API(originally found in theGoogle API docs)Usethis wizardto create or select a project in the Google Developers Console and activate the BigQuery API. Click Continue, then Go to credentials.On theAdd credentials to your projectpage, click theCancelbutton.At the top of the page, select theOAuth consent screentab. Select anEmail address, enter aProduct nameif not already set, and click theSavebutton.Select theCredentialstab, click theCreate credentialsbutton and selectOAuth client ID.Select the application typeOther, enter the name "Singer BigQuery Tap", and click theCreatebutton.ClickOKto dismiss the resulting dialog.Click the Download button to the right of the client ID.Move this file to your working directory and rename itclient_secrets.json.Export the location of the secret file:export GOOGLE_APPLICATION_CREDENTIALS="./client_secret.json"For other authentication method, please see Authentication section.Step 2: InstallFirst, make sure Python 3 is installed on your system or follow these
installation instructions for Mac or Ubuntu.pip install -U tap-bigqueryOr you can install the lastest development version from GitHub:pip install --no-cache-dir https://github.com/anelendata/tap-bigquery/archive/master.tar.gz#egg=tap-bigqueryRunStep 1: ConfigureCreate a file called tap_config.json in your working directory, following
config.sample.json:{
  "streams": [
      {"name": "<some_schema_name>",
       "table": "`<project>.<dataset>.<table>`",
       "columns": ["<col_name_0>", "<col_name_1>", "<col_name_2>"],
       "datetime_key": "<your_key>",
       "filters": ["country='us'", "state='CA'",
                   "registered_on>=DATE_ADD(current_date, INTERVAL -7 day)"
                  ] // also optional: these are parsed in 'WHERE' clause
      }
    ],
  "start_datetime": "2017-01-01T00:00:00Z", // This can be set at the command line argument
  "end_datetime": "2017-02-01T00:00:00Z", // end_datetime is optional
  "limit": 100,
  "start_always_inclusive": false // default is false, optional
}The required parameters is at least one stream (one bigquery table/view) to copy.It is not a recommended BigQuery practice to use*to specify the columns
as it may blow up the cost for a table with a large number of columns.filtersare optional but we strongly recommend using this over a large
partitioned table to control the cost. LIMIT  (The authors of tap-bigquery is not
responsible for the cost incurred by running this program. Always test
thoroughly with small data set first.)start_datetimemust also be set in the config file or as the command line
argument (See the next step).limitwill limit the number of results, but it does not result in reduce
the query cost.The table/view is expected to have a column to indicate the creation or
update date and time so the tap sends the query withORDER BYand use
the column to record the bookmark (See State section).Step 2: Create catalogRun tap-bigquery in discovery mode to let it create json schema file and then
run them together, piping the output of tap-bigquery to target-csv:tap-bigquery -c tap_config.json -d > catalog.jsonStep 3: Runtap-bigquery can be run with any Singer Target. As example, let usetarget-csv.pip install target-csvRun:tap-bigquery -c tap_config.json \
    --catalog catalog.json --start_datetime '2020-08-01T00:00:00Z' \
    --end_datetime '2020-08-02T01:00:00Z' | target-csv --config target_config.json \
    > state.jsonThis example should create a csv file in the same directory.state.jsonshould contain a state (bookmark) after the run. (See State section).Notes:start and end datetimes accept ISO 8601 format, can be date only. start datetime
is inclusive, end datetime is not.It is recommended to inspect the catalog file and fix the auto-type assignment
if necessary.target-csv's target_config.json is optinal.tap-bigquery can produce nested records but it's up to target if the data
writing will be successful. In this example with target-csv, the table is
expected to be flat.AuthenticationIt is recommended to usetap-bigquerywith a service account.Download the client_secrets.json file for your service account, and place it
on the machine wheretap-bigquerywill be executed.Set aGOOGLE_APPLICATION_CREDENTIALSenvironment variable on the machine,
where the value is the fully qualified path to client_secrets.jsonIn environments where it is preferable to supply secrets or other configuration via
environment variables or where it is not feasible to make aclient_secrets.jsonfile available, you may instead provide credential as a JSON string in theGOOGLE_APPLICATION_CREDENTIALS_STRINGenvironment variable. This JSON string
should contain the same contents and be formatted the same way as the contents
of aclient_secrets.jsonfile.In the testing environment, you can also manually authenticate before runnig
the tap. In this case you do not needGOOGLE_APPLICATION_CREDENTIALSdefined:gcloud auth application-default loginYou may also have to set the project:gcloud config set project <project-id>Though not tested, it should also be possible to use the OAuth flow to
authenticate to GCP as well:tap-bigquerywill attempt to open a new window or tab in your default
browser. If this fails, copy the URL from the console and manually open it
in your browser.If you are not already logged into your Google account, you will be prompted
to log in.If you are logged into multiple Google accounts, you will be asked to select
one account to use for the authorization.Click theAcceptbutton to allowtap-bigqueryto access your Google BigQuery
table.You can close the tab after the signup flow is complete.StateThis tap emitsstate.
The command also takes a state file input with--state <file-name>option.
If the state is set, start_datetime config and command line argument are
ignored and the datetime value from last_update key is used as the resuming
point.To avoid the data duplication, start datetime is exclusivestart_datetime < datetime_columnwhen the tap runs with state option. If
you fear a data loss because of this, just use the--start_datetimeoption
instead of state. Or setstart_always_inclusive: truein configuration.The tap itself does not output a state file. It anticipate the target program
or a downstream process to fianlize the state safetly and produce a state file.Original repohttps://github.com/anelendata/tap-bigqueryAbout this projectThis project is developed by
ANELEN and friends. Please check out the ANELEN'sopen innovation philosophy and other projectsCopyright © 2020~ Anelen Co., LLC