Package: macrometa-source-bigquery
macrometa-source-bigqueryExtract data from BigQuery tables.This is a macrometa source bigquery connector that produces JSON-formatted data
following theSinger spec.This tap:Pulls data from Google BigQuery tables/views.Infers the schema for each resource and produce catalog file.InstallationStep 1: Activate the Google BigQuery API(originally found in theGoogle API docs)Usethis wizardto create or select a project in the Google Developers Console and activate the BigQuery API. Click Continue, then Go to credentials.On theAdd credentials to your projectpage, click theCancelbutton.At the top of the page, select theOAuth consent screentab. Select anEmail address, enter aProduct nameif not already set, and click theSavebutton.Select theCredentialstab, click theCreate credentialsbutton and selectOAuth client ID.Select the application typeOther, enter the name "Macrometa Source BigQuery", and click theCreatebutton.ClickOKto dismiss the resulting dialog.Click the Download button to the right of the client ID.Move this file to your working directory and rename itclient_secrets.json.Export the location of the secret file:export GOOGLE_APPLICATION_CREDENTIALS="./client_secret.json"For other authentication method, please see Authentication section.Step 2: InstallFirst, make sure Python 3 is installed on your system or follow these
installation instructions for Mac or Ubuntu.pip install -U macrometa-source-bigqueryRunStep 1: ConfigureCreate a file called tap_config.json in your working directory, following
config.sample.json:{
  "streams": [
      {"name": "<some_schema_name>",
       "table": "`<project>.<dataset>.<table>`",
       "columns": ["<col_name_0>", "<col_name_1>", "<col_name_2>"],
       "datetime_key": "<your_key>",
       "filters": ["country='us'", "state='CA'",
                   "registered_on>=DATE_ADD(current_date, INTERVAL -7 day)"
                  ] // also optional: these are parsed in 'WHERE' clause
      }
    ],
  "start_datetime": "2017-01-01T00:00:00Z", // This can be set at the command line argument
  "end_datetime": "2017-02-01T00:00:00Z", // end_datetime is optional
  "limit": 100,
  "start_always_inclusive": false // default is false, optional
}The required parameters is at least one stream (one bigquery table/view) to copy.It is not a recommended BigQuery practice to use*to specify the columns
as it may blow up the cost for a table with a large number of columns.filtersare optional but we strongly recommend using this over a large
partitioned table to control the cost. LIMIT  (The authors of tap-bigquery is not
responsible for the cost incurred by running this program. Always test
thoroughly with small data set first.)start_datetimemust also be set in the config file or as the command line
argument (See the next step).limitwill limit the number of results, but it does not result in reduce
the query cost.The table/view is expected to have a column to indicate the creation or
update date and time so the tap sends the query withORDER BYand use
the column to record the bookmark (See State section).Step 2: Create catalogRun tap-bigquery in discovery mode to let it create json schema file and then
run them together, piping the output of macrometa-source-bigquery to target-csv:macrometa-source-bigquery -c tap_config.json -d > catalog.jsonStep 3: Runmacrometa-source-bigquery can be run with any Target. As example, let usetarget-csv.pip install target-csvRun:macrometa-source-bigquery -c tap_config.json \
    --catalog catalog.json --start_datetime '2020-08-01T00:00:00Z' \
    --end_datetime '2020-08-02T01:00:00Z' | target-csv --config target_config.json \
    > state.jsonThis example should create a csv file in the same directory.state.jsonshould contain a state (bookmark) after the run. (See State section).Notes:start and end datetimes accept ISO 8601 format, can be date only. start datetime
is inclusive, end datetime is not.It is recommended to inspect the catalog file and fix the auto-type assignment
if necessary.target-csv's target_config.json is optinal.macrometa-source-bigquery can produce nested records but it's up to target if the data
writing will be successful. In this example with target-csv, the table is
expected to be flat.AuthenticationIt is recommended to usemacrometa-source-bigquerywith a service account.Download the client_secrets.json file for your service account, and place it
on the machine wheremacrometa-source-bigquerywill be executed.Set aGOOGLE_APPLICATION_CREDENTIALSenvironment variable on the machine,
where the value is the fully qualified path to client_secrets.jsonIn the testing environment, you can also manually authenticate before runnig
the tap. In this case you do not needGOOGLE_APPLICATION_CREDENTIALSdefined:gcloud auth application-default loginYou may also have to set the project:gcloud config set project <project-id>Though not tested, it should also be possible to use the OAuth flow to
authenticate to GCP as well:macrometa-source-bigquerywill attempt to open a new window or tab in your default
browser. If this fails, copy the URL from the console and manually open it
in your browser.If you are not already logged into your Google account, you will be prompted
to log in.If you are logged into multiple Google accounts, you will be asked to select
one account to use for the authorization.Click theAcceptbutton to allowmacrometa-source-bigqueryto access your Google BigQuery
table.You can close the tab after the signup flow is complete.StateThis source connector emitsstate.
The command also takes a state file input with--state <file-name>option.
If the state is set, start_datetime config and command line argument are
ignored and the datetime value from last_update key is used as the resuming
point.To avoid the data duplication, start datetime is exclusivestart_datetime < datetime_columnwhen the source connector runs with state option. If
you fear a data loss because of this, just use the--start_datetimeoption
instead of state. Or setstart_always_inclusive: truein configuration.The source connector itself does not output a state file. It anticipate the target program
or a downstream process to fianlize the state safetly and produce a state file.Original repohttps://github.com/Macrometacorp/macrometa-source-bigquery