Package: z3-target-bigquery
Target-BigQueryA rare 💎 you have stumbled upontarget-bigqueryis a Singer target for BigQuery.It is the most versatile target for BigQuery. Extremely performant, resource efficient, and fast in all configurations enabling 20 different ingestion patterns. Denormalized variants indicate data is unpacked during load with a resultant schema in BigQuery based on the tap schema. Non-denormalized means we have a fixed schema which loads all data into an unstructuredJSONcolumn. They are both useful patterns. The latter allowing BigQuery to work with schemaless or rapidly changing sources such as MongoDB instantly, while the former is more performant and convenient to start modeling quickly.Patterns🛠 (more details below)Batch Job, Denormalized, OverwriteBatch Job, Denormalized, UpsertBatch Job, Denormalized, AppendBatch Job, Fixed Schema, OverwriteBatch Job, Fixed Schema, AppendGCS Staging Data Lake, Denormalized, OverwriteGCS Staging Data Lake, Denormalized, UpsertGCS Staging Data Lake, Denormalized, AppendGCS Staging Data Lake, Fixed Schema, OverwriteGCS Staging Data Lake, Fixed Schema, AppendStorage Write API, Denormalized, OverwriteStorage Write API, Denormalized, UpsertStorage Write API, Denormalized, AppendStorage Write API, Fixed Schema, OverwriteStorage Write API, Fixed Schema, AppendLegacy Streaming API, Denormalized, OverwriteLegacy Streaming API, Denormalized, UpsertLegacy Streaming API, Denormalized, AppendLegacy Streaming API, Fixed Schema, OverwriteLegacy Streaming API, Fixed Schema, AppendInstallation 📈The package on pypi is namedz3-target-bigquerybut the executable it ships with is simplytarget-bigquery. This allows me to release work without concerns of naming conflicts on the package index.# Use pipx or pippipxinstallz3-target-bigquery# Verify it is installedtarget-bigquery--versionUsage NotesDenormalizationSo denormalized is a loaded term, so lets clarify here what we mean by it:In the context of JSON objects, denormalization refers to the process of flattening a hierarchical or nested JSON object into a simpler, more "denormalized" structure that can be easier to work with for certain use cases.Denormalized=False (default) means we load all data into a singleJSONcolumn. This means all access requires an accessor such asSELECT data.my_column FROM tableinstead ofSELECT my_column FROM table. Hence the term denormalized is relative to thedatacolumn which is the default for this target. This is a tradeoff between convenience/resilience and performance. This is the default because mostresilient. IE a load willneverfail due to a schema change or an invalid schema. You can always denormalize later viadbt. However, it is not the most performant and slower to transform. If your tap has a high quality and consistent schema, denormalization is the way to go to get the best performance and start modeling quickly.Denormalized=True means we unpack the data into a schema which is derived from the tap schema. It doesnotmean we will flatten the data. There is a separate option for flattening. We will convert arrays to repeated fields and records to structs. All top level keys will end up as columns which is was you might expect from more typical targets.Resolver VersionsThere are 2 resolver versions. The config optionschema_resolver_versionlets you select which version you want to use. This versioning exists because we want to support evolving how we resolve schemas whilst not creating breaking changes for long-time users dependent on how a schema is resolved. The default is1which behaves very similarly to existing flavors oftarget-bigquery. It works well enough but has plenty of edge cases where it simply cannot resolve valid jsonschemas to a bq schema. The new version2is much more robust and will resolve most, if not all schemas due to it falling back toJSONwhen in doubt. You must opt-in to this version by settingschema_resolver_version: 2in your config.Overwrite vs AppendSometimes you want to overwrite a table on every load. This can be achieved by either settingoverwrite: truewhich will full refresh ALL tablesorsettingoverwrite: [table1, table2, table_*_other, !table_v1_other]which will only overwrite the specified tables and supports pattern matching. This is useful if you have a table which is a lookup table and you want to overwrite it on every load. You can also setoverwrite: falsewhich will append to the table. This is the default behavior.Upsert (aka Merge)If you want to merge data into a table, you can setmerge: truewhich will use theMERGEstatement to upsert data. This supports pattern matching like the above setting. It requiresdenormalized: truetakes precedence overoverwrite. It will only work on tables which have a primary key as defined by thekey_propertiessent by the tap. There is a supporting config option calleddedupe_before_upsertwhich will dedupe the data before upserting. This is useful if you are replicating data which has a primary key but is not unique. This occurs when you are replicating data from a source which has a primary key but does not enforce it. This is the case for MongoDB. It can also happen when moving data from a data lake in S3/GCS to a database. This is not the default behavior because it is slower and requires more resources.Features ✨Autoscaling self-healing worker pool using either threads (default) or multiprocessing, configurable by the user for thefastestpossible data ingestion. Particularly when leveraging colocated compute in GCP.Denormalized load pattern where data is unpacked in flight into a statically typed BigQuery schema derived from the input stream json schemas.Fix schema load pattern where all data is loaded into aJSONcolumn which has been GA in BigQuery since mid 2022.AutogeneratedVIEWsupport for fixed schema load patterns which essentially overlays a statically typed schema allowing you to get the best of both worlds when using fixed schema ingestion.JIT compilation of protobuf schemas allowing the Storage Write API to use a denormalized load pattern.BATCH message support 😎Load Patterns 🏎Batch Load Jobingestion pattern using in memory compression (fixed schema + denormalized)Storage Write APIingestion pattern using gRPC and protocol buffers supporting both streaming and batch patterns. Capable of JIT compilation of BQ schemas to protobuf to enable denormalized loads of input structures only known at runtime. (fixed schema + denormalized 🎉)GCS Stagingingestion pattern using in memory compression and a GCS staging layer which generates a well organized data lake which backs the data warehouse providing additional failsafes and data sources (fixed schema + denormalized)Legacy Streamingingestion pattern which emphasizes simplicity and fast start up / tear down. I would highly recommend the storage write API instead unless data volume is small (fixed schema + denormalized)Choosing between denormalized and fixed schema (JSON support)?🙇🏾‍♂️The gap between the methods is closed due in part to the target's ability to  automatically generating aVIEWwhich will unpack (or rather provide typing as a more accurate take) a JSON based ingestion source for you. Unless operating at tens of millions of rows with JSON objects containing multiple hundreds of keys, its quite performant. Particularly if accessing a small subset of keys. It does however fall off (quite hard) given enough scale as I mentioned (I've pushed it to the limits). Denormalized is recommended for high volume where the schema is fairly consistent. Fixed is recommended for lower volume, inconsistent schemas or for taps which are inherently schemaless in which case its the ideal (only...) logical pattern. Fixed schema can also be used for taps which routinely break down on BQ due to json schemas being inexpressible in a static way (ie patternProperties, additionalProperties...)Old Header (still true, here for posterity)🧪This is the first truly unstructured sink for BigQuery leveraging the recent GA feature in BigQuery for JSON support. This allows this target to load from essentially any tap regardless of the quality or explicitness of its jsonschema. Observations in existing taps note things such aspatternPropertiesused in jsonschema objects which break down on all existing BigQuery taps due to the previous need for strong typing. Also taps such as MongoDB which inherently deal with unstructured data are seamlessly enabled by this target without klutzy collection scraping of a sample of records which wehopeare repesentative of all documents.Built with theMeltano Target SDK.Configuration 🔨SettingsFirst a valid example to give context to the below including a nested key example (denoted via a.in the setting path) as seen withcolumn_name_transforms.snake_case{"project":"my-bq-project","method":"storage_write_api","denormalized":true,"credentials_path":"...","dataset":"my_dataset","location":"us-central1","batch_size":500,"column_name_transforms":{"snake_case":true}}SettingRequiredDefaultDescriptioncredentials_pathFalseNoneThe path to a gcp credentials json file.credentials_jsonFalseNoneA JSON string of your service account JSON file.projectTrueNoneThe target GCP project to materialize data into.datasetTrueNoneThe target dataset to materialize data into.locationFalseUSThe target dataset location to materialize data into. Applies also to the GCS bucket if usinggcs_stageload method.batch_sizeFalse500The maximum number of rows to send in a single batch to the worker. This should be configured based on load method. Forstorage_write_apiandstreaming_insertit should be<=500, for the LoadJob sinks, it can be much higher, ie>100,000timeoutFalse600Default timeout for batch_job and gcs_stage derived LoadJobs.fail_fastFalseTrueFail the entire load job if any row fails to insert.denormalizedFalseFalseDetermines whether to denormalize the data before writing to BigQuery. A false value will write data using a fixed JSON column based schema, while a true value will write data using a dynamic schema derived from the tap.methodTruestorage_write_apiThe method to use for writing to BigQuery. Must be one ofbatch_job,storage_write_api,gcs_stage,streaming_insertgenerate_viewFalseFalseDetermines whether to generate a view based on the SCHEMA message parsed from the tap. Only valid if denormalized=false meaning you are using the fixed JSON column based schema.upsertFalseFalseDetermines if we should upsert. Defaults to false. A value of true will write to a temporary table and then merge into the target table (upsert). This requires the target table to be unique on the key properties. A value of false will write to the target table directly (append). A value of an array of strings will evaluate the strings in order using fnmatch. At the end of the array, the value of the last match will be used. If not matched, the default value is false (append).overwriteFalseFalseDetermines if the target table should be overwritten on load. Defaults to false. A value of true will write to a temporary table and then overwrite the target table inside a transaction (so it is safe). A value of false will write to the target table directly (append). A value of an array of strings will evaluate the strings in order using fnmatch. At the end of the array, the value of the last match will be used. If not matched, the default value is false. This is mutually exclusive with theupsertoption. If both are set,upsertwill take precedence.dedupe_before_upsertFalseFalseThis option is only used ifupsertis enabled for a stream. The selection criteria for the stream's candidacy is the same as upsert. If the stream is marked for deduping before upsert, we will create a _session scoped temporary table during the merge transaction to dedupe the ingested records. This is useful for streams that are not unique on the key properties during an ingest but are unique in the source system. Data lake ingestion is often a good example of this where the same unique record may exist in the lake at different points in time from different extracts.bucketFalseNoneThe GCS bucket to use for staging data. Only used if method is gcs_stage.cluster_on_key_propertiesFalse0Determines whether to cluster on the key properties from the tap. Defaults to false. When false, clustering will be based on _sdc_batched_at instead.partition_granularityFalse"month"Indicates the granularity of the created table partitioning scheme which is based on_sdc_batched_at. By default the granularity is monthly. Must be one of: "hour", "day", "month", "year".column_name_transforms.lowerFalseNoneLowercase column names.column_name_transforms.quoteFalseNoneQuote column names in any generated DDL.column_name_transforms.add_underscore_when_invalidFalseNoneAdd an underscore to the column name if it starts with a digit to make it valid.column_name_transforms.snake_caseFalseNoneSnake case all incoming column names. Does not apply to fixed schema loads butdoesapply to the view auto-generated over them.options.storage_write_batch_modeFalseNoneBy default, we use the default stream (Committed mode) in thestorage_write_apiload method which results in streaming records which are immediately available and is generally fastest. If this is set to true, we will use the application created streams (pending mode) to transactionally batch data on STATE messages and at end of pipe.options.process_poolFalseNoneBy default we use an autoscaling threadpool to write to BigQuery. If set to true, we will use a process pool.options.max_workersFalseNoneBy default, each sink type has a preconfigured max worker pool limit. This sets an override for maximum number of workers in the pool.schema_resolver_versionFalse1The version of the schema resolver to use. Defaults to 1. Version 2 uses JSON as a fallback during denormalization. This only has an effect if denormalized=truestream_mapsFalseNoneConfig object for stream maps capability. For more information check outStream Maps.stream_map_configFalseNoneUser-defined config values to be used within map expressions.flattening_enabledFalseNone'True' to enable schema flattening and automatically expand nested properties.flattening_max_depthFalseNoneThe max depth to flatten schemas.A full list of supported settings and capabilities is available by running:target-bigquery --aboutConfigure using environment variables ✏️This Singer target will automatically import any environment variables within the working directory's.envif the--config=ENVis provided, such that config values will be considered if a matching
environment variable is set either in the terminal context or in the.envfile.Source Authentication and Authorization 👮🏽‍♂️Authenticate via service account key file or Application Default Credentials (ADC)https://cloud.google.com/bigquery/docs/authenticationCapabilities ✨aboutstream-mapsschema-flatteningbatchUsage 👷‍♀️You can easily runtarget-bigqueryby itself or in a pipeline usingMeltano.Executing the Target Directly 🚧target-bigquery--version
target-bigquery--help# Test using the "Carbon Intensity" sample:tap-carbon-intensity|target-bigquery--config/path/to/target-bigquery-config.jsonDeveloper Resources 👩🏼‍💻Initialize your Development Environmentpipxinstallpoetry
poetryinstallCreate and Run TestsCreate tests within thetarget_bigquery/testssubfolder and
then run:poetryrunpytestYou can also test thetarget-bigqueryCLI interface directly usingpoetry run:poetryruntarget-bigquery--helpTesting withMeltanoNote:This target will work in any Singer environment and does not require Meltano.
Examples here are for convenience and to streamline end-to-end orchestration scenarios.Next, install Meltano (if you haven't already) and any needed plugins:# Install meltanopipxinstallmeltano# Initialize meltano within this directorycdtarget-bigquery
meltanoinstallNow you can test and orchestrate using Meltano:# Test invocation:meltanoinvoketarget-bigquery--version# OR run a test `elt` pipeline with the Carbon Intensity sample tap:meltanoelttap-carbon-intensitytarget-bigquerySDK Dev GuideSee thedev guidefor more instructions on how to use the Meltano SDK to
develop your own Singer taps and targets.