Package: macrometa-target-bigquery
macrometa-target-bigqueryMacrometa target bigquery connector that loads data into BigQuery following theSinger spec.How to use itIf you want to run this target connector independently please read further.InstallFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:makevenvTo runLike any other target connector that's following the singer specification:some-singer-source(tap) | macrometa-target-bigquery --config [config.json]It's reading incoming messages from STDIN and using the properties inconfig.jsonto upload data into BigQuery.Note: To avoid version conflicts runsourceandtargetsin separate virtual environments.Configuration settingsRunning the the target connector requires aconfig.jsonfile. An example with the minimal settings:{"project_id":"mygbqproject"}Full list of options inconfig.json:PropertyTypeRequired?Descriptionproject_idStringYesBigQuery projectlocationStringRegion where BigQuery stores your datasetdefault_target_schemaStringName of the schema where the tables will be created. Ifschema_mappingis not defined then every stream sent by the tap is loaded into this schema.default_target_schema_select_permissionStringGrant USAGE privilege on newly created schemas and grant SELECT privilege on newly createdschema_mappingObjectUseful if you want to load multiple streams from one source to multiple BigQuery schemas.If the source sends thestream_idin<schema_name>-<table_name>format then this option overwrites thedefault_target_schemavalue. Note, that usingschema_mappingyou can overwrite thedefault_target_schema_select_permissionvalue to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.batch_size_rowsInteger(Default: 100000) Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into BigQuery.batch_wait_limit_secondsInteger(Default: None) Maximum time to wait for batch to reachbatch_size_rows.flush_all_streamsBoolean(Default: False) Flush and load every stream into BigQuery when one batch is full. Warning: This may trigger transfer of data with low number of records, and may cause performance problems.parallelismInteger(Default: 0) The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max.max_parallelismInteger(Default: 16) Max number of parallel threads to use when flushing tables.add_metadata_columnsBoolean(Default: False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in bigquery etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix_sdc_. The column names are following the stitch naming conventions documented athttps://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the_sdc_deleted_atmetadata column. Without theadd_metadata_columnsoption the deleted rows from sources will not be recognisable in BigQuery.hard_deleteBoolean(Default: False) Whenhard_deleteoption is true then DELETE SQL commands will be performed in BigQuery to delete rows in tables. It's achieved by continuously checking the_sdc_deleted_atmetadata column sent by the source. Due to deleting rows requires metadata columns,hard_deleteoption automatically enables theadd_metadata_columnsoption as well.hard_delete_mappingObjectThis is useful if you want to sethard_deletefor some streams but not others. This should contain a mapping ofstream_id: <Boolean>. This boolean will override the default behaviour set withhard_deletefor that stream. If a stream is not defined inhard_delete_mappingit will behave according tohard_delete. Whenhard_deleteoption is true then DELETE SQL commands will be performed in BigQuery to delete rows in tables. It's achieved by continuously checking the_sdc_deleted_atmetadata column sent by the singer source. Due to deleting rows requires metadata columns,hard_deleteoption automatically enables theadd_metadata_columnsoption as well.data_flattening_max_levelInteger(Default: 0) Object type RECORD items from sources can be loaded into VARIANT columns as JSON (default) or we can flatten the schema by creating columns automatically.When value is 0 (default) then flattening functionality is turned off.primary_key_requiredBoolean(Default: True) Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined.validate_recordsBoolean(Default: False) Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by BigQuery. Enabling this option will detect invalid records earlier but could cause performance degradation.temp_schemaStringName of the schema where the temporary tables will be created. Will default to the same schema as the target tablesuse_partition_pruningBoolean(Default: False) Iftruethen BigQuery table partition pruning will be used for tables which have partitioning enabled. This partitioning should be on a column which is immutable such as an integer primary key or acreated_atcolumn. The partitioning should be set up manually by the user. This feature can dramatically reduce the cost of eachMERGEfor large tables.Schema ChangesThis macrometa target connector does follow thePipelineWise specificationfor schema changes except versioning columns because of the way BigQuery works.BigQuery does not allow for column renames so a column modification works like this instead:Versioning columnsTarget connectors are versioning columnswhen data type change is detectedin the source
table. Versioning columns means that the old column with the old datatype is kept
and a new column is created by adding a suffix to the name depending of the type (and also a timestamp for struct and arrays)
to the column name with the new data type. This new column will be added to the table.For example if the data type ofCOLUMN_THREEchanges fromINTEGERtoVARCHARPipelineWise will replicate data in this order:Before changing data typeCOLUMN_THREEisINTEGERjust like in in source table:COLUMN_ONECOLUMN_TWOCOLUMN_THREE(INTEGER)texttext1texttext2texttext3After the data type changeCOLUMN_THREEremainsINTEGERwith
the old data and a newCOLUMN_TREE__stcolumn created withSTRINGtype that keeps
data only after the change.COLUMN_ONECOLUMN_TWOCOLUMN_THREE(INTEGER)COLUMN_THREE__st(VARCHAR)texttext111texttext222texttext333texttext444-ABCtexttext555-DEF.. warning::Please note theNULLvalues inCOLUMN_THREEandCOLUMN_THREE__stcolumns.Historical values are not converted to the new data types!If you need the actual representation of the table after data type changes then
you need to resync the table.Column clusteringThis target connector tries to speed up the querying of the resulting tables by clustering the
columns in each table by the primary key of the stream.The choice and ordering of the clustering keys are defined in the same order as thekey_propertiescolumns in the stream'sSCHEMAmessages.Bigquery places a limit on the number of clustering keys (4 as of 2022-08-02), so if the
number of clustering keys is greater than 4, this target will simply use the first 4
columns defined inkey_propertiesproperty.To run tests:Define environment variables that requires running the testsexport GOOGLE_APPLICATION_CREDENTIALS=<credentials-json-file>
  export MACROMETA_TARGET_BIGQUERY_PROJECT=<bigquery project to run your tests on>
  export MACROMETA_TARGET_BIGQUERY_SCHEMA=<temporary schema for running the tests>Install python dependencies in a virtual env and run nose unit and integration testsmake venvTo run unit tests:make unit_testTo run integration tests:make integration_testTo run pylint:Install python dependencies and run python lintermake venv pylintLicenseApache License Version 2.0SeeLICENSEto see the full text.