Package: bigquery-test-kit
bigquery-test-kit, a testing framework to be more confident in your BigQuery SQLBigQuerydoesn't provide any locally runnabled server,
hence tests need to be run in Big Query itself.bigquery-test-kit enables Big Query testing by providing you an almost immutable DSL that allows you to :create and delete datasetcreate and delete table, partitioned or notload csv or json data into tablesrun query templatestransform json or csv data into a data literal or a temp tableYou can, therefore, test your query with data as literals or instantiate
datasets and tables in projects and load data into them.It's faster to run query with data as literals but using materialized tables is mandatory for some use cases.
In fact, data literal may add complexity to your request and therefore be rejected by BigQuery.
In such a situation, temporary tables may come to the rescue as they don't rely on data loading but on data literals.
Complexity will then almost be like you where looking into a real table.Immutability allows you to share datasets and tables definitions as a fixture and use it accros all tests,
adapt the definitions as necessary without worrying about mutations.In order to have reproducible tests, BQ-test-kit add the ability to create isolated dataset or table,
thus query's outputs are predictable and assertion can be done in details.
If you are running simple queries (no DML), you can use data literal to make test running faster.Template queries are rendered viavarsubstbut you can provide your own
interpolator by extendingbq_test_kit.interpolators.base_interpolator.BaseInterpolator. Supported templates are
those supported by varsubst, namely envsubst-like (shell variables) or jinja powered.In order to benefit from those interpolators, you will need to install one of the following extras,bq-test-kit[shell]orbq-test-kit[jinja2].UsageCommon setup with materialized tablesfromgoogle.cloud.bigquery.clientimportClientfrombq_test_kit.bq_test_kitimportBQTestKitfrombq_test_kit.bq_test_kit_configimportBQTestKitConfigfrombq_test_kit.resource_loaders.package_file_loaderimportPackageFileLoaderclient=Client(location="EU")bqtk_conf=BQTestKitConfig().with_test_context("basic")bqtk=BQTestKit(bq_client=client,bqtk_config=bqtk_conf)# project() uses default one specified by GOOGLE_CLOUD_PROJECT environment variablewithbqtk.project().dataset("my_dataset").isolate().clean_and_keep()asd:# dataset `GOOGLE_CLOUD_PROJECT.my_dataset_basic` is created# isolation is done via isolate() and the given context.# if you are forced to use existing dataset, you must use noop().## clean and keep will keep clean dataset if it exists before its creation.# Then my_dataset will be kept. This allows user to interact with BigQuery console afterwards.# Default behavior is to create and clean.schema=[SchemaField("f1",field_type="STRING"),SchemaField("f2",field_type="INT64")]withd.table("my_table",schema=schema).clean_and_keep()ast:# table `GOOGLE_CLOUD_PROJECT.my_dataset_basic.my_table` is created# noop() and isolate() are also supported for tables.pfl=PackageFileLoader("tests/it/bq_test_kit/bq_dsl/bq_resources/data_loaders/resources/dummy_data.csv")t.csv_loader(from_=pfl).load()result=bqtk.query_template(from_=f"select count(*) as nb from `{t.fqdn()}`").run()assertlen(result.rows)>0# table `GOOGLE_CLOUD_PROJECT.my_dataset_basic.my_table` is deleted# dataset `GOOGLE_CLOUD_PROJECT.my_dataset_basic` is deletedAdvanced setup with materialized tablesfromgoogle.cloud.bigquery.clientimportClientfrombq_test_kit.bq_test_kitimportBQTestKitfrombq_test_kit.bq_test_kit_configimportBQTestKitConfigfrombq_test_kit.resource_loaders.package_file_loaderimportPackageFileLoaderclient=Client(location="EU")bqtk_conf=BQTestKitConfig().with_test_context("basic")bqtk=BQTestKit(bq_client=client,bqtk_config=bqtk_conf)p=bqtk.project("it")\.dataset("dataset_foo").isolate()\.table("table_foofoo")\.table("table_foobar")\.project.dataset("dataset_bar").isolate()\.table("table_barfoo")\.table("table_barbar")\.projecttable_foofoo,table_foobar,table_barfoo,table_barbar=None,None,None,NonewithTables.from_(p,p)astables:# create datasets and tables in the order built with the dsl. Then, a tuples of all tables are returned.assertlen(tables)==4table_names=[t.namefortintables]asserttable_names==["table_foofoo","table_foobar","table_barfoo","table_barbar"]table_foofoo,table_foobar,table_barfoo,table_barbar=tablesasserttable_foofoo.show()isnotNoneasserttable_foobar.show()isnotNoneasserttable_barfoo.show()isnotNoneasserttable_barbar.show()isnotNoneSimple BigQuery SQL test with data literalsimportpytzfromdatetimeimportdatetimefromgoogle.cloud.bigquery.clientimportClientfromgoogle.cloud.bigquery.schemaimportSchemaFieldfrombq_test_kit.bq_test_kitimportBQTestKitfrombq_test_kit.bq_test_kit_configimportBQTestKitConfigfrombq_test_kit.data_literal_transformers.json_data_literal_transformerimportJsonDataLiteralTransformerfrombq_test_kit.interpolators.shell_interpolatorimportShellInterpolatorclient=Client(location="EU")bqtk_conf=BQTestKitConfig().with_test_context("basic")bqtk=BQTestKit(bq_client=client,bqtk_config=bqtk_conf)results=bqtk.query_template(from_="""SELECTf.foo, b.bar, e.baz, f._partitiontime as ptFROM${TABLE_FOO}fINNER JOIN ${TABLE_BAR}b ON f.foobar = b.foobarLEFT JOIN ${TABLE_EMPTY}e ON b.foobar = e.foobar""").with_datum({"TABLE_FOO":(['{"foobar": "1", "foo": 1, "_PARTITIONTIME": "2020-11-26 17:09:03.967259 UTC"}'],[SchemaField("foobar","STRING"),SchemaField("foo","INT64"),SchemaField("_PARTITIONTIME","TIMESTAMP")]),"TABLE_BAR":(['{"foobar": "1", "bar": 2}'],[SchemaField("foobar","STRING"),SchemaField("bar","INT64")]),"TABLE_EMPTY":(None,[SchemaField("foobar","STRING"),SchemaField("baz","INT64")])})\.as_data_literals()\.loaded_with(JsonDataLiteralTransformer())\.add_interpolator(ShellInterpolator())\.run()assertlen(results.rows)==1assertresults.rows==[{"foo":1,"bar":2,"baz":None,"pt":datetime(2020,11,26,17,9,3,967259,pytz.UTC)}]Simple BigQuery SQL test with temp tablesimportpytzfromdatetimeimportdatetimefromgoogle.cloud.bigquery.clientimportClientfromgoogle.cloud.bigquery.schemaimportSchemaFieldfrombq_test_kit.bq_test_kitimportBQTestKitfrombq_test_kit.bq_test_kit_configimportBQTestKitConfigfrombq_test_kit.data_literal_transformers.json_data_literal_transformerimportJsonDataLiteralTransformerfrombq_test_kit.interpolators.shell_interpolatorimportShellInterpolatorclient=Client(location="EU")bqtk_conf=BQTestKitConfig().with_test_context("basic")bqtk=BQTestKit(bq_client=client,bqtk_config=bqtk_conf)results=bqtk.query_template(from_="""SELECTf.foo, b.bar, e.baz, f._partitiontime as ptFROM${TABLE_FOO}fINNER JOIN ${TABLE_BAR}b ON f.foobar = b.foobarLEFT JOIN ${TABLE_EMPTY}e ON b.foobar = e.foobar""").with_datum({"TABLE_FOO":(['{"foobar": "1", "foo": 1, "_PARTITIONTIME": "2020-11-26 17:09:03.967259 UTC"}'],[SchemaField("foobar","STRING"),SchemaField("foo","INT64"),SchemaField("_PARTITIONTIME","TIMESTAMP")]),"TABLE_BAR":(['{"foobar": "1", "bar": 2}'],[SchemaField("foobar","STRING"),SchemaField("bar","INT64")]),"TABLE_EMPTY":(None,[SchemaField("foobar","STRING"),SchemaField("baz","INT64")])})\.loaded_with(JsonDataLiteralTransformer())\.add_interpolator(ShellInterpolator())\.run()assertlen(results.rows)==1assertresults.rows==[{"foo":1,"bar":2,"baz":None,"pt":datetime(2020,11,26,17,9,3,967259,pytz.UTC)}]More usage can be found init tests.ConceptsResource LoadersCurrently, the only resource loader available isbq_test_kit.resource_loaders.package_file_loader.PackageFileLoader.
It allows you to load a file from a package, so you can load any file from your source code.You can implement yours by extendingbq_test_kit.resource_loaders.base_resource_loader.BaseResourceLoader.
If so, please create a merge request if you think that yours may be interesting for others.InterpolatorsInterpolators enable variable substitution within a template.
They lay on dictionaries which can be in a global scope or interpolator scope.
While rendering template, interpolator scope's dictionary is merged into global scope thus,
interpolator scope takes precedence over global one.You can benefit from two interpolators by installing the extrasbq-test-kit[shell]orbq-test-kit[jinja2].
Those extra allows you to render you query templates with envsubst-like variable or jinja.
You can define yours by extendingbq_test_kit.interpolators.BaseInterpolator.fromgoogle.cloud.bigquery.clientimportClientfrombq_test_kit.bq_test_kitimportBQTestKitfrombq_test_kit.bq_test_kit_configimportBQTestKitConfigfrombq_test_kit.resource_loaders.package_file_loaderimportPackageFileLoaderfrombq_test_kit.interpolators.jinja_interpolatorimportJinjaInterpolatorfrombq_test_kit.interpolators.shell_interpolatorimportShellInterpolatorclient=Client(location="EU")bqtk_conf=BQTestKitConfig().with_test_context("basic")bqtk=BQTestKit(bq_client=client,bqtk_config=bqtk_conf)result=bqtk.query_template(from_="select ${NB_USER}as nb")\.with_global_dict({"NB_USER":"2"})\.add_interpolator(ShellInterpolator())\.run()assertlen(result.rows)==1assertresult.rows[0]["nb"]==2result=bqtk.query_template(from_="select {{NB_USER}} as nb")\.with_interpolators([JinjaInterpolator({"NB_USER":"3"})])\.run()assertlen(result.rows)==1assertresult.rows[0]["nb"]==3Data loadersSupported data loaders are csv and json only even if Big Query API support more.
Data loaders were restricted to those because they can be easily modified by a human and are maintainable.If you need to support more, you can still load data by instantiatingbq_test_kit.bq_dsl.bq_resources.data_loaders.base_data_loader.BaseDataLoader.Data Literal TransformersSupported data literal transformers are csv and json.
Data Literal Transformers can be less strict than their counter part, Data Loaders.
In fact, they allow to use cast technique to transform string to bytes or cast a date like to its target type.
Furthermore, in json, another format is allowed, JSON_ARRAY.
This allows to have a better maintainability of the test resources.Data Literal Transformers allows you to specify _partitiontime or _partitiondate as well,
thus you can specify all your data in one file and still matching the native table behavior.
If you were using Data Loader to load into an ingestion time partitioned table,
you would have to load data into specific partition.
Loading into a specific partition make the time rounded to 00:00:00.If you need to support a custom format, you may extend BaseDataLiteralTransformer
to benefit from the implemented data literal conversion.bq_test_kit.data_literal_transformers.base_data_literal_transformer.BaseDataLiteralTransformer.Resource strategiesDataset and table resource management can be changed with one of the following :Noop : don't manage the resource at allCleanBeforeAndAfter : clean before each creation and after each usage.CleanAfter : create without cleaning first and delete after each usage. This is the default behavior.CleanBeforeAndKeepAfter : clean before each creation and don't clean resource after each usage.The DSL on dataset and table scope provides the following methods in order to change resource strategy :noop : set to Noopclean_and_keep : set to CleanBeforeAndKeepAfterwith_resource_strategy : set to any resource strategy you wantContributionsContributions are welcome. You can create issue to share a bug or an idea.
You can create merge request as well in order to enhance this project.Local testingTesting is separated in two parts :unit testing : doesn't need interaction with Big Queryintegration testing : validate behavior against Big QueryIn order to run test locally, you must installtox.After that, you are able to run unit testing withtox -e clean, py36-utfrom the root folder.If you plan to run integration testing as well, please use a service account and authenticate yourself withgcloud auth application-default loginwhich will setGOOGLE_APPLICATION_CREDENTIALSenv var. You will have to setGOOGLE_CLOUD_PROJECTenv var as well in order to runtox.Thanks.WARNINGDSL may change with breaking change until release of 1.0.0.Editing in VSCodeIn order to benefit from VSCode features such as debugging, you should type the following commands in the root folder of this project.Linuxpython3 -m venv .venvsource .venv/bin/activatepip3 install -r requirements.txt -r requirements-test.txt -e .Windowspy -3 -m venv .venv.venv\scripts\activatepython -m pip install -r requirements.txt -r requirements-test.txt -e .Changelog0.5.0add clustering support to tables (thanks to @ckchow)0.4.3rename project as python-bigquery-test-kit0.4.2remove load job config in logfix temp tables query generation0.4.1fix empty array generation for data literals0.4.0add ability to rely on temp tables or data literals with query template DSLfix generate empty data literal when json array is empty0.3.1change method typo of with_interpolatorsfix README.md related to interpolators0.3.0add data literal transformer package exports0.2.0Change return type ofTables.__enter__(closes #6)Make jinja's local dictionary optional (closes #7)Wrap query result into BQQueryResult (closes #9)0.1.2Fix time partitioning type in TimeField (closes #3)0.1.1Fix table reference in Dataset (closes #2)0.1.0BigQuery resource DSL to create dataset and table (partitioned or not)isolation of dataset and table namesresults as dict with ease of test on byte arrays. Decoded as base64 string.query templates interpolation made easycsv and json loading into tables, including partitioned one, from code based resources.context manager for cascading creation of BQResourceresource definition sharing accross tests made possible with "immutability".