Package: bq-schema
bq-schemaMotivationAt limehome we are heavy users of python and bigquery. This library was created to mainly solve the following issues:Define table schemas in code and have a migration script to apply changes.On deploy make sure that all schemas were applied, otherwise abort.Guarantee that when we try to write data to a table, the data matches the schema of the table (required / optional, datatypes)Version our tables and enable migrations to a new schemaAdditionally this library aims to help the users through the usage ofpython typing.Specify your schema as apython dataclassOur migration script converts the data class into a bigquery schema definitionDeserialize rows into a dataclass instance, while reading from a tableSerialize a dataclass instance into a dictionary and write it to the table.The main benefit of combining all these features is, that we can guarantee that our code will run, before we deploy to production.QuickstartSince this library makes use of newer features of python, you need at least python3.7.Install the packagepip install bq_schemaCreate a schema and a table definition in my_table.py@dataclassclassSchema:string_field:str=field(metadata={"description":"This is a STRING field."})int_field:Optional[int]some_floats:List[float]bool_field:boolclassMyTable(BigqueryTable):name="my_table_name"schema=SchemaIf you have already tables created in your account, you can use the convert-table script to create a schema.Note: The script produces a file which is meant to be a starting point. You will most likely have to add some imports yourself!Create your tableHint:Make sure to have you credentials set:export GOOGLE_APPLICATION_CREDENTIALS=your_auth.jsonAlternativly you can set the service_file as a environment variable:export GOOGLE_SERVICE_FILE={"type": "service_account", ...}Now create your tablemigrate-tables --project my_project --dataset my_dataset --module-path my_table --applyWrite a rowfromgoogle.cloudimportbigqueryrow=Schema(string_field="foo",int_field=1,some_floats=[1.0,2.0],bool_field=True)row_transformer=RowTransformer[Schema](Schema)serialized_row=RowTransformer.dataclass_instance_to_bq_row(row)bigquery_client=bigquery.Client()table=bigquery_client.get_table("project.dataset.my_table_name")bigquery_client.insert_rows(table,[serialized_row])Validate you code with a type checker likemypymypy my_table.pyRead a rowquery="SELECT * FROM project.dataset.my_table_name"forrowinbigquery_client.query(query=query):deserialized_row=row_transformer.bq_row_to_dataclass_instance(row)assertisinstance(deserialized_row,Schema)DocumentationSchema definitionsFor a full list of supported types check the following schema:fromtypingimportOptionalfromdataclassesimportdataclassfrombq_schema.types.type_mappingimportTimestamp,Geography@dataclassclassRequiredNestedField:int_field:int=field(metadata={"description":"This field is an INT field."})@dataclassclassRequiredSchema:string_field:str=field(metadata={"description":"This field is a STRING field."})string_field_optional=Optional[str]bytes_field:bytesint_field:intfloat_field:floatnumeric_field:Decimalbool_field:booltimestamp_field:Timestampdate_field:datetime_field:timedatetime_field:datetimegeography_field:Geographyrequired_nested_field:RequiredNestedField=field(metadata={"description":"This field is a STRUCT field."})optional_nested_field:Optional[RequiredNestedField]repeated_nested_field:List[RequiredNestedField]Abstract tablesIf you want to have an class that inherits from BigqueryTable but does not actually map to a table in BigQuery (is abstract, common interface etc.), you can have it inherit from ABC and it will not be discovered if you pass the flag "--ignore-abstract"migrate-tables ... --ignore-abstractclassSomeInterface(BigqueryTable,ABC):passclassConcreteImplementation(SomeInterface):name="Some value here"schema=SomeSchemaWithout the flag, this would fail due to name and schema being required in "SomeInterface"TimestampsTimestamps are deserialized into datetime objects, due to the nature of the underlying bq library. To distinguish between datetime and timestamp use bq_schema.types.type_mapping.
Usage:frombq_schema.types.type_mappingimportTimestampfromdatetimeimportdatetimethe_timestamp=Timestamp(datetime.utcnow())GeographyThis library treats the geography data type as a string. BigQuery accepts geography values either in theWKTorGeoJsonformat. To actually parse and work with geodata in python, one could use theshapelylibrary. Here is an example how to load a point from the WKT format:fromshapely.wktimportloadsloads('POINT (0 0)')Table definitionsThe bigquery class is used for:Recursive table discovery by our migrate-tables scriptDefine table properties like name and schemaRequired propertiesname: The name of the tableschema: table schema either as dataclass or a list of schema fieldsOptional propertiesproject: name of the project, can be overwritten by the migrate-tables scriptdataset: name of the dataset, can be overwritten by the migrate-tables scriptVersioning tablesSince bigquery does not allow backwards incompatible schema changes, you might want to version your schemas.projectclassMyTable(BigqueryTable):name="my_table_name"schema=Schemaversion="1"By default the version will be appended to the table name, like so: my_table_name_v1. If you want to overwrite this behaviour,
you can implement the full_table_name method.Time partitioningDefine time partitioning for your table:frombq_schema.types.type_mappingimportTimestampfromgoogle.cloud.bigqueryimportTimePartitioning,TimePartitioningTypeclassMyTable:time_partitioning=TimePartitioning(type_=TimePartitioningType.DAY,field="some_column")Scriptsmigrate-tablesThis script has two uses:Check if locally defined schemas are in sync with the schemas in bigqueryIf a difference is detected, we try to apply the changesThe script will find all defined tables recursivly for a given python module.Note: If you have not defined your project and / or dataset in code, you will have to pass it as a parameter to the script.
Show the help:migrate-tables --helpCheck if tables are in sync. List all changes.migrate-tables --module-path module/If you want the script to fail on a change, add the validate flag. Useful for running inside your CI:migrate-tables --module-path module/ --validateApply changesmigrate-tables --module-path src/jobs/ --applyconvert-tableIf you already have tables created in bigquery, this script print the corresponding dataclass for you.Note: The script produces a file which is meant to be a starting point. You will most likely have to add some imports yourself!Show the help:convert-table --helpPrint a table:convert-table --project project --dataset scraper --table-name table_name >> schema.pyDevelopmentSetting up your dev environmentClone the project.Navigate into the cloned project.Create a virtual environment withpython version >=3.7pipenv --python PYTHON_VERSION$ pipenv --python 3.7orvirtualenv -p /PATH_TO_PYTHON/ /DESIRED_PATH/VENV_NAME$ virtualenv -p /usr/bin/python3.7 placeholderInstall flit via pip$ pip install flitInstall packages$ flit install --symlinkCode qualityRun all code quality checks:inv check-allTestinv testLintinv lintTypesinv type-checkCode formatinv format-codeValidate code is correctly formatted:inv check-code-format