Package: bq-schema-policy-tags
bq-schema-policy-tagsThis Project is a Fork ofbq-schema by Limehome GmbHMotivation (Original GH Repo)At limehome we are heavy users of python and bigquery. This library was created to mainly solve the following issues:Define table schemas in code and have a migration script to apply changes.On deploy make sure that all schemas were applied, otherwise abort.Guarantee that when we try to write data to a table, the data matches the schema of the table (required / optional, datatypes)Version our tables and enable migrations to a new schemaAdditionally this library aims to help the users through the usage ofpython typing.Specify your schema as apython dataclassOur migration script converts the data class into a bigquery schema definitionDeserialize rows into a dataclass instance, while reading from a tableSerialize a dataclass instance into a dictionary and write it to the table.The main benefit of combining all these features is, that we can guarantee that our code will run, before we deploy to production.Motivation for ForkThe primary reason for forking this project was to extend its existing capabilities to support Policy Tags and Taxonomy tags in Google Cloud Platform (GCP). The original codebase provided a strong foundation for working with Python dataclasses, but it lacked features for adding metadata tags that can be crucial for managing sensitive or personally identifiable information (PII).Why Policy Tags?Policy Tags allow for more granular control over data access. When dealing with sensitive or regulated data, such as PII, it's crucial to have mechanisms that enable fine-tuned permissions. Policy Tags in GCP make it possible to associate specific security and access policies with individual data fields, rather than entire data sets, thereby providing a more secure and manageable environment.Taxonomy Tags in GCPTaxonomy Tags in GCP serve as a method for categorizing data based on its type or nature. By attaching taxonomy tags to data fields, organizations can more easily track, manage, and govern their data. This is particularly useful for compliance with data protection laws and regulations, as well as for internal data governance strategies.By incorporating these features into Python dataclasses through this fork, we aim to provide a more robust, secure, and compliant data handling solution.QuickstartSince this library makes use of newer features of python, you need at least python3.7.Install the packagepip install bq-schema-policy-tagsCreate a schema and a table definition in my_table.py@dataclassclassSchema:string_field:str=field(metadata={"description":"This is a STRING field."})int_field:Optional[int]some_floats:List[float]bool_field:boolclassMyTable(BigqueryTable):name="my_table_name"schema=SchemaIf you have already tables created in your account, you can use the convert-table script to create a schema.Note: The script produces a file which is meant to be a starting point. You will most likely have to add some imports yourself!Create your tableHint:Make sure to have you credentials set:export GOOGLE_APPLICATION_CREDENTIALS=your_auth.jsonAlternativly you can set the service_file as a environment variable:export GOOGLE_SERVICE_FILE={"type": "service_account", ...}Now create your tablemigrate-tables --project my_project --dataset my_dataset --module-path my_table --applyWrite a rowfromgoogle.cloudimportbigqueryrow=Schema(string_field="foo",int_field=1,some_floats=[1.0,2.0],bool_field=True)row_transformer=RowTransformer[Schema](Schema)serialized_row=RowTransformer.dataclass_instance_to_bq_row(row)bigquery_client=bigquery.Client()table=bigquery_client.get_table("project.dataset.my_table_name")bigquery_client.insert_rows(table,[serialized_row])Validate you code with a type checker likemypymypy my_table.pyRead a rowquery="SELECT * FROM project.dataset.my_table_name"forrowinbigquery_client.query(query=query):deserialized_row=row_transformer.bq_row_to_dataclass_instance(row)assertisinstance(deserialized_row,Schema)DocumentationSchema definitionsFor a full list of supported types check the following schema:fromtypingimportOptionalfromdataclassesimportdataclassfrombq_schema_policy_tags.types.type_mappingimportTimestamp,Geography@dataclassclassRequiredNestedField:int_field:int=field(metadata={"description":"This field is an INT field."})@dataclassclassRequiredSchema:string_field:str=field(metadata={"description":"This field is a STRING field."})string_field_optional=Optional[str]bytes_field:bytesint_field:intfloat_field:floatnumeric_field:Decimalbool_field:booltimestamp_field:Timestampdate_field:datetime_field:timedatetime_field:datetimegeography_field:Geographyrequired_nested_field:RequiredNestedField=field(metadata={"description":"This field is a STRUCT field."})optional_nested_field:Optional[RequiredNestedField]repeated_nested_field:List[RequiredNestedField]Adding Policy Tags (New)To add policy tags to a field, include policy_tags in the field metadata.fromdataclassesimportdataclass,field@dataclassclassExampleWithPolicyTags:sensitive_data:str=field(metadata={"policy_tags":["sensitive"]})schema=dataclass_to_schema(ExampleWithPolicyTags)Abstract tablesIf you want to have an class that inherits from BigqueryTable but does not actually map to a table in BigQuery (is abstract, common interface etc.), you can have it inherit from ABC and it will not be discovered if you pass the flag "--ignore-abstract"migrate-tables ... --ignore-abstractclassSomeInterface(BigqueryTable,ABC):passclassConcreteImplementation(SomeInterface):name="Some value here"schema=SomeSchemaWithout the flag, this would fail due to name and schema being required in "SomeInterface"TimestampsTimestamps are deserialized into datetime objects, due to the nature of the underlying bq library. To distinguish between datetime and timestamp use bq_schema_policy_tags.types.type_mapping.
Usage:frombq_schema_policy_tags.types.type_mappingimportTimestampfromdatetimeimportdatetimethe_timestamp=Timestamp(datetime.utcnow())GeographyThis library treats the geography data type as a string. BigQuery accepts geography values either in theWKTorGeoJsonformat. To actually parse and work with geodata in python, one could use theshapelylibrary. Here is an example how to load a point from the WKT format:fromshapely.wktimportloadsloads('POINT (0 0)')Table definitionsThe bigquery class is used for:Recursive table discovery by our migrate-tables scriptDefine table properties like name and schemaRequired propertiesname: The name of the tableschema: table schema either as dataclass or a list of schema fieldsOptional propertiesproject: name of the project, can be overwritten by the migrate-tables scriptdataset: name of the dataset, can be overwritten by the migrate-tables scriptVersioning tablesSince bigquery does not allow backwards incompatible schema changes, you might want to version your schemas.projectclassMyTable(BigqueryTable):name="my_table_name"schema=Schemaversion="1"By default the version will be appended to the table name, like so: my_table_name_v1. If you want to overwrite this behaviour,
you can implement the full_table_name method.Time partitioningDefine time partitioning for your table:frombq_schema_policy_tags.types.type_mappingimportTimestampfromgoogle.cloud.bigqueryimportTimePartitioning,TimePartitioningTypeclassMyTable:time_partitioning=TimePartitioning(type_=TimePartitioningType.DAY,field="some_column")Scriptsmigrate-tablesThis script has two uses:Check if locally defined schemas are in sync with the schemas in bigqueryIf a difference is detected, we try to apply the changesThe script will find all defined tables recursivly for a given python module.Note: If you have not defined your project and / or dataset in code, you will have to pass it as a parameter to the script.
Show the help:migrate-tables --helpCheck if tables are in sync. List all changes.migrate-tables --module-path module/If you want the script to fail on a change, add the validate flag. Useful for running inside your CI:migrate-tables --module-path module/ --validateApply changesmigrate-tables --module-path src/jobs/ --applyconvert-tableIf you already have tables created in bigquery, this script print the corresponding dataclass for you.Note: The script produces a file which is meant to be a starting point. You will most likely have to add some imports yourself!Show the help:convert-table --helpPrint a table:convert-table --project project --dataset scraper --table-name table_name >> schema.pyDevelopmentSetting up your dev environmentClone the project.Navigate into the cloned project.Create a virtual environment withpython version >=3.7pipenv --python PYTHON_VERSION$ pipenv --python 3.7orvirtualenv -p /PATH_TO_PYTHON/ /DESIRED_PATH/VENV_NAME$ virtualenv -p /usr/bin/python3.7 placeholderInstall flit via pip$ pip install flitInstall packages$ flit install --symlinkCode qualityRun all code quality checks:inv check-allTestinv testLintinv lintTypesinv type-checkCode formatinv format-codeValidate code is correctly formatted:inv check-code-formatKnown IssuesMissingValueError intest_row_transformer.pyWhen running the pytest suite, aMissingValueErrormay occur, specifically indicating a missing value for the field "string_field". This issue appears to be inherent to the original codebase and may not have a straightforward solution due to the project being no longer maintained.Potential CausesTherowobject in the test case may not contain the "string_field" and its value.TheSchemadata class might be expecting a value for "string_field" which is not provided.The functiondict_to_row(row_as_dict: dict) -> Rowmay not be converting the dictionary to aRowobject correctly.Suggested WorkaroundsDouble-check the test data for completeness.If possible, modify theSchemadata class to provide a default value for "string_field".Check theRowTransformerclass and thefrom_dictfunction for issues related to field handling.Please note that these are suggested workarounds and may not resolve the issue entirely.