Package: apache-airflow-providers-normalizer
Normalizer Provider for Apache AirflowThis package contains operator to normalize JSON as text or JSONB field to nested
tables by YAML specification.InstallationYou can install provider with any package manager compatible with PyPI:pip install apache-airflow-provider-normalizerFeaturesDeclarative specification in YAML.Can extract only specified fields.Work with json/jsonb or general fields.Resistant to the absence of any fields.Automaticaly creates all nested tables.Supports incremental and full-refresh mode.One YAML can contains multiple root tables.Root table is normalized with snowflake schema.ExampleAvailable two specification styles.# Body-style mappingpostgres.public.orders:# full table name or short is supportedstaging.order:# table can be renamed in destinationid*:{origin_id:bigint}# use "*" to mark primary keydt_created:{date_created:timestamp}# fields can be renamed toodate_created:{date_created:timestamp}# use alias for field if changedstate:{state:varchar(5)}details:{details:text}# json-field saved as stringified textdetails.id:{business_key:integer}# conflicts can be resolved with dot notationdetails.order__id:{order__id:integer}# nested fields can be selected with "__"details.client__list:# list will be placed to another tabledetails.purchase__list:{client__list:text}# also list can be saved as stringified textextra**.id:{extra_id:integer}# text field containing valid json is accessble with "**"extra**.user__name:{username:varchar(255)}# after unpacking field is selectable as generic jsonoptions**:# text field with json can be unpacked to separate tablepostgres.orders.options:# the last field of previous specification is placedstaging.order__options:# to nested dedicated table with pk/fk relationships...# generated automaticaly# Header-style mappingpostgres.orders[date_created + state + details** + extra**]:# use explicit list of fields to unpacking in headerstaging.orders:# in this case json is merged to top level with generic fieldsid*:{origin_id:bigint}# it simplifies the declaration and makes it clearerdt_created:{date_created:timestamp}# (be careful with duplicates general and json fields!)date_created:{date_created:timestamp}state:{state:varchar(5)}order__id:{order__id:integer}user_name:{username:varchar(255)}...UsageThis following example demonstrates a typical use case:Usepostgresas source systemSave nested tables tomysqldatabaseUsing mapping specification as shown abovefromairflowimportDAGfromairflow.providers.normalizer.operators.normalizerimportNormalizerOperatorfromairflow.utils.datesimportdays_agowithDAG(dag_id="normalize_example",description="Normalize nested jsons to analytical database",start_date=days_ago(1),schedule="0 * * * *",)asdag:# Simple normalize with mapping fileNormalizerOperator(task_id="normalize_json",source_conn_id="postgres_default",destination_conn_id="mysql_default",mapping="mappings/mapping.yaml",)Another example demonstrates a custom preprocessing use case with full-refresh mode:fromairflowimportDAGfromairflow.providers.normalizer.operators.normalizerimportNormalizerOperatorfromairflow.utils.datesimportdays_agodefpreprocessing(doc):""" Fix bad datetime format. """if"contract"indoc:ifstr(doc["contract"]["date"])=="0001-01-01T00:00:00+00:00":doc["contract"]["date"]="1970-01-01 00:00:00"return[doc]withDAG(dag_id="normalize_example",description="Normalize nested jsons with custom preprocessing",start_date=days_ago(1),schedule="0 * * * *",)asdag:# Simple normalize with drop tables before and preprocessing documentsNormalizerOperator(task_id="normalize_json",source_conn_id="postgres_default",destination_conn_id="mysql_default",mapping="""postgres.orders[order_details + state + dt_created + dt_changed]:staging.orders:# general columnsstate:                       { state: String }dt_created:                  { created_date: DateTime }dt_changed:                  { changed_date: DateTime }init_system:                 { init_system: String }reference_id:                { reference_id: String }# json-fieldsclient__id:                  { client__id: String }client__name:                { client__name: String }contract__id:                { contract__id: String }contract__date:              { contract__date: DateTime }contract__branch__id:        { contract__branch__id: String }""",preprocessing=preprocessing,commit_every=100,)Last example demonstrates a custom SQL-templates usage with extended SQL syntax:fromairflowimportDAGfromairflow.providers.normalizer.operators.normalizerimportNormalizerOperatorfromairflow.utils.datesimportdays_agowithDAG(dag_id="normalize_example",description="Normalize nested jsons with custom SQL-templates",start_date=days_ago(1),schedule="0 * * * *",)asdag:# Customize specific SQL-dialect in deepNormalizerOperator(task_id="normalize_json",source_conn_id="postgres_default",destination_conn_id="clickhouse_default",mapping="mapping/clickhouse.yaml",select_count_query=('''SELECT count(id)FROM {table}'''),select_all_query=('''SELECT {fields}FROM {table}WHERE date_system > {{ date_system }}'''),create_table_query=('''DROP TABLE IF EXISTS {table}''','''CREATE TABLE IF NOT EXISTS {table} ({definition})ENGINE = MergeTreeORDER BY {fk or pk}''',),insert_into_query=('''INSERT INTO {table} ({fields})VALUES {values}''',),incremental=True,retries=2,)NormalizerOperator ReferenceTo importNormalizerOperatoruse:from airflow.providers.normalizer.operators.normalizer import NormalizerOperatorSupported kwargs:source_conn_id: source connection id that contains table with text/json/jsonb.destination_conn_id: destination connection id where nested tables is placed.mapping: YAML specificationselect_count_query: overwrites base query used to calculate total for paginationselect_all_query: overwrites base query used to LIMIT/OFFSET paginationcreate_table_query: overwrites base template queryincremental: set update mode, by default isFalseOther kwargs are inherited from AirflowBaseOperatorIn the query templates placeholders are available:{table}: placeholder for table name{definition}: placeholder for pairs field name and type (e.g. CREATE){fields}: placeholder for list of columns comma separated (e.g. SELECT){values}: placeholder for mulitvalue-INSERT comma separated values{pk}: name of generated primary key{fk}: name of generated foreign key