Package: hnhm
hNhM(highly Normalized hybrid Model) – data modeling methodology that enables iterative and Agile-driven
modifications to your Data Warehouse (DWH). The methodology is based on Anchor Modeling and Data Vault.The idea is to represent each Entity using 3 types of tables:Hubstores Business and Surrogate keysAttributestores the history of changes (e.g. Name, Age, Gender...)Groupis used to reduce the number of underlying tables by grouping multiple Attributes in one Group.The best practice is to define a Group by the same data source.So, the purpose ofhnhmis to:DefineEntities, Links and Flowsdeclarativelyusing PythonDescribe them "logically"The "physical" layer is managed byhnhmApplythe changes to a DWH iterativelyhnhmdetects changes in the entities and creates new Attributes, Groups, or LinksLoaddata from the Staging layer using Flows in the correct orderIgnore, Update, or keep the history usingSCD2Implementation of this package is based on report"How we implemented our data storage model — highly Normalized hybrid Model"by Evgeny Ermakov and Nikolai Grebenshchikov.1) Yandex Report, habr.com.2) SmartData Conference, youtube.comTutorialPrerequisitesInitialize DWHStart modelingAdd Attribute and GroupConceptsLogical level (Python classes)Physical level (Tables)Change types of Attributes and GroupsSupported DatabasesFuture plansContribute!Created withTutorialYou can find the full code in thedwhfolder.PrerequisitesInstall the librarypipinstallhnhmCreate thedwhdirectorymkdirdwhSpin up the Postgres Database# dwh/docker-compose.ymlversion:"3.9"volumes:postgres_data:{}services:postgres:image:postgres:15volumes:-"postgres_data:/var/lib/postgresql/data"ports:-"5433:5432"environment:POSTGRES_DB:hnhmPOSTGRES_USER:hnhmPOSTGRES_PASSWORD:123# $ docker compose up -dInitialize DWHCreate the__hnhm__.pyfile with the following contents:# dwh/__hnhm__.pyfromhnhmimportPostgresPsycopgSql,HnHm,HnhmRegistry,FileStatesql=PostgresPsycopgSql(database="hnhm",user="hnhm",password="123",port=5433)registry=HnhmRegistry(hnhm=HnHm(sql=sql,state=FileState("state.json"),),)PostgresPsycopgSqlgenerates and executes SQLHnhmRegistrystores thehnhmobject, entities and linksHnHmimplements core logic and manages the stateFileStatestores the state of your DWH in a fileApply the changes to your DWH:$hnhmapplydwh

Importing'registry'objectfromthemodule:'dwh.__hnhm__'.

YourDWHisuptodate.Our DWH is up-to-date because we haven't added any entities and links yet.Start modelingLet's add the new Entity. Add thedwh/user.pyfile with the following contents:# dwh/user.pyfromhnhmimport(Layout,LayoutType,String,ChangeType,HnhmEntity,)classUser(HnhmEntity):"""User data."""__layout__=Layout(name="user",type=LayoutType.HNHM)user_id=String("User ID.",change_type=ChangeType.IGNORE)__keys__=[user_id]Add theUserentity to the registry:# dwh/__hnhm__.py
from hnhm import PostgresPsycopgSql, HnHm, HnhmRegistry, FileState+ from dwh.user import Usersql = PostgresPsycopgSql(database="hnhm",user="hnhm",password="123",port=5433
)

registry = HnhmRegistry(+   entities=[User()],hnhm=HnHm(sql=sql,state=FileState("state.json"),),
)Apply the changes to your DWH:$hnhmapplydwh

Importing'registry'objectfromthemodule:'dwh.__hnhm__'.

Plan:

+entity'HNHM.user'+view'user'+hub'user'Applymigrations?[y/N]:y
Applied!We added the new entityUserto our DWH-- View on top the DDS tablesselect*fromentity__user;-- Hubselect*fromhub__user;Add Attribute and GroupLet's add an Attribute and a Group to our Entity. Edit thedwh/user.pyfile:# dwh/user.py
from hnhm import (Layout,LayoutType,String,ChangeType,HnhmEntity,+   Integer)


class User(HnhmEntity):"""User data."""__layout__ = Layout(name="user", type=LayoutType.HNHM)user_id = String("User ID.", change_type=ChangeType.IGNORE)+   age = Integer("Age.", change_type=ChangeType.UPDATE)+   first_name = String("First name.", change_type=ChangeType.NEW, group="name")+   last_name = String("Last name.", change_type=ChangeType.NEW, group="name")__keys__ = [user_id]Apply the changes to your DWH:$hnhmapplydwh

Importing'registry'objectfromthemodule:'dwh.__hnhm__'.

Plan:[u]entity'HNHM.user'+attribute'age'+group'name'|attribute'first_name'|attribute'last_name'[u]view'user'Applymigrations?[y/N]:y
Applied!Take a look at newly created tables-- View on top of the DDS tables was updatedselect*fromentity__user;-- Attribute 'age'select*fromattr__user__age;-- Group 'name'select*fromgroup__user__name;The physical result:view: entity__user
┌────────────────────────────────────────────────────────────────┐
│┌───────────────────┐   ┌────────────────┐   ┌─────────────────┐│
 │ group__user__name │   │ hub__user      │   │ attr__user__age │
 │                   │   │                │   │                 │
 │ + user_sk (FK)    ├──►│ + user_sk (PK) │◄──┤ + user_sk (FK)  │
 │ + first_name      │   │ + user_id_bk   │   │ + age           │
 │ + last_name       │   │ + valid_from   │   │ + valid_from    │
 │ + valid_from      │   │ + _hash        │   │ + _hash         │
 │ + valid_to        │   │ + _source      │   │ + _source       │
 │ + _hash           │   │ + _loaded_at   │   │ + _loaded_at    │
 │ + _source         │   └────────────────┘   └─────────────────┘
 │ + _loaded_at      │
 └───────────────────┘ConceptsLogical level (Python classes)Entity: business entity (User, Review, Order, Booking)Link: the relationship between Entities (UserOrder, UserBooking)Flow: helps to load data from the stage layer to Entities and LinksPhysical level (Tables)Hub: hub table contains Entity's Business Keys and Surrogate Key(MD5 hash of concatenated business keys)Attribute: attribute table contains FK to Entity's surrogate key, history of attribute changes, and thevalid_fromcolumnGroup: group table contains FK to Entity's surrogate key, history of changes to group attributes, and thevalid_fromcolumnLink: link table contains FKs to Entities surrogate keys. Historicity bySCD2Change types of Attributes and GroupsIGNORE: insert the latest new data, ignore updatesUPDATE: insert the latest new data, updateNEW: full history usingSCD2. Adds thevalid_tocolumnSupported DatabasesPostgresGreenPlumSnowflakeBigQueryClickHouseFuture plansNew database connectorsSnowflakeBigQueryGreenPlum(?) ClickHouseTestsMore elegant DSLState managementKeep history in the databaseRefresh: drift detectionRollbackGenerate static documentation for DWHRobustnessReliabilityBenchmarksSQL optimization, refinement of Jinja codeDocumentationDocumentation for Python sourcesGuides, best practices, and recipesCode Style & CITesting matrixContribute!What contributions are welcome? (spoiler: any!)New database connectorsNew featuresTypos correctionDependencies updateDocumentation improvementCode refinementBug fixesCreated withPyCharm ProfessionalNeovim