Package: bomf
BO4E Migration Framework (bomf)BOMF is the BO4E Migration Framework.
This repository contains the code of the Python packagebomf.Rationalebomf is a framework, that allows its users to migrate datafrom source systems (starting with the raw data extracts)into an intermediate, common BO4E based data layer.From there map data to individual target system data modelsand finally create records in target systems (aka "loading").The frameworkencourages user to program consistent data processing pipelines from any source to any target systemenforces users to adapt to structured and consistent patternsand by doing so will lead to higher chances for maintainable and reusable code.Architeture / OverviewThe overall setup for a migration from 1-n source systems (A, B, C...) to 1-m target systems (1,2, 3...) might look like this:graph TD
    A[Source System A] -->|System A DB Dump| A2[Source A Data Model: A JSON Extract]
    B[Source System B] -->|System B CSV Export| B2[Source B Data Model: B CSV Files]
    A2 -->|SourceAToBo4eDataSetMapper| C{Intermediate BO4E Layer aka DataSets}
    B2 -->|SourceBToBo4eDataSetMapper| C
    C -->|validations| C
    C -->|Bo4eDataSetToTarget1Mapper| D1[Target 1 Data Model]
    C -->|Bo4eDataSetToTarget2Mapper| D2[Target 2 Data Model]
    C -->|Bo4eDataSetToTarget3Mapper| D3[Target 3 Data Model]
    D1 -->L1[Target 1 Loader]
    D2 -->L2[Target 2 Loader]
    D3 -->L3[Target 3 Loader]
    L1 -->M1[Target System 1]
    L2 -->M2[Target System 2]
    L3 -->M3[Target System 3]The Intermediate BO4E Layer (that consists of different so called DataSets) is kind of a contract between the code that mapsfrom the source data modeland the code that mapsto the target data model.Data Migration FlowThe migration of specific data from source to target is always the same:graph TD
    A1{Source Data 1} -->|Export| B1(All source data 1 extracts)
    B1 -->C1[Filter on source data 1 model aka Pre-Select 1]
    A2{Source Data 2} -->|Export| B2(All source data 2 extracts)
    B2 -->C2[Filter on source data 2 model aka Pre-Select 2]
    C1 -->|do not match filter predicate| Z{discarded data}
    C1 -->|match filter criteria| M(Custom Logic: SourceDataSetToBo4EDataSetMapper) 
    C2 -->|do not match filter predicate| Z
    C2 -->|match filter criteria| M
    M -->|mapping| E(BO4E Data Sets)
    E -->F[Validation]
    F -->|obeys a validation rule|E
    F -->|violate any validation rule|Z
    F -->|passes all validations| G[BO4E to Target Mapper]
    G -->|mapping| H(target data model)
    H -->I[Target Loader]
    I -->|load target model|L1[Loader: 1. load to target]
    L1 -->|first: load to|T{Target System}
    L1 -->|then|L2[Loader: 2 optionally poll until target has processed data]
    L2 -->|second: poll until|T
    L2 -->|then|L3[Loader: 3 optionally verify the data have been processed correctly]
    L3 -->|finally: verify|T
    L3 -->|verification failed|Z
    L1 -->|loading failed|Z
    L3 -->|verification successful|Y[The End.]
    Z-->Z1[Monitoring and Logging]
    Z1-->Z2[Human Analyst]
    Z2 -.->|manually checks| T
    Z2 -.->|feedback: heuristically define new rules for|F
    Z2 -.->|feedback: heurisically define new filters for|CHow to use this Repository on Your Machine (Development)Please follow theinstructions in our Python Template Repository.
tl;dr:tox.ContributeYou are very welcome to contribute to this template repository by opening a pull request against the main branch.