Package: dotml
dotML⚠️ still in alpha ⚠️
dotML is a lightweight semantic layer with minimal abstractions.
It enables fetching metrics, filtered by dimensions, by parsing metric queries based on a data model.
The data model defines dimensions and metrics similar to LookML and supports joins on primary keys.
The data model is defined in YAML files.
The package generates a SQL query that you can use to fetch your results.InstallationpipinstalldotmlUsagedotML can be used both as a Python package and through its command line interface.As a Python LibraryYou can import theload_cubes, andgenerate_sql_queryfunctions from their respective modules to use in your Python
code.CLI Commands# list all cubesdotmlcubes# list all metrics for a cubedotmlfields<cube_name># query a set of metrics and dimensionsdotmlquery"<query_json>"Thequerycommand expects a JSON5 string as its argument. Here's an example:dotmlquery"{'fields': ['orders.booking_date_month', 'orders.revenue', 'orders_items.quantity'],'filters': ['${orders.country_id}= 67'],'sorts': ['orders.booking_date_month'],'limit': 100}"Is this for me?dotML is for you if are a tool builder and want to:query metrics and dimensions consistently and correctlywant to support highly flexible analytics without writing complex SQLWe build dotML forDot, but believe it can be useful for other tool builders as well.In the future, dotML might also be for you if you are a data analyst and want to:have consistent metrics across all your toolsimplement a semantic layer for your data warehouseWhy dotML?OLAP cubes, hypercubes, metric stores and semantic layers are all different names for the same thing: a data model that
defines dimensions and metrics, and a query engine that can parse metric queries and generate SQL queries to fetch the
results.
All the existing solutions we found are tightly coupled with a specific tool or database, and are not easy to use with
other tools.What about dbt Metricflow?There is a chance that dbt'smetricflowcould take this place, but today (
15.06.2023):it does not have a permissive license for tool builders,is still in beta and,is pretty complex: 1359 files vs 12 files of dotMLseems harder to learn, because of more abstractions in the data modelHowever, dotML only produces sql queries based on the defined data model.
There is no UI, no caching, and no permissions.Today it does:support joins on primary keysnested metrics and dimensionspost-processing of metrics with window function (e.g. average over 7 days, cumulative sum, etc.)Defining CubesCubes are defined in YAML files with the following structure:name:<cube_name>table:<table_path>dimensions:-name:<dimension_name>sql:<sql_expression>...metrics:-name:<measure_name>sql:<sql_expression>...Basically all SQL databases are supported: PostgreSQL, Snowflake, Redshift, BigQuery, Databricks SQL, Trino, Druid,
Oracle, MSSQL ...Full ExampleLet's say we have an online store with the following tables.Ordersidbooking_datecountry_idstatustotal12023-05-1268cancelled10922023-06-0262cancelled3232023-04-2067confirmed21342023-04-3064cancelled2252023-04-2865delivered562023-06-1369shipped9872023-04-1762delivered30Orders Itemsidorder_idproduct_idquantityprice101892620215891212232030319953131310713231542640417910Each table gets modeled as a cube in dotML.
A cube encodes the business logic on how data should be aggregated in dimensions and metrics.cubes:-name:orderstable:my_db.prod.ordersalways_filter:-"${table}.booking_date>='2019-01-01'"-"${table}.status='confirmed'"dimensions:-name:idsql:${table}.idprimary_key:true-name:booking_datesql:date_trunc(${time_frame},${table}.booking_date)variants:-time_frame:[day,week,month,quarter,year]-name:country_idsql:${table}.country_idmetrics:-name:revenuesql:sum(${table}.total)-name:average_order_valuesql:avg(${table}.total)-name:revenue_big_orderssql:sum( iff(${table}.total > 100, ${table}.total, 0) )window_metrics:-name:average_order_value_rolling_30_dayssql:sum(${revenue}) over (order by ${booking_date_day} rows between 30 preceding and current row)-name:orders_itemstable:my_db.prod.orders_line_itemsdimensions:-name:idsql:${table}.idprimary_key:true-name:order_idsql:${table}.order_id-name:productsql:${table}.product_idmetrics:-name:quantitysql:sum(${table}.quantity)Now, we can query the data model with dotML.Notice how we are querying metrics across different cubes.dotML automatically
preventsjoin fanoutsfromdotMLimportload_cubes,generate_sql_querycubes=load_cubes("data_model.yaml")query={"fields":["orders.booking_date_month","orders.revenue","orders_items.quantity"],"filters":["${orders.country_id}= '67'"],"sorts":["orders.booking_date_month"],"limit":100}sql_query=generate_sql_query(cubes,query)# here you can execute the sql query with your favorite database clientThe result looks like this:booking_date_monthrevenuequantity2023-04-018021252023-05-0113292122023-06-011689158The generated SQL query looks like this:withorders_R5S_dimensionas(selectorders_R5S.idaspk0fromdemo_db.public.ordersasorders_R5Swhere(orders_R5S.booking_date>='2019-01-01')and(orders_R5S.status='confirmed')groupby1),orders_items_0B7_dimensionas(selectorders_items_0B7.idaspk0,date_trunc(month,orders_R5S.booking_date)asbooking_date_monthfromdemo_db.public.orders_line_itemsasorders_items_0B7rightjoindemo_db.public.ordersasorders_R5Sonorders_R5S.id=orders_items_0B7.order_idgroupby1,2),orders_R5S_metricsas(selectdate_trunc(month,orders_R5S.booking_date)asbooking_date_month,orders_R5S.country_id,sum(orders_R5S.total)asrevenuefromdemo_db.public.ordersasorders_R5Sjoinorders_R5S_dimensionasorders_R5S_dimensiononorders_R5S.id=orders_R5S_dimension.pk0where(orders_R5S.booking_date>='2019-01-01')and(orders_R5S.status='confirmed')groupby1),orders_items_0B7_metricsas(selectorders_items_0B7_dimension.booking_date_month,orders_items_0B7_dimension.country_id,sum(orders_items_0B7.quantity)asquantityfromdemo_db.public.orders_line_itemsasorders_items_0B7joinorders_items_0B7_dimensionasorders_items_0B7_dimensiononorders_items_0B7.id=orders_items_0B7_dimension.pk0groupby1)selectorders_R5S_metrics.booking_date_month,orders_R5S_metrics.revenue,orders_items_0B7_metrics.quantityfromorders_R5S_metricsasorders_R5S_metricsjoinorders_items_0B7_metricsasorders_items_0B7_metricsonorders_R5S.booking_date_month=orders_items_0B7.booking_date_monthandorders_R5S.country_id=orders_items_0B7.country_idwhere(orders_R5S_metrics.country_id='67')orderby1limit100ContributingIf you have suggestions for how dotML could be improved, or want to report a bug, open an issue!
We'd love all and any contributions.Two areas where we would especially appreciate help are:a Tableau compiler, so that dotML cubes can be used as a data
source in Tableau (or another BI tool, you care about)a dbt_package_wrapper, so that cubes can get materialized by dbtReach out to:hi@sled.so