Package: bifabrik
bifabrikMicrosoft Fabric ETL toolboxThis is anearly build- if you find a problem, please report it here:https://github.com/rjankovic/bifabrik/issues. Thanks!What is the point?make BI development in Microsoft Fabric easier by providing a fluent API for common ETL tasksreduce repetitive code by setting preferences in config filesFor info on all the features, see theproject pageQuickstartFirst, let's install the library. Either add the bifabrik library to an environment in Fabric and attach that environment to your notebook.Or you can add%pip install bifabrikat the beginning of the notebook.Init the classTo load data,bifabrikneeds to access the spark session.frombifabrikimportbifabrikbif=bifabrik(spark)# 'bif' will be used in many code samples as a reference to the bifabrik class instanceAlso,make sure that your notebook is connected to a lakehouse. This is the lakehouse to which bifabrik will save data.Load CSV files (JSON is similar)Simple tasks should be easy.frombifabrikimportbifabrikbif=bifabrik(spark)bif.fromCsv('Files/CsvFiles/annual-enterprise-survey-2021.csv').toTable('Survey2021').run()...and the table is in placedisplay(spark.sql('SELECT * FROM Survey2021'))Or you can make use of pattern matching# take all files matching the pattern and concat thembif.fromCsv('Files/*/annual-enterprise-survey-*.csv').toTable('SurveyAll').run()These are full loads, overwriting the target table if it exists.Configure load preferencesIs your CSV is a bit...special? No problem, we'll tend to it.Let's say you have a European CSV with commas instead of decimal points and semicolons instead of commas as separators.bif.fromCsv("Files/CsvFiles/dimBranch.csv").delimiter(';').decimal(',').toTable('DimBranch').run()The backend uses pandas, so you can take advantage of many other options - seehelp(bif.fromCsv())Keep the configurationWhat, you have more files like that?  Well then, you probably don't want to repeat the setup each time.
Good news is, the bifabrik object can keep all your preferences:frombifabrikimportbifabrikbif=bifabrik(spark)# set the configurationbif.cfg.csv.delimiter=';'bif.cfg.csv.decimal=','# the configuration will be applied to all these loadsbif.fromCsv("Files/CsvFiles/dimBranch.csv").toTable('DimBranch').run()bif.fromCsv("Files/CsvFiles/dimDepartment.csv").toTable('DimDepartment').run()bif.fromCsv("Files/CsvFiles/dimDivision.csv").toTable('DimDivision').run()# (You can still apply configuration in the individual loads, as seen above, to override the general configuration.)If you want to persist your configuration beyond the PySpark session, you can save it to a JSON file - seeConfigurationConsistent configuration is one of the core values of the project.We like our lakehouses to be uniform in terms of loading patterns, table structures, tracking, etc. At the same time, we want to keep itDRY.bifabrik configuration aims to cover many aspects of the lakehouse so that you can define your conventions once, use it repeatedly, and override when neccessary.See thegithub pagefor more details on this.SQL transformationsEnough with the files! Let's make a simple SQL transformation, writing data to another SQL table - a straightforward full load:bif.fromSql('''SELECT Industry_name_NZSIOC AS Industry_Name,AVG(`Value`) AS AvgValueFROM LakeHouse1.Survey2021WHERE Variable_Code = 'H35'GROUP BY Industry_name_NZSIOC''').toTable('SurveySummarized').run()# The resulting table will be saved to the lakehouse attached to your notebook.# You can refer to a different source warehouse in the query, though.