Package: dbt-fal
Welcome to dbt-fal 👋 do more with dbtdbt-fal adapter is the ✨easiest✨ way to run yourdbt Python models.Starting with dbt v1.3, you can now build your dbt models in Python. This leads to some cool use cases that was once difficult to build with SQL alone. Some examples are:Using Python stats libraries to calculate statsBuilding forecastsBuilding other predictive models such as classification and clusteringThis is fantastic! BUT, there is still one issue though! The developer experience with Snowflake and Bigquery is not great, and there is no Python support for Redshift and Postgres.dbt-fal provides the best environment to run your Python models that works with all other data warehouses! With dbt-fal, you can:Build and test your models locallyIsolate each model to run in its own environment with its own dependencies[Coming Soon] Run your Python models in the ☁️ cloud ☁️ with elasticly scaling Python environments.[Coming Soon] Even add GPUs to your models for some heavier workloads such as training ML models.Getting Started1. Install dbt-falpip install dbt-fal[bigquery,snowflake]Add your current warehouse here2. Update yourprofiles.ymland add the fal adapterjaffle_shop:target:dev_with_faloutputs:dev_with_fal:type:faldb_profile:dev_bigquery# This points to your main adapterdev_bigquery:type:bigquery...Don't forget to point to your main adapter with thedb_profileattribute. This is how the fal adapter knows how to connect to your data warehouse.3.dbt run!That is it! It is really that simple 😊4. [🚨 Cool Feature Alert 🚨] Environment management with dbt-falIf you want some help with environment management (vs sticking to the default env that the dbt process runs in), you can create a fal_project.yml in the same folder as your dbt project and have “named environments”:In your dbt project folder:$touchfal_project.yml# Paste the config belowenvironments:-name:mltype:venvrequirements:-prophetand then in your dbt model:$vimodels/orders_forecast.py

defmodel(dbt,fal):dbt.config(fal_environment="ml")# Add this linedf:pd.DataFrame=dbt.ref("orders_daily")Thedbt.config(fal_environment=“ml”)will give you an isolated clean env to run things in, so you dont pollute your package space.5. [Coming Soon™️] Move your compute to the Cloud!Let us know if you are interested in this. We are looking for beta users.dbt-falcommand line toolWith thedbt-falCLI, you can:Send Slack notificationsupon dbt model success or failure.Load data from external data sourcesbefore a model starts running.Download dbt modelsinto a Python context with a familiar syntax:ref('my_dbt_model')usingFalDbtProgramatically access rich metadataabout your dbt project.Head to ourdocumentation sitefor a deeper dive or play within-depth examplesto see how fal can help you get more done with dbt.1. Installdbt-fal$pipinstalldbt-fal[postgres]2. Go to your dbt project directory$cd~/src/my_dbt_project3. Create a Python script:send_slack_message.pyimportosfromslack_sdkimportWebClientfromslack_sdk.errorsimportSlackApiErrorCHANNEL_ID=os.getenv("SLACK_BOT_CHANNEL")SLACK_TOKEN=os.getenv("SLACK_BOT_TOKEN")client=WebClient(token=SLACK_TOKEN)message_text=f"Model:{context.current_model.name}. Status:{context.current_model.status}."try:response=client.chat_postMessage(channel=CHANNEL_ID,text=message_text)exceptSlackApiErrorase:asserte.response["error"]4. Add ametasection in yourschema.ymlmodels:-name:historical_ozone_levelsdescription:Ozone levelsconfig:materialized:tablecolumns:-name:ozone_leveldescription:Ozone level-name:dsdescription:Datemeta:fal:scripts:-send_slack_message.py5. Rundbt-fal flow run$dbt-falflowrun# both your dbt models and python scripts are run6. Alternatively rundbtandfalconsecutively$dbtrun# Your dbt models are run$dbt-falrun# Your python scripts are runRunning scripts before dbt runsRun scripts before the model runs by using thepre-hook:configuration option.Given the following schema.yml:models:
  - name: boston
    description: Ozone levels
    config:
      materialized: table
    meta:
      owner: "@meder"
      fal:
        pre-hook:
          - fal_scripts/trigger_fivetran.py
        post-hook:
          - fal_scripts/slack.pydbt-fal flow runwill runfal_scripts/trigger_fivetran.py, then thebostondbt model, and finallyfal_scripts/slack.py.
If a model is selected with a selection flag (e.g.--select boston), the hooks associated to the model will always run with it.$dbt-falflowrun--selectbostonConceptsprofile.yml and Credentialsfalintegrates withdbt'sprofile.ymlfile to access and read data from the data warehouse. Once you setup credentials in yourprofile.ymlfile for your existingdbtworkflows anytime you usereforsourceto create a dataframefalauthenticates using the credentials specified in theprofile.ymlfile.metaSyntaxmodels:-name:historical_ozone_levels...meta:owner:"@me"fal:post-hook:-send_slack_message.py-another_python_script.pyUse thefalandpost-hookkeys underneath themetaconfig to letfalCLI know where to look for the Python scripts. You can pass a list of scripts as shown above to run one or more scripts as a post-hook operation after adbt run.Variables and functionsInside a Python script, you get access to some useful variables and functionsVariablesAcontextobject with information relevant to the model through which the script was run. For themetaSyntaxexample, we would get the following:context.current_model.name#= historical_ozone_levelscontext.current_model.meta#= {'owner': '@me'}context.current_model.meta.get("owner")#= '@me'context.current_model.status# Could be one of#= 'success'#= 'error'#= 'skipped'contextobject also has access to test information related to the current model. If the previous dbt command was eithertestorbuild, thecontext.current_model.testproperty is populated with a list of tests:context.current_model.tests#= [CurrentTest(name='not_null', modelname='historical_ozone_levels, column='ds', status='Pass')]refandsourcefunctionsThere are also available some familiar functions fromdbt# Refer to dbt models or sources by name and returns it as `pandas.DataFrame`ref('model_name')source('source_name','table_name')# You can use it to get the running model dataref(context.current_model.name)write_to_modelfunction❗️ We recommend using thedbt-faladapter for writing data back to your data-warehouse.It is also possible to send data back to your data-warehouse. This makes it easy to get the data, process it and upload it back into dbt territory.This function is available in fal Python models only, that is a Python script inside afal_modelsdirectory and add afal-models-pathsto yourdbt_project.ymlname:"jaffle_shop"# ...model-paths:["models"]# ...vars:# Add this to your dbt_project.ymlfal-models-paths:["fal_models"]Once added, it will automatically be run by fal without having to add any extra configurations in theschema.yml.source_df=source('source_name','table_name')ref_df=ref('a_model')# Your code heredf=...# Upload a `pandas.DataFrame` back to the datawarehousewrite_to_model(df)write_to_modelalso accepts an optionaldtypeargument, which lets you specify datatypes of columns. It works the same way asdtypeargument forDataFrame.to_sqlfunction.fromsqlalchemy.typesimportInteger# Upload but specifically create the `value` column with type `integer`# Can be useful if data has `None` valueswrite_to_model(df,dtype={'value':Integer()})Importingfalas a Python packageYou may be interested in accessing dbt models and sources easily from a Jupyter Notebook or another Python script.
For that, just import thefalpackage and intantiate a FalDbt project:fromfal.dbtimportFalDbtfaldbt=FalDbt(profiles_dir="~/.dbt",project_dir="../my_project")faldbt.list_sources()# [#    DbtSource(name='results' ...),#    DbtSource(name='ticket_data_sentiment_analysis' ...)#    ...# ]faldbt.list_models()# [#    DbtModel(name='zendesk_ticket_data' ...),#    DbtModel(name='agent_wait_time' ...)#    ...# ]sentiments=faldbt.source('results','ticket_data_sentiment_analysis')# pandas.DataFrametickets=faldbt.ref('stg_zendesk_ticket_data')# pandas.DataFrameWhy are we building this?We thinkdbtis great because it empowers data people to get more done with the tools that they are already familiar with.This library will form the basis of our attempt to more comprehensively enabledata science workloadsdownstream ofdbt. And because having reliable data pipelines is the most important ingredient in building predictive analytics, we are building a library that integrates well with dbt.Have feedback or need help?Join us infal on DiscordJoin thedbt Communityand go into our#tools-fal channel