Package: in-dbt-spark
dbtenables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-aggregate the raw data in your warehouse so that it's ready for analysis.in-dbt-sparkThein-dbt-sparkpackage contains code to connect with Spark Clusters @ LinkedIn.We added a new method to the OSS dbt-spark package to connect dbt with cluster-based spark deployments @ LinkedIn (code name Setu).
This new method implementation will leverage in-houseSetufor the programmatic submission of Spark jobs.Setuprovides a narrow waist Spark job submission interface that in turn would allow for the programmatic submission of Spark jobs from anywhere, subject to authentication, while managing multiple Spark versions and clusters under the hood.
Using Setu for DBT will enable users to move much more independently and faster.Major ImplementationThe Table below represents the mapping of our custom implementation for DB conventions.ClassDBSparkConnectionManagerSQLConnectionManagerSetuSessionDB ConnectionSetuSessionHandlerConnection HandlerSetuStatementCursorDB cursorSparkConnectionManagerThis class is responsible for managing the connections like opening, closing and error handling scenarios.1.  open(cls, connection)open() is a classmethod that gets a connection object (which could be in any state, but will have a Credentials object with the attributes you defined above), creates a new SetuSession using SetuSessionManager and moves it to the 'open' state.Generally this means doing the following:if the connection is open already, log and return it.create a connection handle using the credentialson success:set connection.state to'open'set connection.handle to the handle objectthis is what must have a cursor() method that returns a cursor!on error:set connection.state to'fail'set connection.handle toNoneraise a dbt.exceptions.FailedToConnectException with the error and any other relevant information2. cancel(self, connection)cancel is an instance method that gets a connection object and attempts to cancel any ongoing queries by calling the cancel on the handle object.3.  exception_handler(self, sql, connection_name='master')exception_handler is an instance method that returns a context manager that will handle exceptions raised by running queries, catch them, log appropriately, and then raise exceptions dbt knows how to handle.SetuSessionHandlerEquivalent to a DB connection handler, it is responsible for creating and executing cursors. This class creates SetuStatementCursor and executes the cursor with DBT compiled sqlsSetuSessionThis is a spark Interactive session responsible for creating spark context with requested resources from the yarn RM. This class is responsible for managing a remote SETU session and high-level interactions with it.SetuStatementCursorThis class is responsible for managing the SETU statements and high-level interactions with it. It takes care of creating, executing, waiting till SETU statement results are available and closing the SETU statement.Getting startedInstall dbtRead theintroductionandviewpointcreate virtual environment for dbtcd ~
mkdir dbt_env
python3.9 -m venv ~/dbt_env
source ~/dbt_env/bin/activateInstall DBT Corepip install dbt-coreInstall DBT Spark from Codegit clone this repo
cd in-dbt-spark
pip install -r requirements.txt
python setup.py installverify installationpip list | grep dbt
# Output:
dbt-core          x.x.x
dbt-extractor     x.x.x
in-dbt-spark      x.x.xRunning locallystep 1:Rundbt initcommand to bootstrap dbt project. This step should create a dbt project.step 2:create new folder named profiles and create profiles.yml empty file.step 3:Create new profile in profiles.yml similar to below,dbt_hello_world:
  outputs:
    dev:
      type: spark # connection Type (Spark/Presto/Postgres)
      method: setu  # connection Method (setu/odbc/thrift)
      url: 'https://setu@linkedin.com'
      schema: xxxx  #  schema to persist dbt produced tables
      proxy_user: xxxx  # run spark jobs as proxy user
      queue: xxxx  #  grid queue to submit spark jobs. Defaults to misc_default
      session_name: dbt_hello_world  # unique name for spark session (UUID suffix internally)
      metadata:  # High-level tracking metadata used to provide contextual information about the application
        name: dbt_hello_world
        desciption: hello world project for dbt
        org: xxxx
      spark_conf:  # Additional configs that may be needed for spark app execution (pass-through)
        spark.driver.cores: 1  # Defaults to 2
        spark.executor.cores: 1  # Defaults to 2
        spark.driver.memory: 1G  # Defaults to 4G
        spark.executor.memory: 2G  # Defaults to 8G
      execution_tags:  # Used to determine the target cluster dynamically at runtime.
        gpu: false  # If the spark job requires gpu for processing. Defaults to false
        pool: dev  # Execution environment for the job. Defaults to Dev
      jars: # List of ivy coordinates of the artifacts required by the spark app
        - com.linkedin.xxxx:xxxx:+?transitive=falsestep 5: Add models and run below commands to compile/test/run# COMPILE:
dbt compile --profiles-dir ./profiles --target dev

# RUN:
dbt run --profiles-dir ./profiles --target dev --threads x

# TEST:
dbt test --profiles-dir ./profiles --target dev --threads x

# GENERATE DOCS:
dbt docs generate --profiles-dir ./profiles --target dev

# SERVE DOCS:
dbt docs serve --profiles-dir ./profiles --port 8080Reporting bugs and contributing codeWant to report a bug or request a feature? - Reach out todbt-dev@linkedin.comWant to help us build dbt? Check out theContributing Guide