Package: dbt-loom
dbt-loomdbt-loom is a dbt Core plugin that weaves together multi-project deployments. dbt-loom works by fetching public model definitions from your dbt artifacts, and injecting those models into your dbt project.flowchart LR

    classDef black fill:#f2f2ebff, stroke:#000, color:#000
    classDef background fill:#f2f2ebff, stroke:#000, color:#000
    classDef hidden fill:#BADC3F, stroke:#BADC3F, color:#BADC3F

   style TOP fill:#BADC3F, stroke:#000

  subgraph TOP[Your Infrastructure]
    direction TB
    dbt_runtime[dbt Core]:::background
    proprietary_plugin[Open Source Metadata Plugin]:::background

    files[Local Files]:::background
    object_storage[Object Storage]:::background
    discovery_api[dbt Cloud APIs]:::background

    discovery_api --> proprietary_plugin
    files --> proprietary_plugin
    object_storage --> proprietary_plugin
    proprietary_plugin --> dbt_runtime
  end

  Project:::black --> TOP --> Warehouse:::blackdbt-loom currently supports obtaining model definitions from:Local manifest filesdbt CloudGCSS3-compatible object storage servicesAzure Storage:warning:dbt Core's plugin functionality is still in beta. Please note that this may break in the future as dbt Labs solidifies the dbt plugin API in future versions.Getting StartedTo being, install thedbt-loompython package.pip install dbt-loomNext, create adbt-loomconfiguration file. This configuration file provides the paths for your
upstream project's manifest files.manifests:-name:project_nametype:fileconfig:path:path/to/manifest.jsonBy default,dbt-loomwill look fordbt_loom.config.ymlin your working directory. You can also set theDBT_LOOM_CONFIG_PATHenvironment variable.Using dbt Cloud as an artifact sourceYou can use dbt-loom to fetch model definitions from dbt Cloud by setting up adbt-cloudmanifest in yourdbt-loomconfig, and setting theDBT_CLOUD_API_TOKENenvironment variable in your execution environment.manifests:-name:project_nametype:dbt_cloudconfig:account_id:<YOUR DBT CLOUD ACCOUNT ID># Job ID pertains to the job that you'd like to fetch artifacts from.job_id:<REFERENCE JOB ID>api_endpoint:<DBT CLOUD ENDPOINT># dbt Cloud has multiple regions with different URLs. Update this to# your appropriate dbt cloud endpoint.step_id:<JOB STEP># If your job generates multiple artifacts, you can set the step from# which to fetch artifacts. Defaults to the last step.Using an S3-compatible object store as an artifact sourceYou can use dbt-loom to fetch manifest files from S3-compatible object stores
by setting up abs3manifest in yourdbt-loomconfig. Please note that this
approach supports all standard boto3-compatible environment variables and authentication mechanisms. Please see theboto3 documentationfor more details.manifests:-name:project_nametype:s3config:bucket_name:<YOUR S3 BUCKET NAME># The name of the bucket where your manifest is stored.object_name:<YOUR OBJECT NAME># The object name of your manifest file.Using GCS as an artifact sourceYou can use dbt-loom to fetch manifest files from Google Cloud Storage by setting up agcsmanifest in yourdbt-loomconfig.manifests:-name:project_nametype:gcsconfig:project_id:<YOUR GCP PROJECT ID># The alphanumeric ID of the GCP project that contains your target bucket.bucket_name:<YOUR GCS BUCKET NAME># The name of the bucket where your manifest is stored.object_name:<YOUR OBJECT NAME># The object name of your manifest file.credentials:<PATH TO YOUR SERVICE ACCOUNT JSON CREDENTIALS># The OAuth2 Credentials to use. If not passed, falls back to the default inferred from the environment.Using Azure Storage as an artifact sourceYou can use dbt-loom to fetch manifest files from Azure Storage
by setting up anazuremanifest in yourdbt-loomconfig. Theazuretype implements
theDefaultAzureCredentialclass, supporting all environment variables and authentication mechanisms.
Alternatively, set theAZURE_STORAGE_CONNECTION_STRINGenvironment variable to
authenticate via a connection string.manifests:-name:project_nametype:azureconfig:account_name:<YOUR AZURE STORAGE ACCOUNT NAME># The name of your Azure Storage accountcontainer_name:<YOUR AZURE STORAGE CONTAINER NAME># The name of your Azure Storage containerobject_name:<YOUR OBJECT NAME># The object name of your manifest file.Using environment variablesYou can easily incorporate your own environment variables into the config file. This allows for dynamic configuration values that can change based on the environment. To specify an environment variable in thedbt-loomconfig file, use one of the following formats:${ENV_VAR}or$ENV_VARExample:manifests:-name:revenuetype:gcsconfig:project_id:${GCP_PROJECT}bucket_name:${GCP_BUCKET}object_name:${MANIFEST_PATH}How does it work?As of dbt-core 1.6.0-b8, there now exists adbtPluginclass which defines functions that can
be called by dbt-core'sPluginManger. During different parts of the dbt-core lifecycle (such as graph linking and
manifest writing), thePluginMangerwill be called and all plugins registered with the appropriate hook will be executed.dbt-loom implements aget_nodeshook, and uses a configuration file to parse manifests, identify public models, and
inject those public models when called bydbt-core.Known CaveatsCross-project dependencies are a relatively new development, and dbt-core plugins
are still in beta. As such there are a number of caveats to be aware of when using
this tool.dbt plugins are only supported in dbt-core version 1.6.0-b8 and newer. This means you must be using a dbt adapter
compatible with this version.PluginNodeArgsare not fully-realized dbtManifestNodes, so documentation generated bydbt docs generatemay
be sparse when viewing injected models.