Repository: sqlfluff/sqlfluff
Pull request: Pass dbt vars to dbt
Two small changes related to dbt plugin, both have been documented and tested, but I would be happy to take another look if something does not look right.

## Fixed issue #1262
fixes #1262

Allows sqlfluff to emulate dbt's cli calls with vars flag:
```
dbt run --vars 'variable_not_included_in_dbt_config: 1'
```

If dbt project does not have all of its vars(without defaults) in `dbt_project.yml`, dbt throws the following error:
```
Could not render {{ var('variable_not_included_in_dbt_config') }}: Required var 'variable_not_included_in_dbt_config' not found in config:
  Vars supplied to <Configuration> = {}
```

The way users can include those variables is through `.sqlfuff` in `[templates:dbt:contex]` section( similar to what we do for jinja template).

## Docker Dev container Fixes

 * `CONTRIBUTING.md` mentions that `dbt_profiles.yml` need to be manually updated to work with dev container until a fix comes - this fix just arrived
 *  current Dockerfile uses options that require enabling Buildkit first so I added them to `startup` script
 * `docker-compose` file uses `platform` so I bumped the file to the version that introduced this setting

Problem statement:
Allow for DBT arguments in config
The current DBT config assumes no arguments are passed on ```dbt run```, but many instances where the dbt-profile is configured to uses variables when building, such as for the database name. This prevents  sqlfluff from  being able to build the targets.

Hints from issue discussion:
Can you provide more detail about what arguments should be supported, and link to the dbt docs on those arguments? We can't start work on this until we have these details.
@barrywhart I was just about to raise a similar issue, so I'll add my comment here instead :)

I guess @kamcdonough20 meant DBT's [vars](https://docs.getdbt.com/reference/dbt-jinja-functions/var). DBT allows usage of `var('variable_name')` in its Jinja templates and then substitutes them with variables passed as a yaml-encoded string using `--vars` in the CLI or listed in `dbt_project.yml`. So, e.g.:
```sql
# models/example_model.sql
select * from events where event_type = '{{ var("event_type") }}'
```
```bash
$ dbt run --vars "{\"event_type\":\"activation\"}"
```
yields
```sql
select * from events where event_type = "activation"
```
Similarly, one can use `var` in DBT's `profiles.yml`.

Currently, there is no way to pass those variables when using SQLFluff. Fixing this behaviour will probably require changes in `sqlfluff_templater_dbt.templater` (i.e., its `DbtConfigArgs`), CLI and/or SQLFluff config parser. 

---
[env_var](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var) seems to be picked correctly. However, it makes sense as DBT templater reads those variables directly from `os.environ`.
@kamcdonough20 I encountered the same problem, and I am working on a patch for that.

Assuming that I understood correctly how dbt expects to receive these vars then I simply have to ask sqlfluff to read extra field in config.

Not really sure if anyone else is working on this as well.

Commit message: Updated Dev container config

* fixed missing requirement to use buildkit
* parametrized .dbt/profiles.yml to point to correct container
  automatically

Patch to review:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 208b10984b9..8237356ca23 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -207,7 +207,6 @@ Steps to use the Docker Compose environment:
 * Install Docker on your machine.
 * Run `plugins/sqlfluff-templater-dbt/docker/startup` to create the containers.
 * Run `plugins/sqlfluff-templater-dbt/docker/shell` to start a bash session in the `app` container.
-* Manually edit the `host` value in `plugins/sqlfluff-templater-dbt/test/fixtures/dbt/profiles_yml/profiles.yml`, changing it to `postgres`. (This will likely be automated somehow in the future.)
 
 Inside the container, run:
 ```
diff --git a/bin/act b/bin/act
new file mode 100755
index 00000000000..44da6c9f4b8
Binary files /dev/null and b/bin/act differ
diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst
index 110a834bc5b..f44b4fb0d71 100644
--- a/docs/source/configuration.rst
+++ b/docs/source/configuration.rst
@@ -633,6 +633,19 @@ You can set the dbt project directory, profiles directory and profile with:
     `~/.dbt/`. On Windows, you can determine your default profile directory by
     running `dbt debug --config-dir`.
 
+If your project requires that you pass variables to dbt through command line,
+you can specify them in `template:dbt:contex` section of `.sqlfluff`.
+See below configuration and its equivalend dbt command:
+
+.. code-block:: cfg
+
+    [sqlfluff:templater:dbt:context]
+    my_variable = 1
+
+.. code-block:: text
+
+    dbt run --vars '{"my_variable": 1}'
+
 Known Caveats
 ^^^^^^^^^^^^^
 
diff --git a/plugins/sqlfluff-templater-dbt/docker/docker-compose.yml b/plugins/sqlfluff-templater-dbt/docker/docker-compose.yml
index f628f88e5dd..19a07cf9bbc 100644
--- a/plugins/sqlfluff-templater-dbt/docker/docker-compose.yml
+++ b/plugins/sqlfluff-templater-dbt/docker/docker-compose.yml
@@ -1,4 +1,4 @@
-version: "2"
+version: "2.4"
 
 services:
   app:
@@ -13,6 +13,8 @@ services:
     depends_on:
       - postgres
     entrypoint: /bin/bash
+    environment:
+      - POSTGRES_HOST=postgres
     tty: true
   postgres:
     image: postgres:14-bullseye
diff --git a/plugins/sqlfluff-templater-dbt/docker/startup b/plugins/sqlfluff-templater-dbt/docker/startup
index 4cca2281a7b..97c0a8beeb2 100755
--- a/plugins/sqlfluff-templater-dbt/docker/startup
+++ b/plugins/sqlfluff-templater-dbt/docker/startup
@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
 set -ex
+export COMPOSE_DOCKER_CLI_BUILD=1
+export DOCKER_BUILDKIT=1
 my_path="$( cd "$(dirname "$0")"; pwd -P)"
 ${my_path}/shutdown
 docker-compose -f ${my_path}/docker-compose.yml build
diff --git a/plugins/sqlfluff-templater-dbt/sqlfluff_templater_dbt/templater.py b/plugins/sqlfluff-templater-dbt/sqlfluff_templater_dbt/templater.py
index 21dc514ce24..f4b7a2cde38 100644
--- a/plugins/sqlfluff-templater-dbt/sqlfluff_templater_dbt/templater.py
+++ b/plugins/sqlfluff-templater-dbt/sqlfluff_templater_dbt/templater.py
@@ -52,6 +52,7 @@ class DbtConfigArgs:
     profile: Optional[str] = None
     target: Optional[str] = None
     single_threaded: bool = False
+    vars: str = ""
 
 
 class DbtTemplater(JinjaTemplater):
@@ -100,6 +101,7 @@ def dbt_config(self):
                     project_dir=self.project_dir,
                     profiles_dir=self.profiles_dir,
                     profile=self._get_profile(),
+                    vars=self._get_cli_vars(),
                 ),
                 user_config,
             )
@@ -109,6 +111,7 @@ def dbt_config(self):
                 profiles_dir=self.profiles_dir,
                 profile=self._get_profile(),
                 target=self._get_target(),
+                vars=self._get_cli_vars(),
             )
         )
         register_adapter(self.dbt_config)
@@ -229,6 +232,13 @@ def _get_target(self):
             (self.templater_selector, self.name, "target")
         )
 
+    def _get_cli_vars(self):
+        cli_vars = self.sqlfluff_config.get_section(
+            (self.templater_selector, self.name, "context")
+        )
+
+        return str(cli_vars) if cli_vars else "{}"
+
     def sequence_files(
         self, fnames: List[str], config=None, formatter=None
     ) -> Iterator[str]:
diff --git a/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/dbt_project/dbt_project.yml b/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/dbt_project/dbt_project.yml
index 1dd322e5d09..9ac17c57099 100644
--- a/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/dbt_project/dbt_project.yml
+++ b/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/dbt_project/dbt_project.yml
@@ -14,3 +14,4 @@ vars:
   my_new_project:
     # Default date stamp of run
     ds: "2020-01-01"
+    # passed_through_cli: testing for vars passed through cli('--vars' option) rather than dbt_project
diff --git a/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/dbt_project/models/vars_from_cli.sql b/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/dbt_project/models/vars_from_cli.sql
new file mode 100644
index 00000000000..c6538db893e
--- /dev/null
+++ b/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/dbt_project/models/vars_from_cli.sql
@@ -0,0 +1,2 @@
+-- Issue #1262
+SELECT {{ var('passed_through_cli') }}
diff --git a/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/profiles_yml/profiles.yml b/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/profiles_yml/profiles.yml
index 9cdad544aac..9921a8d3335 100644
--- a/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/profiles_yml/profiles.yml
+++ b/plugins/sqlfluff-templater-dbt/test/fixtures/dbt/profiles_yml/profiles.yml
@@ -3,7 +3,7 @@ default:
   outputs:
     dev:
       type: postgres
-      host: localhost
+      host: "{{ env_var('POSTGRES_HOST', 'localhost') }}"
       user: postgres
       pass: password
       port: 5432
diff --git a/plugins/sqlfluff-templater-dbt/test/templater_test.py b/plugins/sqlfluff-templater-dbt/test/templater_test.py
index aa64699d77b..de603216619 100644
--- a/plugins/sqlfluff-templater-dbt/test/templater_test.py
+++ b/plugins/sqlfluff-templater-dbt/test/templater_test.py
@@ -431,3 +431,27 @@ def test__project_dir_does_not_exist_error(dbt_templater, caplog):  # noqa: F811
         ) in caplog.text
     finally:
         logger.propagate = original_propagate_value
+
+
+@pytest.mark.parametrize(
+    ("model_path", "var_value")
+    , [
+        ("models/vars_from_cli.sql", "expected_value")
+        , ("models/vars_from_cli.sql", [1])
+        , ("models/vars_from_cli.sql", {"nested": 1})
+    ])
+def test__context_in_config_is_loaded(
+        project_dir, dbt_templater, model_path, var_value):  # noqa: F811
+    """Test that variables inside .sqlfluff are passed to dbt."""
+    context = {"passed_through_cli": var_value} if var_value else {}
+
+    config_dict = DBT_FLUFF_CONFIG
+    config_dict["templater"]['dbt']['context'] = context
+    config = FluffConfig(config_dict)
+
+    fname = os.path.abspath(os.path.join(project_dir, model_path))
+
+    processed, violations = dbt_templater.process(in_str="", fname=fname, config=config)
+
+    assert violations == []
+    assert str(var_value) in processed.templated_str


Code review comments:
diff_hunk: @@ -431,3 +431,27 @@ def test__project_dir_does_not_exist_error(dbt_templater, caplog):  # noqa: F811
         ) in caplog.text
     finally:
         logger.propagate = original_propagate_value
+
+
+@pytest.mark.parametrize(
+    ("model_path", "var_value")
+    , [
+        ("models/vars_from_cli.sql", "expected_value")
+        , ("models/vars_from_cli.sql", [1])
+        , ("models/vars_from_cli.sql", {"nested": 1})
+    ])
path: plugins/sqlfluff-templater-dbt/test/templater_test.py
text: @user2:
Keep black happy:

```suggestion
@user1.mark.parametrize(
    ("model_path", "var_value"),
    [
        ("models/vars_from_cli.sql", "expected_value"),
        ("models/vars_from_cli.sql", [1]),
        ("models/vars_from_cli.sql", {"nested": 1}),
    ],
)
```