Repository: dbt-labs/dbt-core

Issue:
[CT-2681] [Feature] Add `explain` or `dry_run` method to BaseAdapter
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Sometimes we want to validate queries for syntactical or attribute reference errors without triggering the query run itself. For example, in the MetricFlow integration, we want to be able to run warehouse validations that do a simple metric query verification to ensure all metrics and expressions are defined using legal SQL expressions, but we do not want to trigger warehouse queries as that would be too costly.

For most SQL engines, this can be achieved by prefixing the query with the `EXPLAIN` keyword. Unfortunately, BigQuery does not support this - instead, callers have to pass in a flag to the BQ client at query time, which is not currently possible with the dbt-bigquery adapters. 

We propose adding a method to BaseAdapter that will allow end users to issue dry run/explain queries so callers - such as MetricFlow - can have a consistent interface for doing so regardless of the underlying engine being used in the dbt project.

Here's a suggested mock up of the API:

```
def dry_run(self, sql: str) -> None:
    """Trigger a dry run of a SQL query against the underlying engine. Useful for things like validating query syntax."""
    raise NotImplementedError
```

In the SQLAdapter we could default this method to prefix the input query with the `EXPLAIN` keyword, and then add an internal implementation in the BigQueryAdapter that works for that.

This method as described would either return None on success or throw an exception on failure. We could also return a result object here instead of requiring callers to try/catch every invocation manually.

Questions:

1. Are there better alternatives?
2. Do we want to call this `dry_run` or `explain`? The reasoning behind `dry_run` is explain might not return a useful plan output result.
3. Do we prefer to let a failure propagate as an exception, or handle the exception and return the errors along with some kind of response object? The former is more typical, the latter is more in line with how adapters manage calls to execute().
4. Should we make this an `@abstractmethod` on the BaseAdapter? That way downstream implementers know they have to add support for this if they are not extending SQLAdapter (or BigQueryAdapter). The drawback is once this goes live existing implementations will probably not work with that version of core.
5. Should we leave room for bind parameters in the function signature? Those appear to only be supported through the add_query() method today, which is not implemented for BigQuery.


### Describe alternatives you've considered

_No response_

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

Yes, I will build this, I may need some guidance on test configuration.

### Anything else?

_No response_
```[tasklist]
### Tasks
- [ ] https://github.com/dbt-labs/dbt-snowflake/issues/682
- [ ] https://github.com/dbt-labs/dbt-redshift/issues/516
- [ ] https://github.com/dbt-labs/dbt-spark/issues/820
- [ ] https://github.com/dbt-labs/dbt-bigquery/issues/805
```



Discussion:

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -289,6 +289,17 @@ def execute(
         """
         return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)
 
+    def validate_sql(self, sql: str) -> AdapterResponse:
+        """Submit the given SQL to the engine for validation, but not execution.
+
+        This should throw an appropriate exception if the input SQL is invalid, although
+        in practice that will generally be handled by delegating to an existing method
+        for execution and allowing the error handler to take care of the rest.
+
+        :param str sql: The sql to validate
+        """
+        raise NotImplementedError("`validate_sql` is not implemented for this adapter!")
+
     @available.parse(lambda *a, **k: [])
     def get_column_schema_from_query(self, sql: str) -> List[BaseColumn]:
         """Get a list of the Columns with names and data types from the given sql."""
@@ -785,7 +796,6 @@ def _make_match(
         schema: str,
         identifier: str,
     ) -> List[BaseRelation]:
-
         matches = []
 
         search = self._make_match_kwargs(database, schema, identifier)
@@ -1063,7 +1073,6 @@ def _get_one_catalog(
         schemas: Set[str],
         manifest: Manifest,
     ) -> agate.Table:
-
         kwargs = {"information_schema": information_schema, "schemas": schemas}
         table = self.execute_macro(
             GET_CATALOG_MACRO_NAME,
@@ -1453,7 +1462,6 @@ def render_model_constraint(cls, constraint: ModelLevelConstraint) -> Optional[s
 def catch_as_completed(
     futures,  # typing: List[Future[agate.Table]]
 ) -> Tuple[agate.Table, List[Exception]]:
-
     # catalogs: agate.Table = agate.Table(rows=[])
     tables: List[agate.Table] = []
     exceptions: List[Exception] = []
diff --git a/core/dbt/adapters/sql/connections.py b/core/dbt/adapters/sql/connections.py
--- a/core/dbt/adapters/sql/connections.py
+++ b/core/dbt/adapters/sql/connections.py
@@ -52,7 +52,6 @@ def add_query(
         bindings: Optional[Any] = None,
         abridge_sql_log: bool = False,
     ) -> Tuple[Connection, Any]:
-
         connection = self.get_thread_connection()
         if auto_begin and connection.transaction_open is False:
             self.begin()
diff --git a/core/dbt/adapters/sql/impl.py b/core/dbt/adapters/sql/impl.py
--- a/core/dbt/adapters/sql/impl.py
+++ b/core/dbt/adapters/sql/impl.py
@@ -1,7 +1,7 @@
 import agate
 from typing import Any, Optional, Tuple, Type, List
 
-from dbt.contracts.connection import Connection
+from dbt.contracts.connection import Connection, AdapterResponse
 from dbt.exceptions import RelationTypeNullError
 from dbt.adapters.base import BaseAdapter, available
 from dbt.adapters.cache import _make_ref_key_dict
@@ -22,6 +22,7 @@
 TRUNCATE_RELATION_MACRO_NAME = "truncate_relation"
 DROP_RELATION_MACRO_NAME = "drop_relation"
 ALTER_COLUMN_TYPE_MACRO_NAME = "alter_column_type"
+VALIDATE_SQL_MACRO_NAME = "validate_sql"
 
 
 class SQLAdapter(BaseAdapter):
@@ -218,6 +219,34 @@ def check_schema_exists(self, database: str, schema: str) -> bool:
         results = self.execute_macro(CHECK_SCHEMA_EXISTS_MACRO_NAME, kwargs=kwargs)
         return results[0][0] > 0
 
+    def validate_sql(self, sql: str) -> AdapterResponse:
+        """Submit the given SQL to the engine for validation, but not execution.
+
+        By default we simply prefix the query with the explain keyword and allow the
+        exceptions thrown by the underlying engine on invalid SQL inputs to bubble up
+        to the exception handler. For adjustments to the explain statement - such as
+        for adapters that have different mechanisms for hinting at query validation
+        or dry-run - callers may be able to override the validate_sql_query macro with
+        the addition of an <adapter>__validate_sql implementation.
+
+        :param sql str: The sql to validate
+        """
+        kwargs = {
+            "sql": sql,
+        }
+        result = self.execute_macro(VALIDATE_SQL_MACRO_NAME, kwargs=kwargs)
+        # The statement macro always returns an AdapterResponse in the output AttrDict's
+        # `response` property, and we preserve the full payload in case we want to
+        # return fetched output for engines where explain plans are emitted as columnar
+        # results. Any macro override that deviates from this behavior may encounter an
+        # assertion error in the runtime.
+        adapter_response = result.response  # type: ignore[attr-defined]
+        assert isinstance(adapter_response, AdapterResponse), (
+            f"Expected AdapterResponse from validate_sql macro execution, "
+            f"got {type(adapter_response)}."
+        )
+        return adapter_response
+
     # This is for use in the test suite
     def run_sql_for_tests(self, sql, fetch, conn):
         cursor = conn.handle.cursor()


Test patch:
diff --git a/tests/adapter/dbt/tests/adapter/utils/test_validate_sql.py b/tests/adapter/dbt/tests/adapter/utils/test_validate_sql.py
new file mode 100644
--- /dev/null
+++ b/tests/adapter/dbt/tests/adapter/utils/test_validate_sql.py
@@ -0,0 +1,75 @@
+from typing import Type
+
+import pytest
+
+from dbt.adapters.base.impl import BaseAdapter
+from dbt.exceptions import DbtRuntimeError, InvalidConnectionError
+
+
+class BaseDryRunMethod:
+    """Tests the behavior of the dry run method for the relevant adapters.
+
+    The valid and invalid SQL should work with most engines by default, but
+    both inputs can be overridden as needed for a given engine to get the correct
+    behavior.
+
+    The base method is meant to throw the appropriate custom exception when dry_run
+    fails.
+    """
+
+    @pytest.fixture(scope="class")
+    def valid_sql(self) -> str:
+        """Returns a valid statement for issuing as a dry run query.
+
+        Ideally this would be checkable for non-execution. For example, we could use a
+        CREATE TABLE statement with an assertion that no table was created. However,
+        for most adapter types this is unnecessary - the EXPLAIN keyword has exactly the
+        behavior we want, and here we are essentially testing to make sure it is
+        supported. As such, we return a simple SELECT query, and leave it to
+        engine-specific test overrides to specify more detailed behavior as appropriate.
+        """
+
+        return "select 1"
+
+    @pytest.fixture(scope="class")
+    def invalid_sql(self) -> str:
+        """Returns an invalid statement for issuing a bad dry run query."""
+
+        return "Let's run some invalid SQL and see if we get an error!"
+
+    @pytest.fixture(scope="class")
+    def expected_exception(self) -> Type[Exception]:
+        """Returns the Exception type thrown by a failed query.
+
+        Defaults to dbt.exceptions.DbtRuntimeError because that is the most common
+        base exception for adapters to throw."""
+        return DbtRuntimeError
+
+    def test_valid_dry_run(self, adapter: BaseAdapter, valid_sql: str) -> None:
+        """Executes a dry run query on valid SQL. No news is good news."""
+        with adapter.connection_named("test_valid_sql_validation"):
+            adapter.validate_sql(valid_sql)
+
+    def test_invalid_dry_run(
+        self,
+        adapter: BaseAdapter,
+        invalid_sql: str,
+        expected_exception: Type[Exception],
+    ) -> None:
+        """Executes a dry run query on invalid SQL, expecting the exception."""
+        with pytest.raises(expected_exception=expected_exception) as excinfo:
+            with adapter.connection_named("test_invalid_sql_validation"):
+                adapter.validate_sql(invalid_sql)
+
+        # InvalidConnectionError is a subclass of DbtRuntimeError, so we have to handle
+        # it separately.
+        if excinfo.type == InvalidConnectionError:
+            raise ValueError(
+                "Unexpected InvalidConnectionError. This typically indicates a problem "
+                "with the test setup, rather than the expected error for an invalid "
+                "validate_sql query."
+            ) from excinfo.value
+
+
+class TestDryRunMethod(BaseDryRunMethod):
+    pass