Repository: dbt-labs/dbt-core

Issue:
[CT-2428] `dbt show` should include `--limit` in compiled query
`dbt show --limit` will limit the number of rows of data previewed on the CLI.

Currently, `dbt show` applies a `limit` _after_ it's already run the query and loaded the results into memory:

https://github.com/dbt-labs/dbt-core/blob/121fa5793fda43aa62ece302114e9cf9ee7bb8a7/core/dbt/task/show.py#L20-L22

I would expect the query templated by `dbt show` to include `limit {limit}`.

Otherwise, dbt could run a query that returns 10k rows, load all of them into memory (in an `agate.Table`), and only then filter down to the default (`5`). My sense is that risks significant slowdown, and OOM issues.

---

**Acceptance criteria:**
- Instead of just directly executing `compiled_node.compiled_code`, template it into a subquery that includes `limit`
- In order to write the subquery in a cross-database-compatible way, we should either
     - Use a (Jinja) macro or materialization to template the subquery with `limit`
     - Use an adapter (Python) method — much easier to implement, and probably sufficient

---

**Out of scope:** Supporting BigQuery's no-cost "preview" API (`bq head`), which would require us to do a diff check / cache validation on the logical/applied state of this model. (logical = current SQL, applied = most recently materialized) Let's keep thinking about if there are ways to support that in the future. **Update:** Opened a separate issue for this: https://github.com/dbt-labs/dbt-core/issues/7391.
[CT-2428] `dbt show` should include `--limit` in compiled query
`dbt show --limit` will limit the number of rows of data previewed on the CLI.

Currently, `dbt show` applies a `limit` _after_ it's already run the query and loaded the results into memory:

https://github.com/dbt-labs/dbt-core/blob/121fa5793fda43aa62ece302114e9cf9ee7bb8a7/core/dbt/task/show.py#L20-L22

I would expect the query templated by `dbt show` to include `limit {limit}`.

Otherwise, dbt could run a query that returns 10k rows, load all of them into memory (in an `agate.Table`), and only then filter down to the default (`5`). My sense is that risks significant slowdown, and OOM issues.

---

**Acceptance criteria:**
- Instead of just directly executing `compiled_node.compiled_code`, template it into a subquery that includes `limit`
- In order to write the subquery in a cross-database-compatible way, we should either
     - Use a (Jinja) macro or materialization to template the subquery with `limit`
     - Use an adapter (Python) method — much easier to implement, and probably sufficient

---

**Out of scope:** Supporting BigQuery's no-cost "preview" API (`bq head`), which would require us to do a diff check / cache validation on the logical/applied state of this model. (logical = current SQL, applied = most recently materialized) Let's keep thinking about if there are ways to support that in the future. **Update:** Opened a separate issue for this: https://github.com/dbt-labs/dbt-core/issues/7391.


Discussion:
@jtcohen6 Per BLG we'd actually like to implement this in the base adapter (option 2).  It strikes a good balance between doing all-the-things (supporting bq head and any other adapter specific logic we might want to use) and doing the quick-and-dirty version where we edit the sql just for the `show` command.  It's been pointed accordingly.
@jtcohen6 Per BLG we'd actually like to implement this in the base adapter (option 2).  It strikes a good balance between doing all-the-things (supporting bq head and any other adapter specific logic we might want to use) and doing the quick-and-dirty version where we edit the sql just for the `show` command.  It's been pointed accordingly.

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -274,7 +274,7 @@ def connection_for(self, node: ResultNode) -> Iterator[None]:
 
     @available.parse(lambda *a, **k: ("", empty_table()))
     def execute(
-        self, sql: str, auto_begin: bool = False, fetch: bool = False
+        self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
     ) -> Tuple[AdapterResponse, agate.Table]:
         """Execute the given SQL. This is a thin wrapper around
         ConnectionManager.execute.
@@ -283,10 +283,11 @@ def execute(
         :param bool auto_begin: If set, and dbt is not currently inside a
             transaction, automatically begin one.
         :param bool fetch: If set, fetch results.
+        :param Optional[int] limit: If set, only fetch n number of rows
         :return: A tuple of the query status and results (empty if fetch=False).
         :rtype: Tuple[AdapterResponse, agate.Table]
         """
-        return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch)
+        return self.connections.execute(sql=sql, auto_begin=auto_begin, fetch=fetch, limit=limit)
 
     @available.parse(lambda *a, **k: [])
     def get_column_schema_from_query(self, sql: str) -> List[BaseColumn]:
diff --git a/core/dbt/adapters/sql/connections.py b/core/dbt/adapters/sql/connections.py
--- a/core/dbt/adapters/sql/connections.py
+++ b/core/dbt/adapters/sql/connections.py
@@ -118,13 +118,16 @@ def process_results(
         return [dict(zip(column_names, row)) for row in rows]
 
     @classmethod
-    def get_result_from_cursor(cls, cursor: Any) -> agate.Table:
+    def get_result_from_cursor(cls, cursor: Any, limit: Optional[int]) -> agate.Table:
         data: List[Any] = []
         column_names: List[str] = []
 
         if cursor.description is not None:
             column_names = [col[0] for col in cursor.description]
-            rows = cursor.fetchall()
+            if limit:
+                rows = cursor.fetchmany(limit)
+            else:
+                rows = cursor.fetchall()
             data = cls.process_results(column_names, rows)
 
         return dbt.clients.agate_helper.table_from_data_flat(data, column_names)
@@ -138,13 +141,13 @@ def data_type_code_to_name(cls, type_code: Union[int, str]) -> str:
         )
 
     def execute(
-        self, sql: str, auto_begin: bool = False, fetch: bool = False
+        self, sql: str, auto_begin: bool = False, fetch: bool = False, limit: Optional[int] = None
     ) -> Tuple[AdapterResponse, agate.Table]:
         sql = self._add_query_comment(sql)
         _, cursor = self.add_query(sql, auto_begin)
         response = self.get_response(cursor)
         if fetch:
-            table = self.get_result_from_cursor(cursor)
+            table = self.get_result_from_cursor(cursor, limit)
         else:
             table = dbt.clients.agate_helper.empty_table()
         return response, table
diff --git a/core/dbt/task/show.py b/core/dbt/task/show.py
--- a/core/dbt/task/show.py
+++ b/core/dbt/task/show.py
@@ -19,8 +19,12 @@ def __init__(self, config, adapter, node, node_index, num_nodes):
 
     def execute(self, compiled_node, manifest):
         start_time = time.time()
+
+        # Allow passing in -1 (or any negative number) to get all rows
+        limit = None if self.config.args.limit < 0 else self.config.args.limit
+
         adapter_response, execute_result = self.adapter.execute(
-            compiled_node.compiled_code, fetch=True
+            compiled_node.compiled_code, fetch=True, limit=limit
         )
         end_time = time.time()
 
@@ -66,13 +70,8 @@ def task_end_messages(self, results):
                     )
 
         for result in matched_results:
-            # Allow passing in -1 (or any negative number) to get all rows
             table = result.agate_table
 
-            if self.args.limit >= 0:
-                table = table.limit(self.args.limit)
-                result.agate_table = table
-
             # Hack to get Agate table output as string
             output = io.StringIO()
             if self.args.output == "json":


Test patch:
diff --git a/tests/functional/show/test_show.py b/tests/functional/show/test_show.py
--- a/tests/functional/show/test_show.py
+++ b/tests/functional/show/test_show.py
@@ -87,6 +87,20 @@ def test_second_ephemeral_model(self, project):
         )
         assert "col_hundo" in log_output
 
+    @pytest.mark.parametrize(
+        "args,expected",
+        [
+            ([], 5),  # default limit
+            (["--limit", 3], 3),  # fetch 3 rows
+            (["--limit", -1], 7),  # fetch all rows
+        ],
+    )
+    def test_limit(self, project, args, expected):
+        run_dbt(["build"])
+        dbt_args = ["show", "--inline", models__second_ephemeral_model, *args]
+        results, log_output = run_dbt_and_capture(dbt_args)
+        assert len(results.results[0].agate_table) == expected
+
     def test_seed(self, project):
         (results, log_output) = run_dbt_and_capture(["show", "--select", "sample_seed"])
         assert "Previewing node 'sample_seed'" in log_output