Repository: dbt-labs/dbt-core
Pull request: fix source selection during catalog generation of over 100 relations
resolves https://github.com/dbt-labs/dbt-core/issues/9755

associated adapter changes: https://github.com/dbt-labs/dbt-adapters/pull/131

<!---
  Include the number of the issue addressed by this PR above if applicable.
  PRs for code changes without an associated issue *will not be merged*.
  See CONTRIBUTING.md for more information.

  Add the `user docs` label to this PR if it will need docs changes.  An 
  issue will get opened in docs.getdbt.com upon successful merge of this PR.
-->

### Problem

<!---
  Describe the problem this PR is solving. What is the application state
  before this PR is merged?
-->

Sources were not included in catalogable_nodes, meaning that they were not included in catalog generation whenever > 100 relations were selected, or if the SchemaMetadataByRelations adapter capability was not supported.

### Solution

<!---
  Describe the way this PR solves the above problem. Add as much detail as you
  can to help reviewers understand your changes. Include any alternatives and
  tradeoffs you considered.
-->

* include sources in catalogable nodes, add a whole bunch of tests that patch adapter.MAX_SCHEMA_METADATA_RELATIONS to be one to easily simulate situations with a 'large' number of relations for cataloging. 

### Checklist

- [x] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me  
- [x] I have run this code in development and it appears to resolve the stated issue  
- [x] This PR includes tests, or tests are not required/relevant for this PR
- [x] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [x] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions


Problem statement:
[Bug] sources not in catalog.json when dbt docs generate selects over 100 nodes
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

If I run `dbt docs generate` in my dbt project selecting over 100 nodes, my sources are not included in `catalog.json`.

### Expected Behavior

The nodes and sources selected in `dbt docs generate` should be in `catalog.json`.

### Steps To Reproduce

1. In a dbt project with more than 100 nodes/sources, run `dbt docs generate`
2. Look at catalog.json, no sources

Hints from issue discussion:

Commit message: changelog entry

Patch to review:
diff --git a/.changes/unreleased/Fixes-20240312-165357.yaml b/.changes/unreleased/Fixes-20240312-165357.yaml
new file mode 100644
index 00000000000..7a391118015
--- /dev/null
+++ b/.changes/unreleased/Fixes-20240312-165357.yaml
@@ -0,0 +1,7 @@
+kind: Fixes
+body: include sources in catalog.json when over 100 relations selected for catalog
+  generation
+time: 2024-03-12T16:53:57.714118-04:00
+custom:
+  Author: michelleark
+  Issue: "9755"
diff --git a/core/dbt/task/docs/generate.py b/core/dbt/task/docs/generate.py
index a8beb145918..800f997268d 100644
--- a/core/dbt/task/docs/generate.py
+++ b/core/dbt/task/docs/generate.py
@@ -283,7 +283,8 @@ def run(self) -> CatalogArtifact:
                         node
                         for node in self.manifest.nodes.values()
                         if (node.is_relational and not node.is_ephemeral_model)
-                    ]
+                    ],
+                    self.manifest.sources.values(),
                 )
                 used_schemas = self.manifest.get_used_schemas()
                 catalog_table, exceptions = adapter.get_filtered_catalog(
diff --git a/dev-requirements.txt b/dev-requirements.txt
index 0c706dddbbe..97da2f29e95 100644
--- a/dev-requirements.txt
+++ b/dev-requirements.txt
@@ -1,4 +1,4 @@
-git+https://github.com/dbt-labs/dbt-adapters.git@main
+git+https://github.com/dbt-labs/dbt-adapters.git@refactor-magic-catalog-int
 git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-tests-adapter
 git+https://github.com/dbt-labs/dbt-common.git@main
 git+https://github.com/dbt-labs/dbt-postgres.git@main
diff --git a/tests/functional/docs/test_generate.py b/tests/functional/docs/test_generate.py
index 4129fc0abb5..7597fdb41f3 100644
--- a/tests/functional/docs/test_generate.py
+++ b/tests/functional/docs/test_generate.py
@@ -153,6 +153,56 @@ def test_select_source(self, project):
         assert len(catalog.nodes) == 0
 
 
+class TestGenerateSelectOverMaxSchemaMetadataRelations(TestBaseGenerate):
+    @pytest.fixture(scope="class")
+    def seeds(self):
+        return {
+            "sample_seed.csv": sample_seed,
+            "second_seed.csv": sample_seed,
+            "source_from_seed.csv": sample_seed,
+        }
+
+    def test_select_source(self, project):
+        run_dbt(["build"])
+
+        project.run_sql("create table {}.sample_source (id int)".format(project.test_schema))
+        project.run_sql("create table {}.second_source (id int)".format(project.test_schema))
+
+        with mock.patch.object(type(project.adapter), "MAX_SCHEMA_METADATA_RELATIONS", 1):
+            # more relations than MAX_SCHEMA_METADATA_RELATIONS -> all sources and nodes correctly returned
+            catalog = run_dbt(["docs", "generate"])
+            assert len(catalog.sources) == 3
+            assert len(catalog.nodes) == 5
+
+            # full source selection respected
+            catalog = run_dbt(["docs", "generate", "--select", "source:*"])
+            assert len(catalog.sources) == 3
+            assert len(catalog.nodes) == 0
+
+            # full node selection respected
+            catalog = run_dbt(["docs", "generate", "--exclude", "source:*"])
+            assert len(catalog.sources) == 0
+            assert len(catalog.nodes) == 5
+
+            # granular source selection respected (> MAX_SCHEMA_METADATA_RELATIONS selected sources)
+            catalog = run_dbt(
+                [
+                    "docs",
+                    "generate",
+                    "--select",
+                    "source:test.my_source_schema.sample_source",
+                    "source:test.my_source_schema.second_source",
+                ]
+            )
+            assert len(catalog.sources) == 2
+            assert len(catalog.nodes) == 0
+
+            # granular node selection respected (> MAX_SCHEMA_METADATA_RELATIONS selected nodes)
+            catalog = run_dbt(["docs", "generate", "--select", "my_model", "alt_model"])
+            assert len(catalog.sources) == 0
+            assert len(catalog.nodes) == 2
+
+
 class TestGenerateSelectSeed(TestBaseGenerate):
     @pytest.fixture(scope="class")
     def seeds(self):


Code review comments:
diff_hunk: @@ -1,4 +1,4 @@
-git+https://github.com/dbt-labs/dbt-adapters.git@main
+git+https://github.com/dbt-labs/dbt-adapters.git@refactor-magic-catalog-int
path: dev-requirements.txt
text: @author:
can be restored once https://github.com/dbt-labs/dbt-adapters/pull/131 is merged.