Repository: colbymchenry/codegraph (2026-06-29T21:54:13Z)

Issue:
v1.1.3 still indexes gitignored embedded repos; ignored benchmark corpus triggers resolving-refs watchdog
## Summary

On CodeGraph v1.1.3, a root `.gitignore` rule for a directory that contains embedded git repositories is not honored by CodeGraph's scanner. The installed package's exported `scanDirectory()` still returns ~138k source files from the ignored embedded repos. A full `codegraph index` then reaches `Resolving refs` and the v1.1.3 command watchdog kills it as wedged.

This looks like a regression / remaining edge case after #970 / #976 / #980, where the intended behavior is that gitignored embedded repos are not discovered unless explicitly opted in with `codegraph.json.includeIgnored`.

## Environment

- CodeGraph: `1.1.3`
- Install path shape: standalone bundle under `~/.codegraph/versions/v1.1.3`
- Node bundled with CodeGraph: `v22.22.3`
- OS: macOS / Darwin arm64
- Hardware class: Apple Silicon workstation, 128 GB RAM
- Project shape: private TypeScript monorepo with a local benchmark/reference corpus of public cloned repositories under a gitignored path

## Relevant `.gitignore`

```gitignore
# benchmark corpus clones (analysis docs stay tracked)
benchmark/corpus/
benchmark/repos/
benchmark/**/node_modules/
```

There is no `codegraph.json` and no `includeIgnored` opt-in.

## Observed behavior

Running `codegraph index` on v1.1.3 scans the ignored embedded repositories anyway:

```text
Scanning files — 140,712 found
Parsing code — done
Resolving refs 6%
[CodeGraph] Main thread unresponsive for ~60s — killing the wedged process so a fresh one can start (#850). Disable with CODEGRAPH_NO_WATCHDOG=1.
```

The existing graph shows the resulting scale:

```text
Files:           140,701
Nodes:         1,595,121
Edges:         1,776,928
Unresolved refs: 5,370,697
DB size:       4.8 GiB
```

Breakdown from the DB:

```text
benchmark/ files:       138,450
benchmark/repos/ files: 138,456 from scanDirectory()
```

The language mix is a strong signal that reference repos are being indexed rather than the first-party project:

```text
typescript 43,478
csharp     17,204
python     14,720
java       12,933
ruby        8,665
yaml        8,531
php         8,404
go          7,193
swift       6,975
rust        6,678
```

## Non-destructive repro / diagnostics

These commands were run from the project root with v1.1.3 installed.

Git agrees the path is ignored:

```bash
git check-ignore -v --no-index \
  benchmark/repos/ \
  benchmark/repos/docusaurus-openapi-docs/ \
  benchmark/repos/docusaurus-openapi-docs/.git
```

Output:

```text
.gitignore:82:benchmark/repos/ benchmark/repos/
.gitignore:82:benchmark/repos/ benchmark/repos/docusaurus-openapi-docs/
.gitignore:82:benchmark/repos/ benchmark/repos/docusaurus-openapi-docs/.git
```

Plain git file enumeration also does not expose files under that path:

```bash
git ls-files -o --exclude-standard benchmark | wc -l
# 0
```

But CodeGraph's scanner still includes them:

```js
const { scanDirectory } = require('/path/to/codegraph/v1.1.3/lib/dist/extraction/index.js');
const files = scanDirectory(process.cwd());
console.log({
  count: files.length,
  benchmarkRepos: files.filter((f) => f.startsWith('benchmark/repos/')).length,
});
```

Output:

```json
{ "count": 140712, "benchmarkRepos": 138456 }
```

`discoverEmbeddedRepoRoots()` also discovers all 19 ignored embedded repos:

```json
{
  "count": 19,
  "benchmarkRepos": 19,
  "sample": [
    "benchmark/repos/RapiDoc/",
    "benchmark/repos/docusaurus-openapi-docs/",
    "benchmark/repos/fern/",
    "benchmark/repos/scalar/",
    "benchmark/repos/zudoku/"
  ]
}
```

One additional clue: `buildScopeIgnore()` does not consider the root `.gitignore` rule to match this path, even though `git check-ignore --no-index` does:

```js
const { buildScopeIgnore } = require('/path/to/codegraph/v1.1.3/lib/dist/extraction/index.js');
const ig = buildScopeIgnore(process.cwd());
console.log(ig.ignores('benchmark/repos/'));
console.log(ig.ignores('benchmark/repos/docusaurus-openapi-docs/'));
console.log(ig.ignores('benchmark/repos/docusaurus-openapi-docs/src/index.ts'));
```

Output:

```text
false
false
false
```

## Expected behavior

With no `codegraph.json.includeIgnored`, `benchmark/repos/` should be excluded because the root `.gitignore` excludes it. The scan should be roughly the first-party project plus tracked benchmark analysis files, not the full contents of every embedded reference repo.

A full `codegraph index` should therefore avoid the 138k-file reference corpus and should not enter the resolver shape that triggers the main-thread watchdog.

## Actual behavior

`scanDirectory()` and `discoverEmbeddedRepoRoots()` include the ignored embedded repos. Full indexing then attempts to resolve millions of low-value cross-repo references from the reference corpus and the v1.1.3 watchdog kills the process during `Resolving refs`.

The hottest unresolved names are all generic test/framework/library names, consistent with unrelated repos being mixed into one graph:

```text
expect      139,596
it           79,113
Assert.That  73,563
toBe         61,454
T            58,188
to_string    39,292
Builder      36,393
```

## Why this seems distinct from already-closed issues

- #970 / #976 / #980 say gitignored embedded repos should be ignored by default again. v1.1.3 still discovers this gitignored embedded-repo directory.
- #999 added tracked-path `exclude` and resolver hardening. This case is not a tracked vendor dir and should not need `exclude`; `.gitignore` should be sufficient per #980.
- The watchdog is doing its job by killing a wedged command, but the root problem is that the ignored reference corpus enters scan scope at all.

## Workaround

A committed `codegraph.json` with an explicit `exclude` for this path may be able to work around it:

```json
{ "exclude": ["benchmark/repos/"] }
```

But this appears to contradict the promised zero-config behavior from #980: root `.gitignore` should exclude gitignored embedded repositories unless `includeIgnored` opts them back in.



Discussion:
Fixed in #1066 (merged to `main`) — will ship in **v1.1.4**.

You nailed the diagnosis. The repos under `benchmark/repos/` are **tracked gitlinks** (mode `160000`) — clones that got `git add`ed (the *"you've added another git repository"* warning) and stayed tracked even after the `.gitignore` rule was added, which is exactly why plain `git check-ignore` reported them as *not* ignored and you needed `--no-index`.

The gitlink-discovery path added in #1031/#1033 indexed those tracked gitlinks **unconditionally** — it never consulted `.gitignore`, unlike the untracked-embedded path. So it bypassed the #970/#976 gitignore-respect default, and once a repo entered `embeddedRoots`, the scope matcher exempted it and refused to prune its ancestors (hence `buildScopeIgnore().ignores('benchmark/repos/')` returning `false`).

The fix gates **both** gitlink-discovery sites on the same rule the untracked-embedded path already uses: a gitlink under a gitignored path is skipped by default, and re-included only via `codegraph.json` `includeIgnored`. Gitlinks in non-ignored locations (the #1031/#1033 case) are unchanged.

So after v1.1.4 your `benchmark/repos/` corpus is excluded with **no config at all** — the `exclude` workaround won't be needed.

Validated end-to-end on macOS, Linux, and Windows: a repo with real clones (express/chalk/slugify) as gitlinks under a gitignored `benchmark/repos/` indexes just the first-party files by default, and pulls the corpus back in only when `includeIgnored` opts it in.

Thanks for the unusually thorough report — the `scanDirectory`/`discoverEmbeddedRepoRoots`/`buildScopeIgnore` diagnostics made the root cause obvious.
v1.1.4 is now live on npm with this fix — `npm i -g @colbymchenry/codegraph@1.1.4` (or your usual upgrade path). Your `benchmark/repos/` corpus should now be excluded with no config. Thanks again!
I upgraded the affected project's checkout to `v1.1.4` and there is still a failure, but the evidence now points to a different remaining path than the original scanner bug.

The good news: the installed `v1.1.4` scanner APIs now exclude `benchmark/repos/` correctly in this repo (`scanDirectory()` returns `2256` files and `0` under `benchmark/repos/`; `discoverEmbeddedRepoRoots()` returns `0`; `buildScopeIgnore()` returns `true` for the ignored benchmark paths). A fresh `v1.1.4` init on a temp copy of the same working tree also succeeds with `2256` files.

The remaining problem is the existing oversized `.codegraph` state created by older versions. The live repo still has a graph with `140,701` files, including `138,445` from `benchmark/repos/`, plus a large DB/WAL (`codegraph.db` about `4.8G`, `codegraph.db-wal` about `4.0G`). On that state, `codegraph index --verbose` with `1.1.4` still prints only `Indexing project`, never reaches visible `Phase: scanning`, and then the watchdog kills it after about 60s with the same message.

I also stopped stale older MCP daemons and retried; same result. So #1065's scanner fix looks valid for fresh state, but users with a pre-fix poisoned/oversized DB cannot recover via the documented `codegraph index` rebuild path.

I opened the follow-up with sanitized repo-specific evidence here: #1067




Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 143f8e92f..ba90a8ec4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### Fixes
+
+- CodeGraph again respects `.gitignore` for nested repositories that git tracks as gitlinks. The recent change that taught CodeGraph to descend into nested repos recorded as `160000` "commit" pointers (#1031, #1033) did so even when your `.gitignore` excludes the directory those repos live in — so a gitignored reference or benchmark corpus full of cloned repositories got pulled into the index anyway. One project with a gitignored `benchmark/repos/` of 19 cloned repos saw over 138,000 files swept in and a 4.8 GiB graph, and a full index then stalled in the "Resolving refs" phase until the watchdog killed it. CodeGraph now treats a gitignored gitlink the same as any other gitignored embedded repo: excluded by default, and re-included only when you opt the directory in with `codegraph.json` `includeIgnored`. Nested repos in non-ignored locations — the case #1031/#1033 fixed — are unchanged. Thanks @AriaShishegaran for the detailed report. (#1065)
 
 ## [1.1.3] - 2026-06-29
 
diff --git a/src/extraction/index.ts b/src/extraction/index.ts
index f96ec6099..65de493a0 100644
--- a/src/extraction/index.ts
+++ b/src/extraction/index.ts
@@ -511,6 +511,40 @@ export function buildScopeIgnore(rootDir: string, embeddedRoots?: Iterable<strin
   );
 }
 
+/**
+ * Whether an embedded repo found as a tracked gitlink (mode 160000, #1031/#1033)
+ * must be SKIPPED rather than indexed. A gitlink is tracked, so `.gitignore`
+ * can't untrack it — but the discovery passes for it must still honor the same
+ * scope rules as every other path, or a gitignored reference/data dir full of
+ * `git add`ed clones gets pulled into the index against the user's stated intent
+ * (#1065). Two reasons to skip:
+ *   1. It sits in a built-in default-ignored location — an npm git-dependency
+ *      under `node_modules` is never project code; not even an explicit opt-in
+ *      revives it (matches `findIgnoredEmbeddedRepos`).
+ *   2. The parent repo's own `.gitignore` covers its path and the project did
+ *      NOT opt that path in via `codegraph.json` `includeIgnored`. The gitignore
+ *      rule is the user's stated intent to keep that path out of scope, exactly
+ *      as for an UNtracked embedded repo — respect it by default, opt back in
+ *      with `includeIgnored` (#514, #970, #976).
+ * `relDir` is repoDir-relative (trailing-slashed); `prefix` is repoDir's
+ * scan-root-relative path so the `includeIgnored` pattern is matched on the full
+ * scan-root-relative path. `defaults` is `defaultsOnlyIgnore()` and `repoIgnore`
+ * is `buildDefaultIgnore(repoDir)` (defaults + the repo's own `.gitignore`),
+ * both passed in so they're built once per repo level rather than per gitlink.
+ */
+function gitlinkEmbeddedRepoSkipped(
+  relDir: string,
+  prefix: string,
+  defaults: Ignore,
+  repoIgnore: Ignore,
+  includeIgnored: Ignore | null,
+): boolean {
+  if (defaults.ignores(relDir)) return true;        // default-ignored — never index, opt-in can't revive
+  if (!repoIgnore.ignores(relDir)) return false;    // not ignored at all — index as before (#1031/#1033)
+  // Gitignored by the repo's own rules — skip unless the project opted it in.
+  return !includeIgnored?.ignores(normalizePath(prefix + relDir));
+}
+
 /**
  * Standalone discovery of every embedded repo root under `rootDir` (relative,
  * trailing-slashed) — the untracked kind (#193) always, and the gitignored kind
@@ -554,13 +588,16 @@ export function discoverEmbeddedRepoRoots(rootDir: string): string[] {
         ['ls-files', '-z', '-s', '--recurse-submodules'],
         { cwd: repoAbs, encoding: 'utf-8', timeout: 30000, maxBuffer: 50 * 1024 * 1024, stdio: ['pipe', 'pipe', 'pipe'], windowsHide: true }
       );
+      const repoIgnore = buildDefaultIgnore(repoAbs);
       for (const entry of staged.split('\0')) {
         if (!entry || entry.slice(0, 6) !== '160000') continue;
         const tab = entry.indexOf('\t');
         if (tab === -1) continue;
         const rel = entry.slice(tab + 1);
         const relDir = rel.endsWith('/') ? rel : rel + '/';
-        if (defaults.ignores(relDir)) continue;
+        // A gitlink under a gitignored path is respected (not indexed) unless the
+        // project opted it in — same rule as the untracked-ignored kind (#1065).
+        if (gitlinkEmbeddedRepoSkipped(relDir, prefix, defaults, repoIgnore, includeIgnored)) continue;
         if (classifyGitDir(path.join(repoAbs, rel)) === 'embedded') candidates.push(relDir);
       }
     } catch { /* staged listing failed — other discovery still runs */ }
@@ -692,14 +729,21 @@ function collectGitFiles(repoDir: string, prefix: string, files: Set<string>, em
   // never sees it (it's tracked, not "other"). A gitlink with no checkout on disk
   // (an uninitialized submodule — empty dir, no `.git`) has nothing to index and
   // is left alone, as is a submodule worktree (a duplicate view, #945). (#1031, #1033)
-  for (const rel of gitlinkRels) {
-    const relDir = rel.endsWith('/') ? rel : rel + '/';
-    if (defaultsOnlyIgnore().ignores(relDir)) continue;
-    const childDir = path.join(repoDir, rel);
-    // 'embedded' = a real .git checkout on disk; 'worktree' and 'none' are skipped.
-    if (classifyGitDir(childDir) !== 'embedded') continue;
-    embeddedRoots?.add(normalizePath(prefix + relDir));
-    collectGitFiles(childDir, prefix + relDir, files, embeddedRoots, includeIgnored);
+  if (gitlinkRels.length > 0) {
+    const defaults = defaultsOnlyIgnore();
+    const repoIgnore = buildDefaultIgnore(repoDir);
+    for (const rel of gitlinkRels) {
+      const relDir = rel.endsWith('/') ? rel : rel + '/';
+      // A gitlink under a gitignored path is respected (not indexed) unless the
+      // project opted it in via `includeIgnored` — keep tracked gitlinks under
+      // the same scope rule as the untracked-ignored kind below (#1065).
+      if (gitlinkEmbeddedRepoSkipped(relDir, prefix, defaults, repoIgnore, includeIgnored)) continue;
+      const childDir = path.join(repoDir, rel);
+      // 'embedded' = a real .git checkout on disk; 'worktree' and 'none' are skipped.
+      if (classifyGitDir(childDir) !== 'embedded') continue;
+      embeddedRoots?.add(normalizePath(prefix + relDir));
+      collectGitFiles(childDir, prefix + relDir, files, embeddedRoots, includeIgnored);
+    }
   }
 
   // Embedded repos hidden by THIS repo's ignore rules (`/packages/` in a


Test patch:
diff --git a/__tests__/extraction.test.ts b/__tests__/extraction.test.ts
index 927ecfef1..15fb9e4b6 100644
--- a/__tests__/extraction.test.ts
+++ b/__tests__/extraction.test.ts
@@ -9,7 +9,7 @@ import * as fs from 'fs';
 import * as path from 'path';
 import * as os from 'os';
 import { CodeGraph } from '../src';
-import { extractFromSource, scanDirectory, buildDefaultIgnore } from '../src/extraction';
+import { extractFromSource, scanDirectory, buildDefaultIgnore, discoverEmbeddedRepoRoots, buildScopeIgnore } from '../src/extraction';
 import { detectLanguage, isLanguageSupported, getSupportedLanguages, initGrammars, loadAllGrammars, isSourceFile } from '../src/extraction/grammars';
 import { stripCppTemplateArgs } from '../src/extraction/languages/c-cpp';
 import { normalizePath } from '../src/utils';
@@ -5659,6 +5659,55 @@ describe('Nested gitlink repos (#1031, #1033)', () => {
     expect(files).toContain('app.ts');
     expect(files).not.toContain('libs/lib/lib.ts'); // not on disk → correctly absent
   });
+
+  // #1065: a gitlink under a path the super-repo's OWN `.gitignore` covers is the
+  // tracked-gitlink twin of the untracked-ignored embedded repo (#514, #970). The
+  // gitlink-discovery pass must honor that `.gitignore` the same way — otherwise a
+  // gitignored reference/benchmark corpus full of `git add`ed clones gets pulled
+  // into the index (the 138k-file blow-up the reporter hit). Respect it by default;
+  // re-include only via `codegraph.json` `includeIgnored`.
+  it('does not index a gitlink under a gitignored directory by default (#1065)', async () => {
+    const { execFileSync } = await import('child_process');
+    const git = (cwd: string, ...args: string[]) => execFileSync('git', args, { cwd, stdio: 'pipe' });
+
+    const root = path.join(tempDir, 'root');
+    await makeRepo(root, 'app');
+    // An embedded clone under a path the super-repo gitignores (a benchmark corpus).
+    await makeRepo(path.join(root, 'benchmark', 'repos', 'ref'), 'ref');
+    git(root, 'add', 'benchmark/repos/ref'); // tracked as a 160000 gitlink
+    fs.writeFileSync(path.join(root, '.gitignore'), 'benchmark/repos/\n');
+    git(root, 'add', '.gitignore');
+    git(root, 'commit', '-q', '-m', 'add gitignored gitlink + ignore rule');
+
+    const files = scanDirectory(root);
+    expect(files).toContain('app.ts');
+    expect(files).not.toContain('benchmark/repos/ref/ref.ts'); // gitignored → excluded
+
+    // The watcher path agrees: the ignored root is never discovered, and the dir is
+    // pruned (the reporter's exact clue — `ignores('benchmark/repos/')` was false).
+    expect(discoverEmbeddedRepoRoots(root)).toEqual([]);
+    expect(buildScopeIgnore(root).ignores('benchmark/repos/')).toBe(true);
+    expect(buildScopeIgnore(root).ignores('benchmark/repos/ref/ref.ts')).toBe(true);
+  });
+
+  it('re-includes a gitignored gitlink when codegraph.json includeIgnored opts in (#1065)', async () => {
+    const { execFileSync } = await import('child_process');
+    const git = (cwd: string, ...args: string[]) => execFileSync('git', args, { cwd, stdio: 'pipe' });
+
+    const root = path.join(tempDir, 'root');
+    await makeRepo(root, 'app');
+    await makeRepo(path.join(root, 'benchmark', 'repos', 'ref'), 'ref');
+    git(root, 'add', 'benchmark/repos/ref');
+    fs.writeFileSync(path.join(root, '.gitignore'), 'benchmark/repos/\n');
+    fs.writeFileSync(path.join(root, 'codegraph.json'), JSON.stringify({ includeIgnored: ['benchmark/repos/'] }));
+    git(root, 'add', '.gitignore', 'codegraph.json');
+    git(root, 'commit', '-q', '-m', 'opt the gitignored gitlink back in');
+
+    const files = scanDirectory(root);
+    expect(files).toContain('app.ts');
+    expect(files).toContain('benchmark/repos/ref/ref.ts'); // opted in → indexed
+    expect(discoverEmbeddedRepoRoots(root)).toContain('benchmark/repos/ref/');
+  });
 });
 
 describe('Nested non-submodule git repos', () => {