Repository: gastownhall/beads (2026-04-03T17:46:21Z)

Issue:
Embedded mode: wisp tables not auto-created (createIgnoredTables not called)
## Bug

`bd dep add` and `bd delete` fail with `table not found: wisp_dependencies` in embedded mode (v1.0.0). The `createIgnoredTables()` function that creates the `dolt_ignore`'d wisp tables (`wisps`, `wisp_dependencies`, `wisp_labels`, `wisp_events`, `wisp_comments`) is never called during embedded Dolt initialization.

## Reproduction

```bash
bd init   # or any existing embedded-mode repo
bd create --title="A" --type=task --priority=2
bd create --title="B" --type=task --priority=2
bd dep add <B-id> <A-id>
# Error: failed to check for dependency cycle: Error 1146: table not found: wisp_dependencies
```

Same for `bd delete`:
```
Error: getting dependencies: get dependencies from wisp_dependencies: Error 1146: table not found: wisp_dependencies
```

## Environment

- `bd version 1.0.0 (Homebrew)`
- macOS, embedded Dolt mode (`metadata.json: dolt_mode: "embedded"`)
- `dolt_ignore` patterns are correctly set (`wisps`, `wisp_%`)
- All 23 schema migrations ran successfully
- The `dependencies` table exists, but `wisp_dependencies` does not

## Root Cause

`createIgnoredTables()` in `internal/storage/dolt/migrations.go` is called during server-mode schema init (`initSchemaOnDB`) but appears to not be called in the embedded Dolt code path. Since wisp tables are `dolt_ignore`'d, they only exist in the working set and must be recreated on every new Dolt session.

The cycle detection CTE in `internal/storage/issueops/dependencies.go` queries both `dependencies` and `wisp_dependencies` in a UNION. Unlike `blocked.go` which gracefully skips missing tables (`isTableNotExistError` → `continue`), the cycle detection query fails hard.

## Workaround

Manually create the tables via dolt CLI:

```bash
dolt --data-dir=.beads/embeddeddolt sql -q "
CREATE TABLE IF NOT EXISTS wisps LIKE issues;
CREATE TABLE IF NOT EXISTS wisp_dependencies LIKE dependencies;
CREATE TABLE IF NOT EXISTS wisp_labels LIKE labels;
CREATE TABLE IF NOT EXISTS wisp_events LIKE events;
CREATE TABLE IF NOT EXISTS wisp_comments LIKE comments;
"
```

Tables persist in the working set after creation.

## Suggested Fix

1. Call `createIgnoredTables()` during embedded Dolt initialization (same as server mode)
2. Additionally: make the cycle detection CTE in `dependencies.go` gracefully handle missing `wisp_dependencies` (like `blocked.go` already does)


Discussion:
What platform are you using beads on? I was not able to reproduce based on the steps above.
```bash
➜  beads_failure git:(main) ✗ bd init
  ✓ Configured Dolt remote: origin → git+https://github.com/coffeegoddd/dotfiles.git
  Repository ID: 66f66af7
  Clone ID: be90256e670cb94b

▶ Already configured as: maintainer
Change role? [y/N]: n
  Hooks installed to: .beads/hooks/
  ✓ Created AGENTS.md with agent instructions
Installing Claude hooks for this project...
✓ Registered SessionStart hook
✓ Registered PreCompact hook
Installing Claude Code integration...
✓ Created new CLAUDE.md with beads integration

✓ Claude Code integration installed
  File: /home/dustin/cursor_src/repros/beads_failure/CLAUDE.md
No additional configuration needed!

✓ Claude Code integration installed
  Settings: /home/dustin/cursor_src/repros/beads_failure/.claude/settings.json

Restart Claude Code for changes to take effect.

⚠ Git upstream not configured
  For sync workflows, set your upstream with:
  git remote add upstream <repo-url>


✓ bd initialized successfully!

  Backend: dolt
  Mode: embedded
  Database: beads_failure
  Issue prefix: beads_failure
  Issues will be named: beads_failure-<hash> (e.g., beads_failure-a3f2dd)

Run bd quickstart to get started.

➜  beads_failure git:(main) ✗ bd create --title="A" --type=task --priority=2

✓ Created issue: beads_failure-xj3 — A
  Priority: P2
  Status: open

➜  beads_failure git:(main) ✗ bd create --title="B" --type=task --priority=2

✓ Created issue: beads_failure-c2q — B
  Priority: P2
  Status: open

💡 Tip: Install the beads plugin for automatic workflow context, or run 'bd setup claude' for CLI-only mode
➜  beads_failure git:(main) ✗ bd list
○ beads_failure-c2q ● P2 B
○ beads_failure-xj3 ● P2 A

--------------------------------------------------------------------------------
Total: 2 issues (2 open, 0 in progress)

Status: ○ open  ◐ in_progress  ● blocked  ✓ closed  ❄ deferred
➜  beads_failure git:(main) ✗ bd dep add beads_failure-c2q beads_failure-xj3
✓ Added dependency: beads_failure-c2q (B) depends on beads_failure-xj3 (A) (blocks)
➜  beads_failure git:(main) ✗ bd list
○ beads_failure-c2q ● P2 B
○ beads_failure-xj3 ● P2 A

--------------------------------------------------------------------------------
Total: 2 issues (2 open, 0 in progress)

Status: ○ open  ◐ in_progress  ● blocked  ✓ closed  ❄ deferred
➜  beads_failure git:(main) ✗ bd version
bd version 1.0.0 (dev: main@72170267e00a)
```
Thanks for the report. I wasn't able to reproduce this with a fresh `bd init` — wisp tables are created correctly in that path. I believe the root cause analysis in the issue description is inaccurate. Here's what I think is actually happening:

### Actual root cause: `backfillMigrations` skips wisp table creation

The embedded mode **does** create wisp tables — via SQL migrations 0020 and 0021. On a fresh `bd init`, all 23 migrations execute and the tables are created in the working set. They persist on disk between `bd` invocations (the `.beads/embeddeddolt/` directory maintains Dolt working set state), so they survive across commands. This is why I can't reproduce it.

The problem is the **backfill code path** in `internal/storage/embeddeddolt/schema.go` (lines 83-91):

```go
// If schema_migrations is empty but core tables already exist (e.g. restored
// from a DoltStore backup that doesn't track embedded migrations), backfill
// all versions so we don't re-run migrations that would fail on "already exists".
if current == 0 {
    var tableCount int
    if err := tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM information_schema.tables WHERE table_name = 'issues' AND table_schema = DATABASE()").Scan(&tableCount); err == nil && tableCount > 0 {
        return backfillMigrations(ctx, tx)
    }
}
```

When the embedded engine finds `schema_migrations` empty but `issues` already exists, it marks **all 23 migrations as applied without executing them**. Migrations 0020-0021 (`CREATE TABLE IF NOT EXISTS wisp_dependencies`, etc.) never run, but they're recorded as version 23. On every subsequent invocation, `migrateUp` fast-paths (`current >= latestVersion()`) and the wisp tables are never created.

This triggers when **upgrading from server-mode to embedded-mode** (which v1.0.0 made the default). The embedded database gets bootstrapped with existing data, the `issues` table is already present, and the backfill path fires.

### Why `bd dep add` fails but `bd list` works

The cycle detection CTE in `issueops/dependencies.go:134` does a hard `UNION ALL` across `dependencies` and `wisp_dependencies`:

```sql
SELECT issue_id, depends_on_id FROM dependencies WHERE type IN ('blocks','conditional-blocks')
UNION ALL
SELECT issue_id, depends_on_id FROM wisp_dependencies WHERE type IN ('blocks','conditional-blocks')
```

If `wisp_dependencies` doesn't exist, this query fails with Error 1146. In contrast, `blocked.go` gracefully handles missing tables via `isTableNotExistError` + `continue`, so `bd list` works fine.

### Suggested fix

1. **After the backfill path, ensure wisp tables exist** — run the wisp table `CREATE TABLE IF NOT EXISTS` statements after `backfillMigrations` returns, since those tables are `dolt_ignore`'d and wouldn't have been imported from the server-mode data.

@sussdorff — can you confirm you upgraded from a pre-v1.0 beads install (server mode) to v1.0.0 (embedded mode)? That would confirm this theory.
Yes, I upgraded from pre-v1.0 (shared-server mode) to v1.0.0 which switched to embedded mode as default. The backfill migration theory matches — the `issues` table already existed from the server-mode data, so the backfill path would have fired and skipped the wisp table creation.

Platform: macOS (Apple Silicon, Darwin 25.3.0), beads v1.0.0.

Separately: the dirty working set issue (dolthub/dolt#10807) also affects us — every remotesapi push from embedded mode dirties the remote SQL server's working set. We've been working around it with `bd dolt pull && bd dolt push --force`. PR dolthub/dolt#10810 should fix that side.



Patch:
diff --git a/internal/storage/embeddeddolt/schema.go b/internal/storage/embeddeddolt/schema.go
index 1e86ec85b1..f08c8389d9 100644
--- a/internal/storage/embeddeddolt/schema.go
+++ b/internal/storage/embeddeddolt/schema.go
@@ -22,9 +22,9 @@ var (
 	latestVer  int
 )
 
-// latestVersion returns the highest version number among the embedded .up.sql files.
+// LatestVersion returns the highest version number among the embedded .up.sql files.
 // Computed once and cached.
-func latestVersion() int {
+func LatestVersion() int {
 	latestOnce.Do(func() {
 		entries, err := fs.ReadDir(upMigrations, "schema")
 		if err != nil {
@@ -76,7 +76,7 @@ func migrateUp(ctx context.Context, tx *sql.Tx) (int, error) {
 	}
 
 	// Fast path: if current version matches the highest embedded migration, nothing to do.
-	if current >= latestVersion() {
+	if current >= LatestVersion() {
 		return 0, nil
 	}
 
@@ -90,17 +90,33 @@ func migrateUp(ctx context.Context, tx *sql.Tx) (int, error) {
 		}
 	}
 
-	// Collect and sort migration files.
+	return runMigrations(ctx, tx, current, false)
+}
+
+// backfillMigrations runs all migrations in order, ignoring "already exists"
+// errors, and records each version. Used when a database is restored from a
+// backup that predates the schema_migrations tracking table — most of the
+// schema is already correct, but dolt_ignore'd tables (wisps) may be missing.
+func backfillMigrations(ctx context.Context, tx *sql.Tx) (int, error) {
+	return runMigrations(ctx, tx, 0, true)
+}
+
+type migrationFile struct {
+	version int
+	name    string
+}
+
+// runMigrations collects all embedded .up.sql files with version > minVersion,
+// sorts them, and executes each one. When tolerateExisting is true, "already
+// exists" SQL errors are ignored and versions are recorded with INSERT IGNORE
+// (backfill path). Otherwise errors are fatal and versions use plain INSERT
+// (normal migration path).
+func runMigrations(ctx context.Context, tx *sql.Tx, minVersion int, tolerateExisting bool) (int, error) {
 	entries, err := fs.ReadDir(upMigrations, "schema")
 	if err != nil {
 		return 0, fmt.Errorf("reading embedded migrations: %w", err)
 	}
 
-	type migrationFile struct {
-		version int
-		name    string
-	}
-
 	var pending []migrationFile
 	for _, e := range entries {
 		if e.IsDir() || !strings.HasSuffix(e.Name(), ".up.sql") {
@@ -110,7 +126,7 @@ func migrateUp(ctx context.Context, tx *sql.Tx) (int, error) {
 		if err != nil {
 			return 0, fmt.Errorf("parsing migration filename %q: %w", e.Name(), err)
 		}
-		if v > current {
+		if v > minVersion {
 			pending = append(pending, migrationFile{version: v, name: e.Name()})
 		}
 	}
@@ -121,22 +137,25 @@ func migrateUp(ctx context.Context, tx *sql.Tx) (int, error) {
 		return 0, nil
 	}
 
-	// Apply each pending migration. The DSN has multiStatements=true,
-	// so each file is executed as a single ExecContext call.
 	for _, mf := range pending {
 		data, err := upMigrations.ReadFile("schema/" + mf.name)
 		if err != nil {
 			return 0, fmt.Errorf("reading migration %s: %w", mf.name, err)
 		}
 
-		sql := strings.TrimSpace(string(data))
-		if sql != "" {
+		if sql := strings.TrimSpace(string(data)); sql != "" {
 			if _, err := tx.ExecContext(ctx, sql); err != nil {
-				return 0, fmt.Errorf("migration %s failed: %w", mf.name, err)
+				if !tolerateExisting || !isAlreadyExistsError(err) {
+					return 0, fmt.Errorf("migration %s failed: %w", mf.name, err)
+				}
 			}
 		}
 
-		if _, err := tx.ExecContext(ctx, "INSERT INTO schema_migrations (version) VALUES (?)", mf.version); err != nil {
+		insertSQL := "INSERT INTO schema_migrations (version) VALUES (?)"
+		if tolerateExisting {
+			insertSQL = "INSERT IGNORE INTO schema_migrations (version) VALUES (?)"
+		}
+		if _, err := tx.ExecContext(ctx, insertSQL, mf.version); err != nil {
 			return 0, fmt.Errorf("recording migration %s: %w", mf.name, err)
 		}
 	}
@@ -144,28 +163,10 @@ func migrateUp(ctx context.Context, tx *sql.Tx) (int, error) {
 	return len(pending), nil
 }
 
-// backfillMigrations marks all known migrations as applied without executing them.
-// Used when a database is restored from a backup that predates the schema_migrations
-// tracking table — the schema is already correct, we just need to record that fact.
-func backfillMigrations(ctx context.Context, tx *sql.Tx) (int, error) {
-	entries, err := fs.ReadDir(upMigrations, "schema")
-	if err != nil {
-		return 0, fmt.Errorf("reading embedded migrations for backfill: %w", err)
-	}
-
-	n := 0
-	for _, e := range entries {
-		if e.IsDir() || !strings.HasSuffix(e.Name(), ".up.sql") {
-			continue
-		}
-		v, err := parseVersion(e.Name())
-		if err != nil {
-			return 0, fmt.Errorf("parsing migration filename %q: %w", e.Name(), err)
-		}
-		if _, err := tx.ExecContext(ctx, "INSERT IGNORE INTO schema_migrations (version) VALUES (?)", v); err != nil {
-			return 0, fmt.Errorf("backfilling migration %s: %w", e.Name(), err)
-		}
-		n++
-	}
-	return n, nil
+// isAlreadyExistsError returns true for MySQL errors indicating a schema
+// object already exists (table, column, index/key).
+func isAlreadyExistsError(err error) bool {
+	msg := strings.ToLower(err.Error())
+	return strings.Contains(msg, "already exists") || // 1050 table, 1061 key
+		strings.Contains(msg, "duplicate column") // 1060
 }
diff --git a/internal/storage/embeddeddolt/store.go b/internal/storage/embeddeddolt/store.go
index d9f0dcc4b2..9d5eadf50b 100644
--- a/internal/storage/embeddeddolt/store.go
+++ b/internal/storage/embeddeddolt/store.go
@@ -9,6 +9,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"strings"
 	"sync/atomic"
 	"time"
 
@@ -242,7 +243,11 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error {
 				return fmt.Errorf("dolt add after migrations: %w", err)
 			}
 			if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', 'schema: apply migrations')"); err != nil {
-				return fmt.Errorf("dolt commit after migrations: %w", err)
+				// Backfill migrations may only create dolt_ignore'd tables (e.g. wisps),
+				// leaving nothing staged for commit. This is expected.
+				if !strings.Contains(err.Error(), "nothing to commit") {
+					return fmt.Errorf("dolt commit after migrations: %w", err)
+				}
 			}
 		}
 		return nil


Test patch:
diff --git a/internal/storage/embeddeddolt/schema_test.go b/internal/storage/embeddeddolt/schema_test.go
index 13125f240d..fe928f85a7 100644
--- a/internal/storage/embeddeddolt/schema_test.go
+++ b/internal/storage/embeddeddolt/schema_test.go
@@ -145,8 +145,8 @@ func TestSchemaAfterInit(t *testing.T) {
 	if err := db.QueryRowContext(ctx, "SELECT MAX(version) FROM schema_migrations").Scan(&maxVersion); err != nil {
 		t.Fatalf("reading max migration version: %v", err)
 	}
-	if maxVersion != 23 {
-		t.Errorf("max migration version: got %d, want 23", maxVersion)
+	if maxVersion != embeddeddolt.LatestVersion() {
+		t.Errorf("max migration version: got %d, want %d", maxVersion, embeddeddolt.LatestVersion())
 	}
 
 	// --- Log all tables for debugging ---
@@ -180,17 +180,93 @@ func TestSchemaAfterInit(t *testing.T) {
 	if err := db2.QueryRowContext(ctx, "SELECT COUNT(*) FROM schema_migrations").Scan(&migrationCount); err != nil {
 		t.Fatalf("counting migrations: %v", err)
 	}
-	if migrationCount != 23 {
-		t.Errorf("migration count after second init: got %d, want 23", migrationCount)
+	if migrationCount != embeddeddolt.LatestVersion() {
+		t.Errorf("migration count after second init: got %d, want %d", migrationCount, embeddeddolt.LatestVersion())
 	}
 
 	if err := db2.QueryRowContext(ctx, "SELECT MAX(version) FROM schema_migrations").Scan(&maxVersion); err != nil {
 		t.Fatalf("reading max version after second init: %v", err)
 	}
-	if maxVersion != 23 {
-		t.Errorf("max version after second init: got %d, want 23", maxVersion)
+	if maxVersion != embeddeddolt.LatestVersion() {
+		t.Errorf("max version after second init: got %d, want %d", maxVersion, embeddeddolt.LatestVersion())
 	}
 
 	cleanup2()
 	store2.Close()
 }
+
+// TestBackfillCreatesWispTables simulates the server-to-embedded upgrade path
+// where core tables exist but schema_migrations is empty. The backfill must
+// actually execute migrations (not just mark them applied) so that dolt_ignore'd
+// wisp tables get created. Regression test for GH#2979.
+func TestBackfillCreatesWispTables(t *testing.T) {
+	if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" {
+		t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt tests")
+	}
+
+	ctx := t.Context()
+	tmpDir := t.TempDir()
+	beadsDir := filepath.Join(tmpDir, ".beads")
+	dataDir := filepath.Join(beadsDir, "embeddeddolt")
+
+	// Step 1: Normal init — creates everything including wisp tables.
+	store, err := embeddeddolt.New(ctx, beadsDir, "testdb", "main")
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+
+	// Step 2: Drop wisp tables and clear schema_migrations to simulate
+	// an upgrade from server-mode (core tables exist, no migration tracking,
+	// no wisp tables since they're dolt_ignore'd and weren't imported).
+	db, cleanup, err := embeddeddolt.OpenSQL(ctx, dataDir, "testdb", "main")
+	if err != nil {
+		store.Close()
+		t.Fatalf("OpenSQL: %v", err)
+	}
+	for _, stmt := range []string{
+		"DROP TABLE IF EXISTS wisp_comments",
+		"DROP TABLE IF EXISTS wisp_events",
+		"DROP TABLE IF EXISTS wisp_labels",
+		"DROP TABLE IF EXISTS wisp_dependencies",
+		"DROP TABLE IF EXISTS wisps",
+		"DELETE FROM schema_migrations",
+	} {
+		if _, err := db.ExecContext(ctx, stmt); err != nil {
+			t.Fatalf("setup: %s: %v", stmt, err)
+		}
+	}
+	cleanup()
+	store.Close()
+
+	// Step 3: Re-open. This triggers the backfill path (schema_migrations
+	// empty, issues table exists). The old code would mark all migrations
+	// as applied without executing them, leaving wisp tables missing.
+	store2, err := embeddeddolt.New(ctx, beadsDir, "testdb", "main")
+	if err != nil {
+		t.Fatalf("New after backfill: %v", err)
+	}
+
+	db2, cleanup2, err := embeddeddolt.OpenSQL(ctx, dataDir, "testdb", "main")
+	if err != nil {
+		store2.Close()
+		t.Fatalf("OpenSQL after backfill: %v", err)
+	}
+	defer func() { cleanup2(); store2.Close() }()
+
+	// Verify all wisp tables were created by the backfill.
+	for _, table := range []string{"wisps", "wisp_dependencies", "wisp_labels", "wisp_events", "wisp_comments"} {
+		var count int
+		if err := db2.QueryRowContext(ctx, "SELECT COUNT(*) FROM `"+table+"`").Scan(&count); err != nil {
+			t.Errorf("wisp table %s missing after backfill: %v", table, err)
+		}
+	}
+
+	// Verify schema_migrations is fully populated.
+	var maxVersion int
+	if err := db2.QueryRowContext(ctx, "SELECT MAX(version) FROM schema_migrations").Scan(&maxVersion); err != nil {
+		t.Fatalf("reading max version: %v", err)
+	}
+	if maxVersion != embeddeddolt.LatestVersion() {
+		t.Errorf("max migration version after backfill: got %d, want %d", maxVersion, embeddeddolt.LatestVersion())
+	}
+}