Repository: juanfont/headscale

Issue:
[Bug] v0.27.0 Headscale break if I using litestream
### Is this a support request?

- [x] This is not a support request

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Current Behavior

When I run headscale in new version 0.27.0 I received migration table error. I have to remove tables: `_litestream_lock` and `_litestream_seq` for start headscale otherwise it wont start.

```sh
2025-10-27T14:51:49Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite3
SQLite schema failed to validate:

>> Remove table "_litestream_lock"

>> Remove table "_litestream_seq"

2025-10-27T14:51:49Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init database: validating schema: invalid schema (-got, +want):\n\n>> Remove table \"_litestream_lock\"\n\n>> Remove table \"_litestream_seq\"\n"
```

### Expected Behavior

Should start normal

### Steps To Reproduce

1. Start headscale with litestream

### Environment

```markdown
- OS:ubuntu 24.0.4
- Headscale version: 0.27.0
- Tailscale version: 1.88.4
```

### Runtime environment

- [x] Headscale is behind a (reverse) proxy
- [x] Headscale runs in a container

### Debug information

 xx

Discussion:
@porelli: I've got a similar problem but I'm not using litestream. When starting with v0.27.0 I get this:
```
2025-10-29T03:42:21Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/serve.go:32 > Error initializing error="creating new headscale: init state: init database: validating schema: invalid schema (-got, +want):\n\n>> Remove table \"namespaces\"\n"

2025-10-29T03:43:22Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite

SQLite schema failed to validate:

>> Remove table "namespaces"
```

Reverting the container to v0.26.1 everything starts properly.@kradalby: >  I have to remove tables: _litestream_lock and _litestream_seq for start headscale otherwise it wont start

Oh yes, this is an issue. I will have to think a bit on how to solve that, as we really do not want to drop down into the previous hole of not-as-strict schema integrity. 
https://github.com/tailscale/squibble/issues/1 has been filed, hopefully we find a good solution.


> I've got a similar problem but I'm not using litestream. When starting with v0.27.0 I get this:

This must have been a schema we missed in the cleanup, I have to investigate how that happened, but it is good to know as this is the exact condition we now should be able to avoid in the future.

@porelli if you drop that table manually, you should be fine to upgrade.

Solution patch:
diff --git a/flake.nix b/flake.nix
index c064c7fe..f8eb6dd1 100644
--- a/flake.nix
+++ b/flake.nix
@@ -19,7 +19,7 @@
       overlay = _: prev: let
         pkgs = nixpkgs.legacyPackages.${prev.system};
         buildGo = pkgs.buildGo125Module;
-        vendorHash = "sha256-GUIzlPRsyEq1uSTzRNds9p1uVu4pTeH5PAxrJ5Njhis=";
+        vendorHash = "sha256-VOi4PGZ8I+2MiwtzxpKc/4smsL5KcH/pHVkjJfAFPJ0=";
       in {
         headscale = buildGo {
           pname = "headscale";
diff --git a/go.mod b/go.mod
index b96cedf1..67c6c089 100644
--- a/go.mod
+++ b/go.mod
@@ -36,7 +36,7 @@ require (
 	github.com/spf13/viper v1.21.0
 	github.com/stretchr/testify v1.11.1
 	github.com/tailscale/hujson v0.0.0-20250226034555-ec1d1c113d33
-	github.com/tailscale/squibble v0.0.0-20250108170732-a4ca58afa694
+	github.com/tailscale/squibble v0.0.0-20251030164342-4d5df9caa993
 	github.com/tailscale/tailsql v0.0.0-20250421235516-02f85f087b97
 	github.com/tcnksm/go-latest v0.0.0-20170313132115-e3007ae9052e
 	go4.org/netipx v0.0.0-20231129151722-fdeea329fbba
@@ -115,7 +115,7 @@ require (
 	github.com/containerd/errdefs v0.3.0 // indirect
 	github.com/containerd/errdefs/pkg v0.3.0 // indirect
 	github.com/coreos/go-iptables v0.7.1-0.20240112124308-65c67c9f46e6 // indirect
-	github.com/creachadair/mds v0.25.2 // indirect
+	github.com/creachadair/mds v0.25.10 // indirect
 	github.com/dblohm7/wingoes v0.0.0-20240123200102-b75a8a7d7eb0 // indirect
 	github.com/digitalocean/go-smbios v0.0.0-20180907143718-390a4f403a8e // indirect
 	github.com/distribution/reference v0.6.0 // indirect
@@ -159,7 +159,7 @@ require (
 	github.com/jinzhu/now v1.1.5 // indirect
 	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/jsimonetti/rtnetlink v1.4.1 // indirect
-	github.com/klauspost/compress v1.18.0 // indirect
+	github.com/klauspost/compress v1.18.1 // indirect
 	github.com/kr/pretty v0.3.1 // indirect
 	github.com/kr/text v0.2.0 // indirect
 	github.com/lib/pq v1.10.9 // indirect
diff --git a/go.sum b/go.sum
index 1b09acc5..e78e9aff 100644
--- a/go.sum
+++ b/go.sum
@@ -126,6 +126,8 @@ github.com/creachadair/flax v0.0.5 h1:zt+CRuXQASxwQ68e9GHAOnEgAU29nF0zYMHOCrL5wz
 github.com/creachadair/flax v0.0.5/go.mod h1:F1PML0JZLXSNDMNiRGK2yjm5f+L9QCHchyHBldFymj8=
 github.com/creachadair/mds v0.25.2 h1:xc0S0AfDq5GX9KUR5sLvi5XjA61/P6S5e0xFs1vA18Q=
 github.com/creachadair/mds v0.25.2/go.mod h1:+s4CFteFRj4eq2KcGHW8Wei3u9NyzSPzNV32EvjyK/Q=
+github.com/creachadair/mds v0.25.10 h1:9k9JB35D1xhOCFl0liBhagBBp8fWWkKZrA7UXsfoHtA=
+github.com/creachadair/mds v0.25.10/go.mod h1:4hatI3hRM+qhzuAmqPRFvaBM8mONkS7nsLxkcuTYUIs=
 github.com/creachadair/taskgroup v0.13.2 h1:3KyqakBuFsm3KkXi/9XIb0QcA8tEzLHLgaoidf0MdVc=
 github.com/creachadair/taskgroup v0.13.2/go.mod h1:i3V1Zx7H8RjwljUEeUWYT30Lmb9poewSb2XI1yTwD0g=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
@@ -278,6 +280,8 @@ github.com/jsimonetti/rtnetlink v1.4.1 h1:JfD4jthWBqZMEffc5RjgmlzpYttAVw1sdnmiNa
 github.com/jsimonetti/rtnetlink v1.4.1/go.mod h1:xJjT7t59UIZ62GLZbv6PLLo8VFrostJMPBAheR6OM8w=
 github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
 github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
+github.com/klauspost/compress v1.18.1 h1:bcSGx7UbpBqMChDtsF28Lw6v/G94LPrrbMbdC3JH2co=
+github.com/klauspost/compress v1.18.1/go.mod h1:ZQFFVG+MdnR0P+l6wpXgIL4NTtwiKIdBnrBd8Nrxr+0=
 github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
 github.com/klauspost/cpuid/v2 v2.0.10/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
 github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
@@ -461,6 +465,8 @@ github.com/tailscale/setec v0.0.0-20250305161714-445cadbbca3d h1:mnqtPWYyvNiPU9l
 github.com/tailscale/setec v0.0.0-20250305161714-445cadbbca3d/go.mod h1:9BzmlFc3OLqLzLTF/5AY+BMs+clxMqyhSGzgXIm8mNI=
 github.com/tailscale/squibble v0.0.0-20250108170732-a4ca58afa694 h1:95eIP97c88cqAFU/8nURjgI9xxPbD+Ci6mY/a79BI/w=
 github.com/tailscale/squibble v0.0.0-20250108170732-a4ca58afa694/go.mod h1:veguaG8tVg1H/JG5RfpoUW41I+O8ClPElo/fTYr8mMk=
+github.com/tailscale/squibble v0.0.0-20251030164342-4d5df9caa993 h1:FyiiAvDAxpB0DrW2GW3KOVfi3YFOtsQUEeFWbf55JJU=
+github.com/tailscale/squibble v0.0.0-20251030164342-4d5df9caa993/go.mod h1:xJkMmR3t+thnUQhA3Q4m2VSlS5pcOq+CIjmU/xfKKx4=
 github.com/tailscale/tailsql v0.0.0-20250421235516-02f85f087b97 h1:JJkDnrAhHvOCttk8z9xeZzcDlzzkRA7+Duxj9cwOyxk=
 github.com/tailscale/tailsql v0.0.0-20250421235516-02f85f087b97/go.mod h1:9jS8HxwsP2fU4ESZ7DZL+fpH/U66EVlVMzdgznH12RM=
 github.com/tailscale/web-client-prebuilt v0.0.0-20250124233751-d4cd19a26976 h1:UBPHPtv8+nEAy2PD8RyAhOYvau1ek0HDJqLS/Pysi14=
diff --git a/hscontrol/db/db.go b/hscontrol/db/db.go
index e18f2e5d..04c6cc0a 100644
--- a/hscontrol/db/db.go
+++ b/hscontrol/db/db.go
@@ -932,6 +932,26 @@ AND auth_key_id NOT IN (
 				},
 				Rollback: func(db *gorm.DB) error { return nil },
 			},
+			{
+				// Drop all tables that are no longer in use and has existed.
+				// They potentially still present from broken migrations in the past.
+				ID: "202510311551",
+				Migrate: func(tx *gorm.DB) error {
+					for _, oldTable := range []string{"namespaces", "machines", "shared_machines", "kvs", "pre_auth_key_acl_tags", "routes"} {
+						err := tx.Migrator().DropTable(oldTable)
+						if err != nil {
+							log.Trace().Str("table", oldTable).
+								Err(err).
+								Msg("Error dropping old table, continuing...")
+						}
+					}
+
+					return nil
+				},
+				Rollback: func(tx *gorm.DB) error {
+					return nil
+				},
+			},
 			// From this point, the following rules must be followed:
 			// - NEVER use gorm.AutoMigrate, write the exact migration steps needed
 			// - AutoMigrate depends on the struct staying exactly the same, which it won't over time.
@@ -962,7 +982,17 @@ AND auth_key_id NOT IN (
 		ctx, cancel := context.WithTimeout(context.Background(), contextTimeoutSecs*time.Second)
 		defer cancel()
 
-		if err := squibble.Validate(ctx, sqlConn, dbSchema); err != nil {
+		opts := squibble.DigestOptions{
+			IgnoreTables: []string{
+				// Litestream tables, these are inserted by
+				// litestream and not part of our schema
+				// https://litestream.io/how-it-works
+				"_litestream_lock",
+				"_litestream_seq",
+			},
+		}
+
+		if err := squibble.Validate(ctx, sqlConn, dbSchema, &opts); err != nil {
 			return nil, fmt.Errorf("validating schema: %w", err)
 		}
 	}



Test patch:
diff --git a/hscontrol/db/testdata/sqlite/headscale_0.26.1_dump_schema-to-0.27.0-old-table-cleanup.sql b/hscontrol/db/testdata/sqlite/headscale_0.26.1_dump_schema-to-0.27.0-old-table-cleanup.sql
new file mode 100644
index 00000000..388fefbc
--- /dev/null
+++ b/hscontrol/db/testdata/sqlite/headscale_0.26.1_dump_schema-to-0.27.0-old-table-cleanup.sql
@@ -0,0 +1,40 @@
+PRAGMA foreign_keys=OFF;
+BEGIN TRANSACTION;
+CREATE TABLE `migrations` (`id` text,PRIMARY KEY (`id`));
+INSERT INTO migrations VALUES('202312101416');
+INSERT INTO migrations VALUES('202312101430');
+INSERT INTO migrations VALUES('202402151347');
+INSERT INTO migrations VALUES('2024041121742');
+INSERT INTO migrations VALUES('202406021630');
+INSERT INTO migrations VALUES('202409271400');
+INSERT INTO migrations VALUES('202407191627');
+INSERT INTO migrations VALUES('202408181235');
+INSERT INTO migrations VALUES('202501221827');
+INSERT INTO migrations VALUES('202501311657');
+INSERT INTO migrations VALUES('202502070949');
+INSERT INTO migrations VALUES('202502131714');
+INSERT INTO migrations VALUES('202502171819');
+INSERT INTO migrations VALUES('202505091439');
+INSERT INTO migrations VALUES('202505141324');
+CREATE TABLE `users` (`id` integer PRIMARY KEY AUTOINCREMENT,`created_at` datetime,`updated_at` datetime,`deleted_at` datetime,`name` text,`display_name` text,`email` text,`provider_identifier` text,`provider` text,`profile_pic_url` text);
+CREATE TABLE `pre_auth_keys` (`id` integer PRIMARY KEY AUTOINCREMENT,`key` text,`user_id` integer,`reusable` numeric,`ephemeral` numeric DEFAULT false,`used` numeric DEFAULT false,`tags` text,`created_at` datetime,`expiration` datetime,CONSTRAINT `fk_pre_auth_keys_user` FOREIGN KEY (`user_id`) REFERENCES `users`(`id`) ON DELETE SET NULL);
+CREATE TABLE `api_keys` (`id` integer PRIMARY KEY AUTOINCREMENT,`prefix` text,`hash` blob,`created_at` datetime,`expiration` datetime,`last_seen` datetime);
+CREATE TABLE IF NOT EXISTS "nodes"  (`id` integer PRIMARY KEY AUTOINCREMENT,`machine_key` text,`node_key` text,`disco_key` text,`endpoints` text,`host_info` text,`ipv4` text,`ipv6` text,`hostname` text,`given_name` varchar(63),`user_id` integer,`register_method` text,`forced_tags` text,`auth_key_id` integer,`expiry` datetime,`last_seen` datetime,`approved_routes` text,`created_at` datetime,`updated_at` datetime,`deleted_at` datetime,CONSTRAINT `fk_nodes_user` FOREIGN KEY (`user_id`) REFERENCES `users`(`id`) ON DELETE CASCADE,CONSTRAINT `fk_nodes_auth_key` FOREIGN KEY (`auth_key_id`) REFERENCES `pre_auth_keys`(`id`));
+CREATE TABLE `policies` (`id` integer PRIMARY KEY AUTOINCREMENT,`created_at` datetime,`updated_at` datetime,`deleted_at` datetime,`data` text);
+DELETE FROM sqlite_sequence;
+INSERT INTO sqlite_sequence VALUES('nodes',0);
+CREATE INDEX `idx_users_deleted_at` ON `users`(`deleted_at`);
+CREATE UNIQUE INDEX `idx_api_keys_prefix` ON `api_keys`(`prefix`);
+CREATE INDEX `idx_policies_deleted_at` ON `policies`(`deleted_at`);
+CREATE UNIQUE INDEX idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL;
+CREATE UNIQUE INDEX idx_name_provider_identifier ON users (name,provider_identifier);
+CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users (name) WHERE provider_identifier IS NULL;
+
+-- Create all the old tables we have had and ensure they are clean up.
+CREATE TABLE `namespaces` (`id` text,PRIMARY KEY (`id`));
+CREATE TABLE `machines` (`id` text,PRIMARY KEY (`id`));
+CREATE TABLE `kvs` (`id` text,PRIMARY KEY (`id`));
+CREATE TABLE `shared_machines` (`id` text,PRIMARY KEY (`id`));
+CREATE TABLE `pre_auth_key_acl_tags` (`id` text,PRIMARY KEY (`id`));
+CREATE TABLE `routes` (`id` text,PRIMARY KEY (`id`));
+COMMIT;
diff --git a/hscontrol/db/testdata/sqlite/headscale_0.26.1_schema-litestream.sql b/hscontrol/db/testdata/sqlite/headscale_0.26.1_schema-litestream.sql
new file mode 100644
index 00000000..3fc2b319
--- /dev/null
+++ b/hscontrol/db/testdata/sqlite/headscale_0.26.1_schema-litestream.sql
@@ -0,0 +1,14 @@
+CREATE TABLE `migrations` (`id` text,PRIMARY KEY (`id`));
+CREATE TABLE `users` (`id` integer PRIMARY KEY AUTOINCREMENT,`created_at` datetime,`updated_at` datetime,`deleted_at` datetime,`name` text,`display_name` text,`email` text,`provider_identifier` text,`provider` text,`profile_pic_url` text);
+CREATE INDEX `idx_users_deleted_at` ON `users`(`deleted_at`);
+CREATE TABLE `pre_auth_keys` (`id` integer PRIMARY KEY AUTOINCREMENT,`key` text,`user_id` integer,`reusable` numeric,`ephemeral` numeric DEFAULT false,`used` numeric DEFAULT false,`tags` text,`created_at` datetime,`expiration` datetime,CONSTRAINT `fk_pre_auth_keys_user` FOREIGN KEY (`user_id`) REFERENCES `users`(`id`) ON DELETE SET NULL);
+CREATE TABLE `api_keys` (`id` integer PRIMARY KEY AUTOINCREMENT,`prefix` text,`hash` blob,`created_at` datetime,`expiration` datetime,`last_seen` datetime);
+CREATE UNIQUE INDEX `idx_api_keys_prefix` ON `api_keys`(`prefix`);
+CREATE TABLE IF NOT EXISTS "nodes"  (`id` integer PRIMARY KEY AUTOINCREMENT,`machine_key` text,`node_key` text,`disco_key` text,`endpoints` text,`host_info` text,`ipv4` text,`ipv6` text,`hostname` text,`given_name` varchar(63),`user_id` integer,`register_method` text,`forced_tags` text,`auth_key_id` integer,`expiry` datetime,`last_seen` datetime,`approved_routes` text,`created_at` datetime,`updated_at` datetime,`deleted_at` datetime,CONSTRAINT `fk_nodes_user` FOREIGN KEY (`user_id`) REFERENCES `users`(`id`) ON DELETE CASCADE,CONSTRAINT `fk_nodes_auth_key` FOREIGN KEY (`auth_key_id`) REFERENCES `pre_auth_keys`(`id`));
+CREATE TABLE `policies` (`id` integer PRIMARY KEY AUTOINCREMENT,`created_at` datetime,`updated_at` datetime,`deleted_at` datetime,`data` text);
+CREATE INDEX `idx_policies_deleted_at` ON `policies`(`deleted_at`);
+CREATE UNIQUE INDEX idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL;
+CREATE UNIQUE INDEX idx_name_provider_identifier ON users (name,provider_identifier);
+CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users (name) WHERE provider_identifier IS NULL;
+CREATE TABLE _litestream_seq (id INTEGER PRIMARY KEY, seq INTEGER);
+CREATE TABLE _litestream_lock (id INTEGER);