Repository: ethereum/go-ethereum (language: go)

Issue:
Fatal error after crlt-c kill: gap in the chain between ancients [...] and leveldb [...]
#### System information

Geth version: 1.15.6-unstable
CL client & version: nimbus
OS & Version: OSX
Commit hash : 881ee4062f93eaa6d0cc95202326caa7af4f88c9

#### Actual behaviour
After killing geth in the init phase with a ctrl-c, it can't restart.

Killed in the init phase like this:
```
INFO [03-26|10:54:44.566]  - Cancun:                      @1710338135 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [03-26|10:54:44.566] 
INFO [03-26|10:54:44.566] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-26|10:54:44.566] 
INFO [03-26|10:54:44.567] Loaded most recent local block           number=22,130,311 hash=15955c..c22cee age=21s
INFO [03-26|10:54:44.567] Loaded most recent local finalized block number=22,130,237 hash=b807f1..b0ba95 age=15m9s
INFO [03-26|10:54:44.567] Loaded last snap-sync pivot marker       number=22,124,002
WARN [03-26|10:54:44.567] Head state missing, repairing            number=22,130,311 hash=15955c..c22cee snaproot=ce4bcb..5631b6
INFO [03-26|10:54:44.574] Rewound to block with state              number=22,129,711 hash=9c1ded..575df6
^C
```

Restarting fails like this
```
INFO [03-26|11:09:38.551] Starting Geth on Ethereum mainnet...
INFO [03-26|11:09:38.551] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [03-26|11:09:38.552] Maximum peer count                       ETH=50 total=50
INFO [03-26|11:09:38.552] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [03-26|11:09:38.554] Set global gas cap                       cap=50,000,000
INFO [03-26|11:09:38.561] Initializing the KZG library             backend=gokzg
INFO [03-26|11:09:38.572] Enabling metrics collection
INFO [03-26|11:09:38.572] Enabling stand-alone metrics HTTP endpoint address=0.0.0.0:6060
INFO [03-26|11:09:38.572] Starting metrics server                  addr=http://0.0.0.0:6060/debug/metrics
INFO [03-26|11:09:38.572] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [03-26|11:09:38.578] Using pebble as the backing database
INFO [03-26|11:09:38.578] Allocated cache and file handles         database=/home/user/.ethereum/geth/chaindata cache=2.00GiB handles=524,288
INFO [03-26|11:09:39.592] Opened ancient database                  database=/home/user/.ethereum/geth/chaindata/ancient/chain readonly=false
Chain metadata
  databaseVersion: 9 (0x9)
  headBlockHash: 0x9c1ded300eff7bd9888ef50efe834ae4872138c499bc0cac9237c66359575df6
  headFastBlockHash: 0x15955ca3b96163347ac361e0f5ed4f030345a95f2d66b5c0f0c7668111c22cee
  headHeaderHash: 0xa08081168d23b5505909ff2d98547b786e27753748d8787a129bd98e9c01953a
  lastPivotNumber: 22124002 (0x15195e2)
  len(snapshotSyncStatus): 282 bytes
  snapshotDisabled: false
  snapshotJournal: 0 bytes
  snapshotRecoveryNumber: 22130119 (0x151adc7)
  snapshotRoot: 0xce4bcb0e32a40ead20270594392d09540d9b881396fb9198997c337fbb5631b6
  txIndexTail: 19780311 (0x12dd2d7)
  SkeletonSyncStatus: {"Subchains":[{"Head":22130270,"Tail":22129712,"Next":"0x9c1ded300eff7bd9888ef50efe834ae4872138c499bc0cac9237c66359575df6"}],"Finalized":22124001}


Fatal: Failed to register the Ethereum service: gap in the chain between ancients [0 - #22129824] and leveldb [#22130074 - #22130073]
```

#### Steps to reproduce the behaviour
Already happened twice killing geth early, but not sure how reproducible.



Discussion / hints:
`geth removedb` and then keeping ancient and removing only the state solves the problem, but the node has to sync the full state again.

Maybe it would be good to have more granular options in removedb (if that's possible and it would work in this case).
This issue is related to Pebble’s async mode.

During chain repair, certain writes to Pebble are lost due to a crash, 
while writes to the freezer persist. This disrupts the write order between
the two storage systems, leading to a scenario where part of the chain 
segment is deleted from the freezer, but the chain head flag remains 
unchanged.

The reason for selecting async mode is https://github.com/ethereum/go-ethereum/issues/29819
and it's implemented in https://github.com/ethereum/go-ethereum/pull/30573

Unfortunately, async mode is not safe enough in this scenario.

The ideal way to resolve it is using a write option which can ensure the durability
across the application level crash. Let's see the response from Pebble. https://github.com/cockroachdb/pebble/issues/4430
But frankly, it's not that easy to ensure the write order between key-value store
and freezer unless a WAL is introduced manually. Probably sync mode is a safer
choice anyway.


Fix patch:
diff --git a/ethdb/pebble/pebble.go b/ethdb/pebble/pebble.go
index b87ecb259501..969e67af5a3f 100644
--- a/ethdb/pebble/pebble.go
+++ b/ethdb/pebble/pebble.go
@@ -144,7 +144,7 @@ func (l panicLogger) Fatalf(format string, args ...interface{}) {
 
 // New returns a wrapped pebble DB object. The namespace is the prefix that the
 // metrics reporting should use for surfacing internal stats.
-func New(file string, cache int, handles int, namespace string, readonly bool) (*Database, error) {
+func New(file string, cache int, handles int, namespace string, readonly bool, ephemeral bool) (*Database, error) {
 	// Ensure we have some minimal caching and file guarantees
 	if cache < minCache {
 		cache = minCache
@@ -185,7 +185,7 @@ func New(file string, cache int, handles int, namespace string, readonly bool) (
 		fn:           file,
 		log:          logger,
 		quitChan:     make(chan chan error),
-		writeOptions: &pebble.WriteOptions{Sync: false},
+		writeOptions: &pebble.WriteOptions{Sync: !ephemeral},
 	}
 	opt := &pebble.Options{
 		// Pebble has a single combined cache area and the write
diff --git a/node/database.go b/node/database.go
index e3ccb9106678..b7d0d856cbd6 100644
--- a/node/database.go
+++ b/node/database.go
@@ -36,6 +36,11 @@ type openOptions struct {
 	Cache             int    // the capacity(in megabytes) of the data caching
 	Handles           int    // number of files to be open simultaneously
 	ReadOnly          bool
+
+	// Ephemeral means that filesystem sync operations should be avoided:
+	// data integrity in the face of a crash is not important. This option
+	// should typically be used in tests.
+	Ephemeral bool
 }
 
 // openDatabase opens both a disk-based key-value database such as leveldb or pebble, but also
@@ -78,7 +83,7 @@ func openKeyValueDatabase(o openOptions) (ethdb.Database, error) {
 	}
 	if o.Type == rawdb.DBPebble || existingDb == rawdb.DBPebble {
 		log.Info("Using pebble as the backing database")
-		return newPebbleDBDatabase(o.Directory, o.Cache, o.Handles, o.Namespace, o.ReadOnly)
+		return newPebbleDBDatabase(o.Directory, o.Cache, o.Handles, o.Namespace, o.ReadOnly, o.Ephemeral)
 	}
 	if o.Type == rawdb.DBLeveldb || existingDb == rawdb.DBLeveldb {
 		log.Info("Using leveldb as the backing database")
@@ -86,7 +91,7 @@ func openKeyValueDatabase(o openOptions) (ethdb.Database, error) {
 	}
 	// No pre-existing database, no user-requested one either. Default to Pebble.
 	log.Info("Defaulting to pebble as the backing database")
-	return newPebbleDBDatabase(o.Directory, o.Cache, o.Handles, o.Namespace, o.ReadOnly)
+	return newPebbleDBDatabase(o.Directory, o.Cache, o.Handles, o.Namespace, o.ReadOnly, o.Ephemeral)
 }
 
 // newLevelDBDatabase creates a persistent key-value database without a freezer
@@ -102,8 +107,8 @@ func newLevelDBDatabase(file string, cache int, handles int, namespace string, r
 
 // newPebbleDBDatabase creates a persistent key-value database without a freezer
 // moving immutable chain segments into cold storage.
-func newPebbleDBDatabase(file string, cache int, handles int, namespace string, readonly bool) (ethdb.Database, error) {
-	db, err := pebble.New(file, cache, handles, namespace, readonly)
+func newPebbleDBDatabase(file string, cache int, handles int, namespace string, readonly bool, ephemeral bool) (ethdb.Database, error) {
+	db, err := pebble.New(file, cache, handles, namespace, readonly, ephemeral)
 	if err != nil {
 		return nil, err
 	}


Test patch:
diff --git a/core/bench_test.go b/core/bench_test.go
index 155fa6c3b547..00f924076ad5 100644
--- a/core/bench_test.go
+++ b/core/bench_test.go
@@ -183,7 +183,7 @@ func benchInsertChain(b *testing.B, disk bool, gen func(int, *BlockGen)) {
 	if !disk {
 		db = rawdb.NewMemoryDatabase()
 	} else {
-		pdb, err := pebble.New(b.TempDir(), 128, 128, "", false)
+		pdb, err := pebble.New(b.TempDir(), 128, 128, "", false, true)
 		if err != nil {
 			b.Fatalf("cannot create temporary database: %v", err)
 		}
@@ -303,7 +303,7 @@ func makeChainForBench(db ethdb.Database, genesis *Genesis, full bool, count uin
 func benchWriteChain(b *testing.B, full bool, count uint64) {
 	genesis := &Genesis{Config: params.AllEthashProtocolChanges}
 	for i := 0; i < b.N; i++ {
-		pdb, err := pebble.New(b.TempDir(), 1024, 128, "", false)
+		pdb, err := pebble.New(b.TempDir(), 1024, 128, "", false, true)
 		if err != nil {
 			b.Fatalf("error opening database: %v", err)
 		}
@@ -316,7 +316,7 @@ func benchWriteChain(b *testing.B, full bool, count uint64) {
 func benchReadChain(b *testing.B, full bool, count uint64) {
 	dir := b.TempDir()
 
-	pdb, err := pebble.New(dir, 1024, 128, "", false)
+	pdb, err := pebble.New(dir, 1024, 128, "", false, true)
 	if err != nil {
 		b.Fatalf("error opening database: %v", err)
 	}
@@ -332,7 +332,7 @@ func benchReadChain(b *testing.B, full bool, count uint64) {
 	b.ResetTimer()
 
 	for i := 0; i < b.N; i++ {
-		pdb, err = pebble.New(dir, 1024, 128, "", false)
+		pdb, err = pebble.New(dir, 1024, 128, "", false, true)
 		if err != nil {
 			b.Fatalf("error opening database: %v", err)
 		}
diff --git a/core/blockchain_repair_test.go b/core/blockchain_repair_test.go
index 6c52d057adf8..3ff1d77fc835 100644
--- a/core/blockchain_repair_test.go
+++ b/core/blockchain_repair_test.go
@@ -1765,7 +1765,7 @@ func testRepairWithScheme(t *testing.T, tt *rewindTest, snapshots bool, scheme s
 	datadir := t.TempDir()
 	ancient := filepath.Join(datadir, "ancient")
 
-	pdb, err := pebble.New(datadir, 0, 0, "", false)
+	pdb, err := pebble.New(datadir, 0, 0, "", false, true)
 	if err != nil {
 		t.Fatalf("Failed to create persistent key-value database: %v", err)
 	}
@@ -1850,7 +1850,7 @@ func testRepairWithScheme(t *testing.T, tt *rewindTest, snapshots bool, scheme s
 	chain.stopWithoutSaving()
 
 	// Start a new blockchain back up and see where the repair leads us
-	pdb, err = pebble.New(datadir, 0, 0, "", false)
+	pdb, err = pebble.New(datadir, 0, 0, "", false, true)
 	if err != nil {
 		t.Fatalf("Failed to reopen persistent key-value database: %v", err)
 	}
@@ -1915,7 +1915,7 @@ func testIssue23496(t *testing.T, scheme string) {
 	datadir := t.TempDir()
 	ancient := filepath.Join(datadir, "ancient")
 
-	pdb, err := pebble.New(datadir, 0, 0, "", false)
+	pdb, err := pebble.New(datadir, 0, 0, "", false, true)
 	if err != nil {
 		t.Fatalf("Failed to create persistent key-value database: %v", err)
 	}
@@ -1973,7 +1973,7 @@ func testIssue23496(t *testing.T, scheme string) {
 	chain.stopWithoutSaving()
 
 	// Start a new blockchain back up and see where the repair leads us
-	pdb, err = pebble.New(datadir, 0, 0, "", false)
+	pdb, err = pebble.New(datadir, 0, 0, "", false, true)
 	if err != nil {
 		t.Fatalf("Failed to reopen persistent key-value database: %v", err)
 	}
diff --git a/core/blockchain_sethead_test.go b/core/blockchain_sethead_test.go
index 424854b2bf81..e998b510df94 100644
--- a/core/blockchain_sethead_test.go
+++ b/core/blockchain_sethead_test.go
@@ -1969,7 +1969,7 @@ func testSetHeadWithScheme(t *testing.T, tt *rewindTest, snapshots bool, scheme
 	datadir := t.TempDir()
 	ancient := filepath.Join(datadir, "ancient")
 
-	pdb, err := pebble.New(datadir, 0, 0, "", false)
+	pdb, err := pebble.New(datadir, 0, 0, "", false, true)
 	if err != nil {
 		t.Fatalf("Failed to create persistent key-value database: %v", err)
 	}
diff --git a/core/blockchain_snapshot_test.go b/core/blockchain_snapshot_test.go
index 1a6fe38af6d7..23effea15e21 100644
--- a/core/blockchain_snapshot_test.go
+++ b/core/blockchain_snapshot_test.go
@@ -66,7 +66,7 @@ func (basic *snapshotTestBasic) prepare(t *testing.T) (*BlockChain, []*types.Blo
 	datadir := t.TempDir()
 	ancient := filepath.Join(datadir, "ancient")
 
-	pdb, err := pebble.New(datadir, 0, 0, "", false)
+	pdb, err := pebble.New(datadir, 0, 0, "", false, true)
 	if err != nil {
 		t.Fatalf("Failed to create persistent key-value database: %v", err)
 	}
@@ -257,7 +257,7 @@ func (snaptest *crashSnapshotTest) test(t *testing.T) {
 	chain.triedb.Close()
 
 	// Start a new blockchain back up and see where the repair leads us
-	pdb, err := pebble.New(snaptest.datadir, 0, 0, "", false)
+	pdb, err := pebble.New(snaptest.datadir, 0, 0, "", false, true)
 	if err != nil {
 		t.Fatalf("Failed to create persistent key-value database: %v", err)
 	}
diff --git a/core/blockchain_test.go b/core/blockchain_test.go
index 04dfa87b8b88..7fc4807844b1 100644
--- a/core/blockchain_test.go
+++ b/core/blockchain_test.go
@@ -2605,7 +2605,7 @@ func testSideImportPrunedBlocks(t *testing.T, scheme string) {
 	datadir := t.TempDir()
 	ancient := path.Join(datadir, "ancient")
 
-	pdb, err := pebble.New(datadir, 0, 0, "", false)
+	pdb, err := pebble.New(datadir, 0, 0, "", false, true)
 	if err != nil {
 		t.Fatalf("Failed to create persistent key-value database: %v", err)
 	}