Repository: hyperledger/besu
Pull request #8352: 7935: add era1 format to blocks importer
Author: login: Matilda-Clerke
Labels: name: history
description: reduce disk reqs thru history mgmt
Created: 2025-02-26T05:46:11Z | Merged: 2025-03-11T02:42:56Z

## PR description
Adds the ability to import blocks from era1 files
## Fixed Issue(s)
https://github.com/hyperledger/besu/issues/7935


Diff:
diff --git a/.gitattributes b/.gitattributes
index 79522fa32e7..2e408995ec2 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -17,4 +17,5 @@
 *.p12 binary
 *.db binary
 *.woff2 binary
+*.era1 binary
 goss-linux-amd64 binary
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d5708b3185d..fffb5940845 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -40,6 +40,7 @@
 - Update `eth_getLogs` to return a `Block not found` error when the requested block is not found. [#8290](https://github.com/hyperledger/besu/pull/8290)
 - Change `Invalid block, unable to parse RLP` RPC error message to `Invalid block param (block not found)` [#8328](https://github.com/hyperledger/besu/pull/8328)
 - Support pending transaction score when saving and restoring txpool [#8363](https://github.com/hyperledger/besu/pull/8363)
+- Add era1 format to blocks import subcommand [#7935](https://github.com/hyperledger/besu/issues/7935)
 
 ### Bug fixes
 - Add missing RPC method `debug_accountRange` to `RpcMethod.java` so this method can be used with `--rpc-http-api-method-no-auth` [#8153](https://github.com/hyperledger/besu/issues/8153)
diff --git a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/ThreadBesuNodeRunner.java b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/ThreadBesuNodeRunner.java
index a0dc5b47b9e..03ff0d1e613 100644
--- a/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/ThreadBesuNodeRunner.java
+++ b/acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/ThreadBesuNodeRunner.java
@@ -19,6 +19,7 @@
 import org.hyperledger.besu.Runner;
 import org.hyperledger.besu.RunnerBuilder;
 import org.hyperledger.besu.chainexport.RlpBlockExporter;
+import org.hyperledger.besu.chainimport.Era1BlockImporter;
 import org.hyperledger.besu.chainimport.JsonBlockImporter;
 import org.hyperledger.besu.chainimport.RlpBlockImporter;
 import org.hyperledger.besu.cli.BesuCommand;
@@ -632,6 +633,7 @@ BesuCommand provideBesuCommand(final BesuPluginContextImpl pluginContext) {
           new BesuCommand(
               RlpBlockImporter::new,
               JsonBlockImporter::new,
+              Era1BlockImporter::new,
               RlpBlockExporter::new,
               new RunnerBuilder(),
               new BesuController.Builder(),
diff --git a/besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java b/besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
new file mode 100644
index 00000000000..cead4e229d1
--- /dev/null
+++ b/besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
@@ -0,0 +1,171 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    final ProtocolSchedule protocolSchedule = controller.getProtocolSchedule();
+    final BlockHeaderFunctions blockHeaderFunctions =
+        ScheduleBasedBlockHeaderFunctions.create(protocolSchedule);
+    final ProtocolContext context = controller.getProtocolContext();
+
+    Era1Reader reader = new Era1Reader(new SnappyFactory());
+
+    final List<Future<BlockHeader>> headersFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<BlockBody>> bodiesFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<List<TransactionReceipt>>> receiptsFutures =
+        new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    reader.read(
+        new FileInputStream(path.toFile()),
+        new Era1ReaderListener() {
+
+          @Override
+          public void handleExecutionBlockHeader(
+              final Era1ExecutionBlockHeader executionBlockHeader) {
+            headersFutures.add(
+                CompletableFuture.supplyAsync(
+                    () ->
+                        BlockHeader.readFrom(
+                            new BytesValueRLPInput(
+                                Bytes.wrap(executionBlockHeader.header()), false),
+                            blockHeaderFunctions)));
+          }
+
+          @Override
+          public void handleExecutionBlockBody(final Era1ExecutionBlockBody executionBlockBody) {
+            bodiesFutures.add(
+                CompletableFuture.supplyAsync(
+                    () ->
+                        BlockBody.readWrappedBodyFrom(
+                            new BytesValueRLPInput(Bytes.wrap(executionBlockBody.block()), false),
+                            blockHeaderFunctions,
+                            true)));
+          }
+
+          @Override
+          public void handleExecutionBlockReceipts(
+              final Era1ExecutionBlockReceipts executionBlockReceipts) {
+            receiptsFutures.add(
+                CompletableFuture.supplyAsync(
+                    () -> {
+                      RLPInput input =
+                          new BytesValueRLPInput(
+                              Bytes.wrap(executionBlockReceipts.receipts()), false);
+                      final List<TransactionReceipt> receiptsForBlock = new ArrayList<>();
+                      input.readList((in) -> receiptsForBlock.add(TransactionReceipt.readFrom(in)));
+                      return receiptsForBlock;
+                    }));
+          }
+
+          @Override
+          public void handleBlockIndex(final Era1BlockIndex blockIndex) {
+            // not really necessary, do nothing
+          }
+        });
+
+    LOG.info("Read {} blocks, now importing", headersFutures.size());
+
+    Block block = null;
+    for (int i = 0; i < headersFutures.size(); i++) {
+      BlockHeader blockHeader = headersFutures.get(i).get(10, TimeUnit.SECONDS);
+      BlockImporter blockImporter =
+          protocolSchedule.getByBlockHeader(blockHeader).getBlockImporter();
+      block = new Block(blockHeader, bodiesFutures.get(i).get(10, TimeUnit.SECONDS));
+
+      BlockImportResult importResult =
+          blockImporter.importBlockForSyncing(
+              context,
+              block,
+              receiptsFutures.get(i).get(10, TimeUnit.SECONDS),
+              HeaderValidationMode.NONE,
+              HeaderValidationMode.NONE,
+              BodyValidationMode.NONE,
+              false);
+      if (importResult.getStatus() != BlockImportResult.BlockImportStatus.IMPORTED) {
+        LOG.warn(
+            "Failed to import block {} due to {}",
+            blockHeader.getNumber(),
+            importResult.getStatus());
+      } else if (i % IMPORT_COUNT_FOR_LOG_UPDATE == 0) {
+        LOG.info("{}/{} blocks imported", i, headersFutures.size());
+      }
+    }
+    LOG.info("Done importing {} blocks", headersFutures.size());
+  }
+
+  @Override
+  public void close() throws IOException {}
+}
diff --git a/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java b/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java
index c92a6a9b25f..c0a50432c10 100644
--- a/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java
+++ b/besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java
@@ -31,6 +31,7 @@
 import org.hyperledger.besu.Runner;
 import org.hyperledger.besu.RunnerBuilder;
 import org.hyperledger.besu.chainexport.RlpBlockExporter;
+import org.hyperledger.besu.chainimport.Era1BlockImporter;
 import org.hyperledger.besu.chainimport.JsonBlockImporter;
 import org.hyperledger.besu.chainimport.RlpBlockImporter;
 import org.hyperledger.besu.cli.config.EthNetworkConfig;
@@ -284,6 +285,7 @@ public class BesuCommand implements DefaultCommandValues, Runnable {
 
   private final Supplier<RlpBlockImporter> rlpBlockImporter;
   private final Function<BesuController, JsonBlockImporter> jsonBlockImporterFactory;
+  private final Supplier<Era1BlockImporter> era1BlockImporter;
   private final Function<Blockchain, RlpBlockExporter> rlpBlockExporterFactory;
 
   // Unstable CLI options
@@ -719,6 +721,7 @@ static class PrivacyOptionGroup {
    *
    * @param rlpBlockImporter RlpBlockImporter supplier
    * @param jsonBlockImporterFactory instance of {@code Function<BesuController, JsonBlockImporter>}
+   * @param era1BlockImporter Era1BlockImporter supplier
    * @param rlpBlockExporterFactory instance of {@code Function<Blockchain, RlpBlockExporter>}
    * @param runnerBuilder instance of RunnerBuilder
    * @param controllerBuilder instance of BesuController.Builder
@@ -729,6 +732,7 @@ static class PrivacyOptionGroup {
   public BesuCommand(
       final Supplier<RlpBlockImporter> rlpBlockImporter,
       final Function<BesuController, JsonBlockImporter> jsonBlockImporterFactory,
+      final Supplier<Era1BlockImporter> era1BlockImporter,
       final Function<Blockchain, RlpBlockExporter> rlpBlockExporterFactory,
       final RunnerBuilder runnerBuilder,
       final BesuController.Builder controllerBuilder,
@@ -738,6 +742,7 @@ public BesuCommand(
     this(
         rlpBlockImporter,
         jsonBlockImporterFactory,
+        era1BlockImporter,
         rlpBlockExporterFactory,
         runnerBuilder,
         controllerBuilder,
@@ -760,6 +765,7 @@ public BesuCommand(
    *
    * @param rlpBlockImporter RlpBlockImporter supplier
    * @param jsonBlockImporterFactory instance of {@code Function<BesuController, JsonBlockImporter>}
+   * @param era1BlockImporter Era1BlockImporter supplier
    * @param rlpBlockExporterFactory instance of {@code Function<Blockchain, RlpBlockExporter>}
    * @param runnerBuilder instance of RunnerBuilder
    * @param controllerBuilder instance of BesuController.Builder
@@ -780,6 +786,7 @@ public BesuCommand(
   protected BesuCommand(
       final Supplier<RlpBlockImporter> rlpBlockImporter,
       final Function<BesuController, JsonBlockImporter> jsonBlockImporterFactory,
+      final Supplier<Era1BlockImporter> era1BlockImporter,
       final Function<Blockchain, RlpBlockExporter> rlpBlockExporterFactory,
       final RunnerBuilder runnerBuilder,
       final BesuController.Builder controllerBuilder,
@@ -798,8 +805,9 @@ protected BesuCommand(
 
     this.logger = commandLogger;
     this.rlpBlockImporter = rlpBlockImporter;
-    this.rlpBlockExporterFactory = rlpBlockExporterFactory;
     this.jsonBlockImporterFactory = jsonBlockImporterFactory;
+    this.era1BlockImporter = era1BlockImporter;
+    this.rlpBlockExporterFactory = rlpBlockExporterFactory;
     this.runnerBuilder = runnerBuilder;
     this.controllerBuilder = controllerBuilder;
     this.besuPluginContext = besuPluginContext;
@@ -1100,6 +1108,7 @@ private void addSubCommands(final InputStream in) {
         new BlocksSubCommand(
             rlpBlockImporter,
             jsonBlockImporterFactory,
+            era1BlockImporter,
             rlpBlockExporterFactory,
             commandLine.getOut()));
     commandLine.addSubcommand(
diff --git a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/blocks/BlockImportFormat.java b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/blocks/BlockImportFormat.java
index 77e58cfb7bd..b9677b1a1cd 100644
--- a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/blocks/BlockImportFormat.java
+++ b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/blocks/BlockImportFormat.java
@@ -19,5 +19,7 @@ public enum BlockImportFormat {
   /** RLP block import format. */
   RLP,
   /** Json block import format. */
-  JSON
+  JSON,
+  /** Era1 block import format. */
+  ERA1,
 }
diff --git a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/blocks/BlocksSubCommand.java b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/blocks/BlocksSubCommand.java
index 4c727f924a0..c05dca5fb4b 100644
--- a/besu/src/main/java/org/hyperledger/besu/cli/subcommands/blocks/BlocksSubCommand.java
+++ b/besu/src/main/java/org/hyperledger/besu/cli/subcommands/blocks/BlocksSubCommand.java
@@ -18,6 +18,7 @@
 import static org.hyperledger.besu.cli.subcommands.blocks.BlocksSubCommand.COMMAND_NAME;
 
 import org.hyperledger.besu.chainexport.RlpBlockExporter;
+import org.hyperledger.besu.chainimport.Era1BlockImporter;
 import org.hyperledger.besu.chainimport.JsonBlockImporter;
 import org.hyperledger.besu.chainimport.RlpBlockImporter;
 import org.hyperledger.besu.cli.BesuCommand;
@@ -51,6 +52,7 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Optional;
+import java.util.concurrent.TimeoutException;
 import java.util.function.Function;
 import java.util.function.Supplier;
 
@@ -92,6 +94,7 @@ public class BlocksSubCommand implements Runnable {
 
   private final Supplier<RlpBlockImporter> rlpBlockImporter;
   private final Function<BesuController, JsonBlockImporter> jsonBlockImporterFactory;
+  private final Supplier<Era1BlockImporter> era1BlockImporter;
   private final Function<Blockchain, RlpBlockExporter> rlpBlockExporterFactory;
 
   private final PrintWriter out;
@@ -101,17 +104,20 @@ public class BlocksSubCommand implements Runnable {
    *
    * @param rlpBlockImporter the RLP block importer
    * @param jsonBlockImporterFactory the Json block importer factory
+   * @param era1BlockImporter the era1 block importer supplier
    * @param rlpBlockExporterFactory the RLP block exporter factory
    * @param out Instance of PrintWriter where command usage will be written.
    */
   public BlocksSubCommand(
       final Supplier<RlpBlockImporter> rlpBlockImporter,
       final Function<BesuController, JsonBlockImporter> jsonBlockImporterFactory,
+      final Supplier<Era1BlockImporter> era1BlockImporter,
       final Function<Blockchain, RlpBlockExporter> rlpBlockExporterFactory,
       final PrintWriter out) {
     this.rlpBlockImporter = rlpBlockImporter;
-    this.rlpBlockExporterFactory = rlpBlockExporterFactory;
     this.jsonBlockImporterFactory = jsonBlockImporterFactory;
+    this.era1BlockImporter = era1BlockImporter;
+    this.rlpBlockExporterFactory = rlpBlockExporterFactory;
     this.out = out;
   }
 
@@ -198,6 +204,7 @@ public void run() {
       checkCommand(parentCommand);
       checkNotNull(parentCommand.rlpBlockImporter);
       checkNotNull(parentCommand.jsonBlockImporterFactory);
+      checkNotNull(parentCommand.era1BlockImporter);
       if (blockImportFiles.isEmpty()) {
         throw new ParameterException(spec.commandLine(), "No files specified to import.");
       }
@@ -214,12 +221,9 @@ public void run() {
           try {
             LOG.info("Importing from {}", path);
             switch (format) {
-              case RLP:
-                importRlpBlocks(controller, path);
-                break;
-              case JSON:
-                importJsonBlocks(controller, path);
-                break;
+              case RLP -> importRlpBlocks(controller, path);
+              case JSON -> importJsonBlocks(controller, path);
+              case ERA1 -> importEra1Blocks(controller, path);
             }
           } catch (final FileNotFoundException e) {
             if (blockImportFiles.size() == 1) {
@@ -297,6 +301,14 @@ private void importRlpBlocks(final BesuController controller, final Path path)
           .get()
           .importBlockchain(path, controller, skipPow, startBlock, endBlock);
     }
+
+    private void importEra1Blocks(final BesuController controller, final Path path)
+        throws IOException,
+            java.util.concurrent.ExecutionException,
+            InterruptedException,
+            TimeoutException {
+      parentCommand.era1BlockImporter.get().importBlocks(controller, path);
+    }
   }
 
   /**
diff --git a/besu/src/main/java/org/hyperledger/besu/components/BesuCommandModule.java b/besu/src/main/java/org/hyperledger/besu/components/BesuCommandModule.java
index ce4f00db737..cef6ebd2d38 100644
--- a/besu/src/main/java/org/hyperledger/besu/components/BesuCommandModule.java
+++ b/besu/src/main/java/org/hyperledger/besu/components/BesuCommandModule.java
@@ -17,6 +17,7 @@
 import org.hyperledger.besu.Besu;
 import org.hyperledger.besu.RunnerBuilder;
 import org.hyperledger.besu.chainexport.RlpBlockExporter;
+import org.hyperledger.besu.chainimport.Era1BlockImporter;
 import org.hyperledger.besu.chainimport.JsonBlockImporter;
 import org.hyperledger.besu.chainimport.RlpBlockImporter;
 import org.hyperledger.besu.cli.BesuCommand;
@@ -50,6 +51,7 @@ BesuCommand provideBesuCommand(final @Named("besuCommandLogger") Logger commandL
         new BesuCommand(
             RlpBlockImporter::new,
             JsonBlockImporter::new,
+            Era1BlockImporter::new,
             RlpBlockExporter::new,
             new RunnerBuilder(),
             new BesuController.Builder(),
diff --git a/besu/src/test/java/org/hyperledger/besu/chainimport/Era1BlockImporterTest.java b/besu/src/test/java/org/hyperledger/besu/chainimport/Era1BlockImporterTest.java
new file mode 100644
index 00000000000..47871d4f9a0
--- /dev/null
+++ b/besu/src/test/java/org/hyperledger/besu/chainimport/Era1BlockImporterTest.java
@@ -0,0 +1,95 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import static org.mockito.Mockito.mock;
+
+import org.hyperledger.besu.cli.config.EthNetworkConfig;
+import org.hyperledger.besu.cli.config.NetworkName;
+import org.hyperledger.besu.components.BesuComponent;
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.cryptoservices.NodeKeyUtils;
+import org.hyperledger.besu.ethereum.api.ImmutableApiConfiguration;
+import org.hyperledger.besu.ethereum.chain.Blockchain;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.InMemoryKeyValueStorageProvider;
+import org.hyperledger.besu.ethereum.core.MiningConfiguration;
+import org.hyperledger.besu.ethereum.core.PrivacyParameters;
+import org.hyperledger.besu.ethereum.eth.EthProtocolConfiguration;
+import org.hyperledger.besu.ethereum.eth.sync.SyncMode;
+import org.hyperledger.besu.ethereum.eth.sync.SynchronizerConfiguration;
+import org.hyperledger.besu.ethereum.eth.transactions.TransactionPoolConfiguration;
+import org.hyperledger.besu.ethereum.p2p.config.NetworkingConfiguration;
+import org.hyperledger.besu.evm.internal.EvmConfiguration;
+import org.hyperledger.besu.metrics.noop.NoOpMetricsSystem;
+import org.hyperledger.besu.testutil.BlockTestUtil;
+import org.hyperledger.besu.testutil.TestClock;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeoutException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class Era1BlockImporterTest {
+  @TempDir Path dataDirectory;
+
+  private final Era1BlockImporter era1BlockImporter = new Era1BlockImporter();
+
+  @Test
+  public void testImport()
+      throws IOException,
+          ExecutionException,
+          InterruptedException,
+          TimeoutException,
+          URISyntaxException {
+    final Path source =
+        Path.of(
+            BlockTestUtil.class
+                .getClassLoader()
+                .getResource("mainnet-00000-5ec1ffb8.era1")
+                .toURI());
+    final BesuController targetController =
+        new BesuController.Builder()
+            .fromEthNetworkConfig(
+                EthNetworkConfig.getNetworkConfig(NetworkName.MAINNET), SyncMode.FAST)
+            .synchronizerConfiguration(SynchronizerConfiguration.builder().build())
+            .ethProtocolConfiguration(EthProtocolConfiguration.defaultConfig())
+            .storageProvider(new InMemoryKeyValueStorageProvider())
+            .networkId(BigInteger.ONE)
+            .miningParameters(MiningConfiguration.newDefault())
+            .nodeKey(NodeKeyUtils.generate())
+            .metricsSystem(new NoOpMetricsSystem())
+            .privacyParameters(PrivacyParameters.DEFAULT)
+            .clock(TestClock.fixed())
+            .transactionPoolConfiguration(TransactionPoolConfiguration.DEFAULT)
+            .evmConfiguration(EvmConfiguration.DEFAULT)
+            .networkConfiguration(NetworkingConfiguration.create())
+            .besuComponent(mock(BesuComponent.class))
+            .apiConfiguration(ImmutableApiConfiguration.builder().build())
+            .dataDirectory(dataDirectory)
+            .build();
+    era1BlockImporter.importBlocks(targetController, source);
+
+    Blockchain blockchain = targetController.getProtocolContext().getBlockchain();
+    BlockHeader chainHeadHeader = blockchain.getChainHeadHeader();
+    Assertions.assertEquals(8191, chainHeadHeader.getNumber());
+  }
+}
diff --git a/besu/src/test/java/org/hyperledger/besu/cli/CommandTestAbstract.java b/besu/src/test/java/org/hyperledger/besu/cli/CommandTestAbstract.java
index b6464e6ff46..e04ec95ca1c 100644
--- a/besu/src/test/java/org/hyperledger/besu/cli/CommandTestAbstract.java
+++ b/besu/src/test/java/org/hyperledger/besu/cli/CommandTestAbstract.java
@@ -31,6 +31,7 @@
 import org.hyperledger.besu.Runner;
 import org.hyperledger.besu.RunnerBuilder;
 import org.hyperledger.besu.chainexport.RlpBlockExporter;
+import org.hyperledger.besu.chainimport.Era1BlockImporter;
 import org.hyperledger.besu.chainimport.JsonBlockImporter;
 import org.hyperledger.besu.chainimport.RlpBlockImporter;
 import org.hyperledger.besu.cli.config.EthNetworkConfig;
@@ -216,6 +217,7 @@ public abstract class CommandTestAbstract {
   @Mock protected RlpBlockExporter rlpBlockExporter;
   @Mock protected JsonBlockImporter jsonBlockImporter;
   @Mock protected RlpBlockImporter rlpBlockImporter;
+  @Mock protected Era1BlockImporter era1BlockImporter;
   @Mock protected StorageServiceImpl storageService;
   @Mock protected TransactionSelectionServiceImpl txSelectionService;
   @Mock protected SecurityModuleServiceImpl securityModuleService;
@@ -472,6 +474,7 @@ private TestBesuCommand getTestBesuCommand(final TestType testType) {
         return new TestBesuCommandWithRequiredOption(
             () -> rlpBlockImporter,
             this::jsonBlockImporterFactory,
+            () -> era1BlockImporter,
             (blockchain) -> rlpBlockExporter,
             mockRunnerBuilder,
             mockControllerBuilderFactory,
@@ -485,6 +488,7 @@ private TestBesuCommand getTestBesuCommand(final TestType testType) {
         return new TestBesuCommand(
             () -> rlpBlockImporter,
             this::jsonBlockImporterFactory,
+            () -> era1BlockImporter,
             (blockchain) -> rlpBlockExporter,
             mockRunnerBuilder,
             mockControllerBuilderFactory,
@@ -498,6 +502,7 @@ private TestBesuCommand getTestBesuCommand(final TestType testType) {
         return new TestBesuCommandWithoutPortCheck(
             () -> rlpBlockImporter,
             this::jsonBlockImporterFactory,
+            () -> era1BlockImporter,
             (blockchain) -> rlpBlockExporter,
             mockRunnerBuilder,
             mockControllerBuilderFactory,
@@ -538,6 +543,7 @@ public static class TestBesuCommand extends BesuCommand {
     TestBesuCommand(
         final Supplier<RlpBlockImporter> mockBlockImporter,
         final Function<BesuController, JsonBlockImporter> jsonBlockImporterFactory,
+        final Supplier<Era1BlockImporter> era1BlockImporter,
         final Function<Blockchain, RlpBlockExporter> rlpBlockExporterFactory,
         final RunnerBuilder mockRunnerBuilder,
         final BesuController.Builder controllerBuilderFactory,
@@ -550,6 +556,7 @@ public static class TestBesuCommand extends BesuCommand {
       super(
           mockBlockImporter,
           jsonBlockImporterFactory,
+          era1BlockImporter,
           rlpBlockExporterFactory,
           mockRunnerBuilder,
           controllerBuilderFactory,
@@ -635,6 +642,7 @@ public static class TestBesuCommandWithRequiredOption extends TestBesuCommand {
     TestBesuCommandWithRequiredOption(
         final Supplier<RlpBlockImporter> mockBlockImporter,
         final Function<BesuController, JsonBlockImporter> jsonBlockImporterFactory,
+        final Supplier<Era1BlockImporter> era1BlockImporter,
         final Function<Blockchain, RlpBlockExporter> rlpBlockExporterFactory,
         final RunnerBuilder mockRunnerBuilder,
         final BesuController.Builder controllerBuilderFactory,
@@ -647,6 +655,7 @@ public static class TestBesuCommandWithRequiredOption extends TestBesuCommand {
       super(
           mockBlockImporter,
           jsonBlockImporterFactory,
+          era1BlockImporter,
           rlpBlockExporterFactory,
           mockRunnerBuilder,
           controllerBuilderFactory,
@@ -669,6 +678,7 @@ public static class TestBesuCommandWithoutPortCheck extends TestBesuCommand {
     TestBesuCommandWithoutPortCheck(
         final Supplier<RlpBlockImporter> mockBlockImporter,
         final Function<BesuController, JsonBlockImporter> jsonBlockImporterFactory,
+        final Supplier<Era1BlockImporter> era1BlockImporter,
         final Function<Blockchain, RlpBlockExporter> rlpBlockExporterFactory,
         final RunnerBuilder mockRunnerBuilder,
         final BesuController.Builder controllerBuilderFactory,
@@ -681,6 +691,7 @@ public static class TestBesuCommandWithoutPortCheck extends TestBesuCommand {
       super(
           mockBlockImporter,
           jsonBlockImporterFactory,
+          era1BlockImporter,
           rlpBlockExporterFactory,
           mockRunnerBuilder,
           controllerBuilderFactory,
diff --git a/besu/src/test/java/org/hyperledger/besu/cli/subcommands/blocks/BlocksSubCommandTest.java b/besu/src/test/java/org/hyperledger/besu/cli/subcommands/blocks/BlocksSubCommandTest.java
index f43d039b4a3..69dd1a4a9d9 100644
--- a/besu/src/test/java/org/hyperledger/besu/cli/subcommands/blocks/BlocksSubCommandTest.java
+++ b/besu/src/test/java/org/hyperledger/besu/cli/subcommands/blocks/BlocksSubCommandTest.java
@@ -72,7 +72,7 @@ public class BlocksSubCommandTest extends CommandTestAbstract {
           + "                               (exclusive).  If not specified all blocks after\n"
           + "                               the start block will be imported.\n"
           + "      --format=<format>      The type of data to be imported, possible values\n"
-          + "                               are: RLP, JSON (default: RLP).\n"
+          + "                               are: RLP, JSON, ERA1 (default: RLP).\n"
           + "      --from[=<FILE>...]     File containing blocks to import.\n"
           + "  -h, --help                 Show this help message and exit.\n"
           + "      --run                  Start besu after importing.\n"
diff --git a/besu/src/test/resources/mainnet-00000-5ec1ffb8.era1 b/besu/src/test/resources/mainnet-00000-5ec1ffb8.era1
new file mode 100755
index 00000000000..dbc1d316cff
Binary files /dev/null and b/besu/src/test/resources/mainnet-00000-5ec1ffb8.era1 differ
diff --git a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/BlockHeader.java b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/BlockHeader.java
index 0eff53c4e11..151fc28c0d3 100644
--- a/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/BlockHeader.java
+++ b/ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/BlockHeader.java
@@ -252,7 +252,10 @@ public void writeTo(final RLPOutput out) {
   public static BlockHeader readFrom(
       final RLPInput input, final BlockHeaderFunctions blockHeaderFunctions) {
     final RLPInput headerRlp = input.readAsRlp();
-    headerRlp.enterList();
+    if (headerRlp.enterList() == 0) {
+      return null;
+    }
+
     final Hash parentHash = Hash.wrap(headerRlp.readBytes32());
     final Hash ommersHash = Hash.wrap(headerRlp.readBytes32());
     final Address coinbase = Address.readFrom(headerRlp);


Reviews:
user: login: garyschulte
body: This is a really tight implementation :+1:  One outstanding comment regarding the pivot block concern that IMO doesn't belong in era1 importer.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T02:31:54Z
user: login: Matilda-Clerke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T02:39:10Z
user: login: garyschulte
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T00:33:27Z
user: login: Matilda-Clerke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T00:44:25Z
user: login: jframe
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T03:12:07Z
user: login: Matilda-Clerke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T03:33:32Z
user: login: jframe
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T04:43:55Z
user: login: matkt
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T14:35:15Z
user: login: Matilda-Clerke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T04:38:02Z
user: login: Matilda-Clerke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T04:39:01Z
user: login: matkt
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T17:42:12Z
user: login: jframe
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T02:25:23Z

Review comments:
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: garyschulte
body: what is the point of the pivot block header file for an era importer?  shouldn't that be strictly a fast/snap sync concern, and far beyond the merge block (which would be the last block in an era1 file) ?

is this just a matter of ensuring we do not truncate block storage when doing a fast/snap sync?  if so - would it make more sense to address it in the fast sync code rather than adding a mock pivot block file here?
created_at: 2025-03-05T02:26:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: Matilda-Clerke
body: When I initially ran the first 100 era1 files and then ran besu, it noticed the pivot block header file was missing and attempted to full sync instead, but failed due to lack of trie logs for the imported blocks. Putting in the pivot block header file allows us to trick besu into doing trying to continue a fast/snap/checkpoint sync. This piece is probably the part I'm least confident in, so I'd be happy to hear of other ways to allow besu to sync after initially importing from era1 files.
created_at: 2025-03-05T02:39:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: garyschulte
body: I will raise with @matkt tomorrow morning at s&t.  I think signaling to snap and fast sync pipelines to behave rationally when we have blockchain data is a good idea.  I am just not sold on a mock pivot block file.  Perhaps the pipelines can check the worldstate storage instead of chain storage to make the determination about whether to continue or start a sync.
created_at: 2025-03-06T00:33:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: Matilda-Clerke
body: Ok, let me know what you come up with. It sounds like maybe modifying the sync pipelines to just be a bit more lenient and forgiving of a missing pivot block header will be a more elegant solution.
created_at: 2025-03-06T00:44:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: jframe
body: Agree with @garyschulte here; I don't think this should be part of this PR. After running the importer, you should be able to use full, snap or checkpoint or whatever you choose. The other importers will also likely be broken in the same way. I think the syncing pipelines should be able to check what existing data we have already imported into the database.
created_at: 2025-03-06T02:43:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
+
+    final ProtocolSchedule protocolSchedule = controller.getProtocolSchedule();
+    final BlockHeaderFunctions blockHeaderFunctions =
+        ScheduleBasedBlockHeaderFunctions.create(protocolSchedule);
+    final ProtocolContext context = controller.getProtocolContext();
+
+    Era1Reader reader = new Era1Reader(new SnappyFactory());
+
+    final List<Future<BlockHeader>> headersFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<BlockBody>> bodiesFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<List<TransactionReceipt>>> receiptsFutures =
+        new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    reader.read(
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: jframe
body: Will this be loading the entire file into memory? It's creating futures for all the headers, bodies and receipts, but it looks like each of those will have a reference to the underlying data in memory. Maybe the file isn't big enough to worry about this though haven't checked the size.
created_at: 2025-03-06T02:54:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
+
+    final ProtocolSchedule protocolSchedule = controller.getProtocolSchedule();
+    final BlockHeaderFunctions blockHeaderFunctions =
+        ScheduleBasedBlockHeaderFunctions.create(protocolSchedule);
+    final ProtocolContext context = controller.getProtocolContext();
+
+    Era1Reader reader = new Era1Reader(new SnappyFactory());
+
+    final List<Future<BlockHeader>> headersFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<BlockBody>> bodiesFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<List<TransactionReceipt>>> receiptsFutures =
+        new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    reader.read(
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: Matilda-Clerke
body: Maximum size for an era1 file is 8192 blocks. It runs fine locally, but may run out of memory on some lower spec machines, especially as the later blocks are more full. Do you think it's worth sacrificing performance to minimise memory usage?
created_at: 2025-03-06T03:33:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
+
+    final ProtocolSchedule protocolSchedule = controller.getProtocolSchedule();
+    final BlockHeaderFunctions blockHeaderFunctions =
+        ScheduleBasedBlockHeaderFunctions.create(protocolSchedule);
+    final ProtocolContext context = controller.getProtocolContext();
+
+    Era1Reader reader = new Era1Reader(new SnappyFactory());
+
+    final List<Future<BlockHeader>> headersFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<BlockBody>> bodiesFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<List<TransactionReceipt>>> receiptsFutures =
+        new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    reader.read(
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: jframe
body: Looking at files here https://era1.ethportal.net/ max I see is about 500MB, not sure what compression ratio we get with snappy but at most probably only a few GB of memory. Probably ok since you won't be running a full Besu while doing the import.
created_at: 2025-03-06T04:43:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: matkt
body: so we are talking about this check 

```java
else if (fastSyncState.getPivotBlockHeader().isEmpty()
        && protocolContext.getBlockchain().getChainHeadBlockNumber()
            != BlockHeader.GENESIS_BLOCK_NUMBER) {
      LOG.info(
          "Checkpoint sync was requested, but cannot be enabled because the local blockchain is not empty.");
      return Optional.empty();
    }
```

this check in my opinion was done to avoid that the node restarts with a fastsync or snapsync when we restart. this is the only reason I see for this check. before there was a reason because we were saving state from top to bottom and we did not really have a way to know if the state was synced or not. now that we save the root node of the trie at the end. I think we can replace this check with rather a check of the existence of the root node.

If it exists we return empty and if it does not exist we launch the sync

created_at: 2025-03-06T14:35:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: Matilda-Clerke
body: Sorry, this part of the work is a little above my head. Where should I be looking for this root node? At first I thought maybe you meant `protocolContext.getWorldStateArchive().getWorldState().rootHash()`, but this seems to be populated already after importing the era1 files.
created_at: 2025-03-07T04:38:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
+
+    final ProtocolSchedule protocolSchedule = controller.getProtocolSchedule();
+    final BlockHeaderFunctions blockHeaderFunctions =
+        ScheduleBasedBlockHeaderFunctions.create(protocolSchedule);
+    final ProtocolContext context = controller.getProtocolContext();
+
+    Era1Reader reader = new Era1Reader(new SnappyFactory());
+
+    final List<Future<BlockHeader>> headersFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<BlockBody>> bodiesFutures = new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    final List<Future<List<TransactionReceipt>>> receiptsFutures =
+        new ArrayList<>(ERA1_BLOCK_COUNT_MAX);
+    reader.read(
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: Matilda-Clerke
body: Great. It's currently making really good use of the CPU, so hopefully the performance is good.
created_at: 2025-03-07T04:39:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,194 @@
+/*
+ * Copyright contributors to Besu.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations under the License.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+package org.hyperledger.besu.chainimport;
+
+import org.hyperledger.besu.controller.BesuController;
+import org.hyperledger.besu.ethereum.ProtocolContext;
+import org.hyperledger.besu.ethereum.core.Block;
+import org.hyperledger.besu.ethereum.core.BlockBody;
+import org.hyperledger.besu.ethereum.core.BlockHeader;
+import org.hyperledger.besu.ethereum.core.BlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.core.BlockImporter;
+import org.hyperledger.besu.ethereum.core.TransactionReceipt;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncState;
+import org.hyperledger.besu.ethereum.eth.sync.fastsync.FastSyncStateStorage;
+import org.hyperledger.besu.ethereum.mainnet.BlockImportResult;
+import org.hyperledger.besu.ethereum.mainnet.BodyValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;
+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;
+import org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions;
+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;
+import org.hyperledger.besu.ethereum.rlp.RLPInput;
+import org.hyperledger.besu.util.era1.Era1BlockIndex;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockBody;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockHeader;
+import org.hyperledger.besu.util.era1.Era1ExecutionBlockReceipts;
+import org.hyperledger.besu.util.era1.Era1Reader;
+import org.hyperledger.besu.util.era1.Era1ReaderListener;
+import org.hyperledger.besu.util.snappy.SnappyFactory;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import org.apache.tuweni.bytes.Bytes;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Tool for importing era1-encoded block data, headers, and transaction receipts from era1 files.
+ */
+public class Era1BlockImporter implements Closeable {
+  private static final Logger LOG = LoggerFactory.getLogger(Era1BlockImporter.class);
+
+  private static final String FASTSYNC_DIRECTORY = "fastsync";
+  private static final String PIVOT_BLOCK_HEADER_FILE_NAME = "pivotBlockHeader.rlp";
+  private static final int ERA1_BLOCK_COUNT_MAX = 8192;
+  private static final int IMPORT_COUNT_FOR_LOG_UPDATE = 1000;
+
+  /** Default Constructor. */
+  public Era1BlockImporter() {}
+
+  /**
+   * Imports the blocks, headers, and transaction receipts from the file found at the supplied path
+   *
+   * @param controller The BesuController
+   * @param path The path
+   * @throws IOException IOException
+   * @throws ExecutionException ExecutionException
+   * @throws InterruptedException InterruptedException
+   * @throws TimeoutException TimeoutException
+   */
+  public void importBlocks(final BesuController controller, final Path path)
+      throws IOException, ExecutionException, InterruptedException, TimeoutException {
+    File fastSyncDir = controller.getDataDirectory().resolve(FASTSYNC_DIRECTORY).toFile();
+    if (!fastSyncDir.exists()) {
+      LOG.info("Creating " + fastSyncDir);
+      if (!fastSyncDir.mkdir()) {
+        throw new RuntimeException("Unable to create " + fastSyncDir + " directory");
+      }
+      File pivotBlockHeaderFile =
+          fastSyncDir.toPath().resolve(PIVOT_BLOCK_HEADER_FILE_NAME).toFile();
+      LOG.info("Creating " + pivotBlockHeaderFile);
+      if (!pivotBlockHeaderFile.createNewFile()) {
+        throw new RuntimeException("Unable to create " + pivotBlockHeaderFile);
+      }
+    }
path: besu/src/main/java/org/hyperledger/besu/chainimport/Era1BlockImporter.java
user: login: matkt
body: Populated ? This part just download the blockchain, right? It should return the hash of an empty tree. https://github.com/hyperledger/besu/blob/f92780859d7c3a31929b370cf6fc23e70f8ffb47/datatypes/src/main/java/org/hyperledger/besu/datatypes/Hash.java#L40

I think it can be a good solution. Bonsai will be better because if the state is empty it should return Hash.ZERO while Forest will return the hash of an empty tree. In both cases I think it will tell if the state has been sync
created_at: 2025-03-07T17:42:12Z
author_association: CONTRIBUTOR

PR comments:
user: login: Matilda-Clerke
created_at: 2025-03-06T04:27:19Z
body: I've removed the code dealing with the pivot block header file, as I anticipate it won't be included in the final code. Easy enough to add back if desired.
author_association: CONTRIBUTOR
user: login: matkt
created_at: 2025-03-07T17:31:46Z
body: Indeed I think it is better to do this in another pr
author_association: CONTRIBUTOR
user: login: Matilda-Clerke
created_at: 2025-03-10T22:24:11Z
body: Ok, now seeking approval to merge this PR as-is, with additional changes to sync pipelines coming in a new PR
author_association: CONTRIBUTOR