Repository: lambdaclass/ethrex
Pull request #2159: feat(l1): import subcommand
Author: login: LeanSerra
Labels: name: cli
description: Command line interface
name: L1
description: Ethereum client
Created: 2025-03-05T20:10:56Z | Merged: 2025-03-07T14:52:05Z

**Motivation**

We want to have a command that imports chain files/ block files to the db but does not start the full node

**Description**

- Move sub commands to module `subcommands`
- Add import.rs that creates a blockchain object and reads files from a path and stores the blocks in the db

**Some considerations**
- With this approach, we are no longer able to load a chain.rlp file while using an in memory db as the process dies after adding the blocks
- I had to leave the `--import` flag for the main command as its used by hive to set up the blockchain for tests removing it would make the ci fail. To remove `--import` we would need to first merge this PR, create a PR in our hive fork that changes the `ethrex.sh` script to use the new `import` sub command, then create a final PR removing `--import` from `ethrex.rs`

[PR](https://github.com/lambdaclass/hive/pull/21) in the hive repo to use the new subcommand, should be merged after this one

**How to test it**
Copy the changes from the PR above then run any hive test that uses the import subcommand like
```Shell
make run-hive SIMULATION=devp2p TEST_PATTERN="snap/client launch"
```
If you just want to test importing some blocks this command should be enough 

```Shell
sudo cargo run --bin ethrex -- --evm levm --network <network> import <rlp file> --removedb
```

Closes #2122



Linked issue #2122: Replace `--import` option with `import` subcommand.
Sometimes you want to import blocks but not start the client. This is the approach other clients also have.

The subcommand should also accept a `--datadir` option. Possible also other options from the main command.

To modularize the code, maybe we should have each command in it's own file, under `subcommands` folder (open to better suggestions though).

Acceptance criteria:
- Add `import` subcommand
- Remove `--import` option
- Update related modules (hive?, ethereum-package?).
Linked issue comments: user: login: LeanSerra
created_at: 2025-03-07T16:17:43Z
body: This should be finally closed with #2177 
author_association: CONTRIBUTOR

Diff:
diff --git a/cmd/ethrex/cli.rs b/cmd/ethrex/cli.rs
index cc3c4747a88..4cc67e33856 100644
--- a/cmd/ethrex/cli.rs
+++ b/cmd/ethrex/cli.rs
@@ -146,6 +146,23 @@ pub fn cli() -> Command {
                     .value_name("DATABASE_DIRECTORY")
                     .action(ArgAction::Set),
             ),
+        )
+        .subcommand(
+            Command::new("import")
+                .about("Import blocks to the database") 
+                .arg(
+                    Arg::new("path")
+                        .required(true)
+                        .value_name("FILE_PATH/FOLDER")
+                        .help("Path to a RLP chain file or a folder containing files with individual Blocks")
+                        .action(ArgAction::Set),
+                )
+                .arg(
+                    Arg::new("removedb")
+                        .long("removedb")
+                        .required(false)
+                        .action(clap::ArgAction::SetTrue)
+                )
         );
 
     cfg_if::cfg_if! {
diff --git a/cmd/ethrex/ethrex.rs b/cmd/ethrex/ethrex.rs
index 5da214c2f7a..9144d0c6925 100644
--- a/cmd/ethrex/ethrex.rs
+++ b/cmd/ethrex/ethrex.rs
@@ -6,17 +6,16 @@ use crate::{
     utils::{set_datadir, store_known_peers},
 };
 use ethrex_p2p::network::peer_table;
-use std::{
-    path::{Path, PathBuf},
-    time::Duration,
-};
+use std::{path::PathBuf, time::Duration};
+use subcommands::{import, removedb};
 use tokio_util::task::TaskTracker;
-use tracing::{info, warn};
+use tracing::info;
 
 mod cli;
 mod decode;
 mod initializers;
 mod networks;
+mod subcommands;
 mod utils;
 
 pub const DEFAULT_DATADIR: &str = "ethrex";
@@ -29,23 +28,30 @@ async fn main() {
         .get_one::<String>("datadir")
         .map_or(set_datadir(DEFAULT_DATADIR), |datadir| set_datadir(datadir));
 
+    init_tracing(&matches);
+
     if matches.subcommand_matches("removedb").is_some() {
-        let path = Path::new(&data_dir);
-        if path.exists() {
-            std::fs::remove_dir_all(path).expect("Failed to remove data directory");
-        } else {
-            warn!("Data directory does not exist: {}", data_dir);
-        }
+        removedb::remove_db(&data_dir);
         return;
     }
 
-    init_tracing(&matches);
+    let evm_engine = matches
+        .get_one::<String>("evm")
+        .unwrap_or(&"revm".to_string())
+        .clone()
+        .try_into()
+        .unwrap_or_else(|e| panic!("{}", e));
 
     let network = get_network(&matches);
 
+    if let Some(subcommand_matches) = matches.subcommand_matches("import") {
+        import::import_blocks_from_path(subcommand_matches, data_dir, evm_engine, &network);
+        return;
+    }
+
     let store = init_store(&data_dir, &network);
 
-    let blockchain = init_blockchain(&matches, store.clone());
+    let blockchain = init_blockchain(&matches, evm_engine, store.clone());
 
     let signer = get_signer(&data_dir);
 
diff --git a/cmd/ethrex/initializers.rs b/cmd/ethrex/initializers.rs
index 62a3c79142e..008355b0ef5 100644
--- a/cmd/ethrex/initializers.rs
+++ b/cmd/ethrex/initializers.rs
@@ -87,22 +87,19 @@ pub fn init_store(data_dir: &str, network: &str) -> Store {
     store
 }
 
-pub fn init_blockchain(matches: &ArgMatches, store: Store) -> Arc<Blockchain> {
-    let evm_engine: EvmEngine = matches
-        .get_one::<String>("evm")
-        .unwrap_or(&"revm".to_string())
-        .clone()
-        .try_into()
-        .unwrap_or_else(|e| panic!("{}", e));
-
-    let blockchain = Blockchain::new(evm_engine, store.clone());
+pub fn init_blockchain(
+    matches: &ArgMatches,
+    evm_engine: EvmEngine,
+    store: Store,
+) -> Arc<Blockchain> {
+    let blockchain = Blockchain::new(evm_engine, store);
 
     if let Some(chain_rlp_path) = matches.get_one::<String>("import") {
         info!("Importing blocks from chain file: {}", chain_rlp_path);
         let blocks = read_chain_file(chain_rlp_path);
         blockchain.import_blocks(&blocks);
     }
-
+    //TODO: remove --import --import_dir when we update hive fork
     if let Some(blocks_path) = matches.get_one::<String>("import_dir") {
         info!(
             "Importing blocks from individual block files in directory: {}",
diff --git a/cmd/ethrex/subcommands/import.rs b/cmd/ethrex/subcommands/import.rs
new file mode 100644
index 00000000000..3ead978b430
--- /dev/null
+++ b/cmd/ethrex/subcommands/import.rs
@@ -0,0 +1,50 @@
+use std::fs::{self, metadata};
+
+use clap::ArgMatches;
+use ethrex_blockchain::Blockchain;
+
+use ethrex_vm::backends::EvmEngine;
+use tracing::info;
+
+use crate::{initializers::init_store, utils};
+
+use super::removedb;
+
+pub fn import_blocks_from_path(
+    matches: &ArgMatches,
+    data_dir: String,
+    evm: EvmEngine,
+    network: &str,
+) {
+    let remove_db = *matches.get_one::<bool>("removedb").unwrap_or(&false);
+    let path = matches
+        .get_one::<String>("path")
+        .expect("No path provided to import blocks");
+    if remove_db {
+        removedb::remove_db(&data_dir);
+    }
+
+    let store = init_store(&data_dir, network);
+
+    // Todo use initializers::init_blockchain when we remove --import from it
+    let blockchain = Blockchain::new(evm, store.clone());
+
+    let path_metadata = metadata(path).expect("Failed to read path");
+    let blocks = if path_metadata.is_dir() {
+        let mut blocks = vec![];
+        let dir_reader = fs::read_dir(path).expect("Failed to read blocks directory");
+        for file_res in dir_reader {
+            let file = file_res.expect("Failed to open file in directory");
+            let path = file.path();
+            let s = path
+                .to_str()
+                .expect("Path could not be converted into string");
+            blocks.push(utils::read_block_file(s));
+        }
+        blocks
+    } else {
+        info!("Importing blocks from chain file: {}", path);
+        utils::read_chain_file(path)
+    };
+    blockchain.import_blocks(&blocks);
+}
diff --git a/cmd/ethrex/subcommands/mod.rs b/cmd/ethrex/subcommands/mod.rs
new file mode 100644
index 00000000000..b26897a3fb4
--- /dev/null
+++ b/cmd/ethrex/subcommands/mod.rs
@@ -0,0 +1,2 @@
+pub mod import;
+pub mod removedb;
diff --git a/cmd/ethrex/subcommands/removedb.rs b/cmd/ethrex/subcommands/removedb.rs
new file mode 100644
index 00000000000..145af7812d0
--- /dev/null
+++ b/cmd/ethrex/subcommands/removedb.rs
@@ -0,0 +1,13 @@
+use std::path::Path;
+
+use tracing::{info, warn};
+
+pub fn remove_db(data_dir: &str) {
+    let path = Path::new(data_dir);
+    if path.exists() {
+        std::fs::remove_dir_all(path).expect("Failed to remove data directory");
+        info!("Succesfully removed database at {}", data_dir);
+    } else {
+        warn!("Data directory does not exist: {}", data_dir);
+    }
+}


Reviews:
user: login: mpaulucci
body: Let's wait for https://github.com/lambdaclass/ethrex/pull/2064
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T16:03:09Z
user: login: fkrause98
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T16:23:35Z
user: login: mpaulucci
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T16:33:04Z
user: login: LeanSerra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:03:52Z
user: login: LeanSerra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:19:44Z
user: login: LeanSerra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:40:32Z
user: login: Oppen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T21:12:52Z
user: login: Oppen
body: LGTM but may be preferred to just normalize before use instead of having more modes of operation. The support can be removed later, so non-blocking.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T21:14:13Z
user: login: LeanSerra
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T21:21:54Z
user: login: mpaulucci
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-07T14:39:05Z

Review comments:
diff_hunk: @@ -0,0 +1,90 @@
+use std::fs::{self, metadata};
+
+use crate::{genesis_file_path_from_network, read_genesis_file, set_datadir, DEFAULT_DATADIR};
+use clap::ArgMatches;
+use ethrex_blockchain::Blockchain;
+use ethrex_common::types::Block;
+use ethrex_rlp::decode::RLPDecode;
+use ethrex_storage::{EngineType, Store};
+use ethrex_vm::backends::EVM;
+use tracing::info;
+
+use crate::decode;
+
+use super::removedb;
+
+pub fn import_blocks_from_path(matches: &ArgMatches) {
+    let remove_db = *matches.get_one::<bool>("removedb").unwrap_or(&false);
+    let path = matches
+        .get_one::<String>("path")
+        .expect("No path provided to import blocks");
+    let data_dir = matches
+        .get_one::<String>("datadir")
+        .map_or(set_datadir(DEFAULT_DATADIR), |datadir| set_datadir(datadir));
+    let evm = matches.get_one::<EVM>("evm").unwrap_or(&EVM::REVM);
+    let mut network = matches
+        .get_one::<String>("network")
+        .expect("network is required")
+        .clone();
+
+    if remove_db {
+        removedb::remove_db_file(&data_dir);
+    }
+
+    let store = {
path: cmd/ethrex/subcommands/import.rs
user: login: mpaulucci
body: let's figure a way to avoid this copy&paste
created_at: 2025-03-06T16:33:03Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,90 @@
+use std::fs::{self, metadata};
+
+use crate::{genesis_file_path_from_network, read_genesis_file, set_datadir, DEFAULT_DATADIR};
+use clap::ArgMatches;
+use ethrex_blockchain::Blockchain;
+use ethrex_common::types::Block;
+use ethrex_rlp::decode::RLPDecode;
+use ethrex_storage::{EngineType, Store};
+use ethrex_vm::backends::EVM;
+use tracing::info;
+
+use crate::decode;
+
+use super::removedb;
+
+pub fn import_blocks_from_path(matches: &ArgMatches) {
+    let remove_db = *matches.get_one::<bool>("removedb").unwrap_or(&false);
+    let path = matches
+        .get_one::<String>("path")
+        .expect("No path provided to import blocks");
+    let data_dir = matches
+        .get_one::<String>("datadir")
+        .map_or(set_datadir(DEFAULT_DATADIR), |datadir| set_datadir(datadir));
+    let evm = matches.get_one::<EVM>("evm").unwrap_or(&EVM::REVM);
+    let mut network = matches
+        .get_one::<String>("network")
+        .expect("network is required")
+        .clone();
+
+    if remove_db {
+        removedb::remove_db_file(&data_dir);
+    }
+
+    let store = {
path: cmd/ethrex/subcommands/import.rs
user: login: LeanSerra
body: The best think I can come up with is removing the repeated flags that already exist in the main command from the subcommand then send the flags to the subcommand from the main command.
This changes a little the syntax of the command from
```Shell
./ethrex import --datadir <Path> --network <genesis-path> --evm <evm> --removedb <path-to-rlp-file>
```

```Shell
./ethrex --datadir <Path> --network <genesis-path> --evm <evm> import --removedb <path-to-rlp-file>
```

As we are using the flags from the main command the import needs to be used after all the flags for the main command.
What do you think? I implemented this already in this [commit](https://github.com/lambdaclass/ethrex/pull/2159/commits/ef18e31a6c7efa4efc997ef3b72072922466adc9) but it can be rolled back if we want to go for other option
created_at: 2025-03-06T18:03:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+use std::fs::{self, metadata};
+
+use crate::{genesis_file_path_from_network, read_genesis_file, set_datadir, DEFAULT_DATADIR};
+use clap::ArgMatches;
+use ethrex_blockchain::Blockchain;
+use ethrex_common::types::Block;
+use ethrex_rlp::decode::RLPDecode;
+use ethrex_storage::{EngineType, Store};
+use ethrex_vm::backends::EVM;
+use tracing::info;
+
+use crate::decode;
+
+use super::removedb;
+
+pub fn import_blocks_from_path(matches: &ArgMatches) {
+    let remove_db = *matches.get_one::<bool>("removedb").unwrap_or(&false);
+    let path = matches
+        .get_one::<String>("path")
+        .expect("No path provided to import blocks");
+    let data_dir = matches
+        .get_one::<String>("datadir")
+        .map_or(set_datadir(DEFAULT_DATADIR), |datadir| set_datadir(datadir));
+    let evm = matches.get_one::<EVM>("evm").unwrap_or(&EVM::REVM);
+    let mut network = matches
+        .get_one::<String>("network")
+        .expect("network is required")
+        .clone();
+
+    if remove_db {
+        removedb::remove_db_file(&data_dir);
+    }
+
+    let store = {
path: cmd/ethrex/subcommands/import.rs
user: login: LeanSerra
body: The changes in main affect this PR a lot so you can disregard the comment above. I'm working on a better solution while merging to main
created_at: 2025-03-06T18:19:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+use std::fs::{self, metadata};
+
+use crate::{genesis_file_path_from_network, read_genesis_file, set_datadir, DEFAULT_DATADIR};
+use clap::ArgMatches;
+use ethrex_blockchain::Blockchain;
+use ethrex_common::types::Block;
+use ethrex_rlp::decode::RLPDecode;
+use ethrex_storage::{EngineType, Store};
+use ethrex_vm::backends::EVM;
+use tracing::info;
+
+use crate::decode;
+
+use super::removedb;
+
+pub fn import_blocks_from_path(matches: &ArgMatches) {
+    let remove_db = *matches.get_one::<bool>("removedb").unwrap_or(&false);
+    let path = matches
+        .get_one::<String>("path")
+        .expect("No path provided to import blocks");
+    let data_dir = matches
+        .get_one::<String>("datadir")
+        .map_or(set_datadir(DEFAULT_DATADIR), |datadir| set_datadir(datadir));
+    let evm = matches.get_one::<EVM>("evm").unwrap_or(&EVM::REVM);
+    let mut network = matches
+        .get_one::<String>("network")
+        .expect("network is required")
+        .clone();
+
+    if remove_db {
+        removedb::remove_db_file(&data_dir);
+    }
+
+    let store = {
path: cmd/ethrex/subcommands/import.rs
user: login: LeanSerra
body: Well I think now it should be better with this [commit](https://github.com/lambdaclass/ethrex/pull/2159/commits/16bf70c1439ef8eaa3d44eb27bddb56fea66f74d)
created_at: 2025-03-06T18:40:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,45 @@
+use std::fs::{self, metadata};
+
+use clap::ArgMatches;
+use ethrex_blockchain::Blockchain;
+
+use ethrex_vm::backends::EVM;
+use tracing::info;
+
+use crate::{initializers::init_store, utils};
+
+use super::removedb;
+
+pub fn import_blocks_from_path(matches: &ArgMatches, data_dir: String, evm: &EVM, network: &str) {
+    let remove_db = *matches.get_one::<bool>("removedb").unwrap_or(&false);
+    let path = matches
+        .get_one::<String>("path")
+        .expect("No path provided to import blocks");
+    if remove_db {
+        removedb::remove_db(&data_dir);
+    }
+
+    let store = init_store(&data_dir, network);
+
+    // Todo use initializers::init_blockchain when we remove --import from it
+    let blockchain = Blockchain::new(evm.clone(), store.clone());
+
+    let path_metadata = metadata(path).expect("Failed to read path");
+    let blocks = if path_metadata.is_dir() {
+        let mut blocks = vec![];
+        let dir_reader = fs::read_dir(path).expect("Failed to read blocks directory");
+        for file_res in dir_reader {
+            let file = file_res.expect("Failed to open file in directory");
+            let path = file.path();
+            let s = path
+                .to_str()
+                .expect("Path could not be converted into string");
+            blocks.push(utils::read_block_file(s));
+        }
+        blocks
+    } else {
+        info!("Importing blocks from chain file: {}", path);
+        utils::read_chain_file(path)
+    };
path: cmd/ethrex/subcommands/import.rs
user: login: Oppen
body: What's the advantage of having two modes here? Can't we normalize to a single chainfile?
created_at: 2025-03-06T21:12:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,45 @@
+use std::fs::{self, metadata};
+
+use clap::ArgMatches;
+use ethrex_blockchain::Blockchain;
+
+use ethrex_vm::backends::EVM;
+use tracing::info;
+
+use crate::{initializers::init_store, utils};
+
+use super::removedb;
+
+pub fn import_blocks_from_path(matches: &ArgMatches, data_dir: String, evm: &EVM, network: &str) {
+    let remove_db = *matches.get_one::<bool>("removedb").unwrap_or(&false);
+    let path = matches
+        .get_one::<String>("path")
+        .expect("No path provided to import blocks");
+    if remove_db {
+        removedb::remove_db(&data_dir);
+    }
+
+    let store = init_store(&data_dir, network);
+
+    // Todo use initializers::init_blockchain when we remove --import from it
+    let blockchain = Blockchain::new(evm.clone(), store.clone());
+
+    let path_metadata = metadata(path).expect("Failed to read path");
+    let blocks = if path_metadata.is_dir() {
+        let mut blocks = vec![];
+        let dir_reader = fs::read_dir(path).expect("Failed to read blocks directory");
+        for file_res in dir_reader {
+            let file = file_res.expect("Failed to open file in directory");
+            let path = file.path();
+            let s = path
+                .to_str()
+                .expect("Path could not be converted into string");
+            blocks.push(utils::read_block_file(s));
+        }
+        blocks
+    } else {
+        info!("Importing blocks from chain file: {}", path);
+        utils::read_chain_file(path)
+    };
path: cmd/ethrex/subcommands/import.rs
user: login: LeanSerra
body: Importing blocks from a directory is being used by hive tests I don't think we can modify those files
created_at: 2025-03-06T21:21:54Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-05T20:11:58Z
body: <h2>Lines of code report</h2>

Total lines added: `75`
Total lines removed: `2`
Total lines changed: `77`

<details>
<summary>Detailed view</summary>

```
+-------------------------------------------+-------+------+
| File                                      | Lines | Diff |
+-------------------------------------------+-------+------+
| ethrex/cmd/ethrex/cli.rs                  | 200   | +17  |
+-------------------------------------------+-------+------+
| ethrex/cmd/ethrex/ethrex.rs               | 98    | +4   |
+-------------------------------------------+-------+------+
| ethrex/cmd/ethrex/initializers.rs         | 376   | -2   |
+-------------------------------------------+-------+------+
| ethrex/cmd/ethrex/subcommands/import.rs   | 41    | +41  |
+-------------------------------------------+-------+------+
| ethrex/cmd/ethrex/subcommands/mod.rs      | 2     | +2   |
+-------------------------------------------+-------+------+
| ethrex/cmd/ethrex/subcommands/removedb.rs | 11    | +11  |
+-------------------------------------------+-------+------+

```
</details>

author_association: NONE
user: login: jrchatruc
created_at: 2025-03-06T15:07:11Z
body: It would be nice to add an additional argument to this subcommand that removes the database before starting the whole import process (you can check the `removedb` command to see how to wipe the db). This is important to us for benchmarking (see #2120), where we want to run the import command multiple times from scratch.
author_association: COLLABORATOR
user: login: LeanSerra
created_at: 2025-03-06T15:24:31Z
body: > It would be nice to add an additional argument to this subcommand that removes the database before starting the whole import process (you can check the `removedb` command to see how to wipe the db). This is important to us for benchmarking (see #2120), where we want to run the import command multiple times from scratch.

Done [Here](https://github.com/lambdaclass/ethrex/pull/2159/commits/c8c3f12bcaaa9e3b7a49e2f59ad329b2d991e056) and [Here](https://github.com/lambdaclass/ethrex/pull/2159/commits/be7371f9ed040d1e8ed357f3b7305b8fb959f261)
author_association: CONTRIBUTOR
user: login: mpaulucci
created_at: 2025-03-06T16:06:09Z
body: > It would be nice to add an additional argument to this subcommand that removes the database before starting the whole import process (you can check the `removedb` command to see how to wipe the db). This is important to us for benchmarking (see #2120), where we want to run the import command multiple times from scratch.

Can't you execute `removedb` in a different command? Maybe an alternative would be to add a `--force` flag instead that says that if there is a chain already then overwrite it
author_association: COLLABORATOR
user: login: jrchatruc
created_at: 2025-03-06T16:35:41Z
body: > > It would be nice to add an additional argument to this subcommand that removes the database before starting the whole import process (you can check the `removedb` command to see how to wipe the db). This is important to us for benchmarking (see #2120), where we want to run the import command multiple times from scratch.
> 
> Can't you execute `removedb` in a different command? 

Not quite because we run benches with hyperfine on the CI, which just receives a binary and runs it a bunch of time as warmup and then as regular runs, so I don't have a way to call a binary in between each run to remove the database.

A `--force` flag would work fine, it would be functionally equivalent to the `removedb` one that just got pushed
author_association: COLLABORATOR