Repository: op-rs/kona
Pull request #1667: fix(node/engine): Round Robin Task Execution
Author: login: refcell
Labels: name: K-fix
description: Kind: fix
name: A-node
description: Area: cl node (eq. Go op-node) handles single-chain consensus
Created: 2025-05-06T21:55:34Z | Merged: 2025-05-07T19:04:13Z

### Description

Allows tasks to be processed in a round-robin fashion based on their type.

Since consolidation tasks may need to wait for data in the derivation pipeline, we still want the engine to execute unsafe payload tasks in the meantime.

This round-robin type-based policy preserves strict ordering of the different task types, while allowing progress to be made on different tasks.


Diff:
diff --git a/crates/node/engine/src/attributes.rs b/crates/node/engine/src/attributes.rs
index 1a023a7ed7..74b1fd02f7 100644
--- a/crates/node/engine/src/attributes.rs
+++ b/crates/node/engine/src/attributes.rs
@@ -127,7 +127,7 @@ impl AttributesMatch {
                 "Checking attributes transaction against block transaction",
             );
             // Let's try to deserialize the attributes transaction
-            let Ok(attr_tx) = OpTxEnvelope::decode_2718(&mut &attr_tx_bytes[..]) else {
+            let Ok(mut attr_tx) = OpTxEnvelope::decode_2718(&mut &attr_tx_bytes[..]) else {
                 error!(
                     "Impossible to deserialize transaction from attributes. If we have stored these attributes it means the transactions where well formatted. This is a bug"
                 );
@@ -135,8 +135,29 @@ impl AttributesMatch {
                 return AttributesMismatch::MalformedAttributesTransaction.into();
             };
 
-            if &attr_tx != block_tx.inner.inner.inner() {
-                // Transaction mismatch!
+            // Unfortunate case where we need to touch the inside of the deposit transaction.
+            //
+            // Since `None` is functionally equivalent to `Some(0)` in revm, we need to make sure
+            // that the attributes produced by the derivation pipeline are consistent with the block
+            // received over the engine api. All we need to do here is set the mint value to be
+            // `Some(0)` in the case where it is `None` and the block is `Some(0)`.
+            //
+            // Notice, we *cannot* set the attributes `mint` to `Some(0)` if it already is `Some` in
+            // case there is truly a mismatch.
+            let block_tx = block_tx.inner.inner.inner();
+            let block_none =
+                if let OpTxEnvelope::Deposit(inner) = &block_tx { inner.mint } else { None };
+            if let OpTxEnvelope::Deposit(inner) = &attr_tx {
+                if block_none == Some(0) {
+                    trace!(target: "engine", "Found `mint` mismatch. Setting attributes `mint` to `Some(0)`");
+                    let mut inner = inner.clone().into_inner();
+                    inner.mint = Some(0);
+                    attr_tx = OpTxEnvelope::Deposit(alloy_primitives::Sealed::new(inner));
+                }
+            }
+
+            if &attr_tx != block_tx {
+                debug!(target: "engine", ?attr_tx, ?block_tx, "Transaction mismatch");
                 return AttributesMismatch::TransactionContent(attr_tx.tx_hash(), block_tx.tx_hash())
                     .into()
             }
diff --git a/crates/node/engine/src/lib.rs b/crates/node/engine/src/lib.rs
index a7704f3292..40a4096fab 100644
--- a/crates/node/engine/src/lib.rs
+++ b/crates/node/engine/src/lib.rs
@@ -12,8 +12,8 @@ extern crate tracing;
 mod task_queue;
 pub use task_queue::{
     BuildTask, BuildTaskError, ConsolidateTask, ConsolidateTaskError, Engine, EngineTask,
-    EngineTaskError, EngineTaskExt, ForkchoiceTask, ForkchoiceTaskError, InsertUnsafeTask,
-    InsertUnsafeTaskError,
+    EngineTaskError, EngineTaskExt, EngineTaskType, ForkchoiceTask, ForkchoiceTaskError,
+    InsertUnsafeTask, InsertUnsafeTaskError,
 };
 
 mod attributes;
diff --git a/crates/node/engine/src/task_queue/core.rs b/crates/node/engine/src/task_queue/core.rs
index 0b849b7839..af7e7761bf 100644
--- a/crates/node/engine/src/task_queue/core.rs
+++ b/crates/node/engine/src/task_queue/core.rs
@@ -1,8 +1,8 @@
 //! The [Engine] is a task queue that receives and executes [EngineTask]s.
 
 use super::{EngineTaskError, EngineTaskExt};
-use crate::{EngineState, EngineTask};
-use std::collections::VecDeque;
+use crate::{EngineState, EngineTask, EngineTaskType};
+use std::collections::{HashMap, VecDeque};
 
 /// The [Engine] task queue.
 ///
@@ -18,7 +18,9 @@ pub struct Engine {
     /// The state of the engine.
     state: EngineState,
     /// The task queue.
-    tasks: VecDeque<EngineTask>,
+    tasks: HashMap<EngineTaskType, VecDeque<EngineTask>>,
+    /// The current task being executed.
+    cursor: EngineTaskType,
 }
 
 impl Engine {
@@ -26,13 +28,17 @@ impl Engine {
     ///
     /// An initial [EngineTask::ForkchoiceUpdate] is added to the task queue to synchronize the
     /// engine with the forkchoice state of the [EngineState].
-    pub const fn new(initial_state: EngineState) -> Self {
-        Self { state: initial_state, tasks: VecDeque::new() }
+    pub fn new(initial_state: EngineState) -> Self {
+        Self {
+            state: initial_state,
+            tasks: HashMap::new(),
+            cursor: EngineTaskType::ForkchoiceUpdate,
+        }
     }
 
     /// Enqueues a new [EngineTask] for execution.
     pub fn enqueue(&mut self, task: EngineTask) {
-        self.tasks.push_back(task);
+        self.tasks.entry(task.ty()).or_default().push_back(task);
     }
 
     /// Clears the task queue.
@@ -40,26 +46,47 @@ impl Engine {
         self.tasks.clear();
     }
 
+    /// Returns the next task type to be executed.
+    pub fn next(&self) -> EngineTaskType {
+        let mut ty = self.cursor;
+        let task_len = self.tasks.len();
+        for _ in 0..task_len {
+            if !self.tasks.contains_key(&ty) {
+                ty = ty.next();
+            } else {
+                break;
+            }
+        }
+        ty
+    }
+
     /// Attempts to drain the queue by executing all [EngineTask]s in-order. If any task returns an
     /// error along the way, it is not popped from the queue (in case it must be retried) and
     /// the error is returned.
     ///
     /// If an [EngineTaskError::Reset] is encountered, the remaining tasks in the queue are cleared.
     pub async fn drain(&mut self) -> Result<(), EngineTaskError> {
-        while let Some(task) = self.tasks.front() {
+        loop {
+            let ty = self.next();
+            self.cursor = self.cursor.next();
+            let Some(task) = self.tasks.get(&ty) else {
+                return Ok(());
+            };
+            let Some(task) = task.front() else {
+                return Ok(());
+            };
             match task.execute(&mut self.state).await {
-                Ok(_) => {
-                    // Dequeue the task if it was successful.
-                    self.tasks.pop_front();
-                }
+                Ok(_) => {}
                 Err(EngineTaskError::Reset(e)) => {
                     self.clear();
                     return Err(EngineTaskError::Reset(e));
                 }
                 e => return e,
             }
+            let ty = task.ty();
+            if let Some(queue) = self.tasks.get_mut(&ty) {
+                queue.pop_front();
+            };
         }
-
-        Ok(())
     }
 }
diff --git a/crates/node/engine/src/task_queue/tasks/consolidate/task.rs b/crates/node/engine/src/task_queue/tasks/consolidate/task.rs
index c6597aeaa3..e62a81e92b 100644
--- a/crates/node/engine/src/task_queue/tasks/consolidate/task.rs
+++ b/crates/node/engine/src/task_queue/tasks/consolidate/task.rs
@@ -77,10 +77,22 @@ impl ConsolidateTask {
         // If this is successful, the forkchoice change synchronizes.
         // Otherwise, the attributes need to be processed.
         if crate::AttributesMatch::check(&self.cfg, &self.attributes, &block).is_match() {
+            debug!(
+                target: "engine",
+                attributes = ?self.attributes,
+                block_hash = %block.header.hash,
+                "Consolidating engine state",
+            );
             return self.execute_forkchoice_task(state).await;
         }
 
         // Otherwise, the attributes need to be processed.
+        debug!(
+            target: "engine",
+            attributes = ?self.attributes,
+            block_hash = %block.header.hash,
+            "No consolidation needed executing build task",
+        );
         self.execute_build_task(state).await
     }
 }
diff --git a/crates/node/engine/src/task_queue/tasks/mod.rs b/crates/node/engine/src/task_queue/tasks/mod.rs
index d2daa3d29c..19d7a1a929 100644
--- a/crates/node/engine/src/task_queue/tasks/mod.rs
+++ b/crates/node/engine/src/task_queue/tasks/mod.rs
@@ -1,7 +1,7 @@
 //! Tasks to update the engine state.
 
 mod task;
-pub use task::{EngineTask, EngineTaskError, EngineTaskExt};
+pub use task::{EngineTask, EngineTaskError, EngineTaskExt, EngineTaskType};
 
 mod forkchoice;
 pub use forkchoice::{ForkchoiceTask, ForkchoiceTaskError};
diff --git a/crates/node/engine/src/task_queue/tasks/task.rs b/crates/node/engine/src/task_queue/tasks/task.rs
index 44d5f738c0..b251702141 100644
--- a/crates/node/engine/src/task_queue/tasks/task.rs
+++ b/crates/node/engine/src/task_queue/tasks/task.rs
@@ -7,6 +7,34 @@ use crate::EngineState;
 use async_trait::async_trait;
 use thiserror::Error;
 
+/// The type of engine task.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]
+#[repr(u8)]
+pub enum EngineTaskType {
+    /// Perform a `engine_forkchoiceUpdated` call with the current [EngineState]'s forkchoice,
+    /// and no payload attributes.
+    ForkchoiceUpdate,
+    /// Inserts an unsafe payload into the execution engine.
+    InsertUnsafe,
+    /// Builds a new block with the given attributes, and inserts it into the execution engine.
+    BuildBlock,
+    /// Performs consolidation on the engine state, reverting to payload attribute processing
+    /// via the [`BuildTask`] if consolidation fails.
+    Consolidate,
+}
+
+impl EngineTaskType {
+    /// Returns the next task type in a round-robin fashion.
+    pub const fn next(&self) -> Self {
+        match self {
+            Self::ForkchoiceUpdate => Self::InsertUnsafe,
+            Self::InsertUnsafe => Self::BuildBlock,
+            Self::BuildBlock => Self::Consolidate,
+            Self::Consolidate => Self::ForkchoiceUpdate,
+        }
+    }
+}
+
 /// Tasks that may be inserted into and executed by the [Engine].
 ///
 /// [Engine]: crate::Engine
@@ -34,6 +62,16 @@ impl EngineTask {
             Self::Consolidate(task) => task.execute(state).await,
         }
     }
+
+    /// Returns the [`EngineTaskType`] for the [`EngineTask`].
+    pub const fn ty(&self) -> EngineTaskType {
+        match &self {
+            Self::ForkchoiceUpdate(_) => EngineTaskType::ForkchoiceUpdate,
+            Self::InsertUnsafe(_) => EngineTaskType::InsertUnsafe,
+            Self::BuildBlock(_) => EngineTaskType::BuildBlock,
+            Self::Consolidate(_) => EngineTaskType::Consolidate,
+        }
+    }
 }
 
 #[async_trait]
diff --git a/crates/node/service/src/actors/engine.rs b/crates/node/service/src/actors/engine.rs
index a2f803af94..03522c69d1 100644
--- a/crates/node/service/src/actors/engine.rs
+++ b/crates/node/service/src/actors/engine.rs
@@ -91,7 +91,7 @@ impl EngineActor {
                     return;
                 }
                 // If the sync status is `None`, begin derivation.
-                info!(target: "engine", "Engine finished syncing, starting derivation.");
+                trace!(target: "engine", "Sending signal to start derivation");
                 channel.send(()).ok();
             }
         });
@@ -175,6 +175,7 @@ impl NodeActor for EngineActor {
                         self.cancellation.cancel();
                         return Err(EngineError::ChannelClosed);
                     };
+                    let hash = envelope.payload_hash;
                     let task = InsertUnsafeTask::new(
                         Arc::clone(&self.client),
                         Arc::clone(&self.sync),
@@ -183,7 +184,7 @@ impl NodeActor for EngineActor {
                     );
                     let task = EngineTask::InsertUnsafe(task);
                     self.engine.enqueue(task);
-                    debug!(target: "engine", "Enqueued unsafe block task.");
+                    debug!(target: "engine", ?hash, "Enqueued unsafe block task.");
                     self.check_sync();
                 }
                 Some(config) = self.runtime_config_rx.recv() => {
diff --git a/crates/protocol/protocol/src/deposits.rs b/crates/protocol/protocol/src/deposits.rs
index 0863f0dcc2..737d54d741 100644
--- a/crates/protocol/protocol/src/deposits.rs
+++ b/crates/protocol/protocol/src/deposits.rs
@@ -203,7 +203,11 @@ pub(crate) fn unmarshal_deposit_version0(
 
     // 0 mint is represented as nil to skip minting code
     if mint == 0 {
-        tx.mint = None;
+        // Since `None` is functionally equivalent to `Some(0)` in revm, we need to make sure
+        // that the attributes produced by the derivation pipeline are consistent with the block
+        // received over the engine api. Instead of setting `mint` to `None`, we set it to
+        // `Some(0)` to ensure that the produced attributes are consistent.
+        tx.mint = Some(0);
     } else {
         tx.mint = Some(mint);
     }


Reviews:
user: login: theochap
body: Looks great, would be nice to have a couple of tests for that
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T22:08:43Z
user: login: refcell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T13:50:00Z
user: login: refcell
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T13:51:06Z
user: login: theochap
body: Looks great
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-07T13:53:36Z

Review comments:
diff_hunk: @@ -34,6 +62,16 @@ impl EngineTask {
             Self::Consolidate(task) => task.execute(state).await,
         }
     }
+
+    /// Returns the [`EngineTaskType`] for the [`EngineTask`].
+    pub const fn ty(&self) -> EngineTaskType {
+        match &self {
+            Self::ForkchoiceUpdate(_) => EngineTaskType::ForkchoiceUpdate,
+            Self::InsertUnsafe(_) => EngineTaskType::InsertUnsafe,
+            Self::BuildBlock(_) => EngineTaskType::BuildBlock,
+            Self::Consolidate(_) => EngineTaskType::Consolidate,
+        }
path: crates/node/engine/src/task_queue/tasks/task.rs
user: login: theochap
body: Nit: to automatically build that type and all the associated impls we can use `strum`. Can also understand if we want to avoid adding external dependencies if not absolutely required
created_at: 2025-05-06T22:05:34Z
author_association: MEMBER
diff_hunk: @@ -46,20 +52,39 @@ impl Engine {
     ///
     /// If an [EngineTaskError::Reset] is encountered, the remaining tasks in the queue are cleared.
     pub async fn drain(&mut self) -> Result<(), EngineTaskError> {
-        while let Some(task) = self.tasks.front() {
-            match task.execute(&mut self.state).await {
-                Ok(_) => {
-                    // Dequeue the task if it was successful.
-                    self.tasks.pop_front();
+        loop {
+            let mut ty = self.cursor;
+            let task_len = self.tasks.len();
+            for _ in 0..task_len {
+                if !self.tasks.contains_key(&ty) {
+                    ty = ty.next();
+                } else {
+                    break;
                 }
+            }
+            let Some(task) = self.tasks.get(&ty) else {
+                // No more tasks to process.
+                return Ok(());
+            };
+            let Some(task) = task.front() else {
+                // No more tasks to_process..
+                continue;
+            };
+            match task.execute(&mut self.state).await {
+                Ok(_) => {}
                 Err(EngineTaskError::Reset(e)) => {
                     self.clear();
                     return Err(EngineTaskError::Reset(e));
                 }
                 e => return e,
             }
+            let ty = task.ty();
+            if let Some(queue) = self.tasks.get_mut(&ty) {
+                queue.pop_front();
+                if queue.is_empty() {
+                    self.tasks.remove(&ty);
+                }
+            };
         }
path: crates/node/engine/src/task_queue/core.rs
user: login: theochap
body: Is there an easy way to test that specific piece of code in isolation? I think this will help make sure this behaves properly and we do get the round robin behavior
created_at: 2025-05-06T22:08:22Z
author_association: MEMBER
diff_hunk: @@ -34,6 +62,16 @@ impl EngineTask {
             Self::Consolidate(task) => task.execute(state).await,
         }
     }
+
+    /// Returns the [`EngineTaskType`] for the [`EngineTask`].
+    pub const fn ty(&self) -> EngineTaskType {
+        match &self {
+            Self::ForkchoiceUpdate(_) => EngineTaskType::ForkchoiceUpdate,
+            Self::InsertUnsafe(_) => EngineTaskType::InsertUnsafe,
+            Self::BuildBlock(_) => EngineTaskType::BuildBlock,
+            Self::Consolidate(_) => EngineTaskType::Consolidate,
+        }
path: crates/node/engine/src/task_queue/tasks/task.rs
user: login: refcell
body: Yea, let's leave the optimization for future PRs. Important to just get this working nicely
created_at: 2025-05-07T13:50:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -46,20 +52,39 @@ impl Engine {
     ///
     /// If an [EngineTaskError::Reset] is encountered, the remaining tasks in the queue are cleared.
     pub async fn drain(&mut self) -> Result<(), EngineTaskError> {
-        while let Some(task) = self.tasks.front() {
-            match task.execute(&mut self.state).await {
-                Ok(_) => {
-                    // Dequeue the task if it was successful.
-                    self.tasks.pop_front();
+        loop {
+            let mut ty = self.cursor;
+            let task_len = self.tasks.len();
+            for _ in 0..task_len {
+                if !self.tasks.contains_key(&ty) {
+                    ty = ty.next();
+                } else {
+                    break;
                 }
+            }
+            let Some(task) = self.tasks.get(&ty) else {
+                // No more tasks to process.
+                return Ok(());
+            };
+            let Some(task) = task.front() else {
+                // No more tasks to_process..
+                continue;
+            };
+            match task.execute(&mut self.state).await {
+                Ok(_) => {}
                 Err(EngineTaskError::Reset(e)) => {
                     self.clear();
                     return Err(EngineTaskError::Reset(e));
                 }
                 e => return e,
             }
+            let ty = task.ty();
+            if let Some(queue) = self.tasks.get_mut(&ty) {
+                queue.pop_front();
+                if queue.is_empty() {
+                    self.tasks.remove(&ty);
+                }
+            };
         }
path: crates/node/engine/src/task_queue/core.rs
user: login: refcell
body: This is a good candidate for a refactor to encapsulate the queue behavior. For now, this is working when running live against `op-reth` so I'd suggest we ticket out a refactor for later.
created_at: 2025-05-07T13:51:06Z
author_association: CONTRIBUTOR

PR comments:
user: login: refcell
created_at: 2025-05-06T21:55:36Z
body: ## 📚 $\text{Stack Overview}$

Pulls submitted in this stack:
* #1667 👈
* #1666
* `main`

_This comment was automatically generated by [`st`](https://github.com/clabby/st)._
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-05-06T21:58:19Z
body: ## [Codecov](https://app.codecov.io/gh/op-rs/kona/pull/1667?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=op-rs) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 92.4%. Comparing base [(`3d8a2c1`)](https://app.codecov.io/gh/op-rs/kona/commit/3d8a2c1b5440a6bedd3a72ced52a73052ab89c73?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=op-rs) to head [(`e5e71b7`)](https://app.codecov.io/gh/op-rs/kona/commit/e5e71b7201eac09710ec6b1cae02e9c0a3fdbed9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=op-rs).
> Report is 2 commits behind head on main.

:white_check_mark: All tests successful. No failed tests found.


[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/op-rs/kona/pull/1667?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=op-rs).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=op-rs).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE