Repository: aptos-labs/aptos-core
Pull request #16621: [Experimental][Orderbook] Add API to decrease the size of the order without changing its priority
Author: login: sitalkedia
Created: 2025-05-14T18:28:44Z | Merged: 2025-05-20T01:14:47Z



## Description

Add API to decrease the size of the order without changing its priority

## How Has This Been Tested?
<!--
- Please ensure that the functionality introduced by this change is well tested and verified to work as expected.
- Ensure tests cover both happy and unhappy paths.
- List and link relevant tests.
-->

## Key Areas to Review
<!--
- Identify any critical parts of the code that require special attention or understanding. Explain why these parts are crucial to the functionality or architecture of the project.
- Point out any areas where complex logic has been implemented. Provide a brief explanation of the logic and your approach to make it easier for reviewers to follow.
- Highlight any areas where you are particularly concerned or unsure about the code's impact on the change. This can include potential performance or security issues, or compatibility with existing features.
-->

## Type of Change
- [ ] New feature
- [ ] Bug fix
- [ ] Breaking change
- [ ] Performance improvement
- [ ] Refactoring
- [ ] Dependency update
- [ ] Documentation update
- [ ] Tests

## Which Components or Systems Does This Change Impact?
- [ ] Validator Node
- [ ] Full Node (API, Indexer, etc.)
- [ ] Move/Aptos Virtual Machine
- [ ] Aptos Framework
- [ ] Aptos CLI/SDK
- [ ] Developer Infrastructure
- [ ] Move Compiler
- [ ] Other (specify)

## Checklist
- [ ] I have read and followed the [CONTRIBUTING](https://github.com/aptos-labs/aptos-core/blob/main/CONTRIBUTING.md) doc
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I identified and added all stakeholders and component owners affected by this change as reviewers
- [ ] I tested both happy and unhappy path of the functionality
- [ ] I have made corresponding changes to the documentation

<!-- Thank you for your contribution! -->



Diff:
diff --git a/aptos-move/framework/aptos-experimental/doc/active_order_book.md b/aptos-move/framework/aptos-experimental/doc/active_order_book.md
index 5047b25981e..aaafb50fa6f 100644
--- a/aptos-move/framework/aptos-experimental/doc/active_order_book.md
+++ b/aptos-move/framework/aptos-experimental/doc/active_order_book.md
@@ -25,7 +25,8 @@
 -  [Function `get_single_match_for_buy_order`](#0x7_active_order_book_get_single_match_for_buy_order)
 -  [Function `get_single_match_for_sell_order`](#0x7_active_order_book_get_single_match_for_sell_order)
 -  [Function `get_single_match_result`](#0x7_active_order_book_get_single_match_result)
--  [Function `increase_maker_order_size`](#0x7_active_order_book_increase_maker_order_size)
+-  [Function `increase_order_size`](#0x7_active_order_book_increase_order_size)
+-  [Function `decrease_order_size`](#0x7_active_order_book_decrease_order_size)
 -  [Function `place_maker_order`](#0x7_active_order_book_place_maker_order)
 
 
@@ -738,13 +739,14 @@ Check if the order is a taker order - i.e. if it can be immediately matched with
 
 </details>
 
-<a id="0x7_active_order_book_increase_maker_order_size"></a>
+<a id="0x7_active_order_book_increase_order_size"></a>
 
-## Function `increase_maker_order_size`
+## Function `increase_order_size`
 
+Increase the size of the order in the orderbook without altering its position in the price-time priority.
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="active_order_book.md#0x7_active_order_book_increase_maker_order_size">increase_maker_order_size</a>(self: &<b>mut</b> <a href="active_order_book.md#0x7_active_order_book_ActiveOrderBook">active_order_book::ActiveOrderBook</a>, price: u64, unique_priority_idx: <a href="order_book_types.md#0x7_order_book_types_UniqueIdxType">order_book_types::UniqueIdxType</a>, size: u64, is_buy: bool)
+<pre><code><b>public</b> <b>fun</b> <a href="active_order_book.md#0x7_active_order_book_increase_order_size">increase_order_size</a>(self: &<b>mut</b> <a href="active_order_book.md#0x7_active_order_book_ActiveOrderBook">active_order_book::ActiveOrderBook</a>, price: u64, unique_priority_idx: <a href="order_book_types.md#0x7_order_book_types_UniqueIdxType">order_book_types::UniqueIdxType</a>, size_delta: u64, is_buy: bool)
 </code></pre>
 
 
@@ -753,21 +755,56 @@ Check if the order is a taker order - i.e. if it can be immediately matched with
 <summary>Implementation</summary>
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="active_order_book.md#0x7_active_order_book_increase_maker_order_size">increase_maker_order_size</a>(
+<pre><code><b>public</b> <b>fun</b> <a href="active_order_book.md#0x7_active_order_book_increase_order_size">increase_order_size</a>(
     self: &<b>mut</b> <a href="active_order_book.md#0x7_active_order_book_ActiveOrderBook">ActiveOrderBook</a>,
     price: u64,
     unique_priority_idx: UniqueIdxType,
-    size: u64,
+    size_delta: u64,
+    is_buy: bool
+) {
+    <b>let</b> tie_breaker = <a href="active_order_book.md#0x7_active_order_book_get_tie_breaker">get_tie_breaker</a>(unique_priority_idx, is_buy);
+    <b>let</b> key = <a href="active_order_book.md#0x7_active_order_book_ActiveBidKey">ActiveBidKey</a> { price, tie_breaker };
+    <b>if</b> (is_buy) {
+        self.buys.borrow_mut(&key).size += size_delta;
+    } <b>else</b> {
+        self.sells.borrow_mut(&key).size += size_delta;
+    };
+}
+</code></pre>
+
+
+
+</details>
+
+<a id="0x7_active_order_book_decrease_order_size"></a>
+
+## Function `decrease_order_size`
+
+Decrease the size of the order in the order book without altering its position in the price-time priority.
+
+
+<pre><code><b>public</b> <b>fun</b> <a href="active_order_book.md#0x7_active_order_book_decrease_order_size">decrease_order_size</a>(self: &<b>mut</b> <a href="active_order_book.md#0x7_active_order_book_ActiveOrderBook">active_order_book::ActiveOrderBook</a>, price: u64, unique_priority_idx: <a href="order_book_types.md#0x7_order_book_types_UniqueIdxType">order_book_types::UniqueIdxType</a>, size_delta: u64, is_buy: bool)
+</code></pre>
+
+
+
+<details>
+<summary>Implementation</summary>
+
+
+<pre><code><b>public</b> <b>fun</b> <a href="active_order_book.md#0x7_active_order_book_decrease_order_size">decrease_order_size</a>(
+    self: &<b>mut</b> <a href="active_order_book.md#0x7_active_order_book_ActiveOrderBook">ActiveOrderBook</a>,
+    price: u64,
+    unique_priority_idx: UniqueIdxType,
+    size_delta: u64,
     is_buy: bool
 ) {
     <b>let</b> tie_breaker = <a href="active_order_book.md#0x7_active_order_book_get_tie_breaker">get_tie_breaker</a>(unique_priority_idx, is_buy);
     <b>let</b> key = <a href="active_order_book.md#0x7_active_order_book_ActiveBidKey">ActiveBidKey</a> { price, tie_breaker };
-    // Assert that this is not a taker order
-    <b>assert</b>!(!self.<a href="active_order_book.md#0x7_active_order_book_is_taker_order">is_taker_order</a>(price, is_buy), <a href="active_order_book.md#0x7_active_order_book_EINVALID_MAKER_ORDER">EINVALID_MAKER_ORDER</a>);
     <b>if</b> (is_buy) {
-        self.buys.borrow_mut(&key).size += size;
+        self.buys.borrow_mut(&key).size -= size_delta;
     } <b>else</b> {
-        self.sells.borrow_mut(&key).size += size;
+        self.sells.borrow_mut(&key).size -= size_delta;
     };
 }
 </code></pre>
diff --git a/aptos-move/framework/aptos-experimental/doc/market.md b/aptos-move/framework/aptos-experimental/doc/market.md
index 78ac088a2f5..6926576357f 100644
--- a/aptos-move/framework/aptos-experimental/doc/market.md
+++ b/aptos-move/framework/aptos-experimental/doc/market.md
@@ -64,9 +64,9 @@ TimeBased(time): The order is triggered when the current time is greater than or
 -  [Function `place_maker_order`](#0x7_market_place_maker_order)
 -  [Function `place_order_with_order_id`](#0x7_market_place_order_with_order_id)
 -  [Function `cancel_order`](#0x7_market_cancel_order)
+-  [Function `decrease_order_size`](#0x7_market_decrease_order_size)
 -  [Function `get_remaining_size`](#0x7_market_get_remaining_size)
 -  [Function `take_ready_price_based_orders`](#0x7_market_take_ready_price_based_orders)
--  [Function `trigger_price_based_orders`](#0x7_market_trigger_price_based_orders)
 -  [Function `take_ready_time_based_orders`](#0x7_market_take_ready_time_based_orders)
 
 
@@ -486,6 +486,15 @@ TimeBased(time): The order is triggered when the current time is greater than or
 
 
 
+<a id="0x7_market_ORDER_SIZE_REDUCED"></a>
+
+
+
+<pre><code><b>const</b> <a href="market.md#0x7_market_ORDER_SIZE_REDUCED">ORDER_SIZE_REDUCED</a>: u8 = 4;
+</code></pre>
+
+
+
 <a id="0x7_market_ORDER_STATUS_CANCELLED"></a>
 
 Order has been cancelled by the user or engine.
@@ -529,15 +538,6 @@ orders are invalid orders. Rejection reasons:
 
 
 
-<a id="0x7_market_SLIPPAGE_TOLERANCE_FOR_TWAP"></a>
-
-
-
-<pre><code><b>const</b> <a href="market.md#0x7_market_SLIPPAGE_TOLERANCE_FOR_TWAP">SLIPPAGE_TOLERANCE_FOR_TWAP</a>: u64 = 300;
-</code></pre>
-
-
-
 <a id="0x7_market_TIME_IN_FORCE_GTC"></a>
 
 Good till cancelled order type
@@ -1823,14 +1823,14 @@ Cancels an order - this will cancel the order and emit an event for the order ca
 
 </details>
 
-<a id="0x7_market_get_remaining_size"></a>
+<a id="0x7_market_decrease_order_size"></a>
 
-## Function `get_remaining_size`
+## Function `decrease_order_size`
 
-Remaining size of the order in the order book.
+Cancels an order - this will cancel the order and emit an event for the order cancellation.
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_get_remaining_size">get_remaining_size</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<a href="market.md#0x7_market_Market">market::Market</a>&lt;M&gt;, user: <b>address</b>, order_id: u64): u64
+<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_decrease_order_size">decrease_order_size</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<b>mut</b> <a href="market.md#0x7_market_Market">market::Market</a>&lt;M&gt;, user: &<a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/signer.md#0x1_signer">signer</a>, order_id: u64, size_delta: u64)
 </code></pre>
 
 
@@ -1839,10 +1839,41 @@ Remaining size of the order in the order book.
 <summary>Implementation</summary>
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_get_remaining_size">get_remaining_size</a>&lt;M: store + <b>copy</b> + drop&gt;(
-    self: &<a href="market.md#0x7_market_Market">Market</a>&lt;M&gt;, user: <b>address</b>, order_id: u64
-): u64 {
-    self.<a href="order_book.md#0x7_order_book">order_book</a>.<a href="market.md#0x7_market_get_remaining_size">get_remaining_size</a>(user, order_id)
+<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_decrease_order_size">decrease_order_size</a>&lt;M: store + <b>copy</b> + drop&gt;(
+    self: &<b>mut</b> <a href="market.md#0x7_market_Market">Market</a>&lt;M&gt;, user: &<a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/signer.md#0x1_signer">signer</a>, order_id: u64, size_delta: u64
+) {
+    <b>let</b> <a href="../../aptos-framework/doc/account.md#0x1_account">account</a> = <a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/signer.md#0x1_signer_address_of">signer::address_of</a>(user);
+    self.<a href="order_book.md#0x7_order_book">order_book</a>.<a href="market.md#0x7_market_decrease_order_size">decrease_order_size</a>(<a href="../../aptos-framework/doc/account.md#0x1_account">account</a>, order_id, size_delta);
+    <b>let</b> maybe_order = self.<a href="order_book.md#0x7_order_book">order_book</a>.get_order(<a href="../../aptos-framework/doc/account.md#0x1_account">account</a>, order_id);
+    <b>assert</b>!(maybe_order.is_some(), <a href="market.md#0x7_market_EORDER_DOES_NOT_EXIST">EORDER_DOES_NOT_EXIST</a>);
+    <b>let</b> (order, _) = maybe_order.destroy_some().destroy_order_from_state();
+    <b>let</b> (
+        order_id_type,
+        _unique_priority_idx,
+        price,
+        orig_size,
+        remaining_size,
+        is_buy,
+        _trigger_condition,
+        _metadata
+    ) = order.destroy_order();
+    <b>let</b> (user, order_id) = order_id_type.destroy_order_id_type();
+    <a href="../../aptos-framework/doc/event.md#0x1_event_emit">event::emit</a>(
+        <a href="market.md#0x7_market_OrderEvent">OrderEvent</a> {
+            parent: self.parent,
+            <a href="market.md#0x7_market">market</a>: self.<a href="market.md#0x7_market">market</a>,
+            order_id,
+            user,
+            orig_size,
+            remaining_size,
+            size_delta: remaining_size,
+            price,
+            is_buy,
+            is_taker: <b>false</b>,
+            status: <a href="market.md#0x7_market_ORDER_SIZE_REDUCED">ORDER_SIZE_REDUCED</a>,
+            details: std::string::utf8(b"Order size reduced")
+        }
+    )
 }
 </code></pre>
 
@@ -1850,15 +1881,14 @@ Remaining size of the order in the order book.
 
 </details>
 
-<a id="0x7_market_take_ready_price_based_orders"></a>
+<a id="0x7_market_get_remaining_size"></a>
 
-## Function `take_ready_price_based_orders`
+## Function `get_remaining_size`
 
-Returns all the pending order ready to be executed based on the oracle price. The caller is responsible to
-call the <code>place_order_with_order_id</code> API to place the order with the order id returned from this API.
+Remaining size of the order in the order book.
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_take_ready_price_based_orders">take_ready_price_based_orders</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<b>mut</b> <a href="market.md#0x7_market_Market">market::Market</a>&lt;M&gt;, oracle_price: u64): <a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;<a href="order_book_types.md#0x7_order_book_types_Order">order_book_types::Order</a>&lt;M&gt;&gt;
+<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_get_remaining_size">get_remaining_size</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<a href="market.md#0x7_market_Market">market::Market</a>&lt;M&gt;, user: <b>address</b>, order_id: u64): u64
 </code></pre>
 
 
@@ -1867,10 +1897,10 @@ call the <code>place_order_with_order_id</code> API to place the order with the
 <summary>Implementation</summary>
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_take_ready_price_based_orders">take_ready_price_based_orders</a>&lt;M: store + <b>copy</b> + drop&gt;(
-    self: &<b>mut</b> <a href="market.md#0x7_market_Market">Market</a>&lt;M&gt;, oracle_price: u64
-): <a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;Order&lt;M&gt;&gt; {
-    self.<a href="order_book.md#0x7_order_book">order_book</a>.<a href="market.md#0x7_market_take_ready_price_based_orders">take_ready_price_based_orders</a>(oracle_price)
+<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_get_remaining_size">get_remaining_size</a>&lt;M: store + <b>copy</b> + drop&gt;(
+    self: &<a href="market.md#0x7_market_Market">Market</a>&lt;M&gt;, user: <b>address</b>, order_id: u64
+): u64 {
+    self.<a href="order_book.md#0x7_order_book">order_book</a>.<a href="market.md#0x7_market_get_remaining_size">get_remaining_size</a>(user, order_id)
 }
 </code></pre>
 
@@ -1878,14 +1908,15 @@ call the <code>place_order_with_order_id</code> API to place the order with the
 
 </details>
 
-<a id="0x7_market_trigger_price_based_orders"></a>
+<a id="0x7_market_take_ready_price_based_orders"></a>
 
-## Function `trigger_price_based_orders`
+## Function `take_ready_price_based_orders`
 
-Triggers all the orders that are ready to be executed based on the oracle price.
+Returns all the pending order ready to be executed based on the oracle price. The caller is responsible to
+call the <code>place_order_with_order_id</code> API to place the order with the order id returned from this API.
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_trigger_price_based_orders">trigger_price_based_orders</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<b>mut</b> <a href="market.md#0x7_market_Market">market::Market</a>&lt;M&gt;, oracle_price: u64, callbacks: &<a href="market_types.md#0x7_market_types_MarketClearinghouseCallbacks">market_types::MarketClearinghouseCallbacks</a>&lt;M&gt;)
+<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_take_ready_price_based_orders">take_ready_price_based_orders</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<b>mut</b> <a href="market.md#0x7_market_Market">market::Market</a>&lt;M&gt;, oracle_price: u64): <a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;<a href="order_book_types.md#0x7_order_book_types_Order">order_book_types::Order</a>&lt;M&gt;&gt;
 </code></pre>
 
 
@@ -1894,36 +1925,10 @@ Triggers all the orders that are ready to be executed based on the oracle price.
 <summary>Implementation</summary>
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_trigger_price_based_orders">trigger_price_based_orders</a>&lt;M: store + <b>copy</b> + drop&gt;(
-    self: &<b>mut</b> <a href="market.md#0x7_market_Market">Market</a>&lt;M&gt;,
-    oracle_price: u64,
-    callbacks: &MarketClearinghouseCallbacks&lt;M&gt;
-) {
-    <b>let</b> ready_orders = self.<a href="order_book.md#0x7_order_book">order_book</a>.<a href="market.md#0x7_market_take_ready_price_based_orders">take_ready_price_based_orders</a>(oracle_price);
-    <b>let</b> i = 0;
-    <b>while</b> (i &lt; ready_orders.length()) {
-        <b>let</b> order = ready_orders[i];
-        <b>let</b> (order_id, _unique_priority_idx, price, orig_size, _, is_buy, _, metadata) =
-
-            order.destroy_order();
-        <b>let</b> (user_addr, order_id) = order_id.destroy_order_id_type();
-        self.<a href="market.md#0x7_market_place_order_with_order_id">place_order_with_order_id</a>(
-            user_addr,
-            price,
-            orig_size,
-            orig_size,
-            is_buy,
-            <a href="market.md#0x7_market_TIME_IN_FORCE_GTC">TIME_IN_FORCE_GTC</a>,
-            <a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/option.md#0x1_option_none">option::none</a>(),
-            metadata,
-            order_id,
-            1000, // TODO(skedia): Add support for fill limit here.
-            <b>false</b>,
-            <b>true</b>,
-            callbacks
-        );
-        i += 1;
-    };
+<pre><code><b>public</b> <b>fun</b> <a href="market.md#0x7_market_take_ready_price_based_orders">take_ready_price_based_orders</a>&lt;M: store + <b>copy</b> + drop&gt;(
+    self: &<b>mut</b> <a href="market.md#0x7_market_Market">Market</a>&lt;M&gt;, oracle_price: u64
+): <a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;Order&lt;M&gt;&gt; {
+    self.<a href="order_book.md#0x7_order_book">order_book</a>.<a href="market.md#0x7_market_take_ready_price_based_orders">take_ready_price_based_orders</a>(oracle_price)
 }
 </code></pre>
 
diff --git a/aptos-move/framework/aptos-experimental/doc/order_book.md b/aptos-move/framework/aptos-experimental/doc/order_book.md
index 9d6b1774ac9..a365bde16d0 100644
--- a/aptos-move/framework/aptos-experimental/doc/order_book.md
+++ b/aptos-move/framework/aptos-experimental/doc/order_book.md
@@ -29,6 +29,7 @@ types of pending orders are supported.
 -  [Function `reinsert_maker_order`](#0x7_order_book_reinsert_maker_order)
 -  [Function `place_pending_maker_order`](#0x7_order_book_place_pending_maker_order)
 -  [Function `get_single_match_for_taker`](#0x7_order_book_get_single_match_for_taker)
+-  [Function `decrease_order_size`](#0x7_order_book_decrease_order_size)
 -  [Function `is_active_order`](#0x7_order_book_is_active_order)
 -  [Function `get_order`](#0x7_order_book_get_order)
 -  [Function `get_remaining_size`](#0x7_order_book_get_remaining_size)
@@ -588,7 +589,7 @@ it is added to the order book, if it exists, it's size is updated.
         order_req.remaining_size
     );
     self.orders.add(order_id, order_with_state);
-    self.active_orders.increase_maker_order_size(
+    self.active_orders.increase_order_size(
         order_req.price,
         order_req.unique_priority_idx.destroy_some(),
         order_req.remaining_size,
@@ -692,6 +693,49 @@ API to ensure that the order is a taker order before calling this API, otherwise
 
 
 
+</details>
+
+<a id="0x7_order_book_decrease_order_size"></a>
+
+## Function `decrease_order_size`
+
+Decrease the size of the order by the given size delta. The API aborts if the order is not found in the order book or
+if the size delta is greater than or equal to the remaining size of the order. Please note that the API will abort and
+not cancel the order if the size delta is equal to the remaining size of the order, to avoid unintended
+cancellation of the order. Please use the <code>cancel_order</code> API to cancel the order.
+
+
+<pre><code><b>public</b> <b>fun</b> <a href="order_book.md#0x7_order_book_decrease_order_size">decrease_order_size</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<b>mut</b> <a href="order_book.md#0x7_order_book_OrderBook">order_book::OrderBook</a>&lt;M&gt;, <a href="../../aptos-framework/doc/account.md#0x1_account">account</a>: <b>address</b>, account_order_id: u64, size_delta: u64)
+</code></pre>
+
+
+
+<details>
+<summary>Implementation</summary>
+
+
+<pre><code><b>public</b> <b>fun</b> <a href="order_book.md#0x7_order_book_decrease_order_size">decrease_order_size</a>&lt;M: store + <b>copy</b> + drop&gt;(
+    self: &<b>mut</b> <a href="order_book.md#0x7_order_book_OrderBook">OrderBook</a>&lt;M&gt;, <a href="../../aptos-framework/doc/account.md#0x1_account">account</a>: <b>address</b>, account_order_id: u64, size_delta: u64
+) {
+    <b>let</b> order_id = new_order_id_type(<a href="../../aptos-framework/doc/account.md#0x1_account">account</a>, account_order_id);
+    <b>assert</b>!(self.orders.contains(&order_id), <a href="order_book.md#0x7_order_book_EORDER_NOT_FOUND">EORDER_NOT_FOUND</a>);
+    <b>let</b> order_with_state = self.orders.remove(&order_id);
+    order_with_state.decrease_remaining_size(size_delta);
+    <b>if</b> (order_with_state.<a href="order_book.md#0x7_order_book_is_active_order">is_active_order</a>()) {
+        <b>let</b> order = order_with_state.get_order_from_state();
+        self.active_orders.<a href="order_book.md#0x7_order_book_decrease_order_size">decrease_order_size</a>(
+            order.get_price(),
+            order_with_state.get_unique_priority_idx_from_state(),
+            size_delta,
+            order.is_buy()
+        );
+    };
+    self.orders.add(order_id, order_with_state);
+}
+</code></pre>
+
+
+
 </details>
 
 <a id="0x7_order_book_is_active_order"></a>
diff --git a/aptos-move/framework/aptos-experimental/doc/order_book_types.md b/aptos-move/framework/aptos-experimental/doc/order_book_types.md
index 80a97b79547..02426f64284 100644
--- a/aptos-move/framework/aptos-experimental/doc/order_book_types.md
+++ b/aptos-move/framework/aptos-experimental/doc/order_book_types.md
@@ -38,6 +38,7 @@
 -  [Function `get_metadata_from_order`](#0x7_order_book_types_get_metadata_from_order)
 -  [Function `get_trigger_condition_from_order`](#0x7_order_book_types_get_trigger_condition_from_order)
 -  [Function `increase_remaining_size`](#0x7_order_book_types_increase_remaining_size)
+-  [Function `decrease_remaining_size`](#0x7_order_book_types_decrease_remaining_size)
 -  [Function `set_remaining_size`](#0x7_order_book_types_set_remaining_size)
 -  [Function `get_remaining_size_from_state`](#0x7_order_book_types_get_remaining_size_from_state)
 -  [Function `get_unique_priority_idx_from_state`](#0x7_order_book_types_get_unique_priority_idx_from_state)
@@ -50,6 +51,7 @@
 -  [Function `destroy_order_id_type`](#0x7_order_book_types_destroy_order_id_type)
 -  [Function `is_active_order`](#0x7_order_book_types_is_active_order)
 -  [Function `get_price`](#0x7_order_book_types_get_price)
+-  [Function `is_buy`](#0x7_order_book_types_is_buy)
 
 
 <pre><code><b>use</b> <a href="../../aptos-framework/../aptos-stdlib/../move-stdlib/doc/bcs.md#0x1_bcs">0x1::bcs</a>;
@@ -410,6 +412,15 @@
 
 
 
+<a id="0x7_order_book_types_EINVALID_ORDER_SIZE_DECREASE"></a>
+
+
+
+<pre><code><b>const</b> <a href="order_book_types.md#0x7_order_book_types_EINVALID_ORDER_SIZE_DECREASE">EINVALID_ORDER_SIZE_DECREASE</a>: u64 = 4;
+</code></pre>
+
+
+
 <a id="0x7_order_book_types_EINVALID_TRIGGER_CONDITION"></a>
 
 
@@ -432,7 +443,7 @@
 
 
 
-<pre><code><b>const</b> <a href="order_book_types.md#0x7_order_book_types_INVALID_MATCH_RESULT">INVALID_MATCH_RESULT</a>: u64 = 1;
+<pre><code><b>const</b> <a href="order_book_types.md#0x7_order_book_types_INVALID_MATCH_RESULT">INVALID_MATCH_RESULT</a>: u64 = 3;
 </code></pre>
 
 
@@ -1082,6 +1093,33 @@
 
 
 
+</details>
+
+<a id="0x7_order_book_types_decrease_remaining_size"></a>
+
+## Function `decrease_remaining_size`
+
+
+
+<pre><code><b>public</b> <b>fun</b> <a href="order_book_types.md#0x7_order_book_types_decrease_remaining_size">decrease_remaining_size</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<b>mut</b> <a href="order_book_types.md#0x7_order_book_types_OrderWithState">order_book_types::OrderWithState</a>&lt;M&gt;, size: u64)
+</code></pre>
+
+
+
+<details>
+<summary>Implementation</summary>
+
+
+<pre><code><b>public</b> <b>fun</b> <a href="order_book_types.md#0x7_order_book_types_decrease_remaining_size">decrease_remaining_size</a>&lt;M: store + <b>copy</b> + drop&gt;(
+    self: &<b>mut</b> <a href="order_book_types.md#0x7_order_book_types_OrderWithState">OrderWithState</a>&lt;M&gt;, size: u64
+) {
+    <b>assert</b>!(self.order.remaining_size &gt; size, <a href="order_book_types.md#0x7_order_book_types_EINVALID_ORDER_SIZE_DECREASE">EINVALID_ORDER_SIZE_DECREASE</a>);
+    self.order.remaining_size -= size;
+}
+</code></pre>
+
+
+
 </details>
 
 <a id="0x7_order_book_types_set_remaining_size"></a>
@@ -1393,6 +1431,30 @@
 
 
 
+</details>
+
+<a id="0x7_order_book_types_is_buy"></a>
+
+## Function `is_buy`
+
+
+
+<pre><code><b>public</b> <b>fun</b> <a href="order_book_types.md#0x7_order_book_types_is_buy">is_buy</a>&lt;M: <b>copy</b>, drop, store&gt;(self: &<a href="order_book_types.md#0x7_order_book_types_Order">order_book_types::Order</a>&lt;M&gt;): bool
+</code></pre>
+
+
+
+<details>
+<summary>Implementation</summary>
+
+
+<pre><code><b>public</b> <b>fun</b> <a href="order_book_types.md#0x7_order_book_types_is_buy">is_buy</a>&lt;M: store + <b>copy</b> + drop&gt;(self: &<a href="order_book_types.md#0x7_order_book_types_Order">Order</a>&lt;M&gt;): bool {
+    self.is_buy
+}
+</code></pre>
+
+
+
 </details>
 
 
diff --git a/aptos-move/framework/aptos-experimental/sources/trading/market/market.move b/aptos-move/framework/aptos-experimental/sources/trading/market/market.move
index 2678b2450d8..99a324e6eec 100644
--- a/aptos-move/framework/aptos-experimental/sources/trading/market/market.move
+++ b/aptos-move/framework/aptos-experimental/sources/trading/market/market.move
@@ -70,9 +70,6 @@ module aptos_experimental::market {
         TIME_IN_FORCE_IOC
     }
 
-    // TODO(skedia): Revisit this slippage tolerance for twap
-    const SLIPPAGE_TOLERANCE_FOR_TWAP: u64 = 300; // 3%
-
     struct Market<M: store + copy + drop> has store {
         /// Address of the parent object that created this market
         /// Purely for grouping events based on the source DEX, not used otherwise
@@ -95,6 +92,7 @@ module aptos_experimental::market {
     /// 1. Insufficient margin
     /// 2. Order is reduce_only but does not reduce
     const ORDER_STATUS_REJECTED: u8 = 3;
+    const ORDER_SIZE_REDUCED: u8 = 4;
 
     public fun order_status_open(): u8 {
         ORDER_STATUS_OPEN
@@ -839,6 +837,44 @@ module aptos_experimental::market {
         }
     }
 
+    /// Cancels an order - this will cancel the order and emit an event for the order cancellation.
+    public fun decrease_order_size<M: store + copy + drop>(
+        self: &mut Market<M>, user: &signer, order_id: u64, size_delta: u64
+    ) {
+        let account = signer::address_of(user);
+        self.order_book.decrease_order_size(account, order_id, size_delta);
+        let maybe_order = self.order_book.get_order(account, order_id);
+        assert!(maybe_order.is_some(), EORDER_DOES_NOT_EXIST);
+        let (order, _) = maybe_order.destroy_some().destroy_order_from_state();
+        let (
+            order_id_type,
+            _unique_priority_idx,
+            price,
+            orig_size,
+            remaining_size,
+            is_buy,
+            _trigger_condition,
+            _metadata
+        ) = order.destroy_order();
+        let (user, order_id) = order_id_type.destroy_order_id_type();
+        event::emit(
+            OrderEvent {
+                parent: self.parent,
+                market: self.market,
+                order_id,
+                user,
+                orig_size,
+                remaining_size,
+                size_delta: remaining_size,
+                price,
+                is_buy,
+                is_taker: false,
+                status: ORDER_SIZE_REDUCED,
+                details: std::string::utf8(b"Order size reduced")
+            }
+        )
+    }
+
     /// Remaining size of the order in the order book.
     public fun get_remaining_size<M: store + copy + drop>(
         self: &Market<M>, user: address, order_id: u64
@@ -854,39 +890,6 @@ module aptos_experimental::market {
         self.order_book.take_ready_price_based_orders(oracle_price)
     }
 
-    /// Triggers all the orders that are ready to be executed based on the oracle price.
-    public fun trigger_price_based_orders<M: store + copy + drop>(
-        self: &mut Market<M>,
-        oracle_price: u64,
-        callbacks: &MarketClearinghouseCallbacks<M>
-    ) {
-        let ready_orders = self.order_book.take_ready_price_based_orders(oracle_price);
-        let i = 0;
-        while (i < ready_orders.length()) {
-            let order = ready_orders[i];
-            let (order_id, _unique_priority_idx, price, orig_size, _, is_buy, _, metadata) =
-
-                order.destroy_order();
-            let (user_addr, order_id) = order_id.destroy_order_id_type();
-            self.place_order_with_order_id(
-                user_addr,
-                price,
-                orig_size,
-                orig_size,
-                is_buy,
-                TIME_IN_FORCE_GTC,
-                option::none(),
-                metadata,
-                order_id,
-                1000, // TODO(skedia): Add support for fill limit here.
-                false,
-                true,
-                callbacks
-            );
-            i += 1;
-        };
-    }
-
     /// Returns all the pending order that are ready to be executed based on current time stamp. The caller is responsible to
     /// call the `place_order_with_order_id` API to place the order with the order id returned from this API.
     public fun take_ready_time_based_orders<M: store + copy + drop>(
diff --git a/aptos-move/framework/aptos-experimental/sources/trading/order_book/active_order_book.move b/aptos-move/framework/aptos-experimental/sources/trading/order_book/active_order_book.move
index f8b9b42264e..614adc4c1b9 100644
--- a/aptos-move/framework/aptos-experimental/sources/trading/order_book/active_order_book.move
+++ b/aptos-move/framework/aptos-experimental/sources/trading/order_book/active_order_book.move
@@ -301,21 +301,37 @@ module aptos_experimental::active_order_book {
         }
     }
 
-    public fun increase_maker_order_size(
+    /// Increase the size of the order in the orderbook without altering its position in the price-time priority.
+    public fun increase_order_size(
         self: &mut ActiveOrderBook,
         price: u64,
         unique_priority_idx: UniqueIdxType,
-        size: u64,
+        size_delta: u64,
+        is_buy: bool
+    ) {
+        let tie_breaker = get_tie_breaker(unique_priority_idx, is_buy);
+        let key = ActiveBidKey { price, tie_breaker };
+        if (is_buy) {
+            self.buys.borrow_mut(&key).size += size_delta;
+        } else {
+            self.sells.borrow_mut(&key).size += size_delta;
+        };
+    }
+
+    /// Decrease the size of the order in the order book without altering its position in the price-time priority.
+    public fun decrease_order_size(
+        self: &mut ActiveOrderBook,
+        price: u64,
+        unique_priority_idx: UniqueIdxType,
+        size_delta: u64,
         is_buy: bool
     ) {
         let tie_breaker = get_tie_breaker(unique_priority_idx, is_buy);
         let key = ActiveBidKey { price, tie_breaker };
-        // Assert that this is not a taker order
-        assert!(!self.is_taker_order(price, is_buy), EINVALID_MAKER_ORDER);
         if (is_buy) {
-            self.buys.borrow_mut(&key).size += size;
+            self.buys.borrow_mut(&key).size -= size_delta;
         } else {
-            self.sells.borrow_mut(&key).size += size;
+            self.sells.borrow_mut(&key).size -= size_delta;
         };
     }
 
diff --git a/aptos-move/framework/aptos-experimental/sources/trading/order_book/order_book.move b/aptos-move/framework/aptos-experimental/sources/trading/order_book/order_book.move
index a6b31bcb618..35811a19279 100644
--- a/aptos-move/framework/aptos-experimental/sources/trading/order_book/order_book.move
+++ b/aptos-move/framework/aptos-experimental/sources/trading/order_book/order_book.move
@@ -210,7 +210,7 @@ module aptos_experimental::order_book {
             order_req.remaining_size
         );
         self.orders.add(order_id, order_with_state);
-        self.active_orders.increase_maker_order_size(
+        self.active_orders.increase_order_size(
             order_req.price,
             order_req.unique_priority_idx.destroy_some(),
             order_req.remaining_size,
@@ -271,6 +271,29 @@ module aptos_experimental::order_book {
         new_single_order_match(order, matched_size)
     }
 
+    /// Decrease the size of the order by the given size delta. The API aborts if the order is not found in the order book or
+    /// if the size delta is greater than or equal to the remaining size of the order. Please note that the API will abort and
+    /// not cancel the order if the size delta is equal to the remaining size of the order, to avoid unintended
+    /// cancellation of the order. Please use the `cancel_order` API to cancel the order.
+    public fun decrease_order_size<M: store + copy + drop>(
+        self: &mut OrderBook<M>, account: address, account_order_id: u64, size_delta: u64
+    ) {
+        let order_id = new_order_id_type(account, account_order_id);
+        assert!(self.orders.contains(&order_id), EORDER_NOT_FOUND);
+        let order_with_state = self.orders.remove(&order_id);
+        order_with_state.decrease_remaining_size(size_delta);
+        if (order_with_state.is_active_order()) {
+            let order = order_with_state.get_order_from_state();
+            self.active_orders.decrease_order_size(
+                order.get_price(),
+                order_with_state.get_unique_priority_idx_from_state(),
+                size_delta,
+                order.is_buy()
+            );
+        };
+        self.orders.add(order_id, order_with_state);
+    }
+
     public fun is_active_order<M: store + copy + drop>(
         self: &OrderBook<M>, account: address, account_order_id: u64
     ): bool {
@@ -1253,4 +1276,48 @@ module aptos_experimental::order_book {
         order_book.destroy_order_book();
     }
 
+
+    #[test]
+    fun test_decrease_order_size() {
+        let order_book = new_order_book<TestMetadata>();
+
+        // Place an active order
+        let order_req = OrderRequest {
+            account: @0xAA,
+            account_order_id: 1,
+            unique_priority_idx: option::none(),
+            price: 100,
+            orig_size: 1000,
+            remaining_size: 1000,
+            is_buy: false,
+            trigger_condition: option::none(),
+            metadata: TestMetadata {}
+        };
+        order_book.place_maker_order(order_req);
+        assert!(order_book.get_remaining_size(@0xAA, 1) ==  1000);
+
+        order_book.decrease_order_size(@0xAA, 1, 700);
+        // Verify order was decreased with updated size
+        assert!(order_book.get_remaining_size(@0xAA, 1) == 300);
+
+        let order_req = OrderRequest {
+            account: @0xBB,
+            account_order_id: 1,
+            unique_priority_idx: option::none(),
+            price: 100,
+            orig_size: 1000,
+            remaining_size: 1000,
+            is_buy: false,
+            trigger_condition: option::some(tp_trigger_condition(90)),
+            metadata: TestMetadata {}
+        };
+        order_book.place_maker_order(order_req);
+        assert!(order_book.get_remaining_size(@0xBB, 1) == 1000);
+        order_book.decrease_order_size(@0xBB, 1, 600);
+        // Verify order was decreased with updated size
+        assert!(order_book.get_remaining_size(@0xBB, 1) == 400);
+
+        order_book.destroy_order_book();
+    }
+
 }
diff --git a/aptos-move/framework/aptos-experimental/sources/trading/order_book/order_book_types.move b/aptos-move/framework/aptos-experimental/sources/trading/order_book/order_book_types.move
index 55263a00e5b..38f2f59c083 100644
--- a/aptos-move/framework/aptos-experimental/sources/trading/order_book/order_book_types.move
+++ b/aptos-move/framework/aptos-experimental/sources/trading/order_book/order_book_types.move
@@ -18,7 +18,8 @@ module aptos_experimental::order_book_types {
 
     const EORDER_ALREADY_EXISTS: u64 = 1;
     const EINVALID_TRIGGER_CONDITION: u64 = 2;
-    const INVALID_MATCH_RESULT: u64 = 1;
+    const INVALID_MATCH_RESULT: u64 = 3;
+    const EINVALID_ORDER_SIZE_DECREASE: u64 = 4;
 
     const SLIPPAGE_PCT_PRECISION: u64 = 100; // 100 = 1%
 
@@ -225,6 +226,13 @@ module aptos_experimental::order_book_types {
         self.order.remaining_size += size;
     }
 
+    public fun decrease_remaining_size<M: store + copy + drop>(
+        self: &mut OrderWithState<M>, size: u64
+    ) {
+        assert!(self.order.remaining_size > size, EINVALID_ORDER_SIZE_DECREASE);
+        self.order.remaining_size -= size;
+    }
+
     public fun set_remaining_size<M: store + copy + drop>(
         self: &mut OrderWithState<M>, remaining_size: u64
     ) {
@@ -295,4 +303,8 @@ module aptos_experimental::order_book_types {
     public fun get_price<M: store + copy + drop>(self: &Order<M>): u64 {
         self.price
     }
+
+    public fun is_buy<M: store + copy + drop>(self: &Order<M>): bool {
+        self.is_buy
+    }
 }


Reviews:
user: login: igor-aptos
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T19:17:56Z
user: login: runtian-zhou
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T19:21:27Z
user: login: runtian-zhou
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T19:22:09Z
user: login: sitalkedia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T19:46:28Z
user: login: runtian-zhou
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T19:48:10Z

Review comments:
diff_hunk: @@ -844,6 +841,15 @@ module aptos_experimental::market {
         }
     }
 
+    /// Cancels an order - this will cancel the order and emit an event for the order cancellation.
+    public fun decrease_order_size<M: store + copy + drop>(
+        self: &mut Market<M>, user: &signer, order_id: u64, size_delta: u64
+    ) {
+        let account = signer::address_of(user);
+        self.order_book.decrease_order_size(account, order_id, size_delta);
+        // TODO(skedia) emit event for order size decrease
path: aptos-move/framework/aptos-experimental/sources/trading/market/market.move
user: login: igor-aptos
body: can you add the event in this PR?
created_at: 2025-05-16T19:09:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -844,6 +841,15 @@ module aptos_experimental::market {
         }
     }
 
+    /// Cancels an order - this will cancel the order and emit an event for the order cancellation.
+    public fun decrease_order_size<M: store + copy + drop>(
+        self: &mut Market<M>, user: &signer, order_id: u64, size_delta: u64
+    ) {
+        let account = signer::address_of(user);
+        self.order_book.decrease_order_size(account, order_id, size_delta);
+        // TODO(skedia) emit event for order size decrease
path: aptos-move/framework/aptos-experimental/sources/trading/market/market.move
user: login: runtian-zhou
body: Yea just add the event in this PR plz
created_at: 2025-05-19T19:22:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -844,6 +841,15 @@ module aptos_experimental::market {
         }
     }
 
+    /// Cancels an order - this will cancel the order and emit an event for the order cancellation.
+    public fun decrease_order_size<M: store + copy + drop>(
+        self: &mut Market<M>, user: &signer, order_id: u64, size_delta: u64
+    ) {
+        let account = signer::address_of(user);
+        self.order_book.decrease_order_size(account, order_id, size_delta);
+        // TODO(skedia) emit event for order size decrease
path: aptos-move/framework/aptos-experimental/sources/trading/market/market.move
user: login: sitalkedia
body: Added. 
created_at: 2025-05-19T19:46:28Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-14T19:06:39Z
body: ### Forge is running suite `compat` on `46ff583f7c392d9a34cc445f7b279c2d7930a374` ==> `e0169e4f344aba880d7bc0c2955e76cc20fd8bdd`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16621&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15028674577)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-14T19:06:41Z
body: ### Forge is running suite `framework_upgrade` on `46ff583f7c392d9a34cc445f7b279c2d7930a374` ==> `e0169e4f344aba880d7bc0c2955e76cc20fd8bdd`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16621&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15028674577)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-14T19:06:46Z
body: ### Forge is running suite `realistic_env_max_load` on `e0169e4f344aba880d7bc0c2955e76cc20fd8bdd`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16621&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15028674577)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-14T19:17:16Z
body: ### :white_check_mark: Forge suite `compat` success on `46ff583f7c392d9a34cc445f7b279c2d7930a374` ==> `e0169e4f344aba880d7bc0c2955e76cc20fd8bdd`
```
Compatibility test results for 46ff583f7c392d9a34cc445f7b279c2d7930a374 ==> e0169e4f344aba880d7bc0c2955e76cc20fd8bdd (PR)
1. Check liveness of validators at old version: 46ff583f7c392d9a34cc445f7b279c2d7930a374
compatibility::simple-validator-upgrade::liveness-check : committed: 8609.00 txn/s, latency: 3613.90 ms, (p50: 3400 ms, p70: 3800, p90: 5100 ms, p99: 11900 ms), latency samples: 284220
2. Upgrading first Validator to new version: e0169e4f344aba880d7bc0c2955e76cc20fd8bdd
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 2778.29 txn/s, latency: 11985.33 ms, (p50: 13600 ms, p70: 13800, p90: 14200 ms, p99: 14200 ms), latency samples: 104080
3. Upgrading rest of first batch to new version: e0169e4f344aba880d7bc0c2955e76cc20fd8bdd
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 2794.71 txn/s, latency: 11921.52 ms, (p50: 13500 ms, p70: 13800, p90: 14100 ms, p99: 14100 ms), latency samples: 105760
4. upgrading second batch to new version: e0169e4f344aba880d7bc0c2955e76cc20fd8bdd
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2411.59 txn/s, latency: 11713.95 ms, (p50: 11100 ms, p70: 14700, p90: 18300 ms, p99: 21700 ms), latency samples: 108100
5. check swarm health
Compatibility test for 46ff583f7c392d9a34cc445f7b279c2d7930a374 ==> e0169e4f344aba880d7bc0c2955e76cc20fd8bdd passed
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16621&var-metrics_source=All&var-chain_name=forge-1&from=1747249606000&to=1747250234000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1747249606000&end=1747250234000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-14T19%3A06%3A46.163019%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-14T19%3A17%3A14.907292%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747249606&until=1747250234&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747249606&until=1747250234&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15028674577)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-14T19:19:48Z
body: ### :white_check_mark: Forge suite `realistic_env_max_load` success on `e0169e4f344aba880d7bc0c2955e76cc20fd8bdd`
```
two traffics test: inner traffic : committed: 11284.00 txn/s, latency: 3191.89 ms, (p50: 2700 ms, p70: 3000, p90: 4700 ms, p99: 6800 ms), latency samples: 4290420
two traffics test : committed: 99.98 txn/s, latency: 1638.21 ms, (p50: 1400 ms, p70: 1600, p90: 2200 ms, p99: 3100 ms), latency samples: 1880
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.255, avg: 2.198", "ConsensusProposalToOrdered: max: 0.303, avg: 0.301", "ConsensusOrderedToCommit: max: 0.339, avg: 0.322", "ConsensusProposalToCommit: max: 0.638, avg: 0.624"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.65s no progress at version 24683 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.88s no progress at version 1820242 (avg 0.88s) [limit 16].
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16621&var-metrics_source=All&var-chain_name=forge-1&from=1747249616000&to=1747250386000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1747249616000&end=1747250386000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-14T19%3A06%3A56.052605%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-14T19%3A19%3A46.524512%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747249616&until=1747250386&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747249616&until=1747250386&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15028674577)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-14T19:22:39Z
body: ### :white_check_mark: Forge suite `framework_upgrade` success on `46ff583f7c392d9a34cc445f7b279c2d7930a374` ==> `e0169e4f344aba880d7bc0c2955e76cc20fd8bdd`
```
Compatibility test results for 46ff583f7c392d9a34cc445f7b279c2d7930a374 ==> e0169e4f344aba880d7bc0c2955e76cc20fd8bdd (PR)
Upgrade the nodes to version: e0169e4f344aba880d7bc0c2955e76cc20fd8bdd
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1698.45 txn/s, submitted: 1703.49 txn/s, failed submission: 5.04 txn/s, expired: 5.04 txn/s, latency: 1699.63 ms, (p50: 1500 ms, p70: 1800, p90: 2700 ms, p99: 3800 ms), latency samples: 154881
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1593.59 txn/s, submitted: 1599.76 txn/s, failed submission: 6.17 txn/s, expired: 6.17 txn/s, latency: 1840.94 ms, (p50: 1600 ms, p70: 1900, p90: 2700 ms, p99: 4300 ms), latency samples: 144643
5. check swarm health
Compatibility test for 46ff583f7c392d9a34cc445f7b279c2d7930a374 ==> e0169e4f344aba880d7bc0c2955e76cc20fd8bdd passed
Upgrade the remaining nodes to version: e0169e4f344aba880d7bc0c2955e76cc20fd8bdd
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1532.81 txn/s, submitted: 1538.04 txn/s, failed submission: 5.23 txn/s, expired: 5.23 txn/s, latency: 1874.57 ms, (p50: 1600 ms, p70: 1900, p90: 2800 ms, p99: 4000 ms), latency samples: 140601
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16621&var-metrics_source=All&var-chain_name=forge-1&from=1747249607000&to=1747250558000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1747249607000&end=1747250558000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-14T19%3A06%3A47.644427%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-14T19%3A22%3A38.496514%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747249607&until=1747250558&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747249607&until=1747250558&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15028674577)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-19T20:11:17Z
body: ### Forge is running suite `framework_upgrade` on `6ff41a58d52e09e11cf6d88c770b376f1108749d` ==> `d4b6506b919654ebc23f63fa8485573df2b9d755`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16621&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15121947877)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-19T20:11:17Z
body: ### Forge is running suite `realistic_env_max_load` on `d4b6506b919654ebc23f63fa8485573df2b9d755`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16621&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15121947877)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-19T20:11:20Z
body: ### Forge is running suite `compat` on `6ff41a58d52e09e11cf6d88c770b376f1108749d` ==> `d4b6506b919654ebc23f63fa8485573df2b9d755`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16621&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15121947877)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-19T20:22:08Z
body: ### :white_check_mark: Forge suite `compat` success on `6ff41a58d52e09e11cf6d88c770b376f1108749d` ==> `d4b6506b919654ebc23f63fa8485573df2b9d755`
```
Compatibility test results for 6ff41a58d52e09e11cf6d88c770b376f1108749d ==> d4b6506b919654ebc23f63fa8485573df2b9d755 (PR)
1. Check liveness of validators at old version: 6ff41a58d52e09e11cf6d88c770b376f1108749d
compatibility::simple-validator-upgrade::liveness-check : committed: 9551.56 txn/s, latency: 3523.08 ms, (p50: 3800 ms, p70: 3900, p90: 4000 ms, p99: 4100 ms), latency samples: 318820
2. Upgrading first Validator to new version: d4b6506b919654ebc23f63fa8485573df2b9d755
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 2553.10 txn/s, latency: 13111.70 ms, (p50: 14700 ms, p70: 15100, p90: 15300 ms, p99: 15400 ms), latency samples: 95980
3. Upgrading rest of first batch to new version: d4b6506b919654ebc23f63fa8485573df2b9d755
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 2703.93 txn/s, latency: 12332.50 ms, (p50: 13900 ms, p70: 14200, p90: 14500 ms, p99: 14600 ms), latency samples: 100960
4. upgrading second batch to new version: d4b6506b919654ebc23f63fa8485573df2b9d755
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 5018.25 txn/s, latency: 6771.95 ms, (p50: 7300 ms, p70: 7600, p90: 8100 ms, p99: 8300 ms), latency samples: 177720
5. check swarm health
Compatibility test for 6ff41a58d52e09e11cf6d88c770b376f1108749d ==> d4b6506b919654ebc23f63fa8485573df2b9d755 passed
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16621&var-metrics_source=All&var-chain_name=forge-1&from=1747685487000&to=1747686127000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1747685487000&end=1747686127000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-19T20%3A11%3A27.522049%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-19T20%3A22%3A07.107688%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747685487&until=1747686127&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747685487&until=1747686127&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15121947877)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-19T20:22:32Z
body: ### :x: Forge suite `realistic_env_max_load` failure on `d4b6506b919654ebc23f63fa8485573df2b9d755`
```
two traffics test: inner traffic : committed: 10186.94 txn/s, submitted: 10187.63 txn/s, expired: 0.68 txn/s, latency: 3555.41 ms, (p50: 3000 ms, p70: 3600, p90: 5100 ms, p99: 9300 ms), latency samples: 3873280
two traffics test : committed: 99.96 txn/s, latency: 1853.52 ms, (p50: 1500 ms, p70: 1700, p90: 2800 ms, p99: 7800 ms), latency samples: 1780
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.853, avg: 2.417", "ConsensusProposalToOrdered: max: 0.324, avg: 0.315", "ConsensusOrderedToCommit: max: 0.387, avg: 0.339", "ConsensusProposalToCommit: max: 0.710, avg: 0.655"]
Max non-epoch-change gap was: 1 rounds at version 2142837 (avg 0.00) [limit 4], 2.23s no progress at version 2142837 (avg 0.22s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.62s no progress at version 2142795 (avg 0.62s) [limit 16].
Test Failed: check for success

Caused by:
    The following errors occurred:
    --------------------------------
    Error 1: "MempoolToBlockCreation" metric violated threshold of 2.85, max_breach_pct: 5, breach_pct: 7 
    Caused by:
    
    

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.98/src/backtrace.rs:27:14
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/result.rs:2014:27
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:393:9
   3: aptos_forge::interface::network::NetworkContext::check_for_success::{{closure}}
             at ./testsuite/forge/src/interface/network.rs:112:10
   4: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:333:14
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
   6: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
   7: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
   8: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:639:37
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  10: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/park.rs:281:63
  11: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:107:5
  12: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:73:5
  13: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/park.rs:281:31
  14: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  15: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:324:22
  16: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  17: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:323:9
  18: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:302:18
  19: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  20: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:430:24
  21: forge::main
             at ./testsuite/forge-cli/src/main.rs:333:21
  22: core::ops::function::FnOnce::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
  23: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:152:18
  24: std::rt::lang_start::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:199:18
  25: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:284:13
  26: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  27: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  28: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  29: std::rt::lang_start_internal::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:168:24
  30: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  31: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  32: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  33: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  34: main
  35: __libc_start_main
  36: _start
Trailing Log Lines:
  31: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  32: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  33: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  34: main
  35: __libc_start_main
  36: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="2d322aa4-20a4-409e-bd15-ac735ae50958">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="CompositeNetworkTest(network:multi-region-network-emulation(two traffics test)) with ">
            <failure message="check for success

Caused by:
    The following errors occurred:
    --------------------------------
    Error 1: &quot;MempoolToBlockCreation&quot; metric violated threshold of 2.85, max_breach_pct: 5, breach_pct: 7 
    Caused by:
    
    

Stack backtrace:
   0: anyhow::error::&lt;impl core::convert::From&lt;E&gt; for anyhow::Error&gt;::from
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.98/src/backtrace.rs:27:14
   1: &lt;core::result::Result&lt;T,F&gt; as core::ops::try_trait::FromResidual&lt;core::result::Result&lt;core::convert::Infallible,E&gt;&gt;&gt;::from_residual
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/result.rs:2014:27
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:393:9
   3: aptos_forge::interface::network::NetworkContext::check_for_success::{{closure}}
             at ./testsuite/forge/src/interface/network.rs:112:10
   4: &lt;dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:333:14
   5: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
   6: &lt;aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
   7: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
   8: &lt;aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:639:37
   9: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  10: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/park.rs:281:63
  11: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:107:5
  12: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:73:5
  13: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/park.rs:281:31
  14: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  15: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:324:22
  16: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  17: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:323:9
  18: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/handle.rs:302:18
  19: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  20: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:430:24
  21: forge::main
             at ./testsuite/forge-cli/src/main.rs:333:21
  22: core::ops::function::FnOnce::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
  23: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:152:18
  24: std::rt::lang_start::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:199:18
  25: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:284:13
  26: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  27: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  28: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  29: std::rt::lang_start_internal::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:168:24
  30: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  31: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  32: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  33: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  34: main
  35: __libc_start_main
  36: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
[2025-05-19T20:22:28Z INFO  aptos_forge::backend::k8s::cluster_helper] Deleting namespace forge-e2e-pr-16621: Some(NamespaceStatus { conditions: None, phase: Some("Terminating") })
[2025-05-19T20:22:28Z INFO  aptos_forge::backend::k8s::cluster_helper] aptos-node resources for Forge removed in namespace: forge-e2e-pr-16621

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.98/src/backtrace.rs:27:14
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:430:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:333:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
   5: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:152:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:199:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
   9: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  10: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:168:24
  12: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  13: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  14: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  15: std::rt::lang_start_internal
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/rt.rs:164:5
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge136-0       1/1     Running     0          10m
aptos-node-0-validator-0                1/1     Running     0          10m
aptos-node-1-fullnode-eforge136-0       1/1     Running     0          10m
aptos-node-1-validator-0                1/1     Running     0          10m
aptos-node-2-fullnode-eforge136-0       1/1     Running     0          10m
aptos-node-2-validator-0                1/1     Running     0          10m
aptos-node-3-fullnode-eforge136-0       1/1     Running     0          10m
aptos-node-3-validator-0                1/1     Running     0          10m
aptos-node-4-fullnode-eforge136-0       1/1     Running     0          10m
aptos-node-4-validator-0                1/1     Running     0          10m
aptos-node-5-validator-0                1/1     Running     0          10m
aptos-node-6-validator-0                1/1     Running     0          10m
forge-testnet-deployer-p7hwq            0/1     Completed   0          10m
genesis-aptos-genesis-eforge136-5bxqx   0/1     Completed   0          10m

```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16621&var-metrics_source=All&var-chain_name=forge-1&from=1747685483000&to=1747686151000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1747685483000&end=1747686151000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-19T20%3A11%3A23.929068%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-19T20%3A22%3A31.242727%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747685483&until=1747686151&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747685483&until=1747686151&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15121947877)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-19T20:24:22Z
body: ### :white_check_mark: Forge suite `framework_upgrade` success on `6ff41a58d52e09e11cf6d88c770b376f1108749d` ==> `d4b6506b919654ebc23f63fa8485573df2b9d755`
```
Forge report malformed: Expecting property name enclosed in double quotes: line 6 column 1 (char 139)
'{\n  "metrics": [\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "submitted_txn",\n[2025-05-19T20:24:18Z INFO  aptos_forge::report] Test Ok\n      "value": 162140.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "expired_txn",\n      "value": 600.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "avg_tps",\n      "value": 1791.1506422280643\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "avg_latency",\n      "value": 1622.0540794849574\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p50_latency",\n      "value": 1500.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p90_latency",\n      "value": 2100.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p99_latency",\n      "value": 3300.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "submitted_txn",\n      "value": 161240.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "expired_txn",\n      "value": 420.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "avg_tps",\n      "value": 1799.728384387917\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "avg_latency",\n      "value": 1660.5792811839324\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p50_latency",\n      "value": 1500.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p90_latency",\n      "value": 2100.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p99_latency",\n      "value": 3300.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "submitted_txn",\n      "value": 161540.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "expired_txn",\n      "value": 520.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "avg_tps",\n      "value": 1789.4592458144389\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "avg_latency",\n      "value": 1642.3766674947212\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p50_latency",\n      "value": 1500.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p90_latency",\n      "value": 2100.0\n    },\n    {\n      "test_name": "framework_upgrade::framework-upgrade::full-framework-upgrade",\n      "metric": "p99_latency",\n      "value": 3000.0\n    }\n  ],\n  "text": "Compatibility test results for 6ff41a58d52e09e11cf6d88c770b376f1108749d ==> d4b6506b919654ebc23f63fa8485573df2b9d755 (PR)\\nUpgrade the nodes to version: d4b6506b919654ebc23f63fa8485573df2b9d755\\nframework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1791.15 txn/s, submitted: 1797.80 txn/s, failed submission: 6.65 txn/s, expired: 6.65 txn/s, latency: 1622.05 ms, (p50: 1500 ms, p70: 1800, p90: 2100 ms, p99: 3300 ms), latency samples: 161540\\nframework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1799.73 txn/s, submitted: 1804.43 txn/s, failed submission: 4.70 txn/s, expired: 4.70 txn/s, latency: 1660.58 ms, (p50: 1500 ms, p70: 1800, p90: 2100 ms, p99: 3300 ms), latency samples: 160820\\n5. check swarm health\\nCompatibility test for 6ff41a58d52e09e11cf6d88c770b376f1108749d ==> d4b6506b919654ebc23f63fa8485573df2b9d755 passed\\nUpgrade the remaining nodes to version: d4b6506b919654ebc23f63fa8485573df2b9d755\\nframework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1789.46 txn/s, submitted: 1795.24 txn/s, failed submission: 5.78 txn/s, expired: 5.78 txn/s, latency: 1642.38 ms, (p50: 1500 ms, p70: 1800, p90: 2100 ms, p99: 3000 ms), latency samples: 161020\\nTest Ok"\n}'
Trailing Log Lines:
Compatibility test results for 6ff41a58d52e09e11cf6d88c770b376f1108749d ==> d4b6506b919654ebc23f63fa8485573df2b9d755 (PR)
Upgrade the nodes to version: d4b6506b919654ebc23f63fa8485573df2b9d755
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1791.15 txn/s, submitted: 1797.80 txn/s, failed submission: 6.65 txn/s, expired: 6.65 txn/s, latency: 1622.05 ms, (p50: 1500 ms, p70: 1800, p90: 2100 ms, p99: 3300 ms), latency samples: 161540
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1799.73 txn/s, submitted: 1804.43 txn/s, failed submission: 4.70 txn/s, expired: 4.70 txn/s, latency: 1660.58 ms, (p50: 1500 ms, p70: 1800, p90: 2100 ms, p99: 3300 ms), latency samples: 160820
5. check swarm health
Compatibility test for 6ff41a58d52e09e11cf6d88c770b376f1108749d ==> d4b6506b919654ebc23f63fa8485573df2b9d755 passed
Upgrade the remaining nodes to version: d4b6506b919654ebc23f63fa8485573df2b9d755
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1789.46 txn/s, submitted: 1795.24 txn/s, failed submission: 5.78 txn/s, expired: 5.78 txn/s, latency: 1642.38 ms, (p50: 1500 ms, p70: 1800, p90: 2100 ms, p99: 3000 ms), latency samples: 161020
Test Ok

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="0" errors="0" uuid="4bc614dc-e265-4729-b59a-dad135117073">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="0">
        <testcase name="framework_upgrade::framework-upgrade">
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===
[2025-05-19T20:24:18Z INFO  aptos_forge::backend::k8s::cluster_helper] Deleting namespace forge-framework-upgrade-pr-16621: Some(NamespaceStatus { conditions: None, phase: Some("Terminating") })
[2025-05-19T20:24:18Z INFO  aptos_forge::backend::k8s::cluster_helper] aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-16621

test result: ok. 1 passed; 0 failed; 0 filtered out

Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          10m
aptos-node-1-validator-0               1/1     Running     0          9m50s
aptos-node-2-validator-0               1/1     Running     0          3m15s
aptos-node-3-validator-0               1/1     Running     0          2m32s
forge-testnet-deployer-82rmr           0/1     Completed   0          12m
genesis-aptos-genesis-eforge60-h27sv   0/1     Completed   0          12m

```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16621&var-metrics_source=All&var-chain_name=forge-1&from=1747685484000&to=1747686261000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1747685484000&end=1747686261000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-19T20%3A11%3A24.317356%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-19T20%3A24%3A21.231722%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747685484&until=1747686261&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747685484&until=1747686261&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15121947877)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-20T00:59:35Z
body: ### Forge is running suite `realistic_env_max_load` on `d4b6506b919654ebc23f63fa8485573df2b9d755`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16621&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15121947877)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-20T01:14:40Z
body: ### :white_check_mark: Forge suite `realistic_env_max_load` success on `d4b6506b919654ebc23f63fa8485573df2b9d755`
```
two traffics test: inner traffic : committed: 11300.74 txn/s, latency: 3206.06 ms, (p50: 2700 ms, p70: 3200, p90: 4800 ms, p99: 6600 ms), latency samples: 4296800
two traffics test : committed: 99.99 txn/s, latency: 1961.30 ms, (p50: 1400 ms, p70: 1600, p90: 3100 ms, p99: 6000 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.258, avg: 1.962", "ConsensusProposalToOrdered: max: 0.311, avg: 0.302", "ConsensusOrderedToCommit: max: 0.339, avg: 0.329", "ConsensusProposalToCommit: max: 0.645, avg: 0.631"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.01s no progress at version 21756 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.05s no progress at version 33604 (avg 0.82s) [limit 16].
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16621&var-metrics_source=All&var-chain_name=forge-1&from=1747702782000&to=1747703678000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16621%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16621%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1747702782000&end=1747703678000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16621%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-20T00%3A59%3A42.308759%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-20T01%3A14%3A38.782839%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747702782&until=1747703678&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1747702782&until=1747703678&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16621&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/15121947877)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR