Repository: aptos-labs/aptos-core
Pull request #16393: Add upsert_ed25519_backup_key_on_keyless_account entry function
Author: login: heliuchuan
Created: 2025-04-17T16:54:41Z | Merged: 2025-04-22T16:46:24Z

## Description
<!-- Please include a summary of the change, including which issue it fixes or what feature it adds. Include relevant motivation, context and documentation as appropriate. List dependencies that are required for this change, if any. -->
This function adds  a second ed25519 public key on a keyless account by utilizing key rotation and MultiKey.

-  The functions requires the caller to provide the original keyless public key which is verified by checking its auth key matches the address. 
- A proof of challenge also needs to be provided for the backup key to install.
- It will verify the challenge with the provided backup key public key (which must be ed25519)
- It will rotate the authentication key by calling `rotate_authentication_key_call`
- Lastly it will emit an event KeyRotationToPublicKey recording the change marking both keys as verified.

## How Has This Been Tested?
<!--
- Please ensure that the functionality introduced by this change is well tested and verified to work as expected.
- Ensure tests cover both happy and unhappy paths.
- List and link relevant tests.
-->

Tested locally

## Type of Change
- [x] New feature
- [ ] Bug fix
- [ ] Breaking change
- [ ] Performance improvement
- [ ] Refactoring
- [ ] Dependency update
- [ ] Documentation update
- [ ] Tests

## Which Components or Systems Does This Change Impact?
- [ ] Validator Node
- [ ] Full Node (API, Indexer, etc.)
- [ ] Move/Aptos Virtual Machine
- [x] Aptos Framework
- [ ] Aptos CLI/SDK
- [ ] Developer Infrastructure
- [ ] Move Compiler
- [ ] Other (specify)

## Checklist
- [x] I have read and followed the [CONTRIBUTING](https://github.com/aptos-labs/aptos-core/blob/main/CONTRIBUTING.md) doc
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I identified and added all stakeholders and component owners affected by this change as reviewers
- [x] I tested both happy and unhappy path of the functionality
- [x] I have made corresponding changes to the documentation

<!-- Thank you for your contribution! -->



Diff:
diff --git a/aptos-move/framework/aptos-framework/doc/account.md b/aptos-move/framework/aptos-framework/doc/account.md
index dbb78bb2798..e3bb969dae1 100644
--- a/aptos-move/framework/aptos-framework/doc/account.md
+++ b/aptos-move/framework/aptos-framework/doc/account.md
@@ -41,6 +41,10 @@
 -  [Function `rotate_authentication_key_internal`](#0x1_account_rotate_authentication_key_internal)
 -  [Function `rotate_authentication_key_call`](#0x1_account_rotate_authentication_key_call)
 -  [Function `rotate_authentication_key_from_public_key`](#0x1_account_rotate_authentication_key_from_public_key)
+-  [Function `upsert_ed25519_backup_key_on_keyless_account`](#0x1_account_upsert_ed25519_backup_key_on_keyless_account)
+    -  [Arguments](#@Arguments_1)
+    -  [Aborts](#@Aborts_2)
+    -  [Events](#@Events_3)
 -  [Function `rotate_authentication_key`](#0x1_account_rotate_authentication_key)
 -  [Function `rotate_authentication_key_with_rotation_capability`](#0x1_account_rotate_authentication_key_with_rotation_capability)
 -  [Function `offer_rotation_capability`](#0x1_account_offer_rotation_capability)
@@ -67,46 +71,46 @@
 -  [Function `create_signer_with_capability`](#0x1_account_create_signer_with_capability)
 -  [Function `get_signer_capability_address`](#0x1_account_get_signer_capability_address)
 -  [Function `verify_signed_message`](#0x1_account_verify_signed_message)
--  [Specification](#@Specification_1)
+-  [Specification](#@Specification_4)
     -  [High-level Requirements](#high-level-req)
     -  [Module-level Specification](#module-level-spec)
-    -  [Function `initialize`](#@Specification_1_initialize)
-    -  [Function `create_account_if_does_not_exist`](#@Specification_1_create_account_if_does_not_exist)
-    -  [Function `create_account`](#@Specification_1_create_account)
-    -  [Function `create_account_unchecked`](#@Specification_1_create_account_unchecked)
-    -  [Function `exists_at`](#@Specification_1_exists_at)
-    -  [Function `get_guid_next_creation_num`](#@Specification_1_get_guid_next_creation_num)
-    -  [Function `get_sequence_number`](#@Specification_1_get_sequence_number)
-    -  [Function `originating_address`](#@Specification_1_originating_address)
-    -  [Function `increment_sequence_number`](#@Specification_1_increment_sequence_number)
-    -  [Function `get_authentication_key`](#@Specification_1_get_authentication_key)
-    -  [Function `rotate_authentication_key_internal`](#@Specification_1_rotate_authentication_key_internal)
-    -  [Function `rotate_authentication_key_call`](#@Specification_1_rotate_authentication_key_call)
-    -  [Function `rotate_authentication_key_from_public_key`](#@Specification_1_rotate_authentication_key_from_public_key)
-    -  [Function `rotate_authentication_key`](#@Specification_1_rotate_authentication_key)
-    -  [Function `rotate_authentication_key_with_rotation_capability`](#@Specification_1_rotate_authentication_key_with_rotation_capability)
-    -  [Function `offer_rotation_capability`](#@Specification_1_offer_rotation_capability)
-    -  [Function `set_originating_address`](#@Specification_1_set_originating_address)
-    -  [Function `is_rotation_capability_offered`](#@Specification_1_is_rotation_capability_offered)
-    -  [Function `get_rotation_capability_offer_for`](#@Specification_1_get_rotation_capability_offer_for)
-    -  [Function `revoke_rotation_capability`](#@Specification_1_revoke_rotation_capability)
-    -  [Function `revoke_any_rotation_capability`](#@Specification_1_revoke_any_rotation_capability)
-    -  [Function `offer_signer_capability`](#@Specification_1_offer_signer_capability)
-    -  [Function `is_signer_capability_offered`](#@Specification_1_is_signer_capability_offered)
-    -  [Function `get_signer_capability_offer_for`](#@Specification_1_get_signer_capability_offer_for)
-    -  [Function `revoke_signer_capability`](#@Specification_1_revoke_signer_capability)
-    -  [Function `revoke_any_signer_capability`](#@Specification_1_revoke_any_signer_capability)
-    -  [Function `create_authorized_signer`](#@Specification_1_create_authorized_signer)
-    -  [Function `assert_valid_rotation_proof_signature_and_get_auth_key`](#@Specification_1_assert_valid_rotation_proof_signature_and_get_auth_key)
-    -  [Function `update_auth_key_and_originating_address_table`](#@Specification_1_update_auth_key_and_originating_address_table)
-    -  [Function `create_resource_address`](#@Specification_1_create_resource_address)
-    -  [Function `create_resource_account`](#@Specification_1_create_resource_account)
-    -  [Function `create_framework_reserved_account`](#@Specification_1_create_framework_reserved_account)
-    -  [Function `create_guid`](#@Specification_1_create_guid)
-    -  [Function `new_event_handle`](#@Specification_1_new_event_handle)
-    -  [Function `register_coin`](#@Specification_1_register_coin)
-    -  [Function `create_signer_with_capability`](#@Specification_1_create_signer_with_capability)
-    -  [Function `verify_signed_message`](#@Specification_1_verify_signed_message)
+    -  [Function `initialize`](#@Specification_4_initialize)
+    -  [Function `create_account_if_does_not_exist`](#@Specification_4_create_account_if_does_not_exist)
+    -  [Function `create_account`](#@Specification_4_create_account)
+    -  [Function `create_account_unchecked`](#@Specification_4_create_account_unchecked)
+    -  [Function `exists_at`](#@Specification_4_exists_at)
+    -  [Function `get_guid_next_creation_num`](#@Specification_4_get_guid_next_creation_num)
+    -  [Function `get_sequence_number`](#@Specification_4_get_sequence_number)
+    -  [Function `originating_address`](#@Specification_4_originating_address)
+    -  [Function `increment_sequence_number`](#@Specification_4_increment_sequence_number)
+    -  [Function `get_authentication_key`](#@Specification_4_get_authentication_key)
+    -  [Function `rotate_authentication_key_internal`](#@Specification_4_rotate_authentication_key_internal)
+    -  [Function `rotate_authentication_key_call`](#@Specification_4_rotate_authentication_key_call)
+    -  [Function `rotate_authentication_key_from_public_key`](#@Specification_4_rotate_authentication_key_from_public_key)
+    -  [Function `rotate_authentication_key`](#@Specification_4_rotate_authentication_key)
+    -  [Function `rotate_authentication_key_with_rotation_capability`](#@Specification_4_rotate_authentication_key_with_rotation_capability)
+    -  [Function `offer_rotation_capability`](#@Specification_4_offer_rotation_capability)
+    -  [Function `set_originating_address`](#@Specification_4_set_originating_address)
+    -  [Function `is_rotation_capability_offered`](#@Specification_4_is_rotation_capability_offered)
+    -  [Function `get_rotation_capability_offer_for`](#@Specification_4_get_rotation_capability_offer_for)
+    -  [Function `revoke_rotation_capability`](#@Specification_4_revoke_rotation_capability)
+    -  [Function `revoke_any_rotation_capability`](#@Specification_4_revoke_any_rotation_capability)
+    -  [Function `offer_signer_capability`](#@Specification_4_offer_signer_capability)
+    -  [Function `is_signer_capability_offered`](#@Specification_4_is_signer_capability_offered)
+    -  [Function `get_signer_capability_offer_for`](#@Specification_4_get_signer_capability_offer_for)
+    -  [Function `revoke_signer_capability`](#@Specification_4_revoke_signer_capability)
+    -  [Function `revoke_any_signer_capability`](#@Specification_4_revoke_any_signer_capability)
+    -  [Function `create_authorized_signer`](#@Specification_4_create_authorized_signer)
+    -  [Function `assert_valid_rotation_proof_signature_and_get_auth_key`](#@Specification_4_assert_valid_rotation_proof_signature_and_get_auth_key)
+    -  [Function `update_auth_key_and_originating_address_table`](#@Specification_4_update_auth_key_and_originating_address_table)
+    -  [Function `create_resource_address`](#@Specification_4_create_resource_address)
+    -  [Function `create_resource_account`](#@Specification_4_create_resource_account)
+    -  [Function `create_framework_reserved_account`](#@Specification_4_create_framework_reserved_account)
+    -  [Function `create_guid`](#@Specification_4_create_guid)
+    -  [Function `new_event_handle`](#@Specification_4_new_event_handle)
+    -  [Function `register_coin`](#@Specification_4_register_coin)
+    -  [Function `create_signer_with_capability`](#@Specification_4_create_signer_with_capability)
+    -  [Function `verify_signed_message`](#@Specification_4_verify_signed_message)
 
 
 <pre><code><b>use</b> <a href="../../aptos-stdlib/../move-stdlib/doc/bcs.md#0x1_bcs">0x1::bcs</a>;
@@ -919,6 +923,26 @@ The current authentication key and the new authentication key are the same
 
 
 
+<a id="0x1_account_ENOT_A_KEYLESS_PUBLIC_KEY"></a>
+
+The provided public key is not a single Keyless public key
+
+
+<pre><code><b>const</b> <a href="account.md#0x1_account_ENOT_A_KEYLESS_PUBLIC_KEY">ENOT_A_KEYLESS_PUBLIC_KEY</a>: u64 = 25;
+</code></pre>
+
+
+
+<a id="0x1_account_ENOT_THE_ORIGINAL_PUBLIC_KEY"></a>
+
+The provided public key is not the original public key for the account
+
+
+<pre><code><b>const</b> <a href="account.md#0x1_account_ENOT_THE_ORIGINAL_PUBLIC_KEY">ENOT_THE_ORIGINAL_PUBLIC_KEY</a>: u64 = 26;
+</code></pre>
+
+
+
 <a id="0x1_account_ENO_ACCOUNT_PERMISSION"></a>
 
 Current permissioned signer cannot perform the privilaged operations.
@@ -1723,6 +1747,116 @@ Note: This function does not update the <code><a href="account.md#0x1_account_Or
 
 
 
+</details>
+
+<a id="0x1_account_upsert_ed25519_backup_key_on_keyless_account"></a>
+
+## Function `upsert_ed25519_backup_key_on_keyless_account`
+
+Upserts an ED25519 backup key to an account that has a keyless public key as its original public key by converting the account's authentication key
+to a multi-key of the original keyless public key and the new backup key that requires 1 signature from either key to authenticate.
+This function takes a the account's original keyless public key and a ED25519 backup public key and rotates the account's authentication key to a multi-key of
+the original keyless public key and the new backup key that requires 1 signature from either key to authenticate.
+
+Note: This function emits a <code>KeyRotationToMultiPublicKey</code> event marking both keys as verified since the keyless public key
+is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+
+
+<a id="@Arguments_1"></a>
+
+### Arguments
+
+* <code><a href="account.md#0x1_account">account</a></code> - The signer representing the keyless account
+* <code>keyless_public_key</code> - The original keyless public key of the account (wrapped in an AnyPublicKey)
+* <code>backup_public_key</code> - The ED25519 public key to add as a backup
+* <code>backup_key_proof</code> - A signature from the backup key proving ownership
+
+
+<a id="@Aborts_2"></a>
+
+### Aborts
+
+* If the any of inputs deserialize incorrectly
+* If the provided public key is not a keyless public key
+* If the keyless public key is not the original public key of the account
+* If the backup key proof signature is invalid
+
+
+<a id="@Events_3"></a>
+
+### Events
+
+* Emits a <code>KeyRotationToMultiPublicKey</code> event with the new multi-key configuration
+
+
+<pre><code>entry <b>fun</b> <a href="account.md#0x1_account_upsert_ed25519_backup_key_on_keyless_account">upsert_ed25519_backup_key_on_keyless_account</a>(<a href="account.md#0x1_account">account</a>: &<a href="../../aptos-stdlib/../move-stdlib/doc/signer.md#0x1_signer">signer</a>, keyless_public_key: <a href="../../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;u8&gt;, backup_public_key: <a href="../../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;u8&gt;, backup_key_proof: <a href="../../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;u8&gt;)
+</code></pre>
+
+
+
+<details>
+<summary>Implementation</summary>
+
+
+<pre><code>entry <b>fun</b> <a href="account.md#0x1_account_upsert_ed25519_backup_key_on_keyless_account">upsert_ed25519_backup_key_on_keyless_account</a>(<a href="account.md#0x1_account">account</a>: &<a href="../../aptos-stdlib/../move-stdlib/doc/signer.md#0x1_signer">signer</a>, keyless_public_key: <a href="../../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;u8&gt;, backup_public_key: <a href="../../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;u8&gt;, backup_key_proof: <a href="../../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>&lt;u8&gt;) <b>acquires</b> <a href="account.md#0x1_account_Account">Account</a> {
+    // Check that the provided <b>public</b> key is a <a href="../../aptos-stdlib/doc/keyless.md#0x1_keyless">keyless</a> <b>public</b> key
+    <b>let</b> keyless_single_key = <a href="../../aptos-stdlib/doc/single_key.md#0x1_single_key_new_public_key_from_bytes">single_key::new_public_key_from_bytes</a>(keyless_public_key);
+    <b>assert</b>!(<a href="../../aptos-stdlib/doc/single_key.md#0x1_single_key_is_keyless_or_federated_keyless_public_key">single_key::is_keyless_or_federated_keyless_public_key</a>(&keyless_single_key), <a href="../../aptos-stdlib/../move-stdlib/doc/error.md#0x1_error_invalid_argument">error::invalid_argument</a>(<a href="account.md#0x1_account_ENOT_A_KEYLESS_PUBLIC_KEY">ENOT_A_KEYLESS_PUBLIC_KEY</a>));
+
+    <b>let</b> addr = <a href="../../aptos-stdlib/../move-stdlib/doc/signer.md#0x1_signer_address_of">signer::address_of</a>(<a href="account.md#0x1_account">account</a>);
+    <b>let</b> account_resource = &<b>mut</b> <a href="account.md#0x1_account_Account">Account</a>[addr];
+    <b>let</b> old_auth_key = account_resource.authentication_key;
+
+    // Check that the provided <b>public</b> key is original <b>public</b> key of the <a href="account.md#0x1_account">account</a> by comparing
+    // its authentication key <b>to</b> the <a href="account.md#0x1_account">account</a> <b>address</b>.
+    <b>assert</b>!(
+        <a href="../../aptos-stdlib/../move-stdlib/doc/bcs.md#0x1_bcs_to_bytes">bcs::to_bytes</a>(&addr) == keyless_single_key.to_authentication_key(),
+        <a href="../../aptos-stdlib/../move-stdlib/doc/error.md#0x1_error_invalid_argument">error::invalid_argument</a>(<a href="account.md#0x1_account_ENOT_THE_ORIGINAL_PUBLIC_KEY">ENOT_THE_ORIGINAL_PUBLIC_KEY</a>)
+    );
+
+    <b>let</b> curr_auth_key_as_address = <a href="../../aptos-stdlib/doc/from_bcs.md#0x1_from_bcs_to_address">from_bcs::to_address</a>(old_auth_key);
+    <b>let</b> challenge = <a href="account.md#0x1_account_RotationProofChallenge">RotationProofChallenge</a> {
+        sequence_number: account_resource.sequence_number,
+        originator: addr,
+        current_auth_key: curr_auth_key_as_address,
+        new_public_key: backup_public_key,
+    };
+
+    // Assert the challenges signed by the provided backup key is valid
+    <a href="account.md#0x1_account_assert_valid_rotation_proof_signature_and_get_auth_key">assert_valid_rotation_proof_signature_and_get_auth_key</a>(
+        <a href="account.md#0x1_account_ED25519_SCHEME">ED25519_SCHEME</a>,
+        backup_public_key,
+        backup_key_proof,
+        &challenge
+    );
+
+    // Get the backup key <b>as</b> a single key
+    <b>let</b> backup_key_ed25519 = <a href="../../aptos-stdlib/doc/ed25519.md#0x1_ed25519_new_unvalidated_public_key_from_bytes">ed25519::new_unvalidated_public_key_from_bytes</a>(backup_public_key);
+    <b>let</b> backup_key_as_single_key = <a href="../../aptos-stdlib/doc/single_key.md#0x1_single_key_from_ed25519_public_key_unvalidated">single_key::from_ed25519_public_key_unvalidated</a>(backup_key_ed25519);
+
+    <b>let</b> new_public_key = <a href="../../aptos-stdlib/doc/multi_key.md#0x1_multi_key_new_multi_key_from_single_keys">multi_key::new_multi_key_from_single_keys</a>(<a href="../../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>[keyless_single_key, backup_key_as_single_key], 1);
+    <b>let</b> new_auth_key = new_public_key.to_authentication_key();
+
+    // Rotate the authentication key <b>to</b> the new multi key <b>public</b> key
+    <a href="account.md#0x1_account_rotate_authentication_key_call">rotate_authentication_key_call</a>(<a href="account.md#0x1_account">account</a>, new_auth_key);
+
+    <a href="event.md#0x1_event_emit">event::emit</a>(<a href="account.md#0x1_account_KeyRotationToPublicKey">KeyRotationToPublicKey</a> {
+        <a href="account.md#0x1_account">account</a>: addr,
+        // This marks that both the <a href="../../aptos-stdlib/doc/keyless.md#0x1_keyless">keyless</a> <b>public</b> key and the new backup key are verified
+        // The <a href="../../aptos-stdlib/doc/keyless.md#0x1_keyless">keyless</a> <b>public</b> key is the original <b>public</b> key of the <a href="account.md#0x1_account">account</a> and the new backup key
+        // <b>has</b> been validated via verifying the challenge signed by the new backup key.
+        // Represents the bitmap 0b11000000000000000000000000000000
+        verified_public_key_bit_map: <a href="../../aptos-stdlib/../move-stdlib/doc/vector.md#0x1_vector">vector</a>[0xC0, 0x00, 0x00, 0x00],
+        public_key_scheme: <a href="account.md#0x1_account_MULTI_KEY_SCHEME">MULTI_KEY_SCHEME</a>,
+        public_key: <a href="../../aptos-stdlib/../move-stdlib/doc/bcs.md#0x1_bcs_to_bytes">bcs::to_bytes</a>(&new_public_key),
+        old_auth_key,
+        new_auth_key,
+    });
+}
+</code></pre>
+
+
+
 </details>
 
 <a id="0x1_account_rotate_authentication_key"></a>
@@ -2939,7 +3073,7 @@ Capability based functions for efficient use.
 
 </details>
 
-<a id="@Specification_1"></a>
+<a id="@Specification_4"></a>
 
 ## Specification
 
@@ -3066,7 +3200,7 @@ Capability based functions for efficient use.
 
 
 
-<a id="@Specification_1_initialize"></a>
+<a id="@Specification_4_initialize"></a>
 
 ### Function `initialize`
 
@@ -3087,7 +3221,7 @@ OriginatingAddress does not exist under <code>@aptos_framework</code> before the
 
 
 
-<a id="@Specification_1_create_account_if_does_not_exist"></a>
+<a id="@Specification_4_create_account_if_does_not_exist"></a>
 
 ### Function `create_account_if_does_not_exist`
 
@@ -3112,7 +3246,7 @@ Ensure that the account exists at the end of the call.
 
 
 
-<a id="@Specification_1_create_account"></a>
+<a id="@Specification_4_create_account"></a>
 
 ### Function `create_account`
 
@@ -3135,7 +3269,7 @@ Limit the new account address is not @vm_reserved / @aptos_framework / @aptos_to
 
 
 
-<a id="@Specification_1_create_account_unchecked"></a>
+<a id="@Specification_4_create_account_unchecked"></a>
 
 ### Function `create_account_unchecked`
 
@@ -3157,7 +3291,7 @@ The Account does not exist under the new address before creating the account.
 
 
 
-<a id="@Specification_1_exists_at"></a>
+<a id="@Specification_4_exists_at"></a>
 
 ### Function `exists_at`
 
@@ -3205,7 +3339,7 @@ The Account does not exist under the new address before creating the account.
 
 
 
-<a id="@Specification_1_get_guid_next_creation_num"></a>
+<a id="@Specification_4_get_guid_next_creation_num"></a>
 
 ### Function `get_guid_next_creation_num`
 
@@ -3223,7 +3357,7 @@ The Account does not exist under the new address before creating the account.
 
 
 
-<a id="@Specification_1_get_sequence_number"></a>
+<a id="@Specification_4_get_sequence_number"></a>
 
 ### Function `get_sequence_number`
 
@@ -3241,7 +3375,7 @@ The Account does not exist under the new address before creating the account.
 
 
 
-<a id="@Specification_1_originating_address"></a>
+<a id="@Specification_4_originating_address"></a>
 
 ### Function `originating_address`
 
@@ -3258,7 +3392,7 @@ The Account does not exist under the new address before creating the account.
 
 
 
-<a id="@Specification_1_increment_sequence_number"></a>
+<a id="@Specification_4_increment_sequence_number"></a>
 
 ### Function `increment_sequence_number`
 
@@ -3282,7 +3416,7 @@ The sequence_number of the Account is up to MAX_U64.
 
 
 
-<a id="@Specification_1_get_authentication_key"></a>
+<a id="@Specification_4_get_authentication_key"></a>
 
 ### Function `get_authentication_key`
 
@@ -3312,7 +3446,7 @@ The sequence_number of the Account is up to MAX_U64.
 
 
 
-<a id="@Specification_1_rotate_authentication_key_internal"></a>
+<a id="@Specification_4_rotate_authentication_key_internal"></a>
 
 ### Function `rotate_authentication_key_internal`
 
@@ -3336,7 +3470,7 @@ The length of new_auth_key is 32.
 
 
 
-<a id="@Specification_1_rotate_authentication_key_call"></a>
+<a id="@Specification_4_rotate_authentication_key_call"></a>
 
 ### Function `rotate_authentication_key_call`
 
@@ -3358,7 +3492,7 @@ The length of new_auth_key is 32.
 
 
 
-<a id="@Specification_1_rotate_authentication_key_from_public_key"></a>
+<a id="@Specification_4_rotate_authentication_key_from_public_key"></a>
 
 ### Function `rotate_authentication_key_from_public_key`
 
@@ -3383,7 +3517,7 @@ The length of new_auth_key is 32.
 
 
 
-<a id="@Specification_1_rotate_authentication_key"></a>
+<a id="@Specification_4_rotate_authentication_key"></a>
 
 ### Function `rotate_authentication_key`
 
@@ -3452,7 +3586,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME
 
 
 
-<a id="@Specification_1_rotate_authentication_key_with_rotation_capability"></a>
+<a id="@Specification_4_rotate_authentication_key_with_rotation_capability"></a>
 
 ### Function `rotate_authentication_key_with_rotation_capability`
 
@@ -3503,7 +3637,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME
 
 
 
-<a id="@Specification_1_offer_rotation_capability"></a>
+<a id="@Specification_4_offer_rotation_capability"></a>
 
 ### Function `offer_rotation_capability`
 
@@ -3557,7 +3691,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME
 
 
 
-<a id="@Specification_1_set_originating_address"></a>
+<a id="@Specification_4_set_originating_address"></a>
 
 ### Function `set_originating_address`
 
@@ -3573,7 +3707,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME
 
 
 
-<a id="@Specification_1_is_rotation_capability_offered"></a>
+<a id="@Specification_4_is_rotation_capability_offered"></a>
 
 ### Function `is_rotation_capability_offered`
 
@@ -3590,7 +3724,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME
 
 
 
-<a id="@Specification_1_get_rotation_capability_offer_for"></a>
+<a id="@Specification_4_get_rotation_capability_offer_for"></a>
 
 ### Function `get_rotation_capability_offer_for`
 
@@ -3609,7 +3743,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME
 
 
 
-<a id="@Specification_1_revoke_rotation_capability"></a>
+<a id="@Specification_4_revoke_rotation_capability"></a>
 
 ### Function `revoke_rotation_capability`
 
@@ -3633,7 +3767,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME
 
 
 
-<a id="@Specification_1_revoke_any_rotation_capability"></a>
+<a id="@Specification_4_revoke_any_rotation_capability"></a>
 
 ### Function `revoke_any_rotation_capability`
 
@@ -3656,7 +3790,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME
 
 
 
-<a id="@Specification_1_offer_signer_capability"></a>
+<a id="@Specification_4_offer_signer_capability"></a>
 
 ### Function `offer_signer_capability`
 
@@ -3710,7 +3844,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME.
 
 
 
-<a id="@Specification_1_is_signer_capability_offered"></a>
+<a id="@Specification_4_is_signer_capability_offered"></a>
 
 ### Function `is_signer_capability_offered`
 
@@ -3727,7 +3861,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME.
 
 
 
-<a id="@Specification_1_get_signer_capability_offer_for"></a>
+<a id="@Specification_4_get_signer_capability_offer_for"></a>
 
 ### Function `get_signer_capability_offer_for`
 
@@ -3746,7 +3880,7 @@ The authentication scheme is ED25519_SCHEME and MULTI_ED25519_SCHEME.
 
 
 
-<a id="@Specification_1_revoke_signer_capability"></a>
+<a id="@Specification_4_revoke_signer_capability"></a>
 
 ### Function `revoke_signer_capability`
 
@@ -3770,7 +3904,7 @@ The value of signer_capability_offer.for of Account resource under the signer is
 
 
 
-<a id="@Specification_1_revoke_any_signer_capability"></a>
+<a id="@Specification_4_revoke_any_signer_capability"></a>
 
 ### Function `revoke_any_signer_capability`
 
@@ -3790,7 +3924,7 @@ The value of signer_capability_offer.for of Account resource under the signer is
 
 
 
-<a id="@Specification_1_create_authorized_signer"></a>
+<a id="@Specification_4_create_authorized_signer"></a>
 
 ### Function `create_authorized_signer`
 
@@ -3832,7 +3966,7 @@ The value of signer_capability_offer.for of Account resource under the signer is
 
 
 
-<a id="@Specification_1_assert_valid_rotation_proof_signature_and_get_auth_key"></a>
+<a id="@Specification_4_assert_valid_rotation_proof_signature_and_get_auth_key"></a>
 
 ### Function `assert_valid_rotation_proof_signature_and_get_auth_key`
 
@@ -3879,7 +4013,7 @@ The value of signer_capability_offer.for of Account resource under the signer is
 
 
 
-<a id="@Specification_1_update_auth_key_and_originating_address_table"></a>
+<a id="@Specification_4_update_auth_key_and_originating_address_table"></a>
 
 ### Function `update_auth_key_and_originating_address_table`
 
@@ -3920,7 +4054,7 @@ The value of signer_capability_offer.for of Account resource under the signer is
 
 
 
-<a id="@Specification_1_create_resource_address"></a>
+<a id="@Specification_4_create_resource_address"></a>
 
 ### Function `create_resource_address`
 
@@ -3951,7 +4085,7 @@ The value of signer_capability_offer.for of Account resource under the signer is
 
 
 
-<a id="@Specification_1_create_resource_account"></a>
+<a id="@Specification_4_create_resource_account"></a>
 
 ### Function `create_resource_account`
 
@@ -3975,7 +4109,7 @@ The value of signer_capability_offer.for of Account resource under the signer is
 
 
 
-<a id="@Specification_1_create_framework_reserved_account"></a>
+<a id="@Specification_4_create_framework_reserved_account"></a>
 
 ### Function `create_framework_reserved_account`
 
@@ -4017,7 +4151,7 @@ The system reserved addresses is @0x1 / @0x2 / @0x3 / @0x4 / @0x5  / @0x6 / @0x7
 
 
 
-<a id="@Specification_1_create_guid"></a>
+<a id="@Specification_4_create_guid"></a>
 
 ### Function `create_guid`
 
@@ -4041,7 +4175,7 @@ The guid_creation_num of the account resource is up to MAX_U64.
 
 
 
-<a id="@Specification_1_new_event_handle"></a>
+<a id="@Specification_4_new_event_handle"></a>
 
 ### Function `new_event_handle`
 
@@ -4075,7 +4209,7 @@ The guid_creation_num of the Account is up to MAX_U64.
 
 
 
-<a id="@Specification_1_register_coin"></a>
+<a id="@Specification_4_register_coin"></a>
 
 ### Function `register_coin`
 
@@ -4093,7 +4227,7 @@ The guid_creation_num of the Account is up to MAX_U64.
 
 
 
-<a id="@Specification_1_create_signer_with_capability"></a>
+<a id="@Specification_4_create_signer_with_capability"></a>
 
 ### Function `create_signer_with_capability`
 
@@ -4122,7 +4256,7 @@ The guid_creation_num of the Account is up to MAX_U64.
 
 
 
-<a id="@Specification_1_verify_signed_message"></a>
+<a id="@Specification_4_verify_signed_message"></a>
 
 ### Function `verify_signed_message`
 
diff --git a/aptos-move/framework/aptos-framework/sources/account/account.move b/aptos-move/framework/aptos-framework/sources/account/account.move
index 3ce1c9cd4be..7ac3144d9f4 100644
--- a/aptos-move/framework/aptos-framework/sources/account/account.move
+++ b/aptos-move/framework/aptos-framework/sources/account/account.move
@@ -214,6 +214,10 @@ module aptos_framework::account {
     const ENO_ACCOUNT_PERMISSION: u64 = 23;
     /// Specified scheme is not recognized. Should be ED25519_SCHEME(0), MULTI_ED25519_SCHEME(1), SINGLE_KEY_SCHEME(2), or MULTI_KEY_SCHEME(3).
     const EUNRECOGNIZED_SCHEME: u64 = 24;
+    /// The provided public key is not a single Keyless public key
+    const ENOT_A_KEYLESS_PUBLIC_KEY: u64 = 25;
+    /// The provided public key is not the original public key for the account
+    const ENOT_THE_ORIGINAL_PUBLIC_KEY: u64 = 26;
 
     /// Explicitly separate the GUID space between Object and Account to prevent accidental overlap.
     const MAX_GUID_CREATION_NUM: u64 = 0x4000000000000;
@@ -489,6 +493,84 @@ module aptos_framework::account {
         });
     }
 
+    /// Upserts an ED25519 backup key to an account that has a keyless public key as its original public key by converting the account's authentication key
+    /// to a multi-key of the original keyless public key and the new backup key that requires 1 signature from either key to authenticate.
+    /// This function takes a the account's original keyless public key and a ED25519 backup public key and rotates the account's authentication key to a multi-key of
+    /// the original keyless public key and the new backup key that requires 1 signature from either key to authenticate.
+    ///
+    /// Note: This function emits a `KeyRotationToMultiPublicKey` event marking both keys as verified since the keyless public key
+    /// is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+    ///
+    /// # Arguments
+    /// * `account` - The signer representing the keyless account
+    /// * `keyless_public_key` - The original keyless public key of the account (wrapped in an AnyPublicKey)
+    /// * `backup_public_key` - The ED25519 public key to add as a backup
+    /// * `backup_key_proof` - A signature from the backup key proving ownership
+    ///
+    /// # Aborts
+    /// * If the any of inputs deserialize incorrectly
+    /// * If the provided public key is not a keyless public key
+    /// * If the keyless public key is not the original public key of the account
+    /// * If the backup key proof signature is invalid
+    ///
+    /// # Events
+    /// * Emits a `KeyRotationToMultiPublicKey` event with the new multi-key configuration
+    entry fun upsert_ed25519_backup_key_on_keyless_account(account: &signer, keyless_public_key: vector<u8>, backup_public_key: vector<u8>, backup_key_proof: vector<u8>) acquires Account {
+        // Check that the provided public key is a keyless public key
+        let keyless_single_key = single_key::new_public_key_from_bytes(keyless_public_key);
+        assert!(single_key::is_keyless_or_federated_keyless_public_key(&keyless_single_key), error::invalid_argument(ENOT_A_KEYLESS_PUBLIC_KEY));
+
+        let addr = signer::address_of(account);
+        let account_resource = &mut Account[addr];
+        let old_auth_key = account_resource.authentication_key;
+
+        // Check that the provided public key is original public key of the account by comparing
+        // its authentication key to the account address.
+        assert!(
+            bcs::to_bytes(&addr) == keyless_single_key.to_authentication_key(),
+            error::invalid_argument(ENOT_THE_ORIGINAL_PUBLIC_KEY)
+        );
+
+        let curr_auth_key_as_address = from_bcs::to_address(old_auth_key);
+        let challenge = RotationProofChallenge {
+            sequence_number: account_resource.sequence_number,
+            originator: addr,
+            current_auth_key: curr_auth_key_as_address,
+            new_public_key: backup_public_key,
+        };
+
+        // Assert the challenges signed by the provided backup key is valid
+        assert_valid_rotation_proof_signature_and_get_auth_key(
+            ED25519_SCHEME,
+            backup_public_key,
+            backup_key_proof,
+            &challenge
+        );
+
+        // Get the backup key as a single key
+        let backup_key_ed25519 = ed25519::new_unvalidated_public_key_from_bytes(backup_public_key);
+        let backup_key_as_single_key = single_key::from_ed25519_public_key_unvalidated(backup_key_ed25519);
+
+        let new_public_key = multi_key::new_multi_key_from_single_keys(vector[keyless_single_key, backup_key_as_single_key], 1);
+        let new_auth_key = new_public_key.to_authentication_key();
+
+        // Rotate the authentication key to the new multi key public key
+        rotate_authentication_key_call(account, new_auth_key);
+
+        event::emit(KeyRotationToPublicKey {
+            account: addr,
+            // This marks that both the keyless public key and the new backup key are verified
+            // The keyless public key is the original public key of the account and the new backup key
+            // has been validated via verifying the challenge signed by the new backup key.
+            // Represents the bitmap 0b11000000000000000000000000000000
+            verified_public_key_bit_map: vector[0xC0, 0x00, 0x00, 0x00],
+            public_key_scheme: MULTI_KEY_SCHEME,
+            public_key: bcs::to_bytes(&new_public_key),
+            old_auth_key,
+            new_auth_key,
+        });
+    }
+
     /// Generic authentication key rotation function that allows the user to rotate their authentication key from any scheme to any scheme.
     /// To authorize the rotation, we need two signatures:
     /// - the first signature `cap_rotate_key` refers to the signature by the account owner's current key on a valid `RotationProofChallenge`,
@@ -1936,6 +2018,40 @@ module aptos_framework::account {
         assert!(Account[alice_addr].authentication_key == new_auth_key, 0);
     }
 
+    #[test(account = @aptos_framework)]
+    public entry fun test_add_ed25519_backup_key_to_keyless_account(
+        account: signer
+    ) acquires Account {
+        initialize(&account);
+        let keyless_pk_bytes: vector<u8> = x"031b68747470733a2f2f6163636f756e74732e676f6f676c652e636f6d2086bc0a0a825eb6337ca1e8a3157e490eac8df23d5cef25d9641ad5e7edc1d514";
+        let curr_pk = single_key::new_public_key_from_bytes(keyless_pk_bytes);
+        let alice_addr = from_bcs::to_address(curr_pk.to_authentication_key());
+        let alice = create_account_unchecked(alice_addr);
+
+        let (new_sk, new_pk) = ed25519::generate_keys();
+
+        let challenge = RotationProofChallenge {
+            sequence_number: Account[alice_addr].sequence_number,
+            originator: alice_addr,
+            current_auth_key: alice_addr,
+            new_public_key: ed25519::validated_public_key_to_bytes(&new_pk),
+        };
+
+        let new_pk_unvalidated = ed25519::public_key_to_unvalidated(&new_pk);
+        let backup_key_as_single_key = single_key::from_ed25519_public_key_unvalidated(new_pk_unvalidated);
+        let new_public_key = multi_key::new_multi_key_from_single_keys(vector[curr_pk, backup_key_as_single_key], 1);
+        let new_auth_key = new_public_key.to_authentication_key();
+
+        let to_sig = ed25519::sign_struct(&new_sk, challenge);
+
+        upsert_ed25519_backup_key_on_keyless_account(
+            &alice,
+            keyless_pk_bytes,
+            ed25519::validated_public_key_to_bytes(&new_pk),
+            ed25519::signature_to_bytes(&to_sig),
+        );
+        assert!(Account[alice_addr].authentication_key == new_auth_key, 0);
+    }
 
     #[test(account = @aptos_framework)]
     public entry fun test_simple_rotation(account: &signer) acquires Account {
diff --git a/aptos-move/framework/cached-packages/src/aptos_framework_sdk_builder.rs b/aptos-move/framework/cached-packages/src/aptos_framework_sdk_builder.rs
index 6518f128079..608e7d4484c 100644
--- a/aptos-move/framework/cached-packages/src/aptos_framework_sdk_builder.rs
+++ b/aptos-move/framework/cached-packages/src/aptos_framework_sdk_builder.rs
@@ -174,6 +174,34 @@ pub enum EntryFunctionCall {
     /// authority of the new authentication key.
     AccountSetOriginatingAddress {},
 
+    /// Upserts an ED25519 backup key to an account that has a keyless public key as its original public key by converting the account's authentication key
+    /// to a multi-key of the original keyless public key and the new backup key that requires 1 signature from either key to authenticate.
+    /// This function takes a the account's original keyless public key and a ED25519 backup public key and rotates the account's authentication key to a multi-key of
+    /// the original keyless public key and the new backup key that requires 1 signature from either key to authenticate.
+    ///
+    /// Note: This function emits a `KeyRotationToMultiPublicKey` event marking both keys as verified since the keyless public key
+    /// is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+    ///
+    /// # Arguments
+    /// * `account` - The signer representing the keyless account
+    /// * `keyless_public_key` - The original keyless public key of the account (wrapped in an AnyPublicKey)
+    /// * `backup_public_key` - The ED25519 public key to add as a backup
+    /// * `backup_key_proof` - A signature from the backup key proving ownership
+    ///
+    /// # Aborts
+    /// * If the any of inputs deserialize incorrectly
+    /// * If the provided public key is not a keyless public key
+    /// * If the keyless public key is not the original public key of the account
+    /// * If the backup key proof signature is invalid
+    ///
+    /// # Events
+    /// * Emits a `KeyRotationToMultiPublicKey` event with the new multi-key configuration
+    AccountUpsertEd25519BackupKeyOnKeylessAccount {
+        keyless_public_key: Vec<u8>,
+        backup_public_key: Vec<u8>,
+        backup_key_proof: Vec<u8>,
+    },
+
     /// Add dispatchable authentication function that enables account abstraction via this function.
     /// Note: it is a private entry function that can only be called directly from transaction.
     AccountAbstractionAddAuthenticationFunction {
@@ -1291,6 +1319,15 @@ impl EntryFunctionCall {
                 cap_update_table,
             ),
             AccountSetOriginatingAddress {} => account_set_originating_address(),
+            AccountUpsertEd25519BackupKeyOnKeylessAccount {
+                keyless_public_key,
+                backup_public_key,
+                backup_key_proof,
+            } => account_upsert_ed25519_backup_key_on_keyless_account(
+                keyless_public_key,
+                backup_public_key,
+                backup_key_proof,
+            ),
             AccountAbstractionAddAuthenticationFunction {
                 module_address,
                 module_name,
@@ -2247,6 +2284,51 @@ pub fn account_set_originating_address() -> TransactionPayload {
     ))
 }
 
+/// Upserts an ED25519 backup key to an account that has a keyless public key as its original public key by converting the account's authentication key
+/// to a multi-key of the original keyless public key and the new backup key that requires 1 signature from either key to authenticate.
+/// This function takes a the account's original keyless public key and a ED25519 backup public key and rotates the account's authentication key to a multi-key of
+/// the original keyless public key and the new backup key that requires 1 signature from either key to authenticate.
+///
+/// Note: This function emits a `KeyRotationToMultiPublicKey` event marking both keys as verified since the keyless public key
+/// is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+///
+/// # Arguments
+/// * `account` - The signer representing the keyless account
+/// * `keyless_public_key` - The original keyless public key of the account (wrapped in an AnyPublicKey)
+/// * `backup_public_key` - The ED25519 public key to add as a backup
+/// * `backup_key_proof` - A signature from the backup key proving ownership
+///
+/// # Aborts
+/// * If the any of inputs deserialize incorrectly
+/// * If the provided public key is not a keyless public key
+/// * If the keyless public key is not the original public key of the account
+/// * If the backup key proof signature is invalid
+///
+/// # Events
+/// * Emits a `KeyRotationToMultiPublicKey` event with the new multi-key configuration
+pub fn account_upsert_ed25519_backup_key_on_keyless_account(
+    keyless_public_key: Vec<u8>,
+    backup_public_key: Vec<u8>,
+    backup_key_proof: Vec<u8>,
+) -> TransactionPayload {
+    TransactionPayload::EntryFunction(EntryFunction::new(
+        ModuleId::new(
+            AccountAddress::new([
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+                0, 0, 0, 1,
+            ]),
+            ident_str!("account").to_owned(),
+        ),
+        ident_str!("upsert_ed25519_backup_key_on_keyless_account").to_owned(),
+        vec![],
+        vec![
+            bcs::to_bytes(&keyless_public_key).unwrap(),
+            bcs::to_bytes(&backup_public_key).unwrap(),
+            bcs::to_bytes(&backup_key_proof).unwrap(),
+        ],
+    ))
+}
+
 /// Add dispatchable authentication function that enables account abstraction via this function.
 /// Note: it is a private entry function that can only be called directly from transaction.
 pub fn account_abstraction_add_authentication_function(
@@ -5534,6 +5616,22 @@ mod decoder {
         }
     }
 
+    pub fn account_upsert_ed25519_backup_key_on_keyless_account(
+        payload: &TransactionPayload,
+    ) -> Option<EntryFunctionCall> {
+        if let TransactionPayload::EntryFunction(script) = payload {
+            Some(
+                EntryFunctionCall::AccountUpsertEd25519BackupKeyOnKeylessAccount {
+                    keyless_public_key: bcs::from_bytes(script.args().get(0)?).ok()?,
+                    backup_public_key: bcs::from_bytes(script.args().get(1)?).ok()?,
+                    backup_key_proof: bcs::from_bytes(script.args().get(2)?).ok()?,
+                },
+            )
+        } else {
+            None
+        }
+    }
+
     pub fn account_abstraction_add_authentication_function(
         payload: &TransactionPayload,
     ) -> Option<EntryFunctionCall> {
@@ -7416,6 +7514,10 @@ static SCRIPT_FUNCTION_DECODER_MAP: once_cell::sync::Lazy<EntryFunctionDecoderMa
             "account_set_originating_address".to_string(),
             Box::new(decoder::account_set_originating_address),
         );
+        map.insert(
+            "account_upsert_ed25519_backup_key_on_keyless_account".to_string(),
+            Box::new(decoder::account_upsert_ed25519_backup_key_on_keyless_account),
+        );
         map.insert(
             "account_abstraction_add_authentication_function".to_string(),
             Box::new(decoder::account_abstraction_add_authentication_function),


Reviews:
user: login: zjma
body: do we want some ut for the new functino?
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T21:13:29Z
user: login: lightmark
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:26:56Z
user: login: heliuchuan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:38:14Z
user: login: lightmark
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:42:23Z
user: login: heliuchuan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T19:59:02Z
user: login: lightmark
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T20:08:10Z
user: login: gregnazario
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T20:52:14Z

Review comments:
diff_hunk: @@ -489,6 +493,84 @@ module aptos_framework::account {
         });
     }
 
+    /// Upserts an ED25519 backup key to a keyless account by converting the account's authentication key to a multi-key.
+    /// This function takes a keyless account (identified by having a keyless public key is the original public key) and updates the
+    /// account's authentication key to a multi-key of the original keyless public key and the new backup key that requires
+    /// 1 signature from either key to authenticate.
+    ///
+    /// Note: This function emits a `KeyRotationToMultiPublicKey` event marking both keys as verified since the keyless public key
+    /// is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+    ///
+    /// # Arguments
+    /// * `account` - The signer representing the keyless account
+    /// * `keyless_public_key` - The original keyless public key of the account (wrapped in an AnyPublicKey)
+    /// * `backup_public_key` - The ED25519 public key to add as a backup
+    /// * `backup_key_proof` - A signature from the backup key proving ownership
+    ///
+    /// # Aborts
+    /// * If the any of inputs deserialize incorrectly
+    /// * If the provided public key is not a keyless public key
+    /// * If the keyless public key is not the original public key of the account
+    /// * If the backup key proof signature is invalid
+    ///
+    /// # Events
+    /// * Emits a `KeyRotationToMultiPublicKey` event with the new multi-key configuration
+    entry fun upsert_ed25519_backup_key_on_keyless_account(account: &signer, keyless_public_key: vector<u8>, backup_public_key: vector<u8>, backup_key_proof: vector<u8>) acquires Account {
+        // Check that the provided public key is a keyless public key
+        let keyless_single_key = single_key::new_public_key_from_bytes(keyless_public_key);
+        assert!(single_key::is_keyless_or_federated_keyless_public_key(&keyless_single_key), error::invalid_argument(ENOT_A_KEYLESS_PUBLIC_KEY));
+
+        let addr = signer::address_of(account);
+        let account_resource = &mut Account[addr];
+        let old_auth_key = account_resource.authentication_key;
+
+        // Check that the provided public key is original public key of the account by comparing
+        // its authentication key to the account address.
+        assert!(
+            bcs::to_bytes(&addr) == keyless_single_key.to_authentication_key(),
+            error::invalid_argument(ENOT_THE_ORIGINAL_PUBLIC_KEY)
+        );
path: aptos-move/framework/aptos-framework/sources/account/account.move
user: login: lightmark
body: why original public key? what if the account has been rotated to a normal account, it's not a keyless account anymore.
created_at: 2025-04-21T19:26:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -489,6 +493,84 @@ module aptos_framework::account {
         });
     }
 
+    /// Upserts an ED25519 backup key to a keyless account by converting the account's authentication key to a multi-key.
+    /// This function takes a keyless account (identified by having a keyless public key is the original public key) and updates the
+    /// account's authentication key to a multi-key of the original keyless public key and the new backup key that requires
+    /// 1 signature from either key to authenticate.
+    ///
+    /// Note: This function emits a `KeyRotationToMultiPublicKey` event marking both keys as verified since the keyless public key
+    /// is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+    ///
+    /// # Arguments
+    /// * `account` - The signer representing the keyless account
+    /// * `keyless_public_key` - The original keyless public key of the account (wrapped in an AnyPublicKey)
+    /// * `backup_public_key` - The ED25519 public key to add as a backup
+    /// * `backup_key_proof` - A signature from the backup key proving ownership
+    ///
+    /// # Aborts
+    /// * If the any of inputs deserialize incorrectly
+    /// * If the provided public key is not a keyless public key
+    /// * If the keyless public key is not the original public key of the account
+    /// * If the backup key proof signature is invalid
+    ///
+    /// # Events
+    /// * Emits a `KeyRotationToMultiPublicKey` event with the new multi-key configuration
+    entry fun upsert_ed25519_backup_key_on_keyless_account(account: &signer, keyless_public_key: vector<u8>, backup_public_key: vector<u8>, backup_key_proof: vector<u8>) acquires Account {
+        // Check that the provided public key is a keyless public key
+        let keyless_single_key = single_key::new_public_key_from_bytes(keyless_public_key);
+        assert!(single_key::is_keyless_or_federated_keyless_public_key(&keyless_single_key), error::invalid_argument(ENOT_A_KEYLESS_PUBLIC_KEY));
+
+        let addr = signer::address_of(account);
+        let account_resource = &mut Account[addr];
+        let old_auth_key = account_resource.authentication_key;
+
+        // Check that the provided public key is original public key of the account by comparing
+        // its authentication key to the account address.
+        assert!(
+            bcs::to_bytes(&addr) == keyless_single_key.to_authentication_key(),
+            error::invalid_argument(ENOT_THE_ORIGINAL_PUBLIC_KEY)
+        );
path: aptos-move/framework/aptos-framework/sources/account/account.move
user: login: heliuchuan
body: Well the function does something very specific.  It rotates a keyless account to a 1 of 2  signer.

If the keyless account has been rotated to a normal account, this function can still be used given that you provide the original public key for the account (which must be keyless).

So the current key doesn't necessarily have to be keyless, but the orginal public key (and the public key that will be used as one of the signers) must be keyless.


created_at: 2025-04-21T19:38:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -489,6 +493,84 @@ module aptos_framework::account {
         });
     }
 
+    /// Upserts an ED25519 backup key to a keyless account by converting the account's authentication key to a multi-key.
+    /// This function takes a keyless account (identified by having a keyless public key is the original public key) and updates the
+    /// account's authentication key to a multi-key of the original keyless public key and the new backup key that requires
+    /// 1 signature from either key to authenticate.
+    ///
+    /// Note: This function emits a `KeyRotationToMultiPublicKey` event marking both keys as verified since the keyless public key
+    /// is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+    ///
+    /// # Arguments
+    /// * `account` - The signer representing the keyless account
+    /// * `keyless_public_key` - The original keyless public key of the account (wrapped in an AnyPublicKey)
+    /// * `backup_public_key` - The ED25519 public key to add as a backup
+    /// * `backup_key_proof` - A signature from the backup key proving ownership
+    ///
+    /// # Aborts
+    /// * If the any of inputs deserialize incorrectly
+    /// * If the provided public key is not a keyless public key
+    /// * If the keyless public key is not the original public key of the account
+    /// * If the backup key proof signature is invalid
+    ///
+    /// # Events
+    /// * Emits a `KeyRotationToMultiPublicKey` event with the new multi-key configuration
+    entry fun upsert_ed25519_backup_key_on_keyless_account(account: &signer, keyless_public_key: vector<u8>, backup_public_key: vector<u8>, backup_key_proof: vector<u8>) acquires Account {
+        // Check that the provided public key is a keyless public key
+        let keyless_single_key = single_key::new_public_key_from_bytes(keyless_public_key);
+        assert!(single_key::is_keyless_or_federated_keyless_public_key(&keyless_single_key), error::invalid_argument(ENOT_A_KEYLESS_PUBLIC_KEY));
+
+        let addr = signer::address_of(account);
+        let account_resource = &mut Account[addr];
+        let old_auth_key = account_resource.authentication_key;
+
+        // Check that the provided public key is original public key of the account by comparing
+        // its authentication key to the account address.
+        assert!(
+            bcs::to_bytes(&addr) == keyless_single_key.to_authentication_key(),
+            error::invalid_argument(ENOT_THE_ORIGINAL_PUBLIC_KEY)
+        );
path: aptos-move/framework/aptos-framework/sources/account/account.move
user: login: lightmark
body: Then it does not match your description saying that "Upserts an ED25519 backup key to a keyless account" since it is not a keyless account anymore.
please refactor this part of the logic to make sure the account is a keyless account. I guess you just need to verify the auth_key and it has to equal to the address. I guess now we don't support an account rotating to keyless account, right?
created_at: 2025-04-21T19:42:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -489,6 +493,84 @@ module aptos_framework::account {
         });
     }
 
+    /// Upserts an ED25519 backup key to a keyless account by converting the account's authentication key to a multi-key.
+    /// This function takes a keyless account (identified by having a keyless public key is the original public key) and updates the
+    /// account's authentication key to a multi-key of the original keyless public key and the new backup key that requires
+    /// 1 signature from either key to authenticate.
+    ///
+    /// Note: This function emits a `KeyRotationToMultiPublicKey` event marking both keys as verified since the keyless public key
+    /// is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+    ///
+    /// # Arguments
+    /// * `account` - The signer representing the keyless account
+    /// * `keyless_public_key` - The original keyless public key of the account (wrapped in an AnyPublicKey)
+    /// * `backup_public_key` - The ED25519 public key to add as a backup
+    /// * `backup_key_proof` - A signature from the backup key proving ownership
+    ///
+    /// # Aborts
+    /// * If the any of inputs deserialize incorrectly
+    /// * If the provided public key is not a keyless public key
+    /// * If the keyless public key is not the original public key of the account
+    /// * If the backup key proof signature is invalid
+    ///
+    /// # Events
+    /// * Emits a `KeyRotationToMultiPublicKey` event with the new multi-key configuration
+    entry fun upsert_ed25519_backup_key_on_keyless_account(account: &signer, keyless_public_key: vector<u8>, backup_public_key: vector<u8>, backup_key_proof: vector<u8>) acquires Account {
+        // Check that the provided public key is a keyless public key
+        let keyless_single_key = single_key::new_public_key_from_bytes(keyless_public_key);
+        assert!(single_key::is_keyless_or_federated_keyless_public_key(&keyless_single_key), error::invalid_argument(ENOT_A_KEYLESS_PUBLIC_KEY));
+
+        let addr = signer::address_of(account);
+        let account_resource = &mut Account[addr];
+        let old_auth_key = account_resource.authentication_key;
+
+        // Check that the provided public key is original public key of the account by comparing
+        // its authentication key to the account address.
+        assert!(
+            bcs::to_bytes(&addr) == keyless_single_key.to_authentication_key(),
+            error::invalid_argument(ENOT_THE_ORIGINAL_PUBLIC_KEY)
+        );
path: aptos-move/framework/aptos-framework/sources/account/account.move
user: login: heliuchuan
body: well i will just update the description to reflect that the requirement that the account is originally a keyless account.  

the current public key does not have to be keyless if it is rotated.

i'd prefer not to name it `upsert_ed25519_backup_key_on_originally_keyless_account`
created_at: 2025-04-21T19:59:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -489,6 +493,84 @@ module aptos_framework::account {
         });
     }
 
+    /// Upserts an ED25519 backup key to a keyless account by converting the account's authentication key to a multi-key.
+    /// This function takes a keyless account (identified by having a keyless public key is the original public key) and updates the
+    /// account's authentication key to a multi-key of the original keyless public key and the new backup key that requires
+    /// 1 signature from either key to authenticate.
+    ///
+    /// Note: This function emits a `KeyRotationToMultiPublicKey` event marking both keys as verified since the keyless public key
+    /// is the original public key of the account and the new backup key has been validated via verifying the challenge signed by the new backup key.
+    ///
+    /// # Arguments
+    /// * `account` - The signer representing the keyless account
+    /// * `keyless_public_key` - The original keyless public key of the account (wrapped in an AnyPublicKey)
+    /// * `backup_public_key` - The ED25519 public key to add as a backup
+    /// * `backup_key_proof` - A signature from the backup key proving ownership
+    ///
+    /// # Aborts
+    /// * If the any of inputs deserialize incorrectly
+    /// * If the provided public key is not a keyless public key
+    /// * If the keyless public key is not the original public key of the account
+    /// * If the backup key proof signature is invalid
+    ///
+    /// # Events
+    /// * Emits a `KeyRotationToMultiPublicKey` event with the new multi-key configuration
+    entry fun upsert_ed25519_backup_key_on_keyless_account(account: &signer, keyless_public_key: vector<u8>, backup_public_key: vector<u8>, backup_key_proof: vector<u8>) acquires Account {
+        // Check that the provided public key is a keyless public key
+        let keyless_single_key = single_key::new_public_key_from_bytes(keyless_public_key);
+        assert!(single_key::is_keyless_or_federated_keyless_public_key(&keyless_single_key), error::invalid_argument(ENOT_A_KEYLESS_PUBLIC_KEY));
+
+        let addr = signer::address_of(account);
+        let account_resource = &mut Account[addr];
+        let old_auth_key = account_resource.authentication_key;
+
+        // Check that the provided public key is original public key of the account by comparing
+        // its authentication key to the account address.
+        assert!(
+            bcs::to_bytes(&addr) == keyless_single_key.to_authentication_key(),
+            error::invalid_argument(ENOT_THE_ORIGINAL_PUBLIC_KEY)
+        );
path: aptos-move/framework/aptos-framework/sources/account/account.move
user: login: lightmark
body: The target use case is very specific and the name is confusing. you only allow an original keyless account to add a backup key even now it is not a keyless account but the name applies only to real keyless account.

I think it only make sense:
either, you only allow real keyless account and abort if the address and auth_key does not match.
or, you make this function more general that it allows any single key account to add another key for backup since your event is also a general one.
created_at: 2025-04-21T20:08:10Z
author_association: CONTRIBUTOR

PR comments:
user: login: trunk-io[bot]
created_at: 2025-04-17T16:54:44Z
body: <!-- Trunk Metrics Report -->



<details>
<summary>
⏱️ 3h 23m total CI duration on this PR
</summary>

| Slowest 15 Jobs | Cumulative Duration | Recent Runs |
|---|---|---|
| [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/14598712024/job/40951946120) | 57m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740514/job/40900191626)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598712024/job/40951946120)&#8201; |
| [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/14598711913/job/40951156769) | 25m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581243512/job/40898237559)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581739968/job/40899716636)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740443/job/40899718116)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711451/job/40951153805)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711913/job/40951156769)&#8201;(+10 more) |
| [rust-targeted-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40954640377) | 23m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40954640377)&#8201; |
| [test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916/job/40951181296) | 22m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531/job/40899731570)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916/job/40951181296)&#8201; |
| [execution-performance / test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14598712024/job/40951159108) | 22m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740514/job/40899719360)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598712024/job/40951159108)&#8201; |
| [rust-cargo-deny](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40951189115) | 20m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14580365547/job/40895575578)&#8201;[⬜](https://github.com/aptos-labs/aptos-core/actions/runs/14581228542/job/40898204579)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581243510/job/40898260962)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740430/job/40899735833)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40951189115)&#8201;(+8 more) |
| [rust-doc-tests](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40951166185) | 7m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40951166185)&#8201; |
| [rust-doc-tests](https://github.com/aptos-labs/aptos-core/actions/runs/14581740430/job/40899723910) | 7m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740430/job/40899723910)&#8201; |
| [general-lints](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40951191005) | 7m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14580365547/job/40895576254)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581228542/job/40898205246)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581243510/job/40898261968)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740430/job/40899736953)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40951191005)&#8201;(+8 more) |
| [semgrep/ci](https://github.com/aptos-labs/aptos-core/actions/runs/14598711920/job/40951156883) | 6m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581243501/job/40898237509)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740426/job/40899718200)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581739961/job/40899716293)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711497/job/40951153851)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711920/job/40951156883)&#8201;(+10 more) |
| [fetch-last-released-docker-image-tag](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916/job/40951181909) | 3m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531/job/40899731873)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916/job/40951181909)&#8201; |
| [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40951165382) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581243510/job/40898248033)&#8201;[⬜](https://github.com/aptos-labs/aptos-core/actions/runs/14581739971/job/40899717759)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740430/job/40899723390)&#8201;[⬜](https://github.com/aptos-labs/aptos-core/actions/runs/14598711487/job/40951155511)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711921/job/40951165382)&#8201;(+10 more) |
| [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/14598711720/job/40951155674) | 43s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581243342/job/40898237022)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740004/job/40899716331)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740378/job/40899718118)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711030/job/40951152634)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711720/job/40951155674)&#8201;(+10 more) |
| [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/14598711787/job/40951156459) | 40s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581243336/job/40898237150)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740355/job/40899718056)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581739956/job/40899716708)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711009/job/40951152849)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711787/job/40951156459)&#8201;(+10 more) |
| [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916/job/40951180120) | 21s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531/job/40899731222)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916/job/40951180120)&#8201; |
</details>

<sub>[settings](https://app.trunk.io/aptoslabs/settings/repo?repo=aptos-labs/aptos-core&utm_source=gh-comment#ci-analytics) ⋅ [feedback](https://form.typeform.com/to/w7Iv34u2?utm_source=gh-comment) ⋅ [docs](https://docs.trunk.io/ci-analytics/pull-request-metrics?utm_source=gh-comment) ⋅ learn more about [trunk.io](https://trunk.io?utm_source=gh-comment)</sub>
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-21T21:46:42Z
body: ### Forge is running suite `framework_upgrade` on `0800002f4bf3f14afd4c0a9c8299e459c7ee351a` ==> `80274fb9bd0e318f3b80ccddfaa3479096cb679b`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16393&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-21T21:46:46Z
body: ### Forge is running suite `realistic_env_max_load` on `80274fb9bd0e318f3b80ccddfaa3479096cb679b`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16393&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-21T21:46:49Z
body: ### Forge is running suite `compat` on `0800002f4bf3f14afd4c0a9c8299e459c7ee351a` ==> `80274fb9bd0e318f3b80ccddfaa3479096cb679b`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16393&var-metrics_source=All&var-chain_name=forge-0&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-21T21:59:09Z
body: ### :white_check_mark: Forge suite `realistic_env_max_load` success on `80274fb9bd0e318f3b80ccddfaa3479096cb679b`
```
two traffics test: inner traffic : committed: 12129.60 txn/s, submitted: 12130.02 txn/s, expired: 0.42 txn/s, latency: 3266.28 ms, (p50: 3000 ms, p70: 3300, p90: 3600 ms, p99: 6000 ms), latency samples: 4611920
two traffics test : committed: 99.99 txn/s, latency: 1544.81 ms, (p50: 1500 ms, p70: 1600, p90: 1800 ms, p99: 2300 ms), latency samples: 1780
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.146, avg: 1.985", "ConsensusProposalToOrdered: max: 0.317, avg: 0.309", "ConsensusOrderedToCommit: max: 0.543, avg: 0.457", "ConsensusProposalToCommit: max: 0.850, avg: 0.766"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.90s no progress at version 26862 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.94s no progress at version 2099478 (avg 0.94s) [limit 16].
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16393&var-metrics_source=All&var-chain_name=forge-1&from=1745272017000&to=1745272747000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1745272017000&end=1745272747000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-21T21%3A46%3A57.220281%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-21T21%3A59%3A07.985079%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745272017&until=1745272747&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745272017&until=1745272747&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-21T22:02:18Z
body: ### :white_check_mark: Forge suite `framework_upgrade` success on `0800002f4bf3f14afd4c0a9c8299e459c7ee351a` ==> `80274fb9bd0e318f3b80ccddfaa3479096cb679b`
```
Compatibility test results for 0800002f4bf3f14afd4c0a9c8299e459c7ee351a ==> 80274fb9bd0e318f3b80ccddfaa3479096cb679b (PR)
Upgrade the nodes to version: 80274fb9bd0e318f3b80ccddfaa3479096cb679b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1142.10 txn/s, submitted: 1145.62 txn/s, failed submission: 3.52 txn/s, expired: 3.52 txn/s, latency: 2641.22 ms, (p50: 1500 ms, p70: 2100, p90: 3700 ms, p99: 12500 ms), latency samples: 103740
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1542.42 txn/s, submitted: 1547.08 txn/s, failed submission: 4.67 txn/s, expired: 4.67 txn/s, latency: 1871.27 ms, (p50: 1500 ms, p70: 1900, p90: 3400 ms, p99: 4500 ms), latency samples: 138761
5. check swarm health
Compatibility test for 0800002f4bf3f14afd4c0a9c8299e459c7ee351a ==> 80274fb9bd0e318f3b80ccddfaa3479096cb679b passed
Upgrade the remaining nodes to version: 80274fb9bd0e318f3b80ccddfaa3479096cb679b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1669.89 txn/s, submitted: 1674.09 txn/s, failed submission: 4.19 txn/s, expired: 4.19 txn/s, latency: 1798.55 ms, (p50: 1600 ms, p70: 2100, p90: 2500 ms, p99: 3100 ms), latency samples: 151280
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16393&var-metrics_source=All&var-chain_name=forge-1&from=1745272008000&to=1745272936000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1745272008000&end=1745272936000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-21T21%3A46%3A48.911102%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-21T22%3A02%3A16.351855%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745272008&until=1745272936&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745272008&until=1745272936&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-21T22:30:35Z
body: ### :x: Forge suite `compat` failure on `0800002f4bf3f14afd4c0a9c8299e459c7ee351a` ==> `80274fb9bd0e318f3b80ccddfaa3479096cb679b`
```
Forge test runner terminated:
Trailing Log Lines:
                 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
      64: std::sys::pal::unix::thread::Thread::new::thread_start
                 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/thread.rs:108:17
      65: start_thread
      66: clone
[2025-04-21T22:29:33Z ERROR aptos_transaction_emitter_lib::emitter] [http://aptos-node-2-validator.forge-compat-pr-16393.svc:8080/v1/] Client cannot catch up to needed timestamp (1745273768), after additional 240s, aborting
[2025-04-21T22:29:33Z WARN  aptos_transaction_emitter_lib::emitter::submission_worker] ["http://aptos-node-2-validator.forge-compat-pr-16393.svc:8080/v1/"] Transactions were not committed before expiration: 20, for [437cc0f422f3bd270f714f905d46f1d7b7fe82353e3e8b53386ab4a1c0fcc699]
[2025-04-21T22:29:49Z ERROR aptos_transaction_emitter_lib::emitter] [http://aptos-node-3-validator.forge-compat-pr-16393.svc:8080/v1/] Client cannot catch up to needed timestamp (1745273768), after additional 240s, aborting
[2025-04-21T22:30:04Z ERROR aptos_transaction_emitter_lib::emitter] [http://aptos-node-2-validator.forge-compat-pr-16393.svc:8080/v1/] Client cannot catch up to needed timestamp (1745273766), after additional 240s, aborting
[2025-04-21T22:30:21Z ERROR aptos_transaction_emitter_lib::emitter] [http://aptos-node-3-validator.forge-compat-pr-16393.svc:8080/v1/] Client cannot catch up to needed timestamp (1745273769), after additional 240s, aborting
Debugging output:

```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16393&var-metrics_source=All&var-chain_name=forge-0&from=1745272028000&to=1745274633000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1745272028000&end=1745274633000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-21T21%3A47%3A08.372631%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-21T22%3A30%3A33.644365%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745272028&until=1745274633&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745272028&until=1745274633&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14581740531)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-22T15:48:06Z
body: ### Forge is running suite `realistic_env_max_load` on `c98b8b1196a1c7211d5da1154c42a51d72467ab1`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16393&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-22T15:48:09Z
body: ### Forge is running suite `framework_upgrade` on `0800002f4bf3f14afd4c0a9c8299e459c7ee351a` ==> `c98b8b1196a1c7211d5da1154c42a51d72467ab1`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16393&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-22T15:48:15Z
body: ### Forge is running suite `compat` on `0800002f4bf3f14afd4c0a9c8299e459c7ee351a` ==> `c98b8b1196a1c7211d5da1154c42a51d72467ab1`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16393&var-metrics_source=All&var-chain_name=forge-0&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-22T15:59:38Z
body: ### :white_check_mark: Forge suite `realistic_env_max_load` success on `c98b8b1196a1c7211d5da1154c42a51d72467ab1`
```
two traffics test: inner traffic : committed: 11489.81 txn/s, latency: 3151.61 ms, (p50: 2700 ms, p70: 3000, p90: 4500 ms, p99: 6700 ms), latency samples: 4368660
two traffics test : committed: 99.98 txn/s, latency: 1559.85 ms, (p50: 1300 ms, p70: 1500, p90: 2200 ms, p99: 3200 ms), latency samples: 1840
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.176, avg: 2.108", "ConsensusProposalToOrdered: max: 0.299, avg: 0.296", "ConsensusOrderedToCommit: max: 0.357, avg: 0.344", "ConsensusProposalToCommit: max: 0.652, avg: 0.640"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.76s no progress at version 20749 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.78s no progress at version 2194258 (avg 0.78s) [limit 16].
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16393&var-metrics_source=All&var-chain_name=forge-1&from=1745336894000&to=1745337576000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1745336894000&end=1745337576000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-22T15%3A48%3A14.206616%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-22T15%3A59%3A36.110695%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745336894&until=1745337576&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745336894&until=1745337576&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-22T16:01:27Z
body: ### :white_check_mark: Forge suite `framework_upgrade` success on `0800002f4bf3f14afd4c0a9c8299e459c7ee351a` ==> `c98b8b1196a1c7211d5da1154c42a51d72467ab1`
```
Compatibility test results for 0800002f4bf3f14afd4c0a9c8299e459c7ee351a ==> c98b8b1196a1c7211d5da1154c42a51d72467ab1 (PR)
Upgrade the nodes to version: c98b8b1196a1c7211d5da1154c42a51d72467ab1
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1884.62 txn/s, submitted: 1892.73 txn/s, failed submission: 8.11 txn/s, expired: 8.11 txn/s, latency: 1520.24 ms, (p50: 1500 ms, p70: 1600, p90: 2000 ms, p99: 2700 ms), latency samples: 171900
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1925.21 txn/s, submitted: 1932.55 txn/s, failed submission: 7.35 txn/s, expired: 7.35 txn/s, latency: 1521.55 ms, (p50: 1500 ms, p70: 1600, p90: 2000 ms, p99: 2700 ms), latency samples: 172961
5. check swarm health
Compatibility test for 0800002f4bf3f14afd4c0a9c8299e459c7ee351a ==> c98b8b1196a1c7211d5da1154c42a51d72467ab1 passed
Upgrade the remaining nodes to version: c98b8b1196a1c7211d5da1154c42a51d72467ab1
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 826.45 txn/s, submitted: 828.88 txn/s, failed submission: 2.43 txn/s, expired: 2.43 txn/s, latency: 3556.88 ms, (p50: 2400 ms, p70: 2700, p90: 11700 ms, p99: 14200 ms), latency samples: 74900
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16393&var-metrics_source=All&var-chain_name=forge-1&from=1745336898000&to=1745337685000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1745336898000&end=1745337685000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-22T15%3A48%3A18.360123%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-22T16%3A01%3A25.724283%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745336898&until=1745337685&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745336898&until=1745337685&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-22T16:17:37Z
body: ### :white_check_mark: Forge suite `compat` success on `0800002f4bf3f14afd4c0a9c8299e459c7ee351a` ==> `c98b8b1196a1c7211d5da1154c42a51d72467ab1`
```
Compatibility test results for 0800002f4bf3f14afd4c0a9c8299e459c7ee351a ==> c98b8b1196a1c7211d5da1154c42a51d72467ab1 (PR)
1. Check liveness of validators at old version: 0800002f4bf3f14afd4c0a9c8299e459c7ee351a
compatibility::simple-validator-upgrade::liveness-check : committed: 9235.45 txn/s, latency: 3458.22 ms, (p50: 3700 ms, p70: 3900, p90: 4200 ms, p99: 4700 ms), latency samples: 319800
2. Upgrading first Validator to new version: c98b8b1196a1c7211d5da1154c42a51d72467ab1
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 2801.08 txn/s, latency: 11879.27 ms, (p50: 13500 ms, p70: 13800, p90: 13900 ms, p99: 14200 ms), latency samples: 105480
3. Upgrading rest of first batch to new version: c98b8b1196a1c7211d5da1154c42a51d72467ab1
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 2934.78 txn/s, latency: 11375.09 ms, (p50: 12800 ms, p70: 13100, p90: 13500 ms, p99: 13500 ms), latency samples: 111900
4. upgrading second batch to new version: c98b8b1196a1c7211d5da1154c42a51d72467ab1
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 4905.00 txn/s, latency: 6867.61 ms, (p50: 7200 ms, p70: 7800, p90: 8400 ms, p99: 8600 ms), latency samples: 175800
5. check swarm health
Compatibility test for 0800002f4bf3f14afd4c0a9c8299e459c7ee351a ==> c98b8b1196a1c7211d5da1154c42a51d72467ab1 passed
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16393&var-metrics_source=All&var-chain_name=forge-0&from=1745336913000&to=1745338656000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16393%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16393%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1745336913000&end=1745338656000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16393%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16393%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-22T15%3A48%3A33.631912%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-22T16%3A17%3A36.113183%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745336913&until=1745338656&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16393&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1745336913&until=1745338656&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16393&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14598711916)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR