Repository: aptos-labs/aptos-core
Pull request #16497: Fix issue #16155
Author: login: junxzm1990
Created: 2025-04-29T17:47:00Z | Merged: 2025-05-02T02:32:55Z

## Description

#16155 reports a recursive structure definition that involves function types and bypasses compiler checks. This is because the existing [recursive struct checker](https://github.com/aptos-labs/aptos-core/blob/jun/bug_16155/third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs#L60) does not check function types nested inside structure definitions.

The fix adds support for function types, as well as other missing types (tuple and reference).

Close #16155

## How Has This Been Tested?
- New [compiler test cases](https://github.com/aptos-labs/aptos-core/blob/jun/bug_16155/third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.move) added
- Existing compile tests and transactional_tests

## Key Areas to Review
- Potential side effects of new supports of other types in [recursive struct checker](https://github.com/aptos-labs/aptos-core/blob/jun/bug_16155/third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs#L60) 

## Type of Change
- [ ] New feature
- [x] Bug fix
- [ ] Breaking change
- [ ] Performance improvement
- [ ] Refactoring
- [ ] Dependency update
- [ ] Documentation update
- [ ] Tests

## Which Components or Systems Does This Change Impact?
- [ ] Validator Node
- [ ] Full Node (API, Indexer, etc.)
- [ ] Move/Aptos Virtual Machine
- [ ] Aptos Framework
- [ ] Aptos CLI/SDK
- [ ] Developer Infrastructure
- [x] Move Compiler
- [ ] Other (specify)

## Checklist
- [x] I have read and followed the [CONTRIBUTING](https://github.com/aptos-labs/aptos-core/blob/main/CONTRIBUTING.md) doc
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I identified and added all stakeholders and component owners affected by this change as reviewers
- [x] I tested both happy and unhappy path of the functionality
- [ ] I have made corresponding changes to the documentation


Linked issue #16155: [Bug] Compiler does not catch recursive type definitions via function types
# 🐛 Bug

Consider the following transactional test:
```
//# publish
module 0xc0ffee::m {
    struct Foo(|Bar|) has copy, drop;
    struct Bar(|Foo|) has copy, drop;
}
```

This results in a bytecode verification error, instead of a nicer compiler error:
```
Error: compilation errors:
 bug: bytecode verification failed with unexpected status code `RECURSIVE_STRUCT_DEFINITION`. This is a compiler bug, consider reporting it.
Error message: none
  ┌─ TEMPFILE:2:1
  │  
2 │ ╭ module 0xc0ffee::m {
3 │ │     struct Foo(|Bar|) has copy, drop;
4 │ │     struct Bar(|Foo|) has copy, drop;
5 │ │ }
  │ ╰─^
```

## Expected Behavior

Compiler error, instead of the above compiler bug message.


Diff:
diff --git a/third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs b/third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs
index 9f4d51838ff..b002cc6ddab 100644
--- a/third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs
+++ b/third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs
@@ -101,7 +101,20 @@ impl<'a> RecursiveStructChecker<'a> {
                         self.report_invalid_field(&struct_env, &field_env);
                     }
                 },
-                Type::Primitive(_) | Type::TypeParameter(_) | Type::Fun(..) => {},
+                Type::Fun(arg_ty, ret_ty, _) => {
+                    if self.ty_contains_struct(path, &arg_ty, loc.clone(), struct_id, checked) {
+                        self.report_invalid_field(&struct_env, &field_env);
+                    }
+                    if self.ty_contains_struct(path, &ret_ty, loc.clone(), struct_id, checked) {
+                        self.report_invalid_field(&struct_env, &field_env);
+                    }
+                },
+                Type::Reference(_, ty) => {
+                    if self.ty_contains_struct(path, &ty, loc.clone(), struct_id, checked) {
+                        self.report_invalid_field(&struct_env, &field_env);
+                    }
+                },
+                Type::Primitive(_) | Type::TypeParameter(_) => {},
                 _ => unreachable!("invalid field type"),
             }
             path.pop();
@@ -195,7 +208,20 @@ impl<'a> RecursiveStructChecker<'a> {
                     .iter()
                     .any(|ty| self.ty_contains_struct(path, ty, loc.clone(), struct_id, checked))
             },
-            Type::Primitive(_) | Type::TypeParameter(_) | Type::Fun(..) => false,
+            Type::Tuple(ty_vec) => {
+                for ty in ty_vec.iter() {
+                    if self.ty_contains_struct(path, ty, loc.clone(), struct_id, checked) {
+                        return true;
+                    }
+                }
+                false
+            },
+            Type::Fun(arg_ty, ret_ty, _abi) => {
+                self.ty_contains_struct(path, arg_ty, loc.clone(), struct_id, checked)
+                    || self.ty_contains_struct(path, ret_ty, loc, struct_id, checked)
+            },
+            Type::Reference(_, ty) => self.ty_contains_struct(path, ty, loc, struct_id, checked),
+            Type::Primitive(_) | Type::TypeParameter(_) => false,
             _ => panic!("ICE: {:?} used as a type parameter", ty),
         }
     }
diff --git a/third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.exp b/third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.exp
new file mode 100644
index 00000000000..595f0cc970c
--- /dev/null
+++ b/third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.exp
@@ -0,0 +1,199 @@
+
+Diagnostics:
+error: cyclic data
+  ┌─ tests/checking/typing/recursive_function_types.move:2:5
+  │
+2 │     struct S(|S|) has copy, drop;
+  │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  │              │
+  │              invalid field `0` of `S` containing `S` itself
+
+error: cyclic data
+  ┌─ tests/checking/typing/recursive_function_types.move:6:5
+  │
+6 │     struct S1(|S2|) has copy, drop;
+  │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  │
+  = field `0` of `S1` contains `S2`
+  = field `0` of `S2` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:11:5
+   │
+11 │     struct S1(|S2|) has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `0` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:17:5
+   │
+17 │     struct S1(|S2|) has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `0` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S4`
+   = field `0` of `S4` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:24:5
+   │
+24 │     struct S1(||S1) has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │               │
+   │               invalid field `0` of `S1` containing `S1` itself
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:28:5
+   │
+28 │     struct S(||S) has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │              │
+   │              invalid field `0` of `S` containing `S` itself
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:32:5
+   │
+32 │     struct S1(|S2|) has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `0` of `S1` contains `S2`
+   = field `0` of `S2` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:37:5
+   │
+37 │     struct S1(|S2|) has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `0` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:43:5
+   │
+43 │     struct S1(|S2|) has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `0` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S4`
+   = field `0` of `S4` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:51:5
+   │
+51 │     struct S1(|S2|) has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `0` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S4`
+   = field `0` of `S4` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:58:5
+   │
+58 │     struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `f` of `S1` contains `S2`
+   = field `0` of `S2` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:63:5
+   │
+63 │     struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `f` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:69:5
+   │
+69 │     struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `f` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S4`
+   = field `0` of `S4` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:76:5
+   │
+76 │     struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `f` of `S1` contains `S2`
+   = field `0` of `S2` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:81:5
+   │
+81 │     struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `f` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:87:5
+   │
+87 │     struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `f` of `S1` contains `S2`
+   = field `0` of `S2` contains `S3`
+   = field `0` of `S3` contains `S4`
+   = field `0` of `S4` contains `S1`, which forms a cycle.
+
+error: cyclic data
+   ┌─ tests/checking/typing/recursive_function_types.move:94:5
+   │
+94 │     struct S1 { x: u64, y: u64, f : S2 } has copy, drop;
+   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   │
+   = field `f` of `S1` contains `S2`
+   = field `0` of `S2` contains `E`
+   = field `s` of `E` contains `S1`, which forms a cycle.
+
+error: cyclic data
+    ┌─ tests/checking/typing/recursive_function_types.move:103:5
+    │
+103 │     struct S1 { x: u64, y: u64, f : S2 } has copy, drop;
+    │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    │
+    = field `f` of `S1` contains `S2`
+    = field `0` of `S2` contains `E`
+    = field `s` of `E` contains `S1`, which forms a cycle.
+
+error: cyclic data
+    ┌─ tests/checking/typing/recursive_function_types.move:112:5
+    │
+112 │     struct S1 { x: u64, y: u64, f : S2 } has copy, drop;
+    │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    │
+    = field `f` of `S1` contains `S2`
+    = field `0` of `S2` contains `S3`
+    = field `0` of `S3` contains `E`
+    = field `s` of `E` contains `S1`, which forms a cycle.
+
+error: cyclic data
+    ┌─ tests/checking/typing/recursive_function_types.move:122:5
+    │
+122 │     struct S1 { x: u64, y: u64, f : S2 } has copy, drop;
+    │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+    │
+    = field `f` of `S1` contains `S2`
+    = field `0` of `S2` contains `S3`
+    = field `0` of `S3` contains `S4`
+    = field `0` of `S4` contains `E`
+    = field `s` of `E` contains `S1`, which forms a cycle.
diff --git a/third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.move b/third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.move
new file mode 100644
index 00000000000..9a451474041
--- /dev/null
+++ b/third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.move
@@ -0,0 +1,130 @@
+module 0xc0ffed::m {
+    struct S(|S|) has copy, drop;
+}
+
+module 0xc0ffee::m1 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S1|) has copy, drop;
+}
+
+module 0xc0ffef::m2 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S1|) has copy, drop;
+}
+
+module 0xc0fff0::m3 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|S1|) has copy, drop;
+}
+
+module 0xc0fff1::m4 {
+    struct S1(||S1) has copy, drop;
+}
+
+module 0xc0fff2::m5 {
+    struct S(||S) has copy, drop;
+}
+
+module 0xc0fff3::m6 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(||S1) has copy, drop;
+}
+
+module 0xc0fff4::m7 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(||S1) has copy, drop;
+}
+
+module 0xc0fff5::m8 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(||S1) has copy, drop;
+}
+
+
+module 0xc0fff6::m9 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(||S1) has copy, drop;
+}
+
+module 0xc0fff7::m10 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S1|) has copy, drop;
+}
+
+module 0xc0fff8::m11 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S1|) has copy, drop;
+}
+
+module 0xc0fff9::m12 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|S1|) has copy, drop;
+}
+
+module 0xc0fff10::m13 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|&S1|) has copy, drop;
+}
+
+module 0xc0fff11::m14 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|&S1|) has copy, drop;
+}
+
+module 0xc0fff12::m15 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|&S1|) has copy, drop;
+}
+
+module 0xc0fff13::m16 {
+    struct S1 { x: u64, y: u64, f : S2 } has copy, drop;
+    enum E {
+        A{s: S1},
+        B{a: u64}
+    } has copy, drop;
+    struct S2(|E|) has copy, drop;
+}
+
+module 0xc0fff14::m17 {
+    struct S1 { x: u64, y: u64, f : S2 } has copy, drop;
+    enum E {
+        A{s: S1},
+        B{a: u64}
+    } has copy, drop;
+    struct S2(|&E|) has copy, drop;
+}
+
+module 0xc0fff15::m18 {
+    struct S1 { x: u64, y: u64, f : S2 } has copy, drop;
+    enum E {
+        A{s: S1},
+        B{a: u64}
+    } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|&E|) has copy, drop;
+}
+
+module 0xc0fff16::m19 {
+    struct S1 { x: u64, y: u64, f : S2 } has copy, drop;
+    enum E {
+        A{s: S1},
+        B{a: u64}
+    } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|&E|) has copy, drop;
+}


Reviews:
user: login: vineethk
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T14:56:39Z
user: login: junxzm1990
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:13:37Z
user: login: junxzm1990
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:14:34Z
user: login: junxzm1990
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T16:14:46Z
user: login: rahxephon89
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T22:45:53Z
user: login: vineethk
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T22:57:27Z

Review comments:
diff_hunk: @@ -101,7 +101,27 @@ impl<'a> RecursiveStructChecker<'a> {
                         self.report_invalid_field(&struct_env, &field_env);
                     }
                 },
-                Type::Primitive(_) | Type::TypeParameter(_) | Type::Fun(..) => {},
+                Type::Tuple(ty_vec) => {
path: third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs
user: login: vineethk
body: Tuple types cannot appear in structs right now, so this changes from `unreachable` in the previous code to having this default implementation here.

We may want to still keep this as unreachable, or at least add a clear note saying this is not currently reachable.
created_at: 2025-05-01T14:47:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -195,7 +214,21 @@ impl<'a> RecursiveStructChecker<'a> {
                     .iter()
                     .any(|ty| self.ty_contains_struct(path, ty, loc.clone(), struct_id, checked))
             },
-            Type::Primitive(_) | Type::TypeParameter(_) | Type::Fun(..) => false,
+            Type::Vector(ty) => self.ty_contains_struct(path, ty, loc, struct_id, checked),
path: third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs
user: login: vineethk
body: Curious: why was this code moved down here?
created_at: 2025-05-01T14:50:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,91 @@
+module 0xc0ffed::m {
+    struct S(|S|) has copy, drop;
+}
+
+module 0xc0ffee::m1 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S1|) has copy, drop;
+}
+
+module 0xc0ffef::m2 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S1|) has copy, drop;
+}
+
+module 0xc0fff0::m3 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|S1|) has copy, drop;
+}
+
+module 0xc0fff1::m4 {
+    struct S1(||S1) has copy, drop;
+}
+
+module 0xc0fff2::m5 {
+    struct S(||S) has copy, drop;
+}
+
+module 0xc0fff3::m6 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(||S1) has copy, drop;
+}
+
+module 0xc0fff4::m7 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(||S1) has copy, drop;
+}
+
+module 0xc0fff5::m8 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(||S1) has copy, drop;
+}
+
+
+module 0xc0fff6::m9 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(||S1) has copy, drop;
+}
+
+module 0xc0fff7::m10 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S1|) has copy, drop;
+}
+
+module 0xc0fff8::m11 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S1|) has copy, drop;
+}
+
+module 0xc0fff9::m12 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|S1|) has copy, drop;
+}
+
+module 0xc0fff10::m13 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|&S1|) has copy, drop;
+}
+
+module 0xc0fff11::m14 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|&S1|) has copy, drop;
+}
+
+module 0xc0fff12::m15 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|&S1|) has copy, drop;
+}
path: third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.move
user: login: vineethk
body: We could perhaps add a test with cycle formation through enums, references, and functions types.
created_at: 2025-05-01T14:56:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -101,7 +101,27 @@ impl<'a> RecursiveStructChecker<'a> {
                         self.report_invalid_field(&struct_env, &field_env);
                     }
                 },
-                Type::Primitive(_) | Type::TypeParameter(_) | Type::Fun(..) => {},
+                Type::Tuple(ty_vec) => {
path: third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs
user: login: junxzm1990
body: I am not aware of that. Will change accordingly.
created_at: 2025-05-01T16:13:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -195,7 +214,21 @@ impl<'a> RecursiveStructChecker<'a> {
                     .iter()
                     .any(|ty| self.ty_contains_struct(path, ty, loc.clone(), struct_id, checked))
             },
-            Type::Primitive(_) | Type::TypeParameter(_) | Type::Fun(..) => false,
+            Type::Vector(ty) => self.ty_contains_struct(path, ty, loc, struct_id, checked),
path: third_party/move/move-compiler-v2/src/env_pipeline/recursive_struct_checker.rs
user: login: junxzm1990
body: I believe it is unintentional.
created_at: 2025-05-01T16:14:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,91 @@
+module 0xc0ffed::m {
+    struct S(|S|) has copy, drop;
+}
+
+module 0xc0ffee::m1 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S1|) has copy, drop;
+}
+
+module 0xc0ffef::m2 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S1|) has copy, drop;
+}
+
+module 0xc0fff0::m3 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|S1|) has copy, drop;
+}
+
+module 0xc0fff1::m4 {
+    struct S1(||S1) has copy, drop;
+}
+
+module 0xc0fff2::m5 {
+    struct S(||S) has copy, drop;
+}
+
+module 0xc0fff3::m6 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(||S1) has copy, drop;
+}
+
+module 0xc0fff4::m7 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(||S1) has copy, drop;
+}
+
+module 0xc0fff5::m8 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(||S1) has copy, drop;
+}
+
+
+module 0xc0fff6::m9 {
+    struct S1(|S2|) has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(||S1) has copy, drop;
+}
+
+module 0xc0fff7::m10 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S1|) has copy, drop;
+}
+
+module 0xc0fff8::m11 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S1|) has copy, drop;
+}
+
+module 0xc0fff9::m12 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|S1|) has copy, drop;
+}
+
+module 0xc0fff10::m13 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|&S1|) has copy, drop;
+}
+
+module 0xc0fff11::m14 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|&S1|) has copy, drop;
+}
+
+module 0xc0fff12::m15 {
+    struct S1  { x: u64, y: u64, f : S2 } has copy, drop;
+    struct S2(|S3|) has copy, drop;
+    struct S3(|S4|) has copy, drop;
+    struct S4(|&S1|) has copy, drop;
+}
path: third_party/move/move-compiler-v2/tests/checking/typing/recursive_function_types.move
user: login: junxzm1990
body: Will add.
created_at: 2025-05-01T16:14:46Z
author_association: CONTRIBUTOR

PR comments:
user: login: trunk-io[bot]
created_at: 2025-04-29T17:47:03Z
body: <!-- Trunk Metrics Report -->



<details>
<summary>
⏱️ 1h 21m total CI duration on this PR
</summary>

| Job | Cumulative Duration | Recent Runs |
|---|---|---|
| [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/14786995458/job/41517208280) | 27m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995458/job/41517208280)&#8201; |
| [rust-cargo-deny](https://github.com/aptos-labs/aptos-core/actions/runs/14786995446/job/41517081170) | 14m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740362539/job/41376663356)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740439855/job/41376919832)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740704517/job/41377756852)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14784853785/job/41511358632)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995446/job/41517081170)&#8201;(+3 more) |
| [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/14786995450/job/41517071486) | 13m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740362522/job/41376630426)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740439838/job/41376888819)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740704526/job/41377723342)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14784853791/job/41511349410)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995450/job/41517071486)&#8201;(+3 more) |
| [rust-doc-tests](https://github.com/aptos-labs/aptos-core/actions/runs/14786995446/job/41517072181) | 7m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995446/job/41517072181)&#8201; |
| [execution-performance / test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14786995458/job/41517071957) | 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995458/job/41517071957)&#8201; |
| [test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517077869) | 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517077869)&#8201; |
| [general-lints](https://github.com/aptos-labs/aptos-core/actions/runs/14786995446/job/41517081710) | 4m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740362539/job/41376664188)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740439855/job/41376920130)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740704517/job/41377757257)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14784853785/job/41511359047)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995446/job/41517081710)&#8201;(+3 more) |
| [semgrep/ci](https://github.com/aptos-labs/aptos-core/actions/runs/14786995488/job/41517071661) | 3m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740362536/job/41376631026)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740439862/job/41376889155)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740704519/job/41377723150)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14784853792/job/41511349675)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995488/job/41517071661)&#8201;(+3 more) |
| [fetch-last-released-docker-image-tag](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517078041) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517078041)&#8201; |
| [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14786995446/job/41517071991) | 1m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740362539/job/41376648417)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740439855/job/41376905824)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740704517/job/41377740150)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14784853785/job/41511350007)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995446/job/41517071991)&#8201;(+3 more) |
| [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/14786995486/job/41517071789) | 22s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740234705/job/41376201661)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740362405/job/41376630446)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740704403/job/41377722859)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14784853599/job/41511348666)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995486/job/41517071789)&#8201;(+2 more) |
| [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/14786995482/job/41517071680) | 22s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740251302/job/41376257221)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740362395/job/41376630265)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14740704391/job/41377722762)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14784853597/job/41511348612)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995482/job/41517071680)&#8201;(+3 more) |
| [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517077496) | 13s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517077496)&#8201; |
| [permission-check](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517072313) | 3s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517072313)&#8201; |
| [determine-docker-build-metadata](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517077681) | 2s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549/job/41517077681)&#8201; |
</details>

🚨 1 job on the last run was significantly faster/slower than expected

| Job | Duration | vs 7d avg | Delta |
|---|---|---|---|
| [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/14786995458/job/41517208280) | 27m | [19m](https://app.trunk.io/aptoslabs/ci-analytics/1_00384b63-ffe4-46c6-9b0b-1034fd44569d/1_4cfc13c8-5fda-4429-b262-c2119596dfd3?repo=aptos-labs/aptos-core&timeBased=true&numericAggregateFunctions=NUMERIC_AGGREGATE_FUNCTION_AVG&timeRange=7+days&conclusion=GIT_HUB_REPO_ACTIVITY_RUN_CONCLUSION_SUCCESS&utm_source=gh-comment) | [![+44%](https://raster.shields.io/badge/+44%25-C84B25)](https://app.trunk.io/aptoslabs/ci-analytics/1_00384b63-ffe4-46c6-9b0b-1034fd44569d/1_4cfc13c8-5fda-4429-b262-c2119596dfd3?repo=aptos-labs/aptos-core&timeBased=true&numericAggregateFunctions=NUMERIC_AGGREGATE_FUNCTION_AVG&timeRange=7+days&conclusion=GIT_HUB_REPO_ACTIVITY_RUN_CONCLUSION_SUCCESS&utm_source=gh-comment) |

<sub>[settings](https://app.trunk.io/aptoslabs/settings/repo?repo=aptos-labs/aptos-core&utm_source=gh-comment#ci-analytics) ⋅ [feedback](https://form.typeform.com/to/w7Iv34u2?utm_source=gh-comment) ⋅ [docs](https://docs.trunk.io/ci-analytics/pull-request-metrics?utm_source=gh-comment) ⋅ learn more about [trunk.io](https://trunk.io?utm_source=gh-comment)</sub>
author_association: NONE
user: login: junxzm1990
created_at: 2025-05-01T22:44:52Z
body: @vineethk all comments addressed. Please re-review and approve.

@rahxephon89 and @JakeSilverman , this is a simple one. So when getting some cycles, could you take a look so that we can close the issue?
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-02T02:01:11Z
body: ### Forge is running suite `compat` on `16808c72122fa780cda3de5447f7d5393dd29f72` ==> `87ca806b1677492d2ab3f26c4890d4327e84956d`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16497&var-metrics_source=All&var-chain_name=forge-0&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16497%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16497%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16497%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16497%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16497&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16497&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-02T02:01:11Z
body: ### Forge is running suite `framework_upgrade` on `16808c72122fa780cda3de5447f7d5393dd29f72` ==> `87ca806b1677492d2ab3f26c4890d4327e84956d`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16497&var-metrics_source=All&var-chain_name=forge-0&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16497%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16497%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16497%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16497%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16497&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16497&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-02T02:01:13Z
body: ### Forge is running suite `realistic_env_max_load` on `87ca806b1677492d2ab3f26c4890d4327e84956d`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16497&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16497%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16497%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16497%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16497%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16497&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16497&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-02T02:12:59Z
body: ### :white_check_mark: Forge suite `compat` success on `16808c72122fa780cda3de5447f7d5393dd29f72` ==> `87ca806b1677492d2ab3f26c4890d4327e84956d`
```
Compatibility test results for 16808c72122fa780cda3de5447f7d5393dd29f72 ==> 87ca806b1677492d2ab3f26c4890d4327e84956d (PR)
1. Check liveness of validators at old version: 16808c72122fa780cda3de5447f7d5393dd29f72
compatibility::simple-validator-upgrade::liveness-check : committed: 9170.30 txn/s, latency: 3584.79 ms, (p50: 3800 ms, p70: 4000, p90: 4300 ms, p99: 4700 ms), latency samples: 309220
2. Upgrading first Validator to new version: 87ca806b1677492d2ab3f26c4890d4327e84956d
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 2439.66 txn/s, latency: 13810.03 ms, (p50: 15000 ms, p70: 15800, p90: 16200 ms, p99: 16400 ms), latency samples: 91660
3. Upgrading rest of first batch to new version: 87ca806b1677492d2ab3f26c4890d4327e84956d
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 2594.85 txn/s, latency: 12811.02 ms, (p50: 14300 ms, p70: 14600, p90: 14800 ms, p99: 15000 ms), latency samples: 97860
4. upgrading second batch to new version: 87ca806b1677492d2ab3f26c4890d4327e84956d
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 4592.62 txn/s, latency: 7431.73 ms, (p50: 7500 ms, p70: 8600, p90: 9500 ms, p99: 9700 ms), latency samples: 161680
5. check swarm health
Compatibility test for 16808c72122fa780cda3de5447f7d5393dd29f72 ==> 87ca806b1677492d2ab3f26c4890d4327e84956d passed
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16497&var-metrics_source=All&var-chain_name=forge-0&from=1746151278000&to=1746151978000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16497%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16497%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1746151278000&end=1746151978000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16497%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16497%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-02T02%3A01%3A18.611098%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-02T02%3A12%3A58.505318%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1746151278&until=1746151978&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16497&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1746151278&until=1746151978&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16497&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-02T02:13:19Z
body: ### :white_check_mark: Forge suite `realistic_env_max_load` success on `87ca806b1677492d2ab3f26c4890d4327e84956d`
```
Forge report malformed: Expecting property name enclosed in double quotes: line 20 column 1 (char 451)
'{\n  "metrics": [\n    {\n      "test_name": "two traffics test: inner traffic",\n      "metric": "submitted_txn",\n      "value": 4415780.0\n    },\n    {\n      "test_name": "two traffics test: inner traffic",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "two traffics test: inner traffic",\n      "metric": "avg_tps",\n      "value": 11613.727449550546\n    },\n    {\n      "test_name": "two traffics test: inner traffic",\n[2025-05-02T02:13:15Z INFO  aptos_forge::report] Test Ok\n      "metric": "avg_latency",\n      "value": 3133.2352139372883\n    },\n    {\n      "test_name": "two traffics test: inner traffic",\n      "metric": "p50_latency",\n      "value": 2700.0\n    },\n    {\n      "test_name": "two traffics test: inner traffic",\n      "metric": "p90_latency",\n      "value": 4600.0\n    },\n    {\n      "test_name": "two traffics test: inner traffic",\n      "metric": "p99_latency",\n      "value": 6200.0\n    },\n    {\n      "test_name": "two traffics test",\n      "metric": "submitted_txn",\n      "value": 44260.0\n    },\n    {\n      "test_name": "two traffics test",\n      "metric": "expired_txn",\n      "value": 0.0\n    },\n    {\n      "test_name": "two traffics test",\n      "metric": "avg_tps",\n      "value": 99.98194844963847\n    },\n    {\n      "test_name": "two traffics test",\n      "metric": "avg_latency",\n      "value": 1911.4505813953488\n    },\n    {\n      "test_name": "two traffics test",\n      "metric": "p50_latency",\n      "value": 1400.0\n    },\n    {\n      "test_name": "two traffics test",\n      "metric": "p90_latency",\n      "value": 3200.0\n    },\n    {\n      "test_name": "two traffics test",\n      "metric": "p99_latency",\n      "value": 6100.0\n    }\n  ],\n  "text": "two traffics test: inner traffic : committed: 11613.73 txn/s, latency: 3133.24 ms, (p50: 2700 ms, p70: 3400, p90: 4600 ms, p99: 6200 ms), latency samples: 4415780\\ntwo traffics test : committed: 99.98 txn/s, latency: 1911.45 ms, (p50: 1400 ms, p70: 1900, p90: 3200 ms, p99: 6100 ms), latency samples: 1720\\nLatency breakdown for phase 0: [\\"MempoolToBlockCreation: max: 1.806, avg: 1.711\\", \\"ConsensusProposalToOrdered: max: 0.298, avg: 0.296\\", \\"ConsensusOrderedToCommit: max: 0.341, avg: 0.333\\", \\"ConsensusProposalToCommit: max: 0.638, avg: 0.629\\"]\\nMax non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.05s no progress at version 25268 (avg 0.20s) [limit 15].\\nMax epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.82s no progress at version 2372226 (avg 0.82s) [limit 16].\\nTest Ok"\n}'
Trailing Log Lines:
networkchaos.chaos-mesh.org "gcp--us-central1-to-aws--sa-east-1-netem" deleted
test CompositeNetworkTest ... ok
Test Statistics: 
two traffics test: inner traffic : committed: 11613.73 txn/s, latency: 3133.24 ms, (p50: 2700 ms, p70: 3400, p90: 4600 ms, p99: 6200 ms), latency samples: 4415780
two traffics test : committed: 99.98 txn/s, latency: 1911.45 ms, (p50: 1400 ms, p70: 1900, p90: 3200 ms, p99: 6100 ms), latency samples: 1720
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.806, avg: 1.711", "ConsensusProposalToOrdered: max: 0.298, avg: 0.296", "ConsensusOrderedToCommit: max: 0.341, avg: 0.333", "ConsensusProposalToCommit: max: 0.638, avg: 0.629"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.05s no progress at version 25268 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.82s no progress at version 2372226 (avg 0.82s) [limit 16].
Test Ok

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="0" errors="0" uuid="77ccdb5d-bfe1-4db9-a3d3-25486ea84269">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="0">
        <testcase name="CompositeNetworkTest(network:multi-region-network-emulation(two traffics test)) with ">
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===
[2025-05-02T02:13:15Z INFO  aptos_forge::backend::k8s::cluster_helper] Deleting namespace forge-e2e-pr-16497: Some(NamespaceStatus { conditions: None, phase: Some("Terminating") })
[2025-05-02T02:13:15Z INFO  aptos_forge::backend::k8s::cluster_helper] aptos-node resources for Forge removed in namespace: forge-e2e-pr-16497

test result: ok. 1 passed; 0 failed; 0 filtered out

Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge193-0       1/1     Running     0          10m
aptos-node-0-validator-0                1/1     Running     0          10m
aptos-node-1-fullnode-eforge193-0       1/1     Running     0          10m
aptos-node-1-validator-0                1/1     Running     0          10m
aptos-node-2-fullnode-eforge193-0       1/1     Running     0          10m
aptos-node-2-validator-0                1/1     Running     0          10m
aptos-node-3-fullnode-eforge193-0       1/1     Running     0          10m
aptos-node-3-validator-0                1/1     Running     0          10m
aptos-node-4-fullnode-eforge193-0       1/1     Running     0          10m
aptos-node-4-validator-0                1/1     Running     0          10m
aptos-node-5-validator-0                1/1     Running     0          10m
aptos-node-6-validator-0                1/1     Running     0          10m
forge-testnet-deployer-lvr4h            0/1     Completed   0          11m
genesis-aptos-genesis-eforge193-dddpb   0/1     Completed   0          11m

```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16497&var-metrics_source=All&var-chain_name=forge-1&from=1746151279000&to=1746151998000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16497%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16497%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1746151279000&end=1746151998000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16497%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16497%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-02T02%3A01%3A19.776342%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-02T02%3A13%3A18.313274%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1746151279&until=1746151998&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16497&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1746151279&until=1746151998&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16497&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-02T02:14:48Z
body: ### :white_check_mark: Forge suite `framework_upgrade` success on `16808c72122fa780cda3de5447f7d5393dd29f72` ==> `87ca806b1677492d2ab3f26c4890d4327e84956d`
```
Compatibility test results for 16808c72122fa780cda3de5447f7d5393dd29f72 ==> 87ca806b1677492d2ab3f26c4890d4327e84956d (PR)
Upgrade the nodes to version: 87ca806b1677492d2ab3f26c4890d4327e84956d
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1573.40 txn/s, submitted: 1579.37 txn/s, failed submission: 5.97 txn/s, expired: 5.97 txn/s, latency: 1808.03 ms, (p50: 1500 ms, p70: 1800, p90: 2800 ms, p99: 4800 ms), latency samples: 142360
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1815.39 txn/s, submitted: 1822.05 txn/s, failed submission: 6.66 txn/s, expired: 6.66 txn/s, latency: 1634.08 ms, (p50: 1500 ms, p70: 1800, p90: 2100 ms, p99: 3500 ms), latency samples: 163564
5. check swarm health
Compatibility test for 16808c72122fa780cda3de5447f7d5393dd29f72 ==> 87ca806b1677492d2ab3f26c4890d4327e84956d passed
Upgrade the remaining nodes to version: 87ca806b1677492d2ab3f26c4890d4327e84956d
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1784.13 txn/s, submitted: 1791.86 txn/s, failed submission: 7.73 txn/s, expired: 7.73 txn/s, latency: 1623.25 ms, (p50: 1500 ms, p70: 1800, p90: 2400 ms, p99: 3300 ms), latency samples: 161543
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16497&var-metrics_source=All&var-chain_name=forge-0&from=1746151278000&to=1746152087000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16497%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16497%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1746151278000&end=1746152087000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16497%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16497%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-05-02T02%3A01%3A18.477358%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-05-02T02%3A14%3A47.043170%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1746151278&until=1746152087&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16497&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1746151278&until=1746152087&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16497&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14786995549)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR