Repository: aptos-labs/aptos-core
Pull request #16366: [compiler] Fix cyclic substitution bug
Author: login: wrwg
Created: 2025-04-15T05:59:44Z | Merged: 2025-04-15T16:42:14Z

## Description

The `Substitution::bind` function could create a cyclic substitution by evaluating constraints after the occurrs check and before the variable was assigned, causing stackoverflow crashes. Kind of a reentrancy problem.

This moves the occurs check after constraint evaluation. As we are specializing types before assignment since a while anyway, the occurs check logic could be simplified.

Closes #16336

## How Has This Been Tested?

New test

## Type of Change
- [ ] New feature
- [x] Bug fix
- [ ] Breaking change
- [ ] Performance improvement
- [ ] Refactoring
- [ ] Dependency update
- [ ] Documentation update
- [ ] Tests

## Which Components or Systems Does This Change Impact?
- [ ] Validator Node
- [ ] Full Node (API, Indexer, etc.)
- [ ] Move/Aptos Virtual Machine
- [ ] Aptos Framework
- [ ] Aptos CLI/SDK
- [ ] Developer Infrastructure
- [x] Move Compiler
- [ ] Other (specify)



Linked issue #16336: [Bug] Assigning function return value to itself causes compiler stack overflow
# 🐛 Bug

```
module 0x42::Test {
    fun f() {
        let x = || {||{}};
        *(&mut x) = x();
    }
}
```

Compiling the above with `aptos move compile --language-version 2.2` will cause stack overflow for the compiler:

```
Compiling, may take a little while to download git dependencies...
BUILDING test

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
[1]    1557755 IOT instruction (core dumped)  aptos move compile
```

Diff:
diff --git a/third_party/move/move-compiler-v2/tests/checking-lang-v2.2/lambda/bug_16336.exp b/third_party/move/move-compiler-v2/tests/checking-lang-v2.2/lambda/bug_16336.exp
new file mode 100644
index 00000000000..40a3f1630f0
--- /dev/null
+++ b/third_party/move/move-compiler-v2/tests/checking-lang-v2.2/lambda/bug_16336.exp
@@ -0,0 +1,7 @@
+
+Diagnostics:
+error: function returns value of type `()` but `|()|` was expected
+  ┌─ tests/checking-lang-v2.2/lambda/bug_16336.move:4:10
+  │
+4 │         *(&mut x) = x();
+  │          ^^^^^^^^
diff --git a/third_party/move/move-compiler-v2/tests/checking-lang-v2.2/lambda/bug_16336.move b/third_party/move/move-compiler-v2/tests/checking-lang-v2.2/lambda/bug_16336.move
new file mode 100644
index 00000000000..64041050bfe
--- /dev/null
+++ b/third_party/move/move-compiler-v2/tests/checking-lang-v2.2/lambda/bug_16336.move
@@ -0,0 +1,6 @@
+module 0x42::Test {
+    fun f() {
+        let x : ||(||) = || {||{}};
+        *(&mut x) = x();
+    }
+}
diff --git a/third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.exp b/third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.exp
index f9ded10b880..0124e9470f1 100644
--- a/third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.exp
+++ b/third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.exp
@@ -3,5 +3,5 @@ Diagnostics:
 error: unable to infer type due to cyclic type constraints (try annotating the type)
   ┌─ tests/checking/typing/recursive_local.move:5:9
   │
-5 │         x = (x, 0);
+5 │         x = vector[x];
   │         ^
diff --git a/third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.move b/third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.move
index 59b08300bf2..2542c26f1c2 100644
--- a/third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.move
+++ b/third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.move
@@ -2,7 +2,7 @@ address 0x42 {
 module M {
     fun t() {
         let x;
-        x = (x, 0);
+        x = vector[x];
     }
 }
 }
diff --git a/third_party/move/move-model/src/ty.rs b/third_party/move/move-model/src/ty.rs
index a23d17a0a3f..a6d8b210e79 100644
--- a/third_party/move/move-model/src/ty.rs
+++ b/third_party/move/move-model/src/ty.rs
@@ -1846,7 +1846,7 @@ impl Substitution {
         ty: Type,
     ) -> Result<(), TypeUnificationError> {
         // Specialize the type before binding, to maximize groundness of type terms.
-        let ty = self.specialize(&ty);
+        let mut ty = self.specialize(&ty);
         if let Some(mut constrs) = self.constraints.remove(&var) {
             // Sort constraints to report primary errors first
             constrs.sort_by(|(_, _, c1), (_, _, c2)| c1.compare(c2).reverse());
@@ -1874,9 +1874,17 @@ impl Substitution {
                     },
                 }
             }
+            // New bindings could have been created, so specialize again.
+            ty = self.specialize(&ty)
+        }
+
+        // Occurs check
+        if ty.get_vars().contains(&var) {
+            Err(TypeUnificationError::CyclicSubstitution(Type::Var(var), ty))
+        } else {
+            self.subs.insert(var, ty);
+            Ok(())
         }
-        self.subs.insert(var, ty);
-        Ok(())
     }
 
     /// Evaluates whether the given type satisfies the constraint, discharging the constraint.
@@ -1906,7 +1914,7 @@ impl Substitution {
         if matches!(ty, Type::Error) {
             Ok(())
         } else if let Type::Var(other_var) = ty {
-            // Transfer constraint on to other variable, which we assert to be free
+            // Transfer constraint on to other variable which we assert to be free
             debug_assert!(!self.subs.contains_key(other_var));
             self.add_constraint(context, *other_var, loc.clone(), order, c, ctx_opt)
         } else if c.propagate_over_reference() && ty.is_reference() {
@@ -2568,39 +2576,17 @@ impl Substitution {
                     break;
                 }
             }
-            // Skip the cycle check if we are unifying the same two variables.
+            // Skip binding if we are unifying the same two variables.
             if t1 == &t2 {
-                return Ok(Some(t1.clone()));
-            }
-            // Cycle check.
-            if !self.occurs_check(&t2, *v1) {
+                Ok(Some(t1.clone()))
+            } else {
                 self.bind(context, *v1, variance, order, t2.clone())?;
                 Ok(Some(t2))
-            } else {
-                Err(TypeUnificationError::CyclicSubstitution(
-                    self.specialize(t1),
-                    self.specialize(&t2),
-                ))
             }
         } else {
             Ok(None)
         }
     }
-
-    /// Check whether the variables occurs in the type, or in any assignment to variables in the
-    /// type.
-    fn occurs_check(&self, ty: &Type, var: u32) -> bool {
-        ty.get_vars().iter().any(|v| {
-            if v == &var {
-                return true;
-            }
-            if let Some(sty) = self.subs.get(v) {
-                self.occurs_check(sty, var)
-            } else {
-                false
-            }
-        })
-    }
 }
 
 /// A context which determines how type unification errors are presented


Reviews:
user: login: vineethk
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T14:57:03Z
user: login: rahxephon89
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:01:27Z
user: login: wrwg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:08:12Z

Review comments:
diff_hunk: @@ -3,5 +3,5 @@ Diagnostics:
 error: unable to infer type due to cyclic type constraints (try annotating the type)
   ┌─ tests/checking/typing/recursive_local.move:5:9
   │
-5 │         x = (x, 0);
+5 │         x = vector[x];
path: third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.exp
user: login: vineethk
body: Why the change in the test case instead of adding it as a new one?
created_at: 2025-04-15T14:56:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3,5 +3,5 @@ Diagnostics:
 error: unable to infer type due to cyclic type constraints (try annotating the type)
   ┌─ tests/checking/typing/recursive_local.move:5:9
   │
-5 │         x = (x, 0);
+5 │         x = vector[x];
path: third_party/move/move-compiler-v2/tests/checking/typing/recursive_local.exp
user: login: wrwg
body: The old one doesn't work anymore since we fixed a bug with tuple assignment and `x = (x, 0)` creates a different error then the cycle error we want to see here.
created_at: 2025-04-15T15:08:12Z
author_association: CONTRIBUTOR

PR comments:
user: login: trunk-io[bot]
created_at: 2025-04-15T05:59:48Z
body: <!-- Trunk Metrics Report -->



<details>
<summary>
⏱️ 2h 7m total CI duration on this PR
</summary>

| Slowest 15 Jobs | Cumulative Duration | Recent Runs |
|---|---|---|
| [execution-performance / single-node-performance](https://github.com/aptos-labs/aptos-core/actions/runs/14473874918/job/40594879013) | 25m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473874918/job/40594879013)&#8201; |
| [rust-targeted-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/14472824353/job/40591005247) | 22m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14472824353/job/40591005247)&#8201; |
| [rust-targeted-unit-tests](https://github.com/aptos-labs/aptos-core/actions/runs/14463342099/job/40589298134) | 21m | [⬜](https://github.com/aptos-labs/aptos-core/actions/runs/14463342099/job/40589298134)&#8201; |
| [forge-compat-test / forge](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007/job/40596750789) | 15m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007/job/40596750789)&#8201; |
| [check-dynamic-deps](https://github.com/aptos-labs/aptos-core/actions/runs/14473874961/job/40594552341) | 9m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14462358975/job/40557166640)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14463342115/job/40560012288)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14472824600/job/40590950360)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473874961/job/40594552341)&#8201; |
| [rust-cargo-deny](https://github.com/aptos-labs/aptos-core/actions/runs/14473874933/job/40594577012) | 7m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14462358959/job/40557177505)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14463342099/job/40560029674)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14472824353/job/40590998772)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473874933/job/40594577012)&#8201; |
| [rust-doc-tests](https://github.com/aptos-labs/aptos-core/actions/runs/14473874933/job/40594555572) | 6m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473874933/job/40594555572)&#8201; |
| [test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007/job/40594569012) | 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007/job/40594569012)&#8201; |
| [execution-performance / test-target-determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14473874918/job/40594553655) | 5m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473874918/job/40594553655)&#8201; |
| [rust-lints](https://github.com/aptos-labs/aptos-core/actions/runs/14472824353/job/40591000125) | 4m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14472824353/job/40591000125)&#8201; |
| [rust-check-merge-base](https://github.com/aptos-labs/aptos-core/actions/runs/14472824353/job/40591001739) | 3m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14472824353/job/40591001739)&#8201; |
| [fetch-last-released-docker-image-tag](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007/job/40594571190) | 2m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007/job/40594571190)&#8201; |
| [general-lints](https://github.com/aptos-labs/aptos-core/actions/runs/14473874933/job/40594579200) | 1m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14462358959/job/40557178484)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14463342099/job/40560030082)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473874933/job/40594579200)&#8201; |
| [semgrep/ci](https://github.com/aptos-labs/aptos-core/actions/runs/14473874937/job/40594552319) | 1m | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14462358974/job/40557166441)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14463342090/job/40560011963)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473874937/job/40594552319)&#8201; |
| [file_change_determinator](https://github.com/aptos-labs/aptos-core/actions/runs/14473874933/job/40594554394) | 32s | [🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14462358959/job/40557167196)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14463342099/job/40560013121)&#8201;[🟩](https://github.com/aptos-labs/aptos-core/actions/runs/14473874933/job/40594554394)&#8201; |
</details>

<sub>[settings](https://app.trunk.io/aptoslabs/settings/repo?repo=aptos-labs/aptos-core&utm_source=gh-comment#ci-analytics) ⋅ [feedback](https://form.typeform.com/to/w7Iv34u2?utm_source=gh-comment) ⋅ [docs](https://docs.trunk.io/ci-analytics/pull-request-metrics?utm_source=gh-comment) ⋅ learn more about [trunk.io](https://trunk.io?utm_source=gh-comment)</sub>
author_association: NONE
user: login: wrwg
created_at: 2025-04-15T06:00:01Z
body: * **#16366** <a href="https://app.graphite.dev/github/pr/aptos-labs/aptos-core/16366?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a> 👈 <a href="https://app.graphite.dev/github/pr/aptos-labs/aptos-core/16366?utm_source=stack-comment-view-in-graphite" target="_blank">(View in Graphite)</a>
* `main`


<h2></h2>

This stack of pull requests is managed by <a href="https://graphite.dev?utm-source=stack-comment"><b>Graphite</b></a>. Learn more about <a href="https://stacking.dev/?utm_source=stack-comment">stacking</a>.
<!-- Current dependencies on/for this PR: -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-15T16:10:13Z
body: ### Forge is running suite `framework_upgrade` on `463dda939a7bdc1266946e824a1eb309f300eaf8` ==> `898e5eac80fb9a01251203aa61cc9044676c7654`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16366&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16366%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16366%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16366%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16366%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16366&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16366&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-15T16:10:14Z
body: ### Forge is running suite `compat` on `463dda939a7bdc1266946e824a1eb309f300eaf8` ==> `898e5eac80fb9a01251203aa61cc9044676c7654`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16366&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16366%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16366%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16366%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16366%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16366&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16366&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-15T16:10:15Z
body: ### Forge is running suite `realistic_env_max_load` on `898e5eac80fb9a01251203aa61cc9044676c7654`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16366&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16366%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16366%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16366%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16366%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16366&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16366&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-15T16:22:59Z
body: ### :white_check_mark: Forge suite `realistic_env_max_load` success on `898e5eac80fb9a01251203aa61cc9044676c7654`
```
two traffics test: inner traffic : committed: 12243.18 txn/s, submitted: 12251.60 txn/s, expired: 8.42 txn/s, latency: 3198.02 ms, (p50: 2700 ms, p70: 3000, p90: 3300 ms, p99: 13300 ms), latency samples: 4655160
two traffics test : committed: 100.00 txn/s, latency: 3797.81 ms, (p50: 2000 ms, p70: 2700, p90: 11200 ms, p99: 14800 ms), latency samples: 1820
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.452, avg: 1.109", "ConsensusProposalToOrdered: max: 0.313, avg: 0.309", "ConsensusOrderedToCommit: max: 0.517, avg: 0.441", "ConsensusProposalToCommit: max: 0.824, avg: 0.750"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.03s no progress at version 30994 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.00s no progress at version 1904374 (avg 1.00s) [limit 16].
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-e2e-pr-16366&var-metrics_source=All&var-chain_name=forge-1&from=1744733431000&to=1744734177000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-e2e-pr-16366%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-e2e-pr-16366%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1744733431000&end=1744734177000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16366%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-e2e-pr-16366%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-15T16%3A10%3A31.985316%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-15T16%3A22%3A57.878983%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1744733431&until=1744734177&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16366&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1744733431&until=1744734177&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-e2e-pr-16366&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-e2e -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-15T16:25:05Z
body: ### :white_check_mark: Forge suite `framework_upgrade` success on `463dda939a7bdc1266946e824a1eb309f300eaf8` ==> `898e5eac80fb9a01251203aa61cc9044676c7654`
```
Compatibility test results for 463dda939a7bdc1266946e824a1eb309f300eaf8 ==> 898e5eac80fb9a01251203aa61cc9044676c7654 (PR)
Upgrade the nodes to version: 898e5eac80fb9a01251203aa61cc9044676c7654
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1708.92 txn/s, submitted: 1714.25 txn/s, failed submission: 5.33 txn/s, expired: 5.33 txn/s, latency: 1724.54 ms, (p50: 1500 ms, p70: 1900, p90: 2400 ms, p99: 3600 ms), latency samples: 153921
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1943.03 txn/s, submitted: 1951.10 txn/s, failed submission: 8.07 txn/s, expired: 8.07 txn/s, latency: 1519.27 ms, (p50: 1400 ms, p70: 1600, p90: 2100 ms, p99: 2900 ms), latency samples: 173380
5. check swarm health
Compatibility test for 463dda939a7bdc1266946e824a1eb309f300eaf8 ==> 898e5eac80fb9a01251203aa61cc9044676c7654 passed
Upgrade the remaining nodes to version: 898e5eac80fb9a01251203aa61cc9044676c7654
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1670.65 txn/s, submitted: 1675.76 txn/s, failed submission: 5.11 txn/s, expired: 5.11 txn/s, latency: 1818.93 ms, (p50: 1600 ms, p70: 1900, p90: 2700 ms, p99: 4300 ms), latency samples: 143782
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-framework-upgrade-pr-16366&var-metrics_source=All&var-chain_name=forge-1&from=1744733424000&to=1744734303000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-framework-upgrade-pr-16366%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-framework-upgrade-pr-16366%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1744733424000&end=1744734303000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16366%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-framework-upgrade-pr-16366%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-15T16%3A10%3A24.811760%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-15T16%3A25%3A03.349637%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1744733424&until=1744734303&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16366&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1744733424&until=1744734303&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-framework-upgrade-pr-16366&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-framework-upgrade -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-15T16:28:11Z
body: ### Forge is running suite `compat` on `463dda939a7bdc1266946e824a1eb309f300eaf8` ==> `898e5eac80fb9a01251203aa61cc9044676c7654`
* [Grafana dashboard (auto-refresh)](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16366&var-metrics_source=All&var-chain_name=forge-1&refresh=10s&from=now-15m&to=now)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16366%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16366%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=true&start=30m)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16366%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16366%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22quickRange%22%3A%20%2230m%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16366&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=now-1h&until=now&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16366&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-15T16:42:08Z
body: ### :white_check_mark: Forge suite `compat` success on `463dda939a7bdc1266946e824a1eb309f300eaf8` ==> `898e5eac80fb9a01251203aa61cc9044676c7654`
```
Compatibility test results for 463dda939a7bdc1266946e824a1eb309f300eaf8 ==> 898e5eac80fb9a01251203aa61cc9044676c7654 (PR)
1. Check liveness of validators at old version: 463dda939a7bdc1266946e824a1eb309f300eaf8
compatibility::simple-validator-upgrade::liveness-check : committed: 8820.37 txn/s, latency: 3648.12 ms, (p50: 3800 ms, p70: 4200, p90: 4500 ms, p99: 6800 ms), latency samples: 300740
2. Upgrading first Validator to new version: 898e5eac80fb9a01251203aa61cc9044676c7654
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 2925.12 txn/s, latency: 11381.58 ms, (p50: 12900 ms, p70: 13200, p90: 13300 ms, p99: 13600 ms), latency samples: 110720
3. Upgrading rest of first batch to new version: 898e5eac80fb9a01251203aa61cc9044676c7654
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 2397.03 txn/s, latency: 14059.06 ms, (p50: 15300 ms, p70: 15900, p90: 16100 ms, p99: 16400 ms), latency samples: 90900
4. upgrading second batch to new version: 898e5eac80fb9a01251203aa61cc9044676c7654
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 4724.09 txn/s, latency: 7183.17 ms, (p50: 7300 ms, p70: 8300, p90: 9000 ms, p99: 9400 ms), latency samples: 165660
5. check swarm health
Compatibility test for 463dda939a7bdc1266946e824a1eb309f300eaf8 ==> 898e5eac80fb9a01251203aa61cc9044676c7654 passed
Test Ok
```
* [Grafana dashboard](https://aptoslabs.grafana.net/d/overview/overview?orgId=1&refresh=10s&var-Datasource=VictoriaMetrics%20Global%20%28Non-mainnet%29&var-BigQuery=Google%20BigQuery&var-namespace=forge-compat-pr-16366&var-metrics_source=All&var-chain_name=forge-1&from=1744734509000&to=1744735326000)
* [Humio Logs](https://cloud.us.humio.com/k8s/search?query=%24forgeLogs%28validator_instance%3D%2A%29+%7C%0A++++%22k8s.namespace%22+%3D+%22forge-compat-pr-16366%22+%2F%2F+filters+on+namespace+which+contains+validator+logs%0A+++OR++%2F%2F+remove+either+side+of+the+OR+operator+to+only+display+validator+or+forge-runner+logs%0A++++%28%22k8s.namespace%22%3Ddefault+AND+%22k8s.labels.forge-namespace%22+%3D+%22forge-compat-pr-16366%22%29+%2F%2F+filters+on+specific+forge-runner+pod+in+default+namespace%0A&widgetType=list-view&columns=%5B%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22%40timestamp%22%2C+%22format%22%3A+%22timestamp%22%2C+%22width%22%3A+180%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22level%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+54%7D%2C+%7B%22type%22%3A+%22link%22%2C+%22openInNewBrowserTab%22%3A+%22%2A%2A%2A%22%2C+%22style%22%3A+%22button%22%2C+%22hrefTemplate%22%3A+%22https%3A%2F%2Fgithub.com%2Faptos-labs%2Faptos-core%2Fpull%2F%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22textTemplate%22%3A+%22%7B%7Bfields%5B%5C%22github_pr%5C%22%5D%7D%7D%22%2C+%22header%22%3A+%22Forge+PR%22%2C+%22width%22%3A+79%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.namespace%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+104%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.pod_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+126%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22k8s.container_name%22%2C+%22format%22%3A+%22text%22%2C+%22width%22%3A+85%7D%2C+%7B%22type%22%3A+%22field%22%2C+%22fieldName%22%3A+%22message%22%2C+%22format%22%3A+%22text%22%7D%5D&newestAtBottom=%2A%2A%2A&showOnlyFirstLine=false&live=false&start=1744734509000&end=1744735326000)
* [Axiom Logs](https://app.axiom.co/aptoslabs-hghf/explorer?initForm=%7B%22apl%22%3A%20%22%5Cn%20%20%20%20%20%20%20%20k8s%5Cn%20%20%20%20%20%20%20%20%7C%20where%20%5B%27k8s.cluster%27%5D%20contains%20%5C%22forge%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22calico-node%5C%22%20and%20%5B%27k8s.namespace%27%5D%20%21%3D%20%5C%22calico-apiserver%5C%22%20and%20%5B%27k8s.container_name%27%5D%20%21%3D%20%5C%22kube-proxy%5C%22%20and%20%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%28%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16366%5C%22%20//%20filters%20on%20namespace%20which%20contains%20validator%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20or%20//%20remove%20either%20side%20of%20the%20OR%20operator%20to%20only%20display%20validator%20or%20forge-runner%20logs%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%5B%27k8s.labels.forge-namespace%27%5D%20%3D%3D%20%5C%22forge-compat-pr-16366%5C%22%20//%20filters%20on%20specific%20forge-runner%20pod%20in%20default%20namespace%5Cn%20%20%20%20%20%20%20%20%20%20%20%20%29%5Cn%20%20%20%20%20%20%20%20%22%2C%20%22queryOptions%22%3A%20%7B%22startTime%22%3A%20%222025-04-15T16%3A28%3A29.669256%2B00%3A00%22%2C%20%22endTime%22%3A%20%222025-04-15T16%3A42%3A06.370924%2B00%3A00%22%7D%7D)
* [Validator CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1744734509&until=1744735326&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16366&var-filters=pod%7C%3D~%7C.%2Avalidator.%2A)
* [Fullnode CPU Profile](https://grafana.aptoslabs.com/a/grafana-pyroscope-app/profiles-explorer?from=1744734509&until=1744735326&maxNodes=16384&explorationType=flame-graph&var-serviceName=ebpf%2Fforge&var-filters=namespace%7C%3D%7Cforge-compat-pr-16366&var-filters=pod%7C%3D~%7C.%2Afullnode.%2A)
* [Test runner output](https://github.com/aptos-labs/aptos-core/actions/runs/14473875007)
* Test run is land-blocking
<!-- Sticky Pull Request Commentforge-compat -->
author_association: CONTRIBUTOR