Repository: dotnet/runtime
Pull request #115957: Enable new exception handling on win-x86
Author: login: filipnavara
Labels: name: area-ExceptionHandling-coreclr
name: community-contribution
description: Indicates that the PR has been added by a community member
Created: 2025-05-24T04:25:01Z | Merged: 2025-05-26T08:42:00Z

Switches JIT to the funclet model that is used by other platforms and VM to use the new exception handling introduced in .NET 9.

Fixes #113985

Linked issue #113985: New exception handling on win-x86
.NET 9 shipped with new exception handling, based on the NativeAOT model, and a NativeAOT runtime support for win-x86 platform. It left us in a position where CoreCLR on win-x86 is the only platform remaining that uses the non-funclet exception model, and also the only platform that didn't get the new exception handling. @jkotas did some [initial work](https://github.com/dotnet/runtime/pull/99191#issuecomment-1987460261) updating CoreCLR to use funclets and new exception handling on x86. Based on this initial work I updated the prototype in https://github.com/dotnet/runtime/pull/113576 to successfully pass the CoreCLR Pri0 and Libraries tests.

The ultimate goal would be to remove all non-funclet code from JIT and VM in order to simplify the code base. It may be aligned with a removal of the legacy exception handling as a part of a huge cleanup. That will, however, require commitment from all the stakeholders and solid plan on how to make the feature reach the product.

Until such plan and commitment happens I propose to upstream the relevant changes from the prototype to the point that enabling it will be flipping a compile time switch.

I'll use this issue in the next couple of weeks to post observations based on the prototype.

---

Implementation check list (incomplete):
- [x] Add win-x86 code paths for `FEATURE_EH_FUNCLET` builds to the runtime (https://github.com/dotnet/runtime/pull/114157)
- [x] Check whether we need to update exception handler registrations around `VSD_ResolveWorker` to mirror `ThePreStub` (Ref: https://github.com/dotnet/runtime/pull/114157/#discussion_r2027457273)
- [x] Investigate and document performance issues
- [x] Fix diagnostic issues and possibly implement no-GC regions (https://github.com/dotnet/runtime/issues/113985#issuecomment-2874826398)
- [x] Fix setjmp/longjmp compatibility in new EH on x86 (https://github.com/dotnet/runtime/pull/115932)
- [x] Update documentation
Linked issue comments: user: login: filipnavara
created_at: 2025-03-27T21:10:54Z
body: Initial assesement of the generated code suggests that we will now benefit from a large range of optimizations which result in smaller code size (on average) and reduced stack space usage. However, these measurements may be offset by the GC info size since more methods are now generated as fully interruptible. This needs to be carefully inspected.

Largest regression in size is for the methods that use `MethodImplOptions.Synchronized` because they now get expanded to use finally blocks with `Monitor.Exit` like on other platforms.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-03-27T21:11:29Z
body: cc @dotnet/jit-contrib @jkotas @janvorli
author_association: MEMBER
user: login: AaronRobinsonMSFT
created_at: 2025-03-27T22:33:42Z
body: @filipnavara Looking forward to this work. It will move us to a place where we can remove the Helper Method Frames in the `IL_Throw` and `IL_Rethrow` jithelpers. 

See https://github.com/dotnet/runtime/issues/95695.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-04-01T11:28:21Z
body: Performance testing as of c440a507cdc56429492f8eb2dfd80729a87b29b5 shows that we still have bottlenecks in processing of deep stacks. This is not necessarily surprising because the code in `EECodeManager` for x86 was going to great length to optimize stack unwinding and reduce unnecessary GC info decoding.

<details>
<summary>Test code</summary>

```csharp
using System.Diagnostics;
using System.Runtime.CompilerServices;

internal class Program
{
    static int exceptionsHandled = 0;

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static void CallMe(int i)
    {
        if (i == 0)
        {
            throw new NotImplementedException();
        }

        CallMe(i - 1);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static void CatchMe()
    {
        try
        {
            CallMe(10); // DEPTH: 10, 100, etc.
        }
        catch (NotImplementedException)
        {
            Interlocked.Increment(ref exceptionsHandled);
        }
    }

    private static void ThreadEntrypoint()
    {
        while (true)
        {
            CatchMe();
        }
    }

    private static void Main(string[] args)
    {
        int savedExceptionsHandled = 0;
        for (int i = 0; i < 10; i++)
        {
            new Thread(ThreadEntrypoint).Start();
        }
        Thread.Sleep(1000);
        savedExceptionsHandled = exceptionsHandled;
        Console.WriteLine($"Exceptions handled: {savedExceptionsHandled}");
        Environment.Exit(0);
    }
}
```

</details>

The test code throws an exception at various depths of stack (depending on configuration) and then catches it. This is run in 10 parallel threads and we measure how many exceptions we processed within a second. It's not using a benchmark framework so it includes some warm up and code tiering. I used a Ryzen 9950X machine to run the test, runtime and libraries were built in Release configuration. The table below includes the best value out of 5 consecutive runs, the variation between runs were minimal.

| Stack depth   |        10 |     100 |
| ------------- | --------- | ------- |
| Old EH        |   961,717 | 360,500 |
| New EH        | 1,198,822 | 180,385 |
| New EH (w/o `PROCESS_EXPLICIT_FRAME_BEFORE_MANAGED_FRAME`) | 1,404,827 | 223,177 |

(Update: Added the numbers with `PROCESS_EXPLICIT_FRAME_BEFORE_MANAGED_FRAME` disabled which is closer to the legacy x86 code paths. This is just meant to measure the impact of always unwinding an extra frame and taking some less optimized code paths. It doesn't work or pass the runtime tests as is.)
author_association: MEMBER
user: login: filipnavara
created_at: 2025-04-02T09:44:20Z
body: > This is not necessarily surprising because the code in `EECodeManager` for x86 was going to great length to optimize stack unwinding and reduce unnecessary GC info decoding.

My current plan is to try adding cache for decoded GC info into `EECodeInfo` (instead of the existing `CodeManState` structure; for X86 only).

**Update:** On its own adding the cache to `EECodeInfo` has very modest improvement (~5%). It would yeild much bigger improvements if it was paired with `PROCESS_EXPLICIT_FRAME_BEFORE_MANAGED_FRAME` but making that configuration work is a substantial effort. One thing that did stood out in the profiles are frequent calls to `EECodeManager::GetFunctionSize`, particularly from `JitTokenToMethodRegionInfo` which in turn was called from a call chain originating in `EECodeInfo::Init`. This essentially means that the GC info for each method was almost always decoded at least twice. We can either refactor this to populate the cache early or rewrite `GetFunctionSize` to decode just the very first word in the x86 GC info. The later option yield another 10% savings.

**Update 2:** Profiles show that another 10%-15% of the regression is caused by this line:
https://github.com/dotnet/runtime/blob/15872212c29cecc8d82da4548c3060f2614665f7/src/coreclr/vm/exceptionhandling.cpp#L4023
We can probably use a cheaper detection at this specific spot (cached GC info would be already decoded, so we know if it's reverse P/Invoke, return addresses in `CallDescrWorkerInternal` / `CallEHFilterFunclet` / `CallEHFunclet` are known).

**Update 3:** Finally getting to the bottom of this. Seems like the `StackTraceArray` reallocations cause GC to kick in which stops the world. This is particularly visible in the multi-threaded benchmark. However, at least part of the speed regression exists in single threaded test as well but it's difficult to capture with a profiler since the ETW events start to skew the timings.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-04-08T17:01:45Z
body: I am still working on analyzing the performance so I can paint a complete picture.

I've done a couple of micro-optimizations which help in reducing the stack walks overhead by around 30% with `FEATURE_EH_FUNCLETS` builds. The biggest one is submitted as PR #114170. I've also tried to micro-optimize few hot code paths (`EECodeManager::IsFunclet` https://github.com/dotnet/runtime/commit/466dd7dad589fd58b8ba4b53d76d06e4e9be8d09, context copies https://github.com/dotnet/runtime/commit/d033b7b7c88e60824ea2a16703d26e427da00f98).

Analysis with a sampling profiler turned out to be difficult. Firstly, the new EH is faster at shallow call stacks and slower at deep call stacks. The implication being that the extra overhead is likely in the stack walking (and `AppendExceptionStackFrame`). I focused on profiling a code that throws an exception in 100-level deep call stack and then catches it. At the high level I see that some things take 30% more time but at the low level the building blocks are usually so fast that the running time is much smaller than the period of the sampling profiler and it's thus hard to measure accurately.

However, here are few observations:
- Copying the `CONTEXT` structure from `pCalleeContext` to `pCallerContext` has measurable overhead. The structure is huge but we only track the integer+control registers so this is quite wasteful.
- A runtime compiled WITHOUT `FEATURE_EH_FUNCLETS` but with `REGDISPLAY` structure changed to match the version with `FEATURE_EH_FUNCLETS` incurs significant overhead in some scenarios. While I didn't see much difference in the single threaded scenarios, in the multithreaded ones the performance was nearly twice as slow (using the "exceptions handled / second" metric for 10 threads continually throwing and catching exceptions over 100-deep call stack). I didn't study it in detail but I know that `StackTraceInfo::AppendElement` triggers a garbage collection quite often in this test case and overhead in stack walking contributes significantly to the stop-the-world time, so it's plausible that this could explain the regression.
- With PR #114170 we do roughly the same number of GC info decodings on `FEATURE_EH_FUNCLETS` runtime as the baseline one. Same goes for the number of stack unwind calls. This means we didn't introduce any structural error but unfortunately the individual calls are still so fast that I have trouble accurately measuring whether they are slower.
- Making `SfiNext` + `AppendExceptionStackFrame` an FCall makes miniscule difference (~1% give or take) so it's unlikely that P/Invoke transitions are significant contributor to the cost.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-04-09T21:41:27Z
body: Finally I got a profiler tool appropriate for the job. With Intel PIN it's possible to get relatively accurate instruction counts spent in different methods and diff it against the baseline: [perf_diff.txt](https://github.com/user-attachments/files/19674752/perf_diff.txt). It needs a bit of a manual analysis since the code paths between the two runtime builds are quite different.

Looks like the best bang for the buck optimizations to try are the following:
- Use `NOTIFY_ON_U2M_TRANSITIONS` flag in `SfiInit` and invert the logic in `SfiNext` to detect unwinding into unmanaged frames. (https://github.com/dotnet/runtime/pull/114496/)
- Get `PROCESS_EXPLICIT_FRAME_BEFORE_MANAGED_FRAME` working with new EH. There's quite a few places that just assume caller context in `REGDISPLAY` is valid without checking.
author_association: MEMBER
user: login: davidwrighton
created_at: 2025-04-11T23:28:35Z
body: I've managed to get 2 PRs out addressing performance issues.

PR #114580  adds a cache around the behavior of EECodeInfo::IsFunclet. I'm not entirely convinced adding ad hoc caches like this is worth it, but it is a nice performance win that should accrue to all platforms, although especially the Windows platforms.

PR #114582 adds a set of micro-optimizations found by examining the code generated for the Windows X86 platform. Some of these optimizations are isolated to the funclets work, but more of them are general performance improvements to this logic. Again, the optimizations are most important for Windows X86, but should have positive impacts on other platforms as well.

With both of these optimizations, I see 15%ish performance improvements, and with them combined I see performance getting within about 10% of the baseline .NET 9 scenario on Windows X86. If we are able to take both of them, I don't see a reason to hold this feature back based on performance. One detail to be aware of is that since we are changing logic significantly, the current set of profile guided optimization data for Windows X86 will be out of date when we turn this feature on, and it will take weeks to months before we have new data to fix that issue.

The issues that I plan to look at next week are the debugging scenarios, which unfortunately are mostly all only testable via internal test suites. I hope to find some time next week to take a look at them. Notably, I have significant concerns around the behavior of mixed mode stacks/throwing and catching across managed/native boundaries, and such. The contract between the runtime/debugger has historically been fairly fragile in this area, and changing EH models entirely may cause a large problem.

In addition, we'll need to bump the R2R version numbers so that old, non-funclet code won't be accepted by the new runtime, and the issues are the exception handler registrations need to be understood.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-04-12T08:21:48Z
body: > I've managed to get 2 PRs out addressing performance issues.

Firstly, huge thanks for looking into the performance issues. There's some small overlap with the optimizations I was testing locally but the two PRs really move the needle. Along with #114496 which was merged yesterday we're looking at roughly +15% instruction count on funclets vs baseline (.NET 10 main) for deep stack exception propagation. There are some smaller micro-optimizations which can reduce that even further, probably worth pursuing but not essential. Notably, the performance actually increases significanly for exception propagation in shallow stack traces. Once all the PRs are reviewed and merged I can run the benchmarks and share some graphs / tables. Notably, #14170 also improved the scenario by a small bit on non-funclet runtime, so comparing to .NET 9 we are now within ~10%, as stated above. I agree with the sentiment that this seems like an acceptable trade-off since we get improved performance and code quality in many other areas and scenarios, including more common exception handling use cases.

 > The issues that I plan to look at next week are the debugging scenarios, which unfortunately are mostly all only testable via internal test suites.

Help on this is much appreciated! I only did very limited tests with netcoredbg to ensure I'm not completely breaking the world, but that doesn't cover any mixed native/managed exceptions.

> In addition, we'll need to bump the R2R version numbers...

Right, I'll bolt that on to the commit in https://github.com/dotnet/runtime/pull/113576 so we don't forget about it.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-04-24T05:16:43Z
body: ### Microbenchmarks from dotnet/performance

Tested with commit 199ae8820492f05169529da1f262e9736007448a (main) and the x86funclets branch rebased on top. 

Funclets are faster on all of them. None of the benchmarks excercise the pathological paths with deep stack traces. The "deep" stack trace benchmarks are only 10-level deep.

| Method                              | Job        | Branch      | kind               | Mean      | Error     | StdDev    | Median    | Min       | Max       | Ratio | MannWhitney(1%) | Gen0   | Allocated | Alloc Ratio |
|------------------------------------ |----------- |----------------------------------------------------------------------------------------- |------------------- |----------:|----------:|----------:|----------:|----------:|----------:|------:|---------------- |-------:|----------:|------------:|
| ThrowAndCatch                       | Job-ICNAGN | x86funclets | Software           |  1.180 us | 0.0077 us | 0.0072 us |  1.177 us |  1.171 us |  1.193 us |  0.21 | Faster          | 0.0328 |     172 B |        1.00 |
| ThrowAndCatch                       | Job-JLGMJO | main        | Software           |  5.500 us | 0.0378 us | 0.0353 us |  5.494 us |  5.441 us |  5.570 us |  1.00 | Baseline        | 0.0216 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatch_ManyCatchBlocks       | Job-ICNAGN | x86funclets | Software           |  1.339 us | 0.0037 us | 0.0032 us |  1.340 us |  1.331 us |  1.343 us |  0.23 | Faster          | 0.0322 |     172 B |        1.00 |
| ThrowAndCatch_ManyCatchBlocks       | Job-JLGMJO | main        | Software           |  5.706 us | 0.0277 us | 0.0231 us |  5.713 us |  5.643 us |  5.738 us |  1.00 | Baseline        | 0.0226 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchFinally                | Job-ICNAGN | x86funclets | Software           |  1.183 us | 0.0043 us | 0.0040 us |  1.182 us |  1.178 us |  1.191 us |  0.22 | Faster          | 0.0283 |     172 B |        1.00 |
| ThrowAndCatchFinally                | Job-JLGMJO | main        | Software           |  5.442 us | 0.0317 us | 0.0296 us |  5.436 us |  5.391 us |  5.495 us |  1.00 | Baseline        | 0.0215 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchWhen                   | Job-ICNAGN | x86funclets | Software           |  1.256 us | 0.0064 us | 0.0060 us |  1.257 us |  1.245 us |  1.265 us |  0.23 | Faster          | 0.0301 |     172 B |        1.00 |
| ThrowAndCatchWhen                   | Job-JLGMJO | main        | Software           |  5.520 us | 0.0274 us | 0.0256 us |  5.519 us |  5.476 us |  5.558 us |  1.00 | Baseline        | 0.0218 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchWhenFinally            | Job-ICNAGN | x86funclets | Software           |  1.256 us | 0.0067 us | 0.0063 us |  1.256 us |  1.245 us |  1.266 us |  0.23 | Faster          | 0.0298 |     172 B |        1.00 |
| ThrowAndCatchWhenFinally            | Job-JLGMJO | main        | Software           |  5.556 us | 0.0343 us | 0.0321 us |  5.548 us |  5.505 us |  5.619 us |  1.00 | Baseline        | 0.0218 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchDeep                   | Job-ICNAGN | x86funclets | Software           |  3.602 us | 0.0131 us | 0.0116 us |  3.599 us |  3.586 us |  3.628 us |  0.51 | Faster          | 0.1282 |     692 B |        1.00 |
| ThrowAndCatchDeep                   | Job-JLGMJO | main        | Software           |  7.123 us | 0.0362 us | 0.0338 us |  7.118 us |  7.075 us |  7.194 us |  1.00 | Baseline        | 0.1128 |     692 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchDeepRecursive          | Job-ICNAGN | x86funclets | Software           |  3.831 us | 0.0128 us | 0.0120 us |  3.830 us |  3.814 us |  3.853 us |  0.53 | Faster          | 0.1224 |     692 B |        1.00 |
| ThrowAndCatchDeepRecursive          | Job-JLGMJO | main        | Software           |  7.283 us | 0.0296 us | 0.0277 us |  7.276 us |  7.235 us |  7.326 us |  1.00 | Baseline        | 0.1160 |     692 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| MultipleNestedTryCatch_FirstCatches | Job-ICNAGN | x86funclets | Software           |  1.178 us | 0.0032 us | 0.0028 us |  1.179 us |  1.173 us |  1.184 us |  0.22 | Faster          | 0.0328 |     172 B |        1.00 |
| MultipleNestedTryCatch_FirstCatches | Job-JLGMJO | main        | Software           |  5.448 us | 0.0175 us | 0.0146 us |  5.447 us |  5.421 us |  5.476 us |  1.00 | Baseline        | 0.0216 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| MultipleNestedTryCatch_LastCatches  | Job-ICNAGN | x86funclets | Software           |  1.294 us | 0.0057 us | 0.0053 us |  1.294 us |  1.286 us |  1.303 us |  0.23 | Faster          | 0.0310 |     172 B |        1.00 |
| MultipleNestedTryCatch_LastCatches  | Job-JLGMJO | main        | Software           |  5.648 us | 0.0315 us | 0.0294 us |  5.659 us |  5.604 us |  5.688 us |  1.00 | Baseline        | 0.0222 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| MultipleNestedTryFinally            | Job-ICNAGN | x86funclets | Software           |  1.336 us | 0.0066 us | 0.0061 us |  1.337 us |  1.328 us |  1.346 us |  0.23 | Faster          | 0.0324 |     172 B |        1.00 |
| MultipleNestedTryFinally            | Job-JLGMJO | main        | Software           |  5.726 us | 0.0433 us | 0.0405 us |  5.730 us |  5.647 us |  5.794 us |  1.00 | Baseline        | 0.0225 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| CatchAndRethrowDeep                 | Job-ICNAGN | x86funclets | Software           | 15.519 us | 0.0499 us | 0.0417 us | 15.522 us | 15.433 us | 15.583 us |  0.26 | Faster          | 0.1236 |     812 B |        1.00 |
| CatchAndRethrowDeep                 | Job-JLGMJO | main        | Software           | 59.260 us | 0.4628 us | 0.4329 us | 59.243 us | 58.495 us | 59.876 us |  1.00 | Baseline        |      - |     812 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| CatchAndThrowOtherDeep              | Job-ICNAGN | x86funclets | Software           | 19.625 us | 0.1129 us | 0.1056 us | 19.621 us | 19.466 us | 19.837 us |  0.32 | Faster          | 0.3129 |    1892 B |        1.00 |
| CatchAndThrowOtherDeep              | Job-JLGMJO | main        | Software           | 61.255 us | 0.3391 us | 0.3172 us | 61.209 us | 60.603 us | 61.619 us |  1.00 | Baseline        | 0.2413 |    1892 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| TryAndFinallyDeep                   | Job-ICNAGN | x86funclets | Software           |  4.918 us | 0.0231 us | 0.0216 us |  4.919 us |  4.875 us |  4.964 us |  0.60 | Faster          | 0.1180 |     692 B |        1.00 |
| TryAndFinallyDeep                   | Job-JLGMJO | main        | Software           |  8.200 us | 0.0668 us | 0.0592 us |  8.185 us |  8.142 us |  8.341 us |  1.00 | Baseline        | 0.1303 |     692 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| TryAndCatchDeep_CaugtAtTheTop       | Job-ICNAGN | x86funclets | Software           |  4.729 us | 0.0242 us | 0.0189 us |  4.730 us |  4.699 us |  4.757 us |  0.61 | Faster          | 0.1306 |     692 B |        1.00 |
| TryAndCatchDeep_CaugtAtTheTop       | Job-JLGMJO | main        | Software           |  7.791 us | 0.0450 us | 0.0399 us |  7.777 us |  7.736 us |  7.882 us |  1.00 | Baseline        | 0.1225 |     692 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatch                       | Job-ICNAGN | x86funclets | Hardware           |  3.585 us | 0.0180 us | 0.0168 us |  3.584 us |  3.560 us |  3.618 us |  0.63 | Faster          | 0.0282 |     172 B |        1.00 |
| ThrowAndCatch                       | Job-JLGMJO | main        | Hardware           |  5.662 us | 0.0184 us | 0.0172 us |  5.658 us |  5.639 us |  5.701 us |  1.00 | Baseline        | 0.0222 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatch_ManyCatchBlocks       | Job-ICNAGN | x86funclets | Hardware           |  3.730 us | 0.0140 us | 0.0124 us |  3.728 us |  3.707 us |  3.756 us |  0.63 | Faster          | 0.0295 |     172 B |        1.00 |
| ThrowAndCatch_ManyCatchBlocks       | Job-JLGMJO | main        | Hardware           |  5.939 us | 0.0181 us | 0.0151 us |  5.935 us |  5.924 us |  5.972 us |  1.00 | Baseline        | 0.0235 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchFinally                | Job-ICNAGN | x86funclets | Hardware           |  3.552 us | 0.0134 us | 0.0125 us |  3.552 us |  3.532 us |  3.580 us |  0.63 | Faster          | 0.0279 |     172 B |        1.00 |
| ThrowAndCatchFinally                | Job-JLGMJO | main        | Hardware           |  5.630 us | 0.0364 us | 0.0341 us |  5.628 us |  5.587 us |  5.698 us |  1.00 | Baseline        | 0.0221 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchWhen                   | Job-ICNAGN | x86funclets | Hardware           |  3.623 us | 0.0187 us | 0.0166 us |  3.625 us |  3.587 us |  3.647 us |  0.63 | Faster          | 0.0286 |     172 B |        1.00 |
| ThrowAndCatchWhen                   | Job-JLGMJO | main        | Hardware           |  5.712 us | 0.0395 us | 0.0370 us |  5.733 us |  5.657 us |  5.762 us |  1.00 | Baseline        | 0.0224 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchWhenFinally            | Job-ICNAGN | x86funclets | Hardware           |  3.639 us | 0.0178 us | 0.0166 us |  3.634 us |  3.622 us |  3.670 us |  0.63 | Faster          | 0.0288 |     172 B |        1.00 |
| ThrowAndCatchWhenFinally            | Job-JLGMJO | main        | Hardware           |  5.735 us | 0.0187 us | 0.0175 us |  5.733 us |  5.703 us |  5.765 us |  1.00 | Baseline        | 0.0225 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchDeep                   | Job-ICNAGN | x86funclets | Hardware           |  6.005 us | 0.0236 us | 0.0220 us |  5.999 us |  5.976 us |  6.045 us |  0.82 | Faster          | 0.1199 |     692 B |        1.00 |
| ThrowAndCatchDeep                   | Job-JLGMJO | main        | Hardware           |  7.301 us | 0.0300 us | 0.0280 us |  7.306 us |  7.245 us |  7.334 us |  1.00 | Baseline        | 0.1150 |     692 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchDeepRecursive          | Job-ICNAGN | x86funclets | Hardware           |  6.260 us | 0.0227 us | 0.0212 us |  6.259 us |  6.219 us |  6.305 us |  0.84 | Faster          | 0.1235 |     692 B |        1.00 |
| ThrowAndCatchDeepRecursive          | Job-JLGMJO | main        | Hardware           |  7.428 us | 0.0245 us | 0.0229 us |  7.433 us |  7.388 us |  7.464 us |  1.00 | Baseline        | 0.1175 |     692 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| MultipleNestedTryCatch_FirstCatches | Job-ICNAGN | x86funclets | Hardware           |  3.556 us | 0.0186 us | 0.0174 us |  3.555 us |  3.528 us |  3.582 us |  0.63 | Faster          | 0.0280 |     172 B |        1.00 |
| MultipleNestedTryCatch_FirstCatches | Job-JLGMJO | main        | Hardware           |  5.646 us | 0.0182 us | 0.0170 us |  5.650 us |  5.617 us |  5.673 us |  1.00 | Baseline        | 0.0221 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| MultipleNestedTryCatch_LastCatches  | Job-ICNAGN | x86funclets | Hardware           |  3.687 us | 0.0172 us | 0.0160 us |  3.679 us |  3.670 us |  3.714 us |  0.63 | Faster          | 0.0291 |     172 B |        1.00 |
| MultipleNestedTryCatch_LastCatches  | Job-JLGMJO | main        | Hardware           |  5.879 us | 0.0225 us | 0.0210 us |  5.880 us |  5.833 us |  5.910 us |  1.00 | Baseline        | 0.0231 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| MultipleNestedTryFinally            | Job-ICNAGN | x86funclets | Hardware           |  3.683 us | 0.0197 us | 0.0184 us |  3.680 us |  3.652 us |  3.712 us |  0.62 | Faster          | 0.0291 |     172 B |        1.00 |
| MultipleNestedTryFinally            | Job-JLGMJO | main        | Hardware           |  5.920 us | 0.0228 us | 0.0202 us |  5.918 us |  5.884 us |  5.961 us |  1.00 | Baseline        | 0.0232 |     172 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| CatchAndRethrowDeep                 | Job-ICNAGN | x86funclets | Hardware           | 18.051 us | 0.0764 us | 0.0715 us | 18.053 us | 17.930 us | 18.172 us |  0.30 | Faster          | 0.1435 |     812 B |        1.00 |
| CatchAndRethrowDeep                 | Job-JLGMJO | main        | Hardware           | 60.568 us | 0.1999 us | 0.1772 us | 60.599 us | 60.311 us | 60.777 us |  1.00 | Baseline        |      - |     812 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| CatchAndThrowOtherDeep              | Job-ICNAGN | x86funclets | Hardware           | 21.770 us | 0.0543 us | 0.0508 us | 21.767 us | 21.690 us | 21.865 us |  0.35 | Faster          | 0.3463 |    1892 B |        1.00 |
| CatchAndThrowOtherDeep              | Job-JLGMJO | main        | Hardware           | 61.553 us | 0.3008 us | 0.2511 us | 61.467 us | 61.262 us | 62.120 us |  1.00 | Baseline        | 0.2441 |    1892 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| TryAndFinallyDeep                   | Job-ICNAGN | x86funclets | Hardware           |  7.182 us | 0.0257 us | 0.0228 us |  7.181 us |  7.133 us |  7.218 us |  0.85 | Faster          | 0.1141 |     692 B |        1.00 |
| TryAndFinallyDeep                   | Job-JLGMJO | main        | Hardware           |  8.489 us | 0.0238 us | 0.0223 us |  8.490 us |  8.447 us |  8.529 us |  1.00 | Baseline        | 0.1005 |     692 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| TryAndCatchDeep_CaugtAtTheTop       | Job-ICNAGN | x86funclets | Hardware           |  7.105 us | 0.0306 us | 0.0271 us |  7.103 us |  7.072 us |  7.158 us |  0.88 | Faster          | 0.1123 |     692 B |        1.00 |
| TryAndCatchDeep_CaugtAtTheTop       | Job-JLGMJO | main        | Hardware           |  8.079 us | 0.0295 us | 0.0276 us |  8.079 us |  8.036 us |  8.127 us |  1.00 | Baseline        | 0.1270 |     692 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatch                       | Job-ICNAGN | x86funclets | ReflectionSoftware |  6.111 us | 0.0294 us | 0.0261 us |  6.115 us |  6.047 us |  6.155 us |  0.52 | Faster          | 0.0969 |     540 B |        1.00 |
| ThrowAndCatch                       | Job-JLGMJO | main        | ReflectionSoftware | 11.742 us | 0.0505 us | 0.0448 us | 11.739 us | 11.646 us | 11.827 us |  1.00 | Baseline        | 0.0923 |     540 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatch_ManyCatchBlocks       | Job-ICNAGN | x86funclets | ReflectionSoftware |  6.191 us | 0.0355 us | 0.0332 us |  6.177 us |  6.128 us |  6.239 us |  0.52 | Faster          | 0.0989 |     540 B |        1.00 |
| ThrowAndCatch_ManyCatchBlocks       | Job-JLGMJO | main        | ReflectionSoftware | 11.958 us | 0.0711 us | 0.0630 us | 11.932 us | 11.886 us | 12.102 us |  1.00 | Baseline        | 0.0946 |     540 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchDeep                   | Job-ICNAGN | x86funclets | ReflectionSoftware |  8.399 us | 0.0775 us | 0.0725 us |  8.366 us |  8.315 us |  8.589 us |  0.63 | Faster          | 0.1677 |     972 B |        1.00 |
| ThrowAndCatchDeep                   | Job-JLGMJO | main        | ReflectionSoftware | 13.331 us | 0.0576 us | 0.0538 us | 13.326 us | 13.231 us | 13.412 us |  1.00 | Baseline        | 0.1585 |     972 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchDeepRecursive          | Job-ICNAGN | x86funclets | ReflectionSoftware |  8.664 us | 0.0288 us | 0.0270 us |  8.671 us |  8.598 us |  8.694 us |  0.64 | Faster          | 0.1738 |     972 B |        1.00 |
| ThrowAndCatchDeepRecursive          | Job-JLGMJO | main        | ReflectionSoftware | 13.483 us | 0.0682 us | 0.0638 us | 13.472 us | 13.394 us | 13.611 us |  1.00 | Baseline        | 0.1593 |     972 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatch                       | Job-ICNAGN | x86funclets | ReflectionHardware |  8.433 us | 0.0378 us | 0.0335 us |  8.432 us |  8.373 us |  8.496 us |  0.71 | Faster          | 0.1005 |     540 B |        1.00 |
| ThrowAndCatch                       | Job-JLGMJO | main        | ReflectionHardware | 11.957 us | 0.0591 us | 0.0553 us | 11.938 us | 11.892 us | 12.069 us |  1.00 | Baseline        | 0.0945 |     540 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatch_ManyCatchBlocks       | Job-ICNAGN | x86funclets | ReflectionHardware |  8.597 us | 0.0356 us | 0.0333 us |  8.588 us |  8.557 us |  8.664 us |  0.70 | Faster          | 0.1020 |     540 B |        1.00 |
| ThrowAndCatch_ManyCatchBlocks       | Job-JLGMJO | main        | ReflectionHardware | 12.307 us | 0.0683 us | 0.0639 us | 12.303 us | 12.213 us | 12.422 us |  1.00 | Baseline        | 0.0968 |     540 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchDeep                   | Job-ICNAGN | x86funclets | ReflectionHardware | 10.828 us | 0.0385 us | 0.0360 us | 10.824 us | 10.737 us | 10.881 us |  0.80 | Faster          | 0.1728 |     972 B |        1.00 |
| ThrowAndCatchDeep                   | Job-JLGMJO | main        | ReflectionHardware | 13.512 us | 0.0582 us | 0.0544 us | 13.521 us | 13.404 us | 13.587 us |  1.00 | Baseline        | 0.1612 |     972 B |        1.00 |
|                                     |            |                                                                                          |                    |           |           |           |           |           |           |       |                 |        |           |             |
| ThrowAndCatchDeepRecursive          | Job-ICNAGN | x86funclets | ReflectionHardware | 11.031 us | 0.0435 us | 0.0363 us | 11.036 us | 10.965 us | 11.097 us |  0.80 | Faster          | 0.1743 |     972 B |        1.00 |
| ThrowAndCatchDeepRecursive          | Job-JLGMJO | main        | ReflectionHardware | 13.704 us | 0.0720 us | 0.0673 us | 13.713 us | 13.596 us | 13.807 us |  1.00 | Baseline        | 0.1623 |     972 B |        1.00 |

author_association: MEMBER
user: login: filipnavara
created_at: 2025-04-24T05:31:12Z
body: ### Deep stack trace benchmark

<details>
<summary>Source code</summary>

```csharp
    static int exceptionsHandled = 0;

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static void CallMe(int i, NotImplementedException inEx)
    {
        if (i == 0)
        {
            throw inEx;
        }

        CallMe(i - 1, inEx);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static void CatchMe(NotImplementedException inEx, int depth = 100)
    {
        try
        {
            CallMe(depth, inEx);
        }
        catch (NotImplementedException)
        {
            Interlocked.Increment(ref exceptionsHandled);
        }
    }

    [Params(1, 10, 100, 1000)]
    public int Depth;

    [Benchmark]
    public void CatchMe1000x100()
    {
        CatchMe(new NotImplementedException(), Depth);
    }    
```
</details>



Main:
| Method          | Depth | Mean       | Error     | StdDev    |
|---------------- |------ |-----------:|----------:|----------:|
| CatchMe1000x100 | 1     |   5.636 us | 0.0139 us | 0.0130 us |
| CatchMe1000x100 | 10    |   7.029 us | 0.0163 us | 0.0145 us |
| CatchMe1000x100 | 100   |  20.581 us | 0.0601 us | 0.0562 us |
| CatchMe1000x100 | 1000  | 153.568 us | 0.4248 us | 0.3974 us |

Funclets:
| Method          | Depth | Mean       | Error     | StdDev    |
|---------------- |------ |-----------:|----------:|----------:|
| CatchMe1000x100 | 1     |   1.488 us | 0.0035 us | 0.0033 us |
| CatchMe1000x100 | 10    |   3.657 us | 0.0115 us | 0.0102 us |
| CatchMe1000x100 | 100   |  23.606 us | 0.1366 us | 0.1211 us |
| CatchMe1000x100 | 1000  | 220.143 us | 0.7091 us | 0.6633 us |


author_association: MEMBER
user: login: filipnavara
created_at: 2025-04-24T05:43:08Z
body: ### Handled exceptions/second metric

This is essentially the same non-scientific benchmark shared in https://github.com/dotnet/runtime/issues/113985#issuecomment-2769050218. We run 10 threads, each of them is throwing an exception at deep stack trace and catching it at the top of the stack trace.

This is showing pretty much the worst case scenario (at depth 100) and also the improvements for the more common scenarios. It also shows that we don't introduce any significant bottleneck by taking some additional lock.

<details>
<summary>Source code</summary>

```csharp
using System.Diagnostics;
using System.Runtime.CompilerServices;

public class Program
{
    static int exceptionsHandled = 0;

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static void CallMe(int i, NotImplementedException inEx)
    {
        if (i == 0)
        {
            throw inEx;
        }

        CallMe(i - 1, inEx);
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static void CatchMe(NotImplementedException inEx, int depth = 100)
    {
        try
        {
            CallMe(depth, inEx);
        }
        catch (NotImplementedException)
        {
            Interlocked.Increment(ref exceptionsHandled);
        }
    }

    private static void ThreadEntrypoint()
    {
        while (true)
        {
            NotImplementedException inEx = new NotImplementedException();
            CatchMe(inEx, 100);
        }
    }

    public static void Main(string[] args)
    {
        exceptionsHandled = 0; 
    
        Stopwatch sw = Stopwatch.StartNew();
        int savedExceptionsHandled = 0;
        int maxThreads = action == "st" ? 1 : 10;
        for (int i = 0; i < maxThreads; i++)
        {
            new Thread(ThreadEntrypoint).Start();
        }
        Thread.Sleep(1000);
        savedExceptionsHandled = Volatile.Read(ref exceptionsHandled);
        sw.Stop();
        Console.WriteLine($"Exceptions handled: {savedExceptionsHandled}");
        Console.WriteLine($"Time: {sw.Elapsed}");
        Console.WriteLine($"Normalized: {(int)(savedExceptionsHandled * 1000 / sw.ElapsedMilliseconds)}");
        Environment.Exit(0);
    }
}
```
</details>

#### Depth: 100
Main: 386,344 ex/sec
.NET 9: 378,816 ex/sec
Funclets: 343,814 ex/sec

#### Depth: 50
Main: 599,575 ex/sec
Funclets: 646,922 ex/sec

#### Depth: 10
Main: 931,671 ex/sec
Funclets: 1,645,489 ex/sec


author_association: MEMBER
user: login: am11
created_at: 2025-05-02T22:02:05Z
body: > The issues that I plan to look at next week are the debugging scenarios, which unfortunately are mostly all only testable via internal test suites. I hope to find some time next week to take a look at them. Notably, I have significant concerns around the behavior of mixed mode stacks/throwing and catching across managed/native boundaries, and such. The contract between the runtime/debugger has historically been fairly fragile in this area, and changing EH models entirely may cause a large problem.

@davidwrighton, would it be possible to get this validated in the current preview time-frame? It will allow us to remove a few more HMFs from jithelpers.
author_association: MEMBER
user: login: davidwrighton
created_at: 2025-05-13T02:18:00Z
body: @am11 I've been working on the debugger space last week and today. I've made good progress, and I'm down to 14 failures in the suite that are unique to this configuration. The fixes have mostly been straightforward although debugging the debugger is always a time-consuming task.

@filipnavara @am11  I did however find something of concern to me. Notably, https://github.com/dotnet/runtime/blob/fdbe38ce16671ae356e2bad21479e38b61d03340/src/coreclr/jit/codegenxarch.cpp#L211  It looks to me like the funclet model may require the implementation of non-gc regions for stress scenarios in fully interruptible code.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-13T09:31:00Z
body: > It looks to me like the funclet model may require the implementation of non-gc regions for stress scenarios in fully interruptible code.

I'll have to study this more in detail. It made me wonder if this specific occasion could be generated as `push finally-return; jmp finally-funclet` but that's not a general solution and I am not sure how it would interact with CFG mitigations (UPD: on closer inspection, the non-funclet code path already does this, so probably it's fine).
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-13T15:05:57Z
body: > It made me wonder if this specific occasion could be generated as `push finally-return; jmp finally-funclet`

I pushed the change (https://github.com/dotnet/runtime/pull/113576/commits/4a587dd61715af903b4586eeed65022a252b87ef) into  https://github.com/dotnet/runtime/pull/113576. Let's see if it passes CI and if it helps with the stress scenarios.

UPD: First time was not the charm...
UPD2: Yeah, I see now why that could not work. If someone throws in the `finally` block we need the stack unwinder to see the BBJ_CALLFINALLY trampoline. I'll have to rethink this.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-14T12:17:55Z
body: > It looks to me like the funclet model may require the implementation of non-gc regions for stress scenarios in fully interruptible code.

I slept a bit on it. We can probably encode this in the x86 GC info. We essentially have two options how to encode it:
- Hijack the last unused 0xBE opcode in argument table as "flip no GC region" and bolt it into `scanArgRegTableI` logic. The upside of this is that it's relatively minor change, and we already have the code offset skipping logic there. The code paths to generate that in the JIT would not be hard to implement but they would be distinct from the non-x86 GC info.
- Introduce a new header flag (part of `infoHdrAdjust2`, just past `SET_RET_KIND_MAX`) and then introduce a separate table for the non-interruptible ranges.

Thoughts?

(Obviously there's also the option to switch to the common GC encoder, define unwind info opcodes for x86, etc., but that's a much larger effort.)

_UPD: I started working on a prototype using the new table. Let's see how it goes._
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-16T16:59:50Z
body: I went through the GC stress failures:
- Some unwinding issues are unrelated and broken in main, fixed by #115660
- `JIT\Generics\Fields\getclassfrommethodparam\getclassfrommethodparam` is a bug in looking up the generic parameter. It needs the fixes in #115630 and I put the additional changes necessary in the PR comments

Not GC stress specific but part of the Pri1 tests:
- Propagation of `longjmp` across managed frames doesn't work. This is officially unsupported so it's not necessarily a blocker but apparently there were some customers who depended on that. It's possible to fix but annoyingly non-trivial. We get `STATUS_LONGJUMP` exception but unlike 64-bit platforms we don't get the jump buffer in the `EXCEPTION_RECORD::ExceptionInformation`. It's expected that we will just return `ExceptionContinueSearch` from the exception handler after we run the funclets for `finally` regions. Retrofitting that into the code base is non-trivial. We would either need to make some of the "no-return" code paths actually return in this specific case or we can come up with some clever hack. I initially tried to hack it by calling `setjmp` in `ProcessCLRException` and then pretending that it's the actual jump buffer for the sake of the current longjmp code paths but that doesn't really work. The other code paths would need plenty of modifications not to unwind the stack and SEH handlers beyond the currently processed record to keep that record and the jump buffer on stack still registered and valid.
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-21T18:35:00Z
body: > Propagation of longjmp across managed frames doesn't work. This is officially unsupported so it's not necessarily a blocker but apparently there were some customers who depended on that. It's possible to fix but annoyingly non-trivial. 

I would be ok with doing nothing (except maybe make it easier to diagnose - fail with a clear message), leaving it broken and waiting for signal that this is important to fix. The current state, where we say that it is unsupported but make it limp along in a semi-broken state, is not good.
author_association: MEMBER
user: login: davidwrighton
created_at: 2025-05-21T22:30:27Z
body: @filipnavara I agree with @jkotas here. If you prepare a PR which makes this into a diagnosable error, that should be enough. I'd like to hear an opinion from @janvorli here before we conclude that is the right approach though.
author_association: MEMBER
user: login: janvorli
created_at: 2025-05-21T23:11:23Z
body: @filipnavara I agree with @jkotas and @davidwrighton. I don't see what else can we do at the moment.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-22T05:18:43Z
body: Making it a diagnosable FailFast is easy. I will submit the PR in the next days.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-23T08:46:02Z
body: I submitted PR to handle the setjmp/longjmp scenario. It seems like we should be in a state where all the known GC stress and Pri1 tests are fixed. Is there a list of things that are still broken on the debugger / diagnostics side so I can update the TODO at the top to keep track?
author_association: MEMBER
user: login: am11
created_at: 2025-05-23T09:42:02Z
body: https://github.com/dotnet/runtime/pull/113576 can be rebased against main, most of the changes were merged.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-23T09:45:40Z
body: > [#113576](https://github.com/dotnet/runtime/pull/113576) can be rebased against main, most of the changes were merged.

I will do it, just didn't want to lose the stress test results (some pipelines delete all output on push of next commit).
author_association: MEMBER
user: login: davidwrighton
created_at: 2025-05-23T12:12:15Z
body: Debugger is fully working as far as I can test without an official build.
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-24T03:49:45Z
body: I think we ready to flip the switch! @filipnavara Could you please prepare the PR?

If everything goes well, the non-funclet implementation can be deleted sometime after .NET 10 release, like end of January 2026.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-24T03:56:39Z
body: Yep. I will try to get it ready today.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-27T13:46:11Z
body: We have some perf wins: https://github.com/dotnet/perf-autofiling-issues/issues/56220

...but possibly some regressions too: https://github.com/dotnet/perf-autofiling-issues/issues/56182
author_association: MEMBER

Diff:
diff --git a/docs/design/coreclr/botr/clr-abi.md b/docs/design/coreclr/botr/clr-abi.md
index 32c89100205049..8d872db6bc188f 100644
--- a/docs/design/coreclr/botr/clr-abi.md
+++ b/docs/design/coreclr/botr/clr-abi.md
@@ -218,14 +218,10 @@ This section describes the conventions the JIT needs to follow when generating c
 
 ## Funclets
 
-For all platforms except Windows/x86 on CoreCLR, all managed EH handlers (finally, fault, filter, filter-handler, and catch) are extracted into their own 'funclets'. To the OS they are treated just like first class functions (separate PDATA and XDATA (`RUNTIME_FUNCTION` entry), etc.). The CLR currently treats them just like part of the parent function in many ways. The main function and all funclets must be allocated in a single code allocation (see hot cold splitting). They 'share' GC info. Only the main function prolog can be hot patched.
+For all platforms, managed EH handlers (finally, fault, filter, filter-handler, and catch) are extracted into their own 'funclets'. To the OS they are treated just like first class functions (separate PDATA and XDATA (`RUNTIME_FUNCTION` entry), etc.). The CLR currently treats them just like part of the parent function in many ways. The main function and all funclets must be allocated in a single code allocation (see hot cold splitting). They 'share' GC info. Only the main function prolog can be hot patched.
 
 The only way to enter a handler funclet is via a call. In the case of an exception, the call is from the VM's EH subsystem as part of exception dispatch/unwind. In the non-exceptional case, this is called local unwind or a non-local exit. In C# this is accomplished by simply falling-through/out of a try body or an explicit goto. In IL this is always accomplished via a LEAVE opcode, within a try body, targeting an IL offset outside the try body. In such cases the call is from the JITed code of the parent function.
 
-For Windows/x86 on CoreCLR, all handlers are generated within the method body, typically in lexical order. A nested try/catch is generated completely within the EH region in which it is nested. These handlers are essentially "in-line funclets", but they do not look like normal functions: they do not have a normal prolog or epilog, although they do have special entry/exit and register conventions. Also, nested handlers are not un-nested as for funclets: the code for a nested handler is generated within the handler in which it is nested.
-
-For Windows/x86 on NativeAOT and Linux/x86, funclets are used just like on other platforms.
-
 ## Cloned finallys
 
 RyuJIT attempts to speed the normal control flow by 'inlining' a called finally along the 'normal' control flow (i.e., leaving a try body in a non-exceptional manner via C# fall-through). This optimization is supported on all architectures.
@@ -234,7 +230,7 @@ RyuJIT attempts to speed the normal control flow by 'inlining' a called finally
 
 In order to have proper forward progress and `Thread.Abort` semantics, there are restrictions on where a call-to-finally can be, and what the call site must look like. The return address can **NOT** be in the corresponding try body (otherwise the VM would think the finally protects itself). The return address **MUST** be within any outer protected region (so exceptions from the finally body are properly handled).
 
-JIT64, and RyuJIT for non-x86, creates something similar to a jump island: a block of code outside the try body that calls the finally and then branches to the final target of the leave/non-local-exit. This jump island is then marked in the EH tables as if it were a cloned finally. The cloned finally clause prevents a Thread.Abort from firing before entering the handler. By having the return address outside of the try body we satisfy the other constraint.
+RyuJIT creates something similar to a jump island: a block of code outside the try body that calls the finally and then branches to the final target of the leave/non-local-exit. This jump island is then marked in the EH tables as if it were a cloned finally. The cloned finally clause prevents a Thread.Abort from firing before entering the handler. By having the return address outside of the try body we satisfy the other constraint.
 
 ## ThreadAbortException considerations
 
@@ -388,100 +384,6 @@ Any register value changes made in the funclet are lost. If a funclet wants to m
 
 Funclets are not required to preserve non-volatile registers.
 
-## Windows/x86 EH considerations
-
-The Windows/x86 model is somewhat different than non-Windows/x86 model. Windows/X86-specific concerns are mentioned here.
-
-### catch / filter-handler regions
-
-When leaving a `catch` or `filter-handler` region, the JIT calls the helper `CORINFO_JIT_ENDCATCH` (implemented in the VM by the `JIT_EndCatch` function) before transferring control to the target location. The code to call to `CORINFO_JIT_ENDCATCH` is within the catch region itself.
-
-### finally / fault regions
-
-"finally" clauses are invoked in the non-exceptional code by the generated JIT code, and in the exceptional case by the VM. "fault" clauses are only executed in exceptional cases by the VM.
-
-On entry to the finally or fault, the top of the stack is the address that should be jumped to on exit from the finally, using a "pop eax; jmp eax" sequence. A simple 'ret' could be used, but we avoid it to avoid potentially creating an unbalanced processor call/ret buffer stack, and messing up call/ret prediction.
-
-There are no register or other stack arguments to a 'finally' or 'fault'.
-
-### ShadowSP slots
-
-X86 exception handlers (e.g., catch, finally) do not establish their own frames. They don't (really) have prologs and epilogs. However, they do use the stack, and need to restore the stack pointer of the enclosing exception handling region when the handler completes executing.
-
-To implement this requirement, for any function with EH, we create a frame-local variable to store a stack of "Shadow SP" values, or ShadowSP slots. In the JIT, the local var is called lvaShadowSPslotsVar, and in dumps it is called "EHSlots". The variable is created in lvaMarkLocalVars() and is sized as follows:
-1. 1 slot is reserved for the VM (for ICodeManager::FixContext(ppEndRegion)).
-2. 1 slot for each handler nesting level (total: ehMaxHndNestingCount).
-3. 1 slot for a filter (we do this even if there aren't any filters; size optimization opportunity to not do this if there are no filters?)
-4. 1 slot for zero termination
-
-Note that the since a slot on x86 is 4 bytes, the minimum size is 16 bytes. The idea is to have 1 slot for each handler that could be possibly be invoked at the same time. For example, for:
-
-```cs
-	try {
-		...
-	} catch {
-		try {
-			...
-		} catch {
-			...
-		}
-	}
-```
-
-When the inner 'catch' is running, the outer 'catch' is also conceptually "on the stack", or in the middle of execution. So the maximum handler nesting count would be 2.
-
-The ShadowSP slots are filled in from the highest address downwards to the lowest address. The highest slot is reserved. The first address with a zero is a zero terminator. So, we always zero terminate by setting the second-to-highest slot to zero in the function prolog (if we didn't zero initialize all locals anyway).
-
-When calling a finally, we set the appropriate level to 0xFC (aka "finally call") and zero terminate the next-lower address.
-
-Thus, calling a finally from JIT generated code looks like:
-
-```asm
-	mov      dword ptr [L_02+0x4 ebp-10H], 0 // This must happen before the 0xFC is written
-	mov      dword ptr [L_02+0x8 ebp-0CH], 252 // 0xFC
-	push     G_M52300_IG07
-	jmp      SHORT G_M52300_IG04
-```
-
-In this case, `G_M52300_IG07` is not the address after the 'jmp', so a simple 'call' wouldn't work.
-
-The code this finally returns to looks like this:
-
-```asm
-	mov      dword ptr [L_02+0x8 ebp-0CH], 0
-	jmp      SHORT G_M52300_IG05
-```
-
-In this case, it zeros out the ShadowSP slot that it previously set to 0xFC, then jumps to the address that is the actual target of the leave from the finally.
-
-The JIT does this "end finally restore" by creating a GT_END_LFIN tree node, with the appropriate EH region ID as an operand, that generates this code.
-
-In the case of an exceptional 'finally' invocation, the VM sets up the 'return address' to whatever address it wants the JIT to return to.
-
-For catch handlers, the VM is completely in control of filling and reading the ShadowSP slots; the JIT just makes sure there is enough space.
-
-### ShadowSP slots frame location
-
-The ShadowSP slots are required to live in a very particular location, reported via the GC info header. Note that the GC info header does not contain an actual pointer or offset to the ShadowSP slots variable. Instead, the VM calculates the location from other data that does exist in the GC info header, as a negative offset from the EBP frame pointer (which must be established in functions with EH) using the function `GetFirstBaseSPslotPtr()` / `GetStartShadowSPSlotsOffset()`. The VM thus assumes the following frame layout:
-
-1. callee-saved registers <= EBP points to the top of this range
-2. GS cookie
-3. 1 slot if localloc is used (Saved localloc SP?)
-4. 1 slot for CORINFO_GENERICS_CTXT_FROM_PARAMTYPEARG -- assumed for any function with EH, to avoid adding a flag to the GC info about whether it exists or not.
-5. ShadowSP slots
-
-(note, these don't have to be in this order for this calculation, but they possibly do need to be in this order for other calculations.) See also `GetEndShadowSPSlotsOffset()`.
-
-The VM walks the ShadowSP slots in the function `GetHandlerFrameInfo()`, and sets it in various functions such as `EECodeManager::FixContext()`.
-
-### JIT implementation: finally
-
-An aside on the JIT implementation for x86.
-
-The JIT creates BBJ_CALLFINALLY/BBJ_ALWAYS pairs for calling the 'finally' clause. The BBJ_CALLFINALLY block will have a series of CORINFO_JIT_ENDCATCH calls appended at the end, if we need to "leave" a series of nested catches before calling the finally handler (due to a single 'leave' opcode attempting to leave multiple levels of different types of handlers). Then, a GT_END_LFIN statement with EH region ID as an argument is added to the step block where the finally returns to. This is used to generate code to zero out the appropriate level of the ShadowSP slot array after the finally has been executed and the final EH nesting depth is known. The BBJ_CALLFINALLY block itself generates the code to insert the 0xFC value into the ShadowSP slot array. If the 'finally' is invoked by the VM, in exceptional cases, then the VM itself updates the ShadowSP slot array before invoking the 'finally'.
-
-At the end of a finally or filter, a GT_RETFILT is inserted. For a finally, this is a TYP_VOID which is just a placeholder. For a filter, it takes an argument which evaluates to the return value from the filter. On legacy JIT, this tree triggers the generation of both the return value load (for filters) and the "funclet" exit sequence, which is either a "pop eax; jmp eax" for a finally, or a "ret" for a filter. When processing the BBJ_EHFINALLYRET or BBJ_EHFILTERRET block itself (at the end of code generation for the block), nothing is generated. In RyuJIT, the GT_RETFILT only loads up the return value (for filters) and does nothing for finally, and the block type processing after all the tree processing triggers the exit sequence to be generated. There is no real difference between these, except to centralize all "exit sequence" generation in the same place.
-
 # EH Info, GC Info, and Hot & Cold Splitting
 
 All GC info offsets and EH info offsets treat the function and funclets as if it was one big method body. Thus all offsets are relative to the start of the main method. Funclets are assumed to always be at the end of (after) all of the main function code. Thus if the main function has any cold code, all funclets must be cold. Or conversely, if there is any hot funclet code, all of the main method must be hot.
diff --git a/src/coreclr/clr.featuredefines.props b/src/coreclr/clr.featuredefines.props
index 883d0bda0f5574..1e4050c8d56e0a 100644
--- a/src/coreclr/clr.featuredefines.props
+++ b/src/coreclr/clr.featuredefines.props
@@ -2,6 +2,7 @@
     <PropertyGroup>
         <FeatureCoreCLR>true</FeatureCoreCLR>
         <FeaturePerfTracing>true</FeaturePerfTracing>
+        <FeatureEHFunclets>true</FeatureEHFunclets>
         <ProfilingSupportedBuild>true</ProfilingSupportedBuild>
     </PropertyGroup>
 
@@ -22,10 +23,6 @@
         <FeatureObjCMarshal>true</FeatureObjCMarshal>
     </PropertyGroup>
 
-    <PropertyGroup Condition="!('$(TargetsWindows)' == 'true' AND '$(Platform)' == 'x86')">
-        <FeatureEHFunclets>true</FeatureEHFunclets>
-    </PropertyGroup>
-
     <PropertyGroup Condition="('$(Platform)' == 'x64' OR '$(Platform)' == 'arm64') AND ('$(Configuration)' == 'debug' OR '$(Configuration)' == 'checked')">
         <FeatureInterpreter>true</FeatureInterpreter>
     </PropertyGroup>
diff --git a/src/coreclr/clrdefinitions.cmake b/src/coreclr/clrdefinitions.cmake
index c7f6ae41d4d8aa..ee5fba323c9c6d 100644
--- a/src/coreclr/clrdefinitions.cmake
+++ b/src/coreclr/clrdefinitions.cmake
@@ -206,9 +206,7 @@ if(CLR_CMAKE_TARGET_WIN32)
   endif(CLR_CMAKE_TARGET_ARCH_AMD64 OR CLR_CMAKE_TARGET_ARCH_I386)
 endif(CLR_CMAKE_TARGET_WIN32)
 
-if (NOT CLR_CMAKE_TARGET_ARCH_I386 OR NOT CLR_CMAKE_TARGET_WIN32)
-  add_compile_definitions($<$<NOT:$<BOOL:$<TARGET_PROPERTY:IGNORE_DEFAULT_TARGET_ARCH>>>:FEATURE_EH_FUNCLETS>)
-endif (NOT CLR_CMAKE_TARGET_ARCH_I386 OR NOT CLR_CMAKE_TARGET_WIN32)
+add_compile_definitions($<$<NOT:$<BOOL:$<TARGET_PROPERTY:IGNORE_DEFAULT_TARGET_ARCH>>>:FEATURE_EH_FUNCLETS>)
 
 if (CLR_CMAKE_TARGET_WIN32 AND (CLR_CMAKE_TARGET_ARCH_AMD64 OR CLR_CMAKE_TARGET_ARCH_ARM64))
   add_definitions(-DFEATURE_SPECIAL_USER_MODE_APC)
diff --git a/src/coreclr/inc/gc_unwind_x86.h b/src/coreclr/inc/gc_unwind_x86.h
index 46d44afec4f6c1..74fe4e753972a0 100644
--- a/src/coreclr/inc/gc_unwind_x86.h
+++ b/src/coreclr/inc/gc_unwind_x86.h
@@ -409,4 +409,9 @@ bool IsInNoGCRegion(hdrInfo   * infoPtr,
                     PTR_CBYTE   table,
                     unsigned    curOffset);
 
+unsigned FindFirstInterruptiblePoint(hdrInfo   * infoPtr,
+                                     PTR_CBYTE   table,
+                                     unsigned    offs,
+                                     unsigned    endOffs);
+
 #endif // _UNWIND_X86_H
diff --git a/src/coreclr/inc/jiteeversionguid.h b/src/coreclr/inc/jiteeversionguid.h
index 5e5f8eb5637a4c..7f6aa4cdc66f1e 100644
--- a/src/coreclr/inc/jiteeversionguid.h
+++ b/src/coreclr/inc/jiteeversionguid.h
@@ -37,11 +37,11 @@
 
 #include <minipal/guid.h>
 
-constexpr GUID JITEEVersionIdentifier = { /* 7ce8764d-ac60-4e05-a6e4-448c1eb8cf35 */
-    0x7ce8764d,
-    0xac60,
-    0x4e05,
-    {0xa6, 0xe4, 0x44, 0x8c, 0x1e, 0xb8, 0xcf, 0x35}
+constexpr GUID JITEEVersionIdentifier = { /* cb23fc5b-e31d-49cb-b4e0-b5666496b4fe */
+    0xcb23fc5b,
+    0xe31d,
+    0x49cb,
+    {0xb4, 0xe0, 0xb5, 0x66, 0x64, 0x96, 0xb4, 0xfe}
   };
 
 #endif // JIT_EE_VERSIONING_GUID_H
diff --git a/src/coreclr/inc/readytorun.h b/src/coreclr/inc/readytorun.h
index a86b6ad7838b69..6aaa43e9175e96 100644
--- a/src/coreclr/inc/readytorun.h
+++ b/src/coreclr/inc/readytorun.h
@@ -19,10 +19,15 @@
 //  src/coreclr/nativeaot/Runtime/inc/ModuleHeaders.h
 // If you update this, ensure you run `git grep MINIMUM_READYTORUN_MAJOR_VERSION`
 // and handle pending work.
-#define READYTORUN_MAJOR_VERSION 13
-#define READYTORUN_MINOR_VERSION 0x0001
+#define READYTORUN_MAJOR_VERSION 14
+#define READYTORUN_MINOR_VERSION 0x0000
 
+// Remove the x86 special case once the general minimum version is bumped
+#ifdef TARGET_X86
+#define MINIMUM_READYTORUN_MAJOR_VERSION 14
+#else
 #define MINIMUM_READYTORUN_MAJOR_VERSION 13
+#endif
 
 // R2R Version 2.1 adds the InliningInfo section
 // R2R Version 2.2 adds the ProfileDataInfo section
@@ -43,6 +48,8 @@
 // R2R Version 13 removes usage of PSPSym, changes ABI for funclets to match NativeAOT, changes register for
 //                exception parameter on AMD64, and redefines generics instance context stack slot in GCInfo v4
 //                to be SP/FP relative
+// R2R Version 13.1 added long/ulong to float helper calls
+// R2R Version 14 changed x86 code generation to use funclets
 
 struct READYTORUN_CORE_HEADER
 {
diff --git a/src/coreclr/jit/emitinl.h b/src/coreclr/jit/emitinl.h
index 02aab9d4ed910e..b3e5fd0246ccf6 100644
--- a/src/coreclr/jit/emitinl.h
+++ b/src/coreclr/jit/emitinl.h
@@ -585,15 +585,15 @@ inline bool insIsCMOV(instruction ins)
  *  false. Returns the final result of the callback.
  */
 template <typename Callback>
-bool emitter::emitGenNoGCLst(Callback& cb, bool skipAllPrologsAndEpilogs /* = false */)
+bool emitter::emitGenNoGCLst(Callback& cb, bool skipMainPrologsAndEpilogs /* = false */)
 {
     for (insGroup* ig = emitIGlist; ig; ig = ig->igNext)
     {
-        if (skipAllPrologsAndEpilogs)
+        if (skipMainPrologsAndEpilogs)
         {
             if (ig == emitPrologIG)
                 continue;
-            if (ig->igFlags & (IGF_EPILOG | IGF_FUNCLET_PROLOG | IGF_FUNCLET_EPILOG))
+            if (ig->igFlags & IGF_EPILOG)
                 continue;
         }
         if ((ig->igFlags & IGF_NOGCINTERRUPT) && ig->igSize > 0)
diff --git a/src/coreclr/jit/emitpub.h b/src/coreclr/jit/emitpub.h
index 5725c04e6f6352..037ea04bccf6ee 100644
--- a/src/coreclr/jit/emitpub.h
+++ b/src/coreclr/jit/emitpub.h
@@ -43,7 +43,7 @@ unsigned emitEndCodeGen(Compiler*         comp,
 unsigned emitGetEpilogCnt();
 
 template <typename Callback>
-bool emitGenNoGCLst(Callback& cb, bool skipAllPrologsAndEpilogs = false);
+bool emitGenNoGCLst(Callback& cb, bool skipMainPrologsAndEpilogs = false);
 
 void     emitBegProlog();
 unsigned emitGetPrologOffsetEstimate();
diff --git a/src/coreclr/jit/gcencode.cpp b/src/coreclr/jit/gcencode.cpp
index a07279ee98bb60..f1eb7cec6b2250 100644
--- a/src/coreclr/jit/gcencode.cpp
+++ b/src/coreclr/jit/gcencode.cpp
@@ -2223,7 +2223,7 @@ size_t GCInfo::gcMakeRegPtrTable(BYTE* dest, int mask, const InfoHdr& header, un
     if (header.noGCRegionCnt != 0)
     {
         NoGCRegionEncoder encoder(mask != 0 ? dest : NULL);
-        compiler->GetEmitter()->emitGenNoGCLst(encoder, /* skipAllPrologsAndEpilogs = */ true);
+        compiler->GetEmitter()->emitGenNoGCLst(encoder, /* skipMainPrologsAndEpilogs = */ true);
         totalSize += encoder.totalSize;
         if (mask != 0)
             dest += encoder.totalSize;
diff --git a/src/coreclr/jit/gcinfo.cpp b/src/coreclr/jit/gcinfo.cpp
index 3f6f5169d3950a..ee08413e714c83 100644
--- a/src/coreclr/jit/gcinfo.cpp
+++ b/src/coreclr/jit/gcinfo.cpp
@@ -585,7 +585,7 @@ void GCInfo::gcCountForHeader(UNALIGNED unsigned int* pUntrackedCount,
     if (compiler->codeGen->GetInterruptible())
     {
         NoGCRegionCounter counter;
-        compiler->GetEmitter()->emitGenNoGCLst(counter, /* skipAllPrologsAndEpilogs = */ true);
+        compiler->GetEmitter()->emitGenNoGCLst(counter, /* skipMainPrologsAndEpilogs = */ true);
         noGCRegionCount = counter.noGCRegionCount;
     }
 
diff --git a/src/coreclr/jit/targetx86.h b/src/coreclr/jit/targetx86.h
index e630a1ae842120..ac0130f5d274ce 100644
--- a/src/coreclr/jit/targetx86.h
+++ b/src/coreclr/jit/targetx86.h
@@ -52,9 +52,6 @@
                                            // target
   #define FEATURE_EH               1       // To aid platform bring-up, eliminate exceptional EH clauses (catch, filter,
                                            // filter-handler, fault) and directly execute 'finally' clauses.
-#if !defined(UNIX_X86_ABI)
-  #define FEATURE_EH_WINDOWS_X86   1       // Enable support for SEH regions
-#endif
   #define ETW_EBP_FRAMED           1       // if 1 we cannot use EBP as a scratch register and must create EBP based
                                            // frames for most methods
   #define CSE_CONSTS               1       // Enable if we want to CSE constants
diff --git a/src/coreclr/nativeaot/Runtime/inc/ModuleHeaders.h b/src/coreclr/nativeaot/Runtime/inc/ModuleHeaders.h
index f8efbe166ff219..2b7464aa4ea97c 100644
--- a/src/coreclr/nativeaot/Runtime/inc/ModuleHeaders.h
+++ b/src/coreclr/nativeaot/Runtime/inc/ModuleHeaders.h
@@ -11,8 +11,8 @@ struct ReadyToRunHeaderConstants
 {
     static const uint32_t Signature = 0x00525452; // 'RTR'
 
-    static const uint32_t CurrentMajorVersion = 13;
-    static const uint32_t CurrentMinorVersion = 1;
+    static const uint32_t CurrentMajorVersion = 14;
+    static const uint32_t CurrentMinorVersion = 0;
 };
 
 struct ReadyToRunHeader
diff --git a/src/coreclr/tools/Common/Internal/Runtime/ModuleHeaders.cs b/src/coreclr/tools/Common/Internal/Runtime/ModuleHeaders.cs
index a0d0370b253c33..e8fdf030893432 100644
--- a/src/coreclr/tools/Common/Internal/Runtime/ModuleHeaders.cs
+++ b/src/coreclr/tools/Common/Internal/Runtime/ModuleHeaders.cs
@@ -15,8 +15,8 @@ internal struct ReadyToRunHeaderConstants
     {
         public const uint Signature = 0x00525452; // 'RTR'
 
-        public const ushort CurrentMajorVersion = 13;
-        public const ushort CurrentMinorVersion = 1;
+        public const ushort CurrentMajorVersion = 14;
+        public const ushort CurrentMinorVersion = 0;
     }
 #if READYTORUN
 #pragma warning disable 0169
diff --git a/src/coreclr/vm/eetwain.cpp b/src/coreclr/vm/eetwain.cpp
index 55a595e4b14e20..8933ebd37117ae 100644
--- a/src/coreclr/vm/eetwain.cpp
+++ b/src/coreclr/vm/eetwain.cpp
@@ -1290,16 +1290,6 @@ bool EECodeManager::EnumGcRefs( PREGDISPLAY     pRD,
     if (relOffsetOverride != NO_OVERRIDE_OFFSET)
     {
         // We've been given an override offset for GC Info
-#ifdef _DEBUG
-        GcInfoDecoder _gcInfoDecoder(
-                            gcInfoToken,
-                            DECODE_CODE_LENGTH
-                      );
-
-        // We only use override offset for wantsReportOnlyLeaf
-        _ASSERTE(_gcInfoDecoder.WantsReportOnlyLeaf());
-#endif // _DEBUG
-
         curOffs = relOffsetOverride;
 
 #ifdef TARGET_ARM
@@ -2505,16 +2495,6 @@ bool InterpreterCodeManager::EnumGcRefs(PREGDISPLAY     pContext,
     if (relOffsetOverride != NO_OVERRIDE_OFFSET)
     {
         // We've been given an override offset for GC Info
-#ifdef _DEBUG
-        InterpreterGcInfoDecoder _gcInfoDecoder(
-                                                gcInfoToken,
-                                                DECODE_CODE_LENGTH
-                                                );
-
-        // We only use override offset for wantsReportOnlyLeaf
-        _ASSERTE(_gcInfoDecoder.WantsReportOnlyLeaf());
-#endif // _DEBUG
-
         curOffs = relOffsetOverride;
 
         LOG((LF_GCINFO, LL_INFO1000, "Adjusted GC reporting offset to provided override offset. Now reporting GC refs for %s at offset %04x.\n",
diff --git a/src/coreclr/vm/gc_unwind_x86.inl b/src/coreclr/vm/gc_unwind_x86.inl
index 62f18a693cf2c5..c92986d358ab24 100644
--- a/src/coreclr/vm/gc_unwind_x86.inl
+++ b/src/coreclr/vm/gc_unwind_x86.inl
@@ -737,6 +737,51 @@ bool IsInNoGCRegion(hdrInfo   * infoPtr,
     return false;
 }
 
+/*****************************************************************************/
+unsigned FindFirstInterruptiblePoint(hdrInfo   * infoPtr,
+                                     PTR_CBYTE   table,
+                                     unsigned    offs,
+                                     unsigned    endOffs)
+{
+    if (!infoPtr->interruptible)
+        return -1;
+
+    _ASSERTE(offs < endOffs);
+
+    // NOTE: We do not check for prologs and epilogs of the main function body. Only
+    // explicit no-GC regions are considered.
+
+    if (infoPtr->noGCRegionCnt == 0)
+        return offs;
+
+#if VERIFY_GC_TABLES
+    _ASSERTE(*castto(table, unsigned short *)++ == 0xBEEF);
+#endif
+
+    unsigned count = infoPtr->noGCRegionCnt;
+    while (count-- > 0) {
+        unsigned regionOffset = fastDecodeUnsigned(table);
+        if (offs < regionOffset)
+        {
+            // Offset is lower than the region start. Since the regions come in
+            // order we can assume that the offset is in GC safe region.
+            return offs;
+        }
+
+        unsigned regionSize = fastDecodeUnsigned(table);
+        if (offs - regionOffset < regionSize)
+        {
+            // Offset is inside the no-GC region, so move it past the region and
+            // check if we still can match something in next iteration.
+            offs = regionOffset + regionSize;
+            if (offs >= endOffs)
+                return -1;
+        }
+    }
+
+    return offs;
+}
+
 /*****************************************************************************/
 static
 PTR_CBYTE skipToArgReg(const hdrInfo& info, PTR_CBYTE table)
@@ -3676,7 +3721,15 @@ bool EnumGcRefsX86(PREGDISPLAY     pContext,
     // Filters are the only funclet that run during the 1st pass, and must have
     // both the leaf and the parent frame reported.  In order to avoid double
     // reporting of the untracked variables, do not report them for the filter.
-    if (!isFilterFunclet)
+    if (isFilterFunclet)
+    {
+        count = info.untrackedCnt;
+        while (count-- > 0)
+        {
+            fastSkipSigned(table);
+        }
+    }
+    else
 #endif // FEATURE_EH_FUNCLETS
     {
         count = info.untrackedCnt;
@@ -3734,7 +3787,6 @@ bool EnumGcRefsX86(PREGDISPLAY     pContext,
                                               info.ebpFrame ? EBP - ptrAddr : ptrAddr - ESP,
                                               true)));
         }
-
     }
 
 #if VERIFY_GC_TABLES
diff --git a/src/coreclr/vm/gcenv.ee.common.cpp b/src/coreclr/vm/gcenv.ee.common.cpp
index 21723724cc5f02..0df511d2eadcbe 100644
--- a/src/coreclr/vm/gcenv.ee.common.cpp
+++ b/src/coreclr/vm/gcenv.ee.common.cpp
@@ -5,7 +5,8 @@
 #include "gcenv.h"
 #include <exinfo.h>
 
-#if defined(FEATURE_EH_FUNCLETS) && defined(USE_GC_INFO_DECODER)
+#if defined(FEATURE_EH_FUNCLETS)
+#if defined(USE_GC_INFO_DECODER)
 
 struct FindFirstInterruptiblePointState
 {
@@ -70,8 +71,22 @@ unsigned FindFirstInterruptiblePoint(CrawlFrame* pCF, unsigned offs, unsigned en
 
     return state.returnOffs;
 }
+#else // USE_GC_INFO_DECODER
 
-#endif // FEATURE_EH_FUNCLETS && USE_GC_INFO_DECODER
+// Find the first interruptible point in the range [offs .. endOffs) (the beginning of the range is inclusive,
+// the end is exclusive). Return -1 if no such point exists.
+unsigned FindFirstInterruptiblePoint(CrawlFrame* pCF, unsigned offs, unsigned endOffs)
+{
+    EECodeInfo *codeInfo = pCF->GetCodeInfo();
+    hdrInfo info = { 0 };
+    PTR_CBYTE table = codeInfo->DecodeGCHdrInfo(&info, offs);
+    // NOTE: We make an assumption that we are not searching for the main function so we can ignore the
+    // prolog/epilog info.
+    return ::FindFirstInterruptiblePoint(&info, table, offs, endOffs);
+}
+
+#endif
+#endif // FEATURE_EH_FUNCLETS
 
 //-----------------------------------------------------------------------------
 // Determine whether we should report the generic parameter context
@@ -317,37 +332,27 @@ StackWalkAction GcStackCrawlCallBack(CrawlFrame* pCF, VOID* pData)
     #endif // _DEBUG
 
             DWORD relOffsetOverride = NO_OVERRIDE_OFFSET;
-#if defined(FEATURE_EH_FUNCLETS) && defined(USE_GC_INFO_DECODER)
+#if defined(FEATURE_EH_FUNCLETS)
             if (pCF->ShouldParentToFuncletUseUnwindTargetLocationForGCReporting())
             {
-                GCInfoToken gcInfoToken = pCF->GetGCInfoToken();
-                GcInfoDecoder _gcInfoDecoder(
-                                    gcInfoToken,
-                                    DECODE_CODE_LENGTH
-                                    );
-
-                if(_gcInfoDecoder.WantsReportOnlyLeaf())
-                {
-                    // We're in a special case of unwinding from a funclet, and resuming execution in
-                    // another catch funclet associated with same parent function. We need to report roots.
-                    // Reporting at the original throw site gives incorrect liveness information. We choose to
-                    // report the liveness information at the first interruptible instruction of the catch funclet
-                    // that we are going to execute. We also only report stack slots, since no registers can be
-                    // live at the first instruction of a handler, except the catch object, which the VM protects
-                    // specially. If the catch funclet has not interruptible point, we fall back and just report
-                    // what we used to: at the original throw instruction. This might lead to bad GC behavior
-                    // if the liveness is not correct.
-                    const EE_ILEXCEPTION_CLAUSE& ehClauseForCatch = pCF->GetEHClauseForCatch();
-                    relOffsetOverride = FindFirstInterruptiblePoint(pCF, ehClauseForCatch.HandlerStartPC,
-                                                                    ehClauseForCatch.HandlerEndPC);
-                    _ASSERTE(relOffsetOverride != NO_OVERRIDE_OFFSET);
-
-                    STRESS_LOG3(LF_GCROOTS, LL_INFO1000, "Setting override offset = %u for method %pM ControlPC = %p\n",
-                        relOffsetOverride, pMD, GetControlPC(pCF->GetRegisterSet()));
-                }
-
+                // We're in a special case of unwinding from a funclet, and resuming execution in
+                // another catch funclet associated with same parent function. We need to report roots.
+                // Reporting at the original throw site gives incorrect liveness information. We choose to
+                // report the liveness information at the first interruptible instruction of the catch funclet
+                // that we are going to execute. We also only report stack slots, since no registers can be
+                // live at the first instruction of a handler, except the catch object, which the VM protects
+                // specially. If the catch funclet has not interruptible point, we fall back and just report
+                // what we used to: at the original throw instruction. This might lead to bad GC behavior
+                // if the liveness is not correct.
+                const EE_ILEXCEPTION_CLAUSE& ehClauseForCatch = pCF->GetEHClauseForCatch();
+                relOffsetOverride = FindFirstInterruptiblePoint(pCF, ehClauseForCatch.HandlerStartPC,
+                                                                ehClauseForCatch.HandlerEndPC);
+                _ASSERTE(relOffsetOverride != NO_OVERRIDE_OFFSET);
+
+                STRESS_LOG3(LF_GCROOTS, LL_INFO1000, "Setting override offset = %u for method %pM ControlPC = %p\n",
+                    relOffsetOverride, pMD, GetControlPC(pCF->GetRegisterSet()));
             }
-#endif // FEATURE_EH_FUNCLETS && USE_GC_INFO_DECODER
+#endif // FEATURE_EH_FUNCLETS
 
             pCM->EnumGcRefs(pCF->GetRegisterSet(),
                             pCF->GetCodeInfo(),
diff --git a/src/coreclr/vm/gcinfodecoder.cpp b/src/coreclr/vm/gcinfodecoder.cpp
index f5e06efff64ef5..00bfa6b96f6d63 100644
--- a/src/coreclr/vm/gcinfodecoder.cpp
+++ b/src/coreclr/vm/gcinfodecoder.cpp
@@ -672,7 +672,7 @@ template <typename GcInfoEncoding> bool TGcInfoDecoder<GcInfoEncoding>::HasTailC
 template <typename GcInfoEncoding> bool TGcInfoDecoder<GcInfoEncoding>::WantsReportOnlyLeaf()
 {
     // Only AMD64 with JIT64 can return false here.
-#ifdef TARGET_AMD64
+#if defined(TARGET_AMD64) && defined(DECODE_OLD_FORMATS)
     return ((m_headerFlags & GC_INFO_WANTS_REPORT_ONLY_LEAF) != 0);
 #else
     return true;
diff --git a/src/coreclr/vm/stackwalk.cpp b/src/coreclr/vm/stackwalk.cpp
index 6e903da8dcae46..9216e1470c345b 100644
--- a/src/coreclr/vm/stackwalk.cpp
+++ b/src/coreclr/vm/stackwalk.cpp
@@ -1684,29 +1684,15 @@ StackWalkAction StackFrameIterator::Filter(void)
 ProcessFuncletsForGCReporting:
                 do
                 {
-                    // When enumerating GC references for "liveness" reporting, depending upon the architecture,
-                    // the responsibility of who reports what varies:
+                    // The funclet reports all references belonging to itself and its parent method.
                     //
-                    // 1) On ARM, ARM64, and X64 (using RyuJIT), the funclet reports all references belonging
-                    //    to itself and its parent method. This is indicated by the WantsReportOnlyLeaf flag being
-                    //    set in the GC information for a function.
-                    //
-                    // 2) X64 (using JIT64) has the reporting distributed between the funclets and the parent method.
-                    //    If some reference(s) get double reported, JIT64 can handle that by playing conservative.
-                    //    JIT64 does NOT set the WantsReportOnlyLeaf flag in the function GC information.
-                    //
-                    // 3) On ARM, the reporting is done by funclets (if present). Otherwise, the primary method
-                    //    does it.
-                    //
-                    // 4) x86 behaves like (1)
-                    //
-                    // For non-x86, the GcStackCrawlCallBack is invoked with a new flag indicating that
-                    // the stackwalk is being done for GC reporting purposes - this flag is GC_FUNCLET_REFERENCE_REPORTING.
+                    // The GcStackCrawlCallBack is invoked with a new flag indicating that the stackwalk is being done
+                    // for GC reporting purposes - this flag is GC_FUNCLET_REFERENCE_REPORTING.
                     // The presence of this flag influences how the stackwalker will enumerate frames; which frames will
                     // result in the callback being invoked; etc. The idea is that we want to report only the
                     // relevant frames via the callback that are active on the callstack. This removes the need to
-                    // double report (even though JIT64 does it), reporting of dead frames, and makes the
-                    // design of reference reporting more consistent (and easier to understand) across architectures.
+                    // double report, reporting of dead frames, and makes the design of reference reporting more
+                    // consistent (and easier to understand) across architectures.
                     //
                     // The algorithm is as follows (at a conceptual level):
                     //
@@ -1726,7 +1712,6 @@ StackWalkAction StackFrameIterator::Filter(void)
                     //
                     // Note: When a flag is passed to the callback indicating that the funclet for a parent frame has already
                     //       reported the references, RyuJIT will simply do nothing and return from the callback.
-                    //       JIT64, on the other hand, will ignore the flag and perform reporting (again).
                     //
                     // Note: For non-filter funclets there is a small window during unwind where we have conceptually unwound past a
                     //       funclet but have not yet reached the parent/handling frame.  In this case we might need the parent to
@@ -3153,9 +3138,19 @@ void StackFrameIterator::PostProcessingForManagedFrames(void)
 #endif // ELIMINATE_FEF
 
 #ifdef TARGET_X86
+#ifdef FEATURE_EH_FUNCLETS
+    bool hasReversePInvoke = false;
+    if (!m_crawl.codeInfo.IsFunclet())
+    {
+        hdrInfo *gcHdrInfo;
+        m_crawl.codeInfo.DecodeGCHdrInfo(&gcHdrInfo);
+        hasReversePInvoke = gcHdrInfo->revPInvokeOffset != INVALID_REV_PINVOKE_OFFSET;
+    }
+#else
     hdrInfo *gcHdrInfo;
     m_crawl.codeInfo.DecodeGCHdrInfo(&gcHdrInfo);
     bool hasReversePInvoke = gcHdrInfo->revPInvokeOffset != INVALID_REV_PINVOKE_OFFSET;
+#endif // FEATURE_EH_FUNCLETS
 #endif // TARGET_X86
 
     ProcessIp(GetControlPC(m_crawl.pRD));


Reviews:
user: login: jkotas
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-24T16:30:10Z
user: login: filipnavara
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-24T16:32:16Z
user: login: jkotas
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-25T14:32:01Z
user: login: filipnavara
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-25T14:47:43Z
user: login: jkotas
body: Thank you for making this happen! Great work!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-26T08:40:15Z

Review comments:
diff_hunk: @@ -585,15 +585,15 @@ inline bool insIsCMOV(instruction ins)
  *  false. Returns the final result of the callback.
  */
 template <typename Callback>
-bool emitter::emitGenNoGCLst(Callback& cb, bool skipAllPrologsAndEpilogs /* = false */)
+bool emitter::emitGenNoGCLst(Callback& cb, bool skipMainPrologsAndEpilogs /* = false */)
 {
     for (insGroup* ig = emitIGlist; ig; ig = ig->igNext)
     {
-        if (skipAllPrologsAndEpilogs)
+        if (skipMainPrologsAndEpilogs)
         {
             if (ig == emitPrologIG)
                 continue;
-            if (ig->igFlags & (IGF_EPILOG | IGF_FUNCLET_PROLOG | IGF_FUNCLET_EPILOG))
+            if (ig->igFlags & IGF_EPILOG)
path: src/coreclr/jit/emitinl.h
user: login: jkotas
body: Is this going to add no-gc regions on all architectures or on x86 only?
created_at: 2025-05-24T16:30:09Z
author_association: MEMBER
diff_hunk: @@ -585,15 +585,15 @@ inline bool insIsCMOV(instruction ins)
  *  false. Returns the final result of the callback.
  */
 template <typename Callback>
-bool emitter::emitGenNoGCLst(Callback& cb, bool skipAllPrologsAndEpilogs /* = false */)
+bool emitter::emitGenNoGCLst(Callback& cb, bool skipMainPrologsAndEpilogs /* = false */)
 {
     for (insGroup* ig = emitIGlist; ig; ig = ig->igNext)
     {
-        if (skipAllPrologsAndEpilogs)
+        if (skipMainPrologsAndEpilogs)
         {
             if (ig == emitPrologIG)
                 continue;
-            if (ig->igFlags & (IGF_EPILOG | IGF_FUNCLET_PROLOG | IGF_FUNCLET_EPILOG))
+            if (ig->igFlags & IGF_EPILOG)
path: src/coreclr/jit/emitinl.h
user: login: filipnavara
body: This code path (`skipMainPrologsAndEpilogs == true`) is only used by the x86 GC info encoder.
created_at: 2025-05-24T16:32:15Z
author_association: MEMBER
diff_hunk: @@ -317,16 +332,21 @@ StackWalkAction GcStackCrawlCallBack(CrawlFrame* pCF, VOID* pData)
     #endif // _DEBUG
 
             DWORD relOffsetOverride = NO_OVERRIDE_OFFSET;
-#if defined(FEATURE_EH_FUNCLETS) && defined(USE_GC_INFO_DECODER)
+#if defined(FEATURE_EH_FUNCLETS)
             if (pCF->ShouldParentToFuncletUseUnwindTargetLocationForGCReporting())
             {
+                bool wantsReportOnlyLeaf = true;
path: src/coreclr/vm/gcenv.ee.common.cpp
user: login: jkotas
body: Nit: This is always set by RyuJIT. We can delete all places that check it in the runtime and change this

https://github.com/dotnet/runtime/blob/8115429a72fa599584135f217f3a34bd4aef8cf4/src/coreclr/vm/gcinfodecoder.cpp#L675

to `#if defined(TARGET_AMD64) && defined(DECODE_OLD_FORMATS)`.
created_at: 2025-05-25T14:32:01Z
author_association: MEMBER
diff_hunk: @@ -317,16 +332,21 @@ StackWalkAction GcStackCrawlCallBack(CrawlFrame* pCF, VOID* pData)
     #endif // _DEBUG
 
             DWORD relOffsetOverride = NO_OVERRIDE_OFFSET;
-#if defined(FEATURE_EH_FUNCLETS) && defined(USE_GC_INFO_DECODER)
+#if defined(FEATURE_EH_FUNCLETS)
             if (pCF->ShouldParentToFuncletUseUnwindTargetLocationForGCReporting())
             {
+                bool wantsReportOnlyLeaf = true;
path: src/coreclr/vm/gcenv.ee.common.cpp
user: login: filipnavara
body: Sounds reasonable. It will likely need some documentation/comments update too.
created_at: 2025-05-25T14:47:42Z
author_association: MEMBER

PR comments:
user: login: jkotas
created_at: 2025-05-24T04:41:20Z
body: /azp run runtime-coreclr outerloop

author_association: MEMBER
user: login: jkotas
created_at: 2025-05-24T04:41:25Z
body: /azp run runtime-coreclr gcstress0x3-gcstress0xc
author_association: MEMBER
user: login: azure-pipelines[bot]
created_at: 2025-05-24T04:41:31Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: azure-pipelines[bot]
created_at: 2025-05-24T04:41:34Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: am11
created_at: 2025-05-24T07:56:14Z
body: `!CREATE_CHECK_STRING(pMT && pMT->Validate())` is https://github.com/dotnet/runtime/issues/110512.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-24T08:08:37Z
body: > `!CREATE_CHECK_STRING(pMT && pMT->Validate())` is #110512.

I'm not so sure it's the same root cause. I'll look at the dumps later on.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-24T10:23:31Z
body: The JIT.opt stress test is failing when reporting GC references at the [`nop` instruction in funclet prolog](https://github.com/dotnet/runtime/blob/cf5369440e236aa783599761750b4f8e1f4aa719/src/coreclr/jit/codegenxarch.cpp#L11053-L11058) that was added in #115630.

More than one value in the dump seems off, but at least it explains why I didn't see this in earlier gcstress pass.

Disassembly:
```
0eedf182 59             pop     ecx
0eedf183 5b             pop     ebx
0eedf184 5e             pop     esi
0eedf185 5f             pop     edi
0eedf186 5d             pop     ebp
0eedf187 c3             ret     
0eedf188 90             nop                                       <--- GC stress here
0eedf189 8945f0         mov     dword ptr [ebp-10h], eax
0eedf18c 8d0517a2d10e   lea     eax, ds:[0ED1A217h]
0eedf192 c3             ret     
0eedf193 0000           add     byte ptr [eax], al
0eedf195 0000           add     byte ptr [eax], al
0eedf197 00cf           add     bh, cl
```

Locals in EnumGCRefsX86:
```
methodStart = 0x000000000eedf0d8 <-- likely wrong
curOffs = 0xb0
funcletStart = 0x000000000ed1a220
```

GC info:
```
!ip2md 0x000000000ed1a220
MethodDesc:   097098f0
Method Name:          Xunit.Assert.RecordException(System.Func`1<System.Object>, System.String)
Class:                0970afb8
MethodTable:          0970afb8
mdToken:              060000CE
Module:               09707674
IsJitted:             yes
Current CodeAddr:     0ed1a170
Version History:
  ILCodeVersion:      00000000
  ReJIT ID:           0
  IL Addr:            09735450
     CodeAddr:           0ed1a170  (Optimized)
     NativeCodeVersion:  00000000
Source file:  /_/src/arcade/src/Microsoft.DotNet.XUnitAssert/src/Record.cs @ 72
0:000> !gcinfo 097098f0
entry point 0ED1A170
Normal JIT generated code
GC info 0EEAE34C
Method info block:
    method      size   = 00BB
    prolog      size   = 11 
    epilog      size   =  6 
    epilog     count   =  2 
    epilog      end    = no  
    callee-saved regs  = EDI ESI EBX EBP 
    ebp frame          = yes  
    fully interruptible= yes  
    double align       = no  
    arguments size     =  0 DWORDs
    stack frame size   =  1 DWORDs
    untracked count    =  0 
    var ptr tab count  =  2 
    exception handlers = yes
    epilog # 0    at   0027
    epilog # 1    at   00AA
    argTabOffset = 7  
81 3B AB B1 9C | 
BF C3 CF 01 02 | 
27 81 03 07    | 

Pointer table:
10 81 27 0A    | 00A7..00B1  [EBP-10H] a  pointer
10 0D 07    ...| 00B4..00BB  [EBP-10H] a  pointer
F0 41    4F ...| 0009        reg EAX becoming live
72    0B 52 ...| 000B        reg ESI becoming live
4F    52 00 ...| 0012        reg ECX becoming live
0B    00 F0 ...| 0015        reg ECX becoming dead
52    F0 42 ...| 0017        reg EDX becoming live
00    42 10 ...| 0017        reg EAX becoming dead
F0 42    F0 ...| 0021        reg EAX becoming live
10    04 30 ...| 0021        reg EDX becoming dead
F0 04    F0 ...| 002D        reg EAX becoming dead
30    42 72 ...| 002D        reg ESI becoming dead
F0 42    F1 ...| 0037        reg EAX becoming live
72    49 F0 ...| 0039        reg ESI becoming live
F1 49    0B ...| 004A        reg ECX becoming live
F0 0B    4A ...| 0055        reg ECX becoming dead
52    06 08 ...| 0057        reg EDX becoming live
4A    08 10 ...| 0059        reg ECX becoming live
06    10 4A ...| 005F        reg EAX becoming dead
08    4A 0D ...| 005F        reg ECX becoming dead
10    0D 30 ...| 005F        reg EDX becoming dead
4A    30 71 ...| 0061        reg ECX becoming live
0D    71 F0 ...| 0066        reg ECX becoming dead
30    F0 42 ...| 0066        reg ESI becoming dead
71    42 7A ...| 0067        reg ESI becoming live
F0 42    F0 ...| 0071        reg EAX becoming live
7A    58 F1 ...| 0073        reg EDI becoming live
F0 58    51 ...| 007B        reg EBX becoming live
F1 51    4A ...| 008C        reg EDX becoming live
81    0E 10 ...| 008D        push ptr  0
4A    10 18 ...| 008F        reg ECX becoming live
0E    18 30 ...| 0095        reg ECX becoming dead
10    30 C8 ...| 0095        reg EDX becoming dead
18    C8 52 ...| 0095        reg EBX becoming dead
30    52 4A ...| 0095        reg ESI becoming dead
C8    4A 06 ...| 0095        pop  1 ptrs
52    06 08 ...| 0097        reg EDX becoming live
4A    08 10 ...| 0099        reg ECX becoming live
06    10 4A ...| 009F        reg EAX becoming dead
08    4A 0D ...| 009F        reg ECX becoming dead
10    0D 38 ...| 009F        reg EDX becoming dead
4A    38 44 ...| 00A1        reg ECX becoming live
0D    44 F1 ...| 00A6        reg ECX becoming dead
38    F1 00 ...| 00A6        reg EDI becoming dead
44    00 FF ...| 00AA        reg EAX becoming live
F1 00    4C ...| 00BA        reg EAX becoming dead
FF    00 00 ...| 
```

Presumably it's trying to report `10 81 27 0A    | 00A7..00B1  [EBP-10H] a  pointer` at the time of the crash which is indeed not valid yet. It only becomes valid on the next instruction.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-24T11:04:28Z
body: Obligatory JIT dump for the above: https://gist.githubusercontent.com/filipnavara/87489fc9b6224706111055ce1ee79583/raw/80122358359dbe95e53fd7708c59611f7024aa87/gistfile1.txt
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-24T12:46:47Z
body: I'm starting to suspect that the liveness of the variable may be misreported on other platforms too. We just get away with it because the funclet prolog is marked as `nogc` which excludes it from GC stress. Until recently we didn't have the capability to report these no-GC regions on x86 and to keep the GC info small I opted to never emit it for any prolog/epilog, main function or funclet. We can change that to emit it for funclet prologs/epilogs but I would like someone to look at the liveness issue first before patching it up with a bandaid. /cc @dotnet/jit-contrib 

x86 JIT dump: https://gist.githubusercontent.com/filipnavara/87489fc9b6224706111055ce1ee79583/raw/80122358359dbe95e53fd7708c59611f7024aa87/gistfile1.txt
x64 JIT dump: https://gist.githubusercontent.com/filipnavara/c76975c784519ed718e1701a24e6a458/raw/88a2fc3e1eddf932d62065cd63d578a38eca3a11/gistfile1.txt
author_association: MEMBER
user: login: SingleAccretion
created_at: 2025-05-24T12:51:55Z
body: > We just get away with it because the funclet prolog is marked as nogc which excludes it from GC stress. Until recently we didn't have the capability to report these no-GC regions on x86 and to keep the GC info small I opted to never emit it for any prolog/epilog, main function or funclet. We can change that to emit it for funclet prologs/epilogs but I would like someone to look at the liveness issue first before patching it up with a bandaid.

It is an expected invariant that prolog/epilog are no-GC regions. Why did this issue not manifest for filter funclets in the old scheme (because filter-live variable are always pinned?)?
author_association: CONTRIBUTOR
user: login: filipnavara
created_at: 2025-05-24T13:05:57Z
body: > It is an expected invariant that prolog/epilog are no-GC regions.

Yeah, and that's fine. x86 GC info always encoded the (non-funclet) prologs/epilogs and treated them as no-GC on the VM side.

Until #115630 the funclet prolog was empty on win-x86 so we didn't run into this problem because there was no code in the no-GC region. Likewise, the funclet epilog is single "ret" instruction so we don't really run into problem either. (These assumptions are not true for linux-x86 which has additional stack alignment instruction and which would have likely run into this problem if someone ever got far enough to run GC stress tests.)

I did, however, expect that calling `gcInfo.gcResetForBB();` from `genFuncletProlog` would reset the liveness of the variables. I'm not sure that's actually happening (properly). Observably, the variable slot gets invalidated only before the `IN002d: 0000B1 mov      gword ptr [V06 ebp-0x10], eax` instruction and then it becomes valid again. I would have expected the previous validity of the slot to be already reset at the beginning of `genFuncletProlog` and then the new one to start at 0xB4 as it does. (UPD: Or rather, I would have expected the `genEpilog` to end the liveness; whoever does it, it's unexpected that the variable is alive past the `ret` instruction and into the following block.)

> Why did this issue not manifest for filter funclets in the old scheme (because filter-live variable are always pinned?)?

Not sure I know the precise answer. I remember stumbling upon some specific `JIT32_GCENCODER` blocks but cannot find it right now.

author_association: MEMBER
user: login: SingleAccretion
created_at: 2025-05-24T13:57:18Z
body: > (UPD: Or rather, I would have expected the genEpilog to end the liveness; whoever does it, it's unexpected that the variable is alive past the ret instruction and into the following block.)

Looks like we have a comment that this is indeed intentional:
```cpp
void CodeGen::genReserveFuncletProlog(BasicBlock* block)
{
    assert(compiler->UsesFunclets());
    assert(block != nullptr);

    /* Currently, no registers are live on entry to the prolog, except maybe
       the exception object. There might be some live stack vars, but they
       cannot be accessed until after the frame pointer is re-established.
       In order to potentially prevent emitting a death before the prolog
       and a birth right after it, we just report it as live during the
       prolog, and rely on the prolog being non-interruptible. Trust
       genCodeForBBlist to correctly initialize all the sets.

       We might need to relax these asserts if the VM ever starts
       restoring any registers, then we could have live-in reg vars...
    */

    noway_assert((gcInfo.gcRegGCrefSetCur & RBM_EXCEPTION_OBJECT) == gcInfo.gcRegGCrefSetCur);
    noway_assert(gcInfo.gcRegByrefSetCur == 0);

    JITDUMP("Reserving funclet prolog IG for block " FMT_BB "\n", block->bbNum);

    GetEmitter()->emitCreatePlaceholderIG(IGPT_FUNCLET_PROLOG, block, gcInfo.gcVarPtrSetCur, gcInfo.gcRegGCrefSetCur,
                                          gcInfo.gcRegByrefSetCur, false);
}
```
author_association: CONTRIBUTOR
user: login: filipnavara
created_at: 2025-05-24T14:01:43Z
body: > Looks like we have a comment that this is indeed intentional:

Thanks. I adjusted emitting the no-GC regions for funclet prologs/epilogs but I will look into this at some later point to see if we can improve the codegen. We no longer establish the frame pointer in funclet prologs on any platform so the reasoning makes little sense.
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-24T15:51:52Z
body: /azp run runtime-coreclr gcstress0x3-gcstress0xc

author_association: MEMBER
user: login: jkotas
created_at: 2025-05-24T15:51:55Z
body: /azp run runtime-coreclr outerloop
author_association: MEMBER
user: login: azure-pipelines[bot]
created_at: 2025-05-24T15:52:07Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: azure-pipelines[bot]
created_at: 2025-05-24T15:52:08Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: am11
created_at: 2025-05-24T19:04:30Z
body: https://github.com/dotnet/runtime/blob/d896e8518a6ddba11bb58634a19a6bcb2a001927/src/coreclr/vm/syncblk.cpp#L2019
failed:

```
> C:\h\w\A2A90912\w\AFCF0954\e\Interop\Interop\../BestFitMapping/BestFitMapping/BestFitMapping.cmd
Xunit.Sdk.TrueException: 
Assert failure(PID 9828 [0x00002664], Thread: 7420 [0x1cfc]): SyncTableEntry::GetSyncTableEntry()[sbIndex].m_Object == obj

CORECLR! ObjHeader::Validate + 0x147 (0x73f2686f)
CORECLR! Object::ValidateInner + 0x20F (0x73ed9e1f)
CORECLR! Object::Validate + 0x98 (0x73ed9bd8)
CORECLR! WKS::GCHeap::Promote + 0x8F (0x741fedaf)
CORECLR! GcEnumObject + 0x6F (0x73ff522f)
CORECLR! EnumGcRefsX86 + 0x10C6 (0x73e00776)
CORECLR! EECodeManager::EnumGcRefs + 0x19B (0x73dff66b)
CORECLR! GcStackCrawlCallBack + 0x2BC (0x73ff55ec)
CORECLR! Thread::MakeStackwalkerCallback + 0x4B (0x73f17fdd)
CORECLR! Thread::StackWalkFramesEx + 0xEC (0x73f1909f)
    File: D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp:2019
    Image: C:\h\w\A2A90912\p\corerun.exe
```
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-24T19:12:44Z
body: Yeah, I downloaded the artifacts for BestFitMapping to investigate. I didn't expect a clean run just yet but it's already looking better than the first attempt.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-24T20:53:13Z
body: The BestFitMapping is not like the other failures I have seen. The object is valid but it has a header pointing to sync block. The sync block table has 2 free entries. That's not really my area of expertise, so any advice is welcome.

The other failure - Interop/COM/NETClients/IDispatch/NETClientIDispatch/NETClientIDispatch - was not reproducible on my machine last time it occurred. However, it does happen consistently on the CI machines. I'll have another look.
author_association: MEMBER
user: login: am11
created_at: 2025-05-25T02:23:29Z
body: > The BestFitMapping is not like the other failures I have seen.

Previous discussion at https://github.com/dotnet/runtime/issues/74741.

> The other failure - Interop/COM/NETClients/IDispatch/NETClientIDispatch/NETClientIDispatch

Is it failing on x86? https://helix.dot.net/api/2019-06-17/jobs/05d45426-3dee-421e-a098-6215d618b5ce/workitems/Interop.0.1/console suggests it has passed.
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-25T03:11:55Z
body: > Interop/COM/NETClients/IDispatch/NETClientIDispatch/NETClientIDispatch

The failure log is here: https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-115957-merge-e222de0ef7c24526a4/Interop.0.1/1/console.0d3011b6.log

It is failing at:
```
Assert failure(PID 7312 [0x00001c90], Thread: 6796 [0x1a8c]): stkOffs < 0

CORECLR! EnumGcRefsX86 + 0x11B1 (0x72fd0861)
CORECLR! EECodeManager::EnumGcRefs + 0x19B (0x72fcf66b)
```

We are doing a GC in the middle of this filter: https://github.com/dotnet/runtime/blob/d896e8518a6ddba11bb58634a19a6bcb2a001927/src/libraries/System.Private.CoreLib/src/System/RuntimeType.cs#L384

My guess is that the following condition has something to do with it https://github.com/dotnet/runtime/blob/d896e8518a6ddba11bb58634a19a6bcb2a001927/src/coreclr/vm/gc_unwind_x86.inl#L3675-L3680
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-25T04:35:13Z
body: > My guess is that the following condition has something to do with it

That sounds extremely plausible. It fails to skip over the table data for untracked variables.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-25T04:50:27Z
body: > > The BestFitMapping is not like the other failures I have seen.
> 
> Previous discussion at #74741.

Interesting. It does sound awfully similar. The difference is that we are dealing with WKS GC here and that we are just about to mark the object, so the initial conditions are quite different. 
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-25T07:01:49Z
body: > Methodical_r1.0.1
 
Failure log: https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-115957-merge-e222de0ef7c24526a4/Methodical_r1.0.1/1/console.8c41ce38.log

Failing at:
```
04d2d03c 73ededaf coreclr!Object::Validate+0x98
04d2d05c 73cd522f coreclr!WKS::GCHeap::Promote+0x8f
04d2d078 73adfbdc coreclr!GcEnumObject+0x6f
04d2d3e0 73adf66b coreclr!EnumGcRefsX86+0x52c
04d2d48c 73cd55ec coreclr!EECodeManager::EnumGcRefs+0x19b
```

We just returned from fault funclet https://github.com/dotnet/runtime/blob/d896e8518a6ddba11bb58634a19a6bcb2a001927/src/tests/JIT/Methodical/Boxing/seh/fault.il#L447-L487 and GC triggered when returning from RhpCallFinallyFunclet back to main EH second pass loop https://github.com/dotnet/runtime/blob/d896e8518a6ddba11bb58634a19a6bcb2a001927/src/coreclr/nativeaot/Runtime.Base/src/System/Runtime/ExceptionHandling.cs#L1218-L1237.

The problem is that we are trying to report GC references based on the IP of call inside the try region (exception was thrown inside this call). These GC references are no longer valid https://github.com/dotnet/runtime/blob/d896e8518a6ddba11bb58634a19a6bcb2a001927/src/coreclr/vm/gc_unwind_x86.inl#L3412.

Fault funclets should work exactly same as finally funclets. Fault funclets are rare since they are not generated by C#. Are we missing check for the fault funclets somewhere?
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-25T07:16:33Z
body: > Failure log: https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-115957-merge-e222de0ef7c24526a4/Methodical_r1.0.1/1/console.8c41ce38.log

<strike>Isn't that from one of the earlier runs that didn't produce no-GC region for funclet prologs? The symptoms match it.</strike>

Nvm, I see it now.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-25T08:32:37Z
body: I suspect the Methodical_r1 test may be failing because this code path is not implemented for `!USE_GC_INFO_DECODER`:
https://github.com/dotnet/runtime/blob/d896e8518a6ddba11bb58634a19a6bcb2a001927/src/coreclr/vm/gcenv.ee.common.cpp#L320-L350

It would have been taken otherwise based on the state of the variables.
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-25T08:33:43Z
body: > BestFitMapping 

Log https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-115957-merge-05d454263dee421ea0/Interop.0.1/1/console.b2222a36.log

Failure: 
```
Assert failure(PID 9828 [0x00002664], Thread: 7420 [0x1cfc]): SyncTableEntry::GetSyncTableEntry()[sbIndex].m_Object == obj

CORECLR! ObjHeader::Validate + 0x147 (0x73f2686f)
CORECLR! Object::ValidateInner + 0x20F (0x73ed9e1f)
CORECLR! Object::Validate + 0x98 (0x73ed9bd8)
CORECLR! WKS::GCHeap::Promote + 0x8F (0x741fedaf)
CORECLR! GcEnumObject + 0x6F (0x73ff522f)
CORECLR! EnumGcRefsX86 + 0x10C6 (0x73e00776)
```

This assert is one of the typical GC hole symptoms. It is likely to be hit anytime we miss reporting an object with a sync block. The offending object in this case is a marshalled delegate. Marshalled delegates have syncblocks.

> The object is valid but it has a header pointing to sync block. 

It likely means that the object was collected, but the space was not reused by the GC for a new object yet. We likely missed reporting the slot in some earlier GC but reporting it again.

The callstack looks like this: BestFitMapping!ILStubClass.IL_STUB_PInvoke -> native code -> BestFitMapping!ILStubClass.IL_STUB_ReversePInvoke

- IL_STUB_ReversePInvoke has finally block. We just returned from calling the finally block (non-exceptionally).
- IL_STUB_PInvoke has a local that stores the invalid delegate object.

Let's take a look what `!DumpLog` says about the previous GC:
```
1cfc   1.930619591 : `GC`GCROOTS`         Ending scan of Thread 02FAD110 ID = 0x2 {
1cfc   1.930618407 : `GCROOTS`            STACKWALK: SKIPPING_TO_FUNCLET_PARENT: not making callback for this frame, SPOfParent = 00B7E8D4,                                      isILStub = 0, m_crawl.pFunc = 08AE0314 (__GeneratedMainWrapper.Main())
1cfc   1.930618280 : `EH`GCROOTS`         CrawlFrame (00B7BE54): Frameless: Yes CallerSP: 00B7E980
1cfc   1.930615468 : `GCROOTS`            STACKWALK: SKIPPING_TO_FUNCLET_PARENT: not making callback for this frame, SPOfParent = 00B7E8D4,                                      isILStub = 0, m_crawl.pFunc = 08AE2960 (BestFitMapping.TestEntryPoint())
1cfc   1.930615330 : `EH`GCROOTS`         CrawlFrame (00B7BE54): Frameless: Yes CallerSP: 00B7E968
1cfc   1.930612350 : `GCROOTS`            STACKWALK: SKIPPING_TO_FUNCLET_PARENT: not making callback for this frame, SPOfParent = 00B7E8D4,                                      isILStub = 1, m_crawl.pFunc = 08DA84E4 (ILStubClass.IL_STUB_PInvoke(SCallBackInOutByRef))
1cfc   1.930612200 : `EH`GCROOTS`         CrawlFrame (00B7BE54): Frameless: Yes CallerSP: 00B7E938
1cfc   1.930608805 : `GC`GCROOTS`             GC Root 00B7E8B8 RELOCATED 0553E480 -> 05538754  MT = 035B8FC4 (System.String)
1cfc   1.930606588 : `GCROOTS`            Scanning Frameless method 08DA89DC (ILStubClass.IL_STUB_ReversePInvoke(IntPtr*)) EIP = 04D973C7 &EIP = 00B7DE90
1cfc   1.930606158 : `EH`GCROOTS`         CrawlFrame (00B7BE54): Frameless: Yes CallerSP: 00B7E87C
1cfc   1.930605891 : `GCROOTS`            STACKWALK: Found Non-Filter funclet @ SP: 00B7E878, m_crawl.pFunc = 08DA89DC; FuncletParentCallerSP: 00B7E8D4
1cfc   1.930605836 : `EH`GCROOTS`         CrawlFrame (00B7BE54): Frameless: Yes CallerSP: 00B7E87C
1cfc   1.930599724 : `GCROOTS`            Scanning ExplicitFrame 00B7D0E8 AssocMethod = 00000000 FrameIdentifier = ResumableFrame
```
This GC was triggered in the finally funclet inside `IL_STUB_ReversePInvoke`. But then the stackwalk got confused and skipped reporting for `ILStubClass.IL_STUB_PInvoke`. That's the GC hole.
author_association: MEMBER
user: login: am11
created_at: 2025-05-25T08:38:14Z
body: BestFitMapping was fixed by https://github.com/dotnet/runtime/pull/115957/commits/b9cd5acf528efb703963e45bc3de8b01450ce541. Could you rerun gcstress?
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-25T08:44:16Z
body: > BestFitMapping was fixed by https://github.com/dotnet/runtime/commit/b9cd5acf528efb703963e45bc3de8b01450ce541.

This was a fix for Interop/COM/NETClients/IDispatch/NETClientIDispatch/NETClientIDispatch. I do not think BestFitMapping is fixed.
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-25T08:44:50Z
body: /azp run runtime-coreclr gcstress0x3-gcstress0xc
author_association: MEMBER
user: login: azure-pipelines[bot]
created_at: 2025-05-25T08:45:05Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: filipnavara
created_at: 2025-05-25T09:28:37Z
body: I have a fix locally for Methodical_r1.0.1. I'll push it after going through more tests and once the current GCStress run finishes.

UPD: Hmm, still needs some refining.

UPD2: As usual, the refining consisted of actually saving a file before starting the compilation... (https://github.com/dotnet/runtime/commit/ac4373f799cd0108948c11d6fb1d3dc3d9b6100f)
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-25T14:29:03Z
body: /azp run runtime-coreclr gcstress0x3-gcstress0xc
author_association: MEMBER
user: login: azure-pipelines[bot]
created_at: 2025-05-25T14:29:17Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: filipnavara
created_at: 2025-05-25T14:49:00Z
body: The BestFitMapping fails in like every 6th try on my machine when running in isolation (w/ GCStress=0xC and TieredCompilation=0). I am trying to capture a time travel trace to investigate it but didn't have luck so far.

UPD: Got lucky.

UPD2: <strike>When the `ILStubClass.IL_STUB_PInvoke` frame is skipped the value of `m_sfParent.SP == 0xbadddc`</strike> (the value is actually a valid one)

UPD3: I found what is the difference in behavior between x86 and x64. Need to run full battery of tests before I change it.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-25T19:12:16Z
body: The issue with `BestFitMapping` is the following:
- Let's start with the analysis in https://github.com/dotnet/runtime/pull/115957#issuecomment-2907691836
- There are two frames of the `ILStubClass.IL_STUB_ReversePInvoke` on the stack. The bottom one is an out-of-line finally, the one above is the trampoline that jumps to it. Next frame is the unmanaged code.
- When we start unwinding, we end up in the `StackFrameIterator::Filter` method in the branch that [sets `m_sfParent = m_sfFuncletParent;`](https://github.com/dotnet/runtime/blob/67a567fbe704655be5387f2fa3ca555d22c39844/src/coreclr/vm/stackwalk.cpp#L1838). The expectation is that this will turn a state where some frames are skipped for reporting until `m_sfParent` is reached. It will then resume the normal reporting. Unfortunately, the parent frame never gets iterated so this mechanism fails.
- The reason why the frame doesn't get iterated is the code introduce in https://github.com/dotnet/runtime/pull/58796/:
https://github.com/dotnet/runtime/blob/67a567fbe704655be5387f2fa3ca555d22c39844/src/coreclr/vm/stackwalk.cpp#L3155-L3180
The assumption is that if previous frame was "reverse P/Invoke" and next frame is managed that we skipped over the `InlinedCallFrame`. Unfortunately, the condition triggers early for the finally jump trampoline, so it reports the `InlinedCallFrame` but never reports the finally jump trampoline itself and that happens to be the one in `m_sfParent`.

<strike>I am not sure how to adequately detect the condition. I'm not inside a funclet so I cannot condition it on that (on second thought, maybe I should see it as funclet?!). I could save `mCrawl.pFunc` and detect that it didn't change but that would still be broken if `[UnmanagedCallersOnly]` method somehow managed to call itself.</strike>

author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-25T19:42:02Z
body: Hmm, this does actually make a difference:
```diff
--- a/src/coreclr/vm/stackwalk.cpp
+++ b/src/coreclr/vm/stackwalk.cpp
@@ -3153,9 +3153,19 @@ void StackFrameIterator::PostProcessingForManagedFrames(void)
 #endif // ELIMINATE_FEF

 #ifdef TARGET_X86
+#ifdef FEATURE_EH_FUNCLETS
+    bool hasReversePInvoke = false;
+    if (!m_crawl.codeInfo.IsFunclet())
+    {
+        hdrInfo *gcHdrInfo;
+        m_crawl.codeInfo.DecodeGCHdrInfo(&gcHdrInfo);
+        hasReversePInvoke = gcHdrInfo->revPInvokeOffset != INVALID_REV_PINVOKE_OFFSET;
+    }
+#else
     hdrInfo *gcHdrInfo;
     m_crawl.codeInfo.DecodeGCHdrInfo(&gcHdrInfo);
     bool hasReversePInvoke = gcHdrInfo->revPInvokeOffset != INVALID_REV_PINVOKE_OFFSET;
+#endif // FEATURE_EH_FUNCLETS
 #endif // TARGET_X86

     ProcessIp(GetControlPC(m_crawl.pRD));
```
...but I still get a GC hole. In fact, I get it even more deterministically.
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-25T19:50:56Z
body: I think this time it's the environment variable `CORE_ROOT` that I wasted hours on 😅 
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-25T20:05:25Z
body: I think it's ready for another pass.
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-26T01:06:48Z
body: /azp run runtime-coreclr outerloop

author_association: MEMBER
user: login: jkotas
created_at: 2025-05-26T01:06:52Z
body: /azp run runtime-coreclr gcstress0x3-gcstress0xc
author_association: MEMBER
user: login: azure-pipelines[bot]
created_at: 2025-05-26T01:07:03Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: azure-pipelines[bot]
created_at: 2025-05-26T01:07:04Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: filipnavara
created_at: 2025-05-26T06:43:57Z
body: We finally have a clean x86 run of the tests 🥳 
author_association: MEMBER
user: login: jkotas
created_at: 2025-05-26T08:41:15Z
body: /ba-g infrastructure and known issues 
author_association: MEMBER
user: login: filipnavara
created_at: 2025-05-26T08:42:19Z
body: Thanks for the help diagnosing the issues, and thanks everyone for helping to make this happen!
author_association: MEMBER