Repository: godotengine/godot (language: cpp)

Issue:
VideoStreamPlayer causes game to crash upon playing video
### Tested versions

- Reproducible in v4.4.beta4.official [93d270693]
- Not reproducible in v4.4.beta3.official [06acfccf8]

The only VideoStreamPlayer-related commit I saw in beta4 was #101958, so I think it is a likely cause of this issue.

### System information

Windows 11 - both Forward+ and Compatibility - both with and without dedicated GPU

### Issue description

Game closes without displaying any in-editor error messages upon playing a VideoStreamPlayer in 4.4 beta4.

### Steps to reproduce

Play an Ogg Theora stream through a VideoStreamPlayer, either with `autoplay=true` or `play()`. Seems to affect both Forward+ and Compatibility renderers (haven't tested Mobile).

### Minimal reproduction project (MRP)

[videoplayerbug.zip](https://github.com/user-attachments/files/18838902/videoplayerbug.zip)
The main scene can be played to reproduce the issue. In beta3, this works as expected and a blue box moves across the screen. In beta4, the game exits immediately.
MinGW GCC LTO internal compiler error when linking: cannot read 'LTO_section_decls'
### Tested versions

- Reproducible in d497631de0c67f22564e1efeb467cf9f061adc80 (commit before merge of workaround #102506) 
- Reproducible in 4.3.beta3 and latest `master` (b607110ad211778a6426a00ca9c8cd99c0eb383c) **with #102506 reverted**
- Not reproducible in 4.3.stable or 4.4.beta2

### System information

Fedora 41, mingw64-gcc-14.2.1-3.fc41, mingw64-headers-12.0.0-3.fc41, mingw64-binutils-2.42-2.fc41

### Issue description

This is the bug report I promised in #102506, describing the actual bug that #102506 just hacks around, but that PR should be reverted and the bug fixed properly. This is very likely to be a GCC or binutils bug, and not a Godot bug.

Reproducible this bug requires (for now) to revert #102506 locally, as it seems like reintroducing this unnecessary code prevents the LTO internal compiler error. It's all fairly brittle though and I'm not confident it won't come back as we add/change other code, hence why this is a release blocker.

When compiling current `master` (b607110ad211778a6426a00ca9c8cd99c0eb383c) with #102506 reverted and the following command, using MinGW-GCC:
```
scons p=windows target=editor arch=x86_64 production=yes module_mono_enabled=yes
```
(mono module and LTO are important)

The linking steps fails with:
```
lto1: internal compiler error: cannot read 'LTO_section_decls' from /tmp/ccf5RNMk.ltrans115.o
Please submit a full bug report, with preprocessed source (by using -freport-bug).
See <http://bugzilla.redhat.com/bugzilla> for instructions.
make: *** [/tmp/cct9c3iK.mk:347: /tmp/ccf5RNMk.ltrans115.ltrans.o] Error 1
make: *** Waiting for unfinished jobs....
lto-wrapper: fatal error: make returned 2 exit status
compilation terminated.
/usr/lib/gcc/x86_64-w64-mingw32/14.2.1/../../../../x86_64-w64-mingw32/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
```

Or this:
```
lto1: error: two or more sections for .gnu.lto__ZZL39_register_variant_builtin_methods_arrayvEN17Method_Array_back17get_argument_typeEi.lto_priv.0.37928630.b9f437417adc9a80
(null):0: confused by earlier errors, bailing out
make: *** [/tmp/cc3CZrbQ.mk:347: /tmp/ccmO0Wsr.ltrans115.ltrans.o] Error 1
make: *** Waiting for unfinished jobs....
lto-wrapper: fatal error: make returned 2 exit status
compilation terminated.
/usr/lib/gcc/x86_64-w64-mingw32/14.2.1/../../../../x86_64-w64-mingw32/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
```

I debugged a bit with @bruvzg, who could also reproduce this with mingw-gcc on macOS with GCC 14.2.0. He tried with `-freport-bug` as advised by the error but that seems to prevent the bug.

@Repiteo noted that the second error is similar to the one in #92585 (note: `x86_32` there, while here it's on `x86_64`), which we just worked around at the time by updating the base Fedora version to get newer mingw/gcc/binutils.

But here we're using the latest version provided on Fedora 41. mingw-headers and mingw-gcc on Fedora 42 aren't newer, but mingw-binutils is at 2.43.1, might be worth testing against.

Since the error popped up between 4.4.beta2 and 4.4.beta3, I bisected it and landed on #102179. There's evidently nothing egregious in that patch that should trigger a LTO crash. Through trial and error, I manage to find a workaround in #102506 by doing a minimal partial revert of #102179. Reintroducing some of that code, even in a path that will never be executed (but could be, as it's checked by an undocumented project setting), seems to prevent the crash.

We also noticed that the presence of LTO objects from a previous build can also be a cause for the issue. But with a `git clean -fxd` this issue is still reproducible when reverting #102179.

Obviously, the problem is deeper so we need to dig. That's where I page @hpvb as the resident expert in debugging GCC bugs :D

### Steps to reproduce

Set up latest mingw with GCC 14.2.
```
git revert e12a424bc569ac5ae9ff2944a8df7fc11b157d71
git clean -fxd
scons p=windows target=editor arch=x86_64 production=yes module_mono_enabled=yes
```

### Minimal reproduction project (MRP)

n/a


Discussion / hints:
CC @berarma 
I can't reproduce it on Linux. The MRP works well here.

I'll try it on a machine with Windows 11. In the mean time if someone could get a crash backtrace it may speed things up.

Does it crash before displaying anything or when the video is already playing?
Can't reproduce on macOS (using provided videoplayerbug project).

Godot v4.4.beta4 - macOS Sequoia (15.3.1) - Multi-window, 2 monitors - Metal (Forward+) - integrated Apple M1 Pro (Apple7) - Apple M1 Pro (10 threads)
Can replicate this on Windows 11, will provide a stacktrace soon
I can't confirm it on a custom build on the same branch, so unsure what is different between the two versions here, so can't provide a stacktrace as it won't occur in a debug build
What compiler and compiler version are you using? What compiler does the official beta use?
> I can't confirm it on a custom build on the same branch, so unsure what is different between the two versions here, so can't provide a stacktrace as it won't occur in a debug build

Thanks for testing it.

> What compiler and compiler version are you using? What compiler does the official beta use?

Looking at the build scripts it seems to be built using ~~mingw-llvm~~ (read comment below from akien-mga).
Testing a production build currently, using MSVC in both cases, so might be MinGW specific, if it doesn't happen on a production MSVC build someone would need to build using MinGW to test (I don't have an editor build for that set up so it'd be easier if someone using it already can test to avoid the build time)
> > What compiler and compiler version are you using? What compiler does the official beta use?
> 
> Looking at the build scripts it seems to be built using mingw-llvm.

It's actually mingw-gcc 14.2.1 from Fedora 41 for x86_64 and x86_32, and [llvm-mingw](https://github.com/mstorsjo/llvm-mingw/) for arm64 only.

Also relevant, official builds are made with GCC LTO (`production=yes`), which may be a factor here.
Doesn't happen in a `production=yes` MSVC build so likely gcc specific
GCC and Windows specific.

I've been able to reproduce the issue on a Windows 10 VM using the official build, tested that beta3 works on the same VM. Now I'm trying to build a custom version for Windows with D3D12 support but it doesn't run on the same VM because it requries OpenGL3.3 or D3D11. It looks like I haven't built with D3D12 support although I've followed the instructions.
You probably need to build with ANGLE to be able to run on that VM. https://docs.godotengine.org/en/stable/contributing/development/compiling/compiling_for_windows.html#compiling-with-angle-support
I've made a custom build with the command `scons platform=windows debug_symbols=yes optimize=debug d3d12=yes angle_libs=/opt/angle/ linker=mold`, and it works.

Now I'm building with `production=yes`.
This is the backtrace and this is the build command: `scons platform=windows d3d12=yes angle_libs=/opt/angle/ linker=mold use_static_cpp=yes lto=auto debug_symbols=yes`
```
CrashHandlerException: Program crashed with signal 11
Engine version: Godot Engine v4.4.beta.custom_build (93d270693079ea7802c9e1334a2e0ecd8529eeed)                          Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] _gnu_exception_handler (./mingw-w64-crt/crt/crt_handler.c:223)
[2] __tmainCRTStartup (./mingw-w64-crt/crt/crtexe.c:321)
[3] WinMainCRTStartup (./mingw-w64-crt/crt/crtexe.c:176)
-- END OF BACKTRACE --
```

Is this an [exception handler crash](https://github.com/godotengine/godot/pull/102900)?
Not sure why the crash handler backtrace is like this, it's probably a separate issue we need to keep looking into @bruvzg.

I ran the reproduction project through gdb and got a more helpful backtrace:
```
Thread 1 received signal SIGSEGV, Segmentation fault.
0x00000001404242e6 in oc_huff_token_decode_c ()
(gdb) bt
#0  0x00000001404242e6 in oc_huff_token_decode_c ()
#1  0x000000014049b09e in VideoStreamPlaybackTheora::update(double) [clone .part.0] ()
#2  0x0000000142cfd9f2 in VideoStreamPlayer::play() ()
#3  0x00000001449648c1 in Object::notification(int, bool) ()
#4  0x0000000142802d1f in Node::_propagate_enter_tree() ()
#5  0x0000000142803015 in Node::_propagate_enter_tree() ()
#6  0x000000014282a0c1 in Node::_set_tree(SceneTree*) ()
#7  0x000000014006f748 in widechar_main(int, wchar_t**) ()
#8  0x0000000145353ba3 in main ()
```

Here's how I did it for the record, which might be helpful for @berarma to debug further from Linux:

- Compiled Godot in a `fedora:41` podman container with mingw-gcc and just `scons p=windows lto=full debug_symbols=yes`
- Installed `wine`, and `mingw64-gdb`, which provides a Windows version of gdb in `/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gdb.exe`
- `cd` to the MRP folder
- `wine /usr/x86_64-w64-mingw32/sys-root/mingw/bin/gdb.exe path/to/godot.windows.editor.x86_64.exe`
- In the gdb prompt, `r --headless video_player.tscn`

I discussed this with @hpvb who had a hunch that it might be related to another GCC LTO bug we're currently debugging in #102867. The flags suggested in https://github.com/godotengine/godot/issues/102867#issuecomment-2663930093 seem to solve this crash indeed in my test.
@akien-mga, I got the same backtrace as you. The process was a bit different since I'm on Debian, but I just had to translate your steps.

There seems to be a gap in the backtrace, I'm not directly calling `oc_huff_token_decode_c` in the code but they're next to each other in the backtrace. Also, I can't see line numbers or the source code, gdb complains that there is no symbol table. Without this information it's hard to know what's happening.

In my tests, setting LTO and default optimization triggered the crash, but it wasn't always the case. Since I couldn't get a good backtrace I put several `printf` in the code to try to narrow down the code causing the crash and then it stopped crashing. The only change was the `printf` lines.

If I could get the debug symbols to work I could investigate more, without this, I'm out of ideas to try. I'll test a build with the flags suggested in #102867 to confirm your results and follow your work there.
Confirmed it doesn't crash with the flags in https://github.com/godotengine/godot/issues/102867#issuecomment-2663930093. Tested also adding `d3d12=yes angle=...` both on Wine and Windows and it also works.

I've resorted to reading the assembly to find out the line, and this is the line that causes the segmentation fault:

https://github.com/godotengine/godot/blob/e567f49cbb292dd6bb628348e67c1f3c2a3a5f99/thirdparty/libtheora/huffdec.c#L482

The second argument (_tree) to the function `oc_huff_token_decode_c` is a bad pointer. This function is called with static int16 arrays defined inside static functions for the second argument, like this:

https://github.com/godotengine/godot/blob/e567f49cbb292dd6bb628348e67c1f3c2a3a5f99/thirdparty/libtheora/decode.c#L319

And this is where the array is defined:

https://github.com/godotengine/godot/blob/e567f49cbb292dd6bb628348e67c1f3c2a3a5f99/thirdparty/libtheora/decode.c#L309

There are several calls to this function in this file, but since I don't have a complete backtrace I can't tell which one is causing the crash. Most of them use these static arrays, while some others use arrays read from the video headers.

I've tried building with `ccflags="-fno-inline"` to try and get a complete backtrace, but it fails with these errors:

```
lto1: error: two or more sections for .gnu.lto__ZZL40_register_variant_builtin_methods_stringvEN28Method_StringName_hex_to_int18get_argument_countEv.34588887.ba4a6b712d465bb9
lto1: error: two or more sections for .gnu.lto__ZZN6embree12parallel_forIyZNS_17ParallelRadixSortINS_4sse212PresplitItemEjE17tbbRadixIterationEjbPKS3_PS3_yEUlyE0_EEvT_RKT0_ENKUlRKNS_5rangeIyEEE_clESG_.9227336.fefa44b2c0907de9
(null):0: confused by earlier errors, bailing out
make: *** [/tmp/ccphOoha.mk:305: /tmp/cc6f9UAr.ltrans101.ltrans.o] Error 1
make: *** Waiting for unfinished jobs....
(null):0: confused by earlier errors, bailing out
make: *** [/tmp/ccphOoha.mk:383: /tmp/cc6f9UAr.ltrans127.ltrans.o] Error 1
./core/templates/sort_array.h: In function 'unguarded_insertion_sort.constprop.isra':
./core/templates/sort_array.h:288:59: warning: iteration 1537228672809129301 invokes undefined behavior [-Waggressive-loop-optimizations]
  288 |                         unguarded_linear_insert(i, p_array[i], p_array);
      |                                                           ^
./core/templates/sort_array.h:287:45: note: within this loop
  287 |                 for (int64_t i = p_first; i != p_last; i++) {
      |                                             ^
thirdparty/misc/polypartition.cpp: In function 'Triangulate_OPT.isra':
thirdparty/misc/polypartition.cpp:711:29: warning: argument 1 value '18446744073709551615' exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
  711 |   dpstates = new DPState *[n];
      |                             ^
/usr/lib/gcc/x86_64-w64-mingw32/12-posix/include/c++/new:128:26: note: in a call to allocation function 'operator new []' declared here
  128 | _GLIBCXX_NODISCARD void* operator new[](std::size_t) _GLIBCXX_THROW (std::bad_alloc)
      |                          ^
lto-wrapper: fatal error: make returned 2 exit status
compilation terminated.
/usr/bin/x86_64-w64-mingw32-ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
scons: *** [bin/godot.windows.editor.x86_64.exe] Error 1
scons: building terminated because of errors.
```
Amended first paragraph in my last comment which didn't make any sense. Had a mental lapse between long builds.
Could someone who reproduces the issue test these builds from #103077 to confirm that it's fixing the bug?

https://github.com/godotengine/godot/pull/103077#issuecomment-2672226330
> Could someone who reproduces the issue test these builds from [#103077](https://github.com/godotengine/godot/pull/103077) to confirm that it's fixing the bug?
> 
> [#103077 (comment)](https://github.com/godotengine/godot/pull/103077#issuecomment-2672226330)

I tested the 64-bit version, and it fixes the crash for me.

However, the new build has some odd lag spikes in the video that weren't present in beta3. I'm unsure if this is related to the new build flags. I'll try to make a custom build of beta3 with these flags to confirm if they are the cause.
> > Could someone who reproduces the issue test these builds from [#103077](https://github.com/godotengine/godot/pull/103077) to confirm that it's fixing the bug?
> > [#103077 (comment)](https://github.com/godotengine/godot/pull/103077#issuecomment-2672226330)
> 
> I tested the 64-bit version, and it fixes the crash for me.
> 
> However, the new build has some odd lag spikes in the video that weren't present in beta3. I'm unsure if this is related to the new build flags. I'll try to make a custom build of beta3 with these flags to confirm if they are the cause.

Does this happen with the MRP? Do they play correctly in `ffplay`?

Videos encoded with FFmpeg could be incorrectly encoded, although they should play the same in `ffplay` and Godot 4.4beta4.

FFmpeg has already been fixed, but you will need a [very recent build](https://github.com/godotengine/godot-docs/pull/10614#issuecomment-2671246526).
> Does this happen with the MRP? Do they play correctly in `ffplay`?
> 
> Videos encoded with FFmpeg could be incorrectly encoded, although they should play the same in `ffplay` and Godot 4.4beta4.
> 
> FFmpeg has already been fixed, but you will need a [very recent build](https://github.com/godotengine/godot-docs/pull/10614#issuecomment-2671246526).

Yes, specifically for the MRP above, on my Windows 11 machine (with RTX3070): beta3 produces smooth video output; rc.1 produces lag spikes. This behavior is repeatable.

The video in the MRP was encoded with `ffmpeg version 7.0.1-full_build-www.gyan.dev`. I'm not sure what you mean regarding ffmpeg being fixed -- was there a bug in their library for Ogg Theora encoding?
> The video in the MRP was encoded with `ffmpeg version 7.0.1-full_build-www.gyan.dev`. I'm not sure what you mean regarding ffmpeg being fixed -- was there a bug in their library for Ogg Theora encoding?

Yes, there were a couple of bugs in FFmpeg that have been recently fixed in their static daily builds.

Please, re-encode the videos from the source with the newest version to make sure it is not due to those bugs.
Re-encoded with version `2025-02-20-git-bc1a3bfd2c` of ffmpeg. New MRP: [videoplayerbug.zip](https://github.com/user-attachments/files/18894572/videoplayerbug.zip)

Same issue. beta3 has smooth video playback while rc.1 has lag spikes. I set up beta3 with #103077 to see if the flags are the issue, but it's still compiling.

Update: Not reproducible in both beta3 and beta3 with #103077 .
I can't reproduce on Linux. There are some frame skips on both beta3 and beta4, but that's to be expected on my computer when playing a 1920x1080/144fps video on a decoder without GPU acceleration.

Can you paste the FPS log for both versions?
Both versions show 144fps consistently for me. I tried setting max fps to 30 and it actually makes the issue worse (the entire video appears to slow down).

Here's a recording of the playback on RC.1 for me: 

https://github.com/user-attachments/assets/0177706a-6d54-49ba-9f0b-0896176038d5
Can you create a new issue with the new MRP and this information? I'm going to investigate it. Thanks!
The issue of videos playing with lag spikes has been submitted as #103106.
> But here we're using the latest version provided on Fedora 41. mingw-headers and mingw-gcc on Fedora 42 aren't newer, but mingw-binutils is at 2.43.1, might be worth testing against.

So I actually tested reproducing the issue in a `fedora:42` podman container, and it seems fixed there. So it might have been a bug in binutils 2.42 that was fixed in 2.43.1 already.

If that's the case, we could just update to Fedora 42 already for official builds, or backport binutils 2.43.1.
It wouldn't solve the problem for end users whose latest MinGW distribution wouldn't have binutils 2.43.1 yet.
> So it might have been a bug in binutils 2.42 that was fixed in 2.43.1 already.

macOS MinGW build I was able to reproduce it with seems to be using binutils 2.43.1.

Sources for the MinGW package are:
```
url "https://downloads.sourceforge.net/project/mingw-w64/mingw-w64/mingw-w64-release/mingw-w64-v12.0.0.tar.bz2"
sha256 "cc41898aac4b6e8dd5cffd7331b9d9515b912df4420a3a612b5ea2955bbeed2f"

url "https://ftp.gnu.org/gnu/binutils/binutils-2.43.1.tar.bz2"
sha256 "becaac5d295e037587b63a42fad57fe3d9d7b83f478eb24b67f9eec5d0f1872f"

url "https://ftp.gnu.org/gnu/gcc/gcc-14.2.0/gcc-14.2.0.tar.xz"
sha256 "a7b39bc69cbf9e25826c5a60ab26477001f7c08d85cec04bc0e29cabed6f3cc9"
```
> macOS MinGW build I was able to reproduce it with seems to be using binutils 2.43.1.

Yeah I tested too in a `fedora:41` podman container, installing the mingw-binutils packages from Fedora 42 (`mingw-binutils-generic-2.43.1-3.fc42.x86_64.rpm` and `mingw64-binutils-2.43.1-3.fc42.x86_64.rpm`) and I still reproduce the bug.

So there might be another component in Fedora 42 that's different, or it's just a coincidence.
I've been able to reproduce this, I'm doing some work with gdb on the wrapper to see if I can work out what's going on.
I have not been able to figure out what the problem is exactly, but I have been able to build Godot successfully with LTO with the following ccflags and linkflags: `-fno-use-linker-plugin -fwhole-program` this should result in at least similar optimization as with the normal lto. 

At least for now I recommend using this while I try and figure out with the gcc folk what is up.
Retested it with current `master` + reverted e12a424bc569ac5ae9ff2944a8df7fc11b157d71 and seems like it's no longer failing on macOS (toolchain is the same), so it's pretty random (probably timing sensitive race condition).
@bruvzg it is not likely timing sensitive. The problem reproduces with or without using multiple threads.

The problem is almost certainly in the "Whole Program Analysis" (WPA) phase of LTO. This is the phase that will partition the LTO work into smaller objects to be processed in series or in parallel. This phase seems to partition the work wrong, which then results in the 115th partition to not be linkable.

The issue is that almost *any* change to the build will cause the problem to go away. For instance just turning LTO off for literally *any* of the .a or .o files, just one, causes the link to succeed.

Because of this the problem is turning out to be very hard to debug. But I'm afraid that it is not a race condition. 

Fix patch:
diff --git a/platform/windows/detect.py b/platform/windows/detect.py
index 6f57b880441f..bbdcb9e79aa1 100644
--- a/platform/windows/detect.py
+++ b/platform/windows/detect.py
@@ -732,7 +732,7 @@ def configure_mingw(env: "SConsEnvironment"):
         if env["use_static_cpp"]:
             env.Append(LINKFLAGS=["-static"])
 
-    if env["arch"] in ["x86_32", "x86_64"]:
+    if env["arch"] == "x86_32":
         env["x86_libtheora_opt_gcc"] = True
 
     env.Append(CCFLAGS=["-ffp-contract=off"])


Test patch: