Repository: godotengine/godot (language: cpp)

Issue:
Godot 4.4 silently closes when using OBS advanced scene switcher (detects active window). GameCapture Mode
### Tested versions

4.4 issue
4.3 works fine

### System information

Godot v4.4.stable - Windows 10 (build 19045) - Multi-window, 2 monitors - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 2070 (NVIDIA; 32.0.15.6636) - Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz (16 threads)

### Issue description

4.4 Godot Crashes when tabbing into other windows using OBS Scene Switcher Plugin(which detects the current window and changes scene)

I will temporarily go back to 4.3 which does not do this.

 I'm not sure what options in the editor have changed or if there are some options I can tick to help with this. I've tried many different options. I know the detection of windows isn't youre problem, but why did it work flawlessly in 4.3?

It only happens if im using GameCapture for Godot 4.4. I just dont want to stream my whole display monitor. I use the advanced scene switcher plugin to make devlogs. Switching scenes between my notes, editor, ect.


No errors. Just my Godot disappears and closes when I tab into other windows. It works fine if Godot isn't Game Capture. 

Did anything change in 4.4 for window detection and compatibility with OBS? Ive even tried setting godot 4.4 to single window mode.

### Steps to reproduce

Open Godot 4.4,

Have scene setup in game capture mode for Godot 4.4.

Use Window Detection with another application ( OBS Automatic Scene Switcher)

Scenes automatically switch based on the active window.

Godot crashes when tabbing into other windows.

Open Godot 4.3

Repeat steps.

Godot does not close/crash


### Minimal reproduction project (MRP)

Happens with any 4.4 project. Even brand new ones.

Unable to replicate in 4.3


Discussion / hints:
To clarify. It closes godot when I tab out of godot and then back into Godot. No errors or crash.

Only thing in the project file is a vulkan cache, no crash logs or anythin.
Confirmed issue is something with Forward+ mode in 4.4 and windows based window detection.

Godot does not seem to crash using these method on Compatibility mode. YAY!! I can work now!
Could you test the versions between 4.3 stable and 4.4 stable to find when this problem started? You can download them here: https://godotengine.org/download/archive/ 
From further testing, my previous comments about 4.3 working was only due to compatibility render mode. If I use forward+ it still happens in 4.3 stable. Let me try to use a different computer perhaps and do further testing. It will be a few days before I can setup my other computer. Please leave this open if you deem necessary. 
Can you reproduce this after switching back to Forward+/Mobile, then changing `rendering_driver.windows` to `d3d12` in the Project Settings and restarting the editor?

This is almost certainly caused by a negative interaction between the OBS Vulkan layer and Godot. This layer should probably be blocked from Godot to avoid crashes, but it's not listed here:

https://github.com/godotengine/godot/blob/30bb49ec1ff3b3fc463074b8d28bc97d4db50b9f/main/main.cpp#L2515-L2535
Thank you. Confirming in 4.3 and 4.4 stables that switching to d3d12 forward+ allows my OBS scene switching plugin to flawlessly detect the windows. The vulkan seems to be culprit.

With vulkan = crash when switching windows
with d3d12 = perfect
To clarify, when in vulkan forward plus. I can tab into Godot fine, but as soon as I tab out, it closes. 

If this is a problem related to the plugin of obs, fair enough. THe plugin that detects windows is

https://obsproject.com/forum/resources/advanced-scene-switcher.395/

specifically the Automatic Scene Switcher tool
You linked `Advanced Scene Switcher`. OBS has `Automatic Scene Switcher` built in. I assume this is about the one you linked?
I can replicate this without any scene switchers. Only affects OBS's `Game Capture` source and Vulkan drivers in Godot. Nothing in the verbose logs. Doesn't even require switching scenes. Hiding and unhiding the source a few times does the trick too.

```
Godot v4.4.stable - Windows 11 (build 22631) - Multi-window, 2 monitors - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3060 Ti (NVIDIA; 32.0.15.6094) - 12th Gen Intel(R) Core(TM) i5-12400F (12 threads)
```

https://github.com/user-attachments/assets/d37412ff-da98-4b98-bb99-b4e364924775

Fix patch:
diff --git a/main/main.cpp b/main/main.cpp
index 441ba3141ef4..032d367bc18b 100644
--- a/main/main.cpp
+++ b/main/main.cpp
@@ -2517,24 +2517,38 @@ Error Main::setup(const char *execpath, int argc, char *argv[], bool p_second_ph
 	OS::get_singleton()->_allow_layered = GLOBAL_DEF_RST("display/window/per_pixel_transparency/allowed", false);
 
 #ifdef TOOLS_ENABLED
-	if (editor || project_manager) {
-		// The editor and project manager always detect and use hiDPI if needed.
-		OS::get_singleton()->_allow_hidpi = true;
-		// Disable Vulkan overlays in editor, they cause various issues.
-		OS::get_singleton()->set_environment("DISABLE_MANGOHUD", "1"); // GH-57403.
-		OS::get_singleton()->set_environment("DISABLE_RTSS_LAYER", "1"); // GH-57937.
-		OS::get_singleton()->set_environment("DISABLE_VKBASALT", "1");
-		OS::get_singleton()->set_environment("DISABLE_VK_LAYER_reshade_1", "1"); // GH-70849.
-		OS::get_singleton()->set_environment("VK_LAYER_bandicam_helper_DEBUG_1", "1"); // GH-101480.
-		OS::get_singleton()->set_environment("DISABLE_VK_LAYER_bandicam_helper_1", "1"); // GH-101480.
-	} else {
-		// Re-allow using Vulkan overlays, disabled while using the editor.
-		OS::get_singleton()->unset_environment("DISABLE_MANGOHUD");
-		OS::get_singleton()->unset_environment("DISABLE_RTSS_LAYER");
-		OS::get_singleton()->unset_environment("DISABLE_VKBASALT");
-		OS::get_singleton()->unset_environment("DISABLE_VK_LAYER_reshade_1");
-		OS::get_singleton()->unset_environment("VK_LAYER_bandicam_helper_DEBUG_1");
-		OS::get_singleton()->unset_environment("DISABLE_VK_LAYER_bandicam_helper_1");
+	{
+		// Synced with https://github.com/baldurk/renderdoc/blob/2b01465c7/renderdoc/driver/vulkan/vk_layer.cpp#L118-L165
+		LocalVector<String> layers_to_disable = {
+			"DISABLE_RTSS_LAYER", // GH-57937.
+			"DISABLE_VULKAN_OBS_CAPTURE", // GH-103800.
+			"DISABLE_VULKAN_OW_OBS_CAPTURE", // GH-104154.
+			"DISABLE_SAMPLE_LAYER", // GH-104154.
+			"DISABLE_GAMEPP_LAYER", // GH-104154.
+			"DISABLE_VK_LAYER_TENCENT_wegame_cross_overlay_1", // GH-104154.
+			"NODEVICE_SELECT", // GH-104154.
+			"VK_LAYER_bandicam_helper_DEBUG_1", // GH-101480.
+			"DISABLE_VK_LAYER_bandicam_helper_1", // GH-101480.
+			"DISABLE_VK_LAYER_reshade_1", // GH-70849.
+			"DISABLE_VK_LAYER_GPUOpen_GRS", // GH-104154.
+			"DISABLE_LAYER", // GH-104154 (fpsmon).
+			"DISABLE_MANGOHUD", // GH-57403.
+			"DISABLE_VKBASALT",
+		};
+
+		if (editor || project_manager) {
+			// The editor and project manager always detect and use hiDPI if needed.
+			OS::get_singleton()->_allow_hidpi = true;
+			// Disable Vulkan overlays in editor, they cause various issues.
+			for (const String &layer_disable : layers_to_disable) {
+				OS::get_singleton()->set_environment(layer_disable, "1");
+			}
+		} else {
+			// Re-allow using Vulkan overlays, disabled while using the editor.
+			for (const String &layer_disable : layers_to_disable) {
+				OS::get_singleton()->unset_environment(layer_disable);
+			}
+		}
 	}
 #endif
 


Test patch: