Repository: microsoft/terminal (language: cpp)

Issue:
The MoAppHang thread
### Windows Terminal version

1.13.2205.23001

### Windows build number

10.0.19043.1766

### Other Software

_No response_

### Steps to reproduce

1. Open few cards in WindowsTerminal, including: cmd, powershell, ipython executed from cmd
2. Interact with those daily, but leave everything running

### Expected Behavior

WindowsTerminal executes smoothly.

### Actual Behavior

WindowsTerminal crashes. It's been crashing for months now. At various intervals. I couldn't determine any action to cause this. Today I switched off remote desktop and when logged in in few minutes, terminal was gone.
The only trace was in Event Viewer:
```Fault bucket 0, type 5
Event Name: MoAppHang
Response: Not available
Cab Id: 0

Problem signature:
P1: Microsoft.WindowsTerminal_1.13.11431.0_x64__8wekyb3d8bbwe
P2: praid:App
P3: 1.13.2205.23001
P4: 628bd6e7
P5: 0c21
P6: 2097152
P7: 
P8: 
P9: 
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER7AF0.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER7B2F.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER7B4D.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER7BBC.tmp.txt
\\?\C:\Users\RGomulk\AppData\Local\Temp\WER7FA5.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\memory.hdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.3f88.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.5990.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.3478.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.46b0.mdmp

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_0c6eae25-ca99-44ed-ac03-564ef4cf04aa

Analysis symbol: 
Rechecking for solution: 0
Report Id: 6ff345e4-bc39-4cf6-8a62-596843d57bd6
Report Status: 268435456
Hashed bucket: 86e421f1a50f55fde4a62f6775b8f8e5
Cab Guid: 0
```

Before which I got:
```
Fault bucket , type 0
Event Name: MoAppHang
Response: Not available
Cab Id: 0

Problem signature:
P1: Microsoft.WindowsTerminal_1.13.11431.0_x64__8wekyb3d8bbwe
P2: praid:App
P3: 1.13.2205.23001
P4: 628bd6e7
P5: 0c21
P6: 2097152
P7: 
P8: 
P9: 
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER7AF0.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER7B2F.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER7B4D.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER7BBC.tmp.txt
\\?\C:\Users\RGomulk\AppData\Local\Temp\WER7FA5.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\memory.hdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.3f88.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.5990.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.3478.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa\minidump.46b0.mdmp

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppHang_Microsoft.Window_4bcfb2ab6ceed78368cad6844db4f7cf44eba4_3ff56f4b_cab_0c6eae25-ca99-44ed-ac03-564ef4cf04aa

Analysis symbol: 
Rechecking for solution: 0
Report Id: 6ff345e4-bc39-4cf6-8a62-596843d57bd6
Report Status: 4
Hashed bucket: 
Cab Guid: 0
```

and
```
The program WindowsTerminal.exe version 1.13.2205.23001 stopped interacting with Windows and was closed. To see if more information about the problem is available, check the problem history in the Security and Maintenance control panel.
 Process ID: 2ff4
 Start Time: 01d8910361242def
 Termination Time: 4294967295
 Application Path: C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.13.11431.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe
 Report Id: 6ff345e4-bc39-4cf6-8a62-596843d57bd6
 Faulting package full name: Microsoft.WindowsTerminal_1.13.11431.0_x64__8wekyb3d8bbwe
 Faulting package-relative application ID: App
 Hang type: Quiesce
```

I have both windows terminal and windows terminal preview installed. I'm not sure which exhibited the behaviour, but if remember correctly, both were crashing similarly.



Discussion / hints:
Huh. This is bucket 86e421f1-a50f-55fd-e4a6-2f6775b8f8e5 which is the moapphang bucket. Usually that's only really hit on upgrade, but there's quite a few more hits than I'd expect...

MSFT:38947155 tracks a variant of this one. a9fa6747-7f80-2130-9fd8-3b2f7d420159 also has a pile of hits and dupes to it.

Y'know though, the crashes are for Terminal version `1.13.11431.0`, but it looks like you've got Terminal `1.13.2205.23001` installed currently. I'd guess that this _IS_ the upgrade moapphang!

(fwiw the dump here had basically nothing useful in it - all our threads looked like they were in perfectly good, unblocked states, and the rest of the threads didn't have symbols)
MoAppHangs seem to be caused by us continuing to do something even after the system asks us to suspend. I wonder if we're seeing a spike of these on 1.13+, because we've got the window state restoration, and we're waiting for the monarch to collect up all the window state. We've apparently got 5s to be ready to be suspended, so I could totally see the RPC calls taking longer than that, or just hanging around. 

Thats at least a theory.

EDIT: Other theories:

> HANG_QUIESCE is used to signify that a packaged Win32 process didn't exit in a timely manner during a servicing operation (uninstall, update, etc). During servicing, processes are sent Ctrl + C and WM_CLOSE messages, afterwards they have 30 seconds to exit.
> 
> If the process IsConsoleApp is True, the Ctrl + C will be sent. If WindowNum > 0, then WM_CLOSE will be sent. Otherwise, there is no way of sending Ctrl + C or window messages, so the app will not be notified. Create a window to receive WM_CLOSE or convert to a console application to receive Ctrl + C.
>
> ### Elevated Processes Always Hang Quiesce
> MSFT:27096508 is tracking the issue. It is present in all versions of Windows that support packaged Win32, and stems from the fact that SiHost (which sends the messages) does not run as elevated.


So we're gonna ALWAYS get a hang quiesce:
* if the terminal is elevated
* If there are multiple tabs and the user has the "are you sure you want to close" dialog open.
NOTE TO SELF - DO THIS vvvv

https://docs.microsoft.com/en-us/windows/win32/rstmgr/guidelines-for-applications


This is literally 50% of our crashes. Amazing that we only have a raw count of crashes equal to the number of updates we get, but let get these out of the metrics entirely.
I bet this is because of an automatic update. My job on Windows Terminal that required several days to run got interrupted by the update yesterday. It's so frustrating.

Powershell, however, seems not to update when running.

Fix patch:
diff --git a/.github/actions/spelling/allow/apis.txt b/.github/actions/spelling/allow/apis.txt
index c7119047e1f..61542db3eaa 100644
--- a/.github/actions/spelling/allow/apis.txt
+++ b/.github/actions/spelling/allow/apis.txt
@@ -15,6 +15,7 @@ BYCOMMAND
 BYPOSITION
 charconv
 CLASSNOTAVAILABLE
+CLOSEAPP
 cmdletbinding
 COLORPROPERTY
 colspan
@@ -35,6 +36,7 @@ DWORDLONG
 DWMSBT
 DWMWA
 endfor
+ENDSESSION
 enumset
 environstrings
 EXPCMDFLAGS
@@ -147,6 +149,7 @@ PEXPLICIT
 PICKFOLDERS
 pmr
 ptstr
+QUERYENDSESSION
 rcx
 REGCLS
 RETURNCMD
diff --git a/src/cascadia/WindowsTerminal/AppHost.cpp b/src/cascadia/WindowsTerminal/AppHost.cpp
index 0ed98add7f2..e0224642173 100644
--- a/src/cascadia/WindowsTerminal/AppHost.cpp
+++ b/src/cascadia/WindowsTerminal/AppHost.cpp
@@ -406,6 +406,12 @@ void AppHost::Initialize()
         }
     });
 
+    _window->AutomaticShutdownRequested([this]() {
+        // Raised when the OS is beginning an update of the app. We will quit,
+        // to save our state, before the OS manually kills us.
+        _windowManager.RequestQuitAll();
+    });
+
     // Load bearing: make sure the PropertyChanged handler is added before we
     // call Create, so that when the app sets up the titlebar brush, we're
     // already prepared to listen for the change notification
diff --git a/src/cascadia/WindowsTerminal/IslandWindow.cpp b/src/cascadia/WindowsTerminal/IslandWindow.cpp
index 7464aae2382..a973b57423d 100644
--- a/src/cascadia/WindowsTerminal/IslandWindow.cpp
+++ b/src/cascadia/WindowsTerminal/IslandWindow.cpp
@@ -658,6 +658,43 @@ long IslandWindow::_calculateTotalSize(const bool isWidth, const long clientSize
         }
         break;
     }
+    case WM_ENDSESSION:
+    {
+        // For WM_QUERYENDSESSION and WM_ENDSESSION, refer to:
+        //
+        // https://docs.microsoft.com/en-us/windows/win32/rstmgr/guidelines-for-applications
+        //
+        // The OS will send us a WM_QUERYENDSESSION when it's preparing an
+        // update for our app. It will then send us a WM_ENDSESSION, which gives
+        // us a small timeout (~30s) to actually shut down gracefully. After
+        // that timeout, it will send us a WM_CLOSE. If we still don't close
+        // after the WM_CLOSE, it'll force-kill us (causing a crash which will be
+        // bucketed to MoAppHang).
+        //
+        // If we need to do anything to prepare for being told to shutdown,
+        // start it in WM_QUERYENDSESSION. If (in the future) we need to prevent
+        // logoff, we can return false there. (DefWindowProc returns true)
+        //
+        // The OS is going to shut us down here. We will manually start a quit,
+        // so that we can persist the state. If we refuse to gracefully shut
+        // down here, the OS will crash us to forcefully terminate us. We choose
+        // to quit here, rather than just close, to skip over any warning
+        // dialogs (e.g. "Are you sure you want to close all tabs?") which might
+        // prevent a WM_CLOSE from cleanly closing the window.
+        //
+        // This will cause a appHost._RequestQuitAll, which will notify the
+        // monarch to collect up all the window state and save it.
+
+        TraceLoggingWrite(
+            g_hWindowsTerminalProvider,
+            "EndSession",
+            TraceLoggingDescription("Emitted when the OS has sent a WM_ENDSESSION"),
+            TraceLoggingLevel(WINEVENT_LEVEL_VERBOSE),
+            TraceLoggingKeyword(TIL_KEYWORD_TRACE));
+
+        _AutomaticShutdownRequestedHandlers();
+        return true;
+    }
     default:
         // We'll want to receive this message when explorer.exe restarts
         // so that we can re-add our icon to the notification area.
diff --git a/src/cascadia/WindowsTerminal/IslandWindow.h b/src/cascadia/WindowsTerminal/IslandWindow.h
index 39e683da0b6..396bfd04b84 100644
--- a/src/cascadia/WindowsTerminal/IslandWindow.h
+++ b/src/cascadia/WindowsTerminal/IslandWindow.h
@@ -77,6 +77,7 @@ class IslandWindow :
     WINRT_CALLBACK(NotifyReAddNotificationIcon, winrt::delegate<void()>);
     WINRT_CALLBACK(ShouldExitFullscreen, winrt::delegate<void()>);
     WINRT_CALLBACK(MaximizeChanged, winrt::delegate<void(bool)>);
+    WINRT_CALLBACK(AutomaticShutdownRequested, winrt::delegate<void(void)>);
 
     WINRT_CALLBACK(WindowMoved, winrt::delegate<void()>);
     WINRT_CALLBACK(WindowVisibilityChanged, winrt::delegate<void(bool)>);


Test patch: