Repository: microsoft/terminal (language: cpp)

Issue:
Windows Terminal crashing, `TermControlAutomationPeer::RecordKeyEvent` and  `InteractivityAutomationPeer::NotifyNewOutput`
### Windows Terminal version

1.15.2212.12006, also 1.16.2212.12004

### Windows build number

10.0.22000.0

### Other Software

all within WSL2:
vim 8.1.3741
node 18.9.1
jest 8.19.1

### Steps to reproduce

I cannot force this problem to occur, but it has happened twice today, and several times in the last few weeks when I am using Terminal on WSL2 and working in Vim. 

I included details that I was running some React tests, but I had this problem several times last week when I was working in Python (no framework).

I went back through the event logs and the earliest date of this error is 8/17/22. There are 8 total logs since then which are nearly identical to the ones below. I've spent a large amount of time on this computer since then, much of it with a vim window open somewhere, so this is a relatively rare issue. However, twice in one day is a bit much.

### Expected Behavior

_No response_

### Actual Behavior

When I am working in Vim, when I am changing focus between Terminal and another app (in this case, I was running a react debugger in chrome), occasionally Terminal will crash without any errors. This happens whether I change focus with alt+tab or with the mouse. 

I cannot be certain changing focus is what is triggering the issue, since I change focus frequently, but it has not happened when I am just typing happily into Vim, nor when I switch between Vim panes.

I was running the terminal preview when it crashed, so I switched to the non-preview (hence the two version numbers for Terminal) and had the same issue. 

Here are the event logs for these crashes:
For the first crash today at 2:52:24 (using Terminal Preview):

ERROR:
***
Faulting application name: WindowsTerminal.exe, version: 1.16.2212.12004, time stamp: 0x6397c31c
Faulting module name: Microsoft.Terminal.Control.dll, version: 1.16.2212.12004, time stamp: 0x6397c12e
Exception code: 0xc0000005
Fault offset: 0x00000000000d5112
Faulting process id: 0x4680
Faulting application start time: 0x01d914b7036683d3
Faulting application path: C:\Program Files\WindowsApps\Microsoft.WindowsTerminalPreview_1.16.3464.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe
Faulting module path: C:\Program Files\WindowsApps\Microsoft.WindowsTerminalPreview_1.16.3464.0_x64__8wekyb3d8bbwe\Microsoft.Terminal.Control.dll
Report Id: efaf5abb-faf8-4f7c-aba5-aefc5999d1e5
Faulting package full name: Microsoft.WindowsTerminalPreview_1.16.3464.0_x64__8wekyb3d8bbwe
Faulting package-relative application ID: App

WINDOWS ERROR REPORTING:
***
Fault bucket 1477357042843758786, type 5
Event Name: MoAppCrash
Response: Not available
Cab Id: 0

Problem signature:
P1: Microsoft.WindowsTerminalPreview_1.16.3464.0_x64__8wekyb3d8bbwe
P2: praid:App
P3: 1.16.2212.12004
P4: 6397c31c
P5: Microsoft.Terminal.Control.dll
P6: 1.16.2212.12004
P7: 6397c12e
P8: c0000005
P9: 00000000000d5112
P10: 

Attached files:
[buncha files]

These files may be available here:
[link to a wer file]

Analysis symbol: 
Rechecking for solution: 0
Report Id: efaf5abb-faf8-4f7c-aba5-aefc5999d1e5
Report Status: 268435456
Hashed bucket: 9f45178bb05e9a32c480a067311fb0c2
Cab Guid: 0



for the second crash at 6:50:59 (with non preview): 

ERROR:
***
Faulting application name: WindowsTerminal.exe, version: 1.15.2212.12006, time stamp: 0x6397f53c
Faulting module name: Microsoft.Terminal.Control.dll, version: 1.15.2212.12006, time stamp: 0x6397f373
Exception code: 0xc0000005
Fault offset: 0x00000000000b2bd9
Faulting process id: 0x7f0
Faulting application start time: 0x01d915801c4096a8
Faulting application path: C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.15.3466.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe
Faulting module path: C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.15.3466.0_x64__8wekyb3d8bbwe\Microsoft.Terminal.Control.dll
Report Id: 47cfed3e-ccc3-44b4-90eb-fbeb40eabd52
Faulting package full name: Microsoft.WindowsTerminal_1.15.3466.0_x64__8wekyb3d8bbwe
Faulting package-relative application ID: App

WINDOWS ERROR REPORTING:
***
Fault bucket 2089511070221908304, type 5
Event Name: MoAppCrash
Response: Not available
Cab Id: 0

Problem signature:
P1: Microsoft.WindowsTerminal_1.15.3466.0_x64__8wekyb3d8bbwe
P2: praid:App
P3: 1.15.2212.12006
P4: 6397f53c
P5: Microsoft.Terminal.Control.dll
P6: 1.15.2212.12006
P7: 6397f373
P8: c000041d
P9: 00000000000b2bd9
P10: 

Attached files:
[buncha files]

These files may be available here:
[link to a wer file]

Analysis symbol: 
Rechecking for solution: 0
Report Id: 8a9de9e7-2bbd-4d46-8a3e-d7c0b3dee0b7
Report Status: 268435456
Hashed bucket: 01b156f5f7ca4192ccff6f3f66d2b950
Cab Guid: 0


Discussion / hints:
Wow for once, the EV logs here actually _did_ point at a crash on the backend! That's amazing!

For MSFT folks, `BucketDetails?BucketIdentifier=9f45178b-b05e-9a32-c480-a067311fb0c2`. 

There's multiple crashes in here, MSFT:38673750 and MSFT:38745829. These are blamed to `...Control::implementation::TermControlAutomationPeer::NotifyNewOutput` and `...Control::implementation::TermControlAutomationPeer::RecordKeyEvent` respectively.

These stacks are weird though, not sure what regressed here
![image](https://user-images.githubusercontent.com/18356694/210384820-799fce3b-70ea-49e2-85e6-06f5c9307420.png)

![image](https://user-images.githubusercontent.com/18356694/210384686-0a8bf9fb-4d84-4565-9f4c-f678adecf29c.png)

I may need to dig into stacks closer. There's plenty of dumps so we should have something to go on. 

Fix patch:
diff --git a/src/cascadia/TerminalControl/TermControlAutomationPeer.cpp b/src/cascadia/TerminalControl/TermControlAutomationPeer.cpp
index 1f670cd1f6d..41c55b37640 100644
--- a/src/cascadia/TerminalControl/TermControlAutomationPeer.cpp
+++ b/src/cascadia/TerminalControl/TermControlAutomationPeer.cpp
@@ -112,7 +112,7 @@ namespace winrt::Microsoft::Terminal::Control::implementation
         {
             if (const auto keyEventChar{ gsl::narrow_cast<wchar_t>(charCode) }; IsReadable({ &keyEventChar, 1 }))
             {
-                _keyEvents.emplace_back(keyEventChar);
+                _keyEvents.lock()->emplace_back(keyEventChar);
             }
         }
     }
@@ -202,27 +202,30 @@ namespace winrt::Microsoft::Terminal::Control::implementation
 
     void TermControlAutomationPeer::NotifyNewOutput(std::wstring_view newOutput)
     {
-        // Try to suppress any events (or event data)
-        // that is just the keypress the user made
         auto sanitized{ Sanitize(newOutput) };
-        while (!_keyEvents.empty() && IsReadable(sanitized))
+        // Try to suppress any events (or event data)
+        // that are just the keypresses the user made
         {
-            if (til::toupper_ascii(sanitized.front()) == _keyEvents.front())
-            {
-                // the key event's character (i.e. the "A" key) matches
-                // the output character (i.e. "a" or "A" text).
-                // We can assume that the output character resulted from
-                // the pressed key, so we can ignore it.
-                sanitized = sanitized.substr(1);
-                _keyEvents.pop_front();
-            }
-            else
+            auto keyEvents = _keyEvents.lock();
+            while (!keyEvents->empty() && IsReadable(sanitized))
             {
-                // The output doesn't match,
-                // so clear the input stack and
-                // move on to fire the event.
-                _keyEvents.clear();
-                break;
+                if (til::toupper_ascii(sanitized.front()) == keyEvents->front())
+                {
+                    // the key event's character (i.e. the "A" key) matches
+                    // the output character (i.e. "a" or "A" text).
+                    // We can assume that the output character resulted from
+                    // the pressed key, so we can ignore it.
+                    sanitized = sanitized.substr(1);
+                    keyEvents->pop_front();
+                }
+                else
+                {
+                    // The output doesn't match,
+                    // so clear the input stack and
+                    // move on to fire the event.
+                    keyEvents->clear();
+                    break;
+                }
             }
         }
 
diff --git a/src/cascadia/TerminalControl/TermControlAutomationPeer.h b/src/cascadia/TerminalControl/TermControlAutomationPeer.h
index c9e00340f9c..e8e62cf953c 100644
--- a/src/cascadia/TerminalControl/TermControlAutomationPeer.h
+++ b/src/cascadia/TerminalControl/TermControlAutomationPeer.h
@@ -80,6 +80,6 @@ namespace winrt::Microsoft::Terminal::Control::implementation
     private:
         winrt::weak_ref<Microsoft::Terminal::Control::implementation::TermControl> _termControl;
         Control::InteractivityAutomationPeer _contentAutomationPeer;
-        std::deque<wchar_t> _keyEvents;
+        til::shared_mutex<std::deque<wchar_t>> _keyEvents;
     };
 }
diff --git a/src/cascadia/TerminalControl/pch.h b/src/cascadia/TerminalControl/pch.h
index cbbe86ac0a9..4d1dd8bb85b 100644
--- a/src/cascadia/TerminalControl/pch.h
+++ b/src/cascadia/TerminalControl/pch.h
@@ -65,6 +65,7 @@ TRACELOGGING_DECLARE_PROVIDER(g_hTerminalControlProvider);
 #include <WinUser.h>
 
 #include "til.h"
+#include <til/mutex.h>
 
 #include "ThrottledFunc.h"
 


Test patch: