Repository: microsoft/terminal (language: cpp)

Issue:
Control+Space not sent to program running in terminal
### Windows Terminal version

1.17.11461.0

### Windows build number

10.0.22621.2134

### Other Software

NeoVim v0.9.1

### Steps to reproduce

1) Inside NeoVim remap <escape> to Control + Space

Inside init.lua:
> vim.keymap.set({'i', 'n', 'v', 'o'}, '<C-Space>', '<ESC>', {noremap=true})

2) Open NeoVim and enter insert mode. 
3) Enter Control + Space and notice it doesn't send the Escape command

NOTE: This Is also the case in VIM as well. It doesn't seem like the Control+Space character is making it to whatever is running in the terminal

### Expected Behavior

I expect Control+Space character to make it to the program running in the terminal

### Actual Behavior

When you hit Control+Space there is no reaction to the program running in the terminal. The command does not appear to be sent.


Discussion / hints:
Note, I see this issue was brought up before (and apparently solved): https://github.com/microsoft/terminal/issues/2865. Either this is a slightly different issue or perhaps it broke again.
Are you running a Windows version of neovim, or in WSL/cygwin/msys/git bash, or over ssh of some sort/? Those can all be complicating factors here. 
Can you try repro'ing this with the [debug tap](https://github.com/microsoft/terminal/wiki/Toubleshooting-Tips#enabling-the-debug-tap) to get a trace of all the input and output? Once the bug starts occurring, send us a screenshot and we might be able to figure out what the Terminal thinks it's getting here.
In pwsh (PSReadline module), for example, `Ctrl-Space` works. This combination opens command completion variants.
@jfedorkiw inside nvim session type `:map *` to check if the keymapping is what you expect, or `:map <c-space>` for the `ctrl+space` mapping only, and post the output.
I'm having the same problem with several CTRL combinations. In testing neovim with the exact same configuration in WSL2, Powershell Core, and nvim-qt:

WSL2: CTRL-space works, CTRL-[ (remapped, not used as escape) doesn't work, CTRL-9 doesn't work (seems to just come through as 9)
Powershell Core: CTRL-space doesn't work, CTRL-[ acts like escape even though it's remapped, CTRL-9 just outputs 9
nvim-qt: All work as expected

I'm on the latest WT preview and Windows 11 insider beta.
@sid-6581 same question to you: Can you try repro'ing this with the [debug tap](https://github.com/microsoft/terminal/wiki/Toubleshooting-Tips#enabling-the-debug-tap) to get a trace of all the input and output?
> Are you running a Windows version of neovim, or in WSL/cygwin/msys/git bash, or over ssh of some sort/?

I'm just running nvim through PowerShell inside the terminal app. The problem is also the same when using "Command Prompt" inside terminal. 

Note that it if I use nvim-qt, which launches NeoVim in it's own window, Ctrl-Space works as expected (sending the escape character). Also - the same problem exists in Vim. 

> Can you try repro'ing this with the [debug tap](https://github.com/microsoft/terminal/wiki/Toubleshooting-Tips#enabling-the-debug-tap) to get a trace of all the input and output?

In this case I opened up the terminal and hit Control+Space to see what was being sent:

![image](https://github.com/microsoft/terminal/assets/2207395/b590f354-391c-4532-8e86-ad5be2f7b4ac)

Here is the text copy & pasted for your debugging pleasure:
```
␛[?9001h␛[?1004h␛[1t␛[?25l␛[2J␛[m␛[HMicrosoft␣Windows␣[Version␣10.0.22621.2134]␍␊
(c)␣Microsoft␣Corporation.␣All␣rights␣reserved.␛[4;1HC:\Users\jfedo>␛]0;Command␣Prompt␇␛[?25h␛[18;56;0;1;35;1_␛[18;56;0;1;35;1_␛[I␛[18;56;0;1;35;1_␛[18;56;0;1;35;1_␛[18;56;0;1;35;1_␛[18;56;0;1;35;1_␛[18;56;0;0;33;1_␛[18;56;0;0;32;1_␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␍␊
C:\Users\jfedo>␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␍␊
C:\Users\jfedo>␛[13;28;13;1;32;1_␛[13;28;13;0;32;1_␍␊
C:\Users\jfedo>␛[17;29;0;1;40;1_␛[17;29;0;1;40;1_␛[32;57;32;1;40;1_␛[32;57;32;0;40;1_␣␛[17;29;0;0;32;1_␛[O
```
As a Breakdown here is what I see happen as I execute the key combination:

Press Control: 
>␛[17;29;0;1;40;1_

Press Space:
>␛[32;57;32;1;40;1_␛[32;57;32;0;40;1_␣

And then when I release Control you get the trailing 
> ␛[17;29;0;0;32;1_

On the left side of the screen, when I hit Control+Space it results in a Space character showing up  (as if I had just pressed space)

Interestingly, if I hit Control+Space on the RIGHT side of the window, where it outputs the debug text, it sends the NUL (␀) character (which I believe is what Control+Space is supposed to send).



@zcobol 
Here is the output of :map <c-space>.
```
v  <C-Space>    @<Plug>VimwikiToggleListItem
n  <C-Space>    @<Plug>VimwikiToggleListItem
o  <C-Space>   * <Esc>
v  <C-Space>   * <Esc>
n  <C-Space>   * <Esc>
Press ENTER or type command to continue
```
Note the mapped vim-wiki functionality doesn't work. Also, as mentioned above, these key combinations do work as expected when running nvim-qt. 
 

Thank-you all for your support in this matter,

John
`Vk` | `Sc` | `Uc` | `Kd` | `Cs` |
-----|------|------|------|------|
VK_CONTROL | 29 | 0 | down | 40
VK_CONTROL | 29 | 0 | down | 40
VK_SPACE | 57 | ' ' | down | 40
VK_SPACE | 57 | ' ' | up | 40
VK_CONTROL | 29 | 0 | up | 32


that all looks right

This might be https://github.com/neovim/neovim/issues/19575
I believe the problem is that when you hit Control+Space nvim is expecting a different character. 

As a similar example, in the terminal when you hit Control+F you get a different set of characters:
```
Control+ F: 
    ␛[17;29;0;1;40;1_␛[70;33;6;1;40;1_␛[70;33;6;0;40;1_^F␛[17;29;0;0;32;1_
f:
    ␛[70;33;102;1;32;1_␛[70;33;102;0;32;1_f


␛[70;33;6;1;40;1_
vs
␛[70;33;102;1;32;1_

```
I *believe* it expects the "Nul" character, but I'm not sure 100% sure about this. Im not sure i there is an easy way to verify this (is it possible to emulate sending a specific character to the terminal ?). 


As an interesting side note (and as noted above)  when you type Control+Space in the right side of the split debug terminal you actual do get the Nul character (␀). For some reason that side of the terminal does something different when you hit space while having control pressed.



@jfedorkiw `<C-Space>` mapping behaves differently depending on the terminal used. This is what I've noticed:

`init.lua` used for testing:

```
vim.keymap.set('i', 'A', 'foo', {noremap=true})
vim.keymap.set('i', '<C-Space>', 'bar', {noremap=true})
```

Only insert mode was mapped since the bug was filed against `<C-Space>` not working in this mode.

For each `A` the string `foo` is substituted and for each `ctrl-space` the string `bar`.

When using the wt.exe only `A` works. There's no action when `ctrl-space` is used:

![nvim_wt_test](https://github.com/microsoft/terminal/assets/3933920/5e5055c6-edd8-4710-b223-09478f6d05a4)

The mapping is correct and it does come from `init.lua`


When using `Alacritty` a space is inserted when pressing `ctrl-space`:

![nvim_alacritty](https://github.com/microsoft/terminal/assets/3933920/a22cf3fe-c50c-425a-863e-48fa4a8e6b2e)


The mapping seems to work okay when using the GUI version of neovim:

![nvim-qt](https://github.com/microsoft/terminal/assets/3933920/2616065c-147c-448a-a3a6-74bc86d46654)

Here's a thought - what keyboard layout are you using?
I'm using a QWERTY Keyboard.

It's not clear to me there is agreement on what the core of the bug is. It might be useful to clarify. 

I'm under the impression the core problem is that the terminal isn't passing along a special key-character when the Control+Space combination is pressed.  The confusion then comes from the fact that presumably this is working on some folks setups but not mine.

It could also be the case that you don't agree the terminal should be converting Control+Space into a special key press (which seems reasonable as this clearly doesn't seem to be well defined behaviour). In that case the "fix" may be more of a feature request to allow users to bind custom key-combinations to specific key codes.


John


There's definitely no disagreement there. The trick here is trying to find what layer in particular is losing the keystroke. 

What we established up in https://github.com/microsoft/terminal/issues/15939#issuecomment-1711657303 is
* The Terminal app is getting the <kbd>ctrl+space</kbd>
* The Terminal is sending sequences corresponding to a <kbd>ctrl+space</kbd> to the connection (this is the `␛[17;29;0;1;40;1_␛[32;57;32;1;40;1_␛[32;57;32;0;40;1_␣␛[17;29;0;0;32;1_` goop)
* _(not necessarily established)_ the console converts that goop to internal `KEY_EVENT_RECORD`s, then into a `NUL` for VT input mode
* _(on my machine)_, a `showkey -a` in WSL does show NUL bytes coming through. 

Now, part of the problem here is probably just that I don't _totally_ get nvim, sorry about that. I've put the following in `.config/nvim/init.lua`, but I honestly don't know if that's working or not. Doesn't seem like it is?

```lua
" vim.keymap.set({'i', 'n', 'v', 'o'}, '', '', {noremap=true})
vim.keymap.set('i', 'A', 'foo', {noremap=true})
vim.keymap.set('i', '<C-Space>', 'bar', {noremap=true})
```

Though, when I do a `C-h,k`,`C-Space` in emacs, I do see it report the help for `C-@` (as I'd expect).

Maybe another part of the problem that could be complicating this is using WSL `nvim`, versus a Windows build of `nvim`. If it's the later, that's a good hypothesis as to the earlier comment about <kbd>ctrl+space</kbd> not working in PowerShell. 

So I'd love to find out is where in the stack the key sequence is getting lost. Is the key making it all the way through conhost? Is conhost sending something to nvim that they're throwing away? 
> So I'd love to find out is where in the stack the key sequence is getting lost. Is the key making it all the way through conhost?

@zadjii-msft I believe this is a conhost issue. I can reproduce the problem with a simple app that reads characters with `ReadConsoleA` while `ENABLE_VIRTUAL_TERMINAL_INPUT` is set.

When I press <kbd>Ctrl</kbd>+<kbd>Space</kbd> I can see it generates the `NUL` character here:

https://github.com/microsoft/terminal/blob/cc2ba5350d115b6898ac8b92482d0af7b8c4d365/src/terminal/input/terminalInput.cpp#L623-L629

But that `NUL` gets converted back into a key event in the `InputBuffer` class here:

https://github.com/microsoft/terminal/blob/cc2ba5350d115b6898ac8b92482d0af7b8c4d365/src/host/inputBuffer.cpp#L817-L820

That key event then gets process by the `GetChar` function here:

https://github.com/microsoft/terminal/blob/cc2ba5350d115b6898ac8b92482d0af7b8c4d365/src/host/stream.cpp#L234-L244

But the `zeroKey` it's trying to match here is <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>OEM_KEY_3</kbd> (at least on my machine), while the key event that was synthesized has a zero virtual key code and no modifiers. As a result, it just gets dropped.

Two possible solutions:

1. In the `InputBuffer` method where we synthesize the key event, check for a null character as a special case and convert that to <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>OEM_KEY_3</kbd> (or whatever `OneCoreSafeVkKeyScanW(0)` translates to).

2. In the `GetChar` function, where we're looking for a match with `OneCoreSafeVkKeyScanW(0)`, also consider a zero virtual key with no modifiers to be a valid match.

Frankly both of those options seem a bit messy, so I'm hoping someone that understands the code base might have a better suggestion, but at least that's a starting point.

Btw, I'm not absolutely certain this is the same issue that the OP is seeing in vim, but it seems likely.
Ah I see, that makes sense, since I made the change to the `/src/host/stream.cpp` code very recently. I made it because:
> In addition, this commit fixes at least one bug: The previous approach to detect the null key via `DoActiveModifierKeysMatch` didn't work. As it compared the modifier keys as a bitset with `==` it failed to match whenever the numpad key was set, which it usually is.

As the culprit, I'll try to fix this issue then. I think the 2nd suggestion is better, because there's no 0 value for the virtual key on any keyboard (the minimum is 1 with `VK_LBUTTON`) and so I think the detection is safe.

It might be interesting how it used to work in conhost v1:
```cpp
#define EITHER_CTRL_PRESSED (LEFT_CTRL_PRESSED | RIGHT_CTRL_PRESSED)
#define EITHER_ALT_PRESSED (LEFT_ALT_PRESSED | RIGHT_ALT_PRESSED)
#define MOD_PRESSED (SHIFT_PRESSED | EITHER_CTRL_PRESSED | EITHER_ALT_PRESSED)

DWORD ConsKbdState[] = {
    0,
    SHIFT_PRESSED,
    EITHER_CTRL_PRESSED,
    SHIFT_PRESSED | EITHER_CTRL_PRESSED,
    EITHER_ALT_PRESSED,
    SHIFT_PRESSED | EITHER_ALT_PRESSED,
    EITHER_CTRL_PRESSED | EITHER_ALT_PRESSED,
    SHIFT_PRESSED | EITHER_CTRL_PRESSED | EITHER_ALT_PRESSED
};

#define KEYEVENTSTATE_EQUAL_WINMODS(Event, WinMods)\
    ((Event.Event.KeyEvent.dwControlKeyState & ConsKbdState[WinMods]) && \
    !(Event.Event.KeyEvent.dwControlKeyState & MOD_PRESSED & ~ConsKbdState[WinMods]))

NTSTATUS GetChar(...)
{
    // ...
    SHORT sTmp = VkKeyScanW(0);
    // ...
    if ((LOBYTE(sTmp) == Event.Event.KeyEvent.wVirtualKeyCode) && KEYEVENTSTATE_EQUAL_WINMODS(Event, HIBYTE(sTmp)))
    {
        // This really is the character 0x0000
        *pwchOut = Event.Event.KeyEvent.uChar.UnicodeChar;
        return STATUS_SUCCESS;
    }
    // ...
}
```

As far as I can tell the intention of the original code is identical to what I now wrote. But the original code was flawed as far as I can tell, because the `KEYEVENTSTATE_EQUAL_WINMODS` macro did a "'has **one** of the modifiers pressed' and 'has **none** of the other modifiers pressed'" check. I believe that's wrong because you'd want to test if all the individual modifiers are matched exactly no? At a minimum I would've expected the former check to be 'has **all** of the modifiers pressed', but it doesn't do that.

Was the original v1 behavior correct?
> Ah I see, that makes sense, since I made the change to the `/src/host/stream.cpp` code very recently.

@lhecker FWIW, I don't think this was broken by any of your recent changes. I've tested in my Window 10 inbox conhost (which I think is several years old), and it was already not working in that version.
Just to note that I rechecked this behaviour and it seems to have started working for me. As [noted on the older issue](https://github.com/microsoft/terminal/issues/2865#issuecomment-1585150273) for this, I saw no keycode for ctrl+space. Now both ctrl+space and ctrl+single quote generate a null which maps to `^@`.

The only thing that seems to have changed since I last tested is the Windows version number: 10.0.22621.1778 didn't work, 10.0.22621.2283 does.
> @zadjii-msft I believe this is a conhost issue. I can reproduce the problem with a simple app that reads characters with `ReadConsoleA` while `ENABLE_VIRTUAL_TERMINAL_INPUT` is set.
> 
> When I press Ctrl+Space I can see it generates the `NUL` character here:
> 
> https://github.com/microsoft/terminal/blob/cc2ba5350d115b6898ac8b92482d0af7b8c4d365/src/terminal/input/terminalInput.cpp#L623-L629

@j4james I `SetConsoleMode` to `ENABLE_VIRTUAL_TERMINAL_INPUT`, it still goes to `_makeWin32Output`: https://github.com/microsoft/terminal/blob/cc2ba5350d115b6898ac8b92482d0af7b8c4d365/src/terminal/input/terminalInput.cpp#L488-L494

Press `ctrl + space` gets `[17;29;0;1;40;1_[32;57;32;1;40;1_[32;57;32;0;40;1_[17;29;0;0;32;1_` from https://github.com/microsoft/terminal/blob/cc2ba5350d115b6898ac8b92482d0af7b8c4d365/src/terminal/input/terminalInput.cpp#L740
@lonnywong If you're testing in Windows Terminal rather than conhost/openconsole then you're dealing with conpty, which adds another layer of redirection. Windows Terminal forwards the individual keyup/keydown events over conpty as win32-input sequences. Those sequences are later received by conhost and converted back into keyboard events. And those keyboard events are then processed by the conhost version of the `TerminalInput` class as described above.
@j4james Thanks. Pressing keys in `OpenConsole.exe`, I got a break in `GetChar`:https://github.com/microsoft/terminal/blob/cc2ba5350d115b6898ac8b92482d0af7b8c4d365/src/host/stream.cpp#L234-L244

~How do you get a break in `InputBuffer::_HandleTerminalInputCallback`?~
I `SetConsoleMode` to `ENABLE_VIRTUAL_TERMINAL_INPUT`, and got a break in `InputBuffer::_HandleTerminalInputCallback`:
https://github.com/microsoft/terminal/blob/cc2ba5350d115b6898ac8b92482d0af7b8c4d365/src/host/inputBuffer.cpp#L817-L820

Fix patch:
diff --git a/src/host/inputBuffer.cpp b/src/host/inputBuffer.cpp
index 3a4e556e440..b4c58851680 100644
--- a/src/host/inputBuffer.cpp
+++ b/src/host/inputBuffer.cpp
@@ -816,6 +816,17 @@ void InputBuffer::_HandleTerminalInputCallback(const TerminalInput::StringType&
 
         for (const auto& wch : text)
         {
+            if (wch == UNICODE_NULL)
+            {
+                // Convert null byte back to input event with proper control state
+                const auto zeroKey = OneCoreSafeVkKeyScanW(0);
+                uint32_t ctrlState = 0;
+                WI_SetFlagIf(ctrlState, SHIFT_PRESSED, WI_IsFlagSet(zeroKey, 0x100));
+                WI_SetFlagIf(ctrlState, LEFT_CTRL_PRESSED, WI_IsFlagSet(zeroKey, 0x200));
+                WI_SetFlagIf(ctrlState, LEFT_ALT_PRESSED, WI_IsFlagSet(zeroKey, 0x400));
+                _storage.push_back(SynthesizeKeyEvent(true, 1, LOBYTE(zeroKey), 0, wch, ctrlState));
+                continue;
+            }
             _storage.push_back(SynthesizeKeyEvent(true, 1, 0, 0, wch, 0));
         }
 


Test patch: