Repository: microsoft/terminal (language: cpp)

Issue:
Characters have black background in Terminal Preview 1.18.1462.0
### Windows Terminal version

1.18.1462.0

### Windows build number

10.0.22622.0

### Other Software

_No response_

### Steps to reproduce

I'm not sure yet what the reproduction steps but the rendering issue seems to present itself immediately upon opening the terminal application.

### Expected Behavior

Rendering of text to be normal

### Actual Behavior

Each character has a black background which is very noticeable if using a light colour scheme, for example:

![image](https://github.com/microsoft/terminal/assets/76320/b45a552c-562d-4f28-93b7-823e198118ca)

Changing the font or font size does not seem to have any effect.


Discussion / hints:
You do have a very curious GPU. 😅 Questions:
* Do you use something like MacType?
* Does it happen if you disable ClearType Anti-Aliasing?
* When you open regedit and navigate to `HKEY_CURRENT_USER\Software\Microsoft\Avalon.Graphics` what keys and values do you see? For instance, I only have `DISPLAY1` there and it looks like that: 
![image](https://github.com/microsoft/terminal/assets/2256941/7d7d6cba-bcda-406b-a471-f466ff971fa6)
  If you have multiple displays there, I'd unfortunately need all unique sets of values (= you can skip those displays that have identical registry values like another one; I'm just interested in the possible combinations that you have as they affect the text shader.)
* Open the settings app and navigate to System > Display > Graphics (it's at the bottom of the page). There should be a "Add an app" dropdown that uses "Desktop apps" by default. Choose "Microsoft store app" instead: 
![image](https://github.com/microsoft/terminal/assets/2256941/c8cd8624-ddb7-4251-908f-ed56db1587da)
  Click browse (marvel at the Windows 8 dialog) and choose Terminal Preview. Then scroll down, click Terminal Preview and choose Options: 
![image](https://github.com/microsoft/terminal/assets/2256941/eb9604b4-c551-4890-96b7-9e2b34a79682)
  In that dialog you can pick the default graphics adapter for an application. Do you have multiple GPUs to choose from? If so, please try the various options. (The difference between "power saving" and "high performance" doesn't matter, only the difference between GPUs does.)

Note to self:
* It can't be the GPU/texture memory, because the burrito emoji has a proper alpha=0 background
* It can't be the dual blend state, because the burrito emoji works
* It could be D2D or a mod for D2D, if it where to fill the clip rect with black with alpha=1 when we call `ID2D1DeviceContext::DrawGlyphRun`. It doesn't do that for other users though so it's unlikely.
* It could be the DWrite (Avalon) settings, because those are being passed to the shader
* Asking for a [PIX](https://devblogs.microsoft.com/pix/download/) recording would be an option, but it requires installing software and takes a bit to set up.
I know right! Why is it always me with the rendering problems? 😬😉

Do you use something like MacType?
Nothing like that, just standard Windows rendering

Does it happen if you disable ClearType Anti-Aliasing?
Same issue with ClearType disabled

Registry values for this single display:
![image](https://github.com/microsoft/terminal/assets/76320/67398f08-d4da-45e5-abec-25d6b2c68e74)

This is the dialog when opening the options:
![image](https://github.com/microsoft/terminal/assets/76320/a604b7f1-652e-4b11-a5cd-fd56f30f3f26)

I then picked Power saving NVIDIA 2070 Super and re-opened Terminal Preview but the black background to the characters is still there.

I also tried adjusting the clear type options until the registry values were the same as yours but the result is the same in Terminal preview.



Ugh... I was hoping that it's a bug with the Avalon settings. If it's not too much to ask, could you please install PIX? You can find it here: https://devblogs.microsoft.com/pix/download/
It'll ask you a couple questions on the first launch and it doesn't really matter what you pick. It's just for their user/market research.

You then need to launch it as Admin (it'll allow you to launch as non-Admin but silently won't record any GPU metrics). Quit all instances of Windows Terminal Preview. Then, do this (the important bits are "Launch UWP" and "Launch for GPU capture"):
![image](https://github.com/microsoft/terminal/assets/2256941/54c0ce4f-5b15-4274-b25f-c213d59ddb7d)

After clicking "Launch", it'd be nice if you could choose a light theme like you did before, insert an emoji like 🌯, and then press Ctrl+A to select all text. Like so:
![image](https://github.com/microsoft/terminal/assets/2256941/275d7a29-b785-4892-b1c3-b0b49b14d059)

Back in PIX, press Ctrl+Shift+N. Make sure the green box looks like this:
![image](https://github.com/microsoft/terminal/assets/2256941/04a618ec-ed21-4c11-a2b4-b5e90090af6d)

Then press Ctrl+N (or click the camera icon) and then focus the terminal again. It should quickly create a capture like so:
![image](https://github.com/microsoft/terminal/assets/2256941/1b792b7a-d8d5-4dd3-865e-6f3f6f0804fd)

Double click the capture and then press Ctrl+S. If you could send this file to `<my github username>@microsoft.com`, that'd be really helpful! (I hope it works when I analyze it on a different GPU though. I never tried that.)

---

You can also investigate it yourself if you're curious or just prefer that. To do that, after opening the capture, click on the little start arrow in the top right:
![image](https://github.com/microsoft/terminal/assets/2256941/88881c0c-77cd-4a60-a559-4c6641f6ee5a)

Then pick graphics queue 1 (WinUI unfortunately uses its own graphics queue = 0):
![image](https://github.com/microsoft/terminal/assets/2256941/6787704e-5d3d-4df8-963f-c42c897e8d87)

Click on the "Pipeline" tab, and click on the blue `DrawIndexedInstanced` entry in the list. At the bottom you'll see all the GPU stages.
* IA: Input Assembler - All the per-frame data that is being sent to the GPU
* VS: Vertex Shader - In our case it translates from absolute coordinates in pixels (top left corner is 0,0) to screen space coordinates that range from -1 to +1 in both the X and Y axis.
* PS: Pixel Shader - The thing that is probably failing right now. Texture 1 is the glyph atlas.

`CBV` are the Constant Buffer Values, i.e. the values that are constant across one entire shader invocation for all pixels. You won't be able to read the CBV unless you tell PIX its layout (I think this is a limitation due to the use of D3D11on12). To read it, double click on the CBV entry on the left and it'll pop up an Active Resource tab. There, you need to enter the struct definition which in case of the Pixel Shader is:
```
float4 backgroundColor;
float2 backgroundCellSize;
float2 backgroundCellCount;
float4 gammaRatios;
float enhancedContrast;
float underlineWidth;
```
Expand item `[0]` and it'll show you all relevant values. In my case:
![image](https://github.com/microsoft/terminal/assets/2256941/7928bc34-fce8-491e-880b-3a287629b67e)
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for **4 days**. It will be closed if no further activity occurs **within 3 days of this comment**.
@lhecker I've emailed you the wpix file directly. I didn't analyse the file but it here is a screen capture of the pix screenshot next to the actual terminal window. Is the pix screenshot supposed to be black like that?

![image](https://github.com/microsoft/terminal/assets/76320/c11ba69c-a7c0-417d-aaea-f8591a74751e)

@tanc The PIX dump and your settings.json file helped me a lot! Quick workaround:
Update to the latest Windows Terminal (even non-Preview is fine) and remove the `passthrough.hlsl` shader from your settings.json (Ctrl+Shift+,). 🙂

I haven't 100% understood yet why it happens but I know what causes this effect. Once I open a PR, I'll comment there what caused this issue.

![image](https://github.com/microsoft/terminal/assets/2256941/85fc8196-be06-4416-8e88-38cd2ffa47d6)

@lhecker ah of course! It was the `passthrough.hlsl` we added to debug #15199 
Removing that has fixed the issue. I'll close this issue as I don't think it'll affect anyone else.
I'm glad it fixed it! For now I'd like to keep the issue open however, since this shouldn't happen if you use a shader and so I still got something to fix. 😅
(Please feel free to unsubscribe if you'd like!)

Fix patch:
diff --git a/src/renderer/atlas/AtlasEngine.api.cpp b/src/renderer/atlas/AtlasEngine.api.cpp
index 4459c97a8e5..c810dd115f6 100644
--- a/src/renderer/atlas/AtlasEngine.api.cpp
+++ b/src/renderer/atlas/AtlasEngine.api.cpp
@@ -490,20 +490,20 @@ void AtlasEngine::UpdateHyperlinkHoveredId(const uint16_t hoveredId) noexcept
 
 void AtlasEngine::_resolveTransparencySettings() noexcept
 {
+    // An opaque background allows us to use true "independent" flips. See AtlasEngine::_createSwapChain().
+    // We can't enable them if custom shaders are specified, because it's unknown, whether they support opaque inputs.
+    const bool useAlpha = _api.enableTransparentBackground || !_api.s->misc->customPixelShaderPath.empty();
     // If the user asks for ClearType, but also for a transparent background
     // (which our ClearType shader doesn't simultaneously support)
     // then we need to sneakily force the renderer to grayscale AA.
-    const auto antialiasingMode = _api.enableTransparentBackground && _api.antialiasingMode == AntialiasingMode::ClearType ? AntialiasingMode::Grayscale : _api.antialiasingMode;
-    const bool enableTransparentBackground = _api.enableTransparentBackground || !_api.s->misc->customPixelShaderPath.empty() || _api.s->misc->useRetroTerminalEffect;
+    const auto antialiasingMode = useAlpha && _api.antialiasingMode == AntialiasingMode::ClearType ? AntialiasingMode::Grayscale : _api.antialiasingMode;
 
-    if (antialiasingMode != _api.s->font->antialiasingMode || enableTransparentBackground != _api.s->target->enableTransparentBackground)
+    if (antialiasingMode != _api.s->font->antialiasingMode || useAlpha != _api.s->target->useAlpha)
     {
         const auto s = _api.s.write();
         s->font.write()->antialiasingMode = antialiasingMode;
-        // An opaque background allows us to use true "independent" flips. See AtlasEngine::_createSwapChain().
-        // We can't enable them if custom shaders are specified, because it's unknown, whether they support opaque inputs.
-        s->target.write()->enableTransparentBackground = enableTransparentBackground;
-        _api.backgroundOpaqueMixin = enableTransparentBackground ? 0x00000000 : 0xff000000;
+        s->target.write()->useAlpha = useAlpha;
+        _api.backgroundOpaqueMixin = useAlpha ? 0x00000000 : 0xff000000;
     }
 }
 
diff --git a/src/renderer/atlas/AtlasEngine.r.cpp b/src/renderer/atlas/AtlasEngine.r.cpp
index e7070b708e6..1fac8f0e834 100644
--- a/src/renderer/atlas/AtlasEngine.r.cpp
+++ b/src/renderer/atlas/AtlasEngine.r.cpp
@@ -329,7 +329,7 @@ void AtlasEngine::_createSwapChain()
         .SwapEffect = DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL,
         // If our background is opaque we can enable "independent" flips by setting DXGI_ALPHA_MODE_IGNORE.
         // As our swap chain won't have to compose with DWM anymore it reduces the display latency dramatically.
-        .AlphaMode = _p.s->target->enableTransparentBackground ? DXGI_ALPHA_MODE_PREMULTIPLIED : DXGI_ALPHA_MODE_IGNORE,
+        .AlphaMode = _p.s->target->useAlpha ? DXGI_ALPHA_MODE_PREMULTIPLIED : DXGI_ALPHA_MODE_IGNORE,
         .Flags = swapChainFlags,
     };
 
@@ -360,6 +360,8 @@ void AtlasEngine::_createSwapChain()
     _p.swapChain.targetSize = _p.s->targetSize;
     _p.swapChain.waitForPresentation = true;
 
+    LOG_IF_FAILED(_p.swapChain.swapChain->SetMaximumFrameLatency(1));
+
     WaitUntilCanRender();
 
     if (_p.swapChainChangedCallback)
diff --git a/src/renderer/atlas/BackendD3D.cpp b/src/renderer/atlas/BackendD3D.cpp
index 846567ca613..19334b3a9d4 100644
--- a/src/renderer/atlas/BackendD3D.cpp
+++ b/src/renderer/atlas/BackendD3D.cpp
@@ -403,23 +403,24 @@ void BackendD3D::_recreateCustomShader(const RenderingPayload& p)
             /* ppCode      */ blob.addressof(),
             /* ppErrorMsgs */ error.addressof());
 
-        // Unless we can determine otherwise, assume this shader requires evaluation every frame
-        _requiresContinuousRedraw = true;
-
         if (SUCCEEDED(hr))
         {
-            THROW_IF_FAILED(p.device->CreatePixelShader(blob->GetBufferPointer(), blob->GetBufferSize(), nullptr, _customPixelShader.put()));
+            THROW_IF_FAILED(p.device->CreatePixelShader(blob->GetBufferPointer(), blob->GetBufferSize(), nullptr, _customPixelShader.addressof()));
 
             // Try to determine whether the shader uses the Time variable
             wil::com_ptr<ID3D11ShaderReflection> reflector;
-            if (SUCCEEDED_LOG(D3DReflect(blob->GetBufferPointer(), blob->GetBufferSize(), IID_PPV_ARGS(reflector.put()))))
+            if (SUCCEEDED_LOG(D3DReflect(blob->GetBufferPointer(), blob->GetBufferSize(), IID_PPV_ARGS(reflector.addressof()))))
             {
+                // Depending on the version of the d3dcompiler_*.dll, the next two functions either return nullptr
+                // on failure or an instance of CInvalidSRConstantBuffer or CInvalidSRVariable respectively,
+                // which cause GetDesc() to return E_FAIL. In other words, we have to assume that any failure in the
+                // next few lines indicates that the cbuffer is entirely unused (--> _requiresContinuousRedraw=false).
                 if (ID3D11ShaderReflectionConstantBuffer* constantBufferReflector = reflector->GetConstantBufferByIndex(0)) // shader buffer
                 {
                     if (ID3D11ShaderReflectionVariable* variableReflector = constantBufferReflector->GetVariableByIndex(0)) // time
                     {
                         D3D11_SHADER_VARIABLE_DESC variableDescriptor;
-                        if (SUCCEEDED_LOG(variableReflector->GetDesc(&variableDescriptor)))
+                        if (SUCCEEDED(variableReflector->GetDesc(&variableDescriptor)))
                         {
                             // only if time is used
                             _requiresContinuousRedraw = WI_IsFlagSet(variableDescriptor.uFlags, D3D_SVF_USED);
@@ -427,6 +428,11 @@ void BackendD3D::_recreateCustomShader(const RenderingPayload& p)
                     }
                 }
             }
+            else
+            {
+                // Unless we can determine otherwise, assume this shader requires evaluation every frame
+                _requiresContinuousRedraw = true;
+            }
         }
         else
         {
@@ -447,8 +453,6 @@ void BackendD3D::_recreateCustomShader(const RenderingPayload& p)
     else if (p.s->misc->useRetroTerminalEffect)
     {
         THROW_IF_FAILED(p.device->CreatePixelShader(&custom_shader_ps[0], sizeof(custom_shader_ps), nullptr, _customPixelShader.put()));
-        // We know the built-in retro shader doesn't require continuous redraw.
-        _requiresContinuousRedraw = false;
     }
 
     if (_customPixelShader)
diff --git a/src/renderer/atlas/common.h b/src/renderer/atlas/common.h
index bb46ff61547..94434bffc9c 100644
--- a/src/renderer/atlas/common.h
+++ b/src/renderer/atlas/common.h
@@ -313,7 +313,7 @@ namespace Microsoft::Console::Render::Atlas
     struct TargetSettings
     {
         HWND hwnd = nullptr;
-        bool enableTransparentBackground = false;
+        bool useAlpha = false;
         bool useSoftwareRendering = false;
     };
 


Test patch: