Repository: wezterm/wezterm (language: rust)

Issue:
WezTerm laggy when under CPU load and terminal output
### What Operating System(s) are you seeing this problem on?

Linux X11

### Which Wayland compositor or X11 Window manager(s) are you using?

xmonad 0.17.1

### WezTerm version

wezterm 20230312-204801-4afedd62

### Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

No, and I'll explain why below

### Describe the bug

I have some embedded devices with serial ports connected to my workstation, which I monitor by running [tio](https://github.com/tio/tio) in a WezTerm terminal window. At the same time I frequently run CPU intensive build jobs.

I found that if one of the embedded devices begins sending debug logs to the serial port at the maximum speed (115200bps) while one of the build jobs is running, that causes all of my WezTerm terminal windows to freeze up for seconds at a time. Looking at `top` in another terminal emulator, I see that the `wezterm-gui` process is running at close to 100% CPU while this happens. My guess is that WezTerm becomes starved for CPU on account of the build jobs and cannot keep up with the output from the serial port.

I tried running the latest nightly but I ran into an unrelated issue that prevented wezterm from starting. I will update the bug if I manage to reproduce with the latest version.

### To Reproduce

1. Start a CPU intensive build job, e.g. build LLVM, Chromium or AOSP.
2. Run the following command in a terminal window to simulate a debug log:
```
sh -c 'while true; do date; sleep 0.001; done'
```
3. Wait a few seconds.
4. Observe that the date printed to the terminal window starts updating less frequently.

### Configuration

```
local wezterm = require 'wezterm'
local act = wezterm.action

local keys = nil
if wezterm.gui then
  keys = wezterm.gui.default_keys()
  table.insert(
    keys,
    { key = 'Home', mods = 'SHIFT', action = act.ScrollToTop })
  table.insert(
    keys,
    { key = 'End', mods = 'SHIFT', action = act.ScrollToBottom })
end

return {
        scrollback_lines = 10000000,
        enable_scroll_bar = true,
        hide_tab_bar_if_only_one_tab = true,
        window_padding = {
    left = 0,
    right = 0,
    top = 0,
    bottom = 0,
  },
  keys = keys,
  font_size = 11.5,
}
```

### Expected Behavior

No lag in (at least) other terminal windows.

### Logs

_No response_

### Anything else?

_No response_


Discussion / hints:
Confirmed that the issue still reproduces with
```
wezterm 20230407-181947-6bd3664d
```
but so far I've only been able to reproduce it with a real serial port and not with the simulated debug logging in my earlier reproducer. I'll see if I can come up with a better reproducer.
Are you saying that when your whole system is overloaded, wezterm is laggy?
Or that wezterm is laggy even though there is still CPU headroom on your system?

I'm saying that while I am running one of these build jobs, WezTerm is the only application that is laggy. I can still run other terminal emulators and Chrome just fine.
Can you find a repro that doesn't involve building LLVM or chrome or anything else?
eg: What if you generate a lot of output in two windows at the same time?

You have the scrollback_lines set really high. Is the lag a manifestation of memory pressure/swap? How many wezterm panes do you have, and how many of them are close to the limit of your scrollback?

I looked at narrowing down the reproducer a bit. I'm not sure if this is the same issue or not, but here goes -- if I run this program in a WezTerm window:
```
#include <stdlib.h>
#include <time.h>
#include <unistd.h>

int main() {
  struct timespec b115200 = { 0, 1000000000 / (115200 / 8) };
  int counter = 0;
  int line_length = rand() % 80;
  while (1) {
    nanosleep(&b115200, 0);
    char buf = '0' + (counter++ % 10);
    write(STDOUT_FILENO, &buf, 1);
    if (line_length-- == 0) {
      buf = '\n';
      write(STDOUT_FILENO, &buf, 1);
      line_length = rand() % 80;
    }
  }
}
```
the terminal window where I run this program usually stops printing output entirely, but occasionally I see a screenful of numbers. I only see updates in that window if I press Ctrl-C. Unlike in the original report, the other terminal windows continue responding properly, and it is independent of any other jobs I have running concurrently. In other terminal emulators (I tried gnome-terminal and xterm), the program produces a continuous stream of numbers on the screen at 115200bps, as I'd expect.

I set scrollback_lines high because I don't want to lose terminal output and I can afford it in terms of RAM (I am fortunate enough to have 192 GiB of RAM on my workstation). Checking `free` while running one of the build jobs, it does not reach half of the total RAM size. I checked the RSS for my `wezterm-gui` process around when I experienced the lag and it was around 600 MiB. Currently I have around 5 panes open.

Fix patch:
diff --git a/mux/src/lib.rs b/mux/src/lib.rs
index 28e211189bc..2fa6226b67e 100644
--- a/mux/src/lib.rs
+++ b/mux/src/lib.rs
@@ -141,7 +141,8 @@ fn parse_buffered_data(pane: Weak<dyn Pane>, dead: &Arc<AtomicBool>, mut rx: Fil
     let mut actions = vec![];
     let mut hold = false;
     let mut action_size = 0;
-    let mut delay_ms = configuration().mux_output_parser_coalesce_delay_ms;
+    let mut delay = Duration::from_millis(configuration().mux_output_parser_coalesce_delay_ms);
+    let mut deadline = None;
 
     loop {
         match rx.read(&mut buf) {
@@ -194,13 +195,20 @@ fn parse_buffered_data(pane: Weak<dyn Pane>, dead: &Arc<AtomicBool>, mut rx: Fil
                     // that we coalesce a full "frame" from an unoptimized
                     // TUI program
                     if action_size < buf.len() {
-                        if delay_ms > 0 {
+                        let poll_delay = match deadline {
+                            None => {
+                               deadline.replace(Instant::now() + delay);
+                               Some(delay)
+                            },
+                            Some(target) => target.checked_duration_since(Instant::now()),
+                        };
+                        if poll_delay.is_some() {
                             let mut pfd = [pollfd {
                                 fd: rx.as_socket_descriptor(),
                                 events: POLLIN,
                                 revents: 0,
                             }];
-                            if let Ok(1) = poll(&mut pfd, Some(Duration::from_millis(delay_ms))) {
+                            if let Ok(1) = poll(&mut pfd, poll_delay) {
                                 // We can read now without blocking, so accumulate
                                 // more data into actions
                                 continue;
@@ -212,12 +220,13 @@ fn parse_buffered_data(pane: Weak<dyn Pane>, dead: &Arc<AtomicBool>, mut rx: Fil
                     }
 
                     send_actions_to_mux(&pane, &dead, std::mem::take(&mut actions));
+                    deadline = None;
                     action_size = 0;
                 }
 
                 let config = configuration();
                 buf.resize(config.mux_output_parser_buffer_size, 0);
-                delay_ms = config.mux_output_parser_coalesce_delay_ms;
+                delay = Duration::from_millis(config.mux_output_parser_coalesce_delay_ms);
             }
         }
     }


Test patch: