Repository: denoland/deno
Pull request: perf: trim glibc malloc arena after module loading

## Summary

- Calls `malloc_trim(0)` on Linux after the main/side module loading phase completes, forcing glibc to release unused heap pages back to the OS
- Addresses 3-5x higher RSS on Linux vs Windows when loading large TypeScript codebases (e.g. generated OpenAPI clients)
- Module compilation creates heavy allocation churn that glibc's allocator retains in its arena; `malloc_trim` reclaims this

### Root cause

glibc's malloc doesn't automatically return freed memory to the OS — it keeps freed pages in its arena for future reuse. On Windows, the NT heap manager coalesces and releases freed memory much more aggressively. After TypeScript compilation/module loading (which creates many temporary allocations for AST nodes, source maps, etc.), glibc holds onto all that memory even though it's been freed.

### Why this is safe

- `malloc_trim` is a standard glibc function, safe to call at any time
- It only releases memory already freed — no impact on live allocations  
- Guarded by `#[cfg(target_os = "linux")]` — no-op on other platforms
- The existing `DENO_USR2_MEMORY_TRIM` mechanism already uses the same call, just triggered externally via SIGUSR2

Closes #25722

## Test plan

- [ ] Verify compilation succeeds on Linux (`cargo check -p deno`)
- [ ] Run spec tests (`cargo test specs`)
- [ ] Test with a large TypeScript project on Linux and measure RSS before/after with `Deno.memoryUsage().rss`
- [ ] Verify no regression on Windows/macOS (the `malloc_trim` call is cfg-gated to Linux only)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Changed files:
filename: Cargo.lock
patch: @@ -2591,6 +2591,7 @@ dependencies = [
  "env_logger",
  "faster-hex",
  "indexmap 2.12.0",
+ "libc",
  "libsui",
  "log",
  "node_resolver",

filename: cli/lib/Cargo.toml
patch: @@ -44,3 +44,6 @@ sys_traits = { workspace = true, features = ["getrandom"] }
 thiserror.workspace = true
 twox-hash.workspace = true
 url.workspace = true
+
+[target.'cfg(target_os = "linux")'.dependencies]
+libc.workspace = true

filename: cli/lib/worker.rs
patch: @@ -874,12 +874,36 @@ impl LibMainWorker {
 
   pub async fn execute_main_module(&mut self) -> Result<(), CoreError> {
     let id = self.worker.preload_main_module(&self.main_module).await?;
-    self.worker.evaluate_module(id).await
+    self.worker.evaluate_module(id).await?;
+
+    // After loading and evaluating all modules, trim the glibc malloc arena.
+    // Module loading/TypeScript compilation creates heavy allocation churn
+    // that glibc's allocator doesn't release back to the OS, causing RSS on
+    // Linux to be much higher than on other platforms (see #25722).
+    #[cfg(all(target_os = "linux", target_env = "gnu"))]
+    {
+      // SAFETY: calling libc malloc_trim which is safe to call at any time.
+      unsafe {
+        libc::malloc_trim(0);
+      }
+    }
+
+    Ok(())
   }
 
   pub async fn execute_side_module(&mut self) -> Result<(), CoreError> {
     let id = self.worker.preload_side_module(&self.main_module).await?;
-    self.worker.evaluate_module(id).await
+    self.worker.evaluate_module(id).await?;
+
+    #[cfg(all(target_os = "linux", target_env = "gnu"))]
+    {
+      // SAFETY: calling libc malloc_trim which is safe to call at any time.
+      unsafe {
+        libc::malloc_trim(0);
+      }
+    }
+
+    Ok(())
   }
 
   pub async fn execute_preload_modules(&mut self) -> Result<(), CoreError> {