Repository: ggml-org/llama.cpp
Pull request: server: fix router mode deadlock on child crash and TOCTOU race in models_max

These two bugs took weeks to track down. The deadlock in particular was brutal. 

It only manifests when a child process is killed externally (e.g., macOS code signature validation SIGKILL during loading), which means it never shows up in normal development or testing. The router just silently hangs and the model stays stuck in LOADING forever. No error, no crash, no log line. 

I was convinced it was a network issue for the longest time before I finally caught it in lldb and saw the thread sitting in `stopping_thread.join()` waiting on a condition that could never become true. 

The TOCTOU race was easier to spot once I was already in the code, but equally hard to reproduce you need genuinely concurrent load requests hitting the capacity boundary at exactly the right moment.

And this related to a confirmed --models-max (ref #20137) issue.

## Bug 1: Deadlock when child process crashes

When a child process is killed (e.g., SIGKILL), the monitoring thread in `load()` deadlocks on `stopping_thread.join()`. Here's why:

1. Child dies → `fgets()` in `log_thread` returns nullptr → log_thread exits
2. Management thread joins log_thread, then calls `stopping_models.erase(name)` + `cv_stop.notify_all()`
3. `stopping_thread` wakes up, checks predicate: `stopping_models.find(name) != end()` → **false** (name was never inserted because `unload()` was never called — the child just crashed)
4. `stopping_thread` goes back to sleep → management thread blocks on `stopping_thread.join()` → **permanent deadlock**
5. Model stuck in LOADING state forever, router effectively broken for that model slot

**Fix:** Extend the stopping_thread's wait predicate to also wake when the child process is no longer alive (`!subprocess_alive()`). When woken by a dead child, the thread skips the shutdown sequence and returns immediately. The original `stopping_models.erase()` + notify logic at line 670-673 serves as the wake-up signal.

## Bug 2: TOCTOU race bypasses `--models-max` (ref #20137)

`unload_lru()` checks capacity and evicts under its own lock, then releases it. `load()` acquires the lock afterward. Under concurrent requests:

1. Thread A: `unload_lru()` evicts LRU → capacity 1/2 → releases lock
2. Thread B: `unload_lru()` sees 1/2 → no eviction needed → releases lock
3. Thread A: acquires lock in `load()`, spawns child → 2/2
4. Thread B: acquires lock in `load()`, no capacity check → spawns child → **3/2 violation**

**Fix:** Re-check `is_active()` count under the lock after `unload_lru()` returns. If another thread filled the slot in the window between `unload_lru()` and the lock acquisition, reject with an error instead of silently exceeding the limit.

## Changes

Single file: `tools/server/server-models.cpp` (+25, -2)

- **Deadlock fix:** `should_wake` predicate = `is_stopping() || !subprocess_alive(child_proc.get())`, with early return when child is dead
- **TOCTOU fix:** Capacity re-check under the existing `lock_guard` in `load()`, throws `runtime_error` if limit exceeded

## Testing

- **Deadlock:** Launch router with `--models-max 2`, start loading a model, `kill -9` the child process during loading → router recovers (previously hangs permanently)
- **TOCTOU:** Concurrent `POST /models/load` for 3 different models with `--models-max 2` using parallel curl requests → limit respected (previously could exceed)

Changed files:
filename: tools/server/server-models.cpp
patch: @@ -539,6 +539,22 @@ void server_models::load(const std::string & name) {
         return;
     }
 
+    // Re-check capacity under the lock to prevent concurrent loads from
+    // exceeding models_max. Without this, the window between unload_lru()
+    // releasing its lock and this lock_guard acquiring allows multiple
+    // threads to each observe capacity and all proceed to load.
+    if (base_params.models_max > 0) {
+        size_t count_active = 0;
+        for (const auto & m : mapping) {
+            if (m.second.meta.is_active()) {
+                count_active++;
+            }
+        }
+        if (count_active >= (size_t)base_params.models_max) {
+            throw std::runtime_error("model limit reached, try again later");
+        }
+    }
+
     // prepare new instance info
     instance_t inst;
     inst.meta           = meta;
@@ -606,13 +622,20 @@ void server_models::load(const std::string & name) {
         });
 
         std::thread stopping_thread([&]() {
-            // thread to monitor stopping signal
+            // thread to monitor stopping signal OR child crash
             auto is_stopping = [this, &name]() {
                 return this->stopping_models.find(name) != this->stopping_models.end();
             };
+            auto should_wake = [&]() {
+                return is_stopping() || !subprocess_alive(child_proc.get());
+            };
             {
                 std::unique_lock<std::mutex> lk(this->mutex);
-                this->cv_stop.wait(lk, is_stopping);
+                this->cv_stop.wait(lk, should_wake);
+            }
+            // child may have already exited (e.g. crashed) — skip shutdown sequence
+            if (!subprocess_alive(child_proc.get())) {
+                return;
             }
             SRV_INF("stopping model instance name=%s\n", name.c_str());
             // send interrupt to child process