Repository: rust-lang/rust
Pull request #139660: compiletest: Add an experimental new executor to replace libtest
Author: login: Zalathar
Labels: name: A-testsuite
description: Area: The testsuite used to check the correctness of rustc
name: T-bootstrap
description: Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
name: S-waiting-on-bors
description: Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
name: A-compiletest
description: Area: The compiletest test runner
Created: 2025-04-11T06:55:51Z | Merged: 2025-04-15T11:10:29Z

This PR adds a new "executor" to compiletest for running the list of collected tests, to eventually replace the current dependency on unstable libtest internals.

The new executor is currently inactive by default. It must be activated explicitly by passing `-n` or `--new-executor` to compiletest, e.g. `./x test ui -- -n`.

(After some amount of wider manual testing, the new executor will hopefully be made the default, and the libtest dependency can be removed. Contributors should not notice any change.)

The new executor is a stripped-down rewrite of the subset of libtest needed by compiletest.

# Supported functionality
- Specifying the number of concurrent tests with `RUST_TEST_THREADS`
- Filtering and skipping tests by name (substring or exact-match)
- Forcibly running ignored tests with `--ignored`
- Optional fail-fast with `--fail-fast`
- JSON output, compatible with bootstrap's parser for libtest output
- Running each test in its own thread
- Short backtraces that ignore the executor itself
- Slow test detection, with a hard-coded timeout of 60 seconds
- Capturing stdout/stderr, via `#![feature(internal_output_capture)]`
- Suppressing output capture with `--no-capture`

# Unsupported functionality
- Non-JSON output, as this is handled by bootstrap instead
- Separate code path for concurrency=1, as the concurrent path should handle this case naturally
- Fallback to running tests synchronously if new threads can't be spawned
- Special handling of hosts that don't support basic functionality like threads or timers
  - Our ability to test *targets* should be unaffected
- Graceful handling of some edge cases that could occur in arbitrary user-written unit tests, but would represent bugs in compiletest
- Due to the current need for output capture, the new executor is still not entirely written in stable Rust

---
r? jieyouxu


Diff:
diff --git a/src/bootstrap/src/core/build_steps/check.rs b/src/bootstrap/src/core/build_steps/check.rs
index b191d0f6b306b..ae9511b786741 100644
--- a/src/bootstrap/src/core/build_steps/check.rs
+++ b/src/bootstrap/src/core/build_steps/check.rs
@@ -3,7 +3,7 @@
 use crate::core::build_steps::compile::{
     add_to_sysroot, run_cargo, rustc_cargo, rustc_cargo_env, std_cargo, std_crates_for_run_make,
 };
-use crate::core::build_steps::tool::{SourceType, prepare_tool_cargo};
+use crate::core::build_steps::tool::{COMPILETEST_ALLOW_FEATURES, SourceType, prepare_tool_cargo};
 use crate::core::builder::{
     self, Alias, Builder, Kind, RunConfig, ShouldRun, Step, crate_description,
 };
@@ -416,7 +416,7 @@ impl Step for Compiletest {
             &[],
         );
 
-        cargo.allow_features("test");
+        cargo.allow_features(COMPILETEST_ALLOW_FEATURES);
 
         // For ./x.py clippy, don't run with --all-targets because
         // linting tests and benchmarks can produce very noisy results
diff --git a/src/bootstrap/src/core/build_steps/test.rs b/src/bootstrap/src/core/build_steps/test.rs
index e23c1ab5a238b..b1a3bba08871d 100644
--- a/src/bootstrap/src/core/build_steps/test.rs
+++ b/src/bootstrap/src/core/build_steps/test.rs
@@ -15,7 +15,7 @@ use crate::core::build_steps::doc::DocumentationFormat;
 use crate::core::build_steps::gcc::{Gcc, add_cg_gcc_cargo_flags};
 use crate::core::build_steps::llvm::get_llvm_version;
 use crate::core::build_steps::synthetic_targets::MirOptPanicAbortSyntheticTarget;
-use crate::core::build_steps::tool::{self, SourceType, Tool};
+use crate::core::build_steps::tool::{self, COMPILETEST_ALLOW_FEATURES, SourceType, Tool};
 use crate::core::build_steps::toolstate::ToolState;
 use crate::core::build_steps::{compile, dist, llvm};
 use crate::core::builder::{
@@ -721,7 +721,7 @@ impl Step for CompiletestTest {
             SourceType::InTree,
             &[],
         );
-        cargo.allow_features("test");
+        cargo.allow_features(COMPILETEST_ALLOW_FEATURES);
         run_cargo_test(cargo, &[], &[], "compiletest self test", host, builder);
     }
 }
diff --git a/src/bootstrap/src/core/build_steps/tool.rs b/src/bootstrap/src/core/build_steps/tool.rs
index ded7220fcedcd..3426da51a808c 100644
--- a/src/bootstrap/src/core/build_steps/tool.rs
+++ b/src/bootstrap/src/core/build_steps/tool.rs
@@ -444,7 +444,11 @@ macro_rules! bootstrap_tool {
                         SourceType::InTree
                     },
                     extra_features: vec![],
-                    allow_features: concat!($($allow_features)*),
+                    allow_features: {
+                        let mut _value = "";
+                        $( _value = $allow_features; )?
+                        _value
+                    },
                     cargo_args: vec![],
                     artifact_kind: if false $(|| $artifact_kind == ToolArtifactKind::Library)* {
                         ToolArtifactKind::Library
@@ -458,6 +462,8 @@ macro_rules! bootstrap_tool {
     }
 }
 
+pub(crate) const COMPILETEST_ALLOW_FEATURES: &str = "test,internal_output_capture";
+
 bootstrap_tool!(
     // This is marked as an external tool because it includes dependencies
     // from submodules. Trying to keep the lints in sync between all the repos
@@ -468,7 +474,7 @@ bootstrap_tool!(
     Tidy, "src/tools/tidy", "tidy";
     Linkchecker, "src/tools/linkchecker", "linkchecker";
     CargoTest, "src/tools/cargotest", "cargotest";
-    Compiletest, "src/tools/compiletest", "compiletest", is_unstable_tool = true, allow_features = "test";
+    Compiletest, "src/tools/compiletest", "compiletest", is_unstable_tool = true, allow_features = COMPILETEST_ALLOW_FEATURES;
     BuildManifest, "src/tools/build-manifest", "build-manifest";
     RemoteTestClient, "src/tools/remote-test-client", "remote-test-client";
     RustInstaller, "src/tools/rust-installer", "rust-installer";
@@ -483,7 +489,8 @@ bootstrap_tool!(
     GenerateCopyright, "src/tools/generate-copyright", "generate-copyright";
     SuggestTests, "src/tools/suggest-tests", "suggest-tests";
     GenerateWindowsSys, "src/tools/generate-windows-sys", "generate-windows-sys";
-    RustdocGUITest, "src/tools/rustdoc-gui-test", "rustdoc-gui-test", is_unstable_tool = true, allow_features = "test";
+    // rustdoc-gui-test has a crate dependency on compiletest, so it needs the same unstable features.
+    RustdocGUITest, "src/tools/rustdoc-gui-test", "rustdoc-gui-test", is_unstable_tool = true, allow_features = COMPILETEST_ALLOW_FEATURES;
     CoverageDump, "src/tools/coverage-dump", "coverage-dump";
     WasmComponentLd, "src/tools/wasm-component-ld", "wasm-component-ld", is_unstable_tool = true, allow_features = "min_specialization";
     UnicodeTableGenerator, "src/tools/unicode-table-generator", "unicode-table-generator";
diff --git a/src/tools/compiletest/src/common.rs b/src/tools/compiletest/src/common.rs
index 604c5fcbddff0..31c696ed41ff4 100644
--- a/src/tools/compiletest/src/common.rs
+++ b/src/tools/compiletest/src/common.rs
@@ -414,6 +414,11 @@ pub struct Config {
     /// cross-compilation scenarios that do not otherwise want/need to `-Zbuild-std`. Used in e.g.
     /// ABI tests.
     pub minicore_path: Utf8PathBuf,
+
+    /// If true, run tests with the "new" executor that was written to replace
+    /// compiletest's dependency on libtest. Eventually this will become the
+    /// default, and the libtest dependency will be removed.
+    pub new_executor: bool,
 }
 
 impl Config {
diff --git a/src/tools/compiletest/src/executor.rs b/src/tools/compiletest/src/executor.rs
index 527d6b8a36eb1..0c173d476affa 100644
--- a/src/tools/compiletest/src/executor.rs
+++ b/src/tools/compiletest/src/executor.rs
@@ -1,22 +1,251 @@
-//! This module encapsulates all of the code that interacts directly with
-//! libtest, to execute the collected tests.
-//!
-//! This will hopefully make it easier to migrate away from libtest someday.
+//! This module contains a reimplementation of the subset of libtest
+//! functionality needed by compiletest.
 
 use std::borrow::Cow;
-use std::io;
-use std::sync::Arc;
+use std::collections::HashMap;
+use std::hash::{BuildHasherDefault, DefaultHasher};
+use std::num::NonZero;
+use std::sync::{Arc, Mutex, mpsc};
+use std::{env, hint, io, mem, panic, thread};
 
 use crate::common::{Config, TestPaths};
 
-/// Delegates to libtest to run the list of collected tests.
+mod deadline;
+mod json;
+pub(crate) mod libtest;
+
+pub(crate) fn run_tests(config: &Config, tests: Vec<CollectedTest>) -> bool {
+    let tests_len = tests.len();
+    let filtered = filter_tests(config, tests);
+    // Iterator yielding tests that haven't been started yet.
+    let mut fresh_tests = (0..).map(TestId).zip(&filtered);
+
+    let concurrency = get_concurrency();
+    assert!(concurrency > 0);
+    let concurrent_capacity = concurrency.min(filtered.len());
+
+    let mut listener = json::Listener::new();
+    let mut running_tests = HashMap::with_capacity_and_hasher(
+        concurrent_capacity,
+        BuildHasherDefault::<DefaultHasher>::new(),
+    );
+    let mut deadline_queue = deadline::DeadlineQueue::with_capacity(concurrent_capacity);
+
+    let num_filtered_out = tests_len - filtered.len();
+    listener.suite_started(filtered.len(), num_filtered_out);
+
+    // Channel used by test threads to report the test outcome when done.
+    let (completion_tx, completion_rx) = mpsc::channel::<TestCompletion>();
+
+    // Unlike libtest, we don't have a separate code path for concurrency=1.
+    // In that case, the tests will effectively be run serially anyway.
+    loop {
+        // Spawn new test threads, up to the concurrency limit.
+        // FIXME(let_chains): Use a let-chain here when stable in bootstrap.
+        'spawn: while running_tests.len() < concurrency {
+            let Some((id, test)) = fresh_tests.next() else { break 'spawn };
+            listener.test_started(test);
+            deadline_queue.push(id, test);
+            let join_handle = spawn_test_thread(id, test, completion_tx.clone());
+            running_tests.insert(id, RunningTest { test, join_handle });
+        }
+
+        // If all running tests have finished, and there weren't any unstarted
+        // tests to spawn, then we're done.
+        if running_tests.is_empty() {
+            break;
+        }
+
+        let completion = deadline_queue
+            .read_channel_while_checking_deadlines(&completion_rx, |_id, test| {
+                listener.test_timed_out(test);
+            })
+            .expect("receive channel should never be closed early");
+
+        let RunningTest { test, join_handle } = running_tests.remove(&completion.id).unwrap();
+        if let Some(join_handle) = join_handle {
+            join_handle.join().unwrap_or_else(|_| {
+                panic!("thread for `{}` panicked after reporting completion", test.desc.name)
+            });
+        }
+
+        listener.test_finished(test, &completion);
+
+        if completion.outcome.is_failed() && config.fail_fast {
+            // Prevent any other in-flight threads from panicking when they
+            // write to the completion channel.
+            mem::forget(completion_rx);
+            break;
+        }
+    }
+
+    let suite_passed = listener.suite_finished();
+    suite_passed
+}
+
+/// Spawns a thread to run a single test, and returns the thread's join handle.
 ///
-/// Returns `Ok(true)` if all tests passed, or `Ok(false)` if one or more tests failed.
-pub(crate) fn execute_tests(config: &Config, tests: Vec<CollectedTest>) -> io::Result<bool> {
-    let opts = test_opts(config);
-    let tests = tests.into_iter().map(|t| t.into_libtest()).collect::<Vec<_>>();
+/// Returns `None` if the test was ignored, so no thread was spawned.
+fn spawn_test_thread(
+    id: TestId,
+    test: &CollectedTest,
+    completion_tx: mpsc::Sender<TestCompletion>,
+) -> Option<thread::JoinHandle<()>> {
+    if test.desc.ignore && !test.config.run_ignored {
+        completion_tx
+            .send(TestCompletion { id, outcome: TestOutcome::Ignored, stdout: None })
+            .unwrap();
+        return None;
+    }
+
+    let runnable_test = RunnableTest::new(test);
+    let should_panic = test.desc.should_panic;
+    let run_test = move || run_test_inner(id, should_panic, runnable_test, completion_tx);
+
+    let thread_builder = thread::Builder::new().name(test.desc.name.clone());
+    let join_handle = thread_builder.spawn(run_test).unwrap();
+    Some(join_handle)
+}
+
+/// Runs a single test, within the dedicated thread spawned by the caller.
+fn run_test_inner(
+    id: TestId,
+    should_panic: ShouldPanic,
+    runnable_test: RunnableTest,
+    completion_sender: mpsc::Sender<TestCompletion>,
+) {
+    let is_capture = !runnable_test.config.nocapture;
+    let capture_buf = is_capture.then(|| Arc::new(Mutex::new(vec![])));
+
+    if let Some(capture_buf) = &capture_buf {
+        io::set_output_capture(Some(Arc::clone(capture_buf)));
+    }
+
+    let panic_payload = panic::catch_unwind(move || runnable_test.run()).err();
+
+    if is_capture {
+        io::set_output_capture(None);
+    }
+
+    let outcome = match (should_panic, panic_payload) {
+        (ShouldPanic::No, None) | (ShouldPanic::Yes, Some(_)) => TestOutcome::Succeeded,
+        (ShouldPanic::No, Some(_)) => TestOutcome::Failed { message: None },
+        (ShouldPanic::Yes, None) => {
+            TestOutcome::Failed { message: Some("test did not panic as expected") }
+        }
+    };
+    let stdout = capture_buf.map(|mutex| mutex.lock().unwrap_or_else(|e| e.into_inner()).to_vec());
+
+    completion_sender.send(TestCompletion { id, outcome, stdout }).unwrap();
+}
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+struct TestId(usize);
+
+struct RunnableTest {
+    config: Arc<Config>,
+    testpaths: TestPaths,
+    revision: Option<String>,
+}
+
+impl RunnableTest {
+    fn new(test: &CollectedTest) -> Self {
+        let config = Arc::clone(&test.config);
+        let testpaths = test.testpaths.clone();
+        let revision = test.revision.clone();
+        Self { config, testpaths, revision }
+    }
+
+    fn run(&self) {
+        __rust_begin_short_backtrace(|| {
+            crate::runtest::run(
+                Arc::clone(&self.config),
+                &self.testpaths,
+                self.revision.as_deref(),
+            );
+        });
+    }
+}
+
+/// Fixed frame used to clean the backtrace with `RUST_BACKTRACE=1`.
+#[inline(never)]
+fn __rust_begin_short_backtrace<T, F: FnOnce() -> T>(f: F) -> T {
+    let result = f();
+
+    // prevent this frame from being tail-call optimised away
+    hint::black_box(result)
+}
 
-    test::run_tests_console(&opts, tests)
+struct RunningTest<'a> {
+    test: &'a CollectedTest,
+    join_handle: Option<thread::JoinHandle<()>>,
+}
+
+/// Test completion message sent by individual test threads when their test
+/// finishes (successfully or unsuccessfully).
+struct TestCompletion {
+    id: TestId,
+    outcome: TestOutcome,
+    stdout: Option<Vec<u8>>,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+enum TestOutcome {
+    Succeeded,
+    Failed { message: Option<&'static str> },
+    Ignored,
+}
+
+impl TestOutcome {
+    fn is_failed(&self) -> bool {
+        matches!(self, Self::Failed { .. })
+    }
+}
+
+/// Applies command-line arguments for filtering/skipping tests by name.
+///
+/// Adapted from `filter_tests` in libtest.
+///
+/// FIXME(#139660): After the libtest dependency is removed, redesign the whole
+/// filtering system to do a better job of understanding and filtering _paths_,
+/// instead of being tied to libtest's substring/exact matching behaviour.
+fn filter_tests(opts: &Config, tests: Vec<CollectedTest>) -> Vec<CollectedTest> {
+    let mut filtered = tests;
+
+    let matches_filter = |test: &CollectedTest, filter_str: &str| {
+        let test_name = &test.desc.name;
+        if opts.filter_exact { test_name == filter_str } else { test_name.contains(filter_str) }
+    };
+
+    // Remove tests that don't match the test filter
+    if !opts.filters.is_empty() {
+        filtered.retain(|test| opts.filters.iter().any(|filter| matches_filter(test, filter)));
+    }
+
+    // Skip tests that match any of the skip filters
+    if !opts.skip.is_empty() {
+        filtered.retain(|test| !opts.skip.iter().any(|sf| matches_filter(test, sf)));
+    }
+
+    filtered
+}
+
+/// Determines the number of tests to run concurrently.
+///
+/// Copied from `get_concurrency` in libtest.
+///
+/// FIXME(#139660): After the libtest dependency is removed, consider making
+/// bootstrap specify the number of threads on the command-line, instead of
+/// propagating the `RUST_TEST_THREADS` environment variable.
+fn get_concurrency() -> usize {
+    if let Ok(value) = env::var("RUST_TEST_THREADS") {
+        match value.parse::<NonZero<usize>>().ok() {
+            Some(n) => n.get(),
+            _ => panic!("RUST_TEST_THREADS is `{value}`, should be a positive integer."),
+        }
+    } else {
+        thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
+    }
 }
 
 /// Information needed to create a `test::TestDescAndFn`.
@@ -35,45 +264,6 @@ pub(crate) struct CollectedTestDesc {
     pub(crate) should_panic: ShouldPanic,
 }
 
-impl CollectedTest {
-    fn into_libtest(self) -> test::TestDescAndFn {
-        let Self { desc, config, testpaths, revision } = self;
-        let CollectedTestDesc { name, ignore, ignore_message, should_panic } = desc;
-
-        // Libtest requires the ignore message to be a &'static str, so we might
-        // have to leak memory to create it. This is fine, as we only do so once
-        // per test, so the leak won't grow indefinitely.
-        let ignore_message = ignore_message.map(|msg| match msg {
-            Cow::Borrowed(s) => s,
-            Cow::Owned(s) => &*String::leak(s),
-        });
-
-        let desc = test::TestDesc {
-            name: test::DynTestName(name),
-            ignore,
-            ignore_message,
-            source_file: "",
-            start_line: 0,
-            start_col: 0,
-            end_line: 0,
-            end_col: 0,
-            should_panic: should_panic.to_libtest(),
-            compile_fail: false,
-            no_run: false,
-            test_type: test::TestType::Unknown,
-        };
-
-        // This closure is invoked when libtest returns control to compiletest
-        // to execute the test.
-        let testfn = test::DynTestFn(Box::new(move || {
-            crate::runtest::run(config, &testpaths, revision.as_deref());
-            Ok(())
-        }));
-
-        test::TestDescAndFn { desc, testfn }
-    }
-}
-
 /// Whether console output should be colored or not.
 #[derive(Copy, Clone, Default, Debug)]
 pub enum ColorConfig {
@@ -83,16 +273,6 @@ pub enum ColorConfig {
     NeverColor,
 }
 
-impl ColorConfig {
-    fn to_libtest(self) -> test::ColorConfig {
-        match self {
-            Self::AutoColor => test::ColorConfig::AutoColor,
-            Self::AlwaysColor => test::ColorConfig::AlwaysColor,
-            Self::NeverColor => test::ColorConfig::NeverColor,
-        }
-    }
-}
-
 /// Format of the test results output.
 #[derive(Copy, Clone, Debug, Default, PartialEq, Eq)]
 pub enum OutputFormat {
@@ -105,52 +285,9 @@ pub enum OutputFormat {
     Json,
 }
 
-impl OutputFormat {
-    fn to_libtest(self) -> test::OutputFormat {
-        match self {
-            Self::Pretty => test::OutputFormat::Pretty,
-            Self::Terse => test::OutputFormat::Terse,
-            Self::Json => test::OutputFormat::Json,
-        }
-    }
-}
-
 /// Whether test is expected to panic or not.
 #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)]
 pub(crate) enum ShouldPanic {
     No,
     Yes,
 }
-
-impl ShouldPanic {
-    fn to_libtest(self) -> test::ShouldPanic {
-        match self {
-            Self::No => test::ShouldPanic::No,
-            Self::Yes => test::ShouldPanic::Yes,
-        }
-    }
-}
-
-fn test_opts(config: &Config) -> test::TestOpts {
-    test::TestOpts {
-        exclude_should_panic: false,
-        filters: config.filters.clone(),
-        filter_exact: config.filter_exact,
-        run_ignored: if config.run_ignored { test::RunIgnored::Yes } else { test::RunIgnored::No },
-        format: config.format.to_libtest(),
-        logfile: None,
-        run_tests: true,
-        bench_benchmarks: true,
-        nocapture: config.nocapture,
-        color: config.color.to_libtest(),
-        shuffle: false,
-        shuffle_seed: None,
-        test_threads: None,
-        skip: config.skip.clone(),
-        list: false,
-        options: test::Options::new(),
-        time_options: None,
-        force_run_in_process: false,
-        fail_fast: config.fail_fast,
-    }
-}
diff --git a/src/tools/compiletest/src/executor/deadline.rs b/src/tools/compiletest/src/executor/deadline.rs
new file mode 100644
index 0000000000000..83b8591a41642
--- /dev/null
+++ b/src/tools/compiletest/src/executor/deadline.rs
@@ -0,0 +1,78 @@
+use std::collections::VecDeque;
+use std::sync::mpsc::{self, RecvError, RecvTimeoutError};
+use std::time::{Duration, Instant};
+
+use crate::executor::{CollectedTest, TestId};
+
+const TEST_WARN_TIMEOUT_S: u64 = 60;
+
+struct DeadlineEntry<'a> {
+    id: TestId,
+    test: &'a CollectedTest,
+    deadline: Instant,
+}
+
+pub(crate) struct DeadlineQueue<'a> {
+    queue: VecDeque<DeadlineEntry<'a>>,
+}
+
+impl<'a> DeadlineQueue<'a> {
+    pub(crate) fn with_capacity(capacity: usize) -> Self {
+        Self { queue: VecDeque::with_capacity(capacity) }
+    }
+
+    pub(crate) fn push(&mut self, id: TestId, test: &'a CollectedTest) {
+        let deadline = Instant::now() + Duration::from_secs(TEST_WARN_TIMEOUT_S);
+        self.queue.push_back(DeadlineEntry { id, test, deadline });
+    }
+
+    /// Equivalent to `rx.read()`, except that if any test exceeds its deadline
+    /// during the wait, the given callback will also be called for that test.
+    pub(crate) fn read_channel_while_checking_deadlines<T>(
+        &mut self,
+        rx: &mpsc::Receiver<T>,
+        mut on_deadline_passed: impl FnMut(TestId, &CollectedTest),
+    ) -> Result<T, RecvError> {
+        loop {
+            let Some(next_deadline) = self.next_deadline() else {
+                // All currently-running tests have already exceeded their
+                // deadline, so do a normal receive.
+                return rx.recv();
+            };
+            let wait_duration = next_deadline.saturating_duration_since(Instant::now());
+
+            let recv_result = rx.recv_timeout(wait_duration);
+            match recv_result {
+                Ok(value) => return Ok(value),
+                Err(RecvTimeoutError::Timeout) => {
+                    // Notify the callback of tests that have exceeded their
+                    // deadline, then loop and do annother channel read.
+                    for DeadlineEntry { id, test, .. } in self.remove_tests_past_deadline() {
+                        on_deadline_passed(id, test);
+                    }
+                }
+                Err(RecvTimeoutError::Disconnected) => return Err(RecvError),
+            }
+        }
+    }
+
+    fn next_deadline(&self) -> Option<Instant> {
+        Some(self.queue.front()?.deadline)
+    }
+
+    fn remove_tests_past_deadline(&mut self) -> Vec<DeadlineEntry<'a>> {
+        let now = Instant::now();
+        let mut timed_out = vec![];
+        while let Some(deadline_entry) = pop_front_if(&mut self.queue, |entry| now < entry.deadline)
+        {
+            timed_out.push(deadline_entry);
+        }
+        timed_out
+    }
+}
+
+/// FIXME(vec_deque_pop_if): Use `VecDeque::pop_front_if` when it is stable in bootstrap.
+fn pop_front_if<T>(queue: &mut VecDeque<T>, predicate: impl FnOnce(&T) -> bool) -> Option<T> {
+    let first = queue.front()?;
+    if predicate(first) { queue.pop_front() } else { None }
+}
diff --git a/src/tools/compiletest/src/executor/json.rs b/src/tools/compiletest/src/executor/json.rs
new file mode 100644
index 0000000000000..c74ed81a36b85
--- /dev/null
+++ b/src/tools/compiletest/src/executor/json.rs
@@ -0,0 +1,111 @@
+//! Collects statistics and emits suite/test events as JSON messages, using
+//! the same JSON format as libtest's JSON formatter.
+//!
+//! These messages are then parsed by bootstrap, which replaces them with
+//! user-friendly terminal output.
+
+use std::time::Instant;
+
+use serde_json::json;
+
+use crate::executor::{CollectedTest, TestCompletion, TestOutcome};
+
+pub(crate) struct Listener {
+    suite_start: Option<Instant>,
+    passed: usize,
+    failed: usize,
+    ignored: usize,
+    filtered_out: usize,
+}
+
+impl Listener {
+    pub(crate) fn new() -> Self {
+        Self { suite_start: None, passed: 0, failed: 0, ignored: 0, filtered_out: 0 }
+    }
+
+    fn print_message(&self, message: &serde_json::Value) {
+        println!("{message}");
+    }
+
+    fn now(&self) -> Instant {
+        Instant::now()
+    }
+
+    pub(crate) fn suite_started(&mut self, test_count: usize, filtered_out: usize) {
+        self.suite_start = Some(self.now());
+        self.filtered_out = filtered_out;
+        let message = json!({ "type": "suite", "event": "started", "test_count": test_count });
+        self.print_message(&message);
+    }
+
+    pub(crate) fn test_started(&mut self, test: &CollectedTest) {
+        let name = test.desc.name.as_str();
+        let message = json!({ "type": "test", "event": "started", "name": name });
+        self.print_message(&message);
+    }
+
+    pub(crate) fn test_timed_out(&mut self, test: &CollectedTest) {
+        let name = test.desc.name.as_str();
+        let message = json!({ "type": "test", "event": "timeout", "name": name });
+        self.print_message(&message);
+    }
+
+    pub(crate) fn test_finished(&mut self, test: &CollectedTest, completion: &TestCompletion) {
+        let event;
+        let name = test.desc.name.as_str();
+        let mut maybe_message = None;
+        let maybe_stdout = completion.stdout.as_deref().map(String::from_utf8_lossy);
+
+        match completion.outcome {
+            TestOutcome::Succeeded => {
+                self.passed += 1;
+                event = "ok";
+            }
+            TestOutcome::Failed { message } => {
+                self.failed += 1;
+                maybe_message = message;
+                event = "failed";
+            }
+            TestOutcome::Ignored => {
+                self.ignored += 1;
+                maybe_message = test.desc.ignore_message.as_deref();
+                event = "ignored";
+            }
+        };
+
+        // This emits optional fields as `null`, instead of omitting them
+        // completely as libtest does, but bootstrap can parse the result
+        // either way.
+        let json = json!({
+            "type": "test",
+            "event": event,
+            "name": name,
+            "message": maybe_message,
+            "stdout": maybe_stdout,
+        });
+
+        self.print_message(&json);
+    }
+
+    pub(crate) fn suite_finished(&mut self) -> bool {
+        let exec_time = self.suite_start.map(|start| (self.now() - start).as_secs_f64());
+        let suite_passed = self.failed == 0;
+
+        let event = if suite_passed { "ok" } else { "failed" };
+        let message = json!({
+            "type": "suite",
+            "event": event,
+            "passed": self.passed,
+            "failed": self.failed,
+            "ignored": self.ignored,
+            // Compiletest doesn't run any benchmarks, but we still need to set this
+            // field to 0 so that bootstrap's JSON parser can read our message.
+            "measured": 0,
+            "filtered_out": self.filtered_out,
+            "exec_time": exec_time,
+        });
+
+        self.print_message(&message);
+        suite_passed
+    }
+}
diff --git a/src/tools/compiletest/src/executor/libtest.rs b/src/tools/compiletest/src/executor/libtest.rs
new file mode 100644
index 0000000000000..032b3f4fa9a87
--- /dev/null
+++ b/src/tools/compiletest/src/executor/libtest.rs
@@ -0,0 +1,111 @@
+//! This submodule encapsulates all of the code that actually interacts with
+//! libtest, so that it can be easily removed after the new executor becomes
+//! the default.
+
+use std::borrow::Cow;
+use std::io;
+
+use crate::common::Config;
+use crate::executor::{CollectedTest, CollectedTestDesc, ColorConfig, OutputFormat, ShouldPanic};
+
+/// Delegates to libtest to run the list of collected tests.
+///
+/// Returns `Ok(true)` if all tests passed, or `Ok(false)` if one or more tests failed.
+pub(crate) fn execute_tests(config: &Config, tests: Vec<CollectedTest>) -> io::Result<bool> {
+    let opts = test_opts(config);
+    let tests = tests.into_iter().map(|t| t.into_libtest()).collect::<Vec<_>>();
+
+    test::run_tests_console(&opts, tests)
+}
+
+impl CollectedTest {
+    fn into_libtest(self) -> test::TestDescAndFn {
+        let Self { desc, config, testpaths, revision } = self;
+        let CollectedTestDesc { name, ignore, ignore_message, should_panic } = desc;
+
+        // Libtest requires the ignore message to be a &'static str, so we might
+        // have to leak memory to create it. This is fine, as we only do so once
+        // per test, so the leak won't grow indefinitely.
+        let ignore_message = ignore_message.map(|msg| match msg {
+            Cow::Borrowed(s) => s,
+            Cow::Owned(s) => &*String::leak(s),
+        });
+
+        let desc = test::TestDesc {
+            name: test::DynTestName(name),
+            ignore,
+            ignore_message,
+            source_file: "",
+            start_line: 0,
+            start_col: 0,
+            end_line: 0,
+            end_col: 0,
+            should_panic: should_panic.to_libtest(),
+            compile_fail: false,
+            no_run: false,
+            test_type: test::TestType::Unknown,
+        };
+
+        // This closure is invoked when libtest returns control to compiletest
+        // to execute the test.
+        let testfn = test::DynTestFn(Box::new(move || {
+            crate::runtest::run(config, &testpaths, revision.as_deref());
+            Ok(())
+        }));
+
+        test::TestDescAndFn { desc, testfn }
+    }
+}
+
+impl ColorConfig {
+    fn to_libtest(self) -> test::ColorConfig {
+        match self {
+            Self::AutoColor => test::ColorConfig::AutoColor,
+            Self::AlwaysColor => test::ColorConfig::AlwaysColor,
+            Self::NeverColor => test::ColorConfig::NeverColor,
+        }
+    }
+}
+
+impl OutputFormat {
+    fn to_libtest(self) -> test::OutputFormat {
+        match self {
+            Self::Pretty => test::OutputFormat::Pretty,
+            Self::Terse => test::OutputFormat::Terse,
+            Self::Json => test::OutputFormat::Json,
+        }
+    }
+}
+
+impl ShouldPanic {
+    fn to_libtest(self) -> test::ShouldPanic {
+        match self {
+            Self::No => test::ShouldPanic::No,
+            Self::Yes => test::ShouldPanic::Yes,
+        }
+    }
+}
+
+fn test_opts(config: &Config) -> test::TestOpts {
+    test::TestOpts {
+        exclude_should_panic: false,
+        filters: config.filters.clone(),
+        filter_exact: config.filter_exact,
+        run_ignored: if config.run_ignored { test::RunIgnored::Yes } else { test::RunIgnored::No },
+        format: config.format.to_libtest(),
+        logfile: None,
+        run_tests: true,
+        bench_benchmarks: true,
+        nocapture: config.nocapture,
+        color: config.color.to_libtest(),
+        shuffle: false,
+        shuffle_seed: None,
+        test_threads: None,
+        skip: config.skip.clone(),
+        list: false,
+        options: test::Options::new(),
+        time_options: None,
+        force_run_in_process: false,
+        fail_fast: config.fail_fast,
+    }
+}
diff --git a/src/tools/compiletest/src/lib.rs b/src/tools/compiletest/src/lib.rs
index b969b22750bc0..4bbd4ab4790d8 100644
--- a/src/tools/compiletest/src/lib.rs
+++ b/src/tools/compiletest/src/lib.rs
@@ -1,7 +1,8 @@
 #![crate_name = "compiletest"]
-// The `test` crate is the only unstable feature
-// allowed here, just to share similar code.
+// Needed by the libtest-based test executor.
 #![feature(test)]
+// Needed by the "new" test executor that does not depend on libtest.
+#![feature(internal_output_capture)]
 
 extern crate test;
 
@@ -202,6 +203,7 @@ pub fn parse_config(args: Vec<String>) -> Config {
             "COMMAND",
         )
         .reqopt("", "minicore-path", "path to minicore aux library", "PATH")
+        .optflag("n", "new-executor", "enables the new test executor instead of using libtest")
         .optopt(
             "",
             "debugger",
@@ -447,6 +449,8 @@ pub fn parse_config(args: Vec<String>) -> Config {
         diff_command: matches.opt_str("compiletest-diff-tool"),
 
         minicore_path: opt_path(matches, "minicore-path"),
+
+        new_executor: matches.opt_present("new-executor"),
     }
 }
 
@@ -570,10 +574,14 @@ pub fn run_tests(config: Arc<Config>) {
 
     tests.sort_by(|a, b| Ord::cmp(&a.desc.name, &b.desc.name));
 
-    // Delegate to libtest to filter and run the big list of structures created
-    // during test discovery. When libtest decides to run a test, it will
-    // return control to compiletest by invoking a closure.
-    let res = crate::executor::execute_tests(&config, tests);
+    // Delegate to the executor to filter and run the big list of test structures
+    // created during test discovery. When the executor decides to run a test,
+    // it will return control to the rest of compiletest by calling `runtest::run`.
+    let res = if config.new_executor {
+        Ok(executor::run_tests(&config, tests))
+    } else {
+        crate::executor::libtest::execute_tests(&config, tests)
+    };
 
     // Check the outcome reported by libtest.
     match res {


Reviews:
user: login: jieyouxu
body: Thanks for working on this! The new executor implementation here is very nice to follow, and having more control over how compiletest actually executes the tests is very much appreciated!

I left two nits, but otherwise the changes look good to me.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-14T02:56:27Z
user: login: jieyouxu
body: Thanks
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-15T03:44:39Z
user: login: Kobzol
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T09:05:39Z
user: login: Zalathar
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T09:10:57Z

Review comments:
diff_hunk: @@ -1,15 +1,245 @@
-//! This module encapsulates all of the code that interacts directly with
-//! libtest, to execute the collected tests.
-//!
-//! This will hopefully make it easier to migrate away from libtest someday.
+//! This module contains a reimplementation of the subset of libtest
+//! functionality needed by compiletest.
 
 use std::borrow::Cow;
-use std::sync::Arc;
+use std::collections::HashMap;
+use std::hash::{BuildHasherDefault, DefaultHasher};
+use std::num::NonZero;
+use std::sync::{Arc, Mutex, mpsc};
+use std::{env, hint, io, mem, panic, thread};
 
 use crate::common::{Config, TestPaths};
 
+mod deadline;
+mod json;
 pub(crate) mod libtest;
 
+pub(crate) fn run_tests(config: &Config, tests: Vec<CollectedTest>) -> bool {
+    let tests_len = tests.len();
+    let filtered = filter_tests(config, tests);
+    // Iterator yielding tests that haven't been started yet.
+    let mut fresh_tests = (0..).map(TestId).zip(&filtered);
+
+    let concurrency = get_concurrency();
+    assert!(concurrency > 0);
+    let concurrent_capacity = concurrency.min(filtered.len());
+
+    let mut listener = json::Listener::new();
+    let mut running_tests = HashMap::with_capacity_and_hasher(
+        concurrent_capacity,
+        BuildHasherDefault::<DefaultHasher>::new(),
+    );
+    let mut deadline_queue = deadline::DeadlineQueue::with_capacity(concurrent_capacity);
+
+    let num_filtered_out = tests_len - filtered.len();
+    listener.suite_started(filtered.len(), num_filtered_out);
+
+    // Channel used by test threads to report the test outcome when done.
+    let (completion_tx, completion_rx) = mpsc::channel::<TestCompletion>();
+
+    // Unlike libtest, we don't have a separate code path for concurrency=1.
+    // In that case, the tests will effectively be run serially anyway.
+    loop {
+        // Spawn new test threads, up to the concurrency limit.
+        // FIXME(let_chains): Use a let-chain here when stable in bootstrap.
+        'spawn: while running_tests.len() < concurrency {
+            let Some((id, test)) = fresh_tests.next() else { break 'spawn };
+            listener.test_started(test);
+            deadline_queue.push(id, test);
+            let join_handle = spawn_test_thread(id, test, completion_tx.clone());
+            running_tests.insert(id, RunningTest { test, join_handle });
+        }
+
+        // If all running tests have finished, and there weren't any unstarted
+        // tests to spawn, then we're done.
+        if running_tests.is_empty() {
+            break;
+        }
+
+        let completion = deadline_queue
+            .read_channel_while_checking_deadlines(&completion_rx, |_id, test| {
+                listener.test_timed_out(test);
+            })
+            .expect("receive channel should never be closed early");
+
+        let RunningTest { test, join_handle } = running_tests.remove(&completion.id).unwrap();
+        if let Some(join_handle) = join_handle {
+            join_handle.join().unwrap_or_else(|_| {
+                panic!("thread for `{}` panicked after reporting completion", test.desc.name)
+            });
+        }
+
+        listener.test_finished(test, &completion);
+
+        if completion.outcome.is_failed() && config.fail_fast {
+            // Prevent any other in-flight threads from panicking when they
+            // write to the completion channel.
+            mem::forget(completion_rx);
+            break;
+        }
+    }
+
+    let suite_passed = listener.suite_finished();
+    suite_passed
+}
+
+/// Spawns a thread to run a single test, and returns the thread's join handle.
+///
+/// Returns `None` if the test was ignored, so no thread was spawned.
+fn spawn_test_thread(
+    id: TestId,
+    test: &CollectedTest,
+    completion_tx: mpsc::Sender<TestCompletion>,
+) -> Option<thread::JoinHandle<()>> {
+    if test.desc.ignore && !test.config.run_ignored {
+        completion_tx
+            .send(TestCompletion { id, outcome: TestOutcome::Ignored, stdout: None })
+            .unwrap();
+        return None;
+    }
+
+    let runnable_test = RunnableTest::new(test);
+    let should_panic = test.desc.should_panic;
+    let run_test = move || run_test_inner(id, should_panic, runnable_test, completion_tx);
+
+    let thread_builder = thread::Builder::new().name(test.desc.name.clone());
+    let join_handle = thread_builder.spawn(run_test).unwrap();
+    Some(join_handle)
+}
+
+/// Runs a single test, within the dedicated thread spawned by the caller.
+fn run_test_inner(
+    id: TestId,
+    should_panic: ShouldPanic,
+    runnable_test: RunnableTest,
+    completion_sender: mpsc::Sender<TestCompletion>,
+) {
+    let is_capture = !runnable_test.config.nocapture;
+    let capture_buf = is_capture.then(|| Arc::new(Mutex::new(vec![])));
+
+    if let Some(capture_buf) = &capture_buf {
+        io::set_output_capture(Some(Arc::clone(capture_buf)));
+    }
+
+    let panic_payload = panic::catch_unwind(move || runnable_test.run()).err();
+
+    if is_capture {
+        io::set_output_capture(None);
+    }
+
+    let outcome = match (should_panic, panic_payload) {
+        (ShouldPanic::No, None) | (ShouldPanic::Yes, Some(_)) => TestOutcome::Succeeded,
+        (ShouldPanic::No, Some(_)) => TestOutcome::Failed { message: None },
+        (ShouldPanic::Yes, None) => {
+            TestOutcome::Failed { message: Some("test did not panic as expected") }
+        }
+    };
+    let stdout = capture_buf.map(|mutex| mutex.lock().unwrap_or_else(|e| e.into_inner()).to_vec());
+
+    completion_sender.send(TestCompletion { id, outcome, stdout }).unwrap();
+}
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+struct TestId(usize);
+
+struct RunnableTest {
+    config: Arc<Config>,
+    testpaths: TestPaths,
+    revision: Option<String>,
+}
+
+impl RunnableTest {
+    fn new(test: &CollectedTest) -> Self {
+        let config = Arc::clone(&test.config);
+        let testpaths = test.testpaths.clone();
+        let revision = test.revision.clone();
+        Self { config, testpaths, revision }
+    }
+
+    fn run(&self) {
+        __rust_begin_short_backtrace(|| {
+            crate::runtest::run(
+                Arc::clone(&self.config),
+                &self.testpaths,
+                self.revision.as_deref(),
+            );
+        });
+    }
+}
+
+/// Fixed frame used to clean the backtrace with `RUST_BACKTRACE=1`.
+#[inline(never)]
+fn __rust_begin_short_backtrace<T, F: FnOnce() -> T>(f: F) -> T {
+    let result = f();
+
+    // prevent this frame from being tail-call optimised away
+    hint::black_box(result)
+}
+
+struct RunningTest<'a> {
+    test: &'a CollectedTest,
+    join_handle: Option<thread::JoinHandle<()>>,
+}
+
+/// Test completion message sent by individual test threads when their test
+/// finishes (successfully or unsuccessfully).
+struct TestCompletion {
+    id: TestId,
+    outcome: TestOutcome,
+    stdout: Option<Vec<u8>>,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+enum TestOutcome {
+    Succeeded,
+    Failed { message: Option<&'static str> },
+    Ignored,
+}
+
+impl TestOutcome {
+    fn is_failed(&self) -> bool {
+        matches!(self, Self::Failed { .. })
+    }
+}
+
+/// Applies command-line arguments for filtering/skipping tests by name.
+///
+/// Adapted from `filter_tests` in libtest.
path: src/tools/compiletest/src/executor.rs
user: login: jieyouxu
body: Suggestion: can you leave me a note here to "reconsider test filtering behavior in new executor"? One gripe I had with libtest's test filtering was that after compiletest passes to libtest the canonicalized test name (based on a relative path), the default filter logic is a naive substring match which can often produce some surprising running more tests than expected behavior. A benefit of rolling our own executor here is to allow us to redesign the filter logic (as a possibility).
created_at: 2025-04-14T02:50:11Z
author_association: MEMBER
diff_hunk: @@ -1,15 +1,245 @@
-//! This module encapsulates all of the code that interacts directly with
-//! libtest, to execute the collected tests.
-//!
-//! This will hopefully make it easier to migrate away from libtest someday.
+//! This module contains a reimplementation of the subset of libtest
+//! functionality needed by compiletest.
 
 use std::borrow::Cow;
-use std::sync::Arc;
+use std::collections::HashMap;
+use std::hash::{BuildHasherDefault, DefaultHasher};
+use std::num::NonZero;
+use std::sync::{Arc, Mutex, mpsc};
+use std::{env, hint, io, mem, panic, thread};
 
 use crate::common::{Config, TestPaths};
 
+mod deadline;
+mod json;
 pub(crate) mod libtest;
 
+pub(crate) fn run_tests(config: &Config, tests: Vec<CollectedTest>) -> bool {
+    let tests_len = tests.len();
+    let filtered = filter_tests(config, tests);
+    // Iterator yielding tests that haven't been started yet.
+    let mut fresh_tests = (0..).map(TestId).zip(&filtered);
+
+    let concurrency = get_concurrency();
+    assert!(concurrency > 0);
+    let concurrent_capacity = concurrency.min(filtered.len());
+
+    let mut listener = json::Listener::new();
+    let mut running_tests = HashMap::with_capacity_and_hasher(
+        concurrent_capacity,
+        BuildHasherDefault::<DefaultHasher>::new(),
+    );
+    let mut deadline_queue = deadline::DeadlineQueue::with_capacity(concurrent_capacity);
+
+    let num_filtered_out = tests_len - filtered.len();
+    listener.suite_started(filtered.len(), num_filtered_out);
+
+    // Channel used by test threads to report the test outcome when done.
+    let (completion_tx, completion_rx) = mpsc::channel::<TestCompletion>();
+
+    // Unlike libtest, we don't have a separate code path for concurrency=1.
+    // In that case, the tests will effectively be run serially anyway.
+    loop {
+        // Spawn new test threads, up to the concurrency limit.
+        // FIXME(let_chains): Use a let-chain here when stable in bootstrap.
+        'spawn: while running_tests.len() < concurrency {
+            let Some((id, test)) = fresh_tests.next() else { break 'spawn };
+            listener.test_started(test);
+            deadline_queue.push(id, test);
+            let join_handle = spawn_test_thread(id, test, completion_tx.clone());
+            running_tests.insert(id, RunningTest { test, join_handle });
+        }
+
+        // If all running tests have finished, and there weren't any unstarted
+        // tests to spawn, then we're done.
+        if running_tests.is_empty() {
+            break;
+        }
+
+        let completion = deadline_queue
+            .read_channel_while_checking_deadlines(&completion_rx, |_id, test| {
+                listener.test_timed_out(test);
+            })
+            .expect("receive channel should never be closed early");
+
+        let RunningTest { test, join_handle } = running_tests.remove(&completion.id).unwrap();
+        if let Some(join_handle) = join_handle {
+            join_handle.join().unwrap_or_else(|_| {
+                panic!("thread for `{}` panicked after reporting completion", test.desc.name)
+            });
+        }
+
+        listener.test_finished(test, &completion);
+
+        if completion.outcome.is_failed() && config.fail_fast {
+            // Prevent any other in-flight threads from panicking when they
+            // write to the completion channel.
+            mem::forget(completion_rx);
+            break;
+        }
+    }
+
+    let suite_passed = listener.suite_finished();
+    suite_passed
+}
+
+/// Spawns a thread to run a single test, and returns the thread's join handle.
+///
+/// Returns `None` if the test was ignored, so no thread was spawned.
+fn spawn_test_thread(
+    id: TestId,
+    test: &CollectedTest,
+    completion_tx: mpsc::Sender<TestCompletion>,
+) -> Option<thread::JoinHandle<()>> {
+    if test.desc.ignore && !test.config.run_ignored {
+        completion_tx
+            .send(TestCompletion { id, outcome: TestOutcome::Ignored, stdout: None })
+            .unwrap();
+        return None;
+    }
+
+    let runnable_test = RunnableTest::new(test);
+    let should_panic = test.desc.should_panic;
+    let run_test = move || run_test_inner(id, should_panic, runnable_test, completion_tx);
+
+    let thread_builder = thread::Builder::new().name(test.desc.name.clone());
+    let join_handle = thread_builder.spawn(run_test).unwrap();
+    Some(join_handle)
+}
+
+/// Runs a single test, within the dedicated thread spawned by the caller.
+fn run_test_inner(
+    id: TestId,
+    should_panic: ShouldPanic,
+    runnable_test: RunnableTest,
+    completion_sender: mpsc::Sender<TestCompletion>,
+) {
+    let is_capture = !runnable_test.config.nocapture;
+    let capture_buf = is_capture.then(|| Arc::new(Mutex::new(vec![])));
+
+    if let Some(capture_buf) = &capture_buf {
+        io::set_output_capture(Some(Arc::clone(capture_buf)));
+    }
+
+    let panic_payload = panic::catch_unwind(move || runnable_test.run()).err();
+
+    if is_capture {
+        io::set_output_capture(None);
+    }
+
+    let outcome = match (should_panic, panic_payload) {
+        (ShouldPanic::No, None) | (ShouldPanic::Yes, Some(_)) => TestOutcome::Succeeded,
+        (ShouldPanic::No, Some(_)) => TestOutcome::Failed { message: None },
+        (ShouldPanic::Yes, None) => {
+            TestOutcome::Failed { message: Some("test did not panic as expected") }
+        }
+    };
+    let stdout = capture_buf.map(|mutex| mutex.lock().unwrap_or_else(|e| e.into_inner()).to_vec());
+
+    completion_sender.send(TestCompletion { id, outcome, stdout }).unwrap();
+}
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+struct TestId(usize);
+
+struct RunnableTest {
+    config: Arc<Config>,
+    testpaths: TestPaths,
+    revision: Option<String>,
+}
+
+impl RunnableTest {
+    fn new(test: &CollectedTest) -> Self {
+        let config = Arc::clone(&test.config);
+        let testpaths = test.testpaths.clone();
+        let revision = test.revision.clone();
+        Self { config, testpaths, revision }
+    }
+
+    fn run(&self) {
+        __rust_begin_short_backtrace(|| {
+            crate::runtest::run(
+                Arc::clone(&self.config),
+                &self.testpaths,
+                self.revision.as_deref(),
+            );
+        });
+    }
+}
+
+/// Fixed frame used to clean the backtrace with `RUST_BACKTRACE=1`.
+#[inline(never)]
+fn __rust_begin_short_backtrace<T, F: FnOnce() -> T>(f: F) -> T {
+    let result = f();
+
+    // prevent this frame from being tail-call optimised away
+    hint::black_box(result)
+}
+
+struct RunningTest<'a> {
+    test: &'a CollectedTest,
+    join_handle: Option<thread::JoinHandle<()>>,
+}
+
+/// Test completion message sent by individual test threads when their test
+/// finishes (successfully or unsuccessfully).
+struct TestCompletion {
+    id: TestId,
+    outcome: TestOutcome,
+    stdout: Option<Vec<u8>>,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+enum TestOutcome {
+    Succeeded,
+    Failed { message: Option<&'static str> },
+    Ignored,
+}
+
+impl TestOutcome {
+    fn is_failed(&self) -> bool {
+        matches!(self, Self::Failed { .. })
+    }
+}
+
+/// Applies command-line arguments for filtering/skipping tests by name.
+///
+/// Adapted from `filter_tests` in libtest.
+fn filter_tests(opts: &Config, tests: Vec<CollectedTest>) -> Vec<CollectedTest> {
+    let mut filtered = tests;
+
+    let matches_filter = |test: &CollectedTest, filter_str: &str| {
+        let test_name = &test.desc.name;
+        if opts.filter_exact { test_name == filter_str } else { test_name.contains(filter_str) }
+    };
+
+    // Remove tests that don't match the test filter
+    if !opts.filters.is_empty() {
+        filtered.retain(|test| opts.filters.iter().any(|filter| matches_filter(test, filter)));
+    }
+
+    // Skip tests that match any of the skip filters
+    if !opts.skip.is_empty() {
+        filtered.retain(|test| !opts.skip.iter().any(|sf| matches_filter(test, sf)));
+    }
+
+    filtered
+}
+
+/// Determines the number of tests to run concurrently.
+///
+/// Copied from `get_concurrency` in libtest.
+fn get_concurrency() -> usize {
+    if let Ok(value) = env::var("RUST_TEST_THREADS") {
path: src/tools/compiletest/src/executor.rs
user: login: jieyouxu
body: Discussion: ~~Since the only intended consumer of compiletest's cli is bootstrap nowadays(`rustdoc-gui-test` uses compiletest programmatically), should this instead be a cli flag instead of an env var?~~ Although I guess even if it was an cli flag, we need also need to process `RUST_TEST_THREADS` anyway.
created_at: 2025-04-14T02:55:25Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,78 @@
+use std::collections::VecDeque;
+use std::sync::mpsc::{self, RecvError, RecvTimeoutError};
+use std::time::{Duration, Instant};
+
+use crate::executor::{CollectedTest, TestId};
+
+const TEST_WARN_TIMEOUT_S: u64 = 60;
+
+struct DeadlineEntry<'a> {
+    id: TestId,
+    test: &'a CollectedTest,
+    deadline: Instant,
+}
+
+pub(crate) struct DeadlineQueue<'a> {
+    queue: VecDeque<DeadlineEntry<'a>>,
+}
+
+impl<'a> DeadlineQueue<'a> {
+    pub(crate) fn with_capacity(capacity: usize) -> Self {
+        Self { queue: VecDeque::with_capacity(capacity) }
+    }
+
+    pub(crate) fn push(&mut self, id: TestId, test: &'a CollectedTest) {
+        let deadline = Instant::now() + Duration::from_secs(TEST_WARN_TIMEOUT_S);
+        self.queue.push_back(DeadlineEntry { id, test, deadline });
+    }
+
+    /// Equivalent to `rx.read()`, except that if any test exceeds its deadline
+    /// during the wait, the given callback will also be called for that test.
+    pub(crate) fn read_channel_while_checking_deadlines<T>(
+        &mut self,
+        rx: &mpsc::Receiver<T>,
+        mut on_deadline_passed: impl FnMut(TestId, &CollectedTest),
+    ) -> Result<T, RecvError> {
+        loop {
+            let Some(next_deadline) = self.next_deadline() else {
+                // All currently-running tests have already exceeded their
+                // deadline, so do a normal receive.
+                return rx.recv();
+            };
+            let wait_duration = next_deadline.saturating_duration_since(Instant::now());
+
+            let recv_result = rx.recv_timeout(wait_duration);
+            match recv_result {
+                Ok(value) => return Ok(value),
+                Err(RecvTimeoutError::Timeout) => {
+                    // Notify the callback of tests that have exceeded their
+                    // deadline, then loop and do annother channel read.
+                    for DeadlineEntry { id, test, .. } in self.remove_tests_past_deadline() {
+                        on_deadline_passed(id, test);
+                    }
+                }
+                Err(RecvTimeoutError::Disconnected) => return Err(RecvError),
+            }
+        }
+    }
+
+    fn next_deadline(&self) -> Option<Instant> {
+        Some(self.queue.front()?.deadline)
+    }
+
+    fn remove_tests_past_deadline(&mut self) -> Vec<DeadlineEntry<'a>> {
+        let now = Instant::now();
+        let mut timed_out = vec![];
+        while let Some(deadline_entry) = pop_front_if(&mut self.queue, |entry| now < entry.deadline)
path: src/tools/compiletest/src/executor/deadline.rs
user: login: Kobzol
body: Shouldn't this be `now >= entry.deadline`?
created_at: 2025-04-18T08:58:20Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,78 @@
+use std::collections::VecDeque;
+use std::sync::mpsc::{self, RecvError, RecvTimeoutError};
+use std::time::{Duration, Instant};
+
+use crate::executor::{CollectedTest, TestId};
+
+const TEST_WARN_TIMEOUT_S: u64 = 60;
+
+struct DeadlineEntry<'a> {
+    id: TestId,
+    test: &'a CollectedTest,
+    deadline: Instant,
+}
+
+pub(crate) struct DeadlineQueue<'a> {
+    queue: VecDeque<DeadlineEntry<'a>>,
+}
+
+impl<'a> DeadlineQueue<'a> {
+    pub(crate) fn with_capacity(capacity: usize) -> Self {
+        Self { queue: VecDeque::with_capacity(capacity) }
+    }
+
+    pub(crate) fn push(&mut self, id: TestId, test: &'a CollectedTest) {
+        let deadline = Instant::now() + Duration::from_secs(TEST_WARN_TIMEOUT_S);
+        self.queue.push_back(DeadlineEntry { id, test, deadline });
+    }
+
+    /// Equivalent to `rx.read()`, except that if any test exceeds its deadline
+    /// during the wait, the given callback will also be called for that test.
+    pub(crate) fn read_channel_while_checking_deadlines<T>(
+        &mut self,
+        rx: &mpsc::Receiver<T>,
+        mut on_deadline_passed: impl FnMut(TestId, &CollectedTest),
+    ) -> Result<T, RecvError> {
+        loop {
+            let Some(next_deadline) = self.next_deadline() else {
+                // All currently-running tests have already exceeded their
+                // deadline, so do a normal receive.
+                return rx.recv();
+            };
+            let wait_duration = next_deadline.saturating_duration_since(Instant::now());
+
+            let recv_result = rx.recv_timeout(wait_duration);
+            match recv_result {
+                Ok(value) => return Ok(value),
+                Err(RecvTimeoutError::Timeout) => {
+                    // Notify the callback of tests that have exceeded their
+                    // deadline, then loop and do annother channel read.
+                    for DeadlineEntry { id, test, .. } in self.remove_tests_past_deadline() {
+                        on_deadline_passed(id, test);
+                    }
+                }
+                Err(RecvTimeoutError::Disconnected) => return Err(RecvError),
+            }
+        }
+    }
+
+    fn next_deadline(&self) -> Option<Instant> {
+        Some(self.queue.front()?.deadline)
+    }
+
+    fn remove_tests_past_deadline(&mut self) -> Vec<DeadlineEntry<'a>> {
+        let now = Instant::now();
+        let mut timed_out = vec![];
+        while let Some(deadline_entry) = pop_front_if(&mut self.queue, |entry| now < entry.deadline)
path: src/tools/compiletest/src/executor/deadline.rs
user: login: Zalathar
body: Good catch, you're right. I'll make a fix.
created_at: 2025-04-18T09:10:57Z
author_association: MEMBER

PR comments:
user: login: rustbot
created_at: 2025-04-11T06:55:58Z
body: Some changes occurred in src/tools/compiletest

cc @jieyouxu
author_association: COLLABORATOR
user: login: rust-log-analyzer
created_at: 2025-04-11T07:46:19Z
body: 
The job **`x86_64-gnu-tools`** failed! Check out the build log: [(web)](https://github.com/rust-lang/rust/actions/runs/14397423803/job/40375615426) [(plain)](https://github.com/rust-lang/rust/commit/b68d7a347b64388ace13c86d219b2e4a4ba630c7/checks/40375615426/logs)

<details><summary><i>Click to see the possible cause of the failure (guessed by this bot)</i></summary>

```plain
[RUSTC-TIMING] rustfix test:false 0.384
error[E0658]: use of unstable library feature `internal_output_capture`: this function is meant for use in the test crate and may disappear in the future
   --> src/tools/compiletest/src/executor.rs:123:9
    |
123 |         io::set_output_capture(Some(Arc::clone(capture_buf)));
    |         ^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add `#![feature(internal_output_capture)]` to the crate attributes to enable
    = note: this compiler was built on 2025-04-01; consider upgrading it if it is out of date

```

</details>

author_association: COLLABORATOR
user: login: jieyouxu
created_at: 2025-04-11T10:11:59Z
body: pro gamer move

I'll look at this tmrw/Sunday.
author_association: MEMBER
user: login: petrochenkov
created_at: 2025-04-11T11:46:03Z
body: >and the libtest dependency can be removed

Is there some history behind this change?
Why is it so important to remove the libtest dependency?
author_association: CONTRIBUTOR
user: login: jieyouxu
created_at: 2025-04-11T12:32:21Z
body: > > and the libtest dependency can be removed
> 
> Is there some history behind this change? Why is it so important to remove the libtest dependency?

This is in the context of the stage 0 std redesign (#119899). Currently, `compiletest` depends on in-tree libtest (unless `compiletest-use-stage0-libtest = true` is specified). For compiler workflows, this means that compiletest will be rebuilt when iterating on compiler changes which feels quite bad with the stage 0 std redesign otherwise (see [#t-infra/bootstrap > Review thread: stage 0 redesign PR @ 💬](https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Review.20thread.3A.20stage.200.20redesign.20PR/near/509663556) discussions).

If we no longer depend on libtest, then we won't need this `compiletest-use-stage0-libtest = true` workaround/hack.
author_association: MEMBER
user: login: Zalathar
created_at: 2025-04-11T12:34:56Z
body: Depending on libtest internals is a big maintenance headache for compiletest. We already have a bunch of annoying hacks in compiletest to work around limitations that libtest imposes. And changing libtest to better meet compiletest's needs is very risky, because libtest is the backbone of testing across most of the wider Rust ecosystem.

What has made this change even more desirable is the upcoming stage 0 redesign (#119899). Currently compiletest relies on an in-tree libtest built with the stage 0 compiler. But after the redesign, stage 0 builds of the standard library will no longer be supported, so compiletest will have to rely on either a stage 1 build of libtest (resulting in unacceptable workflow regressions for compiler contributors), or the pre-built stage 0 libtest (which can go out of sync with the in-tree version, causing various bootstrapping problems).

This replacement still needs unstable library features (for now), but `internal_output_capture` is a much smaller unstable surface area. And in the future it should be possible to work towards removing the need for output capture in compiletest, making it 100% compatible with stable Rust. That removes a lot of the maintenance concerns associated with wanting to build it with the stage 0 compiler.
author_association: MEMBER
user: login: Zalathar
created_at: 2025-04-12T03:54:38Z
body: Tweaked some comments [(diff)](https://github.com/rust-lang/rust/compare/3f2fa860cc020c926c5abf03e5efdd5256da597f..7ebb8a6f0f463014eb267f46300b5f52f90ec842); no functional change.
author_association: MEMBER
user: login: jieyouxu
created_at: 2025-04-14T02:57:43Z
body: @rustbot author
author_association: MEMBER
user: login: rustbot
created_at: 2025-04-14T02:57:47Z
body: Reminder, once the PR becomes ready for a review, use `@rustbot ready`.
author_association: COLLABORATOR
user: login: bors
created_at: 2025-04-14T11:24:49Z
body: :umbrella: The latest upstream changes (presumably #139781) made this pull request unmergeable. Please [resolve the merge conflicts](https://rustc-dev-guide.rust-lang.org/git.html#rebasing-and-conflicts).
author_association: COLLABORATOR
user: login: Zalathar
created_at: 2025-04-15T03:01:52Z
body: Rebased to fix trivial conflicts, then added some FIXMEs to `filter_tests` and `get_concurrency` to address feedback [(diff)](https://github.com/rust-lang/rust/compare/0730ef21a39d18c20d0f2255f09bd6008fe39b4a..e3d68139200da0e5010321b286e3e15e4a9530e0).

@rustbot ready
author_association: MEMBER
user: login: jieyouxu
created_at: 2025-04-15T03:44:58Z
body: @bors r+ rollup
author_association: MEMBER
user: login: bors
created_at: 2025-04-15T03:45:01Z
body: :pushpin: Commit e3d68139200da0e5010321b286e3e15e4a9530e0 has been approved by `jieyouxu`

It is now in the [queue](https://bors.rust-lang.org/homu/queue/rust) for this repository.

<!-- @bors r=jieyouxu e3d68139200da0e5010321b286e3e15e4a9530e0 -->
<!-- homu: {"type":"Approved","sha":"e3d68139200da0e5010321b286e3e15e4a9530e0","approver":"jieyouxu","queue":"https://bors.rust-lang.org/homu/queue/rust"} -->
author_association: COLLABORATOR