Repository: vercel/next.js
Pull request: [turbopack] Create a simple benchmark to measure the overhead of turbotasks

This benchmark works by

* creating a simple busy wait function to simulate cpu bound work
  - This is mostly to prove that we are measuring overheads and not some feature of small tasks.  The overheads for a 1us task are ~identical to the overheads for a 10us task
* running that function directly, as an uncached turbotask as a cached turbo task hitting one key and as a cached turbotask hitting many different cached keys

This allows us to independently estimate the error of the busy loop, the overhead of launching and executing a task and the overhead of getting a cache hit.

I should note, these are _idealized_ conditions and we should expect overheads to be higher during an actual build.  This is because there is no actual contention for CPU resources (no task queueing), the internal hashmapsare small so there is less time waiting to fill caches from RAM, and there are no parallel allocations/deallocations occurring (contenting in mimalloc data structures).

On my machine i see:

Cache-Hit (all same key): 130ns-140ns 
Cache-Hit (all different keys): 240ns-500ns
Turbo-Task-Execution Overhead: 4-8us

The fact that the cost of a cache hit is so different depending on if we are hitting the same key or different keys is interesting.  I would assume that this this is the cost of missing cpu caches (~100ns to fill a cache line from ram), this implies we are paying for the cost of an extra 2-3 dependent ram reads (which in retrospect sounds like a DashMap to me!)

This implies a 'break even' equation for caching

Given a task that takes `Tns` of time  caching becomes worth it after this many executions `ceil((TOverhead-TCache + TTask) / (TTask - TCache))`  plugging in our values we get the following results.

In the optimistic case for caching
 * T <= 130ns:  never worth it
 * T = 1000ns (1us):  7 executions (1 actual, 7 cache hits)
 * T  >= 10μs:  2 executions (aka need at least one cache hit)

In the pessimistic case for caching
 * T <= 500ns:  never worth it
 * T = 1000ns (1us):  11 executions (1 actual, 10 cache hits)
 * T >= 10μs:  2 executions 
 
So really we should be suspicious of all tasks that never get a hit and all tasks faster than 10us need many hits to work.   




Closes PACK-5376

Changed files:
filename: turbopack/crates/turbo-tasks-backend/benches/mod.rs
patch: @@ -3,13 +3,14 @@
 
 use criterion::{Criterion, criterion_group, criterion_main};
 
+pub(crate) mod overhead;
 pub(crate) mod scope_stress;
 pub(crate) mod stress;
 
 criterion_group!(
     name = turbo_tasks_backend_stress;
     config = Criterion::default();
-    targets = stress::fibonacci, scope_stress::scope_stress
+    targets = stress::fibonacci, scope_stress::scope_stress, overhead::overhead
 );
 criterion_main!(turbo_tasks_backend_stress);
 

filename: turbopack/crates/turbo-tasks-backend/benches/overhead.rs
patch: @@ -0,0 +1,141 @@
+use std::time::{Duration, Instant};
+
+use criterion::{BenchmarkId, Criterion, black_box};
+use turbo_tasks::TurboTasks;
+use turbo_tasks_backend::{BackendOptions, TurboTasksBackend, noop_backing_storage};
+
+use super::register;
+
+// Tunable task: busy-wait for a given duration
+#[inline(never)]
+fn busy_task(duration: Duration) {
+    let start = Instant::now();
+    while start.elapsed() < duration {
+        std::hint::spin_loop();
+    }
+}
+
+// Simulate running the task inside turbo-tasks (replace with actual turbo-tasks API)
+#[turbo_tasks::function]
+fn busy_turbo(key: u64, duration: Duration) {
+    busy_task(duration);
+    black_box(key); // consume the key, we need it to be part of the cache key.
+}
+
+pub fn overhead(c: &mut Criterion) {
+    register();
+
+    let mut group = c.benchmark_group("task_overhead");
+    group.sample_size(100);
+    let rt: tokio::runtime::Runtime = tokio::runtime::Builder::new_multi_thread()
+        .enable_all()
+        .build()
+        .unwrap();
+
+    // Test durations between 10us and 1ms.  This enables two things
+    // 1. ensure that our busy task is working correctly, we should see uncached times scale with
+    //    this metric
+    // 2. see if there are effects related to how long await points take
+    for micros in [1, 10, 100, 1000] {
+        let duration = Duration::from_micros(micros);
+
+        group.bench_with_input(BenchmarkId::new("direct", micros), &duration, |b, &d| {
+            b.iter(|| busy_task(black_box(d)))
+        });
+
+        group.bench_with_input(
+            BenchmarkId::new("turbo-uncached", micros),
+            &duration,
+            |b, &d| {
+                run_turbo::<Uncached>(&rt, b, d);
+            },
+        );
+
+        group.bench_with_input(
+            BenchmarkId::new("turbo-cached-same-keys", micros),
+            &duration,
+            |b, &d| {
+                run_turbo::<CachedSame>(&rt, b, d);
+            },
+        );
+
+        group.bench_with_input(
+            BenchmarkId::new("turbo-cached-different-keys", micros),
+            &duration,
+            |b, &d| {
+                run_turbo::<CachedDifferent>(&rt, b, d);
+            },
+        );
+    }
+    group.finish();
+}
+
+trait TurboMode {
+    fn key(index: u64) -> u64;
+    fn is_cached() -> bool;
+}
+struct Uncached;
+impl TurboMode for Uncached {
+    fn key(index: u64) -> u64 {
+        index
+    }
+
+    fn is_cached() -> bool {
+        false
+    }
+}
+struct CachedSame;
+impl TurboMode for CachedSame {
+    fn key(_index: u64) -> u64 {
+        0
+    }
+
+    fn is_cached() -> bool {
+        true
+    }
+}
+struct CachedDifferent;
+impl TurboMode for CachedDifferent {
+    fn key(index: u64) -> u64 {
+        index
+    }
+
+    fn is_cached() -> bool {
+        true
+    }
+}
+fn run_turbo<Mode: TurboMode>(
+    rt: &tokio::runtime::Runtime,
+    b: &mut criterion::Bencher<'_>,
+    d: Duration,
+) {
+    b.to_async(rt).iter_custom(|iters| {
+        // It is important to create the tt instance here to ensure the cache is not shared across
+        // iterations.
+        let tt = TurboTasks::new(TurboTasksBackend::new(
+            BackendOptions {
+                storage_mode: None,
+                ..Default::default()
+            },
+            noop_backing_storage(),
+        ));
+
+        async move {
+            tt.run_once(async move {
+                // If cached run once outside the loop to ensure the tasks are cached.
+                if Mode::is_cached() {
+                    for i in 0..iters {
+                        black_box(busy_turbo(i, black_box(d)).await?);
+                    }
+                }
+                let start = Instant::now();
+                for i in 0..iters {
+                    black_box(busy_turbo(Mode::key(i), black_box(d)).await?);
+                }
+                Ok(start.elapsed())
+            })
+            .await
+            .unwrap()
+        }
+    });
+}