Change:
functions are run multiple times. Some benchmark functions might depend on some runtime initialisation which may fail. For example: Will print “Benchmarks on even hours are forbidden!” a lot of times. If one opts to not report set-up failure, the benchmark function just keeps processor hot for a while without actually benchmarking anything (and reporting 0ns/iter anyway). Would it be sensible to (one of possible solutions): Not run the same function multiple times, since iteration is done by ; Not run the same again if is not called at least once; Provide a method on to cancel all subsequent runs of the function? Or maybe my approach is incorrect?
If one tries to keep benchmarks in separate files and opts for to cancel further runs of the function, the panic will abort the whole benchmark process (i.e. including the benchmarks in other executables).
For , we want to run on the CI infrastructure to ensure all the tests build. However, we don't want to actually benchmark on the CI infrastructure, because the CI infrastructure isn't reliable enough for benchmarking. Thus, doing more than one iteration on the CI infrastructure just wastes time. And, that time, in particular the wall time, negatively affects my other repos, because generally CI services like Travis CI and Appveyor only allow one project to build at a time, for their free open source tiers. Further, even out side of CI infrastructure, we want contributors to run the benchmarks before sending PRs. But, it's not reasonable to expect them to wait through many minutes of perf tests before asking for a PR. Here are two ideas: a that overrides the default targeted benchmark execution time so that we can write to run tests once. a so we can write . I prefer because that also might help with
Never mind. I just learned that will do exactly what I want: run the benchmarks once each.
This was bugging me, so I implemented the first proposed option: As this is my first real rust code, I imagine its pretty bad quality. A fist attempt is here: I'll clean it up a bit, give it some more testing, then send in a pull request unless anyone offers any opinions before I get that far.
Does anyone have an example on how to use this feature ? I'd like to run a benchmark only once.
:frowning_face:

Discussion:
/// This is fed into functions marked with `#[bench]` to allow for /// set-up & tear-down before running a piece of code repeatedly via a /// call to `iter`. <del> #[derive(Copy, Clone)] </del> <ins> #[derive(Clone)] </ins> pub struct Bencher { <del> iterations: u64, dur: Duration, </del> <ins> mode: BenchMode, summary: Option<stats::Summary>, </ins> pub bytes: u64, } <ins> #[derive(Clone, PartialEq, Eq)] pub enum BenchMode { Auto, Single, } </ins> #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)] pub enum ShouldPanic { No,

pub fn iter<T, F>(&mut self, mut inner: F) where F: FnMut() -> T { <del> let start = Instant::now(); let k = self.iterations; for _ in 0..k { black_box(inner()); </del> <ins> if self.mode == BenchMode::Single { ns_iter_inner(&mut inner, 1); return; </ins> } <del> self.dur = start.elapsed(); } pub fn ns_elapsed(&mut self) -> u64 { self.dur.as_secs() * 1_000_000_000 + (self.dur.subsec_nanos() as u64) </del> <ins> self.summary = Some(iter(&mut inner)); </ins> } <del> pub fn ns_per_iter(&mut self) -> u64 { if self.iterations == 0 { 0 } else { self.ns_elapsed() / cmp::max(self.iterations, 1) } } pub fn bench_n<F>(&mut self, n: u64, f: F) where F: FnOnce(&mut Bencher) </del> <ins> pub fn bench<F>(&mut self, mut f: F) -> Option<stats::Summary> where F: FnMut(&mut Bencher) </ins> { <del> self.iterations = n; </del> f(self); <ins> return self.summary; </ins> } <ins> } </ins> <del> // This is a more statistics-driven benchmark algorithm pub fn auto_bench<F>(&mut self, mut f: F) -> stats::Summary where F: FnMut(&mut Bencher) { // Initial bench run to get ballpark figure. let mut n = 1; self.bench_n(n, |x| f(x)); // Try to estimate iter count for 1ms falling back to 1m // iterations if first run took < 1ns. if self.ns_per_iter() == 0 { n = 1_000_000; } else { n = 1_000_000 / cmp::max(self.ns_per_iter(), 1); } // if the first run took more than 1ms we don't want to just // be left doing 0 iterations on every loop. The unfortunate // side effect of not being able to do as many runs is // automatically handled by the statistical analysis below // (i.e. larger error bars). if n == 0 { n = 1; </del> <ins> fn ns_from_dur(dur: Duration) -> u64 { dur.as_secs() * 1_000_000_000 + (dur.subsec_nanos() as u64) } fn ns_iter_inner<T, F>(inner: &mut F, k: u64) -> u64 where F: FnMut() -> T { let start = Instant::now(); for _ in 0..k { black_box(inner()); } return ns_from_dur(start.elapsed()); } pub fn iter<T, F>(inner: &mut F) -> stats::Summary where F: FnMut() -> T { // Initial bench run to get ballpark figure. let ns_single = ns_iter_inner(inner, 1); // Try to estimate iter count for 1ms falling back to 1m // iterations if first run took < 1ns. let ns_target_total = 1_000_000; // 1ms let mut n = ns_target_total / cmp::max(1, ns_single); // if the first run took more than 1ms we don't want to just // be left doing 0 iterations on every loop. The unfortunate // side effect of not being able to do as many runs is // automatically handled by the statistical analysis below // (i.e. larger error bars). n = cmp::max(1, n); let mut total_run = Duration::new(0, 0); let samples: &mut [f64] = &mut [0.0_f64; 50]; loop { let loop_start = Instant::now(); for p in &mut *samples { *p = ns_iter_inner(inner, n) as f64 / n as f64; </ins> } <del> let mut total_run = Duration::new(0, 0); let samples: &mut [f64] = &mut [0.0_f64; 50]; loop { let loop_start = Instant::now(); </del> <ins> stats::winsorize(samples, 5.0); let summ = stats::Summary::new(samples); </ins> <del> for p in &mut *samples { self.bench_n(n, |x| f(x)); *p = self.ns_per_iter() as f64; } </del> <ins> for p in &mut *samples { let ns = ns_iter_inner(inner, 5 * n); *p = ns as f64 / (5 * n) as f64; } </ins> <del> stats::winsorize(samples, 5.0); let summ = stats::Summary::new(samples); </del> <ins> stats::winsorize(samples, 5.0); let summ5 = stats::Summary::new(samples); </ins> <del> for p in &mut *samples { self.bench_n(5 * n, |x| f(x)); *p = self.ns_per_iter() as f64; } </del> <ins> let loop_run = loop_start.elapsed(); </ins> <del> stats::winsorize(samples, 5.0); let summ5 = stats::Summary::new(samples); let loop_run = loop_start.elapsed(); </del> <ins> // If we've run for 100ms and seem to have converged to a // stable median. if loop_run > Duration::from_millis(100) && summ.median_abs_dev_pct < 1.0 && summ.median - summ5.median < summ5.median_abs_dev { return summ5; } </ins> <del> // If we've run for 100ms and seem to have converged to a // stable median. if loop_run > Duration::from_millis(100) && summ.median_abs_dev_pct < 1.0 && summ.median - summ5.median < summ5.median_abs_dev { return summ5; } </del> <ins> total_run = total_run + loop_run; // Longest we ever run for is 3s. if total_run > Duration::from_secs(3) { return summ5; } </ins> <del> total_run = total_run + loop_run; // Longest we ever run for is 3s. if total_run > Duration::from_secs(3) { </del> <ins> // If we overflow here just return the results so far. We check a // multiplier of 10 because we're about to multiply by 2 and the // next iteration of the loop will also multiply by 5 (to calculate // the summ5 result) n = match n.checked_mul(10) { Some(_) => n * 2, None => { </ins> return summ5; } <del> // If we overflow here just return the results so far. We check a // multiplier of 10 because we're about to multiply by 2 and the // next iteration of the loop will also multiply by 5 (to calculate // the summ5 result) n = match n.checked_mul(10) { Some(_) => n * 2, None => return summ5, }; } </del> <ins> }; </ins> } } pub mod bench { use std::cmp; <del> use std::time::Duration; use super::{Bencher, BenchSamples}; </del> <ins> use stats; use super::{Bencher, BenchSamples, BenchMode}; </ins> pub fn benchmark<F>(f: F) -> BenchSamples where F: FnMut(&mut Bencher) { let mut bs = Bencher { <del> iterations: 0, dur: Duration::new(0, 0), </del> <ins> mode: BenchMode::Auto, summary: None, </ins> bytes: 0, }; <del> let ns_iter_summ = bs.auto_bench(f); </del> <ins> return match bs.bench(f) { Some(ns_iter_summ) => { let ns_iter = cmp::max(ns_iter_summ.median as u64, 1); let mb_s = bs.bytes * 1000 / ns_iter; </ins> <del> let ns_iter = cmp::max(ns_iter_summ.median as u64, 1); let mb_s = bs.bytes * 1000 / ns_iter; BenchSamples { ns_iter_summ: ns_iter_summ, mb_s: mb_s as usize, } </del> <ins> BenchSamples { ns_iter_summ: ns_iter_summ, mb_s: mb_s as usize, } } None => { // iter not called, so no data. // FIXME: error in this case? let samples: &mut [f64] = &mut [0.0_f64; 1]; BenchSamples { ns_iter_summ: stats::Summary::new(samples), mb_s: 0, } } }; </ins> } pub fn run_once<F>(f: F) <del> where F: FnOnce(&mut Bencher) </del> <ins> where F: FnMut(&mut Bencher) </ins> { let mut bs = Bencher { <del> iterations: 0, dur: Duration::new(0, 0), </del> <ins> mode: BenchMode::Single, summary: None, </ins> bytes: 0, }; <del> bs.bench_n(1, f); </del> <ins> bs.bench(f); </ins> } }

TestDescAndFn, TestOpts, run_test, MetricMap, StaticTestName, DynTestName, DynTestFn, ShouldPanic}; use std::sync::mpsc::channel; <ins> use bench; use Bencher; </ins> #[test] pub fn do_not_run_ignored_tests() {

m1.insert_metric("in-both-want-upwards-and-improved", 1000.0, -10.0); m2.insert_metric("in-both-want-upwards-and-improved", 2000.0, -10.0); } <ins> #[test] pub fn test_bench_once_no_iter() { fn f(_: &mut Bencher) {} bench::run_once(f); } #[test] pub fn test_bench_once_iter() { fn f(b: &mut Bencher) { b.iter(|| { }) } bench::run_once(f); } #[test] pub fn test_bench_no_iter() { fn f(_: &mut Bencher) {} bench::benchmark(f); } #[test] pub fn test_bench_iter() { fn f(b: &mut Bencher) { b.iter(|| { }) } bench::benchmark(f); } </ins> }

} /// Extracted collection of all the summary statistics of a sample set. <del> #[derive(Clone, PartialEq)] </del> <ins> #[derive(Clone, PartialEq, Copy)] </ins> #[allow(missing_docs)] pub struct Summary { pub sum: f64,

v.sum(); }) } <ins> #[bench] pub fn no_iter(_: &mut Bencher) {} </ins> }