Package: crossbench
CrossbenchCrossbench is a cross-browser/cross-benchmark runner to extract performance
numbers.Mailing list:crossbench@chromium.orgIssues/Bugs:Tests > CrossBenchSupported Browsers: Chrome/Chromium, Firefox, Safari and Edge.Supported OS: macOS, Android, linux and windows.Basic usage:Chromium Devs (with a full chromium checkout)Use the./cb.pyscript directly to run benchmarks (requires chrome'svpython3)Standalone installationUsepip install crossbench,or use the "poetry" package manager, see thedevelopment section.Run the latestspeedometer benchmark20 times with the system default browser (chrome-stable):# Run chrome-stable by default:./cb.pyspeedometer--repeat=20# Compare chrome browser versions and a local chrome build on jetstream:./cb.pyjetstream--browser=chrome-stable--browser=chrome-m90--browser=$PATHProfile individual line items (with pprof on linux):./cb.pyspeedometer--probe='profiling'--separateUse a custom chrome build and only run a subset of the stories:./cb.pyspeedometer--browser=$PATH--probe='profiling'--story='jQuery.*'Profile a website for 17 seconds on Chrome M100 (auto-downloading on macOS and linux):./cb.pyloading--browser=chrome-m100--probe='profiling'--url=www.cnn.com,17sMain ComponentsBrowsersCrossbench supports running benchmarks on one or multiple browser configurations.
The main implementation uses selenium for maximum system independence.You can specify a browser with--browser=<name>. You can repeat the--browserargument to run multiple browser. If you need custom flags for
multiple browsers use--browser-config(or pass simple flags after--to
the browser)../cb.pyspeedometer--browser=$BROWSER----enable-field-trial-configBrowser Config FileFor more complex scenarios you can use abrowser.config.hjsonfile.
It allows you to specify multiple browser and multiple flag configurations in
a single file and produce performance numbers with a single invocation../cb.pyspeedometer--browser-config=config.hjsonTheexample filelists and explains all
configuration details.ProbesProbes define a way to extract arbitrary (performance) numbers from a
host or running browser. This can reach from running simple JS-snippets to
extract page-specific numbers to system-wide profiling.Multiple probes can be added with repeated--probe='XXX'options.
You can use thedescribe probessubcommand to list all probes:# List all probes:./cb.pydescribeprobes# List help for an individual probe:./cb.pydescribeprobev8.logInline Probe ConfigSome probes can be configured, either with inline json when using--probeor
in a separate--probe-confighjson file. Use thedescribecommand to list
all options.# Get probe config details:./cb.pydescribeprobev8.log# Use inline hjson to configure a probe:./cb.pyspeedometer--probe='v8.log:{prof:true}'Probe Config FileFor complex probe setups you can use--probe-config=<file>.
Theexample filelists and explains all
configuration details. For the specific probe configuration properties consult
thedescribecommand.BenchmarksUse thedescribecommand to list all benchmark details:# List all benchmark info:./cb.pydescribebenchmarks# List an individual benchmark info:./cb.pydescribebenchmarkspeedometer_3.0# List a benchmark's command line options:./cb.pyspeedometer_3.0--helpStoriesStories define sequences of browser interactions. This can be simply
loading a URL and waiting for a given period of time, or in more complex
scenarios, actively interact with a page and navigate multiple times.Use--helpor describe to list all stories for a benchmark:./cb.pyspeedometer--helpUse--storiesto list individual story names, or use regular expression
as filter../cb.pyspeedometer--browser=$BROWSER--stories='.*Angular.*'DevelopmentSetupThis project usespoetrydeps and package scripts
to setup the correct environment for testing and debugging.# a) On debian:sudoapt-getinstallpython3.10python3-poetry# b) With python 3.8 to 3.10 installed already:pip3installpoetryCheck that you have poetry on your path and make sure you have the right$PATHsettings.poetry--help||echo"Please update your \$PATH to include poetry bin location";# Depending on your setup, add one of the following to your $PATH:echo"`python3 -m site --user-base`/bin";python3-c"import sysconfig; print(sysconfig.get_path('scripts'))";Install the necessary dependencies from the lock file using poetry:# Select the python version you want to use (3.8 to 3.10):poetryenvuse3.10
poetryinstall# For python 3.11 you have to skip pytype support:poetryenvuse3.11
poetryinstall--without=dev-pytypeCrossbenchFor local development / non-chromium installation you should
usepoetry run cb ...instead of./cb.py ....Side-note, beware that poetry eats up an empty--:# With cb.py:./cb.pyspeedometer...----custom-chrome-flag...# With poetry:poetryruncbspeedometer...------custom-chrome-flag...Testspoetry run pytestRun detailed test coverage:poetryrunpytest--cov=crossbench--cov-report=htmlRunpytypetype checker:poetryrunpytype-jauto.