Package: gradle-profiler-pttest
Paired T-tests for Gradle BenchmarksContextComplete blog post to come. Stay tunnedgradle-profiler-pttestcan analyse the outcomes of two benchmarks generated byGradle Profilerwith thePaired T-test statistical technique.The goal is provide a super easy way to compare two benchmarks for Gradle builds - for the same task - without beingmislead by simple means, since we are leveraging on a more robust statistical evidence on top of the outcomes.This tool is built on top ofpingouin, an opinionated Statistics library which leverages NumPy, Pandas and SciPy. Among other things,gradle-profiler-pttestfeatures :An opinionatedhyphotesis test(left-tailed) where we want to check ifmodified build conditions(h1) have a meanstatistically smallerthan the one we get frombaseline build conditions(h0), which means better execution times given the modifications appliedAuto-correction for benchmark samples with different sizesNote that the ideal size for samples should be a small one - ideally between 10 and 30 measured builds - in order to make sense of T-student analysis.InstallingInstall from your CLI withpippipinstallgradle-profiler-pttestRequiresPython 3.8.5or newer.When running this tool on MacOS Catalina or newer,please check instructionsin order to have SciPy properly installed in your local machine.UsingRun the benchmarks with Gradle profiler for the status quo (baseline) and for the modifications applied the your Gradle project (modified)Supply the generated CSV files togradle-profiler-pttestgradle-profiler-pttest\-b<path/to/baseline/benchmark.csv>\-m<path/to/modified/benchmark.csv>Profit resultsLimitationsRight nowgradle-profiler-pttestsupportsonly one Gradle task per supplied benchmark sample, taking the first task executed as reference for the analysis given a multi-taskbenchmarked scenario.ContributingEnsure you have Python 3.8.5 or newer installedEnsure you haveflake8installed and supported in your text editor / IDEEnsure you havePoetryinstalledCheck thecontribution guidelinesMake sure you have a green buildmake flake8
make testSubmit your PR 🔥CreditsRaphael Vallatforpingouin, it made the task super easyWill McGuganforrich, I wish I had something awesome like this for my JVM projectsAuthorCoded by Ubiratan Soares (follow me onTwitter)LicenseThe MIT License (MIT)

Copyright (c) 2020 Dotanuki Labs

Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.