Package: perun-toolsuite
Perun: Lightweight Performance Version SystemPerun is an open source light-weight Performance Version System, which works as a wrapper over
projects in Version Control Systems (such asgit) and (in parallel) tracks performance profiles
(i.e., collection of performance metrics) corresponding to different versions of underlying project.
Moreover, it offers a wide tool suite that can be used for automation of the performance regression
test runs, postprocessing of resulting profiles (e.g., by creating performance models) or
effective interpretation of the results.In particular, Perun has the following advantages over, e.g., using databases or sole Version Control
Systems for the same purpose:Preserves Context---each performance profile is assigned to a concrete
minor version adding the functional context (i.e., code changes) of profiles.
This way, one knows precisely for which code version, the profile was collected.Provides Automation---Perun allows one to easily automate the process
of profile collection, eventually reducing the whole process to a
single command. The specification of jobs is read from YAML files.Is Highly Generic---supported format of the performance profiles is
based onJSON. Perun tool suite
contains a basic set of visualizations, postprocessing and
collection modules, but it is easily extensible.Is Easy to use---the workflow, interface and storage of Perun is
heavily inspired by the git systems aiming at natural use.Perun is intended to be used in two ways: (1) for a single developer (or a small team) as a complete
solution for automating, storing and interpreting performance of project or (2) as a dedicated store
for a bigger projects and teams. Its git-like design aims at easy distribution and simple interface
makes it a good store of performance profiles along with the functional (or environmental) context.InstallationNote that we are no longer maintaining support for Python 3.8, nor do we support Python 3.12
(this is due to some of its dependencies). Perun may work, but we strongly advise to upgrade your
Python to one of the supported version between Python 3.9 and Python 3.11.You can install Perun from pip as follows:pip3 install perun-toolsuiteAlternatively you can install Perun from the source code as follows:git clone https://github.com/Perfexionists/perun.git
cd perun
make installThese commands install Perun to your system as a runnable python package. You can then run Perun
safely from the command line using theperuncommand. Run eitherperun --helpor see thecli
documentationfor more information about running Perun
commands from command line. Note that depending on your OS and the location of Python libraries, you
might require root permissions to install Perun.It is advised to verify that Perun is running correctly in your environment as follows:# You can run this only once: it will initialize the requirements necessary for testing
make init-test
# Runs all tests using pytest
make testor alternatively using Tox if you wish to test for more Python versions
(see thedeveloping section).Installing Tracer DependenciesIf you wish to use ourtracerprofiler, you have to install its dependencies first. Please refer
toINSTALL-tracer.md.DevelopingIn order to commit changes to the Perun, you have to install Perun in development mode:git clone https://github.com/Perfexionists/perun.git
cd perun
make devThis method of installation allows you to make a changes to the code, which will be then reflected
by the installation.Again, it is advised to verify that Perun is running correctly in your environment as follows:# You can run this only once: it will initialize the requirements necessary for testing
make init-test
# Runs all tests using pytest
make testOr you can useToxto run tests for all the supported
Python versions, as well as run static type checking, code style linting and generating
documentation:tox runIf you wish to test only against a single Python version, run Tox as:tox run -e py3.YwhereYis the python sub-version you wish to test. To see the available Tox environments (and
consequently, the supported Python versions), run:tox listIf you are interested in contributing to Perun project, please refer tocontributingsection. If you think your results could help others, pleasesend us
PR, we will review the code and in case it is
suitable for wider audience, we will include it in ourupstream.But, please be understanding, we cannot fix and merge everything immediately.Getting Started with PerunIn order to start managing performance of your project tracked bygit,
go to your project directory and run the following:perun init --configureThis creates a parallel directory structure for Perun storage (stored in.perun),
and creates initial configuration of the local project settings and opens it in the text editor
(by defaultvim).
The configuration is well commented and will help you with setting Perun up for your project.
You can specify, e.g., the set of collectors (such astimeortrace),
or postprocessors and specify which commands (and with which arguments and workloads) should be profiled.
Seeconfigurationfor more details about Perun's configuration.If you only wish to briefly try Perun out and
assuming you have opened the configuration file from the previous step(or you can open it from the path.perun/local.yaml),
then we recommend to uncomment keys calledcmds,workloadsandcollectors.
This will suffice to demonstrate Perun's capabilities.If you set up the configuration properly you can
now start collecting the profiles for the current version of your project using single command:perun run matrixThis command collects performance data (resulting into so called profiles),
according to the previously set configuration (seespecification of job matrixfor more
details). You can then further manipulate with these profiles: view the list of collected and registered profiles,
and visualize the profiles (seevisualization overview), or check for
possible performance changes (seedegradation
documentation):# Show list of profiles
perun status

# Show the first generated profile using tabular view
perun show 0@p tableof --to-stdout resources

# Register the first generated profile to current minor version
perun add 0@pNow anytime your codebase changes, rerun the collection phase, register new profiles
and you can check whether any change in performance can be detected:# Rerun the collection
perun run matrix

# Register the profiles for current minor version
perun add 0@p

# Run the degradation check
perun check headSelected Features of PerunIn the following, we list the foremost features and advantages of using Perun:Unified performance profile format: our format of performance profiles is baed onJSON; JSON is easily parsable by other programs and languages and provide
human-readable format. The downside is, however, its storage size and potentially slow processing.Natural specification of profiling runs---we allow users to specify how to execute
regular or more complex profiling workflows (the so-called jobs) inYamlformat;
YAML is widely used in CI/CD and other approaches, and is supported by many programs and languages.Git-inspired Interface---we provide command line interface for our tool suite; the commands are inspired by git
version control systems e.g.perun add,perun remove,perun status, orperun log. This allows users to
quickly adapt to using Perun and its tool suite.Efficient storage---performance profiles are stored compressed in the storage in parallel
to versions of the profiled project inspired by git. Note, that the resulting profiles can still be huge; we are
working on compressing and reducing the resulting profiles even further.Performance modeling---Perun's suite contains a postprocessing module for regression
analysis of profiles that construct mathematical models of dependent variables (e.g. runtime) wrt given independent
variable (seeregression analysis documentation),
which supports several different strategies for finding the best predicting model for given data
(such as linear, quadratic, or constant model). We also support more advanced modeling using, e.g. kernel regression
methods; these, however, require more domain specific knowledge.Interactive Visualizations---Perun's tool suite includes support for visualization of the results,
some of them based onholoviewsvisualization library, which
provides nice and interactive plots. However, Perun also supports classical output to console as well.API for profile manipulation---Perun provides modules for working with
profiles in external applications---ranging for executing simple queries over the resources
or other parts of the profiles, to converting or transforming the profiles to different representations
(e.g. pandas data frame). We particularly recommend using this API in the Python REPL.
Seeconversion apiandquery apifor overview.Automatic Detection of Performance Degradation---Perun offers several
heuristics for automatic detection of performance degradation between two project versions (e.g.
between two commits). This can be used either in Continuous Integrations, in debugging sessions or as parts of some
precomming hooks.Performance Fuzz-testing---Perun offers a performance oriented fuzz testing ---i.e. modifying inputs in
order to force error or, in our case, a performance change. This allows users to generate new time-consuming inputs
for theirs projects.As a sneak peek, we are currently working and exploring the following featurues in near future of
the project:Automatic Hooks---a range of automatic hooks, that will allow to automate the runs of job matrix,
automatic detection of degradation and efficient storage.Support for more languages---we are currently working on support for profiling more programming languages. In
particular, we wish to explore profiling languages such as C#, Python, Go or TypeScript. Moreover, we wish to support
some well known profilers such as valgrind toolsuite or perf in our workflow.Optimization of profiling process---in our ongoing work we are trying to optimize the runtime and the resulting
storage space of the profiling process while keeping the precision of the results. Some of our optimizations are
already merged in the upstream, however, we are currently enhancing our methods.Optimal selection of versions---currently we automatically only support the comparison of two latest versions in
git history. Alternatively, one can select two particular profiles to compare. We currently, exploring differt ways
how to select version in history for comparison for some given version.For more information about Perun's feature, please refer to ourextensive list of
features!ContributingIf you'd like to contribute, please first fork our repository and create a dedicated feature branch. Pull requests are
warmly welcome. We will review the contribution (possibly request some changes).In case you run into some unexpected behaviour, error or anything suspicious, either contact us
directly through mail orcreate a new Issue.We build Perun so it is easily extensible. In case you are interested in extending our tool
suite with new kinds of collectors, postprocessors or visualization methods, please refer to
appropriate sections in Perun's documentation (i.e. Create your owncollector,postprocessororvisualization).
Do not hesitate to contact us, if you run into any problems.If you are interested in contributing to Perun project, please first refer tocontributingsection. If you think your custom module could help others, pleasesend us PR, we will review the code and in case
it is suitable for wider audience, we will include it in ourupstream.But, please be understanding, we cannot fix and merge everything.LinksGitHub repository :https://github.com/Perfexionists/perunIssue tracker:https://github.com/Perfexionists/perun/issuesIn case of sensitive bugs like security vulnerabilities, please
contactTomas FiedororJirka Paveladirectly
instead of using issue tracker. We value your effort to improve the security and privacy of our project!Project documentation:Online:https://perfexionists.github.io/perun/Latest Typeset:https://github.com/Perfexionists/perun/blob/devel/docs/pdf/perun.pdfUnrelated links:Check out our research group focusing on program analysis, static and dynamic analysis, formal methods, verification
and many more:VeriFITLicensingThe code in this project is licensed underGNU GPLv3 license.AcknowledgementsWe thank for the support received fromRed Hat(especially branch of Brno), and Brno University of Technology
(BUT FIT).Further we would like to thank the following individuals (in the
alphabetic order) for their (sometimes even just a little)
contributions:Jan Fiedor(Honeywell)---for feedback, and technical
discussions;Jirka Hladkýand his team (RedHat)---for technical discussions and cooperation;Martin Hruška(BUT FIT)---for feedback, and technical
discussions;Viktor Malík(RedHat)---for feedback and support;Petr Müller(SAP)---for nice discussion about the project;Michal Kotoun(BUT FIT)---for feedback, and having faith in
this repo;Hanka Pluháčková(BUT FIT)---for awesome logo, theoretical
discussions about statistics, feedback, and lots of ideas;Adam Rogalewicz(BUT FIT)---for support, theoretical
discussions, feedback;Tomáš Vojnar(BUT FIT)---for support, theoretical discussions,
feedback;Jan Zelený(Red Hat)---for awesome support, and feedback.Development of this tool has been supported by AQUAS project (Aggregated Quality Assurance for
Systems,https://aquas-project.eu/). This project has received funding from the Electronic Component
Systems for European Leadership Joint Undertaking under grant agreement No 737475. This Joint
Undertaking receives support from the European Union’s Horizon 2020 research and innovation
programme and Spain, France, United Kingdom, Austria, Italy, Czech Republic, Germany.This tool as well as the information provided on this web page reflects only the author's view and
ECSEL JU is not responsible for any use that may be made of the information it contains.This project is co-funded by the European Union