Reliable Performance Testing in C++
By: Joshua Weinstein
Tags: Performance, Technology, Cpp, Science, Programming

credits to freewallpapers1

C++ is a language designed for scalability, performance, and control. The language was created over 20 years age, and has since developed a huge community of developers and libraries across the world. In the modern era, C++ is mostly used for systems or performance critical software such as high frequency trading platforms or databases. A requirement for almost any production grade C++ software is reliable and consistent performance testing. Meeting benchmarks and scalability requirements is an essential capability for many C++ applications.

However, building and running C++ performance tests are not so straight forward. For quite a number of years, before the C++ 11 standard was released and incorporated into compilers, the language lacked a cross platform clock or time utility as part of the standard library. The C language has long since possessed a <time.h> header with various time and clock functions. Yet, many guarantees of those functions, such as whether or not a clock is monotonic and will never go backwards, are not promised within the C standard. Thus, any consistent performance tests prior to C++11 would have to have been implemented through operating system specific functions, like those of Windows or Linux.

System Vs Monotonic Clocks

The C++ chrono header deals with two types of clocks, system clocks and monotonic clocks. A system clock in any operating system is set to monitor the ticks since some point in the past, called the epoch. This relationship toward the epoch allows timestamps from a system clock to be easily convertible to human readable format. However, using a system clock leaves a vulnerability toward other programs modifying the system clock. The type std::chrono::system_clock guarantees access to the system clock on any platform or operating system.

The other major type of clock is the monotonic clock. Called a “steady clock”, within std::chrono::steady_clock , monotonic clocks just monitor the number of ticks since the machine the executable or program is running last restarted. Thus, a monotonic clock does not offer the ability to convert it's time points into human readable values, because it counts ticks and time from the moment the system last restarted. This is very different from a system clock, which counts time from an epoch, such as January 1st, 1970, in the case of UNIX time.

Single Interval Tests

The most fundamental performance test for any written code is a single interval test. The test involves recording a point in time, then running the desired code, then recording another point in time. The result of the test is the difference, also called the duration, between the first and second time points. A duration can only be derived from two time points recorded from the same clock type. Since we are only going to use a monotonic clock, all time points will be of the type std::chrono::time_point<std::chrono::steady_clock> . Durations between time points can either be floating point durations or integer durations. The type of duration you should use depends on what purposes the results of the performance tests you are running need to serve. Floating point durations allow statistical measurements like means to be calculated more easily.

Recording two time points and deriving two durations, a floating point and integer one, respectively, would look like this:

For the utilities in the <chrono> header, the - operator between two std::chrono:time_point<> objects evaluates to a duration. To get an integer duration, std::chrono::duration_cast<type> is always needed. Every duration has a Period , indicating the time period it represents. The std::milli is a time period represented by a predefined ratio defined in the <ratio> header. Although custom time periods with customized ratios can be used when creating a duration, it's far more straight forward to use the predefined Period types in the <chrono> header. They are listed below:

Furthermore, the type names for templated chronology types can become rather long, such as std::chrono::milliseconds int_ms = std::chrono::duration_cast<std::chrono::milliseconds>(end - start); If a peformance test is using the same time period and duration type most of the time, there is nothing wrong with using the auto keyword here. Just don't forget you will need an std::chrono::duration_cast for integer durations.

To get the actual value of time from the duration, you have to use it’s count() method. Additionally, a duration has a min() and max() method to indicate the minimum and maximum values for the duration type. For example, here's a way to check the time of a duration, compared to it's minimum and maximum values.

We expect the iterative function to be faster, but let’s see what the test shows:

Mean Interval Testing

Producing reliable performance test results requires not only timing the duration a piece of code takes to execute, but running that code and averaging that timed duration over a period of trials. Performance testing is distinct from functional or regression tests in the sense that all variables which affect the test cannot be accounted for. Where as a functional test can expect the same result given the same input. This is because a performance test depends on the state of several pieces of hardware, what condition they are in, and the many processes running on them. Running the same test many times and taking the average result of those test runs leads to more reliable and trustworthy performance testing.

This test shows that floating point durations from std::chrono can be used to calculate a mean duration. Mean durations over a larger number of test runs produce more confidence in the reliability of the test. Mean interval testing is great for performance tests that need to be very accurate about a specific section of code and specific inputs completing within some time period. However, you might need to test the performance of a program or implementation where the inputs might not be controlled, or there might be many smaller inputs. In those scenarios, mean interval testing won't work well because it only tells you the average, not anything about the distribution of times different runs had. To accomplish that, you need a histogram test.

Histogram Testing

Depending on the program or software being tested, some performance tests need to show the distribution of completion times rather than a mean duration or singular result. You might want to check if your mean completion time for your performance test is being driven by a lot of super fast and super slow runs, as opposed to times close to the mean. Additionally, you might need to guarantee the performance of a particular program runs within a specific range of intervals most of the time across many inputs, rather than ensuring performance on one input. To illustrate such behavior, a histogram is typically used. A histogram is a representation of data that differentiates individual run times within a pool of test runs by the amount of times they have been achieved. Such as, a simple histogram might look like:

For this example, we will use a simple hash function to run on many string inputs to generate the values for our histogram. All inputs will be the same length, but have different characters. Each run consists of generating the hash for one input string. The function is as follows:

For every test run of the hash function, there is an input, and a time taken to hash that input. Since a performance test does not care about the efficiency and collusion rate of a hash function, we shall ignore the outputs. A histogram can be represented in C++ as a collection of test run data points that can be abstracted into different interval arrangements. The data points in a histogram can be organized into different sets of intervals. The more intervals a histogram has, the more detailed picture it can show in terms of the spread and difference of runs.

To write this, three components are needed. One is a struct type to run and store the runs of each input on the hash function and the time taken for the run. In this case, milliseconds will be used. Next, we need a function to construct a vector of all times for all runs, and lastly, a function to query the vector of runs to see how many fall within a given range of times. This will allow the custom distribution against different time ranges to get a more detailed view into different run times that occur.