Performance Testers

==Metric-based Test Framework==
The metric-based test framework can be used to analyze all sorts of data gathered by the viewer at run time. The base implementation offers a variety of services to save and load data, parse LLSD results, compare current performance with a baseline and produce test reports.
The interface is implemented in llcommon/llmetricperformancetester.h. Two abstract classes are available:
* LLMetricPerformanceTesterBasic
* LLMetricPerformanceTesterWithSession
===LLMetricPerformanceTesterBasic===
The abstract class LLMetricPerformanceTesterBasic defines the general metric-based test framework.
This class can be directly inherited from for simple data gathering and provides predefined methods to save, load and compare results of performance sessions.
Below is the detailed doc for this class.
/**
 * @class LLMetricPerformanceTesterBasic
 * @brief Performance Metric Base Class
 */
class LL_COMMON_API LLMetricPerformanceTesterBasic
{
public:
 /**
 * @brief Creates a basic tester instance.
 * @param[in] name - Unique string identifying this tester instance.
 */
	LLMetricPerformanceTesterBasic(std::string name);
	virtual ~LLMetricPerformanceTesterBasic();
 /**
 * @return Returns true if the instance has been added to the tester map.
 * Need to be tested after creation of a tester instance so to know if the tester is correctly handled.
 * A tester might not be added to the map if another tester with the same name already exists.
 */
 BOOL isValid() const { return mValidInstance; }
 /**
 * @brief Write a set of test results to the log LLSD.
 */
	void outputTestResults() ;
 
 /**
 * @brief Compare the test results.
 * By default, compares the test results against the baseline one by one, item by item, 
 * in the increasing order of the LLSD record counter, starting from the first one.
 */
	virtual void analyzePerformance(std::ofstream* os, LLSD* base, LLSD* current) ;
 
 /**
 * @return Returns the number of the test metrics in this tester instance.
 */
	S32 getNumberOfMetrics() const { return mMetricStrings.size() ;}
 /**
 * @return Returns the metric name at index
 * @param[in] index - Index on the list of metrics managed by this tester instance.
 */
	std::string getMetricName(S32 index) const { return mMetricStrings[index] ;}
 
protected:
 /**
 * @return Returns the name of this tester instance.
 */
	std::string getTesterName() const { return mName ;}
 
 /**
 * @brief Insert a new metric to be managed by this tester instance.
 * @param[in] str - Unique string identifying the new metric.
 */
	void addMetric(std::string str) ;
 /**
 * @brief Compare test results, provided in 2 flavors: compare integers and compare floats.
 * @param[out] os - Formatted output string holding the compared values.
 * @param[in] metric_string - Name of the metric.
 * @param[in] v_base - Base value of the metric.
 * @param[in] v_current - Current value of the metric.
 */
	virtual void compareTestResults(std::ofstream* os, std::string metric_string, S32 v_base, S32 v_current) ;
	virtual void compareTestResults(std::ofstream* os, std::string metric_string, F32 v_base, F32 v_current) ;
 
 /**
 * @brief Reset internal record count. Count starts with 1.
 */
	void resetCurrentCount() { mCount = 1; }
 /**
 * @brief Increment internal record count.
 */
	void incrementCurrentCount() { mCount++; }
 /**
 * @return Returns the label to be used for the current count. It's "TesterName"-"Count".
 */
 std::string getCurrentLabelName() const { return llformat("%s-%d", mName.c_str(), mCount) ;}
 
 /**
 * @brief Write a test record to the LLSD. Implementers need to overload this method.
 * @param[out] sd - The LLSD record to store metric data into.
 */
	virtual void outputTestRecord(LLSD* sd) = 0 ;
private:
	void preOutputTestResults(LLSD* sd) ;
	void postOutputTestResults(LLSD* sd) ;
	std::string mName ; // Name of this tester instance
	S32 mCount ; // Current record count
 BOOL mValidInstance; // TRUE if the instance is managed by the map
	std::vector< std::string > mMetricStrings ; // Metrics strings
// Static members managing the collection of testers
public:	
 // Map of all the tester instances in use
	typedef std::map< std::string, LLMetricPerformanceTesterBasic* > name_tester_map_t;	
	static name_tester_map_t sTesterMap ;
 /**
 * @return Returns a pointer to the tester
 * @param[in] name - Name of the tester instance queried.
 */
	static LLMetricPerformanceTesterBasic* getTester(std::string name) ;
 /**
 * @return Returns TRUE if there's a tester defined, FALSE otherwise.
 */
	static BOOL hasMetricPerformanceTesters() { return !sTesterMap.empty() ;}
 /**
 * @brief Delete all testers and reset the tester map
 */
	static void cleanClass() ;
private:
 // Add a tester to the map. Returns false if adding fails.
	static BOOL addTester(LLMetricPerformanceTesterBasic* tester) ; 
};
===LLMetricPerformanceTesterWithSession===
The abstract class LLMetricPerformanceTesterWithSession is derived from the previous one and provides an additional abstraction that allows the definition of ad-hoc comparison methods for reporting. This class should be used when data need to be collated and analyzed in specific ways.
Below is the detailed doc for this class.
/**
 * @class LLMetricPerformanceTesterWithSession
 * @brief Performance Metric Class with custom session 
 */
class LL_COMMON_API LLMetricPerformanceTesterWithSession : public LLMetricPerformanceTesterBasic
{
public:
 /**
 * @param[in] name - Unique string identifying this tester instance.
 */
	LLMetricPerformanceTesterWithSession(std::string name);
	virtual ~LLMetricPerformanceTesterWithSession();
 /**
 * @brief Compare the test results.
 * This will be loading the base and current sessions and compare them using the virtual 
 * abstract methods loadTestSession() and compareTestSessions()
 */
	virtual void analyzePerformance(std::ofstream* os, LLSD* base, LLSD* current) ;
protected:
 /**
 * @class LLMetricPerformanceTesterWithSession::LLTestSession
 * @brief Defines an interface for the two abstract virtual functions loadTestSession() and compareTestSessions()
 */
	class LLTestSession
 {
 public:
 virtual ~LLTestSession() ;
 };
 
 /**
 * @brief Convert an LLSD log into a test session.
 * @param[in] log - The LLSD record
 * @return Returns the record as a test session
 */
	virtual LLMetricPerformanceTesterWithSession::LLTestSession* loadTestSession(LLSD* log) = 0;
 
 /**
 * @brief Compare the base session and the target session. Assumes base and current sessions have been loaded.
 * @param[out] os - The comparison result as a standard stream
 */
	virtual void compareTestSessions(std::ofstream* os) = 0;
 
	LLTestSession* mBaseSessionp;
	LLTestSession* mCurrentSessionp;
};
====Creating/Adding a Basic Test Metrics====
First, you need to create a tester class derived from the LLMetricPerformanceTesterBasic class that will hold your performance data. The key steps are:
*declare your own tester derived from LLMetricPerformanceTesterBasic
*in the constructor, declare all metrics you will use in this tester, the declaration order does not matter
*collect the test data in your own way. The usual way is to define an update() method that gets called and gather the relevant performance data.
*define the abstract virtual method outputTestRecord(LLSD* sd) to output your test data to the LLSD structure. Everything output to the LLSD in this function will be saved to a log file in the log folder.
*the final test report contains the following columns: metric_string, baseline_value, target_value, (target_value - baseline_value) and (100 * target_value / baseline_value).
Below is a complete code example implementation.
class YourOwnTester : public LLMetricPerformanceTesterBasic
{
public:
 YourOwnTester() ;
 ~YourOwnTester() ;
 // This will have to get called in code to update your perf data.
 // Note: you can create as many updateXx() variation as your perf system requires
 void update(const S32 d1, const F32 d2) ;
protected:
 // This is required. It tells the class how to pack the data in an LLSD stream
 /*virtual*/ void outputTestRecord(LLSD* sd) ;
private:
 // Define the relevant perf gathering variables. 
 // Note: the default compare method only supports S32 and F32 comparison. You need to overload the compare if you need to carry something else.
 S32 data1;
 F32 data2;
 ...
};
// Note : the tester name "yourtester" is important to remember as it is the name you will use on the command line
// when launching the viewer in perf metric gathering mode.
YourOwnTester::YourOwnTester() : LLMetricPerformanceTesterBasic("yourtester")
{
 // Declare all the metrics used in the tester.
 addMetric("metric-string-1") ;
 addMetric("metric-string-2") ;
 ...
 // Your own initializations
 data1 = 0;
 data2 = 0.0f;
 ...
}
YourOwnTester::~YourOwnTester()
{
 // You likely need to invalidate the static pointer holding that test instance
 sYourTester = NULL;
}
void YourOwnTester::outputTestRecord(LLSd *sd)
{
 std::string currentLabel = getCurrentLabelName();
 //insert your own code to output test results to sd
 //format like this
 (*sd)[currentLabel]["metric-string-1"] = (LLSD::Integer)data1;
 (*sd)[currentLabel]["metric-string-2"] = (LLSD::Real)data2;
 ...
} 
void YourOwnTester::update(const S32 d1, const F32 d2)
{
 // Do something with the input data to update your perf data
 data1 += d1;
 data2 += d2;
 ...
 // *Important* You need to call outputTestResults() when some perf gathering condition is met
 // Otherwise your data might not be saved to the log ever.
 if (condition)
 {
 outputTestResults();
 }
}
You may check the class LLImageCompressionTester as an example.
====How To Run Metric-based Automated Tests====
Performance metric gathering must be done in 2 main steps:
*Baseline performance data acquisition
*Target run and analysis
For performance data gathering to be performed, the viewer must be run with special command line arguments. See Client parameters to learn how to do this and for more information on all command line arguments.
=====Baseline Performance Data Acquisition=====
During that step, simply run the viewer so to gather initial data. You'll use that run to create a yourtester_baseline.slp file that will be used later to compare new runs with this baseline. If you are for instance on the verge of doing modification to improve the performance of some part of the viewer, this is something you want to do before doing any modification to the code so you can later measure how much your modification improved (or not) the performance measured.
Note that every time you add new performance metric, you need to re-create a baseline. It is therefore advisable to develop performance metric code in a non-modified repository and pull the performance metric code to the modified repository while developing new algorithms. It is actually not uncommon to get ideas of new performance data to gather while doing performance tuning development.
Steps:
*Make sure the log folder does not contain any previously gathered data, i.e. delete or backup the files yourtester.slp and yourtester_baseline.slp if any
*Launch the viewer from the command line with --logmetrics yourtester as an argument. It is required to use a tester parameter here. If the argument passed does not correspond to anything, the resulting log yourtester.slp will simply be empty. If you want to gather all performance metrics defined in the code as one, you can use metric as a parameter. Note though that the viewer is extremely slow and hard to use under this circumstance.
*Perform the test session. It is advisable to repeat a similar scenario between runs though, of course, this really depends about what you want to measure and compare.
*Quit the viewer: at that point, you should have a new yourtester.slp file in the log folder. Note that quitting might take some time as writing the yourtester.slp file is time consuming. Be patient. Do not force quit the viewer during that stage!
*Rename the file yourtester.slp to yourtester_baseline.slp and leave it in the log folder
=====Target Performance Data Acquisition and Analysis=====
You can perform target runs and analysis using the same baseline over and over. The basic data are saved in a yourtester.slp file and the data comparison is stored in a yourtester_report.csv file that can be open with Excel or Open Office.
Note that the code overwrites old yourtester.slp and yourtester_report.csv files so rename or move your reports if you need to.
Steps:
*Make sure the log folder contains a yourtester_baseline.slp file. The perf analysis will fail if that file is not present and no yourtester_report.csv file will be created.
*Launch the viewer from the command line with --logmetrics yourtester --analyzeperformance as arguments
*Perform the test session
*Quit the viewer: you can find the test results in the file yourtester_report.csv located in your secondlife log file folder.
=====More=====
If one wants to gather data automatically, it's possible to use --autologin and --replaysession on the command line.
==Published Testers and Results==
===TextureTester===
To be Completed
===ImageCompressionTester===
This tester is used to measure compression/decompression performance of JPEG2000 images.
It's triggered specifying the ImageCompressionTester performance tracking on the command line, i.e.:
* Mac: ./Second\ Life --logmetrics ImageCompressionTester --analyzeperformance
* Windows: create a shortcut to the executable, edit its properties and add --logmetrics ImageCompressionTester --analyzeperformance to the Target field in the Shortcut tab. 
* Linux: ./secondlife --logmetrics ImageCompressionTester --analyzeperformance
==== Baseline and Test Runs ====
For this series of tests:
* The baseline was created using Kakadu v4.2.1 from the Snowstorm 2.5 216083 build.
* The openjpeg v1.3.0 data were created using that same build minus the dynamic kdu library (no build required with the old dynamic linking strategy).
* The Kakadu v6.4.1 data were created using the Snowstorm 2.6 219259 build.
When comparing various decompression implementations, you must create your own baseline on your own machine, so the comparison is meaningful.
==== Results ====
===== Windows =====
Base = Kakadu v4.2.1 vs. Target = Kakadu v6.4.1	
<tab class=lltable head=top border=1>
Metric	 Base(B)	 Target(T)	 Diff(T-B)	 Percentage(100*T/B)
Time Decompression (s)	 32.7087	 29.1120	 -3.5968	 89.0037
Volume In Decompression (kB)	 69416.0313	 69254.7656	 -161.2656	 99.7677
Volume Out Decompression (kB)	 639489.0625	 654995.5000	 15506.4375	 102.4248
Decompression Ratio (x:1)	 9.2124	 9.4578	 0.2454	 102.6633
Perf Decompression (kB/s)	 2122.2468	 2378.9089	 256.6621	 112.0939
Time Compression (s)	 0.3180	 0.3855	 0.0675	 121.2389
Volume In Compression (kB)	 3145.7280	 3145.7280	 0.0000	 100.0000
Volume Out Compression (kB)	 393.0400	 392.8800	 -0.1600	 99.9593
Compression Ratio (x:1)	 8.0036	 8.0068	 0.0033	 100.0407
Perf Compression (kB/s)	 9892.3506	 8159.3887	 -1732.9619	 82.4818
</tab>
				
Base = Kakadu v4.2.1 vs. Target = openjpeg v1.3.0
<tab class=lltable head=top border=1>
Metric	 Base(B)	 Target(T)	 Diff(T-B)	 Percentage(100*T/B)
Time Decompression (s)	 31.6939	 134.4571	 102.7632	 424.2364
Volume In Decompression (kB)	 67361.5781	 67321.7813	 -39.7969	 99.9409
Volume Out Decompression (kB)	 620220.8750	 640426.3125	 20205.4375	 103.2578
Decompression Ratio (x:1)	 9.2073	 9.5129	 0.3056	 103.3188
Perf Decompression (kB/s)	 2125.3792	 500.6934	 -1624.6858	 23.5578
Time Compression (s)	 0.3180	 1.5392	 1.2212	 484.0213
Volume In Compression (kB)	 3145.7280	 3145.7280	 0.0000	 100.0000
Volume Out Compression (kB)	 393.0400	 314.1560	 -78.8840	 79.9298
Compression Ratio (x:1)	 8.0036	 10.0133	 2.0097	 125.1098
Perf Compression (kB/s)	 9892.3506	 2043.7842	 -7848.5664	 20.6602
</tab>
					
===== Mac =====
Base = Kakadu v4.2.1 vs. Target = Kakadu v6.4.1	
<tab class=lltable head=top border=1>
Metric	 Base(B)	 Target(T)	 Diff(T-B)	 Percentage(100*T/B)
Time Decompression (s)	 60.0396	 47.2449	 -12.7947	 78.6896
Volume In Decompression (kB)	 64200.5156	 64484.8047	 284.2891	 100.4428
Volume Out Decompression (kB)	 617027.9375	 626571.5000	 9543.5625	 101.5467
Decompression Ratio (x:1)	 9.6109	 9.7166	 0.1056	 101.0990
Perf Decompression (kB/s)	 1069.3031	 1364.9049	 295.6018	 127.6443
Time Compression (s)	 0.5290	 0.3624	 -0.1666	 68.5154
Volume In Compression (kB)	 3932.1599	 3145.7280	 -786.4319	 80.0000
Volume Out Compression (kB)	 490.9610	 393.0360	 -97.9250	 80.0544
Compression Ratio (x:1)	 8.0091	 8.0037	 -0.0054	 99.9320
Perf Compression (kB/s)	 7433.1387	 8679.0918	 1245.9531	 116.7621
</tab>
				
Base = Kakadu v4.2.1 vs. Target = openjpeg v1.3.0
<tab class=lltable head=top border=1>
Metric	 Base(B)	 Target(T)	 Diff(T-B)	 Percentage(100*T/B)
Time Decompression (s)	 31.6939	 134.4571	 102.7632	 424.2364
Time Decompression (s)	 60.0396	 215.3034	 155.2638	 358.6024
Volume In Decompression (kB)	 64200.5156	 65215.3086	 1014.7930	 101.5807
Volume Out Decompression (kB)	 617027.9375	 671115.9375	 54088.0000	 108.7659
Decompression Ratio (x:1)	 9.6109	 10.2908	 0.6798	 107.0734
Perf Decompression (kB/s)	 1069.3031	 302.8996	 -766.4036	 28.3268
Time Compression (s)	 0.5290	 0.5691	 0.0401	 107.5776
Volume In Compression (kB)	 3932.1599	 1572.8640	 -2359.2959	 40.0000
Volume Out Compression (kB)	 490.9610	 157.2180	 -333.7430	 32.0225
Compression Ratio (x:1)	 8.0091	 10.0044	 1.9952	 124.9122
Perf Compression (kB/s)	 7433.1387	 2763.8230	 -4669.3154	 37.1824
</tab>
===== Linux =====
Base = Kakadu v4.2.1 vs. Target = Kakadu v6.4.1	
<tab class=lltable head=top border=1>
Metric	 Base(B)	 Target(T)	 Diff(T-B)	 Percentage(100*T/B)
Time Decompression (s)	 45.7413	 34.1557	 -11.5855	 74.6716
Volume In Decompression (kB)	 61884.0898	 62320.6367	 436.5469	 100.7054
Volume Out Decompression (kB)	 593296.5000	 622710.4375	 29413.9375	 104.9577
Decompression Ratio (x:1)	 9.5872	 9.9920	 0.4048	 104.2225
Perf Decompression (kB/s)	 1352.9158	 1824.6021	 471.6863	 134.8644
Time Compression (s)	 0.5304	 0.4495	 -0.0809	 84.7459
Volume In Compression (kB)	 3145.7280	 3145.7280	 0.0000	 100.0000
Volume Out Compression (kB)	 392.8900	 393.0860	 0.1960	 100.0499
Compression Ratio (x:1)	 8.0066	 8.0026	 -0.0040	 99.9501
Perf Compression (kB/s)	 5930.8486	 6998.3916	 1067.5430	 117.9998
</tab>
				
Base = Kakadu v4.2.1 vs. Target = openjpeg v1.3.0
<tab class=lltable head=top border=1>
Metric	 Base(B)	 Target(T)	 Diff(T-B)	 Percentage(100*T/B)
Time Decompression (s)	 49.4230	 156.7737	 107.3508	 317.2082
Volume In Decompression (kB)	 67268.3828	 66718.4844	 -549.8984	 99.1825
Volume Out Decompression (kB)	 640793.8125	 652572.5625	 11778.7500	 101.8381
Decompression Ratio (x:1)	 9.5259	 9.7810	 0.2551	 102.6775
Perf Decompression (kB/s)	 1361.0748	 425.5718	 -935.5030	 31.2673
Time Compression (s)	 0.5304	 1.5683	 1.0379	 295.6786
Volume In Compression (kB)	 3145.7280	 3145.7280	 0.0000	 100.0000
Volume Out Compression (kB)	 392.8900	 313.7420	 -79.1480	 79.8549
Compression Ratio (x:1)	 8.0066	 10.0265	 2.0198	 125.2271
Perf Compression (kB/s)	 5930.8486	 2005.8431	 -3925.0055	 33.8205
</tab>
==== Analysis ====
Kakadu v6.4.1 is decompressing faster than Kakadu v4.2.1 on all platforms:
* 30% faster on Linux and Mac
* 12% faster on Windows
The relatively modest improvement on Windows might be due to the fact that the Windows version of Kakadu was already pretty fast. On the same hardware, v6.4.1 on Linux is still not as fast as v4.2.1 on Windows.
In compression, v6.4.1 is modestly faster on Linux and Mac (16%) but slower on Windows (18%). The amount of data considered though is not as big as for decompression and the significance of this performance not as important for the viewer.
Compared to openjpeg v1.3.0, Kakadu v4.2.1 is 3 times faster on Linux and Mac and 5 times faster on Windows, in both compression and decompression. Kakadu v6.4.1 increases that performance gap even more.
== Links ==
* Performance Tracking: Describes the system that can be used to track general rendering performance within the viewer