Use a stable test plan . The only way to properly compare the results of one test against another is to make sure that the test is the same. For my application, my software took a predetermined path through the landscape. Also, make sure that you are always running the same set of applications during the test. It is best just to not run any other applications on the computer while testing is in progress.
Performance analysis is a process of diminishing returns .
When you first start tuning your application, you will likely see significant improvements with little effort. However, as you continue, the effort will skyrocket and the results will plummet. Only change one thing at a time . Given the dual cache instruction stream and the L1 and L2 memory caches, a single change in one part of the program can have significant effects on other parts of the program. If you keep your changes down to 1 change per test, then you will know exactly how that change affected the whole program.