Experimental design is really important Since the goal of a performance measurement test is to gather valid quantifiable data, your experimental design must be valid as well. This influence is called the experimental effect. However, if other factors are introduced into the design, the effect may be confounded, that is, not statistically valid due to tainting by the other factors. Your design must take into account possible confounding factors and eliminate possible sources of tainting. Data doesn't tell the whole story Testing solely for the purpose of procuring performance data doesn't seem to be as common as it used to be, for several reasons.
Performance testing requires very rigorous test designs and extensive resources.
Most companies don't have the time or money to do research of this kind. Also, the types of things tested are often at a very granular level. Maybe if you're designing call center software, and amortized over thousands of operators across the country, saving each one half a second per call could save millions of dollars per year. But for most office productivity applications. Performance measurement is used in initial stages of design to provide benchmarks for the design process.