Ziff-Davis would have you think. SPEC Quite often, you can get around the problems inherent in just using one benchmarking program by using a collection of them to exercise different parts of a system. This is the idea behind the benchmarking suite . The most popular benchmarking suite for gauging the performance of CPUs is the SPEC suite. The benchmarker then takes this source code, compiles it for the system in question and then can tune the system for the best results.
Vendors get the source for the programs and compile it themselves before running the benchmarks.
The baseline performance measurement restricts the vendor to a single set of flags and a single compiler for all the benchmarks in the same language. The SPEC95 suite specifies a set of tools with which to build and run the benchmark programs, and they limit the number of compiler flags to four, not counting portability flags. In SPEC92, the optimized performance measurement allowed vendors to go wild, using benchmark-specific compiler flags that, if used on a normal program, would generate either illegal code or code that s considerably slower. In some cases, the difference between the optimized and unoptimized scores could be substantial. With the release of SPEC95, some efforts have been made to reign vendors in.