Chris provided some examples of these. The variance of the test runs was small. In summary, Chris listed some of the reasons that good benchmarking is difficult. Computer systems are complex, and the interactions in them are even more complex. Obtaining reproducible, statistically significant numbers is difficult.
Systems are non-deterministic.
Many benchmarks to not accurately represent any real workload. The next speaker was Norm Rubin. Norm has a background in computers and performance. He described two DEC performance tools that he is familiar with. Norm observed that the output from this tools is pretty, but arcane and difficult for anyone but an expert to use.