Computer systems are complex, and the interactions in them are even more complex. Obtaining reproducible, statistically significant numbers is difficult. Systems are non-deterministic. Many benchmarks to not accurately represent any real workload. The next speaker was Norm Rubin.
Norm has a background in computers and performance.
He described two DEC performance tools that he is familiar with. Norm observed that the output from this tools is pretty, but arcane and difficult for anyone but an expert to use. Even with an expert, it is best suited to use with repeated runs of the same program with the same data set. The second tool that Norm discussed was the FX32 translation system. This system interprets programs, collects profile statistics, and then rewrites the programs, exploiting the profile information to improve their performance.