Points of Departure Developing benchmarks is not trivial. For the application level benchmark suite, representative applications must be chosen as well as appropriate specific tasks to be tested within each application. Furthermore with new technology like Transmeta's Crusoe, considerations must be made for dynamic optimizations and dependency on cached translations to speed repetitious activities that dominate typical application usage. In other words, to be fair to Transmeta, the execution of the benchmarks must be slowed down by inserting many, many pauses to create CPU-time for background optimizations. Additionally, repetitive tasks must, in fact, be repeated many times in the benchmark software to take advantage of Crusoe's cached translations.
Synthetic benchmarks are equally tricky.
First, what needs to be tested must be decided, and, as technology changes, so must this list. Each benchmark must be carefully coded and checked to insure that no parasitic code is introduced that can skew the results. Finally, as important as the benchmarks are, we also need extensive documented methodologies, which describe in detail how to set up and use these benchmarks. A few good people are already taking it upon themselves to develop new benchmarks , but these efforts are scattered and embryonic. We must step beyond this stage to create a coordinated, comprehensive effort.