It would be a foolish firm that didn't submit a system at least reasonably well optimised to exploit the way the tests are run. But the nature of Ellison's claim seems to sail remarkably close to suggesting that Microsoft goes a little further than simply optimising the SQL Server database and Windows 2000 operating system. It is certainly the case that, with these two products, Microsoft has a huge amount at stake in the enterprise marketplace. If it fails to demonstrate it can cope with the scale and performance enterprises now require, it will permanently lose to Unix and the big database suppliers. What is more, no matter how well the TPC configures the benchmarks, they are based on simulations of real life, not real life itself.
Companies can develop machines to do better in the tests than under real-life circumstances.
So perhaps it is now time for a different approach to benchmarking. If businesses want to know how machines would perform in a real-world enterprise environment, then surely that is what should be used as a benchmark. Perhaps it is time for the different user groups to come together to form a super-user group that could, through the Internet, create real benchmark tests. So if, for example, an operating system or database supplier claimed its offerings could run over large clusters and thousands of concurrent users, such environments could be constructed for real, and the claim could be put to the ultimate test.