During the last two decades the size both of main memory and of disc storage has grown by several orders of magnitude. While Moore's Law still holds, this will continue. Unfortunately, many of the early algorithms were developed for very small caches and need rethinking for main-memory database situations. Even the indexing solutions that have worked well with discs need re-examining for web search engines with huge amounts of main memory. Too often we assume that scalability will easily follow just by buying more and faster hardware, but all too often it is hindered by extra complexity.
Theoretical approaches can often mislead, by making simplistic assumptions.
Thus experimentation is all important in this area, and it needs to be carried out on a realistically large scale by well-equipped teams funded over longer periods and working closely with leading industrial laboratories.