Rethinking Database System Architecture: Towards a Self-Tuning RISC-Style Database System

Database technology is one of the cornerstones for the new millennium’s IT landscape. However, database systems as a unit of code packaging and deployment are at a crossroad: commercial systems have been adding features for a long time and have now reached complexity that makes them a difficult choice, in terms of their "gain/pain ratio", as a central platform for value-added information services such as ERP or e-commerce. It is critical that database systems be easy to manage, predictable in their performance characteristics, and ultimately self-tuning. For this elusive goal, RISC-style simplification of server functionality and interfaces is absolutely crucial. We suggest a radical architectural departure in which database technology is packaged into much smaller RISC-style data managers with lean, specialized APIs, and with built-in self-assessment and auto-tuning capabilities 1. The Need for a New Departure Database technology has an extremely successful track record as a backbone of information technology (IT) throughout the last three decades. High-level declarative query languages like SQL and atomic transactions are key assets in the cost-effective development and maintenance of information systems. Furthermore, database technology continues to play a major role in the trends of our modern cyberspace society with applications ranging from webbased applications/services, and digital libraries to information mining on business as well as scientific data. Thus, database technology has impressively proven its benefits and seems to remain crucially relevant in the new millennium as well. Success is a lousy teacher (to paraphrase Bill Gates), and therefore we should not conclude that the database system, as the unit of engineering, deploying, and operating packaged database technology, is in good shape. A closer look at some important application areas and major trends in the software industry strongly indicates that database systems have an overly low “gain/pain ratio”. First, with the dramatic drop of hardware and software prices, the expenses due to human administration and tuning staff dominate the cost of ownership for a database system. The complexity and cost of these feed-and-care tasks is likely to prohibit database systems from further playing their traditionally prominent role in the future IT infrastructure. Next, database technology is more likely to be adopted in unbundled and dispersed form within higher-level application services. Both of the above problems stem from packaging all database technology into a single unit of development, maintenance, deployment, and operation. We argue that this architecture is no longer appropriate for the new age of cyberspace applications. The alternative approach that we envision and advocate in this paper is to provide RISC-style, functionally restricted, specialized data managers that have a narrow interface as well as a smaller footprint and are more amenable to automatic tuning. The rest of the paper is organized as follows. Section 2 puts together some important observations indicating that database systems in their traditional form are in crisis. Section 3 briefly reviews earlier attempts for a new architectural departure along the lines of the current paper, and discusses why they did not catch on. Section 4 outlines the envisioned architecture with emphasis on RISC-style simplification of data-management components and consequences for the viability of autotuning. Section 5 outlines a possible research agenda towards our vision. 2. Crisis Indicators To begin our analysis, let us put together a few important observations on how database systems are perceived by customers, vendors, and the research community. Observation 1: Featurism drives products beyond manageability. Database systems offer more and more features, leading to extremely broad and thus complex Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the VLDB copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Very Large Data Base Endowment. To copy otherwise, or to republish, requires a fee and/or special permission from the Endowment. Proceedings of the 26th International Conference on Very Large Databases, Cairo, Egypt, 2000 interfaces. Quite often novel features are more a marketing issue rather than a real application need or technological advance; for example, a database system vendor may decide to support a fancy type of join or spatial index in the next product release because the major competitors have already announced this feature. As a result, database systems become overloaded with functionality, increasing the complexity of maintaining the system’s code base as well as installing and managing the system. The irony of this trend lies in the fact that each individual customer (e.g., a small enterprise) only makes use of a tiny fraction of the system’s features and many high-end features are hardly ever exercised at all. Observation 2: SQL is painful. A big headache that comes with a database system is the SQL language. It is the union of all conceivable features (many of which are rarely used or should be discouraged to use anyway) and is way too complex for the typical application developer. Its core, say selection-projection-join queries and aggregation, is extremely useful, but we doubt that there is wide and wise use of all the bells and whistles. Understanding semantics of SQL (not even of SQL-92), covering all combinations of nested (and correlated) subqueries, null values, triggers, ADT functions, etc. is a nightmare. Teaching SQL typically focuses on the core, and leaves the featurism as a “learning-on-the-job” life experience. Some trade magazines occasionally pose SQL quizzes where the challenge is to express a complicated information request in a single SQL statement. Those statements run over several pages, and are hardly comprehensible. When programmers adopt this style in real applications and given the inherent difficulty of debugging a very high-level “declarative” statement, it is extremely hard if not impossible to gain high confidence that the query is correct in capturing the users’ information needs. In fact, good SQL programming in many cases decomposes complex requests into a sequence of simpler SQL statements. Observation 3: Performance is unpredictable. Commercial database engines are among the most sophisticated pieces of software that have ever been built in the history of computer technology. Furthermore, as product releases have been driven by the time-to-market pressure for quite a few years, these systems have little leeway for redesigning major components so that adding features and enhancements usually increases the code size and complexity and, ultimately, the general “software entropy” of the system. The scary consequence is that database systems become inherently unpredictable in their exact behavior and, especially, performance. Individual components like query optimizers may already have crossed the critical complexity barrier. There is probably no single person in the world who fully understands all subtleties of the complex interplay of rewrite rules, approximate cost models, and search-space traversal heuristics that underlie the optimization of complex queries. Contrast this dilemma with the emerging need for performance and service quality guarantees in ecommerce, digital libraries, and other Internet applications. The PTAC report has rightly emphasized: “our ability to analyze and predict the performance of the enormously complex software systems that lie at the core of our economy is painfully inadequate” [18]. Observation 4: Tuning is a nightmare and auto-tuning is wishful thinking at this stage. The wide diversity of applications for a given database system makes it impossible to provide universally good performance by solely having a well-engineered product. Rather all commercial database systems offer a variety of “tuning knobs” that allow the customer to adjust certain system parameters to the specific workload characteristics of the application. These knobs include index selection, data placement across parallel disks, and other aspects of physical database design, query optimizer hints, thresholds that govern the partitioning of memory or multiprogramming level in a multi-user environment. Reasonable settings for such critical parameters for a complex application often depend on the expertise and experience of highly skilled tuning gurus and/or timeconsuming trial-and-error experimentation; both ways are expensive and tend to dominate the cost of ownership for a database system. “Auto-tuning” capabilities and “zeroadmin” systems have been put on the research and development agenda as high priority topics for several years (see, e.g., [2]), but despite some advances on individual issues (e.g., [4,7,8,10,24]) progress on the big picture of self-tuning system architectures is slow and a breakthrough is not nearly in sight. Although commercial systems have admittedly improved on ease of use, many tuning knobs are merely disguised by introducing internal thresholds that still have to be carefully considered, e.g., at packaging or installation time to take into account the specific resources and the application environment. In our experience, robust, universally working default settings for complex tuning knobs are wishful thinking. Despite the common myth is that a few rules of thumb could be sufficient for most tuning concerns, with complex, highly diverse workloads whose characteristics evolve over time it is quite a nightmare to find appropriate settings for physical design and the various run-time parameters of a database server to ensure at least decent performance. Observation 5: We are not alone in the universe. Database systems are not (or no long