Embracing Failure: A Case for Recovery-Oriented Computing (ROC)

Motivated by the lack of availability demonstrated by current approaches to building servers for the Internet environment, we argue for a new approach to building highly-available systems that better reflects the realities of the modern server environment, namely that failures of hardware, software, and humans are inevitable. Our approach, denoted recovery-oriented computing (ROC), recognizes the inevitability of unanticipated failure and thus emphasizes recovery and repair rather than simple fault-tolerance. We define the properties that a ROC system must provide, and briefly consider how they might be achieved. 1 The importance of availability and why it is lacking Availability is the most important metric for modern computer systems. It is the lifeblood of corporate survival for e-commerce and e-business enterprises, and it is particularly crucial to those built around delivering services directly to the customer over the Internet. In the words of Kal Raman, CEO of drugstore.com, “availability is as important as breathing in and out [is] to human beings” [4]. High availability used to be a solved problem in the transaction-processing community. The solution was simple: buy an expensive fault-tolerant mainframe from a company like Tandem or IBM, deploy the vendor-supplied high-availability database system, and hide the whole thing in the back office behind a firewall of dumb terminals and human agents. With such a well-designed, stable, and controlled environment, high availability was straightforward to engineer. While some applications can still use this approach, today’s most interesting applications do not. The computing world has shifted toward a much more distributed, heterogeneous environment, where transaction processing functionality and data access are exposed directly to customers through a complicated, heterogeneous conglomeration of interconnected systems—databases, application servers, middleware, and web servers—constructed from a multi-vendor mix of off-the-shelf hardware and software. In these systems, perceived availability is defined by the weakest link in the system, and so it is not enough to simply have a robust TP backend. In this kind of environment, the environment of e-commerce and Internet business, availability is not being delivered as promised. Despite high-profile advertising by all major vendors purporting to offer high-availability solutions, and despite entire communities of researchers focusing on topics such as reliability, maintainability, and fault tolerance, outages and failures remain frequent. A recent survey by InternetWeek revealed that 65% of surveyed sites suffered a customer-visible outage at least once in the previous 6-month period; 25% reported three or more outages during that period [15]. What’s wrong with this picture? Why does delivered availability fall so short of what is promised? We claim that the answer is simple: there is a fundamental mismatch between traditional high-availability approaches—fault-tolerant hardware, careful software testing, vendor-supplied technicians—and the realities of modern heterogeneous, distributed server environments, like those backing e-commerce and e-business sites. This mismatch goes as deep as the key assumptions underlying traditional high-availability design: that hardware and software can be built to have negligible failure rates, that failure modes of systems can be predicted and tolerated, and that maintenance and repair are error-free procedures. As we will show in the next section of this paper, none of these assumptions hold true for modern network service environments. If we want to address availability in these environments, then, we must sub