Cloudstone : Multi-Platform , Multi-Language Benchmark and Measurement Tools for Web 2 . 0

Web 2.0 applications place new and different demands on servers compared to their Web 1.0 counterparts. Simultaneously, the definitive arrival of pay-as-you-go “cloud computing” and the proliferation of application development stacks present new and different degrees of freedom in deploying and tuning software-as-a-service. We first identify non-obvious challenges and caveats to performing “apples-to-apples” comparisons of Web 2.0 application deployments. We then offer Cloudstone, a toolkit consisting of an open-source Web 2.0 social application (Olio), a set of automation tools for generating load and measuring its performance in different deployment environments, and a recommended set of constraints for computing a metric we believe makes more sense, dollars per user per month. By way of example we present preliminary measurements of both Rails and PHP versions of Cloudstone on a variety of Amazon Elastic Compute Cloud (EC2) configurations, discussing the challenges of comparing platforms or software stacks and how Cloudstone can help quantify the differences. 1. Why We Need New Workloads Existing Web benchmarking tools (ab, httperf, SPECWeb) and applications (RuBiS, PetStore) are becoming less relevant to current practice in three ways. First, Web 2.0 application functionality induces new characteristics in workloads that servers must handle. Second, the definitive arrival of “pay-as-you-go” cloud computing has brought fast growth and large scale within the reach of independent developers, making the corresponding concerns of benchmarking, stress testing and scalability more broadly applicable. Lastly, debate continues over the performance differences among different development stacks, but we lack tools to investigate such questions systematically. 1.1. Web 2.0 Workloads are Different Following Tim O’Reilly’s widely-cited article [5] , we distinguish dominant architectures and features of “Web 1.0” applications (c.1995-2005) from those of “Web 2.0” applications (c.2005-present), noting their effect on application server workloads and deployment architectures. One-to-many vs. many-to-many: the “mass customization” of Web 1.0 presents the same content heavily customized to each user, but since different users’ activities and profiles rarely affect each other, a natural scaling strategy involves partitioning by user ID. In contrast, the social networking features of Web 2.0, in which each user’s actions and preferences affect many other users in her network, suggest no obvious static partitioning as a scaling strategy. User-contributed content: Whereas Web 1.0 focused on publishing to users, Web 2.0 users publish to each other via blogs, photostreams, tagging (Digg, Del.icio.us, etc.), collaborative filtering (e.g. Amazon book reviews and recommendations), etc. This changes the read/write ratio and write patterns compared to Web 1.0 applications.  The quest for improved interactivity through AJAX (Asynchronous JavaScript And XML), in which JavaScript code communicates with the server in the background, generates extra work on the server during what would otherwise be the Web user’s “think time”. On the other hand, well-coded AJAX may result in less rendering work on the server. A benchmark should help quantify these effects. 1.2. Cloud Computing is Different Pay-as-you-go storage and computing, such as Amazon’s EC2 and S3, change the economics of service deployment in two important ways. One is the much lower cost of “instant” incremental scalability. Another is that capacity can be quickly un-deployed to save money; developers need neither provision for peaks nor waste money on idle capacity during nonpeaks. The net effect is that linear scaling and stress testing at high load, until recently the purview of heavilycapitalized corporations, are now part of the operational landscape for independent developers as well. Even in “locked down” cloud computing environments such as EC2 where the developer has little control over network topology or hardware platform, understanding the performance bottlenecks imposed by the offered infrastructure is valuable.  A Web 2.0 Application, Workload and Metrics The Cloudstone toolkit addresses these requirements with three components: (1) Olio1, an open-source incubator project recently launched by Sun and UC Berkeley, consisting of two complete implementations of a social-event calendar web application (Ruby on Rails and PHP), and a sophisticated opensource workload generator, Faban2, that can scale to thousands of simulated users and supports fine-grained timevarying workloads. Both implementations provide the same Web 2.0 features (user-generated metadata, social networking functions, and a rich AJAX-based GUI) and adhere to each development stack’s idioms, exposing architectural as well as implementation-specific deployment and tuning choices such as caching and database tuning parameters. (2) A set of automation tools, including database population, metric gathering, etc., for running large Olio 1 http://incubator.apache.org/projects/olio.html 2 http://faban.sunsource.net experiments on cloud computing environments such as Amazon Elastic Compute Cloud (EC2). (3) A recommended methodology and set of parameters for computing the key metric of dollars per user per month in a deployment of the application. Cloudstone consists of 100% open source components connected in an architecture representative of datacenterbased deployment, and supplied as a set of virtual machine images for Amazon EC2 that readers are invited to download immediately. We include some preliminary results of Cloudstone measurement on Amazon EC2. It is our expectation that CloudStone can be used to systematically investigate questions such as: How do different development stacks trade single-node performance for code complexity and programmer productivity? What is the relative performance difference between hosting a Web 2.0 application on a large number of modest-capacity servers vs. a smaller number of heavily-provisioned manycore servers? How do two different dynamic provisioning algorithms respond to workload peaks? What is the cost of deployment per user? 2. Cloudstone Overview 2.1. Goals and Non-Goals Our goal is to capture “typical” Web 2.0 functionality in a datacenter or cloud computing environment, and provide for consistent evaluation of how many dollars per user per month an installation can support under typical production conditions. To this end we provide a realistic workload generator, allow flexibility in deployment to support a range of best practices for caching and tuning, and allow for testing and data collection of a variety of scenarios, including stress testing, linear scaling, “hockey stick” surges, etc. Non-goal: arguing for any one development stack over another. Many factors influence the choice of a development stack, and at best CloudStone will help developers systematically quantify some of the effects of those choices. Non-goal: emulating legacy applications. Our choice of application features and development stacks reflects popular design points for Web 2.0 application design today. Non-goal: providing the “best” (fastest, most memoryefficient, etc.) implementation of this particular application. Our goal is code that exploits each platform’s strengths and idioms as a competent developer on that platform would do. 2.2. Components and Typical Workflow Cloudstone follows the now-canonical three-tier Web application architecture: a stateless Web server tier, a stateless or soft-state (caching or affinitized) application server tier, and a persistence tier. A typical experiment consists of: 1) Choose a deployment architecture and arrange for Cloudstone’s included scripts to deploy the components 2) Prepare a workload profile for the workload generator 3) Run the experiment, deploying the workload generator to one or more machines distinct from those on which the application is deployed 4) Collect the resulting data Figure 1. The Olio “social events” application’s functionality and implementation are representative of Web 2.0 in both implementations: Ruby on Rails and PHP. Cloudstone provides various AMI’s (virtual machine image files compatible with Amazon’s Elastic Compute Cloud) that conveniently bundle the components to facilitate this workflow. We describe each of the components briefly.