On predictable performance for distributed systems (2016)

Today, we increasingly rely on a plethora of interactive online services for social networking, e-commerce, video and audio streaming, email, and web search. These services need to deal with billions of visits per day while handling unprecedented volumes of data. At the same time, these services need to deliver fluid response times to their users. Failure to do so impacts the quality of experience for users and directly leads to revenue losses as well as increases in operational costs. Furthermore, the massive compute and storage demands of these large online services have necessitated a shift to the cloud. These services run inside large data-centers, across hundreds to thousands of heterogeneous servers, leading to a paradigm called warehouse-scale computing (WSCs). A complex ecosystem of fault-tolerant distributed systems back these services, each of which run on clusters of hundreds to many thousands of servers. These distributed systems include data-analytics stacks (e.g., Hadoop), distributed storage systems (e.g., Cassandra), cluster schedulers (e.g., Borg), as well as web-applications architected as micro-services (e.g., Netflix). These systems present operational challenges that occur due to several dynamic conditions, such as complex inter-server request-response patterns, server-side performance fluctuations, skews, and hot-spots in data access patterns, as well as multi-tenancy. Several of these dynamic conditions manifest at very small timescales, making operator intervention and tuning an infeasible means of addressing them. As a consequence, end-to-end system performance becomes notoriously difficult to predict and guarantee. This necessitates robust mechanisms by which distributed systems can adapt to these dynamic conditions that occur at very short timescales. To address this challenge, the goal of this dissertation is to develop adaptive mechanisms to achieve performance predictability for distributed systems. We focus our attention on two important classes of distributed systems that permeate the WSC landscape. The first is that of low-latency distributed storage-systems, which are often in the critical path of online services today. Storage servers in these settings experience performance fluctuations due to a multitude of reasons, and this significantly harms application performance. We demonstrate the need for adaptive replica selection, wherein storage clients route requests to replica servers in response to performance heterogeneity among different servers. We then present the design and implementation of the C3 system, which embodies a novel adaptive replica-selection mechanism. In C3, clients, with minimal assistance from servers, adaptively prefer faster replicas while avoiding degenerate behaviors such as load oscillations. In experiments conducted on Amazon EC2, in production settings at Spotify and SoundCloud, as well as through simulations, C3 significantly reduces tail latencies (up to 3x at the 99.9th percentile) and improves system throughput (up to 50\%). The second class of systems we focus on are service-oriented architectures (SOA) and micro-services, an increasingly popular architectural approach to building fault-tolerant online services. We present how complex interactions between different workloads can lead to significantly degraded system performance, potentially leading to widespread application outages. We discuss the challenges in meeting end-to-end performance objectives of interest in these settings, such as efficient and fair resource utilization, meeting deadlines, as well as reducing latencies. We present the design and implementation of Cicero, a system that embodies adaptive, distributed algorithms to enforce end-to-end resource management policies in SOAs and micro-services. In system evaluations wherein we emulated a production SOA, we demonstrated how Cicero can be used to achieve performance isolation among tenants, avoid cascading failures, and meet a high fraction of end-to-end deadlines.