System support for end-to-end performance management

This dissertation introduces, implements, and evaluates the novel concept of "Service Paths", which are system-level abstractions that capture and describe the dynamic dependencies between the different components of a distributed enterprise application. Service paths are dynamic because they capture the natural interactions between application services dynamically composed to offer some desired end user functionality. Service paths are distributed because such sets of services run on networked machines in distributed enterprise data centers. Service paths cross multiple levels of abstraction because they link end user application components like web browsers with system services like http providing communications with embedded services like hardware-supported data encryption. Service paths are system-level abstractions that are created without end user, application, or middleware input, but despite these facts, they are able to capture application-relevant performance metrics, including end-to-end latencies for client requests and the contributions to these latencies from application-level processes and from software/hardware resources like protocol stacks or network devices. Unique service paths are created for different classes of application-level requests, thus making the notion of request class the basic quantum of enterprise management they support. Finally, service paths are constructed by software modules embedded into operating systems on participating machines. These modules dynamically measure and analyze program behavior. Beyond conceiving of service paths and demonstrating their utility, this thesis makes three concrete technical contributions. First, we propose a set of signal analysis techniques called "E2Eprof" that identify the service paths taken by different request classes across a distributed IT infrastructure and the time spent in each such path. It uses a novel algorithm called "pathmap" that computes the correlation between the message arrival and departure timestamps at each participating node and detect dependencies among them. A second contribution is a system-level monitoring toolkit called "SysProf", which captures monitoring information at different levels of granularity, ranging from tracking the systemlevel activities triggered by a single system call, to capturing the client-server interactions associated with a service paths, to characterizing the server resources consumed by sets of clients or client behaviors. SysProf operates enterprise-wide in that it can track the application- and system-level activities and associated resource usage across the multiple machines used in typical enterprise infrastructures. The third contribution of the thesis is a publish-subscribe based monitoring data delivery framework called "QMON". QMON offers high levels of predictability for service delivery and supports utility-aware monitoring while also able to differentiate between different levels of service for monitoring, corresponding to the different classes of SLAs maintained for applications. Service Paths and the E2EProf methods implementing them are shown useful for and are evaluated with multiple enterprise-class applications, which include a high performance shared network file service, a multi-tier auctioning web site and an Operational Information System (OIS) used by our industrial partners. For these applications, service paths are used to automatically detect request paths and then find performance bottlenecks associated with those paths. In addition, they are used to enforce application-level SLAs by adapting the middleware's or application's scheduling and dispatching behavior based on the resources consumed by different request classes. Service paths can also assist in dynamic application tuning and provisioning, by using them when making system management decisions like adding servers, adjusting the number of threads used in applications, changing buffer sizes, etc. Finally, we demonstrate the efficacy of the general approach to runtime analysis taken by our work by first using E2EProf to isolate problematic nodes in a service path and then use the additional and relatively more intrusive capabilities of SysProf to "zoom" into those nodes to perform micro-level analysis and identify the exact workloads, application components, and/or resources causing the performance problem. (Abstract shortened by UMI.)