Benchmark profiling for distributed systems

Embodiments of the invention may be used to generate a benchmark profile for a computing job configured to execute on distributed systems. The benchmark profile may be used to predict the performance of components of a computing job for a variety of different distributed computing system architectures. A profiling tool evaluates the computing job to identify the particular performance characteristics of the application and match this with benchmarks that are most representative of the identified performance characteristics and store them in the benchmark profile. The identified benchmarks may then be run on different configurations of a distributed computing system in order to predict the performance of the application for a variety of scenarios.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the invention relates to benchmark profiling, specifically to benchmark profiling software applications configured to execute on large parallel or distributed computing systems.

2. Description of the Related Art

Distributed computing systems, such as grid computing and computer clusters, are useful tools for breaking down large computing tasks, or jobs, into many smaller tasks that execute concurrently. Used in this manner, distributed systems are highly effective tools to perform large computing tasks in a minimal amount of time.

Distributed systems typically contain a large number of heterogeneous systems containing one or more compute nodes. Because the heterogeneous systems have different hardware architectures, each provides different advantages in executing different types of software. For example, systems with large memories provide good architectures for running database applications. Systems with a number of specialized processors are optimal for specialized processing, such as processing video images.

A benchmark provides a software tool that analyzes the performance of a given hardware architecture, relative to a particular specific performance trait. A benchmark allows users to compare the efficiency of different architectures for the same performance task, allowing a best architecture for the software task to be determined.

Two common types of benchmarks include application benchmarks and synthetic benchmarks. Application benchmarks dynamically record performance metrics while a software application is executing. On the other hand, synthetic benchmarks mimic the performance of a piece of software on a system to predict performance metrics without actually executing the application. Both of these types of benchmarks may be used to analyze how efficient a given computer architecture is regarding different performance traits while executing (either actually or synthetically).

SUMMARY OF THE INVENTION

One embodiment of the invention provides a method for generating a benchmark profile used to predict the performance of a software application. The method generally includes identifying a plurality of performance characteristics associated with the software application. Each performance characteristic specifies a processing activity performed by the software application. The method may also include determining a ratio for each identified performance characteristic, where each ratio specifies a proportion of the processing activity performed by the software application for one of the identified performance characteristics relative to the processing activity of the software application as a whole. For each of the plurality of performance characteristics, at least one benchmark is identified that is configured to evaluate the performance of a computer system relative to the performance characteristic. The method also includes generating the benchmark profile, where the benchmark profile stores the ratio for each identified performance characteristic and an indication of each identified benchmark.

Another embodiment of the invention includes a computer-readable storage medium containing a program configured to generate a benchmark profile used to predict the performance of a software application which, when executed on a processor, performs an operation. The operation generally includes identifying a plurality of performance characteristics associated with the software application. Each performance characteristic specifies a processing activity performed by the software application. The operation may also include determining a ratio for each identified performance characteristic, where each ratio specifies a proportion of the processing activity performed by the software application for one of the identified performance characteristics relative to the processing activity of the software application as a whole. For each of the plurality of performance characteristics, at least one benchmark is identified that is configured to evaluate the performance of a computer system relative to the performance characteristic. The operation may further include generating the benchmark profile, where the benchmark profile stores the ratio for each identified performance characteristic and an indication of each identified benchmark.

Still another embodiment of the invention includes a system having a processor and a memory containing a program configured to generate a benchmark profile used to predict the performance of a software application which, when executed on the processor, performs an operation. The operation generally includes identifying a plurality of performance characteristics associated with the software application. Each performance characteristic specifies a processing activity performed by the software application. The operation may also include determining a ratio for each identified performance characteristic, where each ratio specifies a proportion of the processing activity performed by the software application for one of the identified performance characteristics relative to the processing activity of the software application as a whole. For each of the plurality of performance characteristics, at least one benchmark is identified that is configured to evaluate the performance of a computer system relative to the performance characteristic. The operation may further include generating the benchmark profile, where the benchmark profile stores the ratio for each identified performance characteristic and an indication of each identified benchmark.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Application benchmarks may frequently be impractical to use in evaluating the performance of a software application on distributed system architectures. Application benchmarks require that a software application be running, which in some cases is impractical on a distributed system. Typically, software running on distributed systems requires extensive system setup before execution. Further, executing software on a distributed system may take hours, or even days, making application benchmarking time-consuming and expensive.

While synthetic benchmarks do not require that the software application be configured, loaded, and run on the distributed system, these benchmarks analyze performance in terms of how effectively a particular architecture performs a specific performance trait. For example, the well-known Whetstone benchmark measures the efficiency of a computer architecture with regard to processor use, specifically in terms of how many floating-point operations can be performed per second. Hence, for jobs on distributed systems, which typically require many interrelated performance traits to be evaluated to determine a preferred architecture for a given application, individual synthetic benchmarks may be ineffective.

Embodiments of the invention provide a method to generate a benchmark profile for a computing job configured to execute on distributed systems. The benchmark profile may be used to predict the performance of components of a computing job for a variety of different distributed computing system architectures. The computing job itself may include a plurality of subroutines, each with distinct performance traits, e.g. processor use, or memory consumption, floating point operations, resource accesses etc. For example, consider a case where a computing job includes two subroutines, one heavily dependent on processing power, and the other dependent on available memory. In such a case, a profiling tool could evaluate the computing job and identify these particular performance characteristics, also referred to as performance traits, of these two subroutines. Further, the profiling tool may identify benchmarks that are most representative of these identified performance characteristics and store them in benchmark profile. The identified benchmarks may be then run on different configurations of a distributed computing system to predict the performance of the application for a variety of scenarios.

FIG. 1illustrates the high level architecture of a computing cluster100, according to one embodiment of the invention. Of course, embodiments of the invention may be adapted for use with a variety of other distributed computer systems, including grid computing, stream processing, and adaptive architecture supercomputing. Accordingly, the description of the architecture shown inFIG. 1is not intended to limit the present invention.

Cluster100provides a conceptual illustration of a Beowulf cluster, as well as other clustering architectures. As shown, cluster100includes a user node102, gateway node104, and nodes106connected via high-speed network switch108. Those skilled in the art will recognize thatFIG. 1provides a simplified representation of a computing cluster, and that the nodes of a typical computing cluster include a number of additional elements.

User node102may provide an interface to cluster100. As such, user node102allows users to create, submit, and review the results of computing tasks submitted for execution on the nodes106of system100. Head/gateway node104connects the user node102to the compute nodes106. Compute nodes106provide the processing power of cluster100. As is known, clusters are often built from racks of commonly available personal computer components. Thus, each node106may include one or more CPUs, memory, hard disk storage, a connection to high speed network switch108, and other common personal computer components.

FIG. 1also illustrates a job110running on user node102and subroutines1141-6running on compute nodes106. In one embodiment, job110may include a plurality of separate components, or subroutines, to be dispatched by user node102for execution on the compute nodes106. Users may submit job110for execution through an interface provided on user node102. In turn, user node102may execute job110by dispatching each subroutine114of the job110to the compute nodes106. Each subroutine114may be executed on different nodes106within cluster100.

In one embodiment, profiling component112may be configured to generate a benchmark profile, which provides a profile indicating which of one or more existing benchmarks, or portions of benchmarks, may accurately represent the runtime characteristics of job110, and/or of subroutines1141-6. The profile execution component115may use the benchmark profile to invoke the appropriate benchmarks across one or more available nodes106to predict the likely performance of the job110, given the current system state of cluster100, without actually deploying the job110. As described in greater detail below, profiling component112may be configured to determine a benchmark profile for job110by analyzing the source code of job110to determine the types of operations that are performed by subroutines1141-6, by measuring the characteristics of job110when it is executed on cluster100to generate a profile for future use, and/or via manual configuration by the user. In one embodiment, the benchmark profile may include a composition of individual benchmarks representative of the computing activity performed by job110. Further, the benchmarks in the benchmark profile may be weighted so that the contribution of each benchmark reflects the amount of processing activity performed by job110, relative to other benchmarks in the benchmark profile.

Further, profiling component112may be configured to determine a “best-fit” for a particular subroutine114, based on the benchmark profile for generated for job110and the available nodes106. As used herein, a “best-fit” generally refers to a process of matching one of subroutines114to the node106that may be able to more efficiently execute that subroutine114than other nodes106.

FIG. 2Aillustrates a data flow diagram200for a profiling component to generate a benchmark profile for a computing job, according to one embodiment of the invention. As shown, data flow diagram200includes a job205, a benchmark trait assignment data structure215, a profiling component, and a profile230. Illustratively, the profiling component225receives input in the form of job205and benchmark trait assignment215and uses this input data to generate and output profile230. In one embodiment, job205may be represented as a data structure that contains a textual representation of a computing job, e.g., the program source code of the computing job. Benchmark-trait assignment data structure215may include a collection of identifiers (IDs) for different benchmark applications. Each benchmark application may be used to evaluate the efficiency of a distributed computer system, relative to a particular performance characteristic. For example, for determining the efficiency of processor use, the Whetstone benchmark may be identified in benchmark-trait assignment215. Similarly, other benchmarks may be associated with other performance characteristics.

FIG. 2Billustrates an expanded view of computing job205ofFIG. 2A, according to one embodiment of the invention. In one embodiment, job205may include a collection of 1-N subroutines214. As described, each subroutine214may be part of a computing task configured to execute concurrently on nodes of a distributed computing system (e.g., the nodes106of cluster100ofFIG. 1). Subroutines214may be represented as program source code, executable code, or any other representation used to determine the type of processing activity (or activities) performed by each subroutine214.

FIG. 2Cillustrates an example of benchmark trait assignment data structure215. As shown, benchmark trait assignment data structure215provides a table that includes a program activity column216, a performance trait column217, and a benchmark218column. Each row of this table associates a listed program activity with both a trait (i.e., a computing resource required by that program activity) and a benchmark representative of that program activity.

Entries in program activity column216may represent any computing tasks or actions performed by a given subroutine while executing on a node106. Generally, each program activity listed in column216may be dependent on computing resources of a distributed system relative to one or more performance traits, as listed in column217. For example, the first row of benchmark trait assignment data structure215indicates that the efficiency of a “connect” action may be primarily dependent on input-output (I/O) latency. Similarly, the second row indicates that the efficiency of a “read” action may be primarily dependent on available bandwidth to perform the “read” action.

Entries in benchmark column218indicate a performance benchmark best-suited to measure the efficiency of a node of a distributed system, relative to the corresponding entry in performance trait column217. For example, as shown in the first row, to measure the efficiency of I/O latency, the best benchmark is “Benchmark1.” The other rows show similar data values for other program actions that may be performed by the subroutines214of computing job205.

By determining which benchmarks most accurately correspond to the program action of subroutines214, a benchmark profile may be created that may be used to predict the performance of computing job205when run a particular distributed system. In one embodiment, the benchmark profile may specify which benchmarks are most representative of job205, and further may specify different proportions for the benchmarks included in the benchmark profile. Given the example data structure inFIG. 2C, for a program in which 50% of the processing activity is “reads”, 25% of the activity is “stores”, and 25% of the activity is “connects,” a benchmark profile could include benchmark2, benchmark3, and benchmark1, with a contribution for each overall benchmark weighted respectively at 50/25/25. The benchmarks may then be executed on the computing nodes of a distributed system in a variety of different ways to predict the performance of the application represented by the benchmark profile, without having to prepare, load, and execute the actual application. Thus, the preferred nodes for executing each of the subroutines214of a job205on a distributed cluster (e.g., cluster100ofFIG. 1) may be determined quickly and efficiently using the benchmark profile and associated benchmarks as a proxy for job205.

FIG. 3is a flowchart illustrating a method300for generating a benchmark profile for an application configured to run on a distributed system, according to one embodiment of the invention. Before a user submits a job for processing on a distributed system, benchmark profiling may aid in determining a preferred distribution of the subroutines214of the job205on the nodes of a distributed system.

As shown, the method300begins at step302, where the profiling component evaluates the performance characteristics of each subroutine214included in a computing job205. For example, the characteristics of computing job205may be categorized with respect to processor utilization, memory utilization, I/O requirements, bandwidth, latency, and others. In one embodiment, the ratio of each of these characteristics, with respect to the job as a whole, is determined for each subroutine214and saved within the benchmark profile for computing job205. The overall performance characteristics of the computing job205may be determined from the aggregate performance characteristics of the subroutines.

At step304, the profiling component may compare the characteristics of computing job determined at step302with a collection of available benchmark applications. The benchmarks may be known benchmarks or synthetic benchmarks. For example, known benchmarks such as Linpack, Dhrystone, Whetstone, GliBench. Of course, custom made benchmarks or other known benchmarks may be used. In one embodiment, a benchmark is selected to represent each performance characteristic of the job, or the subroutines of the job, determined at step302. The composition of benchmarks appropriate for computing job205is stored in the benchmark profile.

At step306, a loop begins that includes steps308and310. For each pass through the loop, the profiling component evaluates the relative performance requirements of the computing job for one of the benchmarks to determine a scaling unit. In one embodiment, this scaling unit matches the ratio of computing activity of each performance characteristic, relative to the activity of the computing job as a whole (i.e., the activities and ratios identified at step302). Thus, the scaling unit specifies how important the performance of each benchmark identified at step304is to the benchmark profile being generated. At step308, the profiling component225determines the scaling unit for one of the benchmarks determined at step304. At step310, the profiling component225stores the scaling unit in the benchmark profile. Once a scaling unit is determined for each of the benchmarks in the composition of benchmarks, method300concludes. Additionally, in one embodiment, a user may manually define the scaling unit for performance traits.

For example, assume that performance characteristics for subroutines214of a particular computing job indicate that a program tends to perform I/O actions and memory accesses. Further, as a simplified example, assume that such a computing job includes two subroutines, one that performs six activities for I/O, and another that performs four program for memory access, in such a case, the scaling units for benchmarks used to profile I/O latency and available memory access may be 60% and 40%, respectively. Thus, when generating a benchmark profile for this computing job, method300would specify that 60% of benchmark processing time should be committed to a benchmark that evaluates I/O latency, and 40% should be committed to a benchmark that evaluates available memory. Alternatively, the benchmark evaluates that I/O latency may be dispatched to one node of a parallel system, and the benchmark that evaluates memory access may be dispatched to another. In such a case, the performance of the application on these two compute nodes may be predicted using the results of the benchmarks. Further, a composite score of application performance for a computing job may be calculated from 60% of the benchmark score for I/O latency and 40% of the benchmark score for memory access times. Of course, one of skill in the art may utilize the data generated for a computing job using a benchmark profile in a variety of ways.

FIG. 4illustrates an example benchmark profile data structure430, according to one embodiment of the invention. As shown, benchmark profile data structure430is defined as a table that includes a subroutine ID column431, a performance trait column432, a benchmark column433a ratio column434, and a scaling unit column435.

Each entry in benchmark profile data structure430identifies, for a subroutine identified in subroutine column431and a performance characteristic of that subroutine, a corresponding benchmark, the ratio of programming activity for the subroutine relative to the programming activity for the whole job, and the scaling unit for the performance characteristic relative to the programming activity for the whole subroutine. For example, the first row in table430indicates a subroutine ID of “Sub 1,” a performance trait for this subroutine of “I/O latency,” a benchmark corresponding to “I/O latency” of “Benchmark1,” a ratio of “3:10,” and a scaling unit of “60.” The other entries in benchmark profile data structure430provide similar information for other subroutines and performance activity of computing job205. The scaling ratio may be derived by comparing the number of statements related to a particular performance characteristic, to the total number of statements in the subroutine.

Once a benchmark profile is generated to represent the performance characteristics of a given computing job, the benchmark profile may be used to predict the performance of the application by running the benchmarks specified in the benchmark profile on a given configuration of a distributed system. In one embodiment, the user may invoke the profile execution component to predict or test performance of particular job on a particular system configuration. In turn, the profile execution component accesses the benchmark profile associated with the computing job and executes the benchmarks in the profile across the nodes of a distributed system nodes, with each benchmark running for a time proportional to the scaling unit (i.e. composition ratio) specified in the benchmark profile. The application then measures and saves the results along with the nodes that the benchmarks were run against. The profiles can then be saved, reused, and compared against different versions of the job or against different system configurations.

The benchmark profiles may be based on a combination of source code analysis, initial measured runs, and user input. Further, the predicted performance data may be compared against actual execution data. Still further, the accuracy of a benchmark profile may be improved over time by analysis of the execution performance of the computing job. The actual execution performance the job may then be compared against the benchmark profile to test how well the job matches the benchmarks in the profile. That is, actual execution performance may be used to evaluate the how well the benchmark profile accurately represents the computing activity of the application. If significant differences are found between the actual execution performance and one of more of the benchmarks in the profile, the profile can be updated automatically and/or the user can be notified of the differences.