Methods and apparatus to benchmark a computer system based on executing instructions using different numbers of threads

Example methods, apparatus and articles of manufacture to benchmark hardware and software are disclosed. A disclosed example method includes initiating a first thread to execute a set of instructions on a processor, initiating a second thread to execute the set of instructions on the processor, determining a first duration for the execution of the first thread, determining a second duration for the execution of the second thread, and determining a thread fairness value for the computer system based on the first duration and the second duration.

FIELD OF THE DISCLOSURE

This disclosure relates generally to computer systems and, more particularly, to methods and apparatus to benchmark software and hardware.

BACKGROUND

When selecting a computer system, operating system, and/or environment for implementation in a computing environment (e.g., a business), benchmarks are used to select among the available options. Benchmark data can show how well suited a computer system, operating system, and/or environment is for an intended use without the need to install and operate business software for an extended period of time to analyze performance. For example, benchmark data may be used to select among various computer architectures, various virtual machine environments (e.g., JAVA virtual machine environments), various operating system implementations, etc. Benchmark data is often generated by executing a set of instructions on each of the computer system configurations to be analyzed and monitoring the execution. For example, the amount of time needed to execute the set of instructions on each of the computer system configurations may be recorded and compared to determine which computer system configuration is best at executing the instructions.

DETAILED DESCRIPTION

As disclosed herein, an example benchmarking system includes a client and server operating on a computer system. The client receives user input from a user indicative of a number of threads that should be used during benchmarking. The client transmits the number of threads and a workload to the server. The server initiates the number of threads specified by the user. The threads are instructed to each execute instructions included in the workload on a set of data included in the workload. The client tracks the execution duration of each of the threads as well as garbage collection operations associated with the client and the server. The client then generates benchmarking reports. Example reports include a thread fairness value indicative of the amount of variance in thread execution times. A system with high thread fairness (indicated by a high or low fairness value depending on the particular implementation) will generally execute the same set of instructions in substantially or approximately the same amount of time for each thread. While a system with low thread fairness may execute the same set of instructions in different amounts of time in different threads. Accordingly, a system with a high thread fairness benchmark can be selected where it is desirable for threads to execute in the same amount of time. The example reports may also include a garbage collection benchmark that is indicative of the impact of garbage collection processes (e.g., garbage collection performed by a virtual machine) on the execution of threads. While various benchmarks are described herein, a complete benchmark report may include any number and/or type of benchmarks, including known benchmarks not disclosed herein.

Example methods and apparatus to benchmark software and hardware are disclosed. A disclosed example method includes initiating a first thread to execute a set of instructions on a processor, initiating a second thread to execute the set of instructions on the processor, determining a first duration for the execution of the first thread, determining a second duration for the execution of the second thread, and comparing the first duration and the second duration to determine a thread fairness value.

FIG. 1illustrates an example benchmarking system100that may operate on a computer system to benchmark hardware and software of the operating system. The example benchmarking system100includes a client102that communicates with a workload datastore104and a results datastore106. The example benchmarking system100also includes a server108that is communicatively coupled to the client102. The example system100may, for example, benchmark the operation of a computer programming environment on a computer system. For example, the benchmarking system100may evaluate the performance of a JAVA environment on a computer system.

In general, the client102instructs the server108to execute instructions that are monitored. The server108transmits information about the execution of the instructions to the client102, which generates benchmarking reports from the information. According to the illustrated example, the client102and the server108operate on the same computer system to ensure that the benchmarking information is not influenced by other computer systems or network communications. Alternatively, where a group or network of computer systems is to be monitored, the client102and the server108may be on separate computer systems that are communicatively coupled.

The client102of the illustrated example receives user input from a user directing the operation of the benchmarking system100. For example, the client102may include a user interface that enables a user to input a number of concurrent threads that should be executed by the server108. In addition, the user interface may allow a user to input or specify instructions and/or data that may be used as a workload for the server108to process during the benchmarking procedure. The client102processes the workload for communication and transmits the workload to the server108.

The example client102further generates reports based on benchmarking results received from the server108. For example, the client102may monitor the execution duration of each of multiple threads initiated by the server108based on information receive from the server108and generate reports indicative of the performance of the computer system on which the client102and the server108are operating.

The client102is communicatively coupled to the workload datastore104to retrieve and/or store instructions and/or data that may be transmitted to the server108for use during the benchmarking process. For example, the client102may retrieve a set of instructions and a data set from the workload datastore104and transmit the instructions and the data set to the server108. The server108may execute the received instructions on the received data set and send information about the execution to the client102. The workload datastore104may be implemented by any type(s) of data structure(s) or device(s) for storing the instructions and the data set. Alternatively, the workload datastore104may not be included in the benchmarking system100when the instructions and/or the data set are received from an external source (e.g., from user input).

The client102of the illustrated example is also communicatively coupled to the results datastore106. The results datastore106stores benchmarking results received from the server108. The results datastore106may additionally store intermediate data generated from the benchmarking results that is used in generating reports. The results datastore106may be implemented by any type(s) of data structure(s) and/or device(s).

While the workload datastore104and the results datastore108are illustrated as being separate from the client102, one or both of the workload datastore104and the results datastore108may be included in and/or integrated with the client102.

FIG. 2is a block diagram of an example implementation of the client102ofFIG. 1. The example client102includes a user interface202, a data serializer204, a server interface206, a report generator208, a timer210, a garbage collection monitor212, and a result outputter214.

The user interface202of the illustrated example enables a user to provide user input to the client102. The user interface202may be implemented by any type of user interface. For example, the user interface202may be a graphical user interface, a physical user interface, a command line interface, etc. Alternatively, user input could be received in any other manner. For example, users could provide their input by editing a settings file stored on the client102. According to the illustrated example, the user input is an indication of a number of threads that should be used during the benchmarking process. The number of threads could be an indication of a single number of threads, a fixed plurality of threads, or a range of numbers of threads that should be iteratively used during benchmarking. The user interface202transmits the user input to the data serializer204.

The data serializer204of the illustrated example receives the user input from the user interface202and receives and/or retrieves workload data from the workload datastore104. The data serializer204converts the user input and the workload data to a format that can be transmitted over a communication connection (e.g., over a network). For example, the data serializer204may convert object data into a sequence of bits for network transmission. The data serializer204may perform any type of serializing, data marshalling, data deflating, etc. The output of the data serializer204is transmitted to the server interface206. By way of example, the data serializer204may be implemented by the JAVA remote method invocation (RMI) serialization process.

The example server interface206is communicatively coupled with a server (e.g., the server108ofFIG. 1). According to the illustrated example, the client102and the server108are operated on a single computer system and, thus, the server interface206is communicatively coupled to the server108internal to the computer system. Alternatively, the server interface206could be communicatively coupled to the server108via any type of connection such as, for example, a network connection between two computer systems, which may be, for example, in different geographic locations. The server interface206transmits or otherwise conveys the data received from the data serializer204to the server108. The example server interface206receives data indicative of the operation of the server108from the server108. For example, the data transmitted to the server108may be user input, instructions, and a workload data set that have been serialized, wherein the server108is to initiate a number of threads indicated by the user input to each execute the instructions on the workload data set. Accordingly, the server interface206will then receive an indication from the server108when each of the threads completes execution.

When data received from the server108has been serialized, the example server interface206performs de-serialization of data received from the server108. The received data is transmitted to the report generator208and/or stored in a results datastore (e.g., the results datastore106).

The example report generator208receives data from the server interface206and/or retrieves data from the results datastore106to generate benchmarking reports for the hardware and/or software of the computer system(s) that are under analysis. For example, the report generator208may generate a fairness indication that indicates the deviation in the execution time of multiple threads. To determine the execution time for each thread executed by the server108, the report generator208is communicatively coupled with the timer210. The timer210is started when user interface202receives an instruction from a user to begin benchmarking the computer system. The report generator208retrieves the current time of the timer210when the report generator208receives via the server interface206an indication that a thread has completed execution at the server108. Alternatively, the timer210could be started at any time. For example, when it is not desired to capture the time required for serializing in the benchmark, the timer could be started after receiving a notification from the server108that thread execution has begun or when the instructions have been sent to the server108. As described in conjunction withFIG. 6, the report generator208processes and compares the execution time of each of the threads to determine the fairness indication.

The report generator208may also generate a garbage collection report based on data received via the server interface206and from the garbage collection monitor212that indicates how the garbage collection process(es) of the computer system affect the operation of the computer system's hardware and software. The report generator208may generate any type of report. For example, the report may include graphs, raw results, computed statistics, etc. The generated reports are sent to the result outputter214.

The garbage collection monitor212of the illustrated example, monitors garbage collection process(es) of the client102. For example, the garbage collection monitor212may instruct the operating system or system software (e.g., a JAVA virtual machine environment) of the computer system to report garbage collection information to the garbage collection monitor212. For example, the garbage collection monitor212may receive an indication of the duration of each garbage collection process. Alternatively, the garbage collection monitor212may receive an indication at the start and stop of each garbage collection process and may utilize the timer210to determine the duration of the garbage collection. The garbage collection monitor212may be communicatively coupled with another garbage collection monitor (e.g., a garbage collection monitor at the server108) via the server interface106to enable the garbage collection monitor to obtain information about other garbage collection processes. The garbage collection information is transmitted to the report generator208for analysis.

The result outputter214of the illustrated example receives report information from the report generator208and outputs the results. For example, the result outputter214may output the results to a display, a printer, a file, etc. The result outputter214may additionally or alternatively provide the results to another system or application. For example, the result outputter214may output the results to a program for computations or statistical analysis.

FIG. 3is a block diagram of an example implementation of the server108ofFIG. 1. The example server108includes a client interface302, a thread initiator304, a thread tracker306, a garbage collection monitor308, and a results serializer310.

The client interface302of the illustrated example receives user input and workload data from the client102and transmits thread completion and garbage collection information to the client102. The client interface302transmits received user input and workload data to the thread initiator304.

The thread initiator304of the illustrated example receives the user input and the workload data from the client interface302. The example thread initiator304determines a number of threads to be initiated based on the user input received via the client interface302. For example, the thread initiator304may determine that the user has requested that eight threads be initiated. Alternatively, the thread initiator304may determine that the user has requested that multiple benchmark processes should be run with a variety of numbers of threads (e.g., 2 threads, 4 threads, 8 threads, 16, threads, and 32 threads). Based on the user input, the thread initiator304initiates the number of desired threads to perform instructions included in the workload data that operate on a workload included in the workload data. For example, the thread initiator304may instruct an operating system, virtual machine (e.g., the JAVA virtual machine), processor, or any other system to execute the instructions. The instructions may perform functions that are likely to be performed on the computer system when the computer system is deployed (e.g., in a business). For example, the instructions may perform functions that would be executed by accounting software and the workload data may be sample accounting data. Accordingly, the operation of the computer system can be tested using instructions that simulate the intended use of the computer system. According to the illustrated example, all threads are initialized to execute simultaneously. However, threads could alternatively be initiated one at a time or in groups depending on the analysis desired.

The example thread tracker306monitors the operation of the threads initiated by the thread initiator304and transmits data regarding the threads to the results serializer310. The example thread tracker306sends an indication to the results serializer310when each of the threads completes execution of the workload instructions. Alternatively, the thread tracker306may include or be in communication with a timer to enable the thread tracker306to determine timestamps for and transmit the execution duration of each of the threads. In such an implementation, for example, the timer210ofFIG. 2may, but need not, be eliminated. The thread tracker306may additionally monitor the number of operations performed by each of the threads, the duration of the execution of each of the threads, etc.

The garbage collection monitor308of the illustrated example, monitors garbage collection processes of the server108. For example, the garbage collection monitor308may instruct the operating system or system software (e.g., a JAVA virtual machine environment) of the computer system to report garbage collection information to the garbage collection monitor308. For example, the garbage collection monitor308may receive an indication of the duration of each garbage collection process. Alternatively, the garbage collection monitor308may receive an indication at the start and stop of each garbage collection process and may determine the duration of the garbage collection. The garbage collection information is transmitted to the results serializer310for transmission to the client102.

The results serializer310of the illustrated example receives results from the thread tracker306and the garbage collection monitor308and converts them to a format that can be transmitted over a communication connection (e.g., over a network). For example, the results serializer310may convert object data into a sequence of bits for network transmission. The results serializer310may perform any type(s) of serializing, data marshalling, data deflating, etc. The output of the results serializer310is transmitted to the client102via the client interface302. By way of example, the results serializer310may be implemented by the JAVA remote method invocation (RMI) serialization process.

WhileFIGS. 2 and 3illustrate that certain components are included in the client102and the server108, more or fewer components may be included in the client102and/or the server108. For example, any of the components of the client102could be included in the server108and any components of the server108could be included in the client102. For example, in an example implementation the timer210, report generator208, and result outputter214could be included in the server so that they server could timestamp thread executions and report the results.

While an example benchmarking system100has been illustrated inFIGS. 1-3, the elements illustrated inFIGS. 1-3may be combined, divided, re-arranged, eliminated and/or implemented in any way. For example, the benchmarking system100may include more than one client102and/or more than one server108. Further, the example client102, server108, workload datastore104, and results datastore106ofFIG. 1, the user interface202, the data serializer204, the server interface206, the report generator208, the timer210, the garbage collection monitor212, and the result outputter214ofFIG. 2, and the client interface302, the thread initiator304, the thread stacker306, the results serializer310, and/or the garbage collection monitor312may be implemented by hardware, software, firmware and/or any combination of hardware, software and/or firmware. Thus, for example, any of the example client102, server108, workload datastore104, and results datastore106ofFIG. 1, the user interface202, the data serializer204, the server interface206, the report generator208, the timer210, the garbage collection monitor212, and the result outputter214ofFIG. 2, and the client interface302, the thread initiator304, the thread stacker306, the results serializer310, and/or the garbage collection monitor312may be implemented by one or more circuit(s), programmable processor(s), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)), etc. When any of the appended apparatus claims are read to cover a purely software and/or firmware implementation, at least one of the example client102, server108, workload datastore104, and results datastore106ofFIG. 1, the user interface202, the data serializer204, the server interface206, the report generator208, the timer210, the garbage collection monitor212, and the result outputter214ofFIG. 2, and the client interface302, the thread initiator304, the thread stacker306, the results serializer310, and/or the garbage collection monitor312are hereby expressly defined to include a tangible medium such as a memory, a digital versatile disc (DVD), a compact disc (CD), etc. storing the software and/or firmware. Further still, the example benchmarking system100may include additional devices, servers, systems, networks and/or processors in addition to, or instead of, those illustrated inFIGS. 1-3, and/or may include more than one of any or all of the illustrated devices, servers, networks, systems and/or processors.

FIG. 4is a flowchart representative of example machine readable instructions that may be executed to implement the client102ofFIG. 1.FIG. 5is a flowchart representative of example machine readable instructions that may be executed to implement the server108ofFIG. 1.FIG. 6is a flowchart representative of example machine readable instructions that may be executed by the client102to analyze benchmarking information from the server108. The example processes ofFIGS. 4-7may be carried out by a processor, a controller and/or any other suitable processing device. For example, the processes ofFIGS. 4-7may be embodied in coded instructions stored on any article of manufacture, such as any tangible computer-readable medium. Example tangible computer-readable medium include, but are not limited to, a flash memory, a CD, a DVD, a floppy disk, a read-only memory (ROM), a random-access memory (RAM), a programmable ROM (PROM), an electronically-programmable ROM (EPROM), and/or an electronically-erasable PROM (EEPROM), an optical storage disk, an optical storage device, magnetic storage disk, a magnetic storage device, and/or any other medium which can be used to carry or store program code and/or instructions in the form of machine-accessible instructions or data structures, and which can be electronically accessed by a processor, a general-purpose or special-purpose computer, or other machine with a processor (e.g., the example processor platform P100discussed below in connection withFIG. 10). Combinations of the above are also included within the scope of computer-readable media. Machine-accessible instructions comprise, for example, instructions and/or data that cause a processor, a general-purpose computer, special-purpose computer, or a special-purpose processing machine to implement one or more particular processes. Alternatively, some or all of the example processes ofFIGS. 4-7may be implemented using any combination(s) of ASIC(s), PLD(s), FPLD(s), discrete logic, hardware, firmware, etc. Also, some or all of the example processes ofFIGS. 4-7may instead be implemented manually or as any combination of any of the foregoing techniques, for example, any combination of firmware, software, discrete logic and/or hardware. Further, many other methods of implementing the example operations ofFIGS. 4-7may be employed. For example, the order of execution of the blocks may be changed, and/or one or more of the blocks described may be changed, eliminated, sub-divided, or combined. Additionally, any or all of the example processes ofFIGS. 4-7may be carried out sequentially and/or carried out in parallel by, for example, separate processing threads, processors, devices, discrete logic, circuits, etc.

The example process ofFIG. 4begins when the user interface202of the example client102receives a user input indicating a number of threads to be used during benchmarking (block402). The garbage collection monitor212is then initiated to begin monitoring garbage collection process(es) (block404). The data serializer204then retrieves workload instructions and workload data from the workload datastore104(block406). The example data serializer204serializes the received user input, the workload instructions, and the workload data (block408). The serialized data is then transmitted to the server108via the server interface206(block410).

After transmitting the serialized data to the server108(block410), the server interface206determines if results indicating that a thread has completed execution have been received from the server108(block412). When no results have been received from the server108, control remains at block412to await the return of results. When a result has been received from the server, the report generator208retrieves the current value of the timer (block414). The report generator208stores the timer value in the results datastore106(block416). Then, the report generator208determines if there are additional results (e.g., threads are still executing) (block418). When there are additional results to be received, control returns to block412to await the additional results.

When there are no additional results to be received (block418), the report generator208retrieves the results of the garbage collection monitor212and/or the garbage collection monitor308(block420). The report generator208then generates a report by comparing the received thread execution information and/or the garbage collection information (block422). For example, the report may be generated as described in conjunction withFIG. 6. Once the report has been generated, the result outputter214outputs the results (block424). The process ofFIG. 4then ends. The client102may, for example, wait for the next request from the user.

FIG. 5is a flowchart representative of an example process that may be carried out to implement the server108ofFIG. 1. The process ofFIG. 5begins when the client interface302of the server108receives an indication of a number of threads to be executed from the client (block502). For example, the indication may be serialized data received from the client and the client interface302may de-serialize the data to access the user input. The garbage collection monitor212is then initialized to start monitoring garbage collection processes at the server108(block504). The client interface302then receives a workload (block506). For example, the workload may be a set of instructions to be operated on a data set. The workload may be serialized and the client interface302may de-serialize the data to access the workload data.

Using the received indication of the number of threads to be executed, the thread initiator304causes the initiation of the indicated number of threads to perform the instructions included in the workload data on the data set included in the workload data (block508). For example, the thread initiator304may initiate four threads when the user input indicates a request for four threads. Alternatively, the thread initiator304could iteratively initiate groups of threads based on a number of threads input that requests that a range of threads should be processed to analyze the operation of the computer system with each of the indicated numbers of threads. For example, the number of threads may be indicated to be 2, 4, 8, 16, 32, 64, and 128 to test the system with each of the number of threads.

After the threads have been initiated (block508), the thread tracker306determines if any of the threads have completed (block510). When none of the threads have completed, control remains at block510to continue waiting. When a thread has completed execution, the results serializer310serializes the thread identifier and sends the thread identifier to the client102via the client interface302(block512). The server108may send any type of indication to the client102that a thread has completed execution so that the client102can timestamp the execution of the thread. The thread tracker306then determines if there are additional threads executing (block514). When there are additional threads executing, control returns to block510to wait for the next thread to complete execution.

When there are no additional threads executing (block514), the results serializer310serializes the garbage collection information collected by the garbage collection monitor308and transmits the garbage collection information to the client102via the client interface302(block516). The process ofFIG. 5then ends. The server108may, for example, wait for the next request from the client102.

FIG. 6is a flowchart representative of an example process that may be performed by the client102to analyze benchmarking information from the server108. For example, the process ofFIG. 6may be performed by the report generator208to generate reports during block416ofFIG. 4. The example process begins by determining a total running time for each thread that was executed by the server108(block602). For example, the report generator208may determine the total running time for each thread by subtracting the timer value that was recorded when the thread execution was completed (e.g., in block414) from an initial timer value (e.g., 0 or an initial value for the timer that was recorded in the results datastore106). According to the illustrated example, the initial timer value is stored when the benchmarking process begins. Alternatively, an initial timer value could be stored for each thread when, for example, all of the threads are not started simultaneously.

The report generator208then compares the total running time for each thread by determining the average of the total running time for each thread (block604). For example, the report generator208may determine the sum of the total running times for all threads and then divide the sum by the number of threads. Any desired computed value may be used such as, for example, the average, the mean, the median, etc. The report generator208then subtracts the total running time for each thread (e.g., the total time determined in block602) from the average value (e.g., determined in block604) to determine a variance value for each thread (block606). Alternatively, any other method for determining a variance value for each thread may be used. The report generator208then squares each variance to ensure that only positive values are available (block608). Alternatively, any other method for ensuring that all positive values are available may be used such as, for example, using an absolute value. The squared variance of each of the threads is then summed to determine a total squared variance value (block610). The report generator208then computes the square root of the total squared variance value to determine a square root value (block612). The square root value is then divided by the number of threads to determine a standard deviation (block614). The standard deviation is output as the fairness indication for the system. According to the illustrated example, as the fairness value approaches zero the computer system that was analyzed all threads in approximately the same amount of time. As the fairness value moves away from zero, the computer system executes different threads executing the same instructions in different amounts of time. For example, a first thread may be executed in five seconds while a second thread performing the same instructions on the same workload may be performed in ten seconds. Accordingly, a computer system with a fairness value close to zero may be labeled as having a high fairness while systems with greater fairness values are labeled as having lower fairness.

While an example process for determining a fairness value is illustrated inFIG. 6, any other process may be used. For example, any process for determining a standard deviation of a group of thread execution times may be used.

FIG. 7is a flowchart representative of an example process that may be carried out to analyze garbage collection information at the client102. The process ofFIG. 7begins when the report generator208of the client102receives garbage collection information (block702). For example, the garbage collection information may be received from the garbage collection monitor212and/or the garbage collection monitor308. In the illustrated example, the garbage collection information comprises indications of the duration of each garbage collection process that was performed at the client102and/or the server108. The report generator208then sums the garbage collection durations (block704). The report generator208then divides the sum by the number of threads to determine a garbage collection benchmark value (block706). The benchmark value is indicative of the amount of garbage collection time per executed thread. Accordingly, the benchmark value can be compared with other benchmark values that were determined using a different number of threads.

FIG. 8illustrates an example report generated using the benchmark system100to show garbage collection impact on computer systems. The example report illustrates the garbage collection impact on five different computer systems for different numbers of thread executions. The data for the report ofFIG. 8may be generated by iteratively performing the processes represented byFIGS. 4-5. As shown in the illustrated example, as the number of threads increases the garbage collection impact per thread increases as well.

FIG. 9illustrates an example report generated using the benchmark system100to show fairness of computer systems. The example report illustrates the fairness of five different computer systems for different numbers of thread executions. The data for the report ofFIG. 9may be generated by iteratively performing the processes represented byFIGS. 4-5. As shown in the illustrated example, as the number of threads increases the fairness value generally increases as well. In other words, the thread fairness is generally greatest when fewer threads are analyzed.

FIG. 10is a schematic diagram of an example processor platform P100that may be used and/or programmed to execute the instructions ofFIGS. 4-7to implement the example client102and/or the example server108ofFIGS. 1-3. For example, the processor platform P100can be implemented by one or more general-purpose processors, processor cores, microcontrollers, etc.

The processor platform P100of the example ofFIG. 10includes at least one general purpose programmable processor P105. The processor P105executes coded and/or machine-accessible instructions P110and/or P112stored in main memory of the processor P105(e.g., within a RAM P115and/or a ROM P120). The processor P105may be any type of processing unit, such as a processor core, a processor and/or a microcontroller. The processor P105may execute, among other things, the example processes ofFIGS. 4-6to implement the example methods, apparatus and articles of manufacture described herein.

The processor P105is in communication with the main memory (including a ROM P120and/or the RAM P115) via a bus P125. The RAM P115may be implemented by DRAM, SDRAM, and/or any other type of RAM device, and ROM may be implemented by flash memory and/or any other desired type of memory device. Access to the memory P115and the memory P120may be controlled by a memory controller (not shown).

The processor platform P100also includes an interface circuit P125. The interface circuit P125may be implemented by any type of interface standard, such as an external memory interface, serial port, general-purpose input/output, etc. One or more input devices P130and one or more output devices P130are connected to the interface circuit P125.