Workload performance projection via surrogate program analysis for future information handling systems

A performance projection system includes a test IHS and multiple currently existing IHSs. The performance projection system includes user application software and surrogate programs that execute on currently existing IHSs. The performance projection system measures user application software and surrogate program performance during execution on currently existing IHSs. The performance projection systems measures runtime program performance during execution of surrogate programs on a future semiconductor die IC design model or virtualized future system. Designers normalize and compare surrogate program runtime performance data with user application software performance data. Designers un-normalize the normalized runtime performance data to generate a projection of runtime performance on the future system.

CROSS REFERENCE TO RELATED PATENT APPLICATIONS

This patent application relates to the U.S. Patent Application entitled “WORKLOAD PERFORMANCE PROJECTION FOR FUTURE INFORMATION HANDLING SYSTEMS USING MICROARCHITECTURE DEPENDENT DATA”, inventors Bell, et al. U.S. Ser. No. 12/343,482, the disclosure of which is incorporated herein by reference in its entirety.

BACKGROUND

The disclosures herein relate generally to information handling systems (IHSs), and more specifically, to workload projection methods that IHSs employ.

Customers, designers and other entities may desire to know how their software applications, or workloads, will perform on future IHSs before actual fabrication of the future IHSs. Benchmark programs provide one way to assist in the prediction of the performance of a workload of a future IHS. However, aggregated performance over many benchmarks may result in errors in performance projections for individual software applications on a future IHS. An IHS may operate as an electronic design test system to develop workload performance projections for new processors and other new devices in future IHSs.

SUMMARY

In one embodiment, a method of performance testing is disclosed. The method includes providing a user software program and first and second surrogate software programs. The method also includes executing the user software program on multiple existing information handling systems (IHSs). The method further includes storing runtime data for the user software program as it executes on the multiple existing IHSs. The method still further includes executing the first and second surrogate software programs on the multiple existing IHSs and on a future virtualized IHS. The method also includes storing runtime data for the first surrogate software program as the first surrogate software program executes on the multiple existing IHSs and the future virtualized IHS. The method further includes storing runtime data for the second surrogate software program as the second surrogate program executes on the multiple existing IHSs and the future virtualized IHS. The method also includes normalizing the runtime data for the user software program and the first and second surrogate software programs with respect to runtime data of a particular existing IHS of the multiple existing IHSs, thus providing normalized runtime data. The method further includes comparing the normalized runtime data for the first and second surrogate software programs with respect to the normalized runtime data of the user software program to determine a best fit surrogate software program. The method still further includes selecting the normalized runtime data of the best fit surrogate software program executing on the future virtualized IHS as representing projected runtime data for the user software application.

In another embodiment, a performance projection system is disclosed that includes multiple currently existing information handling systems (IHSs). The performance projection system also includes a test information handling system (IHS). The test IHS includes a processor and a memory coupled to the processor. The memory stores a future virtualized IHS. The performance projection system also includes a user application program that executes on the multiple IHSs. The performance projection system further includes first and second surrogate programs that execute on the multiple IHSs and the future virtualized IHS. The test IHS is configured to store runtime data for the first surrogate software program as the first surrogate software program executes on the multiple existing IHSs and the future virtualized IHS. The test system, also referred to as a performance projection system, is also configured to store runtime data for the second surrogate software program as the second surrogate program executes on the multiple existing IHSs and the future virtualized IHS. The test system is further configured to normalize the runtime data for the user software program and the first and second surrogate software programs with respect to runtime data of a particular existing IHS of the multiple existing IHSs, thus providing normalized runtime data. The test system is also configured to compare the normalized runtime data for the first and second surrogate software programs with respect to the normalized runtime data of the user software program to determine a best fit surrogate software program. The test system is further configured to select the normalized runtime data of the best fit surrogate software program executing on the future virtualized IHS as representing projected runtime data for the user software application.

DETAILED DESCRIPTION

In one embodiment, a performance projection system provides workload performance projection capability for IC designs or hardware (HW) designs under test. These hardware designs may themselves be information handling systems (IHSs). Designers execute application software, such as user application software, as a workload on multiple existing HW designs or existing systems (IHSs). Designers also execute multiple surrogate programs on the multiple existing systems. Surrogate programs include programs that exercise a HW system's functionality, such as benchmark programs for example. Designers or other entities may select surrogate programs that exhibit performance characteristics similar to those of the user application software.

Runtime data, or the amount of time that the application software and each of multiple surrogate programs takes to complete execution, provides a basis for comparison among the multiple existing HW systems or existing IHSs. In a simulation environment, each of the multiple surrogate programs executes on a virtualized future HW design model or future IHS, i.e. a future system. The projected surrogate program runtime data on the virtualized future system enables a comparison with respect to multiple existing systems. That particular comparison may provide for a normalization of data between surrogate program runtime performance on existing systems and that of the virtualized future system. The normalization data provides a way to predict the runtime performance of the application software, or workload, on the future system.

In another embodiment, a performance projection system provides microarchitecture dependent workload performance projection capability for a future hardware (HW) design model or virtualized future IHS under test. Designers or other entities select an existing hardware HW design or existing IHS that most closely resembles the hardware functionality or other criteria of the virtualized future system or future IHS. The virtualized future IHS executes on a test IHS within the performance projection system. Designers execute benchmark software such as user application software on the selected existing IHS. During user application execution, the test IHS records runtime and other hardware counter data. Hardware counter data includes microarchitecture dependent information. Designers select surrogate programs that exhibit similar performance characteristics to those of the user application software. Surrogate programs include programs that exercise an existing IHS's functionality, such as benchmark programs for example. Runtime data, or the amount of time that the application software and each of multiple surrogate programs takes to complete execution, provides a basis for comparison among the multiple existing IHSs. In a simulation environment, each of the multiple surrogate programs runs on a particular future HW design model or virtualized future IHS, i.e. a future system.

Designers or other entities execute the surrogate programs on the selected existing IHS and the virtualized future IHS, collecting runtime and HW counter performance data during execution. A normalization of that performance data, including runtime and HW counter data, allows designers and other entities to select a surrogate program that most closely fits the performance characteristics similar to those of the user application software. Designers and other entities use microarchitecture dependent information as selection criteria to determine the closest fit surrogate program for the user application software performance. Using a scaling process, the surrogate program runtime results provide an offset to generate a performance projection of user application software runtime performance on the future system.

FIG. 1depicts a performance projection system100, that integrated circuit (IC) designers and other entities may employ as a benchmarking tool for existing or new IC designs. Performance projection system100includes a test IHS102having a processor105that includes a hardware (HW) counter107and an L1 cache109. Processor105couples to a bus110. A memory controller115couples a system memory125to bus110via a memory bus120. A video graphics controller130couples a display135to bus110. Test IHS102includes nonvolatile storage140, such as a hard disk drive, CD drive, DVD drive, or other nonvolatile storage that couples to bus110to provide test system100with permanent storage of information. System memory125and nonvolatile storage140are each a form of data store. I/O devices150, such as a keyboard and a mouse pointing device, couple via I/O bus155and an I/O controller160to bus110. Processor105, system memory125and devices coupled to bus110together form test IHS102within performance projection system100.

One or more expansion busses165, such as USB, IEEE 1394 bus, ATA, SATA, PCI, PCIE and other busses, couple to bus110to facilitate the connection of peripherals and devices to test system100. A network interface168couples to bus110to enable test IHS102to connect by wire or wirelessly to other network devices. Test IHS102may take many forms. For example, this IHS may take the form of a desktop, server, portable, laptop, notebook, or other form factor computer or data processing system. Test IHS102may also take other form factors such as a personal digital assistant (PDA), a gaming device, a portable telephone device, a communication device or other devices that include a processor and memory. Test system100includes benchmark software, or other software such as SURROGATE PROGRAM1, and SURROGATE PROGRAM2. Test system100includes existing hardware IHSs, such as an EXISTING IHS A, an EXISTING IHS B, an EXISTING IHS C, and an EXISTING IHS D.

A user or other entity installs software such as FUTURE SYSTEM170in non-volatile storage140of test IHS102prior to conducting testing with APPLICATION SOFTWARE175. APPLICATION SOFTWARE175may be user application software for which it is desirable to determine performance on a FUTURE SYSTEM170. WhileFIG. 1shows APPLICATION SOFTWARE175as installed APPLICATION SOFTWARE175′ within nonvolatile storage140and as APPLICATION SOFTWARE175″ in memory125, performance projection system100may execute APPLICATION SOFTWARE175on multiple existing IHSs, namely an EXISTING IHS A, an EXISTING IHS B, an EXISTING IHS C, and an EXISTING IHS D, as described in more detail below. FUTURE SYSTEM170is a virtual representation of a future hardware system or design, for example a future IHS. FUTURE SYSTEM170may take the form of a software emulation or virtualization of a future hardware system or future IHS.

The designation, FUTURE SYSTEM170′, describes FUTURE SYSTEM170after test system100loads the FUTURE SYSTEM170software into system memory125for execution or analysis. A user or other entity installs software such as APPLICATION SOFTWARE175in non-volatile storage140of test IHS102prior to conducting testing. APPLICATION SOFTWARE175acts as workload software, namely a workload. The designation, APPLICATION SOFTWARE175″, describes APPLICATION SOFTWARE175after test system100loads the APPLICATION SOFTWARE175′ from storage140into system memory125for execution. A user may load programs, such as SURROGATE PROGRAM1and SURROGATE PROGRAM2into non-volatile storage140for execution within test IHS102during simulation of FUTURE SYSTEM170.

FIG. 2depicts runtime performance data for APPLICATION SOFTWARE175and surrogate programs, such as SURROGATE PROGRAM1and SURROGATE PROGRAM2, on multiple HW designs or existing HW systems (IHSs), such as EXISTING IHS A, EXISTING IHS B, EXISTING IHS C, EXISTING IHS D. Existing HW systems include EXISTING IHS A, EXISTING IHS B, EXISTING IHS C, and EXISTING IHS D. A designer or other entity may select an existing system, such as EXISTING IHS A, EXISTING IHS B, EXISTING IHS C and EXISTING IHS D for testing purposes. In one embodiment, designers or other entities may select each existing system to represent a hardware construction similar to FUTURE SYSTEM170. For example, EXISTING IHS A may be an existing HW design of a previous design model of FUTURE SYSTEM170, EXISTING IHS B may be an existing HW design that employs a hardware design or structure similar to FUTURE SYSTEM170. A designer may select the existing systems, namely EXISTING IHS A, EXISTING IHS B. EXISTING IHS C and EXISTING IHS D, for hardware or software commonality with respect to FUTURE SYSTEM170, or for other criteria.

FIG. 2depicts runtime performance data for APPLICATION SOFTWARE175and surrogate programs, such as SURROGATE PROGRAM1and SURROGATE PROGRAM2, on multiple HW designs or existing HW systems (IHSs), such as EXISTING IHS A, EXISTING IHS B, EXISTING IHS C, SYSTEM D196. Existing HW systems include EXISTING IHS A, EXISTING IHS B, EXISTING IHS C, and EXISTING IHS D. A designer or other entity may select an existing system, such as EXISTING IHS A, EXISTING IHS B, EXISTING IHS C and EXISTING IHS D for testing purposes. In one embodiment, designers or other entities may select each existing system to represent a hardware construction similar to FUTURE SYSTEM170. For example, EXISTING IHS A may be an existing HW design of a previous design model of FUTURE SYSTEM170. EXISTING IHS B may be an existing HW design that employs a hardware design or structure similar to FUTURE SYSTEM170. A designer may select the existing systems, namely EXISTING IHS A, EXISTING IHS B, EXISTING IHS C and EXISTING IHS D, for hardware or software commonality with respect to FUTURE SYSTEM170, or for other criteria.

Designers or other entities may load and execute multiple application software or surrogate programs, shown in column205on EXISTING IHS A, and the results are shown in column210ofFIG. 2. As shown in column205, multiple programs, namely APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2provide software for execution in system100. The runtime performance data ofFIG. 2is a number that demonstrates the time that a particular software application or surrogate program consumes as it runs from start to finish. In other words, the runtime performance data is the amount of execution time for each application software or surrogate program. That runtime may be days, hours, or any other time measurement for comparison purposes.

Column210ofFIG. 2shows runtime performance data results for EXISTING IHS A. For example, APPLICATION SOFTWARE175executing on EXISTING IHS A generates a runtime performance data result of 10, as shown in row260, column210. Designers may select surrogate programs for many reasons, such as similarity to application software, standard industry benchmarking software, or other reasons. SURROGATE PROGRAM1executing on EXISTING IHS A generates a runtime performance data result of 15, as shown in row270, column210. SURROGATE PROGRAM2executing on EXISTING IHS A generates a runtime performance data result of 5, as shown in row280, column210.

Column220shows runtime performance data results for EXISTING IHS B. For example, APPLICATION SOFTWARE175executing on EXISTING IHS B generates a runtime performance data result of 20, as shown in row260, column220. The SURROGATE PROGRAM1executing on EXISTING IHS B generates a runtime performance data result of 15, as shown in row270, column220. SURROGATE PROGRAM2executing on EXISTING IHS B generates a runtime performance data result of 11, as shown in row280, column220. Column230shows runtime performance data results for EXISTING IHS C are shown in. For example, APPLICATION SOFTWARE175executing on EXISTING IHS C generates a runtime performance data result of 5, as shown in row260, column230. SURROGATE PROGRAM1executing on EXISTING IHS C generates a runtime performance data result of 10, as shown in row270, column230. SURROGATE PROGRAM2executing on EXISTING IHS C generates a runtime performance data result of 2.5, as shown in row280, column230.

Column240shows runtime performance data results for EXISTING IHS D. For example, APPLICATION SOFTWARE175executing on EXISTING IHS D generates a runtime performance data result of 30, as shown in row260, column240. SURROGATE PROGRAM1executing on EXISTING IHS D generates a runtime performance data result of 40, as shown in row270, column240. SURROGATE PROGRAM2executing on EXISTING IHS D generates a runtime performance data result of 14, as shown in row280and column240. System100executes FUTURE SYSTEM170in a simulation environment. In other words, FUTURE SYSTEM170represents a software or virtual representation of a future hardware IHS or future system. Test IHS102of system100executes FUTURE SYSTEM170in a virtual environment and produces runtime performance data as output.

Column245shows runtime performance data results for FUTURE SYSTEM170. For example, SURROGATE PROGRAM1executing on FUTURE SYSTEM170in test IHS102generates a runtime performance data result of 20, as shown in row270, column245. SURROGATE PROGRAM2executing on FUTURE SYSTEM170generates a runtime performance data result of 55, as shown in row280, column245. Application software is typically relatively large or many lines of code in length. Designers may decide to not execute APPLICATION SOFTWARE175on FUTURE SYSTEM170because that may require extensive amounts of simulation time or runtime on a test IHS, such as test IHS102. In this case, APPLICATION SOFTWARE175executing on FUTURE SYSTEM170as shown in row260, column245is unknown at this time. The determination of the “X” value, namely the runtime performance projection for APPLICATION SOFTWARE175on a future IHS, is described below.

Row290ofFIG. 2shows an aggregate of surrogate program runtime performance data. Aggregate programs, such as aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2provide one method to generate more runtime performance data for analysis. In other words, the results of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data provide input into the generation of aggregate results as shown in row290. Designers may use a sum, geometric mean, host fraction, or other technique to generate aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2data. In one example, designers or other entities generate an aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data as shown in row290ofFIG. 2. For example, row290, column210shows aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data executing on EXISTING IHS A as a value 2.5. Aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data executing on EXISTING IHS B is 6.3, as shown in row290, column220.

Aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data executing on EXISTING IHS C is 3.3, as shown in row290, column230. Aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data executing on EXISTING IHS D is 10.4, as shown in row290, column240. Aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data executing on FUTURE SYSTEM170is 14.7 as shown in row290, column245. Designers may select more surrogate programs, such as benchmark software programs (not shown), thanFIG. 2depicts. In other words, whileFIG. 2shows two surrogate programs, the runtime performance data may includes data from more than two surrogate programs. Designers may generate multiple other aggregates of combinations of surrogate programs (not shown) to provide more runtime performance data for further analysis.

Aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data executing on EXISTING IHS C is 3.3, as shown in row290, column230. Aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data executing on SYSTEM D196is 10.4, as shown in row290, column240. Aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data executing on FUTURE SYSTEM170is 14.7 as shown in row290, column245. Designers may select more surrogate programs, such as benchmark software programs (not shown), thanFIG. 2depicts. In other words, whileFIG. 2shows two surrogate programs, the runtime performance data may includes data from more than two surrogate programs. Designers may generate multiple other aggregates of combinations of surrogate programs (not shown) to provide more runtime performance data for further analysis.

FIG. 3depicts runtime performance data that a designer or other entity normalizes during analysis of runtime performance data, such as that ofFIG. 2. The normalized runtime performance data includes APPLICATION SOFTWARE175performance normalized to EXISTING IHS A, and SURROGATE PROGRAM1performance normalized to EXISTING IHS A. The normalized runtime performance data also includes SURROGATE PROGRAM2performance normalized to EXISTING IHS A, and the aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2performance normalized to EXISTING IHS A, as shown in column310.

Row350shows the normalized runtime performance data for multiple system types, namely EXISTING IHS A, EXISTING IHS B, EXISTING IHS C, EXISTING IHS D, and FUTURE SYSTEM170. A designer may normalize runtime performance data perFIG. 2by identifying one system, such as EXISTING IHS A, to normalize all other data against. In one embodiment, the designer or other entity normalizes all data for EXISTING IHS A from column210inFIG. 2to all 1's. For example the APPLICATION SOFTWARE175runtime performance data perFIG. 2row260, column210shows a particular data value of 10 or a normalization base value equal to 10. The designer normalizes the data value of APPLICATION SOFTWARE175runtime performance to EXISTING IHS A by dividing that particular value of 10 by itself and thus generating a data value of 1, as shown in row360, column310ofFIG. 3.

The designer or other entity normalizes all the remaining data for APPLICATION SOFTWARE175in row360using the particular normalization base value of 10 in this example. The designer normalizes all data for APPLICATION SOFTWARE175by dividing the data as shown inFIG. 2, row260by the particular normalization base value of 10. Each value in row360ofFIG. 3is the same data as row260ofFIG. 2divided by 10, and this provides the APPLICATION SOFTWARE175performance normalized to EXISTING IHS A data. For example, the APPLICATION SOFTWARE175performance normalized to EXISTING IHS A for EXISTING IHS B is equal to 20 divided by 10 or a normalized runtime performance data value of 2, as shown in row360, column320. The APPLICATION SOFTWARE175performance normalized to EXISTING IHS A for EXISTING IHS C is equal to 5 divided by 10 or a normalized runtime performance data value of 0.5 as shown in row360, column330.

The APPLICATION SOFTWARE175performance normalized to EXISTING IHS A for EXISTING IHS D is equal to 30 divided by 10 or a normalized runtime performance data value of 3 as shown in row360, column340. In this manner, a designer determines the complete normalized runtime performance data for APPLICATION SOFTWARE175performance normalized to EXISTING IHS A as shown inFIG. 3, row360. However, the lack of data for APPLICATION SOFTWARE running on FUTURE SYSTEM170inhibits the generation of FUTURE SYSTEM170data as yet in row360, column345. The determination of APPLICATION SOFTWARE running on FUTURE SYSTEM170“X” and APPLICATION SOFTWARE175performance normalized to EXISTING IHS A “XN” is described in more detail below.

The APPLICATION SOFTWARE175performance normalized to EXISTING IHS A for SYSTEM D196is equal to 30 divided by 10 or a normalized runtime performance data value of 3 as shown in row360, column340. In this manner, a designer determines the complete normalized runtime performance data for APPLICATION SOFTWARE175performance normalized to EXISTING IHS A as shown inFIG. 3, row360. However, the lack of data for APPLICATION SOFTWARE running on FUTURE SYSTEM170inhibits the generation of FUTURE SYSTEM170data as yet in row360, column345. The determination of APPLICATION SOFTWARE running on FUTURE SYSTEM170“X” and APPLICATION SOFTWARE175performance normalized to EXISTING IHS A “XN” is described in more detail below.

The designer or other entity also normalizes the runtime performance data of SURROGATE PROGRAM1running on EXISTING IHS A to “1”. In this example, the SURROGATE PROGRAM1runtime performance data perFIG. 2row270, column210shows a particular data value of 15. The designer normalizes this data by dividing that particular value of 15 by itself and thus generates a value of 1 for the data of row370, column310ofFIG. 3. In other words, the normalized data value for SURROGATE PROGRAM1performance normalized to EXISTING IHS A is equal to 1, as shown per row370, column310. The designer normalizes all the remaining data for SURROGATE PROGRAM1in row370using that particular value of 15. In other words, the designer normalizes all data for SURROGATE PROGRAM1by dividing the data as shown inFIG. 2, row270by the particular data value of 15.

Each value in row370ofFIG. 3is the same data as row270ofFIG. 2divided by 15. This division process results in the SURROGATE PROGRAM1performance normalized to EXISTING IHS A data ofFIG. 3. For example, the SURROGATE PROGRAM1performance normalized to EXISTING IHS A for EXISTING IHS B is equal to 15 divided by 15 or a normalized runtime performance data result of 1 as shown in row370, column320. The SURROGATE PROGRAM1performance normalized to EXISTING IHS A for EXISTING IHS C is equal to 10 divided by 15 or a normalized runtime performance data result of approximately 0.67, as shown in row370, column330. The SURROGATE PROGRAM1performance normalized to EXISTING IHS A for EXISTING IHS D is equal to 40 divided by 15 or a normalized runtime performance data value of approximately 2.7, as shown in row370, column340. The SURROGATE PROGRAM1performance normalized to EXISTING IHS A for FUTURE SYSTEM170is equal to 20 divided by 15 or a normalized runtime performance data value of approximately 1.33, as shown in row370, column345.

The designer or other entity also normalizes the runtime performance data value of SURROGATE PROGRAM2running on EXISTING IHS A to “1”. In this example, the SURROGATE PROGRAM2runtime performance data value perFIG. 2row280, column210shows a particular data value of 5. The designer normalizes this data by dividing that particular value of 5 by itself and thus generates a value of 1 for the data of row380, column310ofFIG. 3. In other words, the normalized data value for SURROGATE PROGRAM2performance normalized to EXISTING IHS A is equal to 1, as shown in row380, column310. The designer normalizes all the remaining data for SURROGATE PROGRAM2in row380using that particular divisor value of 5. The designer normalizes all data for SURROGATE PROGRAM1by dividing the data as shown inFIG. 2, row280by the particular data value of 5.

Each value in row380ofFIG. 3is the same data as row280ofFIG. 2divided by 5, generating the SURROGATE PROGRAM2performance normalized to EXISTING IHS A data ofFIG. 3. For example, the SURROGATE PROGRAM2performance normalized to EXISTING IHS A for EXISTING IHS B is equal to 11 divided by 5 or a normalized runtime performance data result of 2.2 as shown in row380, column320. The SURROGATE PROGRAM2performance normalized to EXISTING IHS A for EXISTING IHS C is equal to 2.5 divided by 5 or a normalized runtime performance data result of 0.5, as shown in row380, column330. The SURROGATE PROGRAM2performance normalized to EXISTING IHS A for EXISTING IHS D is equal to 14 divided by 5 or a normalized runtime performance data value of approximately 2.8, as shown in row380, column340. The SURROGATE PROGRAM2performance normalized to EXISTING IHS A for FUTURE SYSTEM170is equal to 55 divided by 5 or a normalized runtime performance data value of 11, as shown in row380, column345.

The designer or other entity normalizes the runtime performance data of aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2running on EXISTING IHS A to “1”. In this example, the aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data perFIG. 2row290, column210shows a particular data value of 2.5. The designer normalizes this data by dividing that particular value of 2.5 by itself and thus generates a value of 1 for the data of row390, column310ofFIG. 3. In other words, the normalized data value for aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2performance normalized to EXISTING IHS A is equal to 1, as shown per row390, column310. The designer normalizes all the remaining data for aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2in row390using that particular value of 2.5. The designer normalizes all data for SURROGATE PROGRAM1by dividing the data as shown inFIG. 2, row290by the particular data value of 2.5.

Each value in row390ofFIG. 3is the same data as row290ofFIG. 2divided by 2.5. This division process, results in the aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2normalized to EXISTING IHS A data ofFIG. 3. For example, the aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2performance normalized to EXISTING IHS A for EXISTING IHS B is equal to 6.3 divided by 2.5 or a normalized runtime performance data result of approximately 2.5 as shown in row390, column320. The aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance normalized to EXISTING IHS A for EXISTING IHS C is equal to 3.3 divided by 2.5 or a normalized runtime performance data result of approximately 1.3, as shown in row390, column330.

The aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance normalized to EXISTING IHS A for EXISTING IHS D is equal to 10.4 divided by 2.5 or a normalized runtime performance data value of approximately 4.2, as shown in row390, column340. The aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2normalized to EXISTING IHS A for FUTURE SYSTEM170is equal to 14.7 divided by 2.5 or a normalized runtime performance data value of approximately 5.9, as shown in row390, column345.

The particular data value of “XN”, or the APPLICATION SOFTWARE175performance normalized to EXISTING IHS A is shown in row360, column345. Designers may generate that particular XN data value using the normalized runtime performance data ofFIG. 3. In other words, designers or other entities may generate the particular data value of XN by using a mathematical relationship of the data values of the normalized runtime performance data ofFIG. 3. For example, in one embodiment, a least-squares-fit mathematical technique using the normalized runtime performance data values ofFIG. 3may determine the value of XN. In other words, a designer selects the closest matching software program as shown in column205in terms of performance data best fit, by using a least-squares-fit mathematical representative technique.

A designer or other entity selects the particular software program of column305that most closely matches or fits the performance of APPLICATION SOFTWARE175performance normalized to EXISTING IHS A as shown in row360. Each of the surrogate programs is a candidate for selection as the best fit. Thus, each surrogate program is a candidate surrogate program for selection as being the best fit or most representative of the performance characteristics of APPLICATION SOFTWARE175running on FUTURE SYSTEM170. In one example, the least-squares-fit technique provides designers with a selection of SURROGATE PROGRAM2performance normalized to EXISTING IHS A as shown in row380as the best fit to APPLICATION SOFTWARE175performance normalized to EXISTING IHS A as shown in row360. In other words, the data ofFIG. 3row380for candidate SURROGATE PROGRAM2280most closely matches the data of row360for the APPLICATION SOFTWARE175. In this manner, designers may determine the value of XN as equal to the value of row380, column345, namely a normalized runtime performance data value of 11. In other words, since row380most closely matches row360, the designer or other entity populates XN with the data value of 11 from row380. An XN value of 11 thus represents the normalized performance of APPLICATION SOFTWARE175on FUTURE SYSTEM170.

With the determination of the normalized XN value as equal to 11 inFIG. 3, the designer or other entity may determine the un-normalized value X, namely the runtime performance value of APPLICATION SOFTWARE175running on FUTURE SYSTEM170in the following manner. Since the normalization of APPLICATION SOFTWARE175data, namely that of row260, uses a value of 10 as the divisor to achieve normalization, that particular value of 10 or the normalization base value enables the determination of X. In other words, multiplying the normalized value of XN=11 by the former divisor 10 yields an un-normalized value that represents the projected performance of APPLICATION SOFTWARE175on FUTURE SYSTEM170. In this example, X is equal to the product of XN, the normalized runtime performance data value of 11 as shown in row380, column345, and the particular normalization base value of 10 shown in row260, column210ofFIG. 2. In other words, X is equal to 11 times 10 and thus110represents the projected runtime performance of APPLICATION SOFTWARE175running on FUTURE SYSTEM170. That data value of X=110 is a projection or prediction and not a precise measurement of actual results. Using the above described methodology, a designer need not execute the actual APPLICATION SOFTWARE175on a FUTURE SYSTEM170either in real hardware or within a simulation environment to project the runtime value X of the APPLICATION SOFTWARE175running on EXISTING IHS A.

FIG. 4is a flowchart that depicts one method for generating a projection of APPLICATION SOFTWARE175performance on FUTURE SYSTEM170. APPLICATION SOFTWARE functions as workload software, namely a workload. The disclosed runtime projection method starts at block410. Designers measure the runtime performance of APPLICATION SOFTWARE175on existing systems, as per block420. For example, a customer may provide a user application software program, such as APPLICATION SOFTWARE175for testing purposes. A designer or other entities may use APPLICATION SOFTWARE175to test runtime performance on multiple existing HW systems or IHSs, such as EXISTING IHS A, EXISTING IHS B, EXISTING IHS C, EXISTING IHS D, and other hardware (HW) system designs not shown. The APPLICATION SOFTWARE175executes on multiple HW design systems, namely multiple IHSs, and a respective total runtime per HW design system from start to finish of execution provides the runtime performance data, such as the data in row260, columns210,220,230, and240ofFIG. 2.

Designers or other entities measure surrogate program performance on existing systems, as per block440. In other words, designers execute SURROGATE PROGRAM1, SURROGATE PROGRAM2, and the aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2on EXISTING IHS A, EXISTING IHS B, EXISTING IHS C, and EXISTING IHS D to generate the runtime performance data ofFIG. 2as shown in rows270,280, and290columns210,220,230, and240. Using the simulation capability of test system100, designers generate surrogate program performance data on FUTURE SYSTEM170, as per block450. In one example, designers execute SURROGATE PROGRAM1, SURROGATE PROGRAM2, and the aggregate of runtime performance data of SURROGATE PROGRAM1, and SURROGATE PROGRAM2on the model of FUTURE SYSTEM170in test IHS102. The results of the runtime simulation provide the runtime performance data for column245ofFIG. 2. Designers or other entities generate an aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2, as per block455. Designers may use a geometric means or other technique to generate an aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2from SURROGATE PROGRAM1and SURROGATE PROGRAM2, as shown in row290, column205ofFIG. 2.

Designers or other entities normalize the runtime performance data as shown inFIG. 2, as per block460. Designers normalize the runtime performance data as shown inFIG. 2to generate the normalized runtime performance data as shown inFIG. 3. In one example, designers select EXISTING IHS A as the HW system with respect to which they normalize all other performance data. In this manner, all normalized runtime performance data for EXISTING IHS A is set equal to 1 as shown in column310ofFIG. 3. Designers normalize the remaining data inFIG. 3with the exception of the unknown XN data that corresponds to APPLICATION SOFTWARE175performance normalized to EXISTING IHS A, a shown in row360, column345.FIG. 3thus shows one example of normalized runtime performance data.

From the multiple surrogate programs, designers or other entities select a particular surrogate program or aggregate that provides the closest fit to APPLICATION SOFTWARE175, as per block465. Designers or other entities select the normalized performance data value of the closest fit surrogate program or aggregate of surrogate programs as the normalized performance data value for the APPLICATION SOFTWARE175on the FUTURE SYSTEM170, as per block470. Designers or other entities may determine the XN data value or APPLICATION SOFTWARE175performance normalized to EXISTING IHS A data value ofFIG. 3by using a least-squares-fit mathematical technique. The least-squares-fit technique provides designer tools for selection of the surrogate program that most closely fits the performance of APPLICATION SOFTWARE175across all systems, as shown per row360ofFIG. 3. In this example, SURROGATE PROGRAM2performance normalized to EXISTING IHS A is the particular surrogate program that best fits that criteria of least-squares-fit. The projected normalized runtime performance data value of 11, as shown in row380, column345provides the XN data value as shown in row360, column345ofFIG. 3. In other words XN equals 11.

Designers or other entities un-normalize or de-normalize the selected normalized performance data value to provide a runtime projection of the APPLICATION SOFTWARE175on the FUTURE SYSTEM170, as per block475. Designers determine the X data value, or APPLICATION SOFTWARE175performance projection on FUTURE SYSTEM170from the XN data value above. Designers use the normalization base value of 10 from row260, column210ofFIG. 2to adjust or un-normalize the XN value. In other words the XN data value of 11 times the normalization base value of 10 provides the projected un-normalized X value. Multiplying the XN data value of 11 by the normalization base value of 10 equals 110, namely the runtime performance projection X for the APPLICATION SOFTWARE175executing on FUTURE SYSTEM170. This step effectively removes the normalization of the XN value and provides an actual projected raw performance value X. The runtime projection method perFIG. 4ends, as per block480. In one embodiment, performance projection system100may perform the functions in the blocks of theFIG. 4flowchart autonomously, or semi-autonomously. Designers or others may configure test IHS102to carry out these functions. In other embodiments, designers or others may manually assist in the performance of the functions of the blocks of theFIG. 4flowchart. Test IHS102may store the data ofFIGS. 2 and 3in system memory125and/or nonvolatile storage140.

FIG. 5depicts performance data for HW systems or IHSs, such as EXISTING IHS A and FUTURE SYSTEM170, that generate runtime performance and microarchitecture dependent hardware counter107data. Microarchitecture dependent data includes data from functional units or micro-architectural units of the performance projection system100such as EXISTING IHS A or FUTURE SYSTEM170. Micro-architectural units include caches, branch misprediction units, instruction flush mechanisms, and other units of IHSs. Microarchitecture dependent data includes cache miss rates, branch misprediction counts, instruction flush counts and other data from microarchitecture units. Hardware counter107or other memory store, such as system memory125or non-volatile storage140, in processor105may store this microarchitecture data. Processor105may include multiple other HW counters (not shown) or other storage that stores microarchitecture data.

From a group of existing IHSs such as EXISTING IHS A and EXISTING IHS B or more existing IHSs, designers select an existing IHS, such as EXISTING IHS A. In one embodiment, designers may select any existing IHS. Designers or other entities execute multiple benchmark or software programs, such as APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2, as shown in column510on EXISTING IHS A. More particularly, each application and surrogate software program shown in column510may execute on EXISTING IHS A. Each surrogate software program shown in column510may execute on FUTURE SYSTEM170.

During execution of software programs on EXISTING IHS A, designers or other entities collect the runtime performance data results. For example, during execution of APPLICATION SOFTWARE175on EXISTING IHS A, designers or other entities collect a runtime performance data value of 15 as shown in row560, column515. SURROGATE PROGRAM1executing on EXISTING IHS A achieves a runtime performance data result of 20, as shown in row570, column515. SURROGATE PROGRAM2executing on EXISTING IHS A achieves a runtime performance data result of 10, as shown in row580, column515. During execution of software programs, such as APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2on EXISTING IHS A, hardware counter107maintains a record of performance data. That hardware counter107performance data may be microarchitecture dependent data of the particular IHS design under test. For example, APPLICATION SOFTWARE175executing on EXISTING IHS A generates hardware counter107data that is microarchitecture data unique to EXISTING IHS A. In one embodiment, hardware counter107performance data may include cycles per instruction (CPI) data as shown in column520.

In one example, CPI is a measure of how much time each instruction takes to complete execution in terms of processor cycles. The CPI measure is a good representation of the efficiency of a particular software program running on a HW design system, such as EXISTING IHS A. For example APPLICATION SOFTWARE175executing on EXISTING IHS A produces CPI data value of 2.5 as shown in row560, column520. SURROGATE PROGRAM1executing on EXISTING IHS A produces CPI data value of 4 as shown in row570, column520. SURROGATE PROGRAM2executing on EXISTING IHS A produces CPI data value of 2 as shown in row580, column520.

Hardware counter107data may also include microarchitecture dependent data such as cache miss rate data for an L1 cache (not shown) in EXISTING IHS A, like that of L1 cache109of test IHS102, as shown in column530. APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2generate miss rate data for L1 cache (not shown), like L1 cache109during execution on EXISTING IHS A, as shown in column530. The L1 cache miss rate data demonstrates the property of L1 cache to either hit or miss on a memory request during execution of a software program, such as APPLICATION SOFTWARE175. The L1 cache is a microarchitecture device of EXISTING IHS A, and thus L1 cache miss rate data is microarchitecture dependent data for EXISTING IHS A. In one example, APPLICATION SOFTWARE175executing on EXISTING IHS A generates L1 cache miss rate data of 2 as shown in row560, column530. SURROGATE PROGRAM1executing on EXISTING IHS A generates an L1 cache miss rate data value of 1 as shown in row570, column530. SURROGATE PROGRAM2executing on EXISTING IHS A generates an L1 cache miss rate data value of 4 as shown in row580, column530.

In a manner similar to EXISTING IHS A, test system100generates performance data for FUTURE SYSTEM170.FIG. 5includes blanks for row560, columns540, and550if designers or other entities do not execute APPLICATION SOFTWARE175on FUTURE SYSTEM170. A “Z” term in row560, column535represents an unknown value for APPLICATION SOFTWARE175runtime performance data for FUTURE SYSTEM170. Designers or other entities project or predict the “Z” term to provide APPLICATION SOFTWARE175performance projection on FUTURE SYSTEM170information as described in more detail below. In a simulation environment, test IHS102executes SURROGATE PROGRAM1and SURROGATE PROGRAM2on a virtual copy or design of FUTURE SYSTEM170. Designers or other entities collect the runtime and hardware counter107performance data to populate columns535,540and550for SURROGATE PROGRAM1, and SURROGATE PROGRAM2ofFIG. 5. For example, during execution of SURROGATE PROGRAM1on FUTURE SYSTEM170, designers collect a runtime performance data result of 30, as shown in row570, column535. Designers or others may configure the test IHS formed by processor105, bus110and system memory125to collect runtime and hardware performance data in hardware counter107. In actual practice, hardware counter107may include multiple hardware counters. Test IHS102may store the data ofFIGS. 5,6and7in system memory125and/or nonvolatile storage140.

SURROGATE PROGRAM2executing on FUTURE SYSTEM170generates a runtime performance data result of 20, as shown in row580, column535. During execution of SURROGATE PROGRAM1and SURROGATE PROGRAM2on FUTURE SYSTEM170, hardware counter107maintains a record of hardware counter107performance data. That hardware counter107performance data may be microarchitecture dependent data of the particular design under test. SURROGATE PROGRAM1executing on FUTURE SYSTEM170generates a CPI data value of 3 as shown in row570, column540. SURROGATE PROGRAM2executing on FUTURE SYSTEM170generates CPI data value of 1 as shown in row580, column540. Test system100may store the microarchitecture dependent data or hardware counter performance data in system memory125and/or non-volatile storage140.

Hardware counter107performance data may also include future system L1 cache (not shown) miss rate data, like that of for L1 cache109as shown in column550. SURROGATE PROGRAM1, and SURROGATE PROGRAM2generate L1 cache miss rate data during execution on FUTURE SYSTEM170, as shown in column550. The L1 cache miss rate data demonstrates the property of the L1 cache to either hit or miss on a memory request during execution of APPLICATION SOFTWARE175. In one example, SURROGATE PROGRAM1executing on FUTURE SYSTEM170generates an L1 cache miss rate data value of 2 as shown in row570, column550. SURROGATE PROGRAM2executing on FUTURE SYSTEM170generates an L1 cache miss rate data value of 1 as shown in row580, column550. Although this example depicts hardware counter107records of CPI and L1 cache miss rates, test IHS102may record other hardware counter performance and microarchitecture dependent data. For example, hardware counter107of test IHS102may record system memory125reload count data, CPI stack breakdown event count data, or other microarchitecture dependent data.

Designers or other entities generate an aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2as shown in row590ofFIG. 5. Designers may use a sum, geometric mean, host fraction, or other technique to generate aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2. In one example, designers generate aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2runtime performance data as shown in row590by use of a geometric mean. For example, aggregate of runtime performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2produces a runtime performance data value of 15 for EXISTING IHS A as shown in row590, column515.

Aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2produces a CPI data value of 3 for EXISTING IHS A, as shown in row590, column520. Aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2produces an L1 cache miss rate data value of 2.5 for EXISTING IHS A, as shown in row590, column530. Aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2produces a runtime performance data value of 25 for FUTURE SYSTEM170as shown in row590, column535. Aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2exhibits a CPI data value of 2 for FUTURE SYSTEM170, as shown in row590, column540.

Aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2produces an L1 cache miss rate data value of 1.5 for FUTURE SYSTEM170, as shown in row590, column550. The data in row590is the result of geometric mean or averaging the data in SURROGATE PROGRAM1row570and SURROGATE PROGRAM2row580data. The result is a unique set of runtime and hardware counter107performance data for the aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2. Designers are not limited to two surrogate programs, such as SURROGATE PROGRAM1and SURROGATE PROGRAM2. In practice, the disclosed methodology may employ more than two surrogate programs. In other words, designers may select multiple benchmark software programs, or other software programs (not shown) beyond the two surrogate programs that representative performance projection system100employs. Designers may generate multiple other aggregates of combinations of surrogate programs (not shown) to provide more performance data for analysis.

FIG. 6depicts normalized performance data from the data results ofFIG. 5. For example, the normalized performance data ofFIG. 6demonstrates the results of normalization in reference to EXISTING IHS A runtime, or column515ofFIG. 5. In other words, designers or other entities normalize the data of EXISTING IHS A runtime in column615to all 1's. Designers or other entities normalize the remaining data ofFIG. 5in reference to the data in column515or EXISTING IHS A runtime performance data. The normalized data ofFIG. 6reflects the performance data from each software application program of column610. Column610includes the software application programs APPLICATION SOFTWARE175, SURROGATE PROGRAM1, SURROGATE PROGRAM2, and the aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2.

Column620ofFIG. 6shows EXISTING IHS A performance data of column520ofFIG. 5normalized to EXISTING IHS A runtime data of column515ofFIG. 5. Column620shows EXISTING IHS A CPI data of 0.2, 0.2, 0.2, and 0.2 normalized to EXISTING IHS A runtime data. Column630shows EXISTING IHS A L1 cache miss rate data of 0.1, 0.1, 0.4, and 0.2 normalized to EXISTING IHS A runtime data. Column635shows FUTURE SYSTEM170runtime data of ZN, 1.5, 2, 1.7 normalized to EXISTING IHS A runtime data. “ZN” represents the normalized data value for Z, or the normalized data value for APPLICATION SOFTWARE175runtime performance data for FUTURE SYSTEM170. Column640shows FUTURE SYSTEM170CPI data of blank/null, 0.2, 0.1, and 0.1 normalized to EXISTING IHS A runtime data. In one embodiment, designers do not measure the FUTURE SYSTEM170CPI data for APPLICATION SOFTWARE175thus resulting in a blank or no data value result (blank/null). Column650shows FUTURE SYSTEM170109L1 cache miss rate data of blank/null, 0.1, 0.1, and 0.1 normalized to EXISTING IHS A runtime data. In one embodiment, designers or other entities perform no measure of the FUTURE SYSTEM170190L1 cache miss rate data for APPLICATION SOFTWARE175, resulting in a blank or no data value result (blank/null).

FIG. 7depicts weighted normalized performance data from the data results ofFIG. 6. Column710depicts the software application programs APPLICATION SOFTWARE175, SURROGATE PROGRAM1, SURROGATE PROGRAM2, and the aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2. Designers or other entities may weight a particular performance data result, such as the normalized performance data for EXISTING IHS A CPI results, shown in column620ofFIG. 6. Designers or other entities may multiply the entire data of column620ofFIG. 6normalized performance data by a weighting factor to obtain weighted normalized performance data. For example, designers or other entities may choose a weighting factor of 10 to increase the effective weight or relative strength of a particular data grouping or column of normalized performance data. In one example, column720shows the results of multiplying the normalized performance data column620inFIG. 6by a weighting factor of 10. Column720shows the weighted normalized performance data for EXISTING IHS A CPI data of 2, 2, 2 and 2. Column740shows the weighted normalized performance data for FUTURE SYSTEM170CPI data of blank/null, 2, 1 and 1.

Designers or other entities may scale a particular surrogate program result to adjust the respective weighted normalized performance data. For example, row795shows the SCALED SURROGATE PROGRAM2results of a 10 percent increase or the 10 percent scaled results of the data of SURROGATE PROGRAM2in row780. Row795shows the SCALED SURROGATE PROGRAM2results of 2.2 and 1.1 for EXISTING IHS A and FUTURE SYSTEM170weighted normalized CPI performance data, respectively. As shown in more detail inFIG. 8below, the weighted normalized CPI performance data provides designers with a method to determine the normalized runtime projection performance data of APPLICATION SOFTWARE175executing on FUTURE SYSTEM170or ZN. Designers or other entities may un-normalize the ZN data value to provide the runtime performance projection of APPLICATION SOFTWARE175executing on FUTURE SYSTEM170or Z as shown below.

FIG. 8is a flowchart that depicts a method of generating a projection of APPLICATION SOFTWARE175performance on an IHS such as FUTURE SYSTEM170using hardware counter107that may record or store microarchitecture dependent data. The runtime projection method starts, as per block810. From a group of existing hardware (HW) IHSs, or existing IHSs, designers select an existing IHS, such as EXISTING IHS A, as per block815. Using EXISTING IHS A as the existing HW design system, designers execute APPLICATION SOFTWARE175and the surrogate programs, namely SURROGATE PROGRAM1, and SURROGATE PROGRAM2on EXISTING IHS A, as per block820. During that exercise or execution, designers or other entities collect the runtime data such as runtime performance data shown in rows560,570, and580, column515ofFIG. 5. For example, the respective runtime data for APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2executing on EXISTING IHS A is 15, 20, and 10 as shown in column515. Designers or other entities measure the performance of APPLICATION SOFTWARE175and surrogate programs, such as SURROGATE PROGRAM1and SURROGATE PROGRAM2, on an existing IHS such as EXISTING IHS A, as per block825.

During the execution of APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2, hardware counter107records CPI data and L1 cache miss rate data in respective columns520and530data ofFIG. 5. For example, the respective CPI data for APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2executing on EXISTING IHS A is 2.5, 4, and 2 as shown in column520. The respective L1 cache miss rate data for APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2executing on EXISTING IHS A are 2, 1, and 4, respectively, as shown in column530ofFIG. 5. Designers or other entities execute surrogate programs, namely SURROGATE PROGRAM1, and SURROGATE PROGRAM2on FUTURE SYSTEM170, as per block830. During execution of SURROGATE PROGRAM1, and SURROGATE PROGRAM2on FUTURE SYSTEM170, designers or other entities measure surrogate program performance data, as per block835. To achieve this, designers, using the simulation capabilities of test system100, execute all surrogate programs, such as SURROGATE PROGRAM1and SURROGATE PROGRAM2on FUTURE SYSTEM170. Designers record runtime performance, CPI, and L1 cache miss rate data from the results of test system100simulation of FUTURE SYSTEM170.

Columns535,540and550show the results of surrogate program performance. For example, the respective runtime data for APPLICATION SOFTWARE175, SURROGATE PROGRAM1, and SURROGATE PROGRAM2executing on FUTURE SYSTEM170is Z, 30, and 20 as shown in column535. At this point in time, the Z runtime result is undetermined, and will be described in more detail below. The CPI data for SURROGATE PROGRAM1, and SURROGATE PROGRAM2executing on FUTURE SYSTEM170are respectively 3 and 1, as shown in column540. The respective L1 cache miss rate data for SURROGATE PROGRAM1, and SURROGATE PROGRAM2executing on FUTURE SYSTEM170is 2, and 1 as shown in column550. Designers or other entities generate aggregate surrogate program performance data, as per block840. By using the performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2, designers may generate an aggregate or merging of the two surrogate program results.

More particularly, designers may generate an aggregate, such as aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2, as shown in row590using simple geometric averaging or other means. For example, the performance data for aggregate of SURROGATE PROGRAM1and SURROGATE PROGRAM2on EXISTING IHS A and FUTURE SYSTEM170is shown in row590. The aggregate data for runtime, CPI, and L1 cache miss rate are respectively 15, 3, 2.5, 25, 2, and 1.5 for EXISTING IHS A and FUTURE SYSTEM170. Although one aggregate, namely aggregate of performance data of SURROGATE PROGRAM1and SURROGATE PROGRAM2is shown in this example, designers may generate many other aggregate results (not shown) for other averaging techniques of surrogate programs. Designer may use combinations of averaging surrogate program data with aggregate program data, and other techniques to generate aggregate programs.

Designers or other entities normalize the performance data, as per block850. Designers normalize the performance data ofFIG. 5generating the normalized performance data ofFIG. 6to properly compare EXISTING IHS A and FUTURE SYSTEM170results. In other words, designers compare EXISTING IHS A and FUTURE SYSTEM170performance results by normalizing all data ofFIG. 5. In one embodiment of the disclosed method, designers may place a weighting scheme on the particular normalized performance data ofFIG. 6to generate weighted normalized performance data ofFIG. 7to provide better strength or weight of one particular metric over another. Designers weight the normalized performance data, as per block860. For example designers may multiply the CPI data in columns520and540by a weighting factor W to generate a weight of W times more strength to the CPI performance data of columns520and540. Designers may use a distance matrix such as the Euclidian distance measure to adjust performance data results. Designers may use other weighting factors and techniques to adjust the relative weight or strength of each performance data type as shown inFIG. 5row555. Applying normalization and weighting techniques to the performance data ofFIG. 5offers designers one method to select a surrogate program that best matches the performance of APPLICATION SOFTWARE175on FUTURE SYSTEM170.

Designers select one surrogate program from the surrogate programs as shown inFIG. 5that best fits the performance results of APPLICATION SOFTWARE175, as per block870. Designers may use any means of comparison between the performance data results of APPLICATION SOFTWARE175on EXISTING IHS A and FUTURE SYSTEM170and each surrogate program to find a best fit. Designers determine a scaling factor, as per block875. The scaling factor provides an offset or comparison between APPLICATION SOFTWARE175and the selected surrogate program, namely SURROGATE PROGRAM2, such as the SCALED SURROGATE PROGRAM2data in row795ofFIG. 7.

Designers determine the APPLICATION SOFTWARE175performance projection on FUTURE SYSTEM170, as per block885. Designers use the scaling factor to generate the runtime performance projection data for APPLICATION SOFTWARE175executing on FUTURE SYSTEM170. For example, using a scaling factor of 10 percent, designers determine the APPLICATION SOFTWARE175performance projection on FUTURE SYSTEM170as 10 percent greater than the runtime performance of SURROGATE PROGRAM2on FUTURE SYSTEM170. In that case the normalized runtime performance data of APPLICATION SOFTWARE175executing on FUTURE SYSTEM170(ZN) is 10 percent greater than 2, or the normalized runtime performance data of SURROGATE PROGRAM2executing on FUTURE SYSTEM170.

The normalized runtime performance projection of APPLICATION SOFTWARE175executing on FUTURE SYSTEM170or ZN is equal to 2.2, as per block880. From the ZN value, designers or other entities determine the runtime performance projection for APPLICATION SOFTWARE175executing on FUTURE SYSTEM170by un-normalizing or de-normalizing the ZN value, as per block885. The un-normalized runtime performance projection for APPLICATION SOFTWARE175executing on FUTURE SYSTEM170“Z” is 10 percent greater than 20 or equal to 22. In this example the runtime performance projection APPLICATION SOFTWARE175executing on FUTURE SYSTEM170is 22. The runtime projection method ends, as per block890. In one embodiment, test system100may perform the functions in the blocks of theFIG. 8flowchart autonomously, or semi-autonomously. Designers or others may configure test system100to carry out these functions. In other embodiments, designers or others may assist in the performance of the functions of the blocks of theFIG. 8flowchart.

The foregoing discloses methodologies wherein an performance projection system employs application software to provide IC design personnel with IC design system tools for simulation, design benchmarking, and other analysis. In one embodiment, designers initiate execution of multiple programs including application software and surrogate programs to generate performance runtime data for future and existing systems. Designers may normalize and evaluate performance runtime data to generate a runtime projection for future system performance.

The foregoing also discloses methodologies wherein an performance projection system employs a hardware counter to collect runtime performance and microarchitecture performance data. The performance projection system employs a future system simulation and existing system test for surrogate program testing. The test system executes application software to provide IC design personnel with runtime performance and microarchitecture data for design benchmarking, and other analysis. In one embodiment, designers execute the surrogate program and application software on the existing system to generate runtime and HW counter data. Designers may normalize and weight the runtime and HW counter data to provide enable a selection of particular surrogate program most similar to the application software. Designers may apply a scaling factor to surrogate program performance results to determine a runtime projection for future system from the particular surrogate program data.

Modifications and alternative embodiments of this invention will be apparent to those skilled in the art in view of this description of the invention. Accordingly, this description teaches those skilled in the art the manner of carrying out the invention and is intended to be construed as illustrative only. The forms of the invention shown and described constitute the present embodiments. Persons skilled in the art may make various changes in the shape, size and arrangement of parts. For example, persons skilled in the art may substitute equivalent elements for the elements illustrated and described here. Moreover, persons skilled in the art after having the benefit of this description of the invention may use certain features of the invention independently of the use of other features, without departing from the scope of the invention.