SPEC in a New Era

Matthijs van Waveren
SPEC Director
Fujitsu Systems Europe

Overview
SS-ken – 22 October 2008
 SPEC in a New Era
 Low Power Consumption
 Compute Intensive Applications
 Server Virtualization
 Conclusion

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)2

SPEC in a New Era
The world is evolving towards green servers, VoIP, grid computing,server virtualization ….

GreenServer

Skype
ServerCluster
VirtualizedServerWindows
SolarisLinuxSMP

SPEC follows this evolution.

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

Web Server

3

SPEC in a New Era
How do you choose suitable systems for this new era ?

Go look on the SPEC web site &compare the published results !
Acquire & run the SPEC benchmark codes !

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

4

SPEC in a New Era
Mission Statements
 Develop metrics that allow end-users to differentiate
hardware and software and find the most suitable systems torun their applications.
 Evolve with the marketplace in order to stay useful for endusers.

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

5

SPEC in a New Era
SPEC develops software that helps you in finding the bestsystem for:
 High Performance Applications on Single and Multiple CPUs
 Low Power Consumption
 Graphics Applications
 Server Virtualization (under development) Instant Messaging (under development) Mail and Web Servers

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

6

SPEC in a Nutshell
What is the Standard Performance Evaluation Corporation ?
 A world-wide non-profit consortium formed 20 years ago to establish,
maintain and endorse a standardized set of relevant benchmarks that canbe applied to the newest generation of high-performance computeequipment.
 Over 80 computer hardware and software vendors, and educational
institutions from all over the world.
 Development of benchmark suites to ensure that the marketplace has a fairand useful set of metrics to differentiate systems.
 Review and publication of submitted results produces a large public
repository of well documented, peer reviewed, benchmark results.

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

7

SPEC is a global organization

RussiaCanadaGermany, France,Italy, Poland, Finland,Sweden, Austria, UK,Belgium, Bulgaria

EUUSJapan
China
TaiwanPhilippines

BrazilAustralia

New Zealand

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)8

SPEC Groups
 High Performance Group (HPC systems)
o OMP (OpenMP benchmark)o MPI (MPI application benchmark) Open Systems Group (desktop systems, high-end workstations and servers)o CPU (CPU benchmarks)o JAVA (java client and server side benchmarks)o MAIL (mail server benchmarks)o SFS (file server benchmarks)o WEB (web server benchmarks)

 Graphics Performance Groups (Graphics)o Apc (Graphics application benchmarks)o Opc (OpenGL performance benchmarks)

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)9

Low Power Consumption
SPECpowerTM
 First industry standard benchmark that measures the power andperformance characteristics of server-class compute-equipment.
 SPEC’s initiative to augment many existing SPEC benchmarks with power
measurements and assist other non-profit industry standards organization.

SPEC Power and Performance Methodology
 An introduction on power and performance for computer systems, theconclusions from the 2 year development of SPECpower_ssj2008
 Guidance for Power and Performance benchmark development

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

10

SPECpowerTM
SPECpower_ssj2008 - Framework
Benchmark Harness - Framework
 Ensures the synchronization of the measured performance, power, andenvironmental data.

PTDaemonController
Local TCP/IPsocket
communication

CCS
Communication
via TCP/IP

PT
DaemonCommunication
via RS-232
or USB

Temperature
Sensor

AC Power In

Workload

Power Analyzer

SUTPower to SUT

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

11SPECpowerTM
SPECpower_ssj2008 – Variable Utilization
Variable System Utilization
SPECpower Workload Iteration
100%
90%
80%
70%
Interval Interval Interval
12
3

60%

Power

50%
40%
30%
20%
10%
Active
Idle
Calibration Intervals

22 October 2008

Measurement Interval (Target Loads)

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

12SPECpowerTM
SPECpower_ssj2008 metric

Performance 1,861,107 ssj_ops
ssj_ops∑
SPECpower_ssj2008 === 265 overall
7,010 wattwatt∑ PowerPower and Performance at mutltiple Target Load Levels (DL360)
1000

350,000

900
800

250,000

700
600

200,000500
150,000

400
300

100,000

200

50,000

100
0

0
100%

22 October 2008

Power [watt]

Performance [ssj_ops]

300,000

90%

80%

70%

60%

50%

40%

30%

20%

10%

active idle

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)13

SPECpowerTM
SPECpower_ssj2008 on PRIMERGY

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

14

Compute Intensive Applications
 Integer and/or floating-point applications
Use SPEC CPU2006
 OpenMP-based floating-point applications
Use SPEC OMP2001
 MPI-based floating-point applications
Use SPEC MPI2007

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)15

SPEC CPU2006
Consists of two benchmark suites: CINT2006
 CFP2006

Large number of submissions since August 2006:
 803 for CINT2006
 743 for CFP2006
 1916 for CINT2006 Rates
 1534 for CFP2006 Rates

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)16

SPEC CPU2006 Metrics
CINT2006 (for integer based performance comparisons): The geometric mean of twelve normalized ratios under peak or base tuning.
 The geometric mean of twelve normalized throughput ratios under peak or basetuning.

CFP2006 (for floating-point based performance comparisons):
 SPECfp2006: The geometric mean of seventeen normalized ratios with peak orbase tuning.
 SPECfp_rate2006: The geometric mean of seventeen normalized throughput
ratios with peak or tuning.
The geometric mean of a data set [a1, a2, a3,..., an] is given by:(a1 * a2 * a3 * ... * an)1/n
The geometric mean of a data set is less than or equal to the data set's arithmetic mean.

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)17

SPEC CPU2006 on PRIMERGY

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)18

SPEC OMP2001 and MPI2007

MPI2007

22 October 2008

OMPM2001

OMPL2001

June 2001

Jan 2003

July 2007

MPI2007 v1.1

Oct 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)

19

SPEC OMP2001

SPEC OMPM2001 is focused on 4-way to 16-way systems
SPEC OMPL2001 targets 32-way and larger systems
Number of submissions to date
 217 for OMPM2001 (since June 2001 release) 56 for OMPL2001 (since May 2002 release)

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)20

Scientific Areas Computational Fluid Dynamics(applu, galgel) Quantum Chromodynamics(wupwise) Air Pollution (apsi)
 Image recognition (art) Crash simulation (fma3d)
 Genetic algorithm (gafort)
 Earthquake modeling (equake) Weather prediction (swim)
 Multigrid solver (mgrid) Molecular Dynamics (ammp)
22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)21

SPEC OPM2001 Metrics
SPEC OMPM2001
 SPECompMpeak2001: The geometric mean of 11 normalized ratios (peak tuning).
 SPECompMbase2001: The geometric mean of 11 normalized ratios (base tuning).

SPEC OMPL2001
 SPECompLpeak2001: The geometric mean of 9 normalized ratios (peak tuning).
 SPECompLbase2001: The geometric mean of 9 normalized ratios (base tuning).

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)22

SPEC OMP2001 on SPARC Enterprise

Overall Result
Detailed Results
Hardware Description
Software Description
Notes/Tuning Information

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)23

Multicore Chips
SPEC’s view on fair comparisons of systems with multicore
chips:
Ask submitters to specify full system description in thesubmissions and in the marketing press releases:
 number of chips
 number of cores number of cores per chip number of threads per core

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)24

SPEC MPI2007
SPEC MPI2007 focuses on performance of applications using
the Message-Passing Interface (MPI), up to 128 ranks.
The following components are emphasized:• the type of computer processor (CPU),• the number of computer processors,• the communication interconnect,• the memory architecture,• the compilers, and• the shared file system.

Number of submissions to date
 61 since July 2007 release
22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)25

Scientific Areas Computational Fluid Dynamics(leslie3d, fds, zeusmp, pop2)
 Quantum Chromodynamics (milc)
 Weather Forecasting (wrf) Parallel Ray Tracing (tachyon)
 Molecular Dynamics (lammps)
 Heat transfer (geofem - codeTokyo Univ) Hydrodynamics (tera_tf)
 Matrix Decomposition (lu)
 Density Functional Theory(socorro)
22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)26

SPEC MPI2007 Metrics
 SPECmpiM_base2007:
The geometric mean of thirteen normalized ratios (base tuning).
 SPECmpiM_peak2007:
The geometric mean of thirteen normalized ratios (peak tuning).

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)27

Virtualization
Virtualizing a number of servers on a single platform helps toreduce costs, save energy and ease IT infrastructuremanagement.

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)28

Virtualization
 In order to help end-users in selecting an optimal system for server
virtualization, a virtualization benchmark is being developed.
 Heterogeneous workloads that are spread across multiple virtual machines
on a single server will be used. A key aspect will be defining amethodology to model the dynamic nature of customer workloads in thisenvironment.
 Goals are:□ Single primary metric□ Scale across a wide range of systems□ Component workloads representing common application categories typical ofvirtualized environments.

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)29Single Tile DesignTILE
InfrastructureServer VirtualMachine

Web Server
IMAP Server
Virtual Machine Virtual Machine

Application
Server VirtualMachine

Database
Server VirtualMachine

Idle Server
Virtual Machine

SMTP Sink
BeSIM

Web Server
Application

IMAP Server
SoftwareApplication
Server Software

Database
Server Software

Guest OS

Guest OS

Guest OS

Guest OS

File Sharing
SoftwareGuest OSBlue = disk IO

Virtualization Layer and Hardware

Guest OSGreen = network IO

SUTClient Hardware
SPECweb2005 driver

SPECimap2007 driver

SPECjappserver driver

Conclusions
 SPEC is continually evolving and adapting to the marketplacein order to help end-users find the most suitable systems fortheir needs.
 If you are looking for systems to run compute-intensive
applications on single CPU, SMP or cluster & have low power
consumption, compare them on the SPEC web site.
 Future developments are for finding the most efficientvirtualized servers, the most efficient systems for VoIP andinstant messaging.

22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)31

SS-ken October 2008

Thank you for your attention !
22 October 2008

SPEC and the SPEC logo are registered trademarks of the Standard Performance Evaluation Corporation (SPEC)32