# Presentation: 429000

## NGF07 Benchmarks

**NGF07 Benchmarks**
** ****David Turner**
- NERSC User Services Group
**Oakland Scientific Facility**
**14 May 2007**

## NGF07 Website

**http://www.nersc.gov/projects/procurements/NGF07/**
- Complete Request For Proposal Package
- Windows “zip” file containing PDF files
- RFP
- Benchmark rules
- “Legal-ise” (sample subcontract, general provisions, etc.)
- Benchmarks
- Run Rules
- MS Word and PDF
- IOR and METABENCH
- gzipped tar files **ior.tgz** and **metabench.tgz**

## NGF07 Website Continued

**http://www.nersc.gov/projects/procurements/NGF07/**
- Hardware Inventory
- Current hardware that should be re-used
- HTML
- MS Word
- Plain text
- Questions and Answers
- Send questions to **ngf07@nersc.gov**
- General interest q/a will be posted
- Slides for May 14 Pre-Proposal Conference
- PowerPoint

## Benchmarks in RFP

- MINIMUM REQUIREMENTS
- Performance
- The facility-wide global file system vendor must:
- Commit to the delivery of sustained aggregate I/O performance as measured by the benchmarks described in the Benchmark Instructions.
- PROPOSAL PREPARATION INSTRUCTIONS
- In the technical proposal, the Offeror shall provide:
- All requested benchmark documentation detailed in the Benchmark Instructions attached to this RFP
- ENCLOSURES
- Result requested in the Benchmark Instructions
- NERSC expects vendors to make “best effort” in providing requested benchmark results

## Benchmarks in SoW

- Benchmark performance levels maintained through life of subcontract
- Additional benchmarks possible, subject to negotiation

## Benchmark Rules

- Run on a small range of processors
- Fully packed
- Obtained from a file system configured as closely as possible to the proposed system
- If the proposal includes multiple file access protocols (e.g., pNFS or NFS), results should be provided for _all_ such protocols
- Performance projections are allowed if they are derived from an “earlier generation” system
- Rigorously derived, easily understood and thoroughly documented
- Modifications only permissible for correct execution
- Must be fully documented and reported
- No changes related to optimization

## Additional Rules

- Intent is to measure performance for that access disk
- Benchmarks utilize smaller working set sizes
- Optimizations may be possible that enable caching or buffering within system memory
- Such optimizations may be beneficial in a production environment
**Please do not engage in such ****optimizations for the benchmark runs!**

## Submission Guidelines

- Tables provided for results
- Results on paper with RFP response
- Results on CD with completed tables, benchmark source and output files
- Source shall be in a form that can be readily compiled
- Do not include object and executable files
- If performance projections are used
- Clearly indicate
- Include output files on which projections are based
- Describe of the projection method

## Additional Guidelines

- Document benchmarking environment
- Client and server configurations, including node and processor counts, processor models, memory size and speed, and OS (names and versions)
- Storage used for global file system and storage configurations
- Network fabric used to connect servers, clients, and storage, including network configuration settings
- Client and server configuration settings, including client and server sysctl settings, driver options, network interface options, and file system configuration options
- Compiler name and version, compiler options, and libraries used to build benchmarks

## The Benchmarks

- IOR
- Developed by Scalable I/O Project at LLNL
- Requires MPI and MPI-IO
- Download version IOR-2.9.3 from
- http://www.llnl.gov/icc/lc/siop/downloads/download.html
- The bandwidth numbers to be reported are the results listed as “Max Write” and “Max Read” measured in “MB/s”
- For more information on IOR , please read the User Guide included with the source distribution
- METABENCH
- MPI program that measures performance of various metadata operations.

## IOR Definitions

- blocksize: size (in bytes) of a contiguous chunk of data accessed by a single client; it is comprised of one or more transfers.  The blocksize is fixed at 1 GB for all multi-client tests.
- transferSize: size (in bytes) of a single data buffer to be transferred in a single I/O call.  Four transfer sizes are use for all multi-client tests: 1 KB, 16 KB, 256 KB, and 4 MB.
- File access: either a single file per MPI process is read/written (“unique”) or a single shared file for all processes is read/written (“shared”).

## IOR Runs

- 65 runs are required
- POSIX API
- Shared file: 4, 16, 32, and 64 processes, transfer size = 1 KB, 16 KB, 256 KB, and 4 MB
- Unique file: 4, 16, 32, and 64 processes, transfer size =  1 KB, 16 KB, 256 KB, and 4 MB
- Unique file: 1 process, blocksize = 8 GB, transfer size = 128 MB
- MPI-IO API
- Shared file: 4, 16, 32, and 64 processes, transfer size = 1 KB, 16 KB, 256 KB, and 4 MB
- Unique file: 4, 16, 32, and 64 processes, transfer size =  1 KB, 16 KB, 256 KB, and 4 MB

## METABENCH Runs

- Three sets of circumstances:
- Creating 30,400 files by one process in one directory;
- Creating 30,400 files by $NP processes in the same directory;
- Creating 30,400 files by $NP processes in separate directories, one directory per process
- $NP will be either 2, 4, 16, or 32
- Other metadata operations (stat and removal) are also timed

- Questions?