Benchmarking Application I / O in the Community

Benchmarking I/O performance of a high performance computer is a tough task. Many sequential and parallel benchmarks exist, however they differ in access patterns, tested interfaces and internal behavior. Interpretation of results on one hand depends on the benchmark, on the other, projection of obtained results to particular applications is not easy or even impossible. Porting scientific applications to an architecture is complicated, library and architecture dependencies are a burden to developers. Therefore, a few application benchmarks exist, which try to mimic application behavior in a small core. However, writing an individual application kernel as a new benchmark for each application leads to a wide diversity in benchmarks. We propose the benchmarking tool Parabench, which allows to mimic a rich variety of application programs. Parabench interprets an easy programming language during runtime to avoid portability or licensing problems. In addition, a future TraceReplay mechanism will allow to replay application traces directly on other systems by interpretation of the traces. Standardized tests of the results ease evaluation and futher result comparison. We will start an open community and collaborations to exchange patterns and measured results on the website. Vendors can take the access patterns and provide results, obtained on new architectures and systems. 1. BENCHMARKING APPLICATIONS In this poster we present the programmable I/O benchmark Parabench [9]. Parabench allows to mimic I/O behavior of a wide range of applications. Additionally, the concept of Trace-Replay is introduced. Trace-Replay allows to execute traces of applications on different systems. Typically, application traces are used for performance optimizations, however an interpreter could reproduce the traced behavior. Trace-Replay will allow to record communication, I/O and quantitative CPU activity i.e. CPU counters and replay the behavior. While there is no implementation so far, we will prepare an implementation. Compared to run an application directly on a system this technique allows to record runs of real applications. These traces can be shared and run without dependency on particular libraries or systems, or without dealing with licensing issues. Without the need to port the scientific application to the new system, this technique offers to record inefficient application runs or store runs as a gold standard. Developers of middleware can use these traces to optimize their software-stack. Depending on the implemented tracing, calls to third party libraries such as HDF5 could be traced and optimized to the same extent. Both Parabench and Trace-Replay check return values of calls to ensure proper benchmarking – by providing a core of functions with correct error handling derived benchmarks will avoid common mistakes in benchmarking. Finally, the exchange of patterns and traces in an open community will allow third parties to check the performance of their systems under realistic conditions. With such an open platform we hope the community will share patterns of common applications to provide a rich and diverse repository for evaluating current and future systems. Versioning of trace files will allow to compare old program version with new version without providing the outdated and even incompatible environment. We are looking for further partners to drive the ideas forward for the sake of the community.