Profiling and Tracing in Linux

Profiling and tracing tools can help mak e application parallelization more ef f ctive and identify performance bottlenecks. Profiling presents summary statistics of performance metrics while tracing highlights the temporal aspect of performance v ariations, sho wing when and where in the code performance is achie ved. A complex challenge is the mapping of performance data g thered during e xecution to high-le vel parallel language constructs in the application source code. Presenting performance data in a meaningful w ay to the user is equally important. This paper presents a brief overview of profiling and tracing tools in the conte x of Linux the operating system most commonly used to build clusters of workstations for high performance computing. back to source-code constructs and entities that the user can comprehend is important. As shown in Figure 1. , instrumentation can be added to the program at an y stage during the compilation/e xecution process from user -added calls in the source code to a completely automated approach. (Ev ents are triggered by the execution of the instrumentation instructions at runtime.) Moving down the stages from source code instrumentation to runtime instrumentation, the instrumentation mechanism changes from language specific to platform specific. Making an instrumentation API specific to a language for source code instrumentation may make the instrumentation portable across multiple compilers and platforms that support the language. However, the instrumentation may not w ork with other languages. T argeting the instrumentation at the e xecutable level ensures that applications written in an y language that generates an e xecutable would benefit from the instrumentation on the specific platform. Ho wever, it is more dif ficult to map performance data back to source-code entities, especially if the application’ s highlevel programming language (such as HPF or ZPL), is translated to an intermediate language (such as F ortran or C) during compilation. There is often a trade-of f between what le vel of abstraction the tool can pro vide and how easily the instrumentation can be added. The JEWEL [19] package requires that calls be added manually to the source code. While pro viding flexibility , the process can be cumbersome and time consuming. A preprocessor can generate instrumented source code by automatically inserting annotations into the original source. This approach is tak en by SvPablo [14] and AIMS [20] for C and Fortran, and by T AU [15] with PDT for C++ [18]. Instrumentation can be added by a compiler as in gprof [5] (with the -pg commandline switch for the GNU compilers). Instrumentation can also be added during linking when the e x cutable image is created by linking multiple object files and libraries together . Packages that use this method include V ampirTrace [13], and PICL [12] for MPI and PVM message passing libraries. When an inter -process communication call is executed, the instrumented library acts as a wrapper that calls the profiling and tracing routines before and after calling the corresponding uninstrumented library call. This is achie ved by weak library bindings. However, often program sources and libraries are unavailable to the users and it is necessary to e xamine the performance of e x cutable images. Pixie [16] and Atom [17] are tools that re write binary executable files while adding the instrumentation, b ut are not currently available under Linux. All these approaches require the programer to modify the application, re-compile and re-e xecute the instrumented application to produce the performance data. Sometimes, re-e xecution of an application using the performance analysis tool is not a viable alternati ve while searching for performance bottlenecks. This is true for long-running tasks such as database serv ers. Paradyne [9] automates the search for bottlenecks in an application at runtime by selecti vely inserting and deleting instrumentation using the DynInst [7] dynamic instrumentation package. After instrumentation, we e x cute the program and perform measurements, in the form of profiling and tracing and analyze the data.