On-Line Automatic Resource Configuration for Distributed Applications

A key problem in executing performance critical applications on distributed computing environments (e.g. the Grid) is the selection of resources. Research related to "automatic resource selection" aims to allocate resources on behalf of users to optimize the execution performance. However, most of current approaches are based on the static principle (i.e. resource selection is performed prior to execution) and need detailed application-specific information. In the paper, we intro- duce a novel on-line automatic resource selection approach. This approach is based on a simple control theory: the application continuously reports the Execution Satisfaction Degree (ESD) to the middleware Application Agent (AA), which relies on the reported ESD values to learn the execution behavior and tune the computing environment by adding/replacing/deleting resources during the execution in order to satisfy users' performance requirements. We introduce the Desired Processing Capability Estimation learning policy to enable AA to learn and tune the computing environment based on the approach. A Mandelbrot computation and a Bulk Synchronization Parallel (BSP) compu- tation are performed to show the effectiveness of the approach. the middleware must select resources prior to commencing the execution. The selection process ends once execution starts. This procedure works fine in the high performance computing where applications are batch jobs and resources are mainframe or homogenous computers. However in the modern distributed computing, an application can be dynamic and its execution behavior could not be predicted in advance, i.e. its resource requirements may change in the middle of execution. Such applications are specially common in astrophysics, mineralogy and oceanography. The applications are called dynamic be- cause they allow significant changes to be made to the structure of the datasets themselves when necessary. Consequently they may require resources dynamically during the execution to meet their performance benchmarks. CFDOG (Computational Fluid Dynamics-Object Graph) (6) is one of the examples in that respect. Moreover, the resource environment could also be dynamic. The required resources may be unavailable and other eligible resources may become available during the execution. Therefore it is hard to plan resource allocation in advance. Another limitation of existing approaches is that they are heavily based on the information principle. Those approaches require users to provide more or less information about the applications to obtain optimized performance, such the communication topology and I/O patterns. However some users may have no knowledge of the parallel applications and the resources. In this paper, we introduce a novel resource selection ap- proach which breaks the rigid process of traditional approaches to provide high resource management transparency to the users. This approach is performed on-line by adding, replacing and deleting resources during application's execution rather than statically selecting resources before commencing execu- tion. Different from the above static approaches, it doesn't need to know the exact execution behavior/characteristics of the application. The application is simply treated as a black-box. The approach listens to the execution performance report (output of the black-box) from the application and according to the report it intelligently decides to re-configure execution environment (input of the black-box) to approach the performance requirements. While inside of the black- box, the application autonomously reforms itself to adapt to the provided execution environment. We provide a specific middleware Application Agent (AA) to support the approach. Specially, we introduce the Desired Processing Capability Estimation policy applied to the approach to achieve the