# An Asynchronous Parallel I/O Framework for Mass Conservation Ocean Model
## 3 Motivation  
At the beginning, MaCOM used PnetCDF to implement I/O functions as MOM (Modular Ocean Model) [22] and sbPOM (Stony Brook Parallel Ocean Model) [23]. Because PnetCDF does not support asynchronous I/O, computation and I/O are executed in the same processes. For the I/O settings, the output frequency is once every 2 h, and the output variables include 6 1D variables (with reduced dimension from horizontal 2D after data optimization), including sea surface height and 5 2D variables (with reduced dimension from horizontal and vertical 3D after data optimization), including sea temperature, salinity, and others. The data volume of each 1D variable is 43.9 MB and 3.3 GB for each 2D variable. The breakdown of the MaCOM wall time with the global 1/12-degree grid resolution for a 7-day forecast with a 2-h output frequency is shown as Figure 2. As the parallelism increases, the performance of the MaCOM is dominated by the I/O efficiency. For example, I/O takes 10.6% of the whole runtime with 128 processes, but it reaches 40.6% with 1024 processes. With an increasing number of computing processes, the computing cost decreases proportionally, while the I/O cost declines slowly compared to computation. Due to the I/O speed limit of a file system and competition to access the same file from all processes, writing data into one file will be a bottleneck for synchronous I/O with massive processes. That is a general performance problem for synchronous parallel I/O. To solve this problem, asynchronous I/O with independent I/O processes implemented in PIO, XIOS, or CFIO is an effective method [24; 25; 26]. In oceanic and meteorological models, the prognostic variables, including temperature, salinity, and three velocity components (zonal, meridional, and vertical), are numerically integrated forward in time by solving the governing equations. This allows for overlapping the storage of computing results from the preceding time step with the dynamical integration of the subsequent time step. This feature enables MaCOM to utilize asynchronous I/O, thereby improving its performance. Meanwhile, to facilitate post-processing of output of MaCOM, the output of all variables from I/O processes in the same period should be written into a single NetCDF file. MaCOM is an operational model that runs daily on a high-performance computer to provide global operational service. To improve the timeliness of this model and save computing resources, maximum improvement in I/O performance is required since it is a bottleneck during the execution of the model. Non-blocking communication in MPI is an effective way to save communication time when sending output data [27; 28; 29]. However, to work with non-blocking communication in MPI, there should be a method to avoid the buffer of sending output data in the current I/O step from being written by the next I/O step before completing communication in the current I/O step. Unfortunately, there is no flexible library including NetCDF, PnetCDF, PIO, and CFIO to meet the requirement of sending I/O data with non-blocking communication and protecting the buffer of sending output data from being overwritten incorrectly by the next step of sending output data. Therefore, it is necessary to develop an effective communication method to meet the above requirements of non-blocking communication for higher I/O performance. Due to the unstructured nature of grid decomposition in MaCOM, I/O data is scattered in different processes. Therefore, it can improve output performance after reordering irregular and discontinuous output data into continuous data. It is also convenient for data post-processing. However, there are no related functions to reorder output data according to the user-defined order in the NetCDF, PnetCDF, PIO, XIOS, and CFIO libraries. Combining I/O operations for multiple array variables has been proven to be a way to improve I/O performance [15]. Unfortunately, this optimizing method is not currently implemented in current NetCDF libraries.