Collaborative Research: Elements: ROCCI: Integrated Cyberinfrastructure for In Situ Lossy Compression Optimization Based on Post Hoc Analysis Requirements PIs: Sheng Di1,2, Franck Cappello1,2, Dingwen Tao3 1University of Chicago, 2Argonne National Laboratory 3 Indiana University Motivation Libpressio Z-Checker/QCAT Objectives This project aims to develop a Requirement-Oriented Compression Cyber-Infrastructure (ROCCI) for data-intensive domains such as astrophysics and materials science. The core purpose is practicality, enabling scientists to efficiently assess and/or optimize the lossy compression quality based on their post hoc analysis requirements. This project advances knowledge with three key thrusts: • It builds an efficient layer to interoperate with different lossy compressors and diverse post hoc analysis requirements on data fidelity by leveraging our developed compression adaptor library (LibPressio) and compression assessment library (Z-checker). • It develops an efficient engine to determine the best compressor with automatically tuned optimized settings. • It develops a user-friendly infrastructure integrating compression optimization and execution via HDF5 dynamic filter mechanism. Acknowledgement Z-checker is a parallel framework for assessing lossy compression of scientific data efficiently. QCAT is the corresponding lightweight utility library and toolkit. This research is supported by NSF OAC # 2104023 and 2104024. We acknowledge the computing resources provided on Bebop (operated by Argonne Laboratory Computing Resource Center and on Theta and JLSE (operated by Argonne Leadership Computing Facility). Publications Estimation error and cost of SECRE for SZx: When using a large sampling stride such as 40 (sampling rate would be 1/41≈2.43%), the estimation error is still very low less than 1% in the worst case). System Architecture of ROCCI with improved & innovative modules. We are towards the end of the second year of the project. In the first two years, we have 20+ papers published in top conference/journal papers, including SC, ICDCS, ICDE, IPDPS, HPDC, ICS, VLDB, EuroSys, IJHPCA, HiPC, etc.. Major Accomplishments First year: We improved Libpressio to support 30+ compressors (both lossy and lossless, both CPU and GPU) and Z-checker/QCAT to support 40+ compression assessment metrics (including ratio, speed, max error, PSNR, SSIM, derivatives, distribution of errors, auto- correlation of errors, visualization, etc.) Second Year: We explored a series compression quality estimation methods, by leveraging machine learning (ML), statistical analysis, compression-principle analysis (surrogate), sampling method, etc. We also improved the performance for Libpressio, Z-checker/QCAT, SZ3, SZx, and developed new capabilities such as QoZ, FAZ. Libpressio-Predict Compression Ratio Estimation Evaluation using 4 real-world science datasets including climate simulation, cosmology research, large-eddy simulation, etc. LibPressio provides a productive interface for integrating compression. It eases experiment with new compressors with less code and complexity. Compression developers benefit from using a common implementation for a parallel compression; a CLI; Python, Rust, Julia, C, C++ bindings; and HDF5, ADIOS2, and other integrations that are complete and consistent with just 1 C++ class. 1 API 32+ compressors Builtin Provenance Support Optional Automatic Configuration and Optimization CPU Parallel and GPU Parallel Compressors Non-measurable overhead CLI, Python, Rust, Julia, C, and C++ bindings LibPressio addresses all these issues by providing a unifying interface with advanced engines for provenance and configuration optimization. LibPressio-predict integrates quite a few compression ratio and metric estimation algorithms for error-bounded lossy compression, including FXRZ, SECRE, Stats-based compresibility estimation method, etc. We have developed multiple efficient compression quality estimation methods and error-con