AstroCompress: A benchmark dataset for multi-purpose compression of astronomical data

Authors: Tuan Truong, Rithwik Sudharsan, Yibo Yang, Peter Xiangyuan Ma, Ruihan Yang, Stephan Mandt, Joshua S. Bloom
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: astronomy, physics, astrophysics, compression, neural compression, computer vision, remote sensing

Abstract:
The site conditions that make astronomical observatories in space and on the ground so desirable---cold and dark---demand a physical remoteness that leads to limited data transmission capabilities. Such transmission limitations directly bottleneck the amount of data acquired and in an era of costly modern observatories, any improvements in lossless data compression has the potential scale to billions of dollars worth of additional science that can be accomplished on the same instrument. Traditional lossless methods for compressing astrophysical data are manually designed. Neural data compression, on the other hand, holds the promise of learning compression algorithms end-to-end from data and outperforming classical techniques by leveraging the unique spatial, temporal, and wavelength structures of astronomical images. This paper introduces [AstroCompress](https://huggingface.co/AstroCompress): a neural compression challenge for astrophysics data, featuring four new datasets (and one legacy dataset) with 16-bit unsigned integer imaging data in various modes: space-based, ground-based, multi-wavelength, and time-series imaging. We provide code to easily access the data and benchmark seven lossless compression methods  (three neural and four non-neural, including all practical state-of-the-art algorithms).
Our results on lossless compression indicate that lossless neural compression techniques can enhance data collection at observatories, and provide guidance on the adoption of neural compression in scientific applications. Though the scope of this paper is restricted to lossless compression, we also comment on the potential exploration of lossy compression methods in future studies.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729591940715

thread_type:
official_review

<thread_object>
creator: reviewer_hypn

<content>
summary:
This paper introduces a benchmark dataset designed for multi-purpose compression of astronomical imagery. A range of image compression codecs is tested and evaluated through comprehensive experiments. The benchmark dataset can serve for researches in astronomy-specific and science-specific lossless neural compression by providing a carefully curated, accessible dataset, along with easily referenceable benchmarks.

strengths:
1. The released dataset is well organized and large in scale and diversity to cover a wide range of astrophysics data.
2. The tested anchors are representative, including both popular learned and conventional codecs.

weaknesses:
1.	Two different methods for handling 16-bit images are discussed and tested. It is recommended that the performance differences between these methods be highlighted.
2.	The proposed dataset’s effectiveness, especially in comparison to existing datasets such as Maireles-González et al. (2023), could be explored more thoroughly.
3.	The performance of RGB-pretrained models on the AstroCompress dataset are suggested to be examined and reported.
4.	The implementation details regarding correlated-frame compression could be clarified further. Specifically, it should be explained whether different frames are compressed independently for each tested anchor, or if a different approach is used.

questions:
As an important aspect in image compression, some preliminary experiments and evaluation on lossy compression are expected.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730224538764

thread_type:
official_review

<thread_object>
creator: reviewer_hihc

<content>
summary:
The paper describes a new dataset for benchmarking neural compression on astronomical data. The paper lays out how the efficacy of astronomical experiments is highly dependent storage and transmission, and that the fidelity of the data must be preserved, motivating the use of neural lossless compression. The dataset itself is apparently unprecedented in scope for astronomical data with five sub-datasets including a mix of 2D still images as well as spatio-temporal and multispectral hypercubes of high resolution. The paper concludes by testing a series of well know neural and non-neural lossless compression algorithms on the dataset.

strengths:
The paper makes a strong contribution to astrological and neural compression research. The paper lays out clearly how astrological experiments depend on good compression and why a dataset specific to astrological data is necessary. The scope of the dataset is also impressive. It is likely that a researcher working on neural compression, lossless or otherwise, would be interested in findings from testing their method on this dataset. Likewise I can imagine astrological researchers using results on this dataset to decide what methods to deploy. Finally the assembly procedure and details in Appendix A shows that great care was put into the curation of the data.

weaknesses:
The only thing which I would have liked to see more discussion on was the point about runtime/compression ratio. Is there any actionable recommendation here? Also given that JPEG-XL is mostly a reference implementation right now and hasn't been optimized while the neural methods require a GPU, is it  fair comparison to say that it is slower?

questions:
* Any actionable recommendation around neural/classical compression based on results?
* Are resource requirements (GPU, memory) factored into speed results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730262852886

thread_type:
official_review

<thread_object>
creator: reviewer_ktqy

<content>
summary:
This paper provide a dataset and benchmark for astronomical data compression. The dataset is relatively large (320 GB). Besides, the dataset composes 5 distinct subsets, which covers images of different dimension, resoluton and content types. The authors evaluate the datasets on different lossless compression methods, including IDF, L3C, PixelCNN++, JPEG series and RICE.

strengths:
The dataset composes of diverse images, from 2D to 4D, which is impressive. The selective of neural lossless codec baseline is smart. In fact. IDF, L3C and PixelCNN++ represent three major paradigms of lossless compression: normalizing flow, latent variable model and auto-regressive model. In fact, I can not think of a better choice if I need to choose three most representative lossless image codec.

weaknesses:
One most evident weakness is that the authors evaluate 7 general lossless codec and only 1 astronomical codec (published in 2009). I have two explainations for this evaluation:
* The authors's evaluation is not thorough, many astronomical codecs are omitted.
* The astronomical data compression is not an active research area, and the only reasonable baseline is published 15 years ago.
Either of the explaination makes me hesitate about accepting this paper.

Besides, I am not really sure about how important astronomical data compression is to ICLR community. From the BACKGROUND AND RELATED WORK section in this paper, I find that most papers of astronomical data compression are published in remote sensing and compression related venues, not machine learning related venues (NIPS, ICLR, ICML, CVPR, ICCV, ECCV, etc.). I am not really sure about:
* How many people in ICLR community is interested in astronomical data compression?
* How many people in astronomical data compression community will read ICLR?

questions:
It is the first time that I ever review a benchmark paper. I am not definitely sure about my evaluation. And I am glad to change my mind if other reviewers are positive toward this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730432550246

thread_type:
official_review

<thread_object>
creator: reviewer_ieqb

<content>
summary:
This paper introduces AstroCompress, a benchmark dataset designed for multi-purpose compression of astronomical images, providing a foundation for advancing astronomy-specific and science-oriented lossless neural compression. Through comprehensive experiments, various image compression codecs are tested and evaluated.

strengths:
1. The released dataset is made open-accessible and well-organized for researchers to follow.
2. Several lossless compression codecs have been tested including both learned/traditional methods. Evaluations of the impact on datasets and methods are presented.

weaknesses:
1. Since many of the datasets are from publicly accessible resources, therefore i suggest that the authors should contribute more over the analysis and deeper evaluation of the re-organized AstroCompress dataset.
2. The cross-dataset evaluation can be improved. In this paper, only brief conclusions about the generalization of different codecs are provided. A deeper evaluation of the similarity between different datasets and the reason for performing differently for different codecs is suggested.

questions:
If there also exists some codecs specially designed for astro data, the authors should also provide some evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733037288379

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General remarks

comment:
We thank all reviewers for their insightful and constructive feedback which has helped strengthen our submission. We are encouraged that reviewers found our dataset “well-organized” [**Reviewer iEQb31**], “unprecedented in scope for astronomical data” [**Reviewer hihc**], “open-accessible and well-organized for researchers to follow” [**Reviewer iEQb**], supported our choice of the “three most representative lossless image codec” which “represent three major paradigms of lossless compression” [**Reviewer kTqy**], and overall valued our paper as “a strong contribution to astronomical and neural compression research” [**Reviewer hihc**].

The reviewers have also raised a few concerns which we address in our revised manuscript. Before we respond to the reviewers’ concerns individually, we summarize our main edits to the manuscript:
1. We performed a new experiment to better understand the cross-data generalization performance of neural compression methods, where we trained IDF on all the single-frame datasets combined. The new results are in Section 4.5 and show that the ensemble approach yields significantly more robust cross-data generalization performance than training on individual datasets alone. 
2. We performed a new empirical analysis in Appendix Section D investigating the compressibility of different datasets. Our results reveal that different datasets have widely varying characteristics of noise in the background pixels, and the amount of background noise is intimately related to compression performance across the datasets. This analysis, together with the new experimental results above, contributes a deeper evaluation of the performance of the different codecs on our datasets as requested by **Reviewer iEQb**.
3. In response to **Reviewer hYPn**’s request, we added additional results in Appendix C.1 comparing the compression performance using the two different proposed methods for handling 16-bit data.
4. In response to **Reviewer kTqy**’s concern about the relevance of our contribution, we updated our Introduction section to provide stronger motivation and evidence for the urgent need for research in astronomy data compression. We refer to a recent audit by NASA on the upcoming Roman space telescope project, which raised the issue of data transmission as their single greatest concern [NASA OIG, 2024].
5. To demonstrate the strong potential of neural compression methods, we experimented with a new state-of-the-art likelihood-based model, Variational Diffusion Model (VDM) [Kingma et al., 2021]. While this method is currently too slow to be a practical codec, its estimated compression ratio dominates all the neural and non-neural methods considered on most datasets, often by a wide margin (e.g., 22% improvement on the LCO dataset), which suggests significant room for future improvement. We include our latest results for VDM on the single-frame datasets in the top portion of our updated Table 1 and expect to finish evaluating VDM on the remaining datasets soon.

Lastly, we want to emphasize the broader significance of neural compression for scientific data and our contribution. Scientific data volumes are experiencing exponential growth, creating a critical need for uniquely advanced compression techniques. Unlike internet-focused compression, scientific domains require fast on-device encoding, minimal reconstruction error, and handling of noisy data. Astronomy presents a particularly compelling case due to its publicly available, vast datasets and unique financial implications, where compression improvements can translate to hundreds of millions of dollars in mission value. We believe this work will be of interest to the broader ICLR and AI for science communities, contributing to the ongoing dialogue on machine learning approaches to real-world scientific computing challenges. We hope to spark a research trajectory towards powerful, practical scientific neural codecs deployed in the real world.

---
References

[NASA OIG, 2024] “Audit of the Nancy Grace Roman Space Telescope Project.” NASA Office of the Inspector General, 31 July 2024, oig.nasa.gov/wp-content/uploads/2024/07/ig-24-014.pdf.

[Kingma et al., 2021] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational Diffusion Models. NeurIPS, 2021.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new dataset designed for benchmarking both neural and non-neural compression methods in the context of astronomical data. The dataset is representative of real-world use cases in astronomy and includes a selection of SOTA neural and non-neural methods. Its well-organized structure makes it easy to access and facilitate the benchmarking of new compression algorithms. Reviewers recognized the dataset as a valuable resource for advancing compression techniques in astronomy, benefiting both the machine learning  and astronomy communities. Moreover, the benchmark results highlight the potential of neural compression methods, which could offer research opportunities for ML researchers.