AION-1: Omnimodal Foundation Model for Astronomical Sciences
Authors: Liam Holden Parker
Francois Lanusse
Jeff Shen
Ollie Liu
Tom Hehir
Leopoldo Sarra
Lucas Thibaut Meyer
Micah Bowles
Sebastian Wagner-Carena
Helen Qu
Siavash Golkar
Alberto Bietti
Hatim Bourfoune
Pierre Cornette
Keiya Hirashima
Geraud Krawezik
Ruben Ohana
Nicholas Lourie
Michael McCabe
Rudy Morel
Payel Mukhopadhyay
Mariel Pettee
Kyunghyun Cho
Miles Cranmer
Shirley Ho
Venue: NeurIPS 2025 poster ()
Keywords: astrophysics
foundation model
multimodal generative model
Primary area: machine_learning_for_sciences
TL;DR: The first large-scale multimodal foundation model for astronomy integrating a large number of diverse observational  modalities.

Abstract: While foundation models have shown promise across a variety of fields, astronomy lacks a unified framework for joint modeling across its highly diverse data modalities. In this paper, we present AION-1, the first large-scale multimodal foundation family of models for astronomy. AION-1 enables arbitrary transformations between heterogeneous data types using a two-stage architecture: modality-specific tokenization followed by transformer-based masked modeling of cross-modal token sequences. Trained on over 200M astronomical objects, AION-1 demonstrates strong performance across regression, classification, generation, and object retrieval tasks. Beyond astronomy, AION-1 provides a scalable blueprint for multimodal scientific foundation models that can seamlessly integrate heterogeneous combinations of real-world observations. Our model release is entirely open source, including the dataset, training script, and weights.

AION-1: Omnimodal Foundation Model for Astronomical Sciences
Liam Parker".1.2.5 Francois Lanusse": Jeff Shen Ollie Liu", Tom Hehir Leopoldo Sarra², Lucas Meyer?, Micah Bowles", Sebastian Wagner-Carena" 2,3 Helen Qu", Siavash Golkar2.3, Alberto Bietti², Hatim Bourfoune Pierre Cornette Keiya Hirashima 2.11 Geraud Krawezik", Ruben Ohana², Nicholas Lourie", Michael McCabe Rudy Morel Payel Mukhopadhyay Mariel Pettee Bruno Regaldo-Saint Blancard', Kyunghyun Choo, Miles Cranmer", Shirley 2,
The Polymathic AI Collaboration
University of California, Berkeley, Flatiron Institute, New York University, 4 Lawrencce Berkeley National Laboratory, Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM, 'Princeton University, Uniiersity of Southern California, University of Cambridge, Univeresity of Oxford, IDRIS, CNRS, RIKEN Center for iTHEMS, University of Wisconsin, Madison
Abstract
While foundation models have shown promise across a variety of fields, as- tronomy lacks unified framework for joint modeling across its highly diverse data modalities. In this paper, we present AION-I, the first family of large- scale multimodal foundation models for astronomy. AION-1 integrates large number of heterogeneous data types using a two-stage architecture: modality- specific tokenization followed by transformer-basee modal token sequences. Trained on over 200M astronomical objects, AION-1 demonstrates strong performance across regression, classification, generation, and object retrieval tasks. Beyond astronomy, AION-1 provides a scalable blueprint for multimodal scientific foundation models that can seamlessly in- tegrate heterogeneous combinations of real-world observations. Our model re- lease entirely open source, including the dataset, training script, and weights: https: / thub. com//PolyymathiccAAAA..
masked modeling of cross-
Introduction
Foundation models have transformed natural language processing and computer vision I| However, they have not been fully explored in scientific domains where data are often complex and heterogeneous, combining multiple instruments, measurement protocols, and noise sources unique to real-world experiments. As result. many scientific analyses employ bespoke models that treat each modality in isolation or rely on strict often hand-crafted schemas for cross-modal data fusion.
Within the broader scientific landscape, astronomy provides particularly compelling testhed for the development of multimodal scientific foundation models owing to both the volume of publicly available data and its extraordinary diversity of measurements, Indeed, recent works have begun to explore multi-modal foundation models in astronomy [8 80 50]; however, these approaches have been limited to single physical phenomena and relied primarily contrastive objectives, which face fundamental limitations including generalization to arbitrary modalities and difficulty in capturing information beyond the mutual information between modalities.
Denotes equal contribution. Contact: 1hparker edu, francois. lanusse@cnns.ff fr
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Data and Tokenization
AION Model
Downstream Tasks
Regression Physical Parameters
Embeddings
Decoler
Iaputs
Transformer
AION
log M.
Encoder
Attention Pooling
Instance Segmentation of Galaxy
Images
Spectm
CNN Decoder
AION
Retrieval of Strong Crravitational Lenses
0.22
Queries
Transformer Decoder
Outpur
Segmectattion Maps
Physical Parameters
AION
Similarity
Cosine
Figure 1: AION-1 integrates 39 different data modalities multiband images, optical spectra, and various properties and measurements into single model usable for wide range of downstream applications. implements a two-step process: first, bespoke tokenization strategies that homoge- nize the diverse scientific data, followed by multimodal masked modeling that learns how different observations relate, inducing a deep understanding of the underlying physical objects. Astronomers can then leverage AION-1 's rich astrophysical understanding for variety of downstream tasks.
In this paper, we introduce AION-1 (Astronomical Omni-modal Network), the first large-scale multimodal foundation model for astronomy designed to handle arbitvary numbers of modalities across multiple physical phenomena. AION-1 unifies imaging, spectroscopy, photometry, and other object-level measurements from major ground- and space-based observatories into a single model for galaxies, stars, and quasars. By bridging these disparate data types, AION-1 addresses key challenge scientific machine learning: the integration multiple heterogeneous datasets spanning different instruments, measurement protocols, noise sources, and physical phenomena into single, unified framework.
At the heart of AION-1 lies two-step approach: Universal Tokenization of Diverse Data, where we homogenize real-world scientific observations with discrete quantization across different data types, instruments, and observatories, followed by Multimodal Masked Modeling, where we train single transformer encoder-decoder with a masked-token objective over all modalities simultaneously. Once trained, we demonstrate emergent behaviors in the AION-1 models that reflect the potential for multimodal scientific foundation models to capture non-trivial physical insights from raw data alone:
Emergent Physical Understanding. AION-1 can solve non-trivial scientific tasks using only simple linear head top of its learned representations.
Superior Performance in the Low-Data Regime. AION-1 achieves competitive results on downstream tasks even with orders of magnitude less data than its supervised counterparts.
Flexible Data Fusion. AION-1 can use arbitrary combinations of observations, enabling seamless data fusion on downstream tasks as well as cross-modal conditional generation.
Physical Structure of the Latent Space: AION- embedding space organizes objects along physically meaningful directions, enabling powerful retrieval of rare observations that surpasses current state-of-the-art retrieval methods in astronomy.
Beyond astronomy, the data tokenization strategies, masked modeling, and cross-modal generation strategies introduced address key challenges in real-world scientific data: namely, heterogeneity, noise, and instrument-specifia idiosyncrasies. Moreover, by focusing purely observational data, our approach is applicable in any data-rich field, even when strong physical models are not available.
1.1 Contributions
In summary, present the following contributions:

---

We introduce AION-I, family of token-based multimodal scientific foundation models ranging in size from 300M 3.1B parameters. AION-1 is designed for arbitrary combina- tions of highly heterogeneous scientific observations.
We develop bespoke tokenization methods to homogenize wide variety of astronomical data into a single coherent corpus. These innovations address the heterogeneity, noise, and instrument-specine peculiarities that challenge standard scientific modeling.
We demonstrate that AION-1 achieves competitive to state-of-the-art performance on broad range of scientific tasks with even simple probing, while significantly outperforming supervised baselines in low-data regimes, rendering the model highly usable by downstream researchers even without dedicated finetuning.
By tackling the challenges of data heterogeneity, noise, and diverse instrumentation, AION-1 offers promising paradigm for future multimodal foundation models beyond astronomy, setting the stage for new era of large-scale, cross-domain scientific exploration.
Related Work
Multimodal foundation models have become cornerstone of modern self-supervised learning 20 [8, 36 6 [0]. Indeed, recent advances like GPT-4V [,, Claude [6], and LLaVA [6 have achieved human-level performance in visual reasoning, while models like Imagen [5 and
Stable Diffusion (60] have enabled high-quality image generation from text. However, these models primarily rely on language bridge modalities, which is often unavailable for scientific data. Recent work on carly-fusion models, such as Chameleon [65] 4M [44], or PercieverIO 27 have demonstrated promising alternatives by learning mappings between modalities.
While these methodological advances in foundation models have transformed many fields, astronomy presents unique challenges, including heterogenous instruments, measurement protocols, and noise. As such, tatronomm--ppeccin efforts have emerged. For example, supervised pre-trained models like Zoobot [68] have leveraged 100M human annotations for galaxy morphology prediction obtained through extensive citizen science campaigns. Large-scale, self-supervised approaches trained on single-modal data have also emerged, including transformer-bat models for Gaia stellar data [4] APOGEE spectra [32 and astronomical images [56] and contrastive approaches for astronomical images 225 62 61). Finally, recent multimodal contrastive approaches have been introduced, starting with galaxy image-spectra pairs in (48 and followed by galaxy images and text 433 and time-series
and photometry 80 50
Relative to these methods, A represents a significant advance in both scale and scope: it is the first effort train multimodal models to billion-paramann parameter scales and the first attempt unify arbitrary modalites or different object types; in this case, 39 modalities across 200 million unique measurements spanning galaxies, stars, and quasars.
3 Universal Tokenization of Diverse Data
Tokenization in AION-1 transforms heterogencous data into unified, transformer-compatbble representation. Astronomical datasets present two key challenges: the variety data types (2D images, ID spectra, scalar values) and the diversity of sources within each type (different telescopes. resolutions, and instrument formats). We address this through modality-specine provide intra-most modality standardization; each modality uses dedicated tokenizer capable of handling multiple instruments, ensuring aligned representations within each data type. Moreover, the need to train multiple tokenizers for modality with multiple survey inputs is removed. We provide the full details of the tokenizers in Appendix C. and the list of tokenized modalities AION-1 understands in Appendix
tokenizers that
Multiband imaging data. Galaxy images vary widely in resolution (physical pixel size), channel number, wavelength range, noise properties, and brightness across surveys. AION's image tokenizer addresses this diversity through a flexible channel-embeddiin accommodates variable channel counts and embeds provenance information (e.g., originating tele- scope) for each channel. Built on ResNet backbone adapted from [77] and using Finite-Scall
scheme adapted from 400 that
3

---

Quantization (FSQ) 42 our tokenizer enables single model to ingest imaging data from multiple diverse observational pipelines.
The tokenizer is trained using an peperse-varinneeeeee weighted Gaussian negative log-likelihood (NLL) that leverages prior knowledge of the noise properties in each image, reported by the data- generation pipelines. The NLL is given by:
LLLL ΣΣ Σ (x; Doco (enc((((()))))) 113
where x; is the input image, and Σ, is the diagonal noise covariance that image provided by the imaging pipeline, which accounts for background noise and shot noise from bright sources.
Spectroscopic Data. A single spectrum measures the wavelength dependent variation of light typically coming from a single object. Measurements from different spectrographs vary in amplitude, wavelength range, and resolution. AION's spectrum tokenizer normalizes and resamples spectra onto a shared latent wavelength grid, enabling joint processing of disparate instruments and objects, e.g. galaxies, stars, and quasars. Built on ConvNeXt V2 (74 with Look-up up Free Quantization (LFQ) [79]. it incorporates survey-specific noise variance through the same loss function used for images.
Tabular/Scalar Data. Scientific data often involve scalar measurements both direct (e.g., size, flux) and derived (e.g., redshift). Instead of continuous embeddings [22]. which struggle with large or variable dynamic ranges, AION-1 tokenizes scalars using uniform binning in their cumulative distribution functions (CDFs). This homogenizes the scalar distributions and minimizes quantization errors in the most important regions.
Segmentation and Property Maps. Beyond photometric images, other spatially resolved scalar fields (e.g., segmentation maps, physical property maps) also of interest in galaxy astronomy. We thus include a tokenizer for normalized maps in [0, 1]. based on a convolutional architecture with FSQ quantization [22 and trained on grayscale galaxy images and segmentation maps.
We also include bounding ellipse catalogs for object localization. Each ellipse in these catalogs is characterized by quintuple (spatial coordinates, ellipticity, radius), tokenized by mapping coordinates to the nearest pixel and quantizing elliptical attributes. Since object counts vary, we lincarize these entries into sequence [12), sorted by distance from the image center, providing consistent reference and reducing ambiguities.
Bounding Ellipses.
Multimodal Masked Modeling
27 21 In particular, it adopts the learn from heterogeneous data (e.g., spectra, images, scalars) by randomly masking tokenized inputs across all available modalities and reconstructing the masked content.
AION-1 builds on recent early fusion multimodal models 65
scalable multimodal masked modeling scheme proposed in 4M 44
Concretely, let X = fx X be token sequences for M modalities available for training example. During training, two disjoint subsets of X are drawn: ohs (observed) and xtst (target). Because these subsets sampled across the entire token pool, the model learns both intra- and cross-modal relationships in each training instance.
The loss for the model is then given by:
(1) = -
Σ log pe (xt x
where pa(- xbb)) is the categorical distribution over the predicted vocabulary, and N is the output
token budget.
4.1 Architecture
We adopt Transformer-based encoder-decoder framework suitable for the multi-moda. masked prediction (see Figure T). Beyond standard encoder-decoder architecture, we emphasize below

---

the modality specific embedding scheme needed at the input of both the encoder and decoder to implement our objective.
Concretely, each modality E (1....,,, 3) has its own token embedding Embed; (.), a learnable modality embedding m,, and positional embedding Pa for token position t. Then, for an observed token from modality i, zy the full embedding is given by
(ene) Embed, m. P
In the decoder, we feed information on the target we are querying tokens for, without providing their value:
o (dec) m, + Pr.
omitting any direct lexical embedding Embed, In our implementation
different modality embedding m. for each modality and each source to identify the unique combination of data type and associated provenance metadata. In other words, two astronomical images from two different instruments will have different modality embeddings even though they are both images. This is to provide the model with important provenance information which implicitly encodes aspects of data quality and resolution of the observations.
4.2 Modality Masking Strategy
key consideration is to select which tokens become inputs (observed) vs. outputs (predicted) for each modality during training. We find that the dirichlet sampling from the original 4M implementation is ineficient when dealing with modalities that vary widely length, and therefrore results in a high frequency of mostly empty batches. Therefore, we follow a simplified approach:
Input Token Budget pick one modality, and then uniformly randomly select number of tokens for inclusion from that modality. We then fill the remaining budget by uniformly sampling tokens from the other modalities.
We select global input token budget B. To populate the budget, we randomly
Output Token Budget. For the remaining unselected tokens, we choose the number of tokens to predict for each modality by sampling from Beta distribution skewed toward zero, which draws down the number of output tokens per sample, aligning with the eventual distribution of output tokens under cosine schedule iterative sampling (e.g., in MaskGiT-style) ensuring that inference-time usage patterns are well covered during training. Similar to the input, one modality is chosen to drawn an unconstrained number of tokens first, and the rest are filled by uniform random draws from the remaining modalities.
AION-1 Family of Models
Dataset. AION-1 pretrained on the publicly available data from the Multimodal Universe (hereafter MMU) [66], a large-scale dataset of MI.-ready, multimodal astronomical data. We use five surveys: Hyper Suprime-Camen Camera (HSC) [5] and Legacy Imaging Survey [16] for galaxy images; DESI 3 and SDSS 766 for high-resolution spectra and cosmic distances on galaxies and stars; and Gaia [13] for tow-resolution spectra and precise photometric and astrometric measurements for stars in the Milky Way. We refer the reader to Appendix A for full details on the pretraining data. The relative contribution of each survey is illustrated in Figure 2a
AION-1's pretraining emphasizes learning relationships between diverse observations of the same astronomical objects. Unlike 4M [44) which requires all modalities simultaneously, we use pairwise associations across surveys, which accommodates uneven associations.
Training. We train three model versions Base (300M), Large (800M), and XLarge (3B) using the A (37 optimizer (31 0.9, B2 0.95, weight decay 0.05) for 205k steps with global batch size of 8096. We use linear warmup and cosine decay schedule, with peak learning rate of 2 x 10-4 Figure 2b shows how model size and data mixture (including or excluding Gaia) impact the evaluation loss.
5

---

Gaia 77.62M
Evaluation loss on Legacy Survey
Dulaect
AION-B AJON-L
AION-LL
4.0
Dulsset GAIA
Legacy Survey
672 IEEe
SDSS
122.74M S
2.5
4206
HSC 2.52M
DEST 1.4MM S
2.0
S
500001
100000
150000
200000
Checkpoint
(a) Training data mixture.
(b) Evaluation loss function
Figure 2: Left: Illustration the diverse data types used in pretraining. By pairing observations from different instruments along with object metadata, the model gains multimodal understanding. The small overlapping sample cizem-common in astrophysies- are a key motivation for AION-1. Right: AION-1 test losses Legacy Survey [16] for various model sizes and dataset selection. is interesting to note the change of losses across all model sizes when Gaia [3] the only dataset that includes only stars (no galaxies) added.
Out-of-the-Box Capabilities
AION-1 enables variety of generative tasks out-of-the-box, from data imputation to cross-survey translation. By representing the joint distribution over all modalities, AION-1 can draw conditional samples of any modality given partial observations, producing physically consistent reconstructions. We provide details using AION-1 as generative model in Appendix D
Cross-Modal Generation AION-1 can conditionally generate high-dimensional modalities using the ROAR framework [44]. This enables scientific analyses like cross- survey translation or super-resolution. Here, demonstrate AION-1's ability to gener- ate high-resolution DESI spectra from coarse Gaia BP/RP coefficients that are 50- 100x sparser than the DESI spectra. Fig. 3 demon- strates that the model accurately reconstructs line centers, widths, and amplitudes within narrow posterior uncertainty bands, enabling detailed abundance analysis from widely available low-resolution data. This capability is valuable given the significantly greater cost of performing high-resolution measurements like DESI and the much broader availability of low-resolution data like Gaia.
Flux
Mh my.
Po lines
Irno
4000 50/00
6000
7000
8000 5000
Wavelength
Figure Super-resolution: AION-1 can generate high-resolution posterior spectra samples (blue) con- ditioned on low-resolution spectra input (red), closely matching the ground-truth high-resolution measure- ments (black) in line location, width, and amplitude.
Scalar Posterior Estimation AION-1 can also directly estimate any scalar value that is natively quantized during pretraining, allowing AION-1 to infer posterior distributions p(=|context) We demonstrate this capability redshift, key measure of cosmic distance. Fiiuure 4 displays posterior samples for a representative galaxy under three increasingly informative contexts: (1) Legacy Survey photometry only, (2) Legacy Survey photometry and multi-band imaging, and (3) high-resolution DESI spectra. The posterior clearly contracts as more information is provided.
7 Evaluation with Downstream Scientific Workflows
AION-1 I can produce physically meaningful, modality-agnoxtic embeddings out of the box, avoiding bespoke supervised pipelines. Because foundation models carry pretraining prior, we use AION-1 as

---

Legacy Survey di f. Photomerry
Legacy Survey aurc fud Imaging
Optical Spectram
Tree
12.5
0.0
0.5
1.0
1.5
20
3.5
3.0
0.0
0.5
100
1.5
3.5
3.55
1.0
0.0
0.5
1.0
3.0
9.5
100
2
Figure 4: Redshift Posterior Estimation. Posterior samples for a single Legacy Survey galaxy under three conditioning scenarios: (left) broadband photometry only, (middle) photometry + imaging, and (right) full spectrum. The posterior contracts as richer information provided.
a frozen feature extractor and perform lightweight task calibration: a small head on a representative calibration set (matching the downstream selection function) and calibrate outputs. At inference, freeze the encoder and discard the decoder; given tokenized inputs, the encoder outputs a contextual sequence Z = {z, 1 Rd which compress to object vector E IR via mean pooling (e + ze) learned attentive pooling (e soft (QK) /V with Q, K, V trainable. Multi-modalipy is native: concatenate tokens from any subset of modalities and pass the union through the frozen encoder-no no fusion is required. This workflow preserves AION-I's representational power while letting researchers inject scientifically relevant priors.
7.1 Physical Property Estimation
Physical properties are used to describe fundamental aspects of astronomical objects, like the stellar mass of galaxies temperature of stars. These often derived using expensive, high-resolution measurements. As such, a common task in astronomy is to bypass this step and estimate these physical properties from low-resolution measurements directly with machine-learning models.
Galaxy Properties We predict five galaxy properties (redshift, 2; stellar mass, .M;; age, tage: star-formation rate, SFR: metallicity, ZMa). We cross-match the SED-fit properties from PROVABGS (24) with the rg Legacy Survey imaging and DESI spectroscopy, yielding roughly 120,000 galaxies. We con- sider three input combinations: pho- tometry, photometry + imaging, and photometry + imaging + spectra, and perform attentive pooling from the AION-1 embeddings as described above. We then fit lightweight, two- layer multilayer perceptron (hidden size 256) using random 80/20 split, to map the input to the target pa- rameters. We quantify performance with 122
M.
tage
log Zetat
sSFR
AION-1-B
Ph
0.75
0.72
0.35
0.41
0.38
Ph+Im
0.93
0.89
0.45
0.49
0.64
Ph+Im+Sp
1.00
0.96
0.53
0.61
0.72
ATON-1-I. Ph
0.76
0.73
0.36
0.39
Ph+Im
0.94
0.89
0.45
0.50
0.64
Ph+Im+
1.00
0.96
0.53
0.62
0.73
AION-I-XL
Ph
0.79
0.76
0.31
0.38
0.48
Ph+Im
0.94
0.89
0.45
0.49
0.64
Ph+Im+Sp
0.99
0.95
0.53
0.62
0.73
AstroCLIP 48
Im
0.78
0.73
0.29
0.36
0.42
Sp
0.99
0.90
0.52
0.60
0.70
DINOV2 47
0.57
0.55
0.17
0.28
0.25
Supervised
(Σ,(0 9,)3)/(, - -)²), where My the ground-truth values, Ú the pre- dicted values, and the mean of the ground-truth sample. We include three supervised basclines informed by the literature: XGBoost on pho- tometry, a NeXt on imaging (66), and a CNN + attention network on spectra 41 We also include results using the same MLP layer embeddings produced by two self self-supervised baselines,
XGBoost (Ph)
0.71
0.69
0.30
0.30
0.38
ConwNeXt (Im)
0.86
0.82
0.45
0.49
0.64
Ctnv+Att (Sp)
1.00
0.85
0.43
0.62
0.68
Table 1: R² (t) for galaxy property estimation. Inputs are photometry (Ph), photometry + imaging (Ph+Im), and photometry + imaging + spectra Im++)))pp).
AstroCLIP (48 and DINOv2 47
NNoe that AstroCLIP and DINOv2 can only ingest 3-channel imaging, so these are ran using correctly normalized (g, r, Legacy Survey images while AION-1 is ran using the full {g, i, 2. channels.
7

---

TnfF
log
Fe/ H
Stellar Properties We also predict stellar parameters (Tami temperature, log gi sur- face gravity, Mmic: microturbulent veloc- ity, [Fe/H]; metallicity, [X/Fe]; elemen- tal abundances). We cross-match Gaia DR3 data with DESI spectra and the asso- ciated derived properties from [80], yield- ing 240,000 stars. We consider three input combinations: photometry. photometry + parallax + sky position, and DESI spectra with parallax. We use the same adapta- tion head and training strategy as above and report R We also include two su- pervised baselines from the literature: XG- Boost on photometry (4 and ConvNeXt on raw spectra. Results are presented in Table 2
AION-I-B
Ph
0.94
0.95
0.58
0.86
Rh+Dee-RA/Dec
0.94
0.95
0.70
0.87
Sp+Pix
0.99
0.98
0.94
0.89
AION-I-L
Ph
0.95
0.96
0,58
0.87
Ph+Dee-RA/Dec
0.95
0.96
0.71
0.88
Sp+Pix
0.99
0.98
0.94
0.89
AION-I-XL
Ph
0.92
0.94
0.56
0.85
Ph+DDee
0,93
0.95
0.68
0.87
Sp+Plx
0.98
0.98
0.92
0.89
Supervised
XGBoost (Ph)
0.94
0.95
0.59
0.87
ConvNeXt (Sp) 0.99
0.98
0.95
0.89
Table 2: R² (1) for stellar-label prediction. Inputs are Gaia photometry (Ph). low-resolution Gaia XP spectra (XP), parallax (Plx), celestial coordinates (RA/Dec), and high-resolution DESI spectra (Sp).
addition to these supervised baselines, we also compare performance to a state- of-the-art self-supervised baseline [34]. Specifically, predict APOGEE-derived stellar parameters from Gaia XP spectral coefficients using cross-matched APOGHF-derived stellar parameters with Gaia, producing roughly 10,000 pairs. We feed only the first 32 BP coefficients and first 32 RP coefficients to AION-1 due to the fact that (34) has a context length of 64, and report results below. Note that (34] has been explicitly pretrained on this task, while AION-1 has never seen this task before; nonetheless, AION-1 outperforms.
Model
Ton (K)
log (dex)
[Fe/H] (dex)
AION-B
94.6
0.206
0.115
Leung, et al.
99.1
0.229
0.143
Table 3: Performance on APOGEE stellar property predictions from Gaia XP coefficients (L),
measured by standard deviation of residuals. K is temperature units in Kelvin, and dex represents scatter on a logarithmic scale.
7.2 Learning from Semantic Human Labels
While some aspects of astronomical objects can be described by physical properties, in many cases they exhibit complex features that are not captured by physical models. We explore leveraging AION-1 to identify such features in galaxy images, given limited set of human annotations.
Galaxy Morphology Classification. We consider here the problem of classifying galaxy images into ten distinct morphology classes (e.g. spiral arms, merging galaxies) defined by Galaxy Zoo 10 [GZ10; We cross-match GZ10 with Legacy Survey images, yielding 8,000 galaxies. We produce embed- dings from AION-1 using mean pooling as described above, and train a 2-layer MLP to classify galaxies from these embeddings. We present classification accuracy Table 4 We see strong performance against dedicated supervised convolutional model (64) trained from scratch, and competitive perfor- mance against a state-of-the-ar model trained on two orders of magnitude more labeled data [70] that we adapt for this task with MLP head on its frozen embeddings. We also demonstrate that AION-1 can be used to transfer learn from Legacy Survey morphologies to HSC morphologies in subsection H.4
Model
Accuracy (%)
AION-1-B AION-1-L
84.0
87.2
AION-1-XL
86.5
69 333.
DINOv2 47
71.4
EfficientNet
80.0
ZooBot 69
89.6
Table 4: Galaxy Morphology Classification
Accuracy (+) Galaxy Zoo 10. AION-1,
DINOv2 (47), and ZooBot (69 use an MLP head on frozen embeddings; EfficientNet-B3 (64] is trained end-to-end from scratch.
8

---

Spiral Arms
Bar
AION-1-B
0.60
0.31
AION-I-L
0.61
0.31
AION-I-XI
0.61
0.32
U-Net
0.55
0.28
Figure 5: Galaxy structure segmentation. Left: Examples of galaxy image segmentation produced from the AION-1 embeddings with a lightweight convolutional neural network compared with the ground-truth true volunteer labels. Right: Mean IoU (1) the held-out test set.
Semantic Segmentation of Galaxy Structures. We also consider image segmentation based on human annotation of prominent galaxy structures obtained through the Galaxy Zoo 3D citizen science campaign (38] which we -rtss with Legacy Survey, yielding 2,800 galaxies. We train simple convolutional upsampling head based on [3 on top of mean-poos meean-poleed AION-1 embeddings predict the segmentation maps. Training is performed using 80/20 split. We compare our results to a simple fully convolutional U-Net baseline following architecture choices from 71. We report Intersection over Union (IoU) between ground truth and growd-annotaie structural components. We also include in (subsection G.3 closely related task consisting in detecting regions of star formation in galaxy images.
7.3
Performance Low-Data Regime
Galazy Property Regression
Morgbology Classification
Property Regresiion
0.7
0.5
0.5
04
Accuraty CT))
0.4
75
04
AXXN-
50
65
0.0
103
1000 2000 3000
4000 5000 6900 7000
107
109
10t
Toilking Soc
Training Set Sioee
Trannng Sei
(a) Galaxy Properties
Morphology Classification
(c) Stellar Properties
Figure 6: Model performance downstream task training set size. We regress (a) galaxy physical properties from images, (b) classify galaxy morphology, and (c) regress stellar properties from spectra. Supervised baselines are trained from scratch the raw input data while AION-1 are results obtained by probing frozen encoder embeddings.
In astronomy, many key classes come with only handful of reliable labels. As result, approaches that perform well in low-data regimes are particularly valuable. To that end, we rerun the property prediction and morphology classification cases described above but artificially limit the training set size while keeping the test set fixed at 20%. For each experiment we retrain the lightweight MLP adapter as well as corresponding supervised baseline. We note that AION-I's performance relative to supervised baselines is more impressive in the low data regime, where it matches or surpasses supervised models that require order of magnitude more training data.

---

Query
Similarity Search Results
Rank
Rank
Rank
Spima
Spirals
Mergers
Lenses
AION-1-B
0.632
0.281
0.012
Bank
Rank
Rank
AION-I-L
0.643
0.303
0.011
Merger
AION-I-XI.
0.621
0.384
0.015
AstroCLIP
48
0.602
0.248
0.006
Sirong Lensing
Bank 6
Runk 20
Rank 50
Stein et al.
62
0,590
0.340
0.007
DINOv2
47
0.477
0.060
0.003
Figure 7: Galaxy Image Retrieval for three astronomical classes of decreasing prevalence; spirals, mergers and strong lenses. Left: Example candidates a given query image. Right: The nDCG@10 score for each of the classes averaged over all queries in the given dataset.
7.4 Similarity- Retrieval
Modern astronomical surveys gather immense volumes of data, however scientists often aim to identify a small set of rare phenomena that have outsized importance for understanding the Universe. For example, strong gravitational lenses of particular interest in astronomy, they provide unique constraints on dark matter and the expansion of the universe [54]. Despite their significance, these objects notoriously hard to find, appearing in only fraction of galaxies (~ 0.1% for strong lenses), and thus lack large, annotated training sets for supervised methods.
AION-1 can address this challenge by searching for rare objects directly in the model's latent space. To illustrate this, we perform rare object retrieval for three types of rare objects; spirals and mergers from Galaxy Zoo DECaL (25,000 2,,000 and 700 out of 171,000 galaxies) [69] and strong lens candidates in the HSC footprint (700 out of 758,000 galaxies) 29 both of which we cross-match with the
Legacy Survey data. For all cases, we embed all images in shared latent space using the mean pooling strategy above, and then for all query objects, we use cosine similarity to rank all possible candidates and compute an nDCG@ @ 10 score (Appendix H..I), H.T.. which then average over the full query set; we provide full details of these datasets in the appendix. Figure 7 shows that AION-1 performs well on both Galaxy Zoo DECaLS objects like spirals (which make up 26% of the sample) and mergers (2%) [69 as well as strong gravitational lens candidates in the HSC footprint, which have just 0.1 % occurrence: Appendix H.3) We compare ATON-I's performance leading self-supervised astronomy models 4%
62] as well as DINOv2 (4)), all ran on the same dataset. We introduce
strategies further improve performance in Appendix H.2
8 Discussion & Conclusion
AION-1 summarises heterogeneous astronomical observations in a single carly-fusion backbone. yielding strong zero-shot performance and anoobe accuracy that rivals (or in sparse-label regimes exceeds) task-specific models. By tokenising images, spectra and catalogues into common sequence and training with masked- prexiccion objectives, the model learns cross-modal representations that support flexible data fusion, generative translation (e.g. sampling high-resolution spectra from low- resolution inputs) and efficient retrieval of rare phenomena, all without fine-tuning. These capabilities foreshadow shift from bespoke, modality-speciffe pipelines toward reusable foundation models that streamline discovery and improve reproducibility across surveys.
Limitations and outlook. AION-1 inherits survey selection biases, lacks explicit temporal reasoning and remains computationally intensive train, but none of these obstacles are fundamental. Continual pre-training on forthcoming facilities, time-aware tokenisers and advances in sparsity distillation can broaden coverage and democratise use. More ambitiously, coupling physics-aware objectives could steer representations toward physically consistent manifolds while accelerating the curation of high-value labels. Even in its current form, AION-1 demonstrates that omnimodal foundation models are both feasible and transformative, providing scalable template for cross cross-domain scientific AI.
10

---

9 Acknowledgments
We would like to acknowledge the support of the Simons Foundation and of Schmidt Sciences. This project was provided with computer and storage resources by GENCI at IDRIS thanks to the grant 2024-GCO1 1015468 on the supercomputer Jean Zay's H100 partition. Additionally, some of the computations in this work were run at facilities supported by the Scientific Computing Core at the Flatiron Institute, a division of the Simons Foundation. Liam Parker also acknowledges support from the National Science Foundation Graduate Research Fellowship Program. Jeff Shen is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), funding reference number 587652. We would like to thank Andrew Engel, Mare Huertas-Company Stephanie Juneau, Andy Morgan, and Mike Smith for their valuable feedback during beta testing and Sophie Barstein for her help with writing the corresponding blog post.
9.1 Data
9.1.1 Legacy Survey
The Legacy Surveys consist of three individual and complementary projects: the Dark Energy Camera Legacy Survey (DECaLS; Proposal ID #2014B-0404; Pls: David Schlegel and Arjun Dey), the Beijing-Arizona Sky Survey (BASS; NOAO Prop. ID -2015A-0801; Pls: Zhou Xu and Xiaohui Fan), and the Mayall z-band Legacy Survey (MzLS; Prop. ID #2016A-0453; PI: Arjun Dey). DECaLS, BASS and MzLS together include data obtained, respectively, the Blanco telescope. Cerro Tololo Inter-Americal Observatory, NSF's NOIRLab: the Bok telescope, Steward Observatory, University of Arizona; and the Mayall telescope, Kitt Peak National Observatory, NOIRLab. Pipeline processing and analyses of the data were supported by NOIRLab and the Lawrence Berkeley National Laboratory (LBNL). The Legacy Surveys project is honored to be permitted to conduct astronomical research on Iolkam Du' ag (Kitt Peak), a mountain with particular significance to the Tohono O' d'odham Nation.
NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA) under a cooperative agreement with the National Science Foundation. LBNL is managed by the Regents of the University of California under contract to the U.S. Department of Energy.
This project used data obtained with the Dark Energy Camera (DECam), which constructed by the Dark Energy Survey (DES) collaboration. Funding for the DES Projects has been provided by the U.S. Department of Energy, the U.S. National Science Foundation, the Ministry of Science and Education of Spain, the Science and Technology Facilities Council of the United Kingdom, the Higher Education Funding Council for England, the National Center for Supercomputing Applications at the University of Illinois at Crhana-Champaign, the Kavli Institute of Cosmological Physics at the University of Chicago, Center for Cosmology and Astro-Particle Physies the Ohio State University, the Mitchell Institute for Fundamental Physics and Astronomy at Texas A&M University. Financiadora de Estudos e Projetos. Fundacao Carlos Chagas Filho de Amparo, Financiadora de Estudos Projetos, Fundacao Carlos Chagas Filho de Amparo a Pesquisa do Estado do Rio de Janeiro, Conselho Nacional de Desenvolvimento Cientifico e Tecnologico and the Ministerio da Ciencia, Tecnologia Inovacao, the Deutsche Forschungsgemeinschatf and the Collaborating Institutions in the Dark Energy Survey. The Collaborating Institutions are Argonne National Laboratory. the University of California at Santa Cruz, the University of Cambridge, Centro de Investigaciones Energeticas, Medioambientales y Tecnologicas-Madinn the University of Chicago, University College London, the DES- DES-Brazil Consortium, the University of Edinburgh, the Fidgenossische Technische Hochschule (ETH) Zurich, Fermi National Accelerator Laboratory, the University of Illinois at Jrbana-Champaigge the Institut de Ciencies de 1'Espai (IBEC/CSIC), the Institut de Fisica d' Altes Energies, Lawrence Berkeley National Laboratory, the Ludwig Maximilians Universitat Munchen and the associated Excellence Cluster Universe, the University of Michigan, NSF's NOIRLab, the University of Nottingham, the Ohio State University, the University of Pennsylvania, the University of Portsmouth, SLAC National Accelerator Laboratory. Stanford University, the University of Sussex, and Texas A&M University.
BASS is a key project of the Telescope Access Program (TAP), which has been funded by the National Astronomical Observatories of China, the Chinese Academy of Sciences (the Strategic Priority Research Program "The Emergence of Cosmological Structures" Grant XDB09000000), and the Special Fund for Astronomy from the Ministry of Finance. The BASS is also supported by the

---

External Cooperation Program of Chinese Academy of Sciences (Grant # 114AIKYSS2016000777 and Chinese National Natural Science Foundation (Grant 12120101003, 11433005).
The Legacy Survey team makes use of data products from the Near- Farth Object Wide-field Infrared Survey Explorer (NEOWISE), which is project of the Jet Propulsion I Laboratory/Califomii Institute of Technology. NEOWISE is funded by the National Aeronautics and Space Administration.
The Legacy Surveys imaging of the DESI footprint is supported by the Director, Office of Science, Office of High Energy Physics of the U.S. Department of Energy under Contract No. DE-ACO2- 05CH1123, by the National Energy Research Scientific Computing Center, DOE Office of Science User Facility under the same contract; and by the U.S. National Science Foundation, Division of Astronomical Sciences under Contract No. AST-0950945 to NOAO.
9.1.2 Hyper Suprime-Cam
The Hyper Suprime-Camy Cam (HSC) collaboration includes the astronomical communities of Japan and Taiwan, and Princeton University. The HSC instrumentation and software were developed by the National Astronomical Observatory of Japan (NAOJ), the Kavli Institute for the Physics and Mathematics of the Universe (Kavli IPMU), the University of Tokyo, the High Energy Accelerator Research Organization (KEK), the Academia Sinica Institute for Astronomy and Astrophysics in Taiwan (ASIAA), and Princeton University. Funding was contributed by the FIRST program from Japanese Cabinet Office, the Ministry of Education, Culture. Sports, Science and Technology (MEXT), the Japan Society for the Promotion of Science (JSPS), Japan Science and Technology Agency (JST), the Toray Science Foundation, NAOJ, Kavli IPMU, KEK, ASIAA, and Princeton University.
This paper makes use of software developed for the Large Synoptic Survey Telescope. We thank the LSST Project for making their code available as free software at http://dm.lst..oo Isst.orr..st.rggggg.o
The Pan-STARRSI Surveys (PS1) have been made possible through contributions of the Institute for Astronomy, the University of Hawaii, the Pan-STARRS Project Office, the Max-Planck Society and its participating institutes, the Max Planck Institute for Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial Physics, Garching, The Johns Hopkins University, Durham University, the University of Edinburgh, Queen's University Belfast, the Harvard-Smittsoniann Center for Astrophysics, the Las Cumbres Observatory Global Telescope Network Incorporated, the National Central University of Taiwan, the Space Telescope Science Institute, the National Aeronautics and Space Administration under Grant No. NNX08AR222 issued through the Planetary Science Division of the NASA Science Mission Directorate. the National Science Foundation under Grant No. AST- 1238877, the University of Maryland, and Eotvos Lorand University (ELTE) and the Los Alamos National Laboratory.
9.1.3 Dark Energy Spectroscopic Instrument
This research used data obtained with the Dark Energy Spectroscopic Instrument (DEST). DESI construction and operations is managed by the Lawrence Berkeley National Laboratory. This material is based upon work supported by the U.S. Department of Energy. Office of Science Office of High-Energy Physies, under Contract No. DE-ACO2-05CHI 1231, and by the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility under the same contract. Additional support for DESI was provided by the U.S. National Science Foundation (NSF), Division of Astronomical Sciences under Contract No. AST-0950945 to the NSF's National Optical-Infrared Astronomy Research Laboratory; the Science and Technology Facilities Council of the United Kingdom; the Gordon and Betty Moore Foundation; the Heising-Simon Foundation; the French Alternative Energies and Atomic Energy Commission (CEA); the National Council of Science and Technology of Mexico (CONACYT); the Ministry of Science and Innovation of Spain (MICINN), and by the DESI Member Institutions: www.desi.lbl.gee wtt.eee......oooolllbbbaaaaaee wwwp/oolaaooaaaiigggaaaao.aa inniabbooattnnnnnsuuttoosssss DESI collaboration is honored to be permitted to conduct scientific research on lolkam Du's ag (Kitt Peak), a mountain with particular significance to the Tohono O' odhamm Nation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. National Science Foundation, the U.S. Department of Energy, or any of the listed funding agencies.
institutions. The
12

---

9.1.4 Sloan Digital Sky Survey
Funding for the Sloan Digital Sky Survey V has been provided by the Alfred P. Sloan Foundation, the Heising-Siman -iinin Foundation, the National Science Foundation, and the Participating Institutions. SDSS acknowledges support and resources from the Center for High-Performance Computing at the University of Utah. SDSS telescopes are located at Apache Point Observatory, funded by the Astrophysical Research Consortium and operated by New Mexico State University, and at Las Campanas Observatory, operated by the Carnegie Institution for Science. The SDSS web site is
www.sdss.org
SDSS is managed by the Astrophysical Research Consortium for the Participating Institutions of the SDSS Collaboration, including Caltech, The Carnegie Institution for Science, Chilean National Time Allocation Committee (CNTAC) ratified researchers, The Flatiron Institute, the Gotham Partici- pation Group, Harvard University, Heidelberg University, The Johns Hopkins University, L'Ecole polytechnique fédérale de Lausanne (EPFL). Leibniz- Institut für Astrophysik Potsdam (AIP), Max- Planck-Institut für Astronomie (MPIA Heidelberg). Max-Pllacck-aia (MPE). Nanjing University, National Astronomical Observatories of China (NAOC), New Mcxico State University, The Ohio State University, Pennsylvania State University, Smithsonian Astrophysi- cal Observatory, Space Telescope Science Institute (STScl), the Stellar Astrophysics Participation Group, Universidad Nacional Autónoma de México, University of Arizona, University of Colorado Boulder, University of Illinois at Trrbaa-Chhapalln University of Toronto, University of Utah, University of Virginia, Yale University, and Yunnan University.
für Extraterrestrische Physik
9.1.5 Gaia
This work has made use of data from the European Space Agency (ESA) mission Gaia (https 7/www. www.:cccoosccoo .sa.. int/gaia), processed by the Gaia Data Processing and Analysis Consor- tium (DPAC,,httss://// http///wwwcccosooooo///oo/ wwwmos.cccoooooooooo esa. intttintttwtteet haab//gia/dppaas dpac/ consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement. The Gaia data are open and free use. provided credit is given to "ESA/Gaiw/DPAC. If you use Gaia DR3 data in your research, please acknowledge it as above.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.082 087744 2023.
[2] A. G. Adame, J. Aguilar, S. Ahlen, S. Alam, G. Aldering, D. M. Alexander, R. Alfarsy
C. Allende Prieto, M. Alvarez, O. Alves, A. Anand, F. Andrade-Oliveira, E. Armengaud,
J. Asorey, S. Avila, A. Aviles, S. Bailey, A. Laagguera-Antlinezz O. Ballester, C. Baltay,
A. Bault, J. Bautista. J. Behera, S. F. Beltran, S. BenZvi, L. Beraldo e Silva, R. Bermejo- Climent, A. Berti, R. Besuner, F. Beutler, D. Bianchi, C. Blake, R. Blum, A. S. Bolton,
S. Brieden, A. Brodzeller, D. Brooks, Z. Brown, E. Buckley-Geer, E. Burtin, L. Cabayol-GGrria,
Z. Cai, R. Canning, L. Cardiel-Sas,, A. Carneoo Rosell, F. J. Castander, J. L. Cervantes-Cota. S. Chabanier, F. Chaussidon, J. Chaves-Monters S. Chen, X. Chen, C. Chuang. T. Claybaugh, S. Cole, A. P. Cooper, A. Cuceu, M. Davis, K. Dawson, R. de Belsunce, R. de la Cruz, A. de la Macorra, J. Della Costa, A. de Mattia, R. Demina, U. Demirbozan, J. DeRose, A. Dey, B. Dey, G. Dhungana, J. Ding. Z. Ding, P. Doel, R. Doshi, K. Douglass, A. Edge, S. Eftekharzadeh, D. J. Eisenstein, A. Elliott, J. Ereza, S. Escoffier, P. Fagrelius, X. Fan, K. Fanning, V. A. Faweett, S. Ferraro. B. Flaugher, A. Font-Ribera, J. E. Forero-Romero, D. Forero-Sánchez, C. Frenk, B. Т. Gänsicke, L. A. García, J. García-Bellido, C. Garcia-Quintero, L. H. Garrison, H. Gil-Marin, J. Golden-Marx. S. Gontcho A Gontcho, A. X. Gonzalez-Morales, Gonzalez- Perez, C. Gordon, O. Graur, D. Green, D. Gruen, J. Guy, B. Hadzhiyska, C. Hahn, J. Han, M. M. S Hanif, H. K. Herrera-Alcantar, K. Honscheid, J. Hou, Howlett, D. Huterer, V. Irăič, M. Ishak, A. Jacques, A. Jana, L. Jiang, J. Jimenez, Y. P. Jing. S. Joudaki, R. Joyce, E. Jullo, Juneau, N. G. Karaçaylı, T. Karim, R. Kehoe, S. Kent, A. Khederlarian, S. Kim, D. Kirkby, T. Kisner, F. Kitaura, N. Kizhuprakkat, J. Kneib, S. E. Koposov, A. Kovács, A. Kremin,
A. Krolewski, B. L'Huiilier, O. Lahav, A. Lambert, C. Lamman, T.-W. Lan, M. Landriau, D. Lang, J. U. Lange, J. Lasker, A. Leauthaud, L. Le Guillou, M. E. Levi, T. S. Li, E. Linder,
13

---

A. Lyons, C. Magneville, M. Manera, J. Manser, D. Margala, P. Martini, P. McDonald, G. E. Medina, L. Medina-Varela, A. Meisner, Mena-Feránndz. J. Meneses-Rine Rizz, M. Mezeua, J. Moon, S. Moore, J. Moustakas, E. Mueller, J. Mundet, A. Mutiere, A. D. Myers, Nadathur, I.. Napolitano, R. Neveux, A. Newman, Nie, R. Nikutta, G. Niz, P. Norberg, H. E. Noriega, E. Paillas, N. Palanque-Delabrouiile, A. Palmese, Z. Pan, D. Parkinson, S. Penmetsa, W. J. Percival, A. Pérez-Fernández, I. Feree-Rafols, M. Pieri, C. Poppett, A. Porredon, S. Pothier, F. Prada, R. Pucha, A. Raichoor, C. Ramirez-Pérez, Ramirez-Solano, M. Rashkovetskyi, C. Ravoux, A. Rocher, C. Rockosi, A. Ross, G. Rossi, R. Ruggeri, V. Ruhlmann- Kleider, C. G. Sabiu, K. Said, A. Saintonge, L. Samushia, E. Sanchez, C. Saulder, E. Schaam, E. F. Schlafly, D. Schlegel, D. Scholte, M. Schubnell, H. Seo, A. Shafieloo, R. Sharples, W. Sheu, J. Silber, F. Sinigaglia, M. Siudek, Z. Slepian, A. Smith, M. T. Soumagnac, D. Sprayberry, L.. Stephey, J. Suárez-Pérez, 7. Sun, T. Tan, G. Tarlé, R. Tojeiro, L. A. Ureña- López, R. Vaisakh, D. Valcin, F. Valdes, M. Valluri, M. Vargas-Magaña, A. Variu, L.. Verde, M. Walther, B. Wang. M. S. Wang, B. A. Weaver, N. Weaverdyck, R. H. Wechsler, M. White, Y. Xie, J. Yang, C. Yèche, J. Yu, S. Yuan, H. Zhang, Z. Zhang, C. Zhao, Z. Zheng, R. Zhou, Z. Zhou, H. Zou, S. Zou, and Y. Zu. The early data release of the dark energy spectroscopic instrument. The Astronomical Journal, 168(2):58, July 2024.
R. Miquel, P. Montero-Camache
[3] Amir Aghamousa, Jessica Aguilar, Steve Ahlen, Shadab Alam, Lori E Allen, Carlos Allende Prieto, James Annis, Stephen Bailey, Christophe Balland, Otger Ballester, et al. The desi experiment part i: science, targeting, and survey design. arXiv preprint orXiv:16110000 2016.
[4] Romina Ahumada, Carlos Allende Prieto, Andrés Almeida, Friedrich Anders, Scott F Anderson, Brett H Andrews, Borja Anguiano, Riccardo Arcodia, Eric Armengaud, Marie Aubert, et al. The 16th data release of the sloan digital sky surveys: first release from the apogee-2 southern survey and full release of eboss spectra. The Astrophysical Journal Supplement Series, (4:(1): 2020.
[5] Hiroaki Aihara, Nobuo Arimoto, Robert Armstrong, Stéphane Arnouts, Neta A Bahcall, Steven Bickerton, James Bosch, Kevin Bundy, Peter I. Capak, James HH Chan, al. The hyper suprime-cam ssp survey: overview and survey design. Publications of the Astronomical Society of Japan, (0(S1):5, 2018.
[6] Anthropic. Claude. https://anthropie cow/ colade 2024. Large language model.
[7] Roman Bachmann, Oğuzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4m-21: An any-to-any vision model for tens of tasks and modalities, 2024.
[8] D. Bina, R. Pelló, J. Richard, J. Lewis, V. Patrício, S. Cantalupo, E. C. Herenz, K. Soto, P. M. Weilbacher, R. Bacon, J. D. R. Vernet, L. Wisotzki, B. Clément, J. G. Cuby, D. Lagattuta, G. Soucail, and A. Verhamme. MUSE observations of the lensing cluster Abell 1689. 590:A14, May 2016.
[9] Adam S. Bolton, Scott Burles, Léon V. E. Koopmans, Tommaso Treu, Raphaël Gavazzi, Leonidas A. Moustakas, Randall Wayth, and David J. Schlegel. The Sloan Lens ACS Survey. V. The Full ACS Strong-Lens Sample. 682(2):964 9844 August 2008.
[10] Adam S. Bolton, Scott Burles, Léon V. E. Koopmans, Tommaso Treu, and Leonidas A. Mous- takas. The Sloan Lens ACS Survey. I. A Large Spectroscopically Selected Sample of Massive Early-Type Lens Galaxies. 638(2):703-724, February 2006.
[11] R. Cañameras, Schuldt, Y. Shu, S. H. Suyu. S. Taubenberger, T. Meinhardt, Leal-Taixé,
D. C. Y. Chao, K. T. Inoue, A. T. Jaclani, and A. More. HOLISMOKES. VI. New galaxy-scale strong lens candidates from the HSC-SSP imaging survey. 653:L6, September 2021.
[12] Ting Chen, Saurabh Saxena, Lala Li, David I Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint orXiv:2109. 10852, 2021.
[13] Gaia Collaboration et al. The gaia mission. arXiv preprint arXiv: (ele::0909044533, 2016.
14

---

[14] DESI Collaboration, A. G. Adame, J. Aguilar, S. Ahlen, S. Alam, G. Aldering. D. M. Alexan- der, R. Alfarsy, C. Allende Prieto, M. Alvarez, O. Alves, A. Anand,
E. Armengaud, J. Asorey, S. Avila, A. Aviles, S. Bailey, A. Balaguera-Antolinez, O. Ballester,
C. Baltay, A. Bault, J. Bautista, J. Behera, S. Beltran, S. BenZvi, L. Beraldo e Silva, J. R. Bermejo-Climent, A. Berti, R. Besuner, F. Beutler, D. Bianchi, C. Blake, R. Blum, A. Bolton, S. Brieden, A. Brodzeller, D. Brooks, Z. Brown, E. Buckley-Geer, E. Burtin, L. Cabayol-Garciaa
Z. Cai, R. Canning, L. Cardiel-Sas, A. Camnro Rosell, F. J. Castander, J. L. Cervantes-Cote Chabanier, E. Chaussidon, Chaves-Montero, S. Chen, X. Chen, C. Chuang. T. Claybaugh, S. Cole, A. P. Cooper, A. Cuceu, T. M. Davis, K. Dawson, R. de Belsunce, R. de la Cruz, A. de la Macorra, J. Della Costa, A. de Mattia, R. Demina, U. Demirbozan, J. DeRose, A. Dey. B. Dey, G. Dhungana, J. Ding, Z. Ding. P. Doel, R. Doshi, K. Douglass, A. Edge. S. Eftekharzadeh, D. Eisenstein, A. Elliott, J. Freza, S. Escoffier, P. Fagrelius, X. Fan, K. Fanning. A. Fawcett, S. Ferraro, B. Flaugher, A. Font-Ribera, J. F. Forero-Romero, D. Forero-Sánchez, C. S. Frenk, B. T. Gänsicke, L. A. García, J. Garcfa-Bellido C. Garcia-Quintera L. H. Garrison, H. Gil- Marín, J. Golden- Marx, S. Gontcho A Gontcho, A. X. Gonzalez-Morales, V. Gonzalez- Perezz C. Gordon, O. Graur, D. Green, D. Gruen, J. Guy, B. Hadzhiyska, C. Hahn, J. Han, M. M. S. Hanif, H. K. Hlrrera-Alcantar, K. Honscheid, J. Hou, C. Howlett, D. Huterer, V. Iršič, M. Ishak, A. Jacques, A. Jana, L.. Jiang, J. Jimenez, Y. P. Jing. S. Joudaki, R. Joyce, E. Jullo, S. Juneau, N. G. Karaçaylı, T. Karim, R. Kehoe, S. Kent, A. Khederlarian, S. Kim, D. Kirkby, T. Kisner, F. Kitaura, N. Kizhuprakkat, J. Kneib, S. F. Koposov, A. Kováes, A. Kremin, A. Krolewski, B. L'Huillier, O. Lahav, A. Lambert, C. Lamman, T. W. Lan, M. Landriau, D. Lang. J. U. Lange,
J. Lasker, A. Leauthaud, L. Le Guillou, M. E. Levi, T. S. Li, E. Linder, A. Lyons, C. Magneville, M. Manera, C. J. Manser, D. Margala, P. Martini, P. McDonald, G. E. Medina, L. Medina-
A. Meisner, J. Mena-Fernnndezz, Meneses-Rizze M. Mezcua, R. Miquel, P. Montero-Camacho, A. D. Myers, S. Nadathur, L. Napolitano, R. Neveux, J. A. Newman, J. Nie, R. Nikutta, G. Niz, Nor- berg, H. E. Noriega, E. Paillas, N. Saanuuee-elabbbullll.... A. Palmese. Z. Pan, D. Parkinson, Penmetsa, W. Percival, A. Férez-Fernández, I. Pérez-Ráfols, M. Pieri, C. Poppett, A. Porre- don, and S. Pothier. The Early Data Release of the Dark Energy Spectroscopic Instrument.. 168(2):58, August 2024.
J. Moon, S. Moore, J. Moustakas, E. Mueller, J. Mundet, A.
[15]
DESI Collaboration, Amir Aghamousa, Jessica Aguilar, Steve Ahlen, Shadab Alam, Lori E. Allen, Carlos Allende Prieto, James Annis, Stephen Bailey, Christophe Balland, Otger Ballester, Charles Baltay, Lucas Beaufore, Chris Bebek, Timothy C. Beers, Eric F. Bell, José Luis Bernal, Robert Besuner, Florian Beutler, Chris Blake, Hannes Bleuler, Michael Blomqvist, Robert Blum, Adam S. Bolton, Cesar Briceno, David Brooks, Joel R. Brownstein, Elizabeth Buckley- Geer, Angela Burden, Etienne Burtin, Nicolas G. Busca, Robert N. Cahn, Yan-Chuan Cai, Laia Cardiel-Sas, Raymond G. Carlberg, Pierre-Henri Carton, Ricard Casas, Francisco J. Castander, Jorge I.. Cervantes- Todd M. Claybaugh, Madeline Close, Carl Coker, Shaun Cole, Johan Comparat, Andrew P. Cooper, M. C. Cousinou, Martin Crocce, Jcan-Gabriel Cuby, Daniel P. Cunningham, Tamara M. Davis, Kyle S. Dawson, Axel de la Macorra, Juan De Vicente. Timothée Delubac, Mark Derwent, Arjun Dey, Govinda Dhungana, Zhejie Ding, Peter Doel, Yutong T. Duan, Anne Ealet, Jerry Edelstein, Sarah Eftekharzzdeh, Daniel J. Eisenstein, Ann Elliott, Stéphanie Escoffier, Matthew Evatt, Parker Fagrelius, Xiaohui Fan, Kevin Fanning, Arya Farahi, Jay Farihi, Ginevra Favole, Yu Feng, Enrique Fernandez, Joseph R. Findlay, Douglas P. Finkbeiner, Michael J. Fitzpatrick, Brenna Flaugher, Samuel Flender, Andreu Font- Ribera, Jaime E. Forero-Romero, Pablo Fosalba, Carlos S. Frenk, Michele Fumagalli, Boris T. Gaensicke, Giuseppe Gallo, Juan Marcia-BBellido Enrique Gaztanaga, Nicola Pietro Gentile Fusillo, Terry Gerard, Irena Gershkovich, Tommaso Giannantonio, Denis Gillet, Guillermo Gonzalez-de-Rivee Violeta Gonzalez-Perez, Shelby Gott, Or Graur, Gaston Gutierrez, Julien Guy, Salman Habib, Henry Heetderks, Ian Heetderks, Katrin Heitmann, Wojciech A. Hellwing, David A. Herrera, Shirley Ho, Stephen Holland, Klaus Honscheid, Eric Huff, Timothy A. Hutchinson, Dragan Huterer, Ho Seong Hwang, Joseph Maria Illa Laguna, Yuzo Ishikawa, Dianna Jacobs, Niall Jeffrey, Patrick Jelinsky, Elise Jennings, Linhua Jiang, Jorge Jimenez, Jennifer Johnson, Richard Joyce, Eric Jullo, Stéphanie Juneau, Sami Kama, Armin Karcher, Sonia Karkar, Robert Kehoe, Noble Kennamer, Stephen Kent, Martin Kilbinger, Alex G. Kim, David Kirkby, Theodore Kisner, Ellie Kitanidis, Jean-Paul Kneib, Sergey Koposov, Eve Kovacs, Kazuya Koyama, Anthony Kremin, Richard Kron, Luzius Kronig, Andrea Kueter- Kueter-Young, Cedric G. Lacey, Robin Lafever, Ofer Lahav, Andrew Lambert, Michael Lampton, Martin
15

---

Landriau, Dustin Lang, Tod R. Lauer, Jean-Mare Le Goff, Laurent Le Guillou, Auguste Le Van Suu, Jae Hyeon Lee, Su-Jeong Lee, Daniela Leitner, Michael Lesser, Michael E. Levi, Benjamin l''uuiliier Baojiu Li, Ming Liang, Huan Lin, Fric Linder, Sarah R. Loebman, Zarija Lukić, Jun Ma, Niall MacCrann, Christophe Magneville, Laleh Makarem, Marc Manera, Christopher J. Manser, Robert Marshall, Paul Martini, Richard Massey, Thomas Matheson, Jeremy McCauley, Patrick McDonald, Ian D. McGreer, Aaron Meisner, Nigel Metcalfe, Timothy N. Miller, Ramon Miquel, John Moustakas, Adam Myers, Milind Naik, Jeffrey A. Newman, Robert C. Nichol, Andrina Nicola, Luiz Nicolati da Costa, Jundan Nie, Gustavo Niz, Peder Norberg, Brian Nord, Dara Norman, Peter Nugent, Thomas O' O'Brien, Minji Oh, and Knut A. G. Olsen. The DESI Experiment Part I: Science, Targeting, and Survey Design. arXiv e-prints, page arXiv:161 1.00003,, October 2016.
[16] Arjun Dey, David Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R Findlay, Doug Finkbeiner, David Herrera, Stéphanie Juneau, et al. Overview of the desi legacy imaging surveys. The Astronomical Journal, 157(5):168, 2019.
[17] Hugh Dickinson, Dominic Adams, Vihang Mehta, Claudia Scarlata, Lucy Fortson, Stephen Serjeant, Coleman Krawezyk, Sandor Kruk, Chris Lintott, Kameswara Bharadwaj Mantha, Brooke D. Simmons, and Mike Walmsley. Galaxy Zoo: Clump Scout Design and first application of two-dimensional aggregation tool for citizen science. 518(4))5882-5911, December 2022.
[18] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[19] Cecile Faure, Jean-Paul Kneib, Giovanni Covone, Lidia Tasca, Alexie Leauthaud, Peter Capak, Knud Jahnke, Vernesa Smolcic, Sylvain de la Torre, Richard Ellis, Alexis Finoguenov, Anton Koekemoer, Oliver Le Fevre, Richard Massey, Yannick Mellier, Alexandre Refregier, Jason Rhodes, Nick Scoville, Eva Schinnerer, James Taylor, Ludovic Van Waerbeke, and Jakob Walcher. First Catalog of Strong Lens Candidates in the COSMOS Field. 176(1):19-38, May 2008.
[20] Petko Gemini Team, Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.055300 2024.
[21] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all, 2023.
[22] Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. xval: A continuous number encoding for large language models. arXiv preprint 1rXiv:2310.02980) 2023.
[23] Charles F. Goullaud, Joseph B. Jensen, John P. Blakeslee, Chung-Pei Ma, Jenny E. Greene, and Jens Thomas. The MASSIVE Survey. IX. Photometric Analysis of 35 High-mass Early-type Galaxies with HST WFC3/IR. 856(1):11, March 2018.
[24] ChangHoon Hahn, KJ Kwon, Rita Tojeiro, Malgorzata Siudek, Rebecca EA Canning, Mar Mezcua, Jeremy L Tinker, David Brooks, Peter Doel, Kevin Fanning, et al. The desi probabilistic value-added bright galaxy survey (provabgs) mock challenge. The Astrophysical Journal, 945(1):16, 2023.
[25] Md Abul Hayat, George Stein, Peter Harrington, Zarija Lukić, and Mustafa Mustafa. Self- supervised Representation Learning for Astronomical Images. 91 (1)(2),33, April 2021.
[26] C. Jacobs, T. Collett, K. Glazebrook, E. Buckley- H. T. Diehl, H. Lin, C. McCarthy, A. K. Qin, C. Odden, M. Caso Escudero, P. Dial, V. J. Yung, S. Gaitsch, A. Pellico, K. A. Lindgren, T. M. C. Abbott, J. Annis, S. Avila, D. Brooks, D. L. Burke, A. Carnero Rosell, M. Carrasco Costa, De Vicente, Fosalba, Frieman, J. García-Bellido, E. Gaztanaga, D. A. Goldstein, D. Gruen, R. A. Gruendl, J. Gschwend, D. L. Hollowood,
Kind, J. Carretero,
16

---

K. Honscheid, B. Hoyle, D. J. James, E. Krause, N. Kuropatkin, O. Lahav, M. Lima, M. A. G. Maia, Marshall, R. Miquel, A. A. Plazas, A. Roodman, E. Sanchez, V. Scarpine, S. Serrano, I. Sevilla-Noarbe, M. Smith, F. Sobreira, E. Suchyta, M. F. C. Swanson, G. Tarle, V. Vikram, A. R. Walker, Zhang, and DES Collaboration. An Extended Catalog of Galaxy-Galaxy Strong Gravitational Lenses Discovered in DES Using Convolutional Neural Networks. 243(1):17, July 2019.
[27] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver io: A general architecture for structured inputs & outputs, 2022.
[28] Anton T. Jaelani, Anuprecta More, Masamune Oguri, Alessandro Sonnenfeld, Sherry H. Suyu, Cristian E. Rusu, Kenneth Wong, James H. H. Chan, Issha Kayo, Chien-Hsiu Lee, Dani C. Y. Chao, Jean Coupon, Kaiki T. Inoue, and Toshifumi Futamase. Survey of Gravitationally lensed Objects in HSC Imaging (SuGOHII- V. Group-to-clusee scale lens search from the HSC-SSP Survey. 495(1):1291-1310) June 2020.
[29] Anton T. Jaelani, Anupreeta More, Kenneth C. Wong, Kaiki T. Inoue, Dani C. Y. Chao, Premana W. Premadi, and Raoul Cañameras. Survey of gravitationally lensed objects in HSC imaging (SuGOHI)- X. Strong lens finding in the HSC-SSP using convolutional neural networks.
(535()): 1639, December 2024.
[30] Diederik P Kingma, J Adam Ba, and J Adam. A method for stochastic optimization. arxiv 2014. arXiv preprini arXiv: 14/2.690, 106:6, 2020.
[31] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yer Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015 4026, 2023.
[32] Nolan Koblischke and Jo Bovy. SpectraFM: Tuning into Stellar Foundation Models. arXiv e-prints, page Stt.::4111.047555 November 2024.
[33] Henry W Leung and Jo Bovy. Deep learning of multi- malti-ellement abundances from high- resolutooo spectroscopic data. Monthly Notices of the Royal Astronomical Society, November 2018.
[34] Henry W. Leung and Jo Bovy. Towards an astronomical foundation model for stars with transformer- baasdd model. 527(())1494-155)))) January 2024.
[35] Rui Li, Yiping Shu, Jianlin Su, Haicheng Feng, Guobao Zhang, Jiancheng Wang, and Hongtao Liu. Using deep Residual Networks search for galaxy-Ly emitter lens candidates based on spectroscopic selection. (482(1):113-320, January 2019.
[36] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 2384022304.084455, 2023.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[38] Karen L Masters, Coleman Krawezyk, Shoaib Shamsi, Alexander Todd, Daniel Finnegan, Matthew Bershady, Kevin Bundy, Brian Cherinka, Amelia Fraser-McKelvie Dhanesh Krish- narao, Sandor Kruk, Richard R Lane, David Law, Chris Lintott, Michael Merrifield, Brooke Simmons, Anne-Marie Weijmans, and Renbin Yan. Galaxy Zoo: 3D crowdsourced bar, spiral, and foreground star masks for MaNGA target galaxies. Monthly Notices of the Royal Astronomical Society, 30733):923333333 08 2021.
[39] Karen L. Masters, Coleman Krawezyk, Shoaib Shamsi, Alexander Todd, Daniel Finnegan, Matthew Bershady, Kevin Bundy, Brian Cherinka, Amelia Freeve-MeKelvie, Dhanesh Krish- narao, Sandor Kruk, Richard R. Lane, David Law, Chris Lintott, Michael Merrifield, Brooke Simmons, Anne- Marie Weijmans, and Renbin Yan. Galaxy Zoo: 3D crowdsourced bar, spiral, and foreground star masks for MaNGA target galaxies. 507(3):3923-39)) November 2021.
17

---

[40] Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physies pretraining for physical surrogate models. arXiv preprint drXiv:2310.02994, 2023.
[41] Peter Melchior, Yan Liang, ChangHoon Hahn, and Andy Goulding. Autoencoding galaxy spectra. i. architecture. The Astronomical Journal, 166(2):74, 2023.
[42] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint Stt1:2309.155005 2023.
[43] Siddharth Mishra-Sharma, Yiding Song, and Jesse Thaler. PAPERCLIP: Associating Astro- nomical Observations and Natural Language with Multi-Modal Models. arXiv e-prints, page MEXiv:2403.0885/
March 2024.
[44] David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling, 2023.
[45] A. More, R. Cabanac, S. More, C. Alard, M. Limousin, J. P. Kneib, R. Gavazzi, and V. Motta. The CFHTLS-Strony Lensing Legacy Survey (SL2S): Investigating the group-scale lenses with the SARCS sample. In Journal of Physics Conference Series, volume 484 of Journal of Physics Conference Series, page 01 012041. IOP, March 2014.
[46] Masamune Oguri, Matthew B. Bayliss, Håkon Dahle, Keren Sharon, Michael D. Gladders, Priyamvada Natarajan, Joseph Hennawi, and Benjamin Koester. Combined strong and weak lensing analysis of 28 clusters from the Sloan Giant Ares Survey. -20((4)33213-323) March 2012.
[47] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/994 2023.
[48] Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg. Geraud Krawezik, Michael McCabe, Rudy Morel, Ruben Ohana, Mariel Pettee, Bruno Régaldo-Saut Blancard, Kyunghyun Cho, Shirley Ho, and Polymathic AI Collaboration. AstroCLII: cross-modal foundation model for galaxies. 49900)1490--011, July 2024.
[49] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[50] Mariia Rizhko and Joshua S. Bloom. astronomy. 2024.
Astrom self-supervised multimodal model for
[51] Andrea J. Ruff, Raphaël Gavazzi, Philip J. Marshall, Tommaso Treu, Matthew W. Auger, and Florence Brault. The SL2S Galaxy-scale Lens Sample. Cosmic Evolution of Dark and Luminous Mass in Early-type Galaxies. 727(2):96, February 2011.
[52] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Imagen: Photorealistic iext-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 22t.:2205..14877 2022.
[53] Stefan Schuldt, Raoul Cañameras, Irham Andika, Satadru Bag, Alejandra Melo, Yiping Shu, Sherry H. Suyu, Stefan Taubenberger, and Claudio Grillo. HOLISMOKES XIII: Strong-lens candidates all mass scales and their environments from the Hyper-Suprime Cam and deep learning. arXiv e-prints, page arXiv:2405.203: 203833 May 2024.
[54] Anowar J. Shajib, Pritom Mozumdar, Geoff C. F. Chen, Tommaso Treu, Michele Cappellari, Shawn Knabel, Sherry H. Suyu, Vardha N. Bennert, Joshua A. Frieman, Dominique Sluse, Simon Birrer, Frederic Courbin, Christopher D. Fassnacht, Lizvette Villafaña, and Peter R. Williams. TDCOSMO. XII. Improved Hubble constant measurement from lensing time delays using spatially resolved stellar kinematics of the lens galaxy. 673:A9, May 2023.
18

---

[55] Yiping Shu, Raoul Cañameras, Stefan Schuldt, Sherry H. Suyu, Stefan Taubenberger, Kaiki Taro Inoue, and Anton Jaelani. HOLISMOKES, VIII. High-redshift, strong-lens search in the Hyper Suprime-Cam Subaru Strategic Program. 662:A4, June 2022.
[56] Michael J. Smith, Ryan J. Roberts, Eirini Angeloudi, and Mare Mmertas-Compana AstroPT: Scaling Large Observation Models for Astronomy. arXiv e-prints, page MrXiv:2405.14950 May 2024.
[57] Alessandro Sonnenfeld, James H. H. Chan, Yiping Shu, Anupreeta More, Masamune Oguri, Sherry H. Suyu, Kenneth C. Wong, Chien-Hsiu Lee, Jean Coupon, Atsunori Yonehara, Adam S. Bolton, Anton T. Jaelani, Masayuki Tanaka, Satoshi Miyazaki, and Yutaka Komiyama, Survey of Gravitationally-Innse Objects in HSC Imaging (SuGOHI). Automatic search for galaxy-scale strong lenses. 70:S29, January 2018.
[58] Alessandro Sonnenfeld, Anton T. Jaelani, James Chan, Anupreeta More, Sherry H. Suyu, Kenneth C. Wong, Masamune Oguri, and Chien-Hsiu Lee. Survey of gravitationally-lensed objects in HSC imaging (SuGOHI). III. Statistical strong lensing constraints on the stellar IMF of CMASS galaxies. 630:A71, October 2019.
[59] Alessandro Sonnenfeld, Aprajita Verma, Anuprecta More, Elisabeth Baeten, Christine Macmil- lan, Kenneth C. Wong, James H. H. Chan, Anton T. Jaelani, Chien-Hsiu Lee, Masamune Oguri, Cristian E. Rusu, Marten Veldthuis, Laura Trouille, Philip J. Marshall, Roger Hutchings, Campbell Allen, James Donnell, Claude Cornen, Christopher P. Davis, Adam McMaster, Chris Lintott, and Grant Miller. Survey of Gravitationally--enned Objects in HSC Imaging (SuGOHI). VI. Crowdsourced lens finding with Space Warps. 642:A1 148, October 2020.
[60] Stability Stable diffusion, 2022.
[61] George Stein, Jacqueline Blaum, Peter Harrington, Tomislav Medan, and Zarija Lukić. Min- ing for strong gravitational lenses with self-supervised learning. The Astrophysical Journal, 937, 2022.
[62] George Stein, Peter Harrington, Jacqueline Blaum, Tomislav Medan, and Zarija Lukic. Self- supervised similarity search for large scientific datasets. arXiv preprint 10.111110 2021.
[63] Michael S. Talbot, Joel Brownstein, Kyle Dawson, Jean-Paul Kneib, and Julian Bautista. The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: catalogue of strong dalaxx-gaalxxy lens candidates. 502(3):4617-4644) April 2021.
[64] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, PMLR, 2019.
[65]
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024.
[66] The Multimodal Universe Collaboration, Jeroen Audenaert, Micah Bowles, Benjamin M. Boyd, David Chemaly, Brian Cherinka, Ioana Ciucă, Miles Cranmer, Aaron Do, Matthew Grayling, Erin E. Hayes, Tom Hehir, Shirley Ho, Mare Huertas-Company, Kartheik G. Iyer, Maja Jablonska, Francois Lanusse, Henry W. Leung. Kaisey Mandel, Juan Rafael Martínez- Galarza, Peter Melchior, Lucas Meyer, Liam H. Parker, Helen Qu, Jeff Shen, Michael J. Smith, Connor Stone, Mike Walmsley, and John F. Wu. The Multimodal Universe: Enabling Large- Scale Machine Learning with 100TB of Astronomical Scientifie Data. arXiv e-prints, page arXiv:2412.02527, December 2024.
[67] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. CoRR, abs/1711.00937, 2017.
[68] Mike Walmsley, Micah Bowles, Anna M. M. Scaife, Jason Shingirai Makechemu, Alexander J. Gordon, Annette M. N. Ferguson, Robert G. Mann, James Pearson, Jürgen J. Popp, Jo Bovy, Josh Speagle, Hugh Dickinson, Lucy Fortson, Tobias Géron, Sandor Kruk, Chris J. Lintott, Kameswara Mantha, Devina Mohan, David O'Ryan, and Inigo V. Slijepevic. Scaling Laws for Galaxy Images. arXiv e-prints, page arXiv:2404.02977 April 2024.
19

---

[69] Mike Walmsley, Chris Lintott, Tobias Géron, Sandor Kruk, Coleman Krawezyk, Kyle W Willett, Steven Bamford, Lee S Kelvin, Lucy Fortson, Yarin Gal, al. Galaxy zoo decals: Detailed visual morphology measurements from volunteers and deep learning for 314 000 galaxies. Monthly Notices of the Royal Astronomical Society, 509()):3966-399) 2022.
[70] Mike Wahmsley and Ashley Spindler. Deep Learning Segmentation of Spiral Arms and Bars. arXiv e-prints, page arXiv:2312.029900 December 2023.
[71] Mike Walmsley and Ashley Spindler. Deep learning segmentation of spiral arms and bars, 2023.
[72] Kenneth C. Wong, James H. H. Chan, Dani C. Y. Chao, Anton T. Jaelani, Issha Kayo, Chien- Hsiu Lee, Anupreeta More, and Masamune Oguri. Survey of Gravitationally lensed objects in HSC Imaging (SuGOHI), VIII. New galaxy-scale lenses from the HSC SSP. 7405):1209-12199 October 2022.
[73] Kenneth C. Wong, Alessandro Sonnenfeld, James H. H. Chan, Cristian E. Rusu, Masayuki Tanaka, Anton T. Jaelani, Chien-Hsiu Lee, Anupreeta More, Masamune Oguri, Sherry H. Suyu, and Yutaka Komiyama. Survey of Gravitationally Lensed Objects in HSC Imaging (SuGOHI). II. Environments and Line-of-Sight Structure of Strong Gravitational Lens Galaxies to z ~ 0.8. 867(2):107, November 2018.
[74] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders, 2023.
[75] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11133-16142, 2023.
[76] Donald G York, Adelman, John E. Anderson Jr, Scott F Anderson, James Annis, Neta A Bahcall, JA Bakken, Robert Barkhouser, Steven Bastian, Eileen Berman, et The sloan digital sky survey: Technical summary. The Astronomical Journal, 120(3):1779, 2000.
[77] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-10469, 2023.
[78] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng. Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion-tokcnizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.
[79] Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation, 2024.
[80] Meng Zhang, Maosheng Xiang. Yuan-Sen Ting, Jiahui Wang, Haining Li, Hu Zou, Jundan Nie, Lanya Mou, Tianmin Wu, Yaqian Wu, and Jifeng Liu. Determining Stellar Elemental Abundances from DESI Spectra with the Data-driven Payne. 273(2):19, August 2024.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Demonstration broad applicability of our foundation model astrophysics is illustrated in Section 6.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, comment on limitations and future work in our conclusion.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: No theoretical results involved in this paper.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide extensive information in appendix.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is included in supplemnetal material.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Information provided in appendix. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Limited evaluation resources have not given us the opportunity to compute error bars.
8. Experiments compute resources
21

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification: We report in appendix the compute required to train our models, including margin for experimentation.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: impact limited to astronomical sciences.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:
Justification:
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code included in supplementary material is documented.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: N/A
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
23

---

A Details on Pretraining Data
Below, we provide a brief overview of the details and data types from each survey, but refer the reader to [66] or the original source dataset paper for more exhaustive descriptions. We note that the per-survey magnitude/quality thresholds, footprint choices, Gaia XP availability, and our reciprocal cross-match together define the effective selection function of the pre-training corpus; this shapes which morphologies, redshifts, environments, and S/N regimes might be emphasized in AION-1's ''s embeddings.
A.1 Legacy Surveys
The DESI Legacy Imaging Surveys combine three wide-field programs on the Blanco, Mayall, and Bok telescopes, delivering uniform {g, r, i, 2] imaging over ~20 ~20 (00dee""-ru
half the sky. The
galaxies are imaged at a pixel scale of 0.262 arcsec. MMU provides 160 x 160 pixel postage stamp cut-outs centered on galaxies from the Data Release 10 [T6], which we crop 96696 images; we use only objects in the Southern Galactic Cap, and retain objects with mag_z < 21 that pass the MMU quality cuts, corresponding roughly 122 million galaxies.
Modalities Each object is packaged as: (1) calibrated {g. r, i, 2)} integrated fluxes and their inverse- variance estimates, (2) mid-IR fluxes W1-W4 from WISE, (3) Milky- -Way reddening E(B-V), and (4) basic shape/size descriptors the ellipticity components (et, e2) and circularised half-light radius Rott derived from the Legacy tractor model fits.
A.2 Hyper Suprime-Cam (HSC)
The Hyper Suprime-Cam Subaru Strategic Program delivers deep, high-resolution {g, r, i, 2, y} imaging over ~ 200 deg?. We use only the wide subset from PDR3 [5. From the co co-added calexp frames, MMU extracts 160 x 160-pixel cut-outs at pixel scale of 0.162 aresec centered on catalog sources, which we crop to 96 96, as with the Legacy Survey images. Objects are kept when they satisfy: mag_i < 22.5; at least three visits in every band ("full-depth full-colour"); and the standard HSC quality flags remove bright-star contamination, edge artefacts, saturation and unreliable cmodel photometry. The resulting sample contains roughly 2.5 million galaxies.
Modalities Each object is packaged as: (I) calibrated {g. r, i, 2, y integrated fluxes and their inverse- variance estimates, (2) PSF-homogenised forced photometry in each band with extinction corrections, and (3) the moment-based SDSS shape tensor components (711, 712 722) computed by the HSC pipeline.
A.3 Sloan Digital Sky Survey (SDSS)
The Sloan Digital Sky Survey (SDSS) has obtained medium-resolution (R~~ 2 000) optical spectra for millions of objects. We use the aggregated public optical spectra from the Legacy, SEGUE-1/2, BOSS and eBOSS program covering 3 650-10 600-10400Ā with resolutions of R = A/ = 1,500 at 3. 800A and R = 9, 000 at 9,000Å We keep only primary, science- target spectra from plates flagged PLATEQUALITY= good'. This yields 44 million galaxies and stars.
Modalities. Each object is packaged as: (1) the optical spectrum, its inverse-variance estimates, and its wavelength and (2) the pipeline redshift.
A.4 Dark Energy Spectroscopic Instrument (DESI)
The Dark Energy Spectroscopic Instrument 115] survey is collecting spectra for <40 million galaxies and quasars; MMU presently ingests the Early Data Release (EDR, 1% of the full survey) [1]. Each spectrum spans 1600-9 9 800A on a fixed 7,081-pixel grid resolutions of R 2000 at 3, 600Å and R 5, 500 at 9, 000À and is distributed with flux, wavelength and inverse-varianee arrays. We select spectra from the SV3 "one-peree pene-perennt survey where SV PRIMAAYY is true, DEJTYPE='TGT* and giving roughly 1 million galaxies, stars, and quasars.
We note that the SDSS and BOSS instruments have different fiber aperture sizes, but in the present work we include them in the same dataset.
24

---

Modalities, Each object is packaged as: (1) the optical spectrum, its inverse-varianee estimates, and its wavelength and (2) the pipeline redshift.
A.5
Gaia
Gaia DR3 [3] provides low-resolution prism spectra from its blue (BP) and red (RP) photometers for
220 million Milky-Way sources in addition to precise astrometry and broad-band photometry. MMU stores each BP/RP spectrum as the 110 Gauss Hermite coefficients released by the mission (55 BP + 55 RP). which can be resampled onto an 1 101-pixel wavelength grid via GaiaXPy. We include all DR3 objects that have mean BP/RP spectrum, retaining the full set of associated photometric, astrometric and cellar-parameter metadata.
Modalities. Each object is packaged as: (I) The 110 BP/RP spectral coefficients, (2) four-parameter astrometry (sky coordinates and parallax), and (3) mean fluxes in the QP BP and RP bands.
A.6 Cross-matching strategy
For each pair of surveys we perform a nearest-neighhour match within arcsec radius on the sky and keep only reciprocal matches. Every resulting match is materialised as its own dataset. Each object these datasets therefore aggregates all modalities from both parent surveys so that a single file read yields fully fused, multi-survey view of the same astrophysices object. During AION-1 pre-training we draw samples both from the individual survey datasets and from these cross-matched sets, detailed in the next sections. We note that this procedure may preferentially retain bright, isolated, well-centered sources and may de-emphasize blended or offset systems, introducing further selection effect on the joint training distribution (see subsec:limitations).
B Details on AION-1 Models
We provide in Table 5
description of the different configurations used for suite of models.
Table 5: Model size variants:
Following the choice of 4M model, adopt the conventional T5
model sizes and naming schemes
49
Model
Encoder Blocks
Decoder Blocks
Model Dim
Num Heads
Total Params
AION-1-B
12
12
768
12
300M
AION-1-L
24
24
1024
16
800M
AION-1-XL
24
24
2048
32
3B
Pretraining Details. We adopt an input budget of 256 tokens, and output budget of 128 tokens for all our models during pretraining. All models are trained with 6 mixed precision, and model distribution under PyTorch's Fully Sharded Data Parallel (FSDP) ZeRO-2 strategy. To achieve batch size of 8192 in all cases, we train AION-1-4 1-B using 64 H100 GPUs for 1.5 days, AION-1-L. using 100 H100 GPUs for 2.5 days, and 288 H100 GPUs for 3.5 days.
C Details on Tokenizers
C.1 Multi-Survey Image Tokenizer
C.1.1
Preprocessing
Each input from an imaging survey provides (i) a per-band flux map x, (ii) a pixel-wise inverse variance map Σ, and (iii) per-pixel mask m for given source. Our tokenizer ingests heterogeneous measurements drawn from both HSC (five filters {g. r. i, v)) and the Legacy Survey SGC (LS; four filters {g. r, z)). The two pipelines vary in central wavelength, pixel scale, zero-point, and noise. Therefore, we treat all bands from the surveys as distinct from eachother; i.e. from Legacy
'G is the mission's very broad "white-ligh light" band measured by the astrometric field CCDs.
25

---

Survey is treated as a different band than from HSC. We stack all distinct bands into single fixed set of channels (5 from HSC and from LS), assigning a specific index to each channel. Next, we map every image into 9-channel tensor, filling the subset of channels corresponding to that image's bands with flux values and setting any unused channels to zero; a binary mask m, E (0, 1] tracks which bands are zeroed out. The result of this process is that all images drawn have the same dimension, and can be stacked into a single, heterogenous batch, while maintaining survey-specific provenance information. We then normalize the zero-points between surveys by rescaling HSC to the Legacy Survey zero- prro-point of 22.5 mag via s = 10(2P-22.6)222 and multiply by the ratio of pixel scales. While these steps are not strictly necessary as the bands are already separated above we find that it helps with training stability. Finally, we apply an arcsinh normalization to the images to account for their high dynamic range, which we invert before computing the autoencoding loss. We find that adequate range compression is crucial for training stability.
C.1.2 Architecture and quantization.
Subsampled Linear Projection Given a batch of images, x E BB (batch by channels by height by width), we project it to higher-dimensionall space of size dimout s 6C using
x am + b),
with learnable W E Cxdimonnt and E Rddimoos The scale factor am)) keeps the feature norm invariant to missing channels. The projection is inverted after decoding. We introduce the subsampled linear projection to expand each image into higher-dimensional embedding that disentangles survey-specific channel information while preserving feature norms even when some bands are missing.
Autoencoder Once subsampled, we feed the output of the subsampled linear projection, which is now 54-dimensional image, into a baseed autoencoder. Specifically, we use the MagVit ViT architecture adapted from [77]. in which we remove transformer blocks. The encoder therefore consists of downsampling ResNet blocks, which reduce the dimensionality of the input image by factor of 16, resulting in a laten space that is 24 x 24 x 512; this is compressed to d = 4 dimensions before being fed the quantizer. The output of the quantizer then projected back to 512 dimensions, before being upsampled in the decoder. In total, the ResNet-based autoencoder has roughly 50M learnable parameters.
At the bottleneck of the tokenizer, we quantize features into a discrete set of codes. We experiment with multiple approaches, but empirically, we find that Finite Scale Quantiza- tion [FSQ: 42) yields the best performance in terms of reconstruction fidelity and training sta- bility. Further, to explore the trade-off between reconstruction loss and codebook utilization, we vary the codebook size from smaller (e.g.. 24) to larger (e.g.. 2-4)- following the recommended configurations in the FSQ paper - and observe that a size of 12 offers a desirable balance: the reconstruction loss plateaus with larger code- books, while code usage remains sufficiently high to avoid underfitting with smaller code- books; see (Fiuur for reference. Consequently, our final configuration employs FSQ with code- book levels of n, {8, 5, 5, 5), equating to rough size of 22 codes.
Quantizer
Image Tokenizer Reconstruction Error
0.00925
0.300000
0.00875
0.00sc0
0.00626
0.30SEU
0.30776
10
12
14
FSQ Quantizer Size (log 2)
Tokenizer Codebook Scaling: We present the image tokenizer reconstruction error (MSE) as a function of FSQ quantizer size. We choose 212 as our ultimate codebook size for im- ages, as it reconstruction loss appears to plateau around this point.
Figure 8:
C.1.3 Loss Function and Per-band Weighting
The tokenizer is trained using an seeeee-vrranee-weiie Gaussian negative log-likelihood (NLL.) that leverages prior knowledge of the noise properties in each image, as reported by the data-
26

---

Original
Tokenizer Recon.
Tractor Recon.
Tokenizer Residual
Tractor Residual
Figure 9: Image Tokenizer Performance: Reconstruction quality of the image tokenizer on three representative Legacy Survey images. The columns show, left to right, the original image, the reconstruction from the tokens, the reconstruction from the Legacy Survey tractor, the r-band residual for the tokenizer, and the r-band residual for the tractor.
generation pipelines. The NLL is given by:
DNLL Σ E m (x; Decs Deco(Enc,,xx))) 112
(5)
where is the input image, E; is the diagonal noise covariance provided by the imaging pipeline, accounting for background and shot noise from bright sources, and m, is the survey pipeline mask which removes masked pixels in the image.
C.1.4 Training Details
We train with the image tokenizer using the Adam 30 optimizer with a learning rate of 5 x 10-4 on batches of 256 images, sampling LS:HSC
20:1 ratio reflect the relative size of the two datasets. The learning rate is warmed up over 1k steps before being decayed for 400k steps using a cosine decay. of CNLL 0.00775. We show some representative samples of the tokenizer's reconstruction quality in Figure and include reconstructions from the Legacy Survey pipeline tractor for comparison.
Training converges in ~5 days on 4 x NVIDIA H100 GPUs, yielding a final reconstruction
C.2 Multi-Survey Spectrum Tokenizer
C.2.1 Preprocessing
Each input spectrum provides (i) observed-frame flux density per unit wavelength f()), (ii) inverse standard deviation (A). and (iii) per-pixel mask m mAA). Our tokenizer ingests heterogeneous measurements drawn from both DESI and SDSS. For each survey, (ignoring masked pixels), use log10 range compression, and quantize it with a 1-D scalar tokenizer (codebook size = 1024). We then normalize f and istd by f and stack them into a 2-channel array. The array is linearly interpolated onto fixed latent grid 8704 points covering 8500-10462.44 4A at 0.8A spacing; this common grid then shared between SDSS and DESI, and removes survey-specific wavelength/ddsserrioon differences.
compute robust median flux
27

---

Original
Plux
Flax
1000
3000
6000
7000
5000
5000
1000
3000
5000
5000
5000
Wavelength (A)
Wavelength (A)
Figure 10: Spectrum Tokenizer Performance: Reconstruction quality the spectrum tokenizer on two representative DESI EDR spectra. Top panels show the normalized flux as a function of wavelength, with original input (black) and reconstructed output (red) overlaid. Bottom panels show the residuals in units of the input uncertainty (o).
C.2.2 Architecture & Quantization
Autoencoder The stacked spectrum flux and inverse standard deviation IR Bx288700 is encoded by a 4-stage ConvNeXt- V2 backbone [75]. consisting of an initial downsampling stack composed of a 4 4 convolution and LayerNorm, followed by three downsampling stacks of 2 convolutions and LayerNorms. Each of the four downsampling stacks is followed by multiple ConvNeXt V2 processing blocks. This compresses the spectrum into a 273 512 latent space, which is then further downsampled to 10 dimension before being fed the quantizer; this dimensionality is chosen to conform to the 20 codebook size used by the quantizer. Like with the image, these steps are inverted during the decoding part of the autoencoder.
Quantizer We use Look-up- Free Quantizer [LFQ: 78 with an embedding dimension of ten (equating to codebook size of 1024 codes) to convert the latent sequence into discrete codes. Contrary to the images, we find here here that LFQ quantization slightly outperforms FSQ for the spectrum.
C.2.3 Losses
For each spectrum we project the decoder output back to its native wavelength grid and apply three losses:
1. Flux likelihood. Gaussian NLL weighted by inverse variance wA),, identical to Eq. (5)
2. Mask accuracy. Binary cross-entropy between the predicted reliability map mi() and the ground-truth mask m()).
3. Commitment. LFQ commitment loss with weight Bgg 0.25.
C.2.4 Training Details
We train with using the Adam W optimizer with constant 10-4 learning rate, 0.1 weight decay
penalty, and global batch size of 128. Training for 215 steps (~24 hours on 4 x NVIDIA H100) yields token reconstruction R² 0.994 and mask AUC of 0.92. Reconstruction quality two representative spectra from DESI are shown in Figure 10
C.3 Scalar Tokenizer
While the scalar values could be quantized directly, equal width binning directly in the data space would lead an uneven probability mass assignment and potentially imbalance training. Therefore, map every scalar value to unit normal Gaussian before quantization. To that end, we first need
28

---

to tabulate the empirical cumulative distribution function Fg on the training se map scalar value z, to a standard normal variate via
Once tabulated, we
2, + (Fa(t)))
(6)
where Φ-1 is the inverse CDF of (0, 1). Because 2 ~ (0, 1). equal-width binning in 2-space allocates the same probability mass to every bin, automatically adapting to long tails or sharp peaks in the original distribution. Each Gaussianised scalar 2; is quantised independently with an FSQ codebook of K 1024 centroids. Centroids are fixed a priori: we place them the K equally spaced quantiles of the standard normal, i.e. C₁ 4b (-k( (k- 3)/K). No parameters are learned and no loss required. To recover an approximate scalar value f; from its token G₁.
4 Fr (*(c))-
With K 1024 bins the median absolute reconstruction enror is below typical measurement certainties, ensuring that tokenisation fidelity is sufficient for downstream tasks while keeping the representation compact and parameter-free. Note that for some of the scalars with large dynamic ranges, we also apply a 1og or aresinh transform before CDF mapping and tokenization. We apply the scalar tokenizer the following scalars from each survey:
Legacy Survey: rg, fluxes, WISE W1-W4 fluxes, E(B -8) extinction, ellipticity components (e1, 62), circularized half-light radius Ren.
HSC: {g. r, 2, fluxes, shape tensor components.
SDSS & DESI: pipeline-reported redshift (z).
Gaia: 110 BP/RP coefficients, parallax, sky coordinates (ra, dec), G, BP, RP fluxes.
C.4
Scalar Field Tokenizer
In addition to images, we included an additional tokenizer specialized for scalar maps with values in [0, 1]. This tokenizer is particularly adapted to handle segmentation maps, could also be used to generate any property map scaled between 0 and I, such as Star Formation Rate maps derived from Integral Field Spectroscopy.
C.4.1 Data & Preprocessing
The scalar field tokenizer trained to autoencode mixture of 5 categories of normalized single- channel images derived from Legacy Survey photometry: RGB cutouts converted to grayscale; individual red, green, and blue channels from the RGB cutouts; and an 'object mask' indicating the silhouettes of sources detected in each cutout. The object mask is generated from the Tractor model photometry included in the Legacy Survey data release. The Tractor classifies each detected source as one of 5 morphological types and fits corresponding elliptical surface brightness model to the light emitted by the source. After fitting, parameters can be extracted from the surface brightness profile define a centered ellipse enclosing 50% of the total emission from given source. We generate an object mask for each cutout by painting such ellipses onto a null background for all sources detected in the cutout. The ellipses are filled with constant value selected from (0.2, 0.4, 0.6, 0.8, 1.0] which corresponds to the morphological type.
C.4.2 Architecture & Quantization
Autoencoder We base our architecture VQ-VAE 667; the encoder is comprised of stack of convolutional downsampling layers followed by 2 residual blocks, and this arrangement is mirrored in the decoder (with upsampling transpose convolutions replacing the downsampling convolutions). The downsampling convolutions have kernel size 4, stride 2, padding 1, and 128 256 / 512 kernels, respectively. Each residual block consists of sequence of 2 convolutional layers separated by batch norm layer, where the convolutions have kernel size 3 stride 1, and padding / 0, respectively. All layers are ReLU-activated.
'We estimate F' with fixed-size reservoir (N ~ 100 samples) maintained online during data streaming Reservoir sampling (Algorithm R) produces an unbiased CDF while keeping memory ONN). independent of the full catalog size.
29

---

Quantizer We use Finite Scale Quantizer 42 to quantize 4-dimensional codes with Adim 5, discrete levels available in the respective dimensions, yielding codebook size of 1000.
C.4.3 Training Details
This tokenizer was optimized under a mean squared error objective using the AdamW optimizer with the weight decay parameter set to 0.01. The model was trained with batch size 256 for 114,000 steps at base learning rate of 10-1. The base learning rate was modified by linear warmup phase in the first 1,000 steps and a cosine decay over the final 72,000 steps. The model weights were updated as an exponential moving average of previous values with decay parameter of 0.9999. Under these conditions, the loss converged to LMSE R 0.0017.
D Details on Generative Capabilities
AION-1 is generative model: once pretrained, it represents the joint distribution of all 39 tokenised modalities. At inference time we can therefore draw posterior samples of any modality in the training set by passing the appropriate query tokens and iteratively resampling them conditioned on the visible context. To generate these posteriors, we perform the following steps.
First, we pass the query modality through its appropriate tokenizer, producing a set of input tokens
x ,.,. NN,). These are passed to the AION-1 encoder, while the decoder receives sequence of query tokens x = (I.......... IN) whose values are to be inferred. At test-time we need
samples from
Po(x99
xx*) = II Po( xx))
jE
where Q indexes the query positions and pa is the categorical distribution produced by the frozen decoder. To perform this sampling, we follow the ROAR generation scheme introduced in 4M [44]: each iteration we
1. Draw a fresh random permulation म, Q + O. of the still-unknown query indices Q.;
2. Reveal the first P: = [r" [Qa]] positions of this permutation,
{m)( <p(p)));
3. Sample those tokens once from the model,
pal 1 xn Ux!!! (1-1) (.))))) E S;; 1
4. Promote them to inputs: x + t U (x/50) Jies, and update 2++1 = 2 S..
With a decay factor e [0, 1) the number of unresolved tokens drops exponentially, the full sample is generated = (log [)) decoder calls. After sampling is complete, every query token routed back through its sppelic tokenizer to recover the original data representation. Repeating the entire ROAR loop M times with a non-zero sampling temperature + > 0 yields M i.idd. draws (A)) from the conditional distribution in Equation 7
Importantly, we note that these draws are plausibility samples from the decoder's categorical outputs under an iterative reveal schedule; they are not guaranteed to be well-calibrated joint posteriors for sequences of tokens longer than single token. We discuss this limitation in further detail in subsec:limitations.
E
Details on Galaxy and Stellar Parameter Inference
Data (Galaxy Parameter Inference). For inferring galaxy parameters, we use the data from the PRObabilistic Value-Added Bright Galaxy Survey (PROVABGS) (24), which provides derived physical properties from galaxy photometry and spectroscopy using Bayesian Spectral Energy Density (SED) physical model. In particular, we extract:
2: Redshift of the galaxy
30

---

met: Metallicity
M,: Stellar mass
tage: Stellar population age
SFR: Star formation rate
We cross-match these galaxies with the Legacy Survey [66] imaging and photometry in the southern hemisphere and DESI spectroscopy (2). Then, we apply several quality cuts ensure reliable parameter estimates. In particular, we remove any objects with M, < 0 as well as objects with any unphysical photometric magnitudes (m < 0). After these cuts, we arrive at relatively clean sample of = 100, 000 galaxies suitable for testing and validating our inference pipeline.
To accommodate the large dynamic range in certain parameters, we take the logarithm of both Zmet. and M.. We also convert SFR into the specific star formation rate (sSFR), given by
SFR
sSFR = log
(8)
M.
which is often more insightful for characterizing the relative growth of stellar mass in the galaxy.
Data (Stellar Parameter Inference). The Gaia dataset that we train AION-1 is subset of the available data in Gaia DR3. We cross-match this subset with DESI to produce the sample for which evaluate stellar parameter inference. The training and validation sets consist of the cross-matched stars belonging to the training and validation healpixes during pretraining to ensure that the validation performed on stars that AION-1 has not seen during pretraining.
While the MMU dataset for DESI provides stellar spectra, we turn to the catalog [80] (hereafter Z24) for stellar parameters. Z24 uses data-driven method with regularization from physical models to provides estimates for basic stellar parameters like Tef, log g. and [Fe/H], as well as for various abundances [X/Fe] from DESI spectra.
Models.
We perform parameter inference by training linear models on top of AION-1 embeddings. Given some data inputs, we first tokenize them, then extract the corresponding embeddings from the pretrained and frozen AION-1 encoder. We experiment with both mean-pooling and cross-attention to compress the sequence dimension of the embeddings before applying a linear projection layer to the channel dimension to produce estimates for each quantity of interest.
Data Fusion. We train all downstream models on various input combinations as demonstration of data fusion. In particular, for galaxies we train downstream models on galaxy photometry, galaxy photometry and imaging, and galaxy photometry, imaging and spectroscopy. For stars: we train downstream models on stellar photometry, stellar photometry and low resolution spectroscopy, and low resolution spectroscopy and parallax.
Supervised Baselines In the following section we detail the dedicated, supervised baselines used for galaxy and stellar property prediction.
train two baseline models for stellar property prediction:
ConvNeXt Regressor on Raw Spectra. This model uses multiple stacks of alternating ConvNeXt (74) processing blocks and downsampling the encoder used for AION-1 spectrum final linear projection. It is trained directly the raw, pixel-level stellar spectra and noise estimates.
in architecture by attention pooling and a
BGBoost on Token Representations. We train an XGBoost regressor on the mean-pooled tokens produced by the AION-1 spectrum tokenizer. Because these tokens are already a compressed, high-level representation of the spectra, the XGBoost model's 's task is. in principle, simpler than that of the ConvNeXt regressor, which must simultaneously learn to extract features and perform the final prediction.
For galaxy property prediction, train three different baseline models:
31

---

XGBoost on Photometry. We train an XGBoost regressor directly on the photometric measurements of galaxies (e.g. magnitudes or fluxes in various bands).
ConvNeeXt-iny on Images. As second baseline, we adopt the Conv-NeXt-tnn architecture trained on galaxy images. This approach matches the baseline provided in the Multi-Modal Universe (MMU) framework.
Convolutional + Attention Network on Spectra. Inspired by the method of 41 and used as baseline in 48), we train a network that combines convolutional layers with attention-bass based pooling on the galaxy spectra.
Self-Suuee Baselines. We also use a number state-of-the-arr self-supervised baselines from the literature. For galaxies, we use the following models:
AstroCLIP 48 a previous state-of-the-an multimodal foundation model for galaxies; we follow the authors' recommended protocol, extracting frozen embeddings from the CLIP image encoder and training a lightweight MLP ontop of the embeddings. Note that AstroCLIP was trained on Legacy Survey {g, r, 2} cut-outs only, so in our setting it has access to one fewer band (i) than AION-1.
represents widely used vision model; we feed RGB-converted (g, =} images to the ViT-g/14 backbone and again attach the same MLP probe.
DINOv2 47
For stars, we compare performance against current -fate-of-the-arr baseline from 34 who de- veloped Transformer- foundation model for stellar data. More specifically, the task is to predict APOGEE deriieed stellar parameters namely Teh, log and [Fe/H] from Gaia XP spectral coefficients. We use the same data as (34), and cross-match APOGEE-derived stellar parameters with the MMU Gaia data, producing set of roughly 10, 000 APOGEE parameter-Gain XP spectral pairs. We feed as input to both AION-1 and the 34 model only the first BP coefficients and first 32 RP coefficients due to the fact that the 34 model only has context length of 64; we artificially handicap AION- which does not have this restriction in order to perform a fair comparison. We note here that (34] has been explicitly given APO APOGEE-derived stellar parameters and Gaia XP coefficients during its pretraining stage, and so this task is one that it has effectively been trained for. On the other hand, the pretraining dataset for AION-B does not contain APOGEE data, nor any other stellar parameters; we simply train a simple linear projection layer with cross-attention pooling on 5000 paired examples, and leave the weights of the AION model itself frozen.
Performance and Scaling. We present below an extended set of results on stellar and galaxy parameter inference.
we show the performance of AION-1-B, AION-I-L, and AION-1-XL well of the two baselines, on the prediction of individual properties as a function of training set size. We find that in general, AION-1 outperforms the baselines across the board, with the XGBoost baseline typically being slightly worse and the ConvNet baseline being significantly worse, especially in the low-data regime where only 100 training samples are available. However, while the ConvNet baseline performs very poorly with few samples, its performance improves with training set size up until the full training sample of 50, 000 examples, at which point it performs essentially identically to-and in some cases better than-AlON-1 MION-I-B/L/X -BBLL/LL and the XGB baseline. is possible that the regression task is not particularly difficult, and thus model performance saturates early; with AION- 1-B performing essentially the same as AION-1-L, we find that scaling up to a large model provides little benefit for this task.
In Figure 12
Figure 11 shows that overall, independent of input modalities, cross-attention pooling (solid lines) significantly outperforms mean-pooling (dashed lines).
F Details on Morphology Classification
Data. We construct the downstream sample by cross-matching the Galaxy Zoo 10 catalog 69
33
GZ10:: with the Legacy Survey DR10 imaging footprint, yielding ~ 8, 000 galaxies with (g. r, 2) cutouts.
32

---

M.
Tago
Met
SFR
Mean
Attention
Mean
Attention
Mean
Attention
Mean
Attention
Mean
Attention
AION-I-B
Ph
0.736
Phitin
0.910
0,934
0.857
0.886
0.394
0.445
0.453
0.490
0.611
0.637
Sp
0.779
0.995
0.739
0.956
0.258
0,532
0,365
0.610
0.439
0,720
AION-I-L
Ph
0.659
0,761
0.630
0.734
0,268
0.357
0,302
0.411
0.228
0.387
Phi+Im
0,922
0,940
0,870
0.889
0,412
0.454
0.460
0.496
0.621
0.642
Ph Sp
0,800
0,995
0,760
0,955
0.276
0.534
0,375
0.620
0,461
0,727
AION-1-XI.
Ph
0.679
0.792
0.647
0.757
0.266
0.314
0,318
0.379
0.240
0.475
Phc+In
0.910
0,940
0.857
0.888
0.394
0.450
0.439
0.490
0.610
0.644
Ph+In+Spp
0.795
0.992
0.759
0.947
0,273
0.534
0.374
0.621
0.454
0.731
Supervised
Supervised
Supervised
Supervised
Supervised
PM Im?
0.708
0.692
0,301
0,301
0.377
0.864
0.821
0,445
0.489
0.638
Sp
0.998
0.852
0.433
0.621
0.675
Table 6: R (1) for galaxy property estimation. Inputs the model are: photometry (Ph),
photometry and imaging (Ph+Im), and photometry, imaging, and spectra (Ph+/m4Sp), Mean implies AION-1 embeddings followed by linear projection head, while Attention
taking the average implies training a cross-attention layer with a linear projection head on the full set of embeddings. Supervised models are: XGBooost ConnNNex, *Convolution Attention Network. All models trained on 100, 000 examples.
[FoH]
[Oiee]
[ANTe]
[CMFe]
[TMFF]
[NNTe]]
[NUTe]
250
-0.25
4.125
22
5.1.0
5.089
-0.360
6..000
6.272
0.27
200
10.08
2.14
E,070
60.66
175
-0.2550
1,4
6.385
4.10A
350
5,24
0.380
-0.an
00.0k
0.15
4.005
0.260
125
6.15
0.275
0,315
6.377
2.12
0.150
4.055
16.155
-0.70
16:93
1.090
0.750
3.00
0.35
0.200
(olens Mar 14-
Deeer Saee
DWegr Seec Ric
Uilord Pie
Hilord EP Piee Nr.
Figure 11: Predictive performance measured by standard deviation of residuals held-out evaluation set of models with various input combinations on various physical properties of stars. Lower is better. Each line represents different model, with dashed lines indicating mean-pooled linear probes and solid lines indicating cross-attention pooled linear probes. The color of the line represents the inputs that the model is given. All embeddings are generated from the frozen, pretrained encoder of Aion-1-L.
33

---

lop,9
Taf
[Fe/H]
1.4
0.45
AITN-
ARON-I
ARON-S
1.2
AIUNAL
ARUN-L
AIUN-L
Sfd (1)
CNN
Sid of residuual (1)
950
Sidd of (J)
1.0
XLII
200
XGIS
0.30
XLII
0.8
0.25
0.6
200
150
0.4
0.10
0.2
10
10F
102
10
102
10P
10a
102
30
10f
Training Size [C/Pe]
Training Size
[Ai/Fe]
[Ca/Fe]
0.130
AION-B AION-XL CNN
0.125
AION-XL
AION-XL
Sel resiital (1)
of of resichal (L)
CNN
of resihall (L)
CNN
0.120
0.14
0.10
0:115
0.110
0.12
0.08
0.10
0.08
0.04
10a
10
10a
10"
10a
10a
10'
1df
Training Set Size [Cn/Fo]
Training Sei Size
Triinnn Set Siee [Ma/Fe]
[Mg/Fe]
ABON-B
AION-B
ADON-B
ARON--
0.10
ANTN-
ARTI-
0.14
XL
AION-LL
AION-ML
Sid of residoal (1)
Sid of resduull (U)
Sid of reschal (1)
XGI
0.07
0.10
0.06
0.09
0.08
0.08
10
10a
10a
10at
10a
10a
10
Training Size
Triining Size
Trininn
Figure 12: Predictive performance of AION-I-B/L/XL. convolutional baseline from raw spectra, and an XGBoost baseline from spectrum tokens as function of training set size for different stellar properties. Performance measured in terms of the standard deviation of the residuals, and thus lower is better.
Models. For AION-1, tokenize each cutout with the multi-survey image tokenizer, mean-pool the resulting embedding and pass the 768-d mean vector to a two-layer MLP head (hidden size 256, GELU, dropout 0.T). The head is trained 80% of the sample with class-stratified splits and evaluated on the remaining 20%.
Baselines. We replicate this protocol with the DINOv2 baseline, replacing the tokenizer with the ViT-g/14 backbone and applying the RGB normalization recommended by [47] EfficientNet-B3 trained end-to-an -od-to-end from random initialization using the same splits and standard data augmentations. Finally, we adapt ZooBot [69] by finettuning the penultimate layer on our 000 samples; although ZooBot was never exposed to GZ10 labels, it benefits from pre-training on ~ 300, 000 images covering the broader, harder GZ-5 decision tree, and thus acts as an approximate upper bound on achievable accuracy.
Results. As Table 4 shows, AION-1-L tops all baselines except ZooBot, exceeding EfficientNet by +7. pp and DINOv2 by +15.8 pp. while using only a lightweight MLP head. Moreover,
'Although we experiment with attentive pooling in this setting, unlike with property estimation, find that attentive pooling does provide any meaningful gain in accuracy.
34

---

[N/Fe]
[Ni/Fe]
[O/Fe]
0,100
0.24
AICN-II
AIUN-
0.096
AIUN-
AIONA
AIUN-L
ARTN-HII
0.22
Std rf (L)
0.000
CNN
0.16
of
XCIII
XLBI
Sid of
XGI
0.080
0.18
0. 0.075
0.12
0.06
0,070
0.065
0.14
104
10a
10
102
104
10a
10
10
Training
Training Size
[Sü/Fe]
[TV/F]]
0.10
ADON-B
AION-B
0.825
ARON-B
AJON-L
0.14
AION-XL
AION-NA XL
0.900
AION-XL
CNN
Sal resiual (1)
CNN
0.275
CNN
XGB
of resiiall
0.260
0.12
0.225
0.200
0.10
0.175
0.08
0.08
0.150
0.06
0.125
10P
10a
10a
10a
10a
10a
101
Trininn Set Size
Training Set Size
Training Sel Siop
Figure 12: (continued)
reaches close to the ZooBot accuracy, only under--erforming by -2.4 pp. despite seeing two orders of magnitude fewer labeled galaxy images during inference process.
G Details on Dense Predictions
In this section, provide additional details on our dense fine-tuning experiments, which involve predicting segmentation maps and detecting sets of objects from image inputs. While prior work by as an autoregressive sequence generation problem-we we refer to it as dense prediction in contrast to our scalar prediction tasks. This distinction emphasizes the structured nature of segmentation and object detection compared to simpler regression-based outputs.
[4 7 classifies object detection as sparse prediction cask-treatinn
G.1 Architecture
Semantic Segmentation. We implement a lightweight convolutional upsampler trained on top of AION-1'S 's encoder representations. Our upsampler design is largely inspired by the mask decoder from B1 but with a key modification: we do not include hypernetworks instantiated from additional register tokens. Instead, desired number of segmentation maps, simplifying the architecture while maintaining efficiency.
single convolutional layer to project the upsampled output the
Clump Detection. For clump detection, we introduce no additional model parameters. Instead, we finetune AION-1's decoder to autoregressively generate lincarized object tokens. following the same tokenization scheme used in our pre-training catalog data. This approach enables the model to predict structured object sequences without requiring task-specific modifications.
G.2 Galaxy Zoo 3D Segmentation
Galaxy Zoo 3D is a dataset derived from volunteer annotations of galaxies, originally presented in 139 and collected through the Zooniverse citizen science platform. Each galaxy sample was
http::/wwww..ooovvee.rorrr//// hny.://eeeearrb..eeeeeeeaaaaaage
35

---

Spiral Segmentation
Bar Segmentation
0.82
0.60
9.31
0.39
0.30
InloU (1)
0.68
0.22
0.57
0.56
0.27
AION-B
AION-L
0.26
Sapervised Baclin
0.20
1750
2000
2250
2600
2750
2260
2500
2750
Training Set
Training Set Size
Figure 13: IoU scores for spiral arm and bar segmentation across different models, evaluated at three fractions of the available training data 33% (937), 66% (1874), and 100% (2811).
annotated by 15 volunteers, who were asked to mark the galactic center, any stars in the frame, and to draw bounding boxes around galactic bars and spiral arms.
For this study, we focus on the vote maps, which consist of four dense arrays containing pixel-wise annotation counts (ranging from 0 to 15) indicating how many annotators included given pixel in their annotations. Segmenting bars and spiral arms is particularly challenging, which is why human annotations are crucial for this task.
Following 771), we filter out samples that do not reach a confidence level of 0.2 and compare results to model proposed in the same study, which trained ourselves. The ground truth segmentation is defined as the set of pixels that received any number of votes from annotators, normalized by the maximum number of votes received per sample.
For our dense predictions, we finetune small convolutional head on top of AION's frozen encoding module and optimize the trainable parameters using mean squared error (MSE).
We evaluate model using the Intersection Union (IoU) metric, reporting separate IoU scores for spiral arms and bars rather than a combined mean IoU (mloU). The IoU is defined as:
M M M U M
IoU
where M, M E [o, x| denote the predicted and ground truth segmentation masks, respectively.
We determine separate segmentation thresholds for spiral arms and bars using 20% of our validation set. These thresholds are computed independently, as annotators tend to agree more consistently on bar structures than spiral arms. This approach ensures optimal segmentation performance for both components.
G.3 Galaxy Zoo Clump Detection
We additionally investigate the Seq2Seq problem autoregressively generating galaxy clumps [17 Specifically, we finetune AION-1 to generate an ordered sequence of clumps where the number of clumps varies across different examples - by conditioning on Legacy Survey Images. To achieve this, we use the catalog tokenizer used during pre- pre-training, which encodes catalog objects into structured sequence of quintuples, each consisting of pixel coordinates, elliptical shapes, and radius. We then finetune the model with a causal language modeling objective, conditioning on both the previously generated clumps and the corresponding Legacy Survey images. This setup allows model to learn spatial and morphological dependencies among clumps, ultimately improving its ability to generate realistic clump distributions for galaxy images. This dataset consists of 3727 cross-matched samples. Some qualitative examples are shown in Figure 14
36

---

Comparison Predicted vs. Ground Truth Clumps
Predicted Clump
Ground Truth
Figure 14: Qualitative examples of ground truth and predicted clump objects in Legacy Survey.
H Details on Retrieval Evaluation
H.1 Evaluation Metric
For each query galaxy, we first produce embedding using AION-1. We then generate embeddings for all other galaxies in the search corpus. All embeddings are averaged together to produce a single vector, E Rd, where is the embedding dimension of the specific AION model used. Next, compute the cosine similarity between the query embedding, and each candidate embedding X
[xq
X
Sa(xq, xe)
(10)
||x=
The entire corpus (excluding the embedding) then ranked in descending order of similarity. We compute the normalized Discounted Cumulative Gain (nDCG) retrieved objects, where the relevance, r, of each object, i, is determined by the criteria described in the sections below. Specifically, the DCG at rank is defined
n -
DCG@k = Σ
(11)
log, ( 1)
The ideal DCG (IDCG) is computed by sorting the items by descending relevance. The normalized DCG is then
nDCG@k DCG@k
(12)
IDCG@k
37

---

Mergers
Spirals
1.0
0.8
0.5
top- ap-10
0.4
AION-B
0.2
0.2
AION-L
AION-X
Stein,
10
175
20
10
14
20
Query Vector Subset Size
Query Voctor Subseet Size
Figure 15: nDCG@ @ 10 as a Function of Aggregated Query Size. Overall, we find that mean- pooling multiple queries of the same object type form our query vector dramatically improves model performance when searching for mergers and spirals, up to a query count of roughly 15. This holds for all models.
In our experiments, we focus 10 and report nDCG@10 our evaluation metric.
H.2 Galaxy Zoo Object Labels
Data. We use the Galaxy Zoo DECaLS catalog [69], which provides citizen-science morphological classifications for large set of galaxies observed in the Legacy Imaging Survey [6]. We then cross-match these classified objects with corresponding Legacy Survey South images. To ensure that each galaxy has sufficiently reliable volunteer annotations, we discard any objects with fewer than three total volunteer votes. This leaves & 171, 000 galaxies.
Within this filtered dataset, we focus on two morphological classes of interest: mergers and spirals. We select nigh-confidence examples of these two classes by identifying galaxies for which more than 0.9 of the volunteers have voted for the corresponding morphology (merger or spiral). These high-confidence galaxies form our set of query objects in the retrieval experiments. In total, we have 726 merging galaxies and 24, 622 spiral galaxies.
For each object in the dataset, we define a relevance label based on the fraction ff of volunteers who voted for the same morphological class. Thus, object whose volunteer vote distribution aligns more strongly with merger or spiral morphology is assigned a higher relevance label than one whose distribution more ambiguous. This setup encourages the retrieval model prioritize both the correct morphological class and the degree of confidence in that classification.
Aggregated Query Results. addition to evaluating single query objects, we investigate whether aggregating multiple queries can improve retrieval performance. Specifically, instead of using single galaxy embedding as the query, we compute a single query embedding by averaging the embeddings of multiple galaxies from the same morphological class (merger spiral). The rationale is that features indicative of the given morphology will be reinforced across several galaxy embeddings, while idiosyneratic features unrelated to that morphology will be muted.
shows how the nDCG@ 10 score varies with the number of query embeddings being averaged. We observe that performance systematically increases up to about 15 queries, after which gains plateau. particular, for merger queries, this approach more than doubles the nDCG@ 10 compared to single-query retrieval, while for spirals it boosts nDCG@ @ 10 by approximately 0.25. Notably, even with aggregated queries, our best models still outperform the baseline method 62 underscoring the effectiveness of this aggregation strategy.
Figure 15
38

---

LS LS
HSC HSC
LS HSC
HSC LS
AION-I-B
0.012
0.018
0.004
0.016
AION-1-L
0,011
0.019
0.004
0.017
AION-I-XL 0.015
0.015
0.004
0.012
0.007
Table 7: nDCG @ 10 for Strong Lensing Retrieval. We evaluate retrieval performance strong gravitational lenses, measuring nDCG for galaxies retrieved via cosine similarity of AION-I 's average token embeddings. Each column is labeled X - Y, X being the modality used to produce the query embedding and Y the key embedding. For all four columns, we show the results for the three AION-1 variants (B, L, XL). Since the state-of-the-art self-supervised baseline only generates legacy survey (LS) embeddings, we only show its results on the LS + LS task.
H.. Strong Lens Finding
Data. For the strong lensing retrieval task, we start by filtering the cross-matched catalog of objects within the Legacy Survey and HSC datasets to approximately reproduce the parent sample used in the HSC strong lensing searches (2)]. Specifically, we impose three additional cuts: (1) objects with photometric redshifts between 0.2 and 1.2, (2) objects with an estimated stellar mass above x M and (3) objects with a star formation rate to stellar mass ratio less than 1 x 10 In order to identify the strong gravitational lenses within the resulting parent sample, we pross-match
with previous lens-finding catalogs (57 73 58 59
72 29
55 53 35
9 51 19
26
Most strong lensing catalogs offer grade for each candidate. Since the criteria for this grading varies between catalogs, we ignore these grades and instead assign a relevance score of 1.0 to all the strong lensing candidates found within each catalog. All other objects in our parent sample are given relevance score of 0.0. Even with the additional filtering, strong gravitational lenses make up only 0.1% of our parent sample.
23
46
For each object in the parent sample, we have both the HSC and Legacy Survey observations. For AION-1, we extract the embeddings by passing the tokemized observation through the frozen encoder. We then average the output over the patch dimension. For the state-of-the-aa baseline model [62], we directly use the representation output by the model. Unlike AION-1, the baseline is trained solely for legacy survey images. The resulting AION-1 embeddings have 768 dimensions for both modalities, whereas the baseline model embeddings have 128 dimensions for the Legacy Survey modality.
Results. Since we have both HSC and Legacy Survey (LS) embeddings for our strong lens catalog, we can perform two retrieval tasks within modality: LS query with LS keys (LS + LS) and HSC query with HSC keys HSC + HSC. We can also explore two retrieval tasks between modalities: L.S query with HSC keys (LS + HSC) and HSC query with LS keys (HSC + baseline only enables LS - LS. The nDCG@ 10 metrics for these tasks are reported in Table 7
LS). The state-of-the tfe----rtttt
We find that we outperform the state-of-the-art on the LS + LS task for all three AION-1 model sizes, with the largest model leading the greatest performance improvement. Switching from the LS + LS task to the HSC + HSC task leads to further gains in the retrieval metric, confirming that AION-1 successfully leveraging the higher-resolution information present in the HSC images. For the cross-modality tasks we find mixed performance for AION-1. On the LS - HSC task we get the worst retrieval performance of any modality combination, but on the HSC - LS task we get equivalent retrieval performance to the HSC - HSC task. One possible cause is that our retrieval depends on informative query embeddings. For example, in Appendix |subscction H.2 we find that aggregating query embeddings leads significant improvements in performance galaxy morphology retrieval. For strong lensing retrieval, the HSC embeddings are derived from more informative (higher-resolution)) observations than the LS embeddings. This leads to better performance when we query with the more informative HSC embeddings over the LS embeddings.
The low overall nDCG@ 10 score for all models and tasks reflects the inherent challenge in retrieving strong-lensing images. The Einstein ring that characterizes a strong gravitational lens is often dim compared to other features in the image, and strong lenses themselves are incredibly rare events. Despite this challenge, the state-of-the-an model we outperform has already been used to identify 1 1192 new strong lensing candidates [61].
39

---

True DESI spectrum
2=0.588
Sampled spectra
z-0.174
H5C Image
4000
5000
6000
7000
8000
9000
10000
Observed envehength (A)
Figure 16: Out-of-distrbbution conditional generation. DESI spectra sampled from AION-1 (blue) conditioned on HSC images (insets), overlaid on the true DESI spectra (black). Even though HSC-DESI pairs were never seen during pre-training, the model reproduces key spectral features, demonstrating emergent transitive understanding.
H.4
Emergent Transfer Properties
H.4.1 Generative Transfer of Multimodal Understanding
Our training mixture contains many pairwise matches, e.g. HSC images SDSS spectra and SDSS spectra < DESI spectra. However, the model is never trained to on the DESI-HSC pairs, and so never learns produce HSC images from DESI spectra directly. Nevertheless, when we condition use AION-1 1 to sample a DESI spectrum from an HSC image, the generated spectrum (blue) closely tracks the ground-truth DESI spectrum (black); see Figure 16) Both quiescent (red) galaxy and star- forming (blue) galaxy are reproduced with realistic absorption and emission features, demonstrating that AION-1 has learned transitive mapping across modalities. This is likely due to the fact that AION-1 already understands the mapping between HSC and other spectroscopic (SDSS) or imaging (Legacy Survey) surveys, as well as the mapping between those intermediate surveys and the final target survey, DESI.
H.4.2 Survey-to-survey Transfer in Embedding Space
Beyond conditional generation, ask whether AION-1's 's frozen image encoder produces survey- invariant representations that let us port knowledge from one telescope to another. Specifically, train single linear classifier Legacy Survey embeddings morphology classes. The encoder weights remain fixed; only the 10-way soft-max layer is optimized. We then apply this exact head-withut any fine fine-tuing--t embeddings of Hyper Suprime-Can (HSC) images. To create the evaluation set we cross-match the HSC wide catalog (see (seeescbbnttion A.2 with Galaxy Zoo-10 10 (GZ10) volunteer votes and explicitly remove any targets that overlap with the Legacy Survey-GZI0 training set to prevent test leakage. The resulting sample contains ~1,000 galaxies. As reported in Table 8, the zero-shot classifier attains 84 86% accuracy across all AION-1 scales, essentially matching its performance on the native Legacy Survey domain. This robustmess holds despite factor-of-~2 differences depth, distinct filter sets ((g, r, 2, 2, vs. {g, i, =)), and a different pixel scale. The result underscores that AION-I embeddings capture morphology in way that largely agnostic to survey-specific imaging characteristics, enabling workflows that recycle scarce labeled data from one survey to bootstrap science in another.
predict the Galaxy Zoo- 10
40

---

Legacy Survey (Train)
HSC (Eval.)
AION-I-B
83.95
84.15
AION-1-L.
87.16
85.66
AION-I-XL
86.99
85,91
Table 8: Zero-shot morpholggy-clasiiccaiinn accuracy (%). A classifier trained on Legacy Survey
images transfers directly to HSC, indicating that AION-1 produces survey-invariant representations.
I Details on Model Scaling
We evaluate how the model size impact its performances. The evaluation computes the categorical cross-entropy on predicted token outputs given input tokens that are randomly selected among every available modality. The overall evaluation loss is the weighted average of each modality loss. Figure 17 reports the overall evaluation loss and the image and spectrum losses for the legacy and SDSS surveys respectively. The decrease we see in the overall evaluation loss for the legacy survey indicates the model performs better when its size increases. However, the evaluation loss on SDSS remains similar regardless the size of the model. When checking for modality specifically (second row Figure 177. appears the image evaluation loss decreases with the size of the model while the spectrum evaluation loss stagnates. This indicates the amelioration observed while scaling the model largely due better performances predicting token images. This trend could be explained by the fact that the complete dataset contains much more samples with images than with spectra (Figure 2a). There might not enough spectrum data to observe improvement while scaling the model.
Additionally, we evaluate the impact of adding the GAIA dataset to the training set. GAIA contains 77M objects with spectrum data, thus should theoretically bring more information about this modality. Figure 17 compares the evaluation losses for different model sizes trained on all surveys except GAIA (Base Dataset i.e. Legacy survey + SDSS + DESI + HSW) and on all surveys including GAIA (Base Dataset + GAIA). For both evaluations on Legacy and SDSS surveys the loss is higher when training with Base Dataset GAIA. Adding GAIA survey to the training dataset seems thus to impacts negatively the performances of the model on the other surveys. It is the case for image and even spectrum modality. The decreased performance on spectrum modality of SDSS survey, while adding spectrum information from GAIA in the training dataset, could be explained by the fact that GAIA spectrum is of much lower resolution than the one of SDSS.
Full Modality Tokens
41

---

Category Imaging (2)
Description
Modality
Legacy Survey imaging
tok image
HSC Wide imaging
tok image hsc
Catalog (1)
Legacy Survey catalog
catalog
Spectra (2)
SDSS spectra
tok_ lik_spectrum_sate sdss
DESI spectra
tok si__pperrum_dess
Gaia (4)
Gaia BP spectrum
tok_xp_bp tok _ok_xp_rp tok lok_parallax
Gaia RP spectrum
Gaia parallax
Sky coordinates
tok_ra ra, tok_dec
Gaia Photometry (3)
Gaia G-band flux
tok flux g g_aaaa
Gaia BP-band flux
tok flux fww._ppluueeeeeee
Gaia RP-band flux
tok flux_rrp _au__p__ga rp__aia
Legacy Survey (9)
g-band flux
tok tok_flux flux_ g
r-band flux
tok f__uu___r
i-band flux
tok flux i
z- z-band flux
tok flux
WISE WI flux
tok flux w1
WISE W2 flux
tok fllx_flwwwww___ w2
WISE W3 flux
tok_flux_wa tok flux_w44444440 w4
WISE W4 flux
E(B-V) extinction
tok_ebv
Legacy Survey Shape (3)
Ellipticity component
tok shape_es el
Ellipticity component 2
tok shape_ea e2 cok_shape_r tok mag g tok mag tok mag_ tok_mag
Effective radius
HSC Photometry (5)
g-band magnitude
r-band magnitude
i-band magnitude
z-band magnitude y-band magnitude g g-band extinction
tok _ok_mag_y tok a_g
HSC Extinction (5)
r-band extinction
tok a
i-band extinction
tok a
z-band extinction
tok_a_ tok ______y
y-band extinction
HSC Shape (3)
Shape component
tok shape
Shape component 22
tok okk_shape22
Shape component 12 Redshift
tok tok_shape12
Other (1)
tok_
Table 9: Complete list of modalities used in our multi-survey analysis. The modalities are grouped by
their source surveys and measurement types. Numbers in parentheses indicate the count of modalities in each category.
42

---

Base +
AION-1
Evaleation Loss Legacy Servey
Evaluation Loss SDSS
3.1
3.0
2.9
3.0
2.8
2.7
2.5
2.5
2.5
Loss
Image Evaluation Loss Legacy Survey
Speetrum Evaluation Loss SDSS
1,75
0..0
1,70
5.8
1.05
1.00
5.2
1.55
5.0
1.50
4.8
50000 100000
L50000
200000
60000
L00000
130000
200000
Checkpoint
Figure 17: Evaluation losses for different model sizes on Legacy and SDSS surveys including image and spectrum modality. Base dataset refers the training all surveys except GAIA, while Base Dataset + GAIA are the models trained on all surveys described Figure 2a
43


Reviews:
[{"review_id": "ZDGMOX0IWO", "reviewer": "Reviewer_SExS", "summary": "This paper presents AION-1, the first large-scale, multimodal foundation model for astronomy. The authors propose a two-stage architecture to handle the diverse data types in this domain, such as images, spectra, and tabular data. First, modality-specific tokenizers convert the heterogeneous inputs into a common discrete token space. Then, a single transformer-based model is trained using a masked modeling objective on sequences of these tokens from over 200 million astronomical objects. The main contribution is a unified framework that can ingest arbitrary combinations of astronomical data to perform a wide range of downstream tasks, including parameter regression, classification, data generation, and object retrieval.", "questions": "The introduction makes a strong claim about the superiority of masked modeling over contrastive learning for this scientific domain. However, the experimental comparisons are primarily against supervised, single-modality baselines. To truly substantiate this claim, a more direct comparison would be beneficial. Could you discuss how AION-1's representations or downstream performance compare to a strong contrastive model, like a CLIP-style architecture, trained on the same multimodal data (e.g., images and spectra)?", "limitations": "yes", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "I am not familiar with the area and don't know how to judge such a paper.\n\nPros\n* It’s especially impressive in situations where you don't have a lot of data, which is a common headache in scientific research.\n* Emergent Abilities: It can do some really cool things that aren't just simple predictions. For example, it can create high-resolution spectra from low-res ones and is great at finding rare objects, which really showcases its advanced capabilities.\n* Clear and Well-Written: the paper easy to follow.\n* Open and Accessible: they are open-sourcing everything – the models, the data, and the code.\n\nCons:\n\n* **Not Entirely New:** The core idea isn't brand new. It builds heavily on recent models like the 4M model. The main innovation here is applying these ideas to a new scientific field and the clever engineering they did, not a completely new machine learning concept.\n* **Insanely Expensive to Train:** The amount of computer power needed to train this model is a major practical drawback. To their credit, the authors admit this and suggest that future work could focus on making it more efficient.", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed review and insightful feedback. \n\n## 1. Methodological Novelty\nThe NeurIPS 2025 Call for Papers explicitly lists “Machine learning for sciences” among its core topical areas, so contributions that unlock new scientific capabilities with ML fall squarely within scope. Although AION‑1 adopts the two‑stage tokenizer‑plus‑transformer design introduced by 4M, it delivers a decisive advance for astronomy. Earlier astronomical FMs, such as AstroCLIP (galaxies; Parker, et al., 2024) and Maven (supernovae; Zhang, et al., 2024), use contrastive losses to align only 2-3 modalities, must train on the small subset of objects where those modalities co‑exist, and target a single physical process. AION‑1, by contrast, scales to 39 modalities, enables training over all available data, and is the first model in astronomy to reach the billion-parameter scale and to process multiple physical phenomena simultaneously (e.g. galaxies, stars, and quasars). This omnimodal approach tackles astronomy’s inherently heterogeneous, weakly overlapping datasets, making AION‑1 the first foundation model capable of scaling across the entire domain, and it already sets new state‑of‑the‑art marks on a variety of tasks.\n\n## 2. Cost of Training\nThe reviewer is correct that pre‑training AION‑1 is non‑trivially expensive. However, its compute footprint is well inside the norms already accepted for scientific foundation models. Our largest release, AION-1-XL, is trained for 3.5 days on 288 H100 GPUs, equivalent to roughly 1000 GPU-days; by comparison, the ESM-2 3B protein model used 512 V100s for ~30 days (15,000 GPU-days), and the 15B version doubled this budget. \n\nMoreover, while the compute cost is high, it is a one-off expense that will be amortized by community use, especially relative to the entire zoo of single-use deep learning models trained for end-to-end specific tasks throughout astronomy. We will release the full pretrained weights of the model, tokenizers, and all model code, enabling users to easily use the model embeddings for downstream tasks, which we demonstrate already beat end‑to‑end supervised baselines on a variety of benchmark tasks. \n\nIn short, the upfront compute is comparable to other successful scientific FMs, and the community can inherit the benefits at a tiny fraction of the cost.\n\n## Comparison with CLIP-style architecture\nTo address the reviewer’s request, we benchmarked AION‑1 against the strongest publicly‑available contrastive model for astronomy, AstroCLIP (Parker, et al., 2024), and an out‑of‑the‑box computer‑vision baseline, DINOv2‑B (Oquab, et al., 2023). All models were evaluated on the same train/test splits and galaxy‑property tasks.\n\nGalaxy Images -> Property (R^2)\n| Model | $z$ | $M_\\text{star}$ | $t_\\text{age}$ | $Z_\\text{met}$ | $\\text{SFR}$  |\n|---|---|---|---|---|--|\n| AION-B + Attn | **0.934**  | **0.886** | **0.445** | **0.490** | **0.637** |\n| AstroCLIP | 0.783  | 0.725 | 0.293 | 0.362 | 0.424 |\n| DINOv2-B | 0.571  | 0.550 | 0.166 | 0.284 | 0.251|\n\nGalaxy Spectrum -> Property (R^2)\n| Model   | $z$   | $M_\\text{star}$ | $t_\\text{age}$ | $Z_\\text{met}$ | $\\text{SFR}$  |\n|---|---|---|---|---|---|\n| AION-B + Attn | **0.995** | **0.956** | **0.532** | **0.610**  | **0.720** |\n| AstroCLIP | 0.985  | 0.898 | 0.518 | 0.601  | 0.704 |\n\nGalaxy Image Rare Object Retrieval (nDCG@10)\n| Model | Spirals | Mergers | Strong Lenses |\n|---|---|---|---|\n| AION-XL | **0.621** | **0.384**  | **0.015**  |\n| AstroCLIP | 0.602   | 0.248   | 0.006  |\n| DINOv2-B  | 0.477   | 0.060   | 0.003  |\n\nUltimately, we find that AION-1’s embeddings outperform those produced by both AstroCLIP and DINOv2 in terms of R^2 for the property prediction tasks and nDCG@10 for the retrieval tasks."}, {"review_id": "cSbvuM8ulA", "reviewer": "Reviewer_1y7Y", "summary": "This paper introduces AION-1, which is a family of large multimodal foundation models designed for astronomy’s diverse data landscape. Unlike prior approaches that often leverage natural language to connect modalities, AION-1 directly integrates up to 39 heterogeneous data modalities within a single unified model. Interestingly, after unsupervised pretraining, AION-1 demonstrates several emergent capabilities without task-specific finetuning. For example, its learned latent space encodes rich astrophysical information that can be exploited by simple linear probes: on predicting galaxy properties (redshift, stellar mass, etc.) and stellar parameters, frozen AION-1 embeddings achieve performance on par with or exceeding dedicated supervised models, especially when only limited labeled examples are available. Notably, in low-data regimes (tens to a few hundred samples), AION-1’s representations often surpass specialized models that were trained from scratch on each modality.", "questions": "AION-1 is purposely made for arbitrary combinations of data modality, but how does it handle objects with substantially longitudinal missing data? For example, if an object has only photometry and no images or spectra, how does the model perform?\nIn a similar vein, beyond the spectral super-resolution example, can AION-1 perform more exotic cross-modal generation? \nWhat was the motivation against fine-tuning the entire AION-1 model on a downstream task?  Is it due to a loss in generality?", "limitations": "one of the biggest weaknesses/limitations of the paper is the lack of accounting of temporal dynamics. This is indeed crucial for certain domains (e.g. monitoring variable stars or exoplanet transits). A suggestion is to develop a time-series tokenizer that could, for example, convert a light curve into a set of tokens (perhaps via discrete wavelet coefficients or sequence VQ encoding). The authors do acknowledge this fervently, and I do hope to see future work on this.\n\nA weakness not touched upon is interpretability. Foundation models can be black boxes, however, in science it’s important to extract understanding. The paper does a bit of this by showing latent space structure and that certain attention heads attend to spectral lines (in SpectraFM style), but I believe more could be done. Maybe through a more principled approach towards taking in the data, or by providing tools or visualizations for scientists to interpret what AION-1 has learned (e.g. which token dimensions correspond to physical properties). This would increase trust and adoption in the community.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper’s methodology is sound and represents a rigorous application of state-of-the-art self-supervised learning to a novel domain. The authors carefully address the challenges of heterogeneous data: each modality is standardized via appropriate tokenization (e.g. ResNet+VQ for images, ConvNeXt for spectra) with built-in awareness of noise and units. \nTheir models are comprehensively tested and across the board show very strong performance against other standard models, particularly shining when data is scarce (as is often the case in astronomy).\nAION-1 is also a significant step up at the intersection of AI and astronomy. It is the first model to unify such an expansive range of data modalities in astronomy and scale to the billion-parameter regime. This is a substantial step from earlier efforts like AstroCLIP or PAPERCLIP.\n\nWeaknesses:\nWhile the experiments are extensive, one area that could strengthen the quality further is full fine-tuning or end-to-end evaluation on downstream tasks. The study primarily uses frozen representations with simple heads (or small added conv layers) to probe performance. \nAnother minor concern is that the model’s treatment of temporal data remains absent but the authors acknowledge this as a current limitation.\nThe novelty in algorithmic terms is somewhat limited. The architecture is an encoder-decoder Transformer with a BERT-like objective. Maybe more specific physics informed elements could have been incorporated, but overall the methods are motivated well.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful, detailed feedback and for recommending acceptance. We are encouraged that you found the methodology rigorous, the omnimodal tokenization sound, and the low‑data performance particularly impactful.\n\n## Handling Heavily Missing Modalities\nAION‑1’s masked‑token pre‑training objective is modality‑agnostic: at both training and inference, any modality can be absent and the sequence simply contains no tokens of that type. Empirically, this robustness holds: when we feed photometry only for the galaxy‑property benchmark, AION‑1 still outperforms an XGBoost model trained on the same photometric features (Table 2). We ran the same ablation for stellar‑parameter prediction, again supplying only photometry, and similarly found that AION-1 generally outperforms an XGBoost baseline. \n\n| Model | $T_\\text{eff}$   | $\\log g$ | $[Fe/H]$ |\n|---|---|---|---|\n| AION-L + Attn  | **0.946**| **0.955** | 0.582 |\n| XGBoost | 0.943  | 0.950 | **0.589**  |\n\n## More exotic cross-modal generation\nThank you for raising this point, and we agree that it would be interesting to see if we can perform more exotic cross-modal generations. As a meaningful test, we considered a combination of modalities that are never seen during pre-training: HSC imaging and DESI spectra. Our training strategy included pairs of modality examples HSC imaging <-> SDSS spectra and SDSS spectra <-> DESI spectra, but never directly seeing HSC imaging <-> DESI spectra. This allows us to test whether the model is building a transitive understanding of data modalities which would ultimately allow us not to have a combinatorial explosion of modality combinations needed for training. \n\nAt test time, asking the model to generate DESI spectra from HSC imaging (which, again, is an out of distribution task) produces physically plausible DESI‑resolution spectra whose key absorption/emission features align with the ground truth for both quiescent red galaxy and a star forming blue galaxy. We will include examples of these generations in the revision of the paper. \n\n## Temporal Data\nWe thank the reviewer for emphasizing the need to incorporate time‑domain information and fully agree that incorporating this data in the future will be critical, allowing us to extend our model to a range of use cases from stellar variability to supernovae characterisation. As a matter of fact, our AION-2 model, currently under development, includes a dedicated time-series tokenizer.\n\n## Finetuning\nWe explore using AION‑1 primarily as a frozen embedding resource as we try to match the low-data, low‑compute workflow we expect most astronomers to adopt. Indeed, the main point we want to make to the community is that adapting such Foundation Model to their use case is trivial. And  we  note that the cross-attention heads on the frozen embeddings already surpass supervised baselines on most of the tasks that we evaluate on. Nonetheless, the transformer can be fine‑tuned end‑to‑end, and we are including utilities in our open source library to enable several fine-tuning strategies (from only adapting embedding layers to full fine-tuning) which we expect will be useful to adapt the model to  tasks and modalities that depart substantially from those we evaluated. \n\n## Interpretability\nWe agree this is an important and exciting future direction. One avenue we are already exploring is training sparse autoencoders on the latent space to disentangle which dimensions capture specific physical features, and we look forward to reporting on these results in future work. More generally, we recognise that neural networks in science must be applied with methodological care, so our workflow always includes a dedicated calibration stage: a small, carefully curated downstream‑task training set is used to train a calibration head on the embeddings for the specific physical question, ensuring that the learned representations remain trustworthy."}, {"review_id": "bm19Pmcwwp", "reviewer": "Reviewer_KFN2", "summary": "The paper describes the implementation of a multimodal model for astronomical sources, trained on curated astronomy data. It introduces a specialized tokenization scheme to train multimodal masked modeling, and shows multiple astronomy analysis tasks to assess the quality of the trained model.", "questions": "1. Would the authors consider adding limitations (not just selection biases) in the main text, perhaps with an hallucinated example?\n2. I understand ablation studies justifying choices can be very costly for these type of models, but would the authors add text justifying the choices of tokenizer and models, and perhaps give a short list of the things which did not improve or did not work?\n3.  Decrease of performance when adding Gaia spectra to the other modalities: authors explanation blame the lower resolution, even though it add information. This would perhaps require a bit more explanation or proof that is not an actual limitation of the chosen architecture.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* Performant model evaluated on a large set of astronomy analysis tasks\n* Large engineering effort, and well detailed (in appendix)\n* Tokenization framework is interesting\n\n**Weaknesses**\n* Could show the limits of the model: where it hallucinates, where it underperforms\n* Most results are in the appendix. Actually the appendix is quite a long read, which makes me think if this is the right venue.\n* No ablation study at either modelling or tokenization is shown\n* Evaluation metrics could be better chosen. $R^2$ is not well suited (perhaps adjusted-$R^2$ and showing a dispersion metric would help)\n* Some overselling that does not help the text or is not convincing\n\n*Various small issues*:\n- line 57: \"AION-1 is the first large-scale model designed for arbitrary combinations of highly heterogeneous scientific observations\"  no it is not. Weather, remote sensing, biology models have already been developed and published with multi-modal data\n- line93:  \"two key challenges: the variety of data types (2D images, 1D spectra, scalar values) and the diversity of sources within each type (different telescopes, resolutions, and instrument formats).\" I do not see how variety and source diversity make it special. Natural images also come from different camera, formats, resolutions...I think the authors mean the quantitative tasks they are doing with the data are indeed much different and could require more careful on the source, but data diversity or variety is not really more heterogeneous for any other domain\n- \"Built on a ResNet backbone adapted from [71]\" : reference 71 points to MagViT which does not use ResNet but VQGAN\n line237: \"While we see that AION-1 performs competively against supervised baselines in the full-data regime, its performance is much stronger in the low-data regimes, where it matches or surpasses traditional models that require an order of magnitude more training data\" : the authors meant \"low-label regimes\" and \"order of magnitude more labeled data\"  \n- Figure 2a: no links between Legacy survey and HSC or between Legacy Survey and Gaia: was this an omission?", "was_revised": true, "final_justification": "I increased my score. I was pretty close to a higher score initially and the authors addressed enough (limited metrics, missing limitations) to raise it.", "rebuttal": "We would like to thank the reviewer for the detailed review and insightful feedback.\n\n## Model Limitations/Hallucinations\n Thank you for highlighting the need to characterize where the model under‑performs. We will add some examples discussing hallucination to the paper. For example, when we ask AION‑1 to super‑resolve low‑resolution LegacySurvey images into high‑resolution HSC imaging, the model occasionally “hallucinates” faint background galaxies that are not present in the ground truth. This behavior is an illustration that the conditional distributions between complex modalities are not perfectly captured, which is always to be expected (no machine learning model is perfect). We are aware of these imperfections in a generative framework, as well as the need for using AION-1 in a framework enforcing explicit conditioning on observed data with an explicit likelihood term which prevents hallucinations. We will clarify this in the main body of the text, and emphasize that in the absence of this framework we envision using AION-1 primarily an embedding model, which will be calibrated for downstream tasks with linear/attention heads that are trained on the specific task; this strategy to use the model is not subject to hallucinations.\n\n## Quality of Evaluation Metrics \nUsing R^2 is the primary regression metric in recent multimodal‐astronomy studies such as AstroCLIP (Parker, et al., 2024) and MAVeN (Zhang, et al., 2024). However, we agree with the reviewer that a dispersion‑based measure is also valuable. Accordingly, we have re‑evaluated galaxy‑ and stellar‑property predictions using the standard deviation of the residuals (see tables below). We will also add residual‑dispersion plots in the appendix for visual inspection.\n\nGalaxy Measurements -> Property (Std. Dev. of Residuals):\n| Model | $z$ | $M_\\text{star}$ | $t_\\text{age}$ | $Z_\\text{met}$ | $\\text{SFR}$  |\n|---|---|---|---|---|--|\n| AION-1-B (Ph) | 0.056 | 0.354 | 1.326 | 0.694 | 1.778 |\n| AION-1-B (Ph+Im) | 0.029 | 0.228 | 1.226 | 0.648 | 1.360 |\n| AION-1-B (Ph+Im+sp) | 0.008 | 0.141 | 1.128 | 0.566 | 1.195 |\n\nStellar Spectra -> Property (Std. Dev. of Residuals):\n| Model | $T_{eff}$ | $\\text{logg}$ | $v_\\text{mic}$ | $\\text{Fe/H}$  |\n|---|---|---|---|--|\n| AION-1-B | 72.48 | 0.246 | 0.129 | 0.080 | \n| Baseline | 73.28 | 0.239 | 0.129 | 0.070 |\n\n\n## Ablation Studies\nFor the 4M architecture and training choices, we rely on the extensive ablation studies reported in 4M and therefore adopt the same encoder-decoder transformer configuration without re‑running those costly sweeps. Our own ablations focus on the new tokenisers. In the appendix we present a codebook‑size sweep for the image tokenizer: sizes ranging between 2^4 and 2^14 show a steady increase in performance up to 2^12, after which gains in performance begin to saturate. We therefore selected 2^\t12 as the size for the image tokenizer, and fixed the other tokenizer sizes according to this based on a rough estimation of the relative complexity of that data. \n\n## Other Issues\n**Length of Appendix:** We followed the NeurIPS 9‑page limit by keeping the core findings in the main manuscript, while placing extensions in the appendix for completeness. For example, galaxy‑property prediction with cross‑attention pooling appears in the main text (Table 2); the appendix then adds a comparison to the mean‑pooling variant. A similar approach is used when presenting the rest of the results. \n\n**Overselling **: Thank you for flagging wording that felt overstated. In the camera‑ready version we will remove superlatives that are not strictly supported by evidence and address the individual points raised below. \n\n**Line 57: “AION-1 is the first [...]”:** We will delete the sentence at line 57 and instead clarify that AION‑1 is, to our knowledge, the first omnimodal foundation model for astronomy - while multimodal models already exist in weather, remote sensing, and biology.\n\n**Line 93: “two key challenges [...]”:** We agree that natural‑image corpora also features multiple cameras, formats and resolutions, so the phrasing was imprecise. What is distinctive in astronomy is the combination of cross‑dimensional heterogeneity (2‑D images, 1‑D spectra, irregular time‑series, scalar catalogue values) with instrument‑specific systematics that vary by orders of magnitude across wavelength, aperture and detector type. These differences carry physical meaning (flux calibration, spectral resolution, cadence) and directly affect the quantitative tasks we target (e.g. redshift, stellar‑mass inference). We will revise the sentence to emphasise this physical and dimensional heterogeneity rather than implying that source diversity alone makes astronomy unique.\n\n**“Built on a ResNet backbone [...]”:** We thank the reviewer for pointing this out. VQGAN does have residual blocks but it’s not technically built on a ResNet backbone; we will modify the wording in the final paper to reference VQGAN instead.\n\n**Line 237: “While we see that AION-1 performs [...]”:** Thank you for pointing this out. We will replace “low‑data regimes” and “training data” with “low‑label regimes” and “labelled data” to accurately reflect that the advantage is in settings with scarce annotations, not limited raw observations.\n\n**Figure 2a: no links [...]:** No link is drawn between our Legacy Survey catalog and our Gaia catalog because the LS catalog specifically targets galaxies, while the Gaia catalog targets stars; consequently, no individual objects are explicitly targeted by both catalogs."}]

Author rebuttal:

Meta-review: The paper presents a family of multimodal foundation models for astronomy, named AINO-1. 
The final ratings are two "accept" and one "borderline reject".
Reviewer KFN2 raised the score to “accept” after the author's rebuttal addressed the concerns of limited metrics and missing limitations.
Reviewer SExS maintained the score of "borderline reject" considering the limited novelty, while also noting that the reviewer is not familiar with the area.
The AC acknowledges that this work introduces a unified framework and baselines into the area of astronomy. The results looks good.
Thus, AC would like to suggest to accept.
However, the authors need to revise the paper according to all reviews.
Decision: Accept (poster)