**Collection Process** We first download all coadded images (specifically calexp files from the deepCoadd_result folder) for the Deep/UltraDeep fields from the HSC PDR3 data archive<fnref n="11" />. Each of these images is of size 4200x4200, from which we extract cutouts centered on the coordinates of the objects in the sample selected above. We associate each of these cutouts with their corresponding entry in the object catalog.

**Preprocessing/cleaning** No further processing is applied to the images beyond the selection cuts mentioned above. Similarly to previous imaging datasets, images can contain bad pixel or artifacts surrounding the central object, those can be identified through the mask field that identifies good pixels.

**Uses** It is the first time that this specific dataset is compiled, it will have similar use cases as the other imaging datasets, i.e. machine learning methods for detecting rare objects, or classifying or characterizing galaxy properties.

**Acknowledgements and Terms of Use** Data from the HSC SSP can be used without prior permission within the following scopes:

* Extent of free use stipulated by Japanese copyright law (private use, educational use, news reporting, etc.)

* Usage in academic research, education, and learning activities

* Usage by news organizations

* Usage in printed media, or in websites and social networks

The full description of restricted use-cases is provided here: https://www.nao.ac.jp/en/terms/copyright.html. In all cases, please explicitly include the credit "NAOJ / HSC Collaboration".

### A.3.4 BTSbot: btsbot

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

**Description** The Zwicky Transient Facility (ZTF) [15, 16] Bright Transient Survey (BTS) [56, 114] is the largest spectroscopic supernova survey ever constructed, surveying the entire nighttime Northern sky every 2-3 nights in two filters aiming to acquire spectra for a complete sample of bright, local transients. While identification of new transient sources has traditionally been led by human scanners assessing images, BTS have developed the BTSbot model and accompanying dataset [120] with the goal of automating transient identification. Each example in the BTSbot dataset corresponds to an 'alert', which will be generated automatically after observations and passed to the community to identify interesting objects and plan potential followup observations. The BTSbot model was trained specifically to identify sources expected to fit BTS's selection criteria.

**Composition** The BTSbot dataset contains 409,107 transient alerts. Note that while each alert is treated separately, the same transient objects will feature multiple times in the data set. For example, real transients will be harder to identify in earlier alerts when they are fainter and data is noisier but will become more apparent in later alerts as they get brighter. Each transient alert comprises a set of 3 single-band images referred to as the 'science', 'reference', and 'difference' images. The science image shows the latest image of a potential transient event and its surrounding environment, for example the galaxy it is located in. The reference image is an archival image of the same area before the transient was present, while the difference image is the residual between the two which isolates the transient. We collect these views as different channels in a 3×63×63 image in the same way that other multi-band imaging datasets organise images seen through different filters. The dataset is also multi-modal as each image triplet is accompanied by metadata containing extracted features of the transient source which are informative for classification.

<footnote n="11">¹¹https://hsc-release.mtk.nao.ac.jp/archive/filetree/pdr3_dud/ (requires authentication)</footnote>
**Collection Process** A description of how the BTSbot dataset was compiled is given in the original paper [120]. We accessed the dataset at the authors' zenodo page<fnref n="12" />.

**Preprocessing/Cleaning** The BTSbot dataset is included within the MULTIMODAL UNIVERSE without any modifications (aside from reformatting).

**Uses** The BTSbot dataset was used to develop the BTSbot model described in the original paper [120].

**Acknowledgements and Terms of Use** The BTSbot dataset is distributed under the CC BY 4.0 license. Attribution is given to the authors of the BTSbot paper [120].

### A.3.5 JWST: jwst

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

**Description** The JWST dataset contains NIRCam images from several of the first James Webb Space Telescope (JWST) deep field observations: CEERS [52], PRIMER [48], JADES [50] and NGDEEP [14].

**Composition** Fixed size postage stamps (96 × 96 pixels) centered on the positions of sources in the photometric catalogs. Each deep survey is an independent instance of the JWST imaging survey and has a different wavelength coverage with 6 or 7 filters in the ~0.9µm to ~4.4µm range. The combined dataset contains of the order of 300K images. The list of available filters is: F090W (0.9µm), F115W (1.15µm), F150W (1.5µm), F200W (2µm), F277W (2.7µm), F356W (3.56µm), F444W (4.4µm).

**Collection Process** For consistency, we use the public reductions and associated photometric catalogs from the Dawn JWST Archive<fnref n="13" />. Basic details of the NIRCam data reduction are presented in [141]. Images have been reduced with the grizli pipeline<fnref n="14" />.

**Preprocessing/cleaning** Where bands are missing, we store an array of zeros instead to simplify data loading.

**Uses** Images from these deep JWST surveys have been used in a large number of scientific publications. However, the specific dataset compiled in this work has not been used in any prior publication. Some examples of machine learning works using JWST images are: [77, 142]

**Acknowledgements and Terms of Use** All the data included in this dataset is of the public domain. The data products presented herein were retrieved from the Dawn JWST Archive (DJA). DJA is an initiative of the Cosmic Dawn Center (DAWN), which is funded by the Danish National Research Foundation under grant DNRF140.

## A.4 Time Series

### A.4.1 PLAsTiCC: plasticc

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

**Description** The PLAsTiCC dataset is a collection of simulated light curves from the Photometric LSST Astronomical Time-Series Classification Challenge. This diverse dataset contains 14 types of astronomical time-varying objects, simulated using the expected instrument characteristics and survey strategy of the upcoming Legacy Survey of Space and Time [LSST 79] conducted at the Vera C. Rubin Observatory (Figure 4). The dataset includes two overall categories of time-series objects: *transients*, which are short-lived events such as supernovae, and *variable* sources, which are objects with fluctuating brightness such as pulsating stars. Specifically, the dataset includes the following transients: type Ia supernovae (SNIa), SNIax, SNIa-91bg, SNIbc, SNII, superluminous supernovae (SLSN), tidal disruption

<footnote n="12">¹²https://zenodo.org/records/10839691</footnote>

<footnote n="13">¹³https://dawn-cph.github.io/dja/index.html</footnote>

<footnote n="14">¹⁴https://github.com/gbrammer/grizli, https://zenodo.org/records/8370018</footnote>
<figure><img src="image_30.png" /><figcaption>Figure 4: Examples of the 14 object types included in the PLAsTiCC dataset. Observations taken in different wavelength ranges, or *filters*, are shown in different colors. We overplot lines from Gaussian process interpolations of the data for visibility.</figcaption></figure>

events (TDE), and single lens microlensing events (μLens-Single); and the following variable objects: active galactic nuclei (AGN), Mira variables, eclipsing binary systems (EB), and RR Lyrae (RRL); details in Table 7.

<table><caption>Table 7: Breakdown of the PLAsTiCC train and test datasets by object type. The small training set size coupled with the class imbalances make this a challenging classification dataset.</caption><thead><tr><th>Object Type</th><th>Train Samples</th><th>Test Samples</th></tr></thead><tbody><tr><td>SNIa</td><td>2,313</td><td>1,659,831</td></tr><tr><td>SNIa-91bg</td><td>208</td><td>40,193</td></tr><tr><td>SNIax</td><td>183</td><td>63,664</td></tr><tr><td>SNII</td><td>1193</td><td>1,000,150</td></tr><tr><td>SNIbc</td><td>484</td><td>175,094</td></tr><tr><td>SLSN-I</td><td>175</td><td>35,782</td></tr><tr><td>TDE</td><td>495</td><td>13,555</td></tr><tr><td>KN</td><td>102</td><td>133</td></tr><tr><td>AGN</td><td>370</td><td>101,424</td></tr><tr><td>RRL</td><td>239</td><td>197,155</td></tr><tr><td>M-dwarf</td><td>981</td><td>93,494</td></tr><tr><td>EB</td><td>924</td><td>96,572</td></tr><tr><td>Mira</td><td>30</td><td>1,453</td></tr><tr><td>μ-Lens-Single</td><td>151</td><td>1,303</td></tr><tr><td>Total</td><td>7,848</td><td>3,492,890</td></tr></tbody></table>

**Composition** The dataset contains a collection of 3,492,890 time-series, providing in each case several fields: time of observation, flux, flux error, and filter in which the flux is measured. Observations in 6 filters are included (u,g,r,i,z,Y). In addition, each time series is associated with a class representing the type of object, and a redshift for that object.

**Collection Process** The PLAsTiCC dataset is available on Zenodo at https://zenodo.org/records/2539456. We simply download and reformat the data to match the MULTI-MODAL UNIVERSE standard.

**Preprocessing/cleaning** No further pre-processing is applied to the data.

**Uses** The PLAsTiCC dataset has been widely used to develop and test supernova classification model [e.g. 116, 20].

**Acknowledgements and Terms of Use** The dataset is released under the CC BY 4.0 License, it can be cited using the following DOI: 10.5281/zenodo.2539456.
### A.4.2 TESS: tess

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

**Description** The NASA Transiting Exoplanet Survey Satellite (TESS)<fnref n="15" /> [121] is an all-sky photometric survey observing millions of sources to discover exoplanets and study variable stars. Stars are observed for a period of about 27 days at a time, before the satellite shifts to its next observing field (sector). Stars can be observed multiple times in different sectors over the course of the mission. The observations have a cadence (sampling time) between 20 sec and 30 min, depending on the mission cycle and data product.

**Composition** The TESS data set in the MULTIMODAL UNIVERSE currently contains the light curves (brightness over time) for around 1,120,000 stars observed in sectors 56, 58, 60, 63, 64, 67 and 69, but the data can be downloaded for all available sectors by running the respective download scripts. The light curves consist of six fields: `object_id`, `time` (BTJD; Barycenter corrected TESS Julian Date), `flux` (electrons per second) and `flux_error` (electrons per second). In the future, we will also include the much larger set of light curves delivered by the MIT Quick-look Pipeline (QLP)<fnref n="16" /> [73, 74, 87, 88], which contains up to a few million light curves per sector.

**Collection Process** The TESS Science Processing Operations Center (SPOC) pipeline generates light curves [80] for up to 160,000 targets per sector [33] from the shorter cadence Target Pixel Files (TPFs) and the longer cadence Full-Frame Images (FFIs). The data sets were accessed through the Mikulski Archive for Space Telescopes (MAST) servers.

**Preprocessing/cleaning** We use the Pre-search Data Conditioning (PDC_SAP) flux measurements, in which long term systematic trends have been removed using Co-trending Basis Vectors. We have further cleaned the light curves by applying a hard quality flag bitmask that removes data points with possible anomalies as described in the TESS Science Data Products Description Document and the TESS Data Release Notes.

**Uses** Light curves delivered by TESS, or light curves with similar characteristics, such as those from the NASA Kepler mission [21, 85], have been used in a number of machine learning publications. Light curve classification (e.g., transiting exoplanet, pulsating star or eclipsing binary star) examples are [11, 72, 12, 137, 110, 37], while a stellar parameter estimation example is [111].

**Acknowledgements and Terms of Use** The data collected by the TESS mission are publicly available from the Mikulski Archive for Space Telescopes (MAST). The light curves in this data set have been released under the CC BY 4.0 License. More detailed information on the data products can be found on MAST: https://archive.stsci.edu/hlsp/tess-ssoc. Funding for the TESS mission is provided by NASA's Science Mission directorate.

### A.4.3 CfA Datasets: cfa

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

**Description** The Harvard-Smithsonian Center for Astrophysics (CfA) Supernova Group has published light curves of various classes of supernovae observed between 2000-2011.

**Composition** The four datasets in the MULTIMODAL UNIVERSE are as follows: CfA3 presents light curves of 185 Type Ia supernovae in the UBVRIr'i' bands in the standard photometric system, CfA4 provides light curves of and additional 94 Type Ia supernovae in the u'UBVr'i' bands in the standard photometric system, CfA-SECCSN presents light curves of 64 stripped-envelope core-collapse supernovae in the u'UBVRIr'i'JHKs bands. Lastly CfA-SNII has light curves of 60 Type II supernovae in the UBVRIu'r'i'JHKs bands.

We present an equivalent breakdown in Table 8.

Each dataset has a row for each supernova with two float fields for sky coordinates (ra, dec), two string fields (`object_id` and `obj_type`) describing each supernova's name and classification,

<footnote n="15">¹⁵https://tess.mit.edu/</footnote>

<footnote n="16">¹⁶https://archive.stsci.edu/hlsp/qlp</footnote>
<table><caption>Table 8: Breakdown of the CfA Samples.</caption><thead><tr><th>Source Survey</th><th>Shape</th><th>Number of samples</th><th>Main science</th></tr></thead><tbody><tr><td>CfA 4 SNe Ia</td><td>Variable</td><td>341</td><td>Type I Supernovae</td></tr><tr><td>CfA 3 SNe Ia</td><td>Variable</td><td>185</td><td>Type I Supernovae</td></tr><tr><td>CfA Stripped-Core SNe</td><td>Variable</td><td>249</td><td>Stripped-envelope core-collapse Supernovae</td></tr><tr><td>CfA SNe II</td><td>Variable</td><td>400</td><td>Type II Supernovae</td></tr></tbody></table>

and a lightcurve field with rows for each observation of that particular supernova. Each row has four subfields. The time subfield contains float values for the modified Julian date of observation (in units of days). The band subfield contains strings describing the band of the observation. The flux and flux_err fields are replaced by mag and mag_err fields, which present the original photometric measurements in the standard photometric system.

**Collection Process** The compilation of each dataset is described in the corresponding papers [68, 69, 18, 70]. The datasets were accessed from the CfA Supernova Group's website https://lweb.cfa.harvard.edu/supernova/.

**Preprocessing/cleaning** We have included supplemental data for each object, including RA, Dec, and spectroscopic classifications from the transient name server (TNS)<fnref n="17" />, the International Astronomical Union's official repository for reporting astronomical transients.

**Uses** These datasets were introduced in their respective papers, and have been used in many studies since. In particular, the Type Ia supernovae from CfA3 and CfA4 are included in the Pantheon [127] and Pantheon+ samples [128], the Joint Light-curve Analysis sample [17], the DESSN-3YR sample [25], the Union2 [9] and Union3 samples [123], and many more.

**Acknowledgements and Terms of Use** This research has made use of the CfA Supernova Archive, which is funded in part by the National Science Foundation through grant AST 0907903. If you make use of CfA data, please include the same acknowledgement as was made here and cite the appropriate references for the data (see the 'Collection Process' paragraph above) in your publication. The CfA3 data is provided under the CC BY 4.0 license and the CfA-SNII data under the CC BY 3.0 license. Refer to the licensing policies of IOPscience<fnref n="18" /> and AAS<fnref n="19" /> for the CfA4 and CfA-SECCSN datasets.

#### A.4.4 YSE: yse

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

**Description** A collection of 1975 supernova light curves from the Young Supernova Experiment Data Release 1 (YSE DR1) [7]. An example use of this dataset would be for photometric classification.

**Composition** We present the data as light curve objects containing time of observation, flux, flux error and band pass filter. Measurements are in griz band pass filters. The supernova classification is included as metadata. Additional metadata includes information on coordinates, redshifts and host galaxy masses.

**Collection Process** The data is presented in full as shown in the original paper [7] and is accessed through Zenodo<fnref n="20" />.

**Acknowledgements and Terms of Use** Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI [7].

<footnote n="17">¹⁷https://www.wis-tns.org</footnote>

<footnote n="18">¹⁸https://publishingsupport.iopscience.iop.org/creative-commons-licences/</footnote>

<footnote n="19">¹⁹https://journals.aas.org/article-charges-and-copyright/</footnote>

<footnote n="20">²⁰https://zenodo.org/records/7317476</footnote>
### A.4.5 PS1 SNe Ia: ps1_sne_ia

<figure><img src="image_43.png" /></figure>

**Description** A collection of 369 spectroscopically confirmed type Ia supernova light curves from Pan-STARRS1 (PS1 SNe Ia) [127]. This dataset can be used for photometric redshift prediction or light curve inpainting.

**Composition** We present the data as light curve objects containing time of observation, flux, flux error and band pass filter. Measurements are in *griz* band pass filters. The supernova classification is included as metadata. Additional metadata includes information on coordinates, redshifts and host galaxy masses.

**Collection Process** The data is collected from the Pantheon+ compilation [128] that also performs a series of selection cuts. The data is accessed through the compilation's GitHub: https://github.com/PantheonPlusSHOES/DataRelease/tree/main/Pantheon%2B_Data/1_DATA/photometry/Pantheon_PS1MD.

**Acknowledgements and Terms of Use** Original content from the work [127] may be used under the terms of the Creative Commons Attribution 4.0 license. If you use this data, please cite the appropriate papers and acknowledge the dataset as stated on the Pan-STARRS webpage<fnref n="21" />. The Pan-STARRS1 Surveys (PS1) and the PS1 public science archive have been made possible through contributions by the Institute for Astronomy, the University of Hawaii, the Pan-STARRS Project Office, the Max-Planck Society and its participating institutes, the Max Planck Institute for Astronomy, Heidelberg and the Max Planck Institute for Extraterrestrial Physics, Garching, The Johns Hopkins University, Durham University, the University of Edinburgh, the Queen's University Belfast, the Harvard-Smithsonian Center for Astrophysics, the Las Cumbres Observatory Global Telescope Network Incorporated, the National Central University of Taiwan, the Space Telescope Science Institute, the National Aeronautics and Space Administration under Grant No. NNX08AR22G issued through the Planetary Science Division of the NASA Science Mission Directorate, the National Science Foundation Grant No. AST-1238877, the University of Maryland, Eotvos Lorand University (ELTE), the Los Alamos National Laboratory, and the Gordon and Betty Moore Foundation.

### A.4.6 DES Y3 SNe Ia: des_y3_sne_ia

<figure><img src="image_44.png" /></figure>

**Description** A collection of 248 spectroscopically confirmed type Ia supernova light curves from the Dark Energy Survey Year 3 (DES Y3 SNe Ia) dataset [24]. This dataset can be used for photometric redshift prediction or light curve inpainting.

**Composition** We present the data as light curve objects containing time of observation, flux, flux error and band pass filter. Measurements are in *griz* band pass filters. The supernova classification is included as metadata. Additional metadata includes information on coordinates, redshifts and host galaxy masses.

**Collection Process** The data is collected from the Pantheon+ compilation [128] that also performs a series of selection cuts. The data is accessed through the compilation's GitHub: https://github.com/PantheonPlusSHOES/DataRelease/tree/main/Pantheon%2B_Data/photometry/DES3YR_DES_COMBINED_TEXT.

**Acknowledgements and Terms of Use** Original content from the work [24] may be used under the terms of the Creative Commons Attribution 4.0 license. If you use this data, please cite [34, 24] and add the following acknowledgement to your work: This project used public archival data from the Dark Energy Survey (DES). Funding for the DES Projects has been provided by the U.S. Department of Energy, the U.S. National Science Foundation, the Ministry of Science and Education of Spain, the Science and Technology Facilities Council of the United Kingdom, the Higher Education Funding Council for England, the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign, the Kavli Institute of Cosmological Physics at the University of Chicago, the

<footnote n="21">²¹https://outerspace.stsci.edu/display/PANSTARRS/</footnote>
Center for Cosmology and Astro-Particle Physics at the Ohio State University, the Mitchell Institute for Fundamental Physics and Astronomy at Texas A&M University, Financiadora de Estudos e Projetos, Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro, Conselho Nacional de Desenvolvimento Científico e Tecnológico and the Ministério da Ciência, Tecnologia e Inovação, the Deutsche Forschungsgemeinschaft, and the Collaborating Institutions in the Dark Energy Survey. The Collaborating Institutions are Argonne National Laboratory, the University of California at Santa Cruz, the University of Cambridge, Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas-Madrid, the University of Chicago, University College London, the DES-Brazil Consortium, the University of Edinburgh, the Eidgenössische Technische Hochschule (ETH) Zürich, Fermi National Accelerator Laboratory, the University of Illinois at Urbana-Champaign, the Institut de Ciències de l'Espai (IEEC/CSIC), the Institut de Física d'Altes Energies, Lawrence Berkeley National Laboratory, the Ludwig-Maximilians Universität München and the associated Excellence Cluster Universe, the University of Michigan, the National Optical Astronomy Observatory, the University of Nottingham, The Ohio State University, the OzDES Membership Consortium, the University of Pennsylvania, the University of Portsmouth, SLAC National Accelerator Laboratory, Stanford University, the University of Sussex, and Texas A&M University. Based in part on observations at Cerro Tololo Inter-American Observatory, National Optical Astronomy Observatory, which is operated by the Association of Universities for Research in Astronomy (AURA) under a cooperative agreement with the National Science Foundation.

### A.4.7 SNLS: snls

<figure><img src="image_45.png" /></figure>

**Description** A collection of 239 spectroscopically confirmed type Ia supernova light curves from the Supernova Legacy Survey (SNLS) [63]. This dataset can be used for photometric redshift prediction or light curve inpainting.

**Composition** We present the data as light curve objects containing time of observation, flux, flux error and band pass filter. Measurements are in *griz* band pass filters. The supernova classification is included as metadata. Additional metadata includes information on coordinates, redshifts and host galaxy masses.

**Collection Process** The data is collected from the Pantheon+ compilation [128] that also performs a series of selection cuts. The data is accessed through the compilation's GitHub: https://github.com/PantheonPlusSHOES/DataRelease/tree/main/Pantheon%2B_Data/1_DATA/photometry/JLA2014_SNLS_DS17.

**Acknowledgements and Terms of Use** Original content from the work [63] may be used under the terms of the GNU General Public License v3.0. If you use this data, please attribute the original data source [63] appropriately.

### A.4.8 Foundation: foundation

<figure><img src="image_46.png" /></figure>

**Description** A collection of 180 spectroscopically confirmed type Ia supernova light curves from Foundation Data Release 3 [Foundation DR3 53, 81]. This dataset can be used for photometric redshift prediction or light curve inpainting.

**Composition** We present the data as light curve objects containing time of observation, flux, flux error and band pass filter. Measurements are in *griz* band pass filters. The supernova classification is included as metadata. Additional metadata includes information on coordinates, redshifts and host galaxy masses.

**Collection Process** The data is collected from the Pantheon+ compilation [128] that also performs a series of selection cuts. The data is accessed through the compilation's GitHub: https://github.com/PantheonPlusSHOES/DataRelease/tree/main/Pantheon%2B_Data/photometry/Foundation_DJ17.
**Acknowledgements and Terms of Use** Original content from the work [53, 81] may be used under the terms of the Creative Commons Attribution 3.0 license. Any further distribution of this work must include attribution to the author(s), the title of the respective work, journal citation and DOI.

### A.4.9 CSP: csp

<figure><img src="image_47.png" /></figure>

**Description** The third data release of the first stage of the Carnegie Supernova Project (CSP) contains light curves of 134 spectroscopically confirmed Type Ia supernovae observed between 2004-2009 [86].

**Composition** Aside from the use of different bands (ugriBVYJH), the data fields are identical to those of the CfA datasets.

**Collection Process** The compilation of the CSP data is described in the data release paper [86]. The dataset was accessed from the CSP website's data products page https://csp obs.carnegiescience.edu/data.

**Preprocessing/cleaning** No pre-processing has been applied.

**Uses** The CSP dataset has been used in many studies including those mentioned in the CfA section, as well as several CSP studies [e.g. 32, 115, 55, 140].

**Acknowledgements and Terms of Use** The CSP dataset is provided under the CC BY 4.0 license. If you use CSP, please cite the respective data release paper [86] in your paper.

### A.4.10 Swift SNe Ia: swift_sne_ia

<figure><img src="image_48.png" /></figure>

**Description** A collection of 117 spectroscopically confirmed type Ia supernova light curves from the Swift Type Ia Supernova (Swift SN Ia) dataset [27]. This dataset can be used for photometric redshift prediction or light curve inpainting.

**Composition** We present the data as light curve objects containing time of observation, flux, flux error and band pass filter. Measurements are in $UV_{w1}$, $u$, $b$, $UV_{w2}$, $v$ and $UV_{m2}$ band pass filters. The supernova classification is included as metadata. Additional metadata includes information on coordinates, redshifts and host galaxy masses.

**Collection Process** The data is collected from the Pantheon+ compilation [128] that also performs a series of selection cuts. The data is accessed through the compilation's GitHub: https://github.com/PantheonPlusSHOES/DataRelease/tree/main/Pantheon%2B_Data/1_DATA/photometry/SWIFT.

**Acknowledgements and Terms of Use** Original content from the work [27] may be used under the terms of the GNU Lesser General Public License.

## A.5 Spectra Datasets

### A.5.1 Gaia BP/RP: gaia

<figure><img src="image_49.png" /></figure>

Refer to Subsection A.7.1.

### A.5.2 SDSS: sdss

<figure><img src="image_50.png" /></figure>

**Description** The Sloan Digital Sky Survey (SDSS; [153]) is a comprehensive astronomical survey that has mapped the sky using a dedicated 2.5-meter wide-angle optical telescope located at the Apache Point Observatory in New Mexico, USA. SDSS has collected detailed images and spectra of millions of celestial objects, covering a significant portion of the sky in multiple bands. In this dataset we collect data from all SDSS optical spectra including SDSS Legacy survey, SEGUE-1, SEGUE-2, BOSS, and eBOSS surveys [153, 49, 19].
**Composition** The SDSS dataset includes spectra spanning a wavelength range from 3,800 to 9,200 for spectra acquired with the SDSS spectrograph and 3,650-10,400 for data from the BOSS spectrograph. In addition, for various processing reasons, not all spectra have the same lengths. Each spectrum comes with associated metadata, including flux measurements, wavelength values, and inverse variance (ivar) for the flux. Additionally, pixel-level masks indicate potential quality issues. The dataset contains a total of roughly 4 million spectra.

We apply a minimal selection on this data by only imposing the following cuts:

* Avoid duplicated spectra for the same object: SPECPRIMARY = 1

* Only good plates: PLATEQUALITY = 'good'

* Only science targets (no sky, calibration, etc.): TARGETTYPE = 'science'

**Collection Process** We use the publicly accessible data from the SDSS 18th data release (SDSS DR18)<fnref n="22" />, and systematically download all processed spectra files for spectra that pass the selection cut described above.

**Preprocessing/Cleaning** No further cuts of processing steps are performed.

**Uses** SDSS spectra have been used in numerous scientific publications. Notable examples involving machine learning include building data-driven representations of galaxy spectra and looking for outliers [104].

**Acknowledgements and Terms of Use** All SDSS data released in public data releases is considered to be in the public domain. When using the data included in this compilation, the following acknowledgements should be included:

* **SDSS and SEGUE-1 samples:** Funding for the SDSS and SDSS-II has been provided by the Alfred P. Sloan Foundation, the Participating Institutions, the National Science Foundation, the U.S. Department of Energy, the National Aeronautics and Space Administration, the Japanese Monbukagakusho, the Max Planck Society, and the Higher Education Funding Council for England. The SDSS Web Site is http://www.sdss.org/. The SDSS is managed by the Astrophysical Research Consortium for the Participating Institutions. The Participating Institutions are the American Museum of Natural History, Astrophysical Institute Potsdam, University of Basel, University of Cambridge, Case Western Reserve University, University of Chicago, Drexel University, Fermilab, the Institute for Advanced Study, the Japan Participation Group, Johns Hopkins University, the Joint Institute for Nuclear Astrophysics, the Kavli Institute for Particle Astrophysics and Cosmology, the Korean Scientist Group, the Chinese Academy of Sciences (LAMOST), Los Alamos National Laboratory, the Max-Planck-Institute for Astronomy (MPIA), the Max-Planck-Institute for Astrophysics (MPA), New Mexico State University, Ohio State University, University of Pittsburgh, University of Portsmouth, Princeton University, the United States Naval Observatory, and the University of Washington.

* **BOSS and SEGUE-2 samples:** Funding for SDSS-III has been provided by the Alfred P. Sloan Foundation, the Participating Institutions, the National Science Foundation, and the U.S. Department of Energy Office of Science. The SDSS-III web site is http://www.sdss3.org/. SDSS-III is managed by the Astrophysical Research Consortium for the Participating Institutions of the SDSS-III Collaboration including the University of Arizona, the Brazilian Participation Group, Brookhaven National Laboratory, Carnegie Mellon University, University of Florida, the French Participation Group, the German Participation Group, Harvard University, the Instituto de Astrofísica de Canarias, the Michigan State/Notre Dame/JINA Participation Group, Johns Hopkins University, Lawrence Berkeley National Laboratory, Max Planck Institute for Astrophysics, Max Planck Institute for Extraterrestrial Physics, New Mexico State University, New

<footnote n="22">²²https://www.sdss.org/dr18/data_access/bulk/</footnote>
York University, Ohio State University, Pennsylvania State University, University of Portsmouth, Princeton University, the Spanish Participation Group, University of Tokyo, University of Utah, Vanderbilt University, University of Virginia, University of Washington, and Yale University.

* **eBOSS sample:** Funding for the Sloan Digital Sky Survey IV has been provided by the Alfred P. Sloan Foundation, the U.S. Department of Energy Office of Science, and the Participating Institutions. SDSS acknowledges support and resources from the Center for High-Performance Computing at the University of Utah. The SDSS web site is www.sdss4.org.

SDSS is managed by the Astrophysical Research Consortium for the Participating Institutions of the SDSS Collaboration including the Brazilian Participation Group, the Carnegie Institution for Science, Carnegie Mellon University, Center for Astrophysics | Harvard & Smithsonian (CfA), the Chilean Participation Group, the French Participation Group, Instituto de Astrofísica de Canarias, The Johns Hopkins University, Kavli Institute for the Physics and Mathematics of the Universe (IPMU) / University of Tokyo, the Korean Participation Group, Lawrence Berkeley National Laboratory, Leibniz Institut für Astrophysik Potsdam (AIP), Max-Planck-Institut für Astronomie (MPIA Heidelberg), Max-Planck-Institut für Astrophysik (MPA Garching), Max-Planck-Institut für Extraterrestrische Physik (MPE), National Astronomical Observatories of China, New Mexico State University, New York University, University of Notre Dame, Observatório Nacional / MCTI, The Ohio State University, Pennsylvania State University, Shanghai Astronomical Observatory, United Kingdom Participation Group, Universidad Nacional Autónoma de México, University of Arizona, University of Colorado Boulder, University of Oxford, University of Portsmouth, University of Utah, University of Virginia, University of Washington, University of Wisconsin, Vanderbilt University, and Yale University.

### A.5.3 DESI: desi

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

**Description** The Dark Energy Spectroscopic Instrument (DESI; [41]) is designed to measure the effect of dark energy on the expansion of the universe by mapping the distributions of galaxies in three dimensions. To do so, DESI uses a 4-meter telescope at Kitt Peak National Observatory to collect spectra from millions of galaxies and quasars over a five-year period. This experiment is still on-going and most of the data is not yet public. In this dataset, we specifically make use of the DESI Early Data Release (EDR) [40] which contains 1% survey of the final survey.

**Composition** The DESI EDR sample includes spectra with a fixed wavelength range from 3,600 to 9,800 and 7081 pixels. Each spectrum is accompanied by flux measurements, wavelength values, and inverse variance (ivar) for the flux. To select the sample including in this dataset we only apply the following minimal set of selection cuts:

* Only use data from the one percent survey: `SURVEY = 'sv3'`

* Only use the primary spectrum for each object: `SV_PRIMARY` is true

* Only use targets (ignore sky and others): `OBJTYPE = 'TGT'`

* Only use fibers with good status: `COADD_FIBERSTATUS = 0`

This results in a set of approximately 1M spectra of galaxies, quasars, and stars.

**Collection Process** We systematically download all spectra files for the SV3 sample through GLOBUS from the DESI EDR archive https://data.desi.lbl.gov/doc/releases/.

**Preprocessing/Cleaning** We apply no further pre-processing beyond the set of cuts described above.
**Uses** The DESI EDR being still very recent, few machine learning works have been developed on this dataset. Still, a few examples are the AstroClip multimodal representation learning model [112], and the detection of outliers with thespender spectrum auto-encoding model [96].

**Acknowledgements and Terms of Use** The Dark Energy Spectroscopic Instrument (DESI) data are licensed under the Creative Commons Attribution 4.0 International License ("CC BY 4.0"). Users are free to share, copy, redistribute, adapt, transform and build upon the DESI data available through https://data.desi.lbl.gov/ for any purpose, including commercially.

When DESI data are used, the appropriate credit is required by using both the following reference [40] and acknowledgements text:

This research used data obtained with the Dark Energy Spectroscopic Instrument (DESI). DESI construction and operations is managed by the Lawrence Berkeley National Laboratory. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of High-Energy Physics, under Contract No. DE-AC02-05CH11231, and by the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility under the same contract. Additional support for DESI was provided by the U.S. National Science Foundation (NSF), Division of Astronomical Sciences under Contract No. AST-0950945 to the NSF's National Optical-Infrared Astronomy Research Laboratory; the Science and Technology Facilities Council of the United Kingdom; the Gordon and Betty Moore Foundation; the Heising-Simons Foundation; the French Alternative Energies and Atomic Energy Commission (CEA); the National Council of Science and Technology of Mexico (CONACYT); the Ministry of Science and Innovation of Spain (MICINN), and by the DESI Member Institutions: www.desi.lbl.gov/collaborating-institutions. The DESI collaboration is honored to be permitted to conduct scientific research on Iolkam Du'ag (Kitt Peak), a mountain with particular significance to the Tohono O'odham Nation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. National Science Foundation, the U.S. Department of Energy, or any of the listed funding agencies.

### A.5.4 APOGEE: apogee

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

**Description** Apache Point Observatory Galactic Evolution Experiment (APOGEE; [19, 2]) within the Sloan Digital Sky Survey (SDSS) is a high-resolution ($R \sim 22,000$), high signal-to-noise (100 per pixel typically) stellar spectroscopic survey with 2.5-m telescopes in both northern and southern hemisphere in the near infrared H-band wavelength region [102, 151].

**Composition** Fixed size APOGEE stellar spectra (1 × 7514) spanning a wavelength range of 1.5 to 1.7µm. Each stellar spectrum comes with both flux and continuum normalized flux (i.e. flux with continuum removed), its associated λ vector representing the corresponding wavelength of each pixel, as well the inverse variance (ivar) for both flux and continuum normalized flux and a pixel-level mask indicating potential quality issues. The dataset contains a total of roughly 700,000 stellar spectra. Surface temperature $T_{\text{eff}}$, surface gravity $\log g$, overall metallicity $[M/H]$, overall alpha abundance $[\alpha/H]$ and radial velocity are derived using the APOGEE Stellar Parameter and Chemical Abundances Pipeline [61] from the APOGEE spectra.

**Collection Process** We use the publicly accessible data from the APOGEE 17th data release (APOGEE DR17) catalog$^{23}$.

**Preprocessing/cleaning** The dataset only include APOGEE spectra that has signal-to-noise (SNR) greater than 30, where duplicated stars in the public data released are removed due to some stars being observed in multiple programs multiple times. Preprocessing of the raw data are done internally hence minimal processing are needed for the public data release which we are using.

<footnote>$^{23}$https://www.sdss4.org/dr17/irspec/</footnote>
**Uses** The public APOGEE DR17 dataset has been used in a number of scientific publications in both stellar and Galactic astronomy, as well as in machine learning specific applications.

**Acknowledgements and Terms of Use** Funding for the Sloan Digital Sky Survey IV has been provided by the Alfred P. Sloan Foundation, the U.S. Department of Energy Office of Science, and the Participating Institutions. SDSS-IV acknowledges support and resources from the Center for High Performance Computing at the University of Utah. The SDSS website is www.sdss.org. All SDSS data released in our public data releases is considered in the public domain. If you use APOGEE data in your work, please cite and acknowledge it as above.

### A.5.5 GALAH: galah

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

**Description** Galactic Archaeology with HERMES (GALAH; [39]) is an Australian-led survey aimed at obtaining high-quality one-dimensional stellar spectra for deriving stellar atmospheric parameters and elemental abundances.

**Composition** The dataset includes spectra and derived stellar parameters such as effective temperature and several elemental abundances. The publicly available Data Release 3 (DR3; [28]) contains spectra for 325,518 objects after applying recommended quality cuts.

**Collection Process** The GALAH survey utilizes the HERMES spectrograph on the Anglo-Australian Telescope to collect high-resolution spectra. The observations are processed and made publicly available through periodic data releases, with DR3 being the latest release used for this dataset. We download the data through https://www.galah-survey.org/dr3/the_spectra/

**Preprocessing/Cleaning** We apply recommended cuts on spectra quality<fnref n="24" /> to ensure the reliability of the data, resulting in a dataset with high-quality spectra and accurate stellar parameters.

**Uses** The GALAH dataset is used in studies of stellar populations, Galactic archaeology, and the chemical evolution of the Milky Way.

**Acknowledgements and Terms of Use** The GALAH data is openly accessible, When using GALAH data, please cite [39, 28] and provide the following acknowledgements:

This work made use of the Third Data Release of the GALAH Survey (Buder et al. 2021). The GALAH Survey is based on data acquired through the Australian Astronomical Observatory, under programs: A/2013B/13 (The GALAH pilot survey); A/2014A/25, A/2015A/19, A2017A/18 (The GALAH survey phase 1); A2018A/18 (Open clusters with HERMES); A2019A/1 (Hierarchical star formation in Ori OB1); A2019A/15 (The GALAH survey phase 2); A/2015B/19, A/2016A/22, A/2016B/10, A/2017B/16, A/2018B/15 (The HERMES-TESS program); and A/2015A/3, A/2015B/1, A/2015B/19, A/2016A/22, A/2016B/12, A/2017A/14 (The HERMES K2-follow-up program). We acknowledge the traditional owners of the land on which the AAT stands, the Gamilaray people, and pay our respects to elders past and present. This paper includes data that has been provided by AAO Data Central (datacentral.org.au).

### A.5.6 VIPERS: vipers

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

**Description** The “VIMOS Public Extragalactic Redshift Survey” (VIPERS) [126] dataset contains measured optical spectra of galaxies in the 0.5 < z < 1.0 range.

**Composition** Fixed size spectra (1 × 557) spanning a wavelength range of 5514 Å to 9484 Å, taken at the center of each galaxy. Each spectrum comes with its associated λ vector,

<footnote n="24">²⁴https://www.galah-survey.org/dr3/using_the_data/</footnote>
as well the inverse variance (ivar) and a mask of each flux measurement in the spectrum. The dataset continns a total of roughly 90,000 galaxies with red magnitude I(AB) brighter than 22.5 over an overall area of nearly 24 square degrees.

**Collection Process** For consistency, we use the publicly accessible data from the second VIPERS public data release (PDR-2) catalog<fnref n="25" />.

**Preprocessing/cleaning** The VIPERS spectra flux is natively stored in units of erg cm⁻²s⁻¹Å⁻¹, and correspondingly the noise is stored in erg cm⁻²s⁻¹Å⁻¹ and the wavelength in angstroms (Å). To be consistent with the other spectra datasets, we normalize the spectra fluxes to 10⁻¹⁷erg cm⁻²s⁻¹Å⁻¹. Correspondingly, we convert the spectra noise to inverse variance, also in units 10⁻¹⁷erg cm⁻²s⁻¹Å⁻¹.

**Uses** This dataset has been used in a number of scientific publications, as well as in machine learning specific applications, including source identification with SVMs [103] and galaxy classification [129] with unsupervised methods.

**Acknowledgements and Terms of Use** This paper uses data from the VIMOS Public Extragalactic Redshift Survey (VIPERS). VIPERS has been performed using the ESO Very Large Telescope, under the "Large Programme" 182.A-0886. The participating institutions and funding agencies are listed at http://vipers.inaf.it.

<figure><img src="image_64.png" /><figcaption>A.5.7 Chandra: chandra</figcaption></figure>

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /><figcaption>Figure 5: Examples of Chandra X-ray spectra included in MULTIMODAL UNIVERSE. Spectra of sources 2CXO J002029.1+591651 (left) and 2CXO J023437.8-084716 (right) are shown.</figcaption></figure>

**Description** The Chandra X-ray Observatory is one of NASA's flagship observatories, studying the X-ray emission from the most extreme environments in the Universe, including the surroundings of stellar-mass black holes and super massive black holes (SMBHs), super-nova remnants, and violent coronal mass ejections from young stars [150]. The observatory records these observations using two instruments, the Advanced CCD Imaging Spectrometer (ACIS), and the High Resolution Camera (HRC), with CCD chips arranged to cover the telescope's field of view. The Chandra processing pipeline produces the individual X-ray photon recordings for each observation in the form of event files, which can be thought of as multivariate time series of the photon's energies and spatial location on the detector, with the energies covering the range between approximately 0.5 keV and 8 keV. MULTIMODAL UNIVERSE contains ACIS X-ray spectra of astrophysical sources, derived from the event files through the proper application of the instrument's spectral response.

X-ray spectra (photon energy versus count rate) from astrophysical sources detected by the Chandra X-ray Observatory in the energy range between 0.5keV and 8keV, binned at 10 counts per energy bin. The spectral files are downloaded from the Chandra Source Catalog

<footnote n="25">²⁵http://www.vipers.inaf.it/rel-pdr2.html</footnote>
(CSC) version 2.1 server. (https://cxc.cfa.harvard.edu/csc/). We provide all CSC
spectra that have at least 40 total counts and a detection signal to noise ratio of 4 or larger.

**Composition** Each entry consist of the source identifiers and coordinates, as well as arrays that contain the low and high limits of the energy bins, the center of the bin, the measured count rate per bin, and the error in that count rate. In addition, we provide features that characterize the observed source: the aperture photometry X-ray flux in physical units (ergs s⁻¹ cm⁻²), the flux estimated using a blackbody fit to the spectrum, the flux significance, or signal-to-noise ratio, three hardness ratios which characterize the spectral shape, and measures of the flux variability as a function of time, both as a probability and as an index that quantifies the statistical significance of the variability.

**Collection Process** The data was originally collected as part of the Chandra Source Catalog project, which detects and characterizes all sources serendipitously detected by Chandra, and produces both tables and data products for those sources, including spectral files, storing them as a database in a dedicated server hosted at the Smithsonian Astrophysical Observatory. For MULTIMODAL UNIVERSE, we have accessed these files using International Virtual Observatory (IVOA) interfaces and HTTP requests via command line interface.

**Preprocessing/cleaning** The observatory records these observations using two instruments, the Advanced CCD Imaging Spectrometer (ACIS), and the High Resolution Camera (HRC), with CCD chips arranged to cover the telescope's field of view. The Chandra processing pipeline produces the individual X-ray photon recordings for each observation in the form of event files, which can be thought of as multivariate time series of the photon's energies and spatial location on the detector, with the energies covering the range between approximately 0.5 keV and 8 keV. MULTIMODAL UNIVERSE contains ACIS X-ray spectra of astrophysical sources, derived from the event files through the proper application of the instrument's spectral response, and processed using the Chandra Interactive Analysis of Observations (CIAO) software (https://cxc.cfa.harvard.edu/ciao/), in particular the Sherpa package.

**Uses** Chandra studies the X-ray emission from the most extreme environments in the Universe, including the surroundings of stellar-mass black holes and super massive black holes (SMBHs), supernova remnants, and violent coronal mass ejections from young stars [150]. Many of the serendipitous sources compiled in the CSC lack and astrophysical classification. A similar dataset to the one presented here has been used to perform supervised machine learning classification of Chandra sources [67].

**Acknowledgements and Terms of Use** This research has made use of data obtained from the Chandra Source Catalog, provided by the Chandra X-ray Center (CXC). The CSC (doi:10.25574/csc2) is publicly available. The current version is CSC 2.1. CXC provides up to date guidance<fnref n="26" /> on acknowledging and citing their work, which we ask you do if you use Chandra data.

## A.6 HyperSpectral Imaging Datasets

### A.6.1 MaNGA manga

<figure><img src="image_67.png" /></figure>

**Description** The SDSS-IV MaNGA survey[29] is a wide-field, optical, IFU survey of 10,000 nearby galaxies. MaNGA's goal is to understand the “life history” of present day galaxies from imprinted clues of their birth and assembly, through their ongoing growth via star formation and merging, to their death from quenching at late times.

The primary MaNGA data products are composed of 3-D calibrated data cubes produced by the DRP[89] and 2-D maps of derived quantities, such as emission line fluxes, gas and stellar kinematics, and stellar population properties, produced by the DAP[149] from those cubes. The 3-D data cubes are constructed from a few tens to a few thousands of individual spectra that have been combined onto a regular grid. The 2-D maps of derived quantities

<footnote n="26">²⁶https://cxc.cfa.harvard.edu/csc2.1/cite.html</footnote>
<figure><img src="image_68.png" /><figcaption>Figure 6: Left: gri image of MaNGA 1-596678 with the IFU field of view shown in purple. Middle: IFU observations produce three-dimensional data cubes, with one spectral and two spatial dimensions (credit: Stephen Todd and Douglas Pierce-Price; IFS Wiki http://ifs.wikidot.com). Right: spectral analysis of individual spaxels produces hundreds of two-dimensional maps for each galaxy spanning a wide range of physical properties. The four example maps shown for 1-596678 are flux (top left), log([N ii] 6585/) flux (top right), stellar velocity (bottom left), and stellar velocity dispersion corrected for instrumental broadening (bottom right). Original figure from the SDSS MaNGA Marvin paper, [35].</figcaption></figure>

are constructed by analyzing individual or binned groups of spaxels and constructing maps of the quantities at the relevant on-sky location.

This dataset contains information pulled from the MaNGA DRP LOGCUBE<fnref n="27" /> files, as well as the DAP "HYB10-MILESHC-MASTARSSP" MAPS<fnref n="28" /> files.

**Composition** Each entry in the dataset is a single MaNGA galaxy observation, identified by its unique id designation, the "plate-IFU", e.g. "8485-1901", with information aggregated from both the IFU data cubes and the derived analysis maps, for that target. For each galaxy, the following features are provided:

* **object_id**: the MaNGA plate-IFU

* **ra**: the Right Ascension of the target

* **dec**: the Declination of the target

* **healpix**: the healpix id

* **z**: the target redshift

* **spaxel_size**: the size of the spaxel (spatial pixel)

* **spaxel_size_units**: the units of the spaxel size

* **spaxels**: a list of spaxels, with properties

* **images**: a list of reconstructed images, with properties

* **maps**: a list of derived maps, with properties

Each spaxel item has the following features:

* **flux**: the flux spectrum array in the spaxel

* **ivar**: the inverse variance array in the spaxel

* **mask**: the pixel quality mask array in the spaxel

* **lsf**: the line-spread-function array in the spaxel

<footnote n="27">²⁷https://www.sdss4.org/dr17/manga/manga-data/data-model/#CUBEFiles</footnote>

<footnote n="28">²⁸https://www.sdss4.org/dr17/manga/manga-data/data-model/#DataAnalysisPipelineOutput</footnote>
* `lambda`: the wavelength array, same for all spaxels

* `x`: the x-array index spaxel coordinate

* `y`: the y-array index spaxel coordinate

* `spaxel_idx`: the 1d array index of the numpy.raveled spaxel

* `flux_units`: the physical units of the flux array

* `lambda_units`: the physical units of the wavelength array

* `skycoo_x`: the sky x offset in RA, in arcsec from galaxy center

* `skycoo_y`: the sky y offset in DEC, in arcsec from galaxy center

* `ellcoo_r`: the elliptical polar coordinate radial offset, from galaxy center, R in arcsec

* `ellcoo_rre`: the elliptical polar coordinate radial offset, from galaxy center, as R/Re

* `ellcoo_rkpc`: the elliptical polar coordinate radial offset, from galaxy center, R in h-1 kpc

* `ellcoo_theta`: the elliptical polar coordinate angle offset, from galaxy center, theta in degrees

* `skycoo_units`: the physical units of skycoo_x/y

* `ellcoo_r_units`: the physical units of ellcoo_r

* `ellcoo_rre_units`: the physical units of ellcoo_rre

* `ellcoo_rkpc_units`: the physical units of ellcoo_rkpc

* `ellcoo_theta_units`: the physical units of ellcoo_theta

Each **images** item has the following features:

* `filter`: the image band or filter, e.g. griz

* `array`: the 2d array pixel values

* `array_units`: the physical units of the data

* `psf`: the reconstructed PSF pixel values

* `psf_units`: the physical units of the data

* `scale`: the image pixel scale

* `scale_units`: the units of the image scale

Each **maps** item has the following features:

* `group`: the original MaNGA DAP extension, e.g. `emline_gflux`

* `label`: the full map label, made from the HDU extension name + channel, e.g. `emline_gflux_ha_6564`.

* `array`: the 2d array of pixel values

* `ivar`: the 2d array of inverse variance values

* `mask`: the 2d array of pixel quality mask values

* `array_units`: the physical units of the map data

**Collection Process** We use the publicly accessible data for MaNGA from the SDSS-IV 17th data release (DR17)<fnref n="29" />.

**Preprocessing/cleaning** During processing all IFU cubes, and maps have been resized to the same spatial grid of 96 x 96, with zero-padded elements added around the edges of the data. Thus all cubes have shape 96 x 96 x 4563 (42,882,048 pixels), and all maps have shape 96 x 96 (9,216 pixels).

<footnote n="29">²⁹https://www.sdss4.org/dr17/manga/</footnote>
**Uses** The public MaNGA DR17 dataset has been used in a number of scientific publications in extra-galactic astronomy, as well as in machine learning specific applications.

**Acknowledgements and Terms of Use** All SDSS data released in our public data releases is considered in the public domain. When using the data included in this compilation, the following acknowledgements should be included:

Funding for the Sloan Digital Sky Survey IV has been provided by the Alfred P. Sloan Foundation, the U.S. Department of Energy Office of Science, and the Participating Institutions. SDSS-IV acknowledges support and resources from the Center for High Performance Computing at the University of Utah. The SDSS website is www.sdss.org.

SDSS-IV is managed by the Astrophysical Research Consortium for the Participating Institutions of the SDSS Collaboration including the Brazilian Participation Group, the Carnegie Institution for Science, Carnegie Mellon University, Center for Astrophysics | Harvard & Smithsonian, the Chilean Participation Group, the French Participation Group, Instituto de Astrofísica de Canarias, The Johns Hopkins University, Kavli Institute for the Physics and Mathematics of the Universe (IPMU) / University of Tokyo, the Korean Participation Group, Lawrence Berkeley National Laboratory, Leibniz Institut für Astrophysik Potsdam (AIP), Max-Planck-Institut für Astronomie (MPIA Heidelberg), Max-Planck-Institut für Astrophysik (MPA Garching), Max-Planck-Institut für Extraterrestrische Physik (MPE), National Astronomical Observatories of China, New Mexico State University, New York University, University of Notre Dame, Observatório Nacional / MCTI, The Ohio State University, Pennsylvania State University, Shanghai Astronomical Observatory, United Kingdom Participation Group, Universidad Nacional Autónoma de México, University of Arizona, University of Colorado Boulder, University of Oxford, University of Portsmouth, University of Utah, University of Virginia, University of Washington, University of Wisconsin, Vanderbilt University, and Yale University.

## A.7 Tabular Datasets

### A.7.1 Gaia DR3: gaia

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

**Description** The Gaia mission is primarily intended for very precisely measuring astrometry of stars in the Milky Way (MW) from space at an unprecedented scale. Data Release 3 (DR3), in addition to astrometric measurements (position on the sky, parallaxes, proper motions, as well as associated uncertainties), also provides, for example: photometry; physical parameters (stellar surface properties like surface gravity and effective temperature); epoch photometry for variables; orbital elements for binary systems; high-resolution spectra from the radial velocity (RV) spectrometer and low-resolution spectra from the blue and red photometers (BP and RP).

**Composition** The dataset currently contains the following:

* Low-resolution optical spectra [38]: the spectra are stored as 110 coefficients of Gauss-Hermite polynomials (55 for BP, and 55 for RP). These can be sampled to an arbitrary wavelength grid using the GaiaXPy tool<fnref n="30" />.

* Astrometry: on-sky position, parallax, and proper motions, along with the full covariance matrix.

* Photometry: mean *G*, *G*<sub>BP</sub>, and *G*<sub>RP</sub> magnitudes and fluxes, along with errors.

* Photometrically-estimated parameters: stellar parameters (e.g., distance, surface gravity, metallicity, surface temperature) and lower/upper bounds derived from BP/RP spectra, apparent *G* magnitudes, and parallax.

* Radial velocities: RVs, errors, and stellar parameters of the template used to derive the RVs.

* Corrections and flags: commonly used variables used to make, for example, radial velocity and parallax zero-point corrections.

<footnote n="30">³⁰https://www.cosmos.esa.int/web/gaia/gaiaxpy</footnote>
**Collection Process** We use the publicly accessible DR3 data through the mirror hosted by the Flatiron Institute<fnref n="31" />.

**Preprocessing/cleaning** We select the subset of 220M stars in DR3 for which Gaia mean BP/RP spectra are provided. For these stars, we provide the contents described above. In the future we will include the entire dataset of the nearly 2B sources measured in DR3.

**Uses** DR3 data has already been used extensively in MW science and in machine learning applications. For example, using DR3, [57] explored and identified non-axisymmetric features in the disc, [58] have created a detailed chemodynamical map of the MW, and [94] have trained a transformer-based model that can generate stellar spectra, estimate physical parameters, and inpaint unobserved spectral regions.

**Acknowledgements and Terms of Use** This work has made use of data from the European Space Agency (ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and Analysis Consortium (DPAC, https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement. The Gaia data are open and free to use, provided credit is given to ‘ESA/Gaia/DPAC’. If you use Gaia DR3 data in your research, please acknowledge it as above.

### A.7.2 Galaxy10 DECaLS gz10

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

**Description** Galaxy Zoo (https://www.zooniverse.org/projects/zookeeper/galaxy-zoo/) is a long running citizen science (crowd-sourcing) effort recruiting volunteers to annotate images of galaxies online according to the galaxies structure such as whether or not they have spiral arms or bulges [99]. These structures are useful, and traditionally expensive, tracers of underlying physics such as galaxy evolution [124]. Galaxy10 DECaLS [93] represents a subset of the Galaxy Zoo data, where the original decision tree for Galaxy Zoo is split into ten classes described below.

<figure><img src="image_76.png" /><figcaption>Figure 7: Examples of each class of the GalaxyZoo10 DECaLS data set with their RGB composite image. Adapted from original figure from [93].</figcaption></figure>

**Composition** [92] The data set is comprised of the images which volunteers saw and an aggregation of their resulting classifications into 10 subsets class labels for the objects spanning 17,736 samples split among the following classes:

1. Disturbed Galaxies (1,081)

<footnote n="31">³¹https://sdsc-users flatironinstitute.org/~gaia/dr3/</footnote>
2. Merging Galaxies (1,853)

3. Round Smooth Galaxies (2,645)

4. In-between Round Smooth Galaxies (2,027)

5. Cigar Shaped Smooth Galaxies (334)

6. Barred Spiral Galaxies (2,043)

7. Unbarred Tight Spiral Galaxies (1,829)

8. Unbarred Loose Spiral Galaxies (2,628)

9. Edge-on Galaxies without Bulge (1,423)

10. Edge-on Galaxies with Bulge (1,873)

The data can be loaded in two broad configurations: Firstly, as just the labels with auxiliary data, or the labels with the respective *uint8* PNGs (3x256x256). The auxiliary data includes the right ascension *ra*, declination *dec*, redshift, and MULTIMODAL UNIVERSE specific object id *object_id*. Additionally, if the images are loaded, the pixel scale of the images is included as *rgbPixelScale*.

**Collection Process** The images presented in this dataset are RGB composites [145] of data collected by the DESI legacy imaging survey [44] and are analogous to what the volunteers were shown while assigning the labels to the galaxies. The collection process for the labels of the underlying data are detailed in [145, 97].

**Preprocessing/cleaning** There are a number of selection effects made on what data received labels, see [145]. The labels, constructed from the original decision tree (https://data.galaxyzoo.org/gz Trees/gz Trees.html) are collated to select confident and clearly distinguishable classes [92] for simplicity. Only confident labels are maintained as this dataset was intended as a benchmark dataset with minimal or no label noise. Several baseline models for this benchmark are provided, see subsection B.1 Additionally, the images are not the full scientific measurement of the respective object, as done by DESI [44], but rather a RGB composite designed to highlight the respective structure. These images are what the volunteers would have seen whilst labelling the respective objects.

**Uses** A number of publications have made use of this data, presented on the homepage (https://astronn.readthedocs.io/en/latest/galaxy10.html#some-papers-that-used-galaxy-10). These efforts are largely bench marking the efficacy of various approaches for galaxy morphology classification, as the labels in this dataset are clean and relatively simple in comparison to the raw labels of the decision trees, such as those presented in [145].

**Acknowledgements and Terms of Use** Galaxy Zoo is described in [99], the GalaxyZoo Data Release 2 is described in [98], Galaxy Zoo DECaLS Campaign is described in Galaxy Zoo is described in [99], the GalaxyZoo Data Release 2 is described in [98], Galaxy Zoo DECaLS Campaign is described in [145], DESI Legacy Imaging Surveys is described in [42].

The Legacy Surveys consist of three individual and complementary projects: the Dark Energy Camera Legacy Survey (DECaLS; Proposal ID #2014B-0404; PIs: David Schlegel and Arjun Dey), the Beijing-Arizona Sky Survey (BASS; NOAO Prop. ID #2015A-0801; PIs: Zhou Xu and Xiaohui Fan), and the Mayall z-band Legacy Survey (MzLS; Prop. ID #2016A-0453; PI: Arjun Dey). DECaLS, BASS and MzLS together include data obtained, respectively, at the Blanco telescope, Cerro Tololo Inter-American Observatory, NSF's NOIRLab; the Bok telescope, Steward Observatory, University of Arizona; and the Mayall telescope, Kitt Peak National Observatory, NOIRLab. The Legacy Surveys project is honored to be permitted to conduct astronomical research on Iolkam Du'ag (Kitt Peak), a mountain with particular significance to the Tohono O'odham Nation.

### A.7.3 PROVABGS desi.provabs

<figure><img src="image_77.png" /></figure>

**Description** The PRObabilistic Value-Added Bright Galaxy Survey (PROVABGS) [65] dataset provides measurements of galaxy properties, such as log stellar mass (log $M_*$), star
formation rate (SFR), mass-weighted stellar metallicity ($Z_{MW}$), and mass-weighted stellar age ($t_{age,MW}$) for galaxy spectra in the DESI Bright Galaxy Survey (BGS). These are inferred using a full Bayesian inference framework and state-of-the-art Spectral Energy Distribution (SED) modeling of the DESI spectra. At present, as the DESI survey is still on-going, the PROVABGS catalog reports these properties for the early data release (EDR) of the DESI sample.

**Composition** The best-fit parameters of the PROVABGS SED modeling for $log M_*, SFR, Z_{MW}, t_{age,MW}$. Additionally, it also includes the 100 × 13 array of 100 samples drawn from the 13-dimensional PROVABGS posterior. The 13 parameters correspond to $log(M_{form})^{32}$, $\beta_1, \beta_2, \beta_3, \beta_4, f_{burst}, t_{burst}, \gamma_1, \gamma_2, \tau_{BC}, \tau_{ISM}, n_{dust}, f_{fiber}$, from which the above properties are derived. We refer the reader to the original PROVABGS documentation$^{33}$ for the full details. Overall, the PROVABGS modeling of the DESI EDR contains physical properties for roughly 221,000 galaxy spectra.

**Collection Process** We use the reported PROVABGS posteriors from the publicly available DESI EDR BGS PROVABGS catalog$^{34}$.

**Preprocessing/cleaning** We extract the best fit parameters of the PROVABGS SED modeling for $log M_*, SFR, Z_{MW}, t_{age,MW}$ using the PROVABGS SPS model with non-parametric star formation and metallicity histories and flexible dust attenuation model.

**Uses** The PROVABGS catalog has been used for physical property estimation from both images and spectra in both self-supervised learning and supervised learning contexts [112].

**Acknowledgements and Terms of Use** All of the data included in this dataset is of the public domain. We are grateful to the PROVABGS team for making the data available to the community. This research used data obtained with the Dark Energy Spectroscopic Instrument (DESI). DESI construction and operations is managed by the Lawrence Berkeley National Laboratory. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of High-Energy Physics, under Contract No. DE-AC02-05CH11231, and by the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility under the same contract. Additional support for DESI was provided by the U.S. National Science Foundation (NSF), Division of Astronomical Sciences under Contract No. AST-0950945 to the NSF's National Optical-Infrared Astronomy Research Laboratory; the Science and Technology Facilities Council of the United Kingdom; the Gordon and Betty Moore Foundation; the Heising-Simons Foundation; the French Alternative Energies and Atomic Energy Commission (CEA); the National Council of Science and Technology of Mexico (CONACYT); the Ministry of Science and Innovation of Spain (MICINN), and by the DESI Member Institutions: www.desi.lbl.gov/collaborating-institutions. The DESI collaboration is honored to be permitted to conduct scientific research on Iolkam Du'ag (Kitt Peak), a mountain with particular significance to the Tohono O'odham Nation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. National Science Foundation, the U.S. Department of Energy, or any of the listed funding agencies.

## A.8 Maintenance

To ensure the continued evolution and relevance of MULTIMODAL UNIVERSE, the project is organized as an open and collaborative effort. Contributions from researchers are actively encouraged, with a well-defined process for submitting new data, benchmarks, and improvements via GitHub, as described in the project's CONTRIBUTING document. A dedicated team of maintainers oversees the integration of these contributions, ensuring that the dataset remains up-to-date and continues to meet the needs of the scientific and machine learning communities. This framework will enable regular updates to incorporate new data from

<footnote>$^{32}$Note that $log(M_{form})$ is different from log(stellar mass).</footnote>

<footnote>$^{33}$https://data.desi.lbl.gov/doc/releases/edr/vac/provabgs/</footnote>

<footnote>$^{34}$https://data.desi.lbl.gov/public/edr/vac/edr/provabgs/v1.0/README.md</footnote>
ongoing and future astronomical surveys, as well as periodic evaluations and improvements
based on feedback from the research community.

**A.9 Author statement**

The authors of this scientific paper bear full responsibility for any violation of rights that
may arise from the collection of the data included in this research.
# B Details on Benchmark tasks

## B.1 Galaxy10 DECaLS Morphology Classification

For the Galaxy10 DECaLS morphology classification baselines, we train all models using an 80/20 train-test split on the GalaxyZoo10 DECaLS classification labels and the corresponding Legacy Survey RGB-converted images. We train three canonical computer vision baseline networks - ResNet-18 [66], EfficientNetB0 [136], and DenseNet121 [75] - to classify these images. As they are already range compressed in their conversion to RGB images, we do not apply any range compression. To prevent overfitting, we use data augmentation: random horizontal and vertical flips, random gaussian blurring, random affine translation, and color jittering<fnref n="35" />. We train all models using a cross-entropy loss on the training dataset with the Adam optimizer [84]. Training is performed for 10 epochs on a single A100 GPU for each model with a batch size of 128 and a learning rate of $\lambda = 10^{-4}$. We report the Top-1 accuracy on the held-out validation test dataset.

## B.2 Physical Property Estimation

For the physical property inference baselines, we train the image and spectrum models using an 80/20 train-test split on the cross-match between the PROVABGS dataset and the Legacy Survey imaging datasets and the DESI spectrum datasets respectively.

For the image models, we use canonical computer vision baseline networks - ResNet-18 [66], EfficientNetB0 [136], and DenseNet121 [75] - to regress the PROVABGS galaxy properties from the Legacy Survey images. Due to the high dynamic range of the images, we use a range compression scheme to reduce the dynamic range for better optimization. Specifically, we apply arcsinh range compression and some additional clamping to the images. We also use data augmentation - random horizontal and vertical rotations and random gaussian blurring - to reduce overfitting. For the spectrum model, we use a convolutional+attention network with the same architecture as the encoder from the state-of-the-art spectrum encoder from [104]. For the photometry model, we use a simple 5-layer MLP with $d = 512$ hidden dimension. Additionally, we $Z$-score the photometry to normalize the data.

We train all models using a mean-squared error (MSE) loss on the training dataset with the Adam optimizer [84]. We $Z$-score the PROVABGS galaxy properties. Additionally, we convert metallicity ($Z_{MW}$) to log metallicity and star-formation rate ($SFR$) to $sSFR = \log SFR - \log Z_{MW}$. We train all models for 25 epochs on a single A100 GPU with a batch size of 256 and a learning rate of $\lambda = 10^{-4}$. We report the $R^2$ performance of all models on the held-out test datasets.

## B.3 PLAsTiCC Photometric Classification and Redshift Estimation

For the PLAsTiCC baseline model, we use an encoder-only Informer model [154] with 8 encoder layers of 12 attention heads each. The model hidden dimension was chosen to be 768 and the layer MLPs have hidden dimension 256. Due to the 2-dimensional position data (each element of the time-series has an associated time and photometric band/wavelength) and irregular sampling of our dataset, we train a positional encoding based on learnable Fourier features following [95]. We also select a random window of length 300 from each example (and zero-pad examples with fewer than 300 observations) to produce inputs of uniform shape. We first pretrain with a masked autoencoding objective:

$$ \mathcal{L}_{\text{MAE}}(\phi) = \mathbb{E}_{x \sim P_U, x' \sim \mathcal{A}_{\text{pre}}(\cdot|x)}[(\phi(x') - x)^2] \quad (1) $$

Here, $P_U$ is the distribution of unlabeled data that we pretrain with, $\mathcal{A}_{\text{pre}}$ is the distribution of augmented (masked) inputs, and $\phi$ is the model. We perform pretraining by randomly masking 60% of observations of each time-series, a batch size of 256, and learning rate 1e-4

<footnote n="35">³⁵In other astrophysical contexts color jittering tends to be avoided due to its potential to corrupt valuable astrophysical information present in the galaxy colors; however, as we are only concerned with galaxy morphology in this application, we apply it here.</footnote>
(selected from 1e-3 ~ 1e-6) for 75,000 steps. We finetune the pretrained model with linear
probing for 20,000 steps and learning rate 1e-4, then fine-tuning for 10,000 steps at learning
rate of 4e-5. The redshift estimation task uses LP learning rate of 5e-4 and FT learning rate
of 1e-4. We decrease the learning rate per step with a linear scheduler.

## B.4 YSE Photometric classification

We demonstrate an implementation of the snmachine package [100, 8] on the YSE data set with classes mapped to SNe Ia, SNe II, SNe Ib/c, or other. snmachine offers feature extraction methods based on SN Ia templates [64], parametric models [108, 82], and wavelet analysis [90], along with wrappers for several classifiers from scikit-learn [113]. We find that wavelet-based feature extraction combined with the Random Forest classifier, with or without the ensemble AdaBoost classifier, achieves a AUC of 0.90in SN Ia one-vs-rest classification.

## B.5 BTSbot Candidate Identification

We report on the performance of BTSbot - a multimodal convolutional neural network which combines images of potential new transient detections with associated metadata - on the BTSbot training sample [119, 120]. Applied to a validation data set comprising 10 per cent of the entire data set, BTSbot achieves an AUC of 0.985 [120]. Given that the BTSbot training set includes multiple alerts for the same transient objects as they evolve, it is important to ensure that train/test splits are based on unique transient objects rather than just unique transient alerts, as in the public data set. While the overall AUC is an important measure, it is also valuable to consider the performance on an object-by-object basis rather than just considering all the individual alerts to determine how many unique transient objects are being identified. Early classification is also of value for this data set to allow transients to be targeted for further observations, therefore a model which is able to accurately identify real transient candidates as early as possible in their light curves will be of great benefit.

## B.6 Contrastive Image-Spectrum Pretraining

We replicate the results of AstroCLIP method [112] using the MULTIMODAL UNIVERSE cross-matched DESI and Legacy Surveys. We split the dataset using an 80/20 train-test split. We use the same hyperparameter and training suite as in [112] for training the model on the training suite. We then evaluate model performance by evaluating the zero-shot performance of a *k*-NN algorithm with *k* = 16 nearest neighbors on the cross-matched DESI and Legacy Survey and PROVABGS dataset. We report the *R*² performance of the model on the held-out test set. Note that the model has never seen the test set during either pretraining or zero-shot "training".

## B.7 From MaNGA Hyperspectral Datacubes to Physical Parameter Images

We demonstrate how 3D hyperspectral datacubes of galaxies can be used to infer 2D image "maps" corresponding to physical parameters, a process typically performed through complete spectrum analysis at each spaxel. In this use case, we train the canonical computer vision baseline network, U-Net [122], on a subset of ~ 1100 galaxy datacubes from the MaNGA dataset (with a shape of 96 x 96 x 4563) to infer the stellar velocity dispersion map of each galaxy (96 x 96). We split the dataset using an 90/10 train-test split.

The model is trained using a mean absolute error (MAE) loss on the training dataset with the Adam optimizer [84] for 10 epochs. The $R^2$ performance of the model on the held-out test dataset is 0.78, which drops to 0.35 when zero pixels are not considered in the calculation. The reconstruction of the 2D stellar velocity dispersion map for four random galaxies from the test set is shown in Figure 8.
<figure><img src="image_78.png" /><figcaption>Figure 8: Reconstruction of the stellar velocity dispersion maps by the U-net using the hyperspectral datacubes as input for four galaxies from the MaNGA test set.</figcaption></figure>

## C Infrastructure

To handle data at the 100TB scale, key decisions around data storage were made to accommodate both I/O and flexibility requirements. The data is stored in HDF5 files for consistency and performance. To allow efficient data access across various file systems, the samples are partitioned into HEALPix<fnref n="36" /> regions. This enables simple spatial indexing for objects within similar regions (constrained by the grid size). This partitioning also allows downstream users to download only samples pertaining to specific regions or to match samples across different MULTIMODAL UNIVERSE modalities or datasets. Finally, the data is chunked into HDF5 files with a variable number of samples to prevent individual files from becoming too large.

### C.1 Cross Matching Utility

The MULTIMODAL UNIVERSE framework provides utilities for cross-matching between different astronomical datasets. Below we demonstrate the simplicity of performing cross-matching operations using our provided tools:

```python
from datasets import load_datasetBuilder
from mmu.utils import cross_match_datasetS

# Load the dataset descriptions from local copy of the data
sdss = load_datasetBuilder("MultimodalUniverse/sdss",
                          trust_remote_code=True)
hsc = load_datasetBuilder("MultimodalUniverse/hsc",
                          trust_remote_code=True)

# Use the cross matching utility to return a new HF dataset
dset = cross_match_datasetS(
    sdss,  # Left dataset, contains spectroscopy
    hsc,   # Right dataset, contains imaging data
    matchingradius=1.0, # Distance in arcsec
)
```

### C.2 Contributing New Datasets

Detailed guidelines for contributing new datasets to the MultimodalUniverse<fnref n="37" /> framework are provided in the project's DESIGN.md and CONTRIBUTING.md documentation. These

<footnote n="36">³⁶https://healpix.sourceforge.io/</footnote>

<footnote n="37">³⁷https://github.com/MultimodalUniverse/MultimodalUniverse/</footnote>
documents are regularly updated with new tutorials and examples to facilitate community
contributions.

For the most up-to-date examples and tutorials, users can refer to our getting started
notebook<fnref n="38" />.

<footnote n="38">³⁸https://colab.research.google.com/github/MultimodalUniverse/MultimodalUniverse/
blob/main/notebooks/getting_started.ipynb</footnote>