PAPERCLIP: Associating Astronomical Observations and Natural Language with Multi-Modal Models
Authors: Siddharth Mishra-Sharma
YIDING SONG
Jesse Thaler
Venue: COLM (2024)
Keywords: CLIP, astronomy, astrophysics, physics, science, space, telescopes, vision, language, scientific discovery, contrastive learning, guided generation, data curation, fine tuning, foundation models
TL;DR: We develop a method to associate astronomical images observed by telescopes with natural language in a common, semantically informative embedding space.

Abstract: We present PAPERCLIP (Proposal Abstracts Provide an Effective Representation for Contrastive Language-Image Pre-training), a method which associates astronomical observations imaged by telescopes with natural language using a neural network model. The model is fine-tuned from a pre-trained Contrastive Language-Image Pre-training (CLIP) model using successful observing proposal abstracts and corresponding downstream observations, with the abstracts optionally summarized via guided generation using large language models (LLMs). Using observations from the Hubble Space Telescope (HST) as an example, we show that the fine-tuned model embodies a meaningful joint representation between observations and natural language through tests targeting image retrieval (i.e., finding the most relevant observations using natural language queries) and description retrieval (i.e., querying for astrophysical object classes and use cases most relevant to a given observation). Our study demonstrates the potential for using generalist foundation models rather than task-specific models for interacting with astronomical data by leveraging text as an interface.

Published as a conference paper at COLM 2024
PAPERCLIP:
Associating Astronomical Observations and Natural Language with Multi-Modal Models
Siddharth Mishra-Sharma, Yiding Song, & Jesse Thaler
MIT & IAIFI
"smsharm, e /emit.edu
Abstract
We present PAPERCLIP (Proposal Abstracts Provide an Effective Repre- sentation for Contrastive Language-Image Pree-raining), a method which associates astronomical observations imaged by telescopes with natural language using a neural network model. The model fine-tuned from a pre-trained Contrastive Language-Iman Pre-training (CLIP) model using successful observing proposal abstracts and corresponding downstream ob- servations, with the abstracts optionally summarized via guided generation using large language models (LLMs). Using observations from the Hubble Space Telescope (HST)
example, we show that the fine-tuned model embodies a meaningful joint representation between observations and nat- ural language through quantitative evaluation well as tests targeting image retrieval (i.e., finding the most relevant observations using natural language queries). and description retrieval (i.e., querying for astrophysical object classes and use cases most relevant to given observation). Our study demonstrates the potential for using generalist foundation models rather than task-specific models for interacting with astronomical data by leveraging text as an interface.
1 Introduction
Machine learning (ML) is starting to have significant impact in the sciences, with astro- physics being no exception. ML methods have demonstrated promise every stage of the research pipeline, from instrument design, to data acquisition, to its analysis (Huertas- Company & Lanusse, 2022). Most applications of ML within astrophysics have focused augmenting traditional techniques in order to improve performance specific tasks. The foundation model paradigm, contrast, seeks to develop generalist models which can be deployed to simultaneously tackle wide range of tasks (Bommasani et al., 2021). These models are typically pre-trained on massive amounts of unlabeled data using self- supervised or weakly-supervised learning techniques, enabling them to learn powerful representations which can then be used downstream. Foundation models can often benefit from additional training tunen) using a relatively small amount of domain-specific data in order to increase their usefulness when applied to specialized domains.
There is considerable interest in developing custom foundation models for the sciences (e.g., Batatia et al., 2023; Subramanian et al., 2023; McCabe et al., 2023; Birk et al., 2024; Vig al., 2024; Heinrich al., 2024), with astrophysics being ripe for such an effort given the large amounts of publicly-available data and diverse ways of interacting with it. The multi-modality inherent to astrophysical observations, with different types of data (e-g., images, spectra, light curves, textual descriptions) often available for given target object, presents a unique opportunity.
In this paper, we describe PAPERCLIP (Proposal Abstracts Provide an Effective Represen- tation for Contrastive Language-Image Pre-training), a method that connects, for the first time, astronomical image observations with natural language by leveraging the association between abstracts of successful observing proposals written by astronomers and images corresponding to downstream observations imaged by telescopes. This approach demon-
languaae

---

Published as a conference paper at COLM 2024
CLIP fine-tuning
Downstream task: observation retrieval
With Nubble obscrvation proposal abstract pairs
Gieen natural language text query
Quey: "berred optrat gainey"
Mobble proposal
Auttrnt Catiiees
anstrssed 14B grecation
Text
Dragayy now 1ile sagen
prepee
cowe mistar
ta dieia dwn
Tist
entoehee
Lungs
monder
Hobble olsevvatios
Lunge
cucodar
Closs ulserstiuu
Chorhos enchadiing
Figure 1: Overview of the PAPERCLIP method. (Left) A pre-trained CLIP model is fine- tuned using dataset of Hubble observations and corresponding proposal abstracts. The proposal abstracts are optionally summarized using guided large language model gen- eration. (Right) The fine-tuned model can then be used for downstream tasks such as observation retrieval i.e, finding the observations most relevant to given text query. The proposal abstract snippet shown here corresponds to proposal ID 16914.
strates the potential of adapting generalist multi-modal foundation models to astronomy, complementing task-specific models by providing a flexible, language-based interface for interacting with observational data. Concretely, we showcase the method using observa- tions imaged by the Hubble Space Telescope (HST). We show that fine-tuning pre-trained CLIP (Contrastive Language-Image Pre-training; Radford et al., 2021) image-text model on Abservation-absteen pairs results in meaningful joint representations through quantitative and qualitative evaluation tests. Our method opens up the possibility of interacting with interface, which is cor-
astronomical survey data using free-form natural language nerstone of the success of the modern foundation model paradigm. A high-level overview of the method is shown in Fig.
Related Work The concept of learning task-agnostic representations via self-supervised and contrastive learning has been applied within astrophysics (Slijepcevic et al., 2024; Stein et al., 2021; Hayat et al., 2021b; Slijepcevic et al., 2022) and used for downstream tasks like object similarity search (Stein et al., 2021), gravitational lens finding (Stein et al., 2022), estimation of Galactic distances (Hayat et al., 2021a), identification of rare galaxies (Walmsley & Scaife, 2023), and data compression (Akhmetzhanova et al., 2024). For a recent review of contrastive learning in astrophysics, see Cooeraas-Compayy et al. (2023). Beyond applications to a single modality, ASTROCLIP (Lanusse et al., 2023) recently used contrastive learning to learn a joint representation between galaxy images and associated spectra, showing that the learned representation embodies relevant physical properties and can be effectively used for downstream tasks like redshift and mass estimation. Bowles et al. (2023; 2022) introduced method to associate radio galaxy images with natural language description of their morphology by using human- uuman-generated descriptions, with the goal of deriving semantic morphology classes and using them for classification. In contrast with previous work, our application is the first to associate astronomical observation with the text modality in task-agnostic manner, showcasing the potential of language models in specialized scientific domains like astronomy.
The rest of this paper is organized follows. In Sec. 2, describe the Hubble dataset used in this work, including the curation and processing of observations as well as text captions. In Sec. 3, describe the methodology used to train and evaluate the model. In Sec.
2

---

Published as a conference paper at COLM 2024
present quantitative and qualitative results of our experiments retrieval tasks. We discuss future prospects and conclude in Sec.
2 Dataset Construction
We curate dataset of Hubble Space Telescope (HST) image observations and corresponding text descriptions from publicly available sources. We rely on proposal abstracts from the Proposal Abstracts Catalog' catalog of successful HST proposals to generate captions for the observations, optionally summarizing them via guided generation using L.LMs (described in Sec. 2.2 below). The HST has been operational since its launch on April 24, 1990, and we use available proposals and observations up to the Cycle 30 science program, which commenced data-taking in 2022.
Table shows examples of images and their corresponding (clipped) proposal abstracts. can be seen that the images in this dataset exhibit specific characteristics as well as artifacts particular to HST data-taking and processing which distinguishes them from the distribution of natural images typically used for large-scale pre-training of foundation models. This further motivates fine-tuning on domain-speciff data.
Hubble image
Obs. cycle (Year)
Prop. ID
Proposal abstract (clipped)
Category: STELLAR EJECTA. We propose the WFPC2 and STIS CCD to obtain maximum spa- tial resolution emission-line images young, oxygen- rich supernova remnants SN0540069.3 in the LMC B0102.2-
7 (1999)
7340
the SMC.
ambda5007, S III ambdal ambda6724 and O III amb- dal ambda277772 images SN0540-692 be used to characterize the ionization structure and...
Category: HOT STARS. Type supernovae (SN Ia) have enormous importance to cosmology and astrophysics, but their progenitors and explosion mechanisms are not known in detail. Recently, ab- servations and theoretical models have suggested that not all thermonuclear white-dwar warf supemova explosions are supernovae (peculiar cousins to SN la), are...
22
13757
(2016)
particular, type
Table 1: Examples of Hubble images (left-most column) and corresponding clipped proposal abstracts (right-most column). The observation cycle and corresponding year, as well as proposal ID, are shown in the second and third columns, respectively. The proposal IDs link to the Mikulski Archive for Space Telescopes (MAST) page corresponding to the proposal.
2.1 Hubble Data Selection and Pre-processing
Observations corresponding to individual proposal IDs are queried through the Mikulski Archive for Space Telescopes (MAST)² via the Astroquery (Ginsburg et al., 2019) APL Products of type PREVIEW are filtered in, corresponding to preview postcard images. We note that these are not science-grade observations, but rather lower-resolution images useful for diagnostic or preview purposes. A maximum of 20 images are downloaded per proposal ID, selected at random, in order to avoid biasing the model towards proposals with larger number of observations and survey-style campaigns. Images are centered and resized to a resolution-per-side of 512 pixels. Color previews (i.e., observations taken with
Inttps://archiee.et http:/chww..teeeeeeece/////rrroo abstracts.
2nttps://mast.ttaet stsci, edu/
3

---

Published as a conference paper at COLM 2024
Prop. ID
LLMeextracted summary
Objects and phenomena
Science use cases
characterize ionization structure distribution chemically pe- culiar debris in SNU540-6933, determine ionization structure in the debris 80102.2-7221 provide benchmarks models nu- cleosynthesis massive excitation mechanisms extremely metal-rich plasmas, and supernova explosion dynamics, study the pulsar and synchrotron nebula 5N0540-69.3, investigate SN0540- 69.3 3's proximity to SN 1987A in both space and time, and relation the same extended complex young stars
young oxygen-rich superova remmants SN0540-69.3, I.MC, SMC, supernova debris, active pulsar, synchrotron nebula
7340
type lax supernovae, white dwarfs, possible companion stars, accretion disks, luminous blue stars
constrain progenitor systems of type lax supernovae, distinguish between explosion mechanisms, investigate mass transfer processes in accretion disks, determine if type lax supemovae originate from massive stars
13757
Table 2: For the Hubble proposal abstracts shown in Tab. the LLM (MIXTRAL-8X7B) extracted summaries showing objects and phenomena (middle column) as well as potential downstream science cases (last column) separately. The proposal IDs (left column) contain hyperlinks to the MAST page corresponding to the proposal.
multiple wavelength filters assigned to individual RGB channels) are manually excluded via filename filter in order to maintain consistency across the samples. If no appropriate images corresponding to an abstract are found, it is excluded from the dataset.
In total, 31 $1,859 images corresponding to 4,438 abstracts are included in the fine-tuning dataset. 3,194 images are held out for validation, with no abstract being common between training and validation sets in order to ensure independent set of image-text pairs for evaluation. The held out images correspond to 429 unique abstracts. Due to practical limitations associated with the small size of the fine-tuning dataset, we did not use different datasets for validation and testing, deeming the current approach sufficient for proof-of- principle exposition.
We note that some fraction of the image-caption pairs in the constructed dataset will primarily concern instrumentation and or calibration rather than scientific content. We choose to not filter out these pairs, in order to have a larger sample of HST observations that the model can leverage to adapt the distinctive characteristics of Hubble images.
2.2 Abstract Summarization via Guided Generation
Raw proposal abstracts summarize the corresponding successful HST observing proposals, which intend to make the case for allocating Hubble telescope time towards a particular set of observations. These abstracts are written in diversity of styles, formats, and lengths while also being highly variable in their content. Although the abstracts can be used as-is image captions, we experiment with summarizing them via guided large language model (LLM) generation to standardize the captions used for fine-tuning the CLIP model. Captions are summarized by extracting list of objects and phenomena, well potential downstream science use cases, corresponding to the eventual imaged observation. The intended goal of the summarization process is to increase the strength of the association signal between text and images.
The method from Willard & Louf (2023) is used to produce LLM-generated summary of the abstract conforming to a particular schema, specified in in JSON format. The schema is designed to represent list of the objects (e.g., 'Type la supernova') and phenomena
4

---

Published as a conference paper at COLM 2024
(e.g., 'gravitational lensing'), as well potential downstream science uses cases (e.g., 'set constraints on supernoaa explosion models') that could correspond to the eventual imaged observation given the abstract text, with minimum of 1 and maximum of 5 elements per list. The procedure guides the generation of LLM outputs while ensuring that the schema is respected at every step in the generation process by masking out tokens that would violate the intended format. By framing the problem in terms of transitions between a set of finite states (i.e., finite-state machine), Willard Louf (2023) showed that guided generation can be performed with negligible overhead compared to unconstrained generation. See App. A.1 for a more detailed description of the guidance generation method used here, including an overview of technical details. While the schema-guided generation ensures the format of the output, the prompt and choice of LLM will dictate the content of the generated summaries. We use the open-weights, instruction-tunat
(liang et al., 2024) to generate the summaries, with guided generation performed using the Outlines package. Further details on the summarization procedure, including the prompts and schema used, are provided in App. A.2.
model MIXTRAL 111-188--NSSRUUU
Examples of LLM-generated abstract summaries are shown in Tab. 2, for the same set of abstracts as shown in Tab. 1. We train separate models using the raw abstracts and the L.MM-gnnerted summaries, and compare their performance on downstream tasks in Sec. 4. We note that, even after summarization, the association signal is expected to be noisy, since parts of the summarized caption may not be directly descriptive of the observed images. The goal of the fine-tuning process to leverage the signal contained in this noisy association.
3 Methodology
3.1 Contrastive Language-Image Pre-training
Contrastive Language-Image Pre-training (CLIP; Radford et al., 2021) is a multi-modal neural network model pre-trained on a large corpus of image-text pairs via weak supervision using a contrastive loss. Given minibatch B of [B] image-text pairs {[, ], the goal is to align the learned representations of corresponding (positive) pairs (4, Ti) while repelling the representations of unaligned (negative) pairs (4, T,4;). Image and text encoders f + Re and g T -> Reme are used to map images and text to a common embedding space of dimension nmmb We use the standard bidirectional variant of the InfoNCE (Oord et al., 2018) contrastive loss function introduced for training CLIP-style architectures (Radford et al.,
1 2021), L(B) = 2|3 -=
log
log 0/y//
where x; (T)/III (TIII
and y/ g (T:) (Ti)l are the normalized representations of the i-th image and text caption, respectively, and is a learnable temperature hyperparameter. Note that this loss treats the image and text representations symmetrically, ensuring that the two modalities are considered on the same footing.
3.2 Fine-tuning Procedure
The base CLIP model is fine-tuned using the dataset described in Sec. using either the LLM-summarized abstracts or raw proposal abstracts paired with observations. When using raw proposal abstracts, random chunks of the text delimited by periods are selected on the fly to fit within the maximum token length of the text encoder. Images are augmented via random four-fold rotations (increments of 90%) and randomly cropped to the native resolution of the image encoder, maintaining ~ 20% of the area of the original image, at each training step. Given the relatively modest size of the fine-tuning dataset, batch size [B| = 32 is used throughout; larger batch sizes were observed to be susceptible to overfitting. The temperature hyperparameler T was initialized to its pre-trained value. We emphasize that the positive and negative image-text association is noisy and imperfect, since multiple images can be associated with the same abstract, and the goal of the fine- tine-tuning process is to leverage the signal contained in this noisy association.
hhttps: cww/out ineei dew.eeeeddeootlliiieeeee
5

---

Published as a conference paper at COLM 2024
We use the CLIP-ViT-B/16 (Radford et al., 2021) variant as the base pre-trained CLIP model. We explore three different methods of training the model on our domain dataset: (1) Fine- tuning the entire network starting from the pre-trained base model; (2) Freezing the base image/tex, text encoders and training a small projection head; and (3) Training the entire model from scratch. For (2), we 2-layer MLP with 1024 hidden units and GELU activation layer, projecting onto the 512-dimensional common embedding space. Additional details on the CLIP model and fine-tuning procedure are provided in App. B.
3.3 Evaluation Metrics
The model is evaluated by tracking the contrastive loss as well as the top-k% retrieval accuracy the held out validation set over the course of training. The retrieval accu- racy is defined as the fraction of associated captions (either raw or LLM-summarized abstracts) which fall within the top k% of captions by cosine similarity of the normal- ized image and caption embeddings, averaged over the images in the validation set:
k [ where [V] is the total number of images in the validation set, [-] is the indicator function that returns 1 if the condition inside the brackets is true and 0 otherwise, rank (x; yii {x; // is a function that returns the rank of the cosine similarity between x; and y; among the cosine similarities between x; and all captions yj in the validation set, and k is the percentage of top captions considered for the retrieval accuracy. Note that this metric is symmetric in the image and text modalities.
1 +M rank (x; yo (x: y/) <
We also qualitatively evaluate the learned embeddings through image retrieval (i.e., retriev- ing the most relevant images from the validation set using natural language queries).
Results and Discussion
4.1 Quantitative Evaluation
Validation metrics during training Figure 2 shows the contrastive loss (left) and the top-10% retrieval accuracy (right) evaluated on the held out validation set over the course of training, for different training configurations considered. The dashed orange lines show the metrics evaluated when training with batches where the image-text associations are randomly shuffled. This randomized baseline is seen do on par with random expectation (i.e., 10% retrieval accuracy), unlike the others, validating the presence of significant association signal between images and text the dataset. Interestingly, the base pre-trained model performs better than random expectation, with top-10% retrieval accuracy of ~ 15% (as see from the left-most datum in Fig. 2 right, for the curves corresponding to fine-tuned models). We therefore also compare the qualitative performance of the base model with the fine-tuned models on downstream retrieval tasks.
The model trained using LLM-summarized abstracts (red lines) is seen to perform slightly worse than the model using raw abstracts as captions (blue lines), despite the curation of the cummarizze-abstract dataset intended to provide stronger image-text association signal. Fine-tuning a small ML. head over frozen vision and text backbones (dotted green lines) and training from scratch with summarized abstracts as captions (yellow lines) show a non- trivial improvement compared to the base model, although with deteriorated performance compared to fine-tuning with either summarized or raw abstracts.
Distribution of text-image cosine similarities Figure (left) shows the distribution of cosine similarities between corresponding image and text embeddings, x; and y;, for the base CLIP model (purple line), and for the LLM-summarized abstracts using the fine-tuned CLIP model (red line). Distributions evaluated for a shuffled order of text embeddings therefore randomizing the image-text correspondence during evaluation - are shown as dashed lines. We note that the shuffling here is performed at the evaluation stage, and not the training stage. The distributions for the base model is seen to be sharply peaked at a specific value, showing little diversity and being very similar between the shuffled (dashed
6

---

Published as a conference paper at COLM 2024
Validation metrics over training
Validation loss
1op-10% 10% retrieval
0.8
4.6
0.7
M 7 w
4.7
0.6
Retrieval antranacy
Valiiation lee
4.0
0.5
Staffi wathin batch
0.4
(ssmmurtie)) Fianskaue hoad (umuuzii))
3.5
Fize-tate (stumezaa)
Firestate (oumusaie))
Fiue-tane lasa (sumuizzo))
34
(aleteac)
Trun from (esmmuune)
Streffe vilin harch
3.2
Rundeck esplctatios
3.0
5000
10000
1500
200000
5000
10000
1500
2000
Training steps
Training stege
Figure 2: The CLIP contrastive loss (left) and the top-10% retrieval accuracy (right) computed on the validation set over the course of training. Shown for the dataset with summarized abstracts as captions (red), dataset using raw proposal abstracts as captions (blue), only fine- tuning small MLP head (dotted green), training from scratch with summarized abstracts as captions (yellow), and trained with shuffled image-text pairs (dashed orange).
purple) and non-shuffled (solid purple) versions. Distributions for the fine-tuned model, on the other hand, show a clear separation when evaluated on shuffled (dashed red) and corresponding (solid red) text-image pairs.
Retrieval accuracy Figure 3 (right) shows the retrieval accuracy as a function of the re- trieval fraction k%. In this case, we evaluate all four models (fine-tuned on raw abstracts (blue), fine-tuned on L.M--ummarized abstracts (red), trained on LLM-summarized ab- stracts from scratch (yellow), and the base model (purple)) on the same captions dataset- the summarized abstracts - for direct comparison. Remarkably, the model trained on raw abstracts shows very similar performance when evaluated on the summarized abstracts compared to that trained on the summarized abstracts themselves, indicating that (1) the image-text association signal is preserved in the summarization process, and (2) the model is able to effectively leverage meaningful concepts in the noisy raw abstracts through weak su- pervision. The significantly worse performance of the model trained from scratch, compared to the fine-tuned models, highlights the crucial role of the inductive bias inherited from the base pre-trained model, which effectively captures rich associations between images and language.
We show retrieval accuracy performance for additional variations on the model and training configuration in App. C.
4.2 Image Retrieval
Having aligned the image and text representations, we can embed natural language query using the model and show the closest images by embedding from the validation set when ranked by cosine similarity. A sketch of this procedure is shown in Fig. (right). We show these in Tabs. 3 and 4 for the base and fine-tuned models respectively using two simple curated queries: Jupiter and SN1987A (a specific supernova). The proposal ID corresponding to the retrieved images is shown below each image, and contains hyperlink to the MAST page corresponding to the proposal for further details.
While the base model shows some signs of meaningful retrieval (e-g., the image of Jupiter in the first row of Tab. 3), associations between the retrieved images and corresponding queries are not consistent.
The model fine-tuned with summarized abstracts, meanwhile, shows strikingly different behavior (Tab. 4). Images looking like Jupiter are returned for the Jupiter query. However,
7

---

Published as a conference paper at COLM 2024
Casine Similarities (Summaarined Abstracts)
Retrieval Accuracy on Sunmarized Abetracts
Hase (jucoenst pains;
Dase (onttect peins)
1730
Finestned (incermet pains)
0,8
1500
Fine-tened (oureet
Retrivaal accunery
1000
0.0
730
Finns (eummarin))
0.2
Fine tuze (alarract)
250
Than from wratch (ammmrted))
0.0
Retricual fraction
Figure 3: (Left) Distribution of cosine similarities between corresponding image and text embeddings, x; and y/, shown when using the base CLIP model (purple lines), and the summary fine-tuned CLIP model (red line). Dashed lines correspond to models evaluated on image-text pairs with associations shuffled. (Right) Retrieval accuracy as a function of the retrieval fraction k for the fine-tuned model on the summarized abstracts (red), fine-tuned on raw abstracts (blue), trained on summarized abstracts from scratch (yellow), and the base model (purple).
this example also illustrates the model's potential to misidentify objects, with the first and third image actually showing Saturn with artifacts on the planet and partially obscured rings. Supernova SN1987 itself can be seen in the three closest images for the SN1987A query.
We also evaluate the observation retrieval task more quantitatively. We design a prompt which lets us evaluate whether the abstract corresponding to retrieved observation is relevant or not, with the output constrained to be boolean using Outlines. We then evaluated this prompt, for the base as well as fine-tuned models, on the top 10 closest images by cosine similarity returned for 10 different queries. 38% and 77% of the retrieved observations are deemed relevant when using the base and fine-tuned models, respectively. The fine-tuned model thus significantly more likely to return images relevant to the query. The prompt and curated queries for this test are described in App. A.3.
Note that we chose to illustrate qualitative performance on image retrieval using the model fine-tuned on summarized abstracts, rather than raw abstracts. Although the two models show very similar quantitative performance on retrieval metrics (as shown in Fig. 3), they exhibit characteristically different behaviors in terms of images retrieved, with the summary fine-tuned models generally retrieving images that look more visually "relevant" to domain expert. We emphasize that for scientific usefulness, the goal is not necessarily to correctly retrieve the most relevant objects, but rather to identify diverse set of interesting candidates for manual follow-up and further analysis. By diverse, we mean that retrieved observations may contain different types of objects or phenomena, which may be relevant to the query in distinct ways.
The fine-tuned model can similarly be used for description / text retrieval, akin to the traditional zero-shot classification setting, where the closest text snippets from a curated list are returned given an observed astronomical image. We show examples of the text retrieval task App. D.
Outlook and Conclusions
We present PAPERCLIP, a method for training domain-specific multi-modal models for astrophysics that associates observations imaged by telescopes with natural language in common embedding space. We showcase application to Hubble Space Telescope (HST) observations, where the model fine-tuned from a pre-trained CLIP model using abstracts of successful Hubble proposals, optionally summarized, leveraging noisy association
8

---

Published as a conference paper at COLM 2024
Query
Top-3 most similar images using base off-the-shelf CLIP model
Jupiter
6028
10170
10170
SN1 987A
13830
15475
14594
Table 3: For two text queries (left-most column), the three most similar images from the validation dataset by cosine similarity when using the base (off-the-shelf) CLIP model (CLIP-ViT-B/16) The proposal ID associated with each image is given below the image and contains hyperlink to the MAST page corresponding to the proposal.
Query
Top-3 most similar images using summary fine-tuned CLIP model
Jupiter
11956
6028
11956
SN1 987A
11653
11653
8648
Table 4: Same as Tab. 3, but using the summary fine-tuned CLIP model.
signal between text and images. We show that PAPERCLIP significantly outperforms the base CLIP model quantitative metrics, such as retrieval accuracy, as well as quality of text-to-image and image-to-text retrieval. We also introduce novel LLM summarization process which leverages guided generation to distill the content of proposal abstracts while preserving salient information. Overall, the procedure demonstrates the efficacy of fine- tuning generalist pre-trained models on small amounts of domain-specific data, in particular astronomical datasets, and leveraging text
interface for interacting with the data.
Although the model explored here is fine-tuned using postage stamp images (i.e., preview- quality and not science-grade data), we highlight potential immediate well downstream

---

Published as a conference paper at COLM 2024
use cases. A model trained using suaely-s image-text pairs can be used to query large amounts of unlabeled survey data egg, PHANGS (Lee et al., 2022), COSMOS et al., 2007) for objects or use-cases of interest using natural language, well as to efficiently find patterns in such data that may not be apparent using specialized models or manual inspection. The learned representations, having shown to correlate with physical charac- teristics of imaged objects, can also be fine-tuned via transfer learning to adapt to either specific tasks e.g,, classification (Wei et al., 2020) or segmentation (Hausen & Robertson, 2020), or observations imaged by other telescopes.
Finally, while the CLIP model is restricted retrieving nearest-neighbour associations within and across text eoxt/image modalities, the learned embeddings can be used as a starting point for training or fine-tuning multi-modal large-language models for interacting with survey data and receiving responses in natural language form, as well as grounding the responses based on an existing set of observations.
Acknowledgments
We thank Michael Brenner, François Lanusse, and Julian Muñoz for helpful conversations. This work is supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF Al Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.ors
This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of High Energy Physics of U.S. Department of Energy under grant Contract Number DE-SC0012566
YS was supported by the Research Science w "Interpretation of Multimodal Images from Astronomy" This research was supported by the Munich Institute for Astro-, Particle and BioPhysics (MIAPbP'), which funded by the (DFG, German Research Foundation) under Germany's Excellence Strategy -XXC-2094 390783311. The computations this paper were run on the FASRC Cannon cluster supported by the FAS Division of Science Research Computing Group at Harvard University.
Institute (RST) program at MIT. This research was supported by an award from Google,
Deutsche Forschungsgemeinschane
This research is based on observations made with the NASA/ESA Hubble Space Telescope obtained from the Space Telescope Science Institute, which is operated by the Associa- tion of Universities for Research in Astronomy, Inc., under NASÀ contract NAS 5-26555. Based on observations made with the NASA/ ESA Hubble Space Telescope, and obtained from the Hubble Legacy Archive, which collaboration between the Space Telescope Science Institute (STScI/NAL) NASA), the Space Telescope European Coordinating Facility (ST- ECF/ES (C//ESAC/ESA) and the Canadian Astronomy Data Centre (CADC/NRC/CSA),
This work relied on the Astroquery (Ginsburg et al., 2019), BitsAndBytes (Dettmers et al., 2022), Flax (Heek al., 2023), Jax (Bradbury al., 2018), Jupyter (Kluyver et al., 2016), Matplotlib (Hunter, 2007), Numpy (Harris et al., 2020), Optax (Babuschkin et al., 2020), Outlines, Pandas (Virtanen et al., 2020), Pydantic, PyTorch (Paszke et al., 2019), SciPy (Virtanen et al., 2020), Transformers (Wolf et al., 2019), and Wandb (Biewald, 2020) software packages.
Reproducibility Statement
Code used to reproduce the results in this work is available at https://github. com/ Mishharmm//AAEERRLI---tbbblleee
Ethics Statement
This work relies on using abstracts from successful Hubble Space Telescope observing proposals as part of a dataset for training and evaluating machine learning models. While these abstracts are publicly available, the authors likely did not anticipate their text being used in this manner, raising questions around consent, attribution, and appropriate use of data. Since this research intends to develop methods to aid astronomical research and does not use sensitive personal information or target commercial gain, we believe that the scientific benefits outweigh the potential concerns in this case, while acknowledging good- faith arguments to the contrary. As the use of foundation models in the sciences increases,
10

---

Published as a conference paper at COLM 2024
it will be important for the community to consider norms and guidelines around the appropriate use and attribution of various data sources for model training and evaluation, including qualitative textual data, to ensure transparency and maintain trust.
References
Aizhan Akhmetzhanova, Siddharth Mishra-Sharma, and Cora Dvorkin. Data compression and inference in cosmology with self-supervised machine learning. Monthly Notices of the Royal Astronomical Society, 527(3):7459-745) -4481 2024.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Turii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http://ttt: thub. com/ com/deepmind.
Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M Elena, Dávid P Kovács, Janosh Riebesell, Xavier R Advincula, Mark Asta, William J Baldwin, Noam Bernstein, et al. A foundation model for atomistic materials chemistry. ar Xiv preprint ar XiX::22401.000 1000000 2023.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.
wandb. com/. Software available from wandb.com.
Joschka Birk, Anna Hallin, and Gregor Kasieczka. Omnijet-a: The first cross-task foundation model for particle physics. 3 2024.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint ar ftte:2008.02554 2021.
Micah Bowles, Hongming Tang, Eleni Vardoulaki, Emma L Alexander, Yan Luo, Lawrence Rudnick, Mike Walmsley, Fiona Porter, Anna MM Scaife, Inigo Val Slijepcevic, et al. A new task: Deriving semantic class targets for the physical sciences. ar preprint ar WxXi:2210.14760, 2022.
Micah Bowles, Hongming Tang, Eleni Vardoulaki, Emma L Alexander, Yan Luo, Lawrence Rudnick, Mike Walmsley, Fiona Porter, Anna MM Scaife, Inigo Val Slijepcevic, et al. Radio galaxy zoo emu: towards semantic radio galaxy morphology taxonomy. Monthly Notices of the Royal Astronomical Society, 522(2):2584 2600, 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github. com/ wwt.coooooooooeecc google/jox.....o
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. ar preprint ar arXio:2208.07339 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint ar Xr1:20000011192, 2020.
A. Ginsburg, B. M. Sipöcz, C. Brasseur, P. S. Cowperthwaite, M. W. Craig, C. Deil, J. Guillochon, G. Guzman, S. Liedtke, P. Lian Lim, K. E. Lockhart, M. Mommert, B. M. Morris, H. Norman, M. Parikh, M. V. Persson, T. P. Robitaille, J.-C. Segovia, L. P. Singer, E. J. Tollerud, M. de Val-Borro, I. Valtchanov, J. Woillez, The Astroquery collaboration, and subset of the astropy collaboration. astroquery: An Astronomical Web-querying Package in Python. Astrophysical Journal, 157:98, March 2019. doi: 10.3847 018847/1534 /1538//11133333 3881 /aff333.
11

---

Published as a conference paper at COLM 2024
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Vir- tanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Na- fure, 585(7825) September 2020. doi: 10.1038/ (00008/$4156-0000 9411886-220--2699-222 URL https: //doi. orw..0.g/11.. 10000/1188858888 0008866020-264-0
Ryan Hausen and Brant E Robertson. Morpheus: A deep learning framework for the pixel-level analysis of astronomical image data. The Astrophysical Journal Supplement Series, 248(1):20, 2020.
Md Abul Hayat, Peter Harrington, George Stein, Zarija Lukić, and Mustafa Mustafa. Es- timating galactic distances from images using self-supervised representation learning. ar arviv preprint xxXiv:2101.04293 2021a.
Md Abul Hayat, George Stein, Peter Harrington, Zarija Lukić, and Mustafa Mustafa. Self- supervised representation learning for astronomical images. The Astrophysical Journal Letters, 911 2021b.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http://github comm/oogle//fer
Lukas Heinrich, Michael Kagan, Samuel Klein, Matthew Leigh, Tobias Golling, John An- drew Raine, and Margarita Osadchy. Masked particle modeling on sets: Towards self- supervised high energy physics foundation models. arXio preprint ar arXiv:2401.13537, 2024.
Kagan,
Marc Huertas- Sompanyy learning for the analysis of galaxy surveys. arXiv preprint 2rXi::2200001813, 2022.
Cooerpay- and François Lanusse. The dawes review 10: The impact of deep
Marc Huertas-Company, Regina Sarmiento, and Johan H Knapen. A brief review of con- trastive learning applied to astrophysics. RAS Techniques and Instruments, 2(1):441-452, 2023.
applied
D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, (3)990-95, 2007. doi: 10.1109 MM010///2..2000.555...
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arv preprint ar XrXi:2440100008 2024.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv..n org. abs//bs//// /et.//111..89.
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Busson- nier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing. Jupyter notebooks format for reproducible computational workflows. In Loizides and B. Schmidt (eds.), Positioning and Power in Academic Publishing: Players, Agents and Agendas, pp. 87 90. IOS Press, 2016.
publishing
Francois Lanusse, Liam Parker, Siavash Golkar, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Ruben Ohana, Mariel Pettee, et al. Astroclip: Cross-modal pre-training for astronomical foundation models. arviv preprint arXio:2310.0302, 2023.
12

---

Published as a conference paper at COLM 2024
Janice C Lee, Bradley C Whhtmore,, David A Thilker, Sinan Deger, Kirsten L Larson, Leonardo Ubeda, Gagandeep S Anand, Médéric Boquien, Rupali Chandar, Daniel A Dale, et al. The phangs-hst survey: Physics at high angular resolution in nearby galaxies with the hubble space telescope. The Astrophysical Journal Supplement Series, 258(1):10, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th Interna- tional Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: https:://pperevieew http:///peeeeeeeeeeeee///on/aiia weet/forummid-Bkg@RRiiie
Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models. arXiv preprint arXiv:2310.02994, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive codiing. ariiv preprint arXit:1807.03777 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, Pp. 8748-8763. PMLR, 2021.
Nick Scoville, H Aussel, Marcella Brusa, Peter Capak, C Marcella Carollo, M Elvis, M Gi- avalisco, L Guzzo, Hasinger, C Impey, et al. The cosmic evolution survey (cosmos): overview. The Astroplnysiaal Journal Supplement Series, 172(1):1, 2007.
Inigo V Slijepcevic, Anna MM Scaife, Mike Walmsley, Micah Bowles, O Ivy Wong, Stanislav S Shabala, and Sarah V White. Radio galaxy zoo: towards building the first multipurpose foundation model for radio astronomy with self-supervised learning. RAS Techniques and Instruments, 311:119-3,, 2024.
Inigo Val Slijepcevic, Anna MM Scaife, Mike Walmsley, and Micah Bowles. Learning useful representations for radio astronomy" in the wild" with contrastive learning. ar arviv preprint arXiv:2007.08666 2022.
George Stein, Peter Harrington, Jacqueline Blaum, Tomislav Medan, and Zarija Lukic. Self- supervised similarity search for large scientific datasets. arXio preprint arXi::2110.13151, 2021.
George Stein, Jacqueline Blaum, Peter Harrington, Tomislav Medan, and Zarija Lukić. Mining for strong gravitational lenses with self-supervised learning. The Astrophysical Journal, 932(2):107, 2022.
Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. ar Xiv preprint ar arXi::2306.00258, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Matthias Vig, Nicole Hartman, and Lukas Heinrich. Finetuning foundation models for joint analysis optimization. arXiv preprini ar Xix::22401. 13536,, 2024.
13

---

Published as a conference paper at COLM 2024
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cour- napeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, An- drew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, īlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, I'aul van Mulbregt, and SciPy 1.0 Contributors. Scily 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10.1038/ 000088/$41550 141592-019-066-2.
Mike Walmsley and Anna MM Scaife. Rare galaxy classes identified in foundation model representations. Xrv preprini ar arXiv:2312.00911 2023.
Wei Wei, EA Huerta, Bradley C Whitmore, Janice C Lee, Stephen Hannon, Rupali Chandar, Daniel A Dale, Kirsten L Larson, David A Thilker, Leonardo Ubeda, al. Deep transfer learning for star cluster classification: I. application to the phangs-hst survey. Monthly Notices of the Royal Astronomical Society, 493(3):317-3319, 2020.
Brandon T Willard and Rémi Louf. Efficient guided generation for 1lms. arxi preprint WwXit:2307.09702 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
A Details on the Abstract Summarization Procedure
We provide additional details of the abstract summarization procedure, including a brief overview of the guided generation method used, well as the prompts and schema used for the summarization task.
A.1 Guided LLM Generation with Outlines
As mention in Sec. 2.2, we employ the guided generation method introduced by Willard Louf (2023) and implemented in Outlines to ensure that the LLM summarization of the raw proposal abstracts adheres to specific pattern, specified in JSON format (Sec. A.2 below), which we briefly describe here. This approach represents the desired output format finite-state machine (FSM) that encodes the JSON schema as regular expression. The JSON schema constraint is therefore first converted into regular expression.
The key idea then is to pre-compute an index that maps each state of the FSM to the subset of tokens from the LLM's vocabulary that can be generated from that state while still allowing for a valid completion of the pattern. By doing so, we can efficiently determine the valid next tokens at each step of the generation process without having to check the entire vocabulary.
Formally, let M = (Q, Σ, 6, 90, F) be the FSM representing the regular expression, where Q the set of states, Σ is the alphabet of the regular expression, d: Q Σ - Q is the transition function between states, 4o is the start state, and F C Q is the set of accept states which terminate the generation. An index o Q + P(V) is first constructed, where V is the LLM's token vocabulary and PVV) denotes the power set of V. For each state q Q (q)) contains the allowed tokens that can be generated from state q while maintaining the possibility of reaching accept state. The construction of o involves finding all token sequences that, when processed by the FSM starting from each state 9, lead to an accept state.
During the sequential generation process, the current FSM state 4: is kept track of after sampling each token vy. At each step f, the LLM's output logits are masked based on the valid next tokens (q:),, setting the logits of invalid tokens to cco The next token is then sampled from the categorical distribution defined by the unmasked logits, and the FSM transitions to the next state 4t = 5(9,, D1 where D Σ is the token in the regular
14

---

Published as a conference paper at COLM 2024
expression alphabet corresponding to the sampled token. This process continues until accept state with no outgoing transitions is reached, indicating a valid completion of the pattern.
A.2 Prompts and Schema Used for Summarization
We list here the prompts and schema (i.e., desired output formats) used for guided text generation via Outlines package interfacing with the MIXTRA--XX7-INSTRRCCT open- weights large language model.
The following schema, specified using the data-validation package Pydantic, is used to guide the generation of the summaries, intended to produce between one and five objects and hypotheses, as well as science use cases, given raw proposal abstract. Both fields are of type conlist, Pydantic type that represents constrained list.
from pydantic import BaseModel, conlist
2
class ConstraiindRResooneeiSTT (esrspp (BaseModel):
pad__phnonnenna: conlist(str, min min_length=1, max_length=5) science_use_casee conlist(str, min_length=1, max_length=5)
The following prompt function is used to produce list of one to five possible objects and phenomena shown in HST observations downstream of proposal abstract, well as one to five possible science use cases, in the format native to Outlines. "<>>
" and " "
" are start and end instruction delimiters, respectively, for the MIXTRAL-6X7B model.
import outlines
poutlines..rompt
prompt_fn(abstract)
"""Cs>
 You are an expert astrophysicist, with broad expertise across observational and theoretical astrophysics. You are able to extract core information from astrophysical texts.
7 Abstract: "((abstract))"
Based the above observational proposal abstract, your task is to summarize the nature of the eventual observations. You will identify the astrophysical objects phenomena, as well as potential science use cases described in the abstract.
10
Follow these instructions exactly: Mention up items for both categories; do not mention more than itees either category.
12
13
Choase the most relevant ones if there are more than 5 items in category.
Never mention the Hubble Space Telescope, HST, the HST archive. Mention the class (e.g., barred spiral galaxy) and not just the specific instance (e.g., Andromeda).
16
Name the objects in the science use cases, if appropriate.
17
Write out full names of objects in addition to acronyms.
Do not list irrelevant objects which do not describe the eventual observation, such units proposal Cycle numbers. List fewer but more relevant objects if in doubt.
19 Each science case listed must be self-contained but succinct.
Only write in English.
not list items that
generic (e.g., galaxy, faint object, kinematics)
total length text should exceed 80 words.
23 Present your lists in comma- semar-eep format; no dashed or numbered lists.
Example output: ('objects_and__gg phe_ppppeeeenaa 'spiral galaxies, galaxy clusters, supernova remnants'. 'science _se_eases_eeeeeeeeeeeeeeeee 'mdeel galactic structure and
15

---

Published as a conference paper at COLM 2024
evolution, characterize dark matter distribution in clusters, analyze expansion rates of supernova remnants }
26
27
Answer in JSON format. The JSON should be a dictionary with keys objjetts_and__pennmeen and "science_use_casee


30
A.3 Prompt Used for Quantitative Evaluation of Observation Retrieval
The following prompt was used to evaluate the relevance of abstracts corresponding to retrieved images to query, when quantitatively assessing the observation retrieval task.
import outlines
3
@outlines, @outlines.prompt prompt
def prompt_fn (abbttatc, query):
"""

expert astrophysicist, with broad expertise observational and theoretical astrophysics.
7
Abstract: "((abstract))"
Query: "((query))"
10
11
The above is an abstract for a proposed observation taken by the Hubble Space Telescope (labeled "Abstract"), and an object or concept (labeled "Query").
Could the observations corresponding the abstract contain the query? Be precise, and do not contain related concepts or objects.
14
15
Your response should be either True or False. Only return True if the query is closely related the abstract, and the downstream observation could be relevant to the query.
16


17
The queries used in the evaluation ["globular cluster", "dwarf galaxy", "SN1987A", "strong lensing", "galaxy clusters", "interstellar medium", "dark matter", "spiral galaxies", "lyman alpha", "comets"].
B Additional Model and Training Details
We use the CLIP-ViT-B/16 (Radford et al., 2021) variant the base pre-trained CLIP model. This model uses a 12-layer, 12-head, 768-embedding dimension vision transformer with patch size 16 x 16 as the image encoder (Dosovitskiy et al., 2020) and 12-layer, 8-head, 512-embedding dimension text sequence transformer as the text backbone (Vaswani et al., 2017). The text encoder has maximum length of 77 tokens and the image encoder has native resolution of 224 x 224 pixels. Linear projection layers map the outputs of the image and text encoders
common embedding space of dimension Hemb 512. In total, the model contains ~ 149 million trainable parameters. This model was originally pre-trained on ~ 400 million tmate-text pairs from internet data.
All models were trained over 20,000 steps with 2000 linear warmup steps using the AdamW optimizer (Loshchilov & Hutter, 2019; Kingma & Ba, 2015) with learning rate 10-5 and weight decay 10-3, Training takes approximately 3 hours on Nvidia A100 GPUs. Models were instantiated using the Transformers (Wolf et al., 2019) library and trained using packages from the Jax (Bradbury et al., 2018) ecosystem.
16

---

Published as a conference paper at COLM 2024
C Additional Variations on Model and Training
Figure 4 shows the retrieval accuracy a function of the retrieval fraction for further variations the model or training, evaluated and trained on summarized abstracts. The red line corresponds to the model trained on summarized abstract described in the main text (fine-tuned CLIP-ViT-B/16 with constant learning rate LR 10 after linear warmup). The purple line corresponds to the base CLIP-ViT-B/16 model.
Curves for the model fine-tuned on the larger base CLIP model CLIP-ViT-L/14 (dotted red), with smaller learning rate LR 10 (dashed green), and with cosine learning rate schedule (green) are also shown. All these models are seen to perform similarly, with the exception of the model trained with smaller learning rate showing degraded performance. Given the similar performance between CLIP-ViT-L/14 (~~ 428 million parameters) and CLIP-VIT-B/16 (~ 149 million parameters), we chose the latter as the base model in the main text for computational efficiency.
Ablation Tests (Using Summarized Abstracts)
1.0
0.8
Retrieval accuracy
0.6
0.4
Fine-tune (summaries)
Base
0.2
Fine-tune larger CLIP model
LR
10-6
Cosine LR schedule
0.0 0.0
0.2
0.4
0.6
0.8
1.0
Retrieval fraction
Figure 4: Same Fig. 3 (right) retrieval accuracy as a function of the retrieval fraction for further variations on the model or training. The red and purple lines correspond to the model trained summarized abstract, described in the main text, and the base CLIP-ViT-B/16 model, respectively. Curves for the model fine-tuned on the larger base CLIP model CLIP-ViT-L/14 (dotted red), with a smaller learning rate LR = 16 (dashed green), and with cosine learning rate schedule (green) also shown.
D Text Retrieval Task
We images from the validation set queries and retrieve the most relevant text chunks (e.g., objects and use cases) from a curated list.
The following curated categories are used in the text retrieval experiment in Sec. 4. These are derived by initially prompting CLAUDE 2 having attached a subsample of 30 proposal abstracts in the online interface to be used as context, to produce list of categories corresponding to typical HST observations. The list then manually curated to remove similar entries and ensure representative sample of categories.
https::/ccladd.
17

---

Published as a conference paper at COLM 2024
["star forming galaxies", "lyman alpha", "dust", "crowded stellar field", "core- collapse supernova", "cosmology", "gravitational lensing", "supernovae", diffuse galaxies", "globular clusters", , "stellar populations", "interstellar mediumm,, "black holes", "dark matter", "galaxy clusters", "galaxy evolution", "galaxy formation", "quasars", "circumstellar disks", "exoplanets", "Kuiper Belt objects", "solar system objects", "cosmic web structure", "distant galaxies", "galaxy mergers", "galaxy interactions", "star formation", stellar winds", "brown dwarfs", "white dwarfs", "nebulae", "star clusters", galaxy archeology", "galactic structure", "active galactic nuclei", "ganma- ray bursts", "stellar nurseries", "intergalactic medium", "dark energy", dwarf galaxies", "barred spiral galaxies", "irregular galaxies", "starburst galaxies", "low surface brightness galaxies", "ultra diffuse galaxies", circumgalactic medium", "intracluster medium", "cosmic dust", "interstellar chemistry", "star formation histories", "initial proper motions", "binary star systems", "open clusters", "pre-main sequence stars", "protostars", "protoplanetary disks", "jets and outflows", interstellar shocks", "planetary nebulae", "supernova remnants", "red giants" "Cepheid variables", "RR Lyrae variables", "stellar abundances", "stellar dynamics", "compact stellar remnants", "Einstein rings", "trans-Neptunian objects", "cosmic microwave background", "reionization epoch", "first stars", "first galaxies", "high-redshift quasars", "primordial black holes", resolved binaries", "binary stars"]
function", "stellar
The following prompt used to generate the initial list before manual curation: "Here is a list of Hubble proposals. Base on this, please provide list of about 100 strings, each describing a science target or use case for obseroations imaged by the Hubble Space Telescope. You may use these proposals and also rely on your general knowledge. For example, I" l'gravitational lensing", "supernozae", "diffuse galaxies"....
We show the result of image-to-text retrieval in Tab. 5, for the base (second column) as well as summary tine-tuned (third column) models, using four observations (left-most column) from the validation set.
The top four text associations are shown for each image query. The 'ground truth' sum- marized abstract is shown in the right column. The base well fine-tuned models are seen to return a mix of relevant and less-relevant associations, although showing different qualitative behavior. Purely qualitatively, the fine-tuned model is seen to consistently return more relevant associations compared to the base model.
The second row (an image of supernova 1987A) highlights an interesting pattern the base model erroneously attributes the object at the center of the image to a gravitational lens, while the fine-tuned model correctly identifies reasonable misattribution common when querying the base model, and largely absent in the fine-tuned model.
supernova remnant. This kind of
18

---

Published as a conference paper at COLM 2024
Summarized abstract
Top43 text (summary fine-tuned)
Top-4 text (base off-the-shelf)
Hubble image
(objects; 'ground
truth')
1. high-redshift quasars
dwarf galaxies
isolated dwarf galaxies, WLM, Pegasus Dwarf Irregular Galaxy, stellar mass, main sequence stars
2. gravitational lensing
Lyrue variables
3. white dwarfs
3. red giants
4. dearf galaxies
4. trans- Neptunnnp objects
1. gravitational lensing supernovae
supernova remants
supemova SN 1987A, circum- stellar ring, superova rem- nant, shocked ring, radioac- isotopes
protostars
3. binary star systems cirsunstellar disks
3. galactic structure core-collapse supernova
X-ray luminous galaxy clus- ters, clusters, Balmer Break Galaxies, Lyman-break galaxies, gravitational tele- scopes
1. gravitational lensing
galaxy clusters
2. high-redshift quasars
lynan alpha
3. ultra diffuse galaxies galaxy clusters
3. intracluster nediun energy
pre-main sequence stars, Large Magellanic Cloud, young clusters, color- magnitude diagrams, main-sequence turn offs
1. star clusters
globular clusters
2. globular clusters
clusters
3. open clusters
3. galactic structure
populations
field
Table 5: Text snippets from curated list most closely matching given image query (left- most column) by cosine similarity of respective embeddings, shown for the base off-the-shelf (CLIP-VIT-B/16) and summary fine-tuned models. The 'ground truth' LLM-summarized abstract (only objects/ phenomena) is shown in the right-most column.
19


Reviews:
[{"review_id": "VsUSOs6F8Z", "reviewer": "Reviewer_dJXX", "summary": "The paper presents a new model called PAPERCLIP. The model is trained by fine-tuning CLIP on astronomical observations and abstracts from the Hubble telescope data. They train a variant of this model on summaries of the abstracts. Both these models show significantly higher performance than the base model on the held-out validation set. Further, the qualitative results on image retrieval are promising. Overall, this paper makes a unique contribution by adapting foundation models to astronomical data.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The need for PAPERCLIP needs to be better motivated in the paper.**\n\nTraditionally, astronomers train specialized models to search for phenomena of interest. This requires the curation of training data and separate models for each application. The motivation for this work is to introduce a method to train a single model that can be used to query relevant images from a set of observations, based on natural language.\n\n**Shouldn’t the quantitative experiments be designed to test image retrieval [...]?**\n**[...] one might query for images [...] It would be great if the authors extended this setting to a formal benchmark for quantitative evaluation.**\n\nThe retrieval accuracy, as defined in Sec. 3.3, is precisely the image retrieval accuracy for an image-text pair validation set. Note that this metric is symmetric in text to/from image retrieval, when averaged over the entire validation set.\n\n**[...] the models trained with abstract should also be tested on the validation set containing the summaries.**\n\nWe do exactly this in Fig. 3 (right) – here, all models (including trained with abstracts) are tested on the summaries, for a direct comparison.\n\n**Will the model perform well on other datasets not included in the training?**\n**Table 1 has two images: one from 1999 and the other from 2016. There are some obvious quality differences. Did you notice if these differences propagate to the model?**\n\nThe dataset includes diverse images from Hubble, with different filters, targets, and post-processings. The differences between the images in Tab. 1 is coming from these variations, rather than differences in observation years. Our method is intended to be applied to a diversity of observations from a specific telescope, and we do not expect it to immediately generalize to different telescopes without further fine-tuning.\n\n**The CLIP text encoder has a maximum text length of 77 tokens. What do you do when the abstract length is greater than 77 tokens? Do you discard it?**\n\nWe note in Sec. 3.2: “When using raw proposal abstracts, random chunks of the text delimited by periods are selected on the fly to fit within the maximum token length of the text encoder.” This also serves as a form of text data augmentation, given the relatively small dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Paper writing.** The paper is well written. All the sections are neatly organized, making it easy to understand the paper. The results and figures are very easy to follow. \n\n**Unique contribution.** The paper presents a new dataset containing Hubble Space Telescope image observations and their abstracts. This dataset will be valuable to the community. The authors develop a specialized foundation model called PAPERCLIP by further fine-tuning CLIP on this dataset. \n\n**Promising results.** The results of PAPERCLIP on the validation set show PAPERCLIP performs well whereas CLIP struggles with specialized datasets like the one introduced in the paper.", "reasons_to_reject": "**Motivation.**\n\nThe need for PAPERCLIP needs to be better motivated in the paper. The authors say there is a “considerable interest in developing custom foundation models for the sciences”. While this may be true, it doesn’t say why creating a model like PAPERCLIP is necessary. \nThe qualitative evaluation focuses on image retrieval, but this setup is not well motivated. Are practitioners going to use the model to retrieve observations? If so, shouldn’t the quantitative experiments be designed to test image retrieval rather than showing accuracy on the validation set? (see evaluation weaknesses below).\n\n**Technical contribution.**\n\nPAPERCLIP which adapts a pretrained CLIP to the astronomical data with a contrastive loss objective. While this is a unique model, it is a classic example of transfer learning. Transfer learning is a well-studied area in machine learning. There is no new insight here. \n\nAnother contribution of the paper is summarizing the abstracts. The authors use the Outlines package to summarize the abstracts by adding constraints. But, the results in Figure 2 suggest that there is no advantage to using LLM-guided summaries. In Figure 3, the improvements are rather marginal. This suggests that the summaries may not be doing much to improve the model performance. \n\n\n**Experiment and Evaluation.**\n\nThe quantitative evaluation is limited to one dataset. There is no evidence that this model works beyond this one dataset. Will the model perform well on other datasets not included in the training?\n\nEvaluating the model’s performance on the validation set is not convincing. The performance curve only shows that loss drops on the validation set drawn from the same distribution. Furthermore, for completeness, the models trained with abstract should also be tested on the validation set containing the summaries. \n\nThe qualitative evaluation looks useful as one might query for images. However, the evaluation is limited to two objects. It would be great if the authors extended this setting to a formal benchmark for quantitative evaluation.", "questions_to_authors": "Table 1 has two images: one from 1999 and the other from 2016. There are some obvious quality differences. Did you notice if these differences propagate to the model? \n\nLong abstracts: The CLIP text encoder has a maximum text length of 77 tokens. What do you do when the abstract length is greater than 77 tokens? Do you discard it?", "extra_scores": {}}, {"review_id": "g5NTGJcvHN", "reviewer": "Reviewer_5XaM", "summary": "This paper introduces PAPAERCLIP, a fine-tuned version of the CLIP model designed to link proposal abstracts with their corresponding astronomical observations. Various fine-tuning methods and text processing strategies are explored. The authors demonstrate that their adapted model performs effectively in both image and description retrieval tasks on the HSB dataset.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The paper does not include any baseline comparisons. [...] 1), Are there existing tools capable of performing similar tasks? 2) Assuming each observation has an abstract, could description/image retrieval be achieved by calculating text similarities [...]**\n\nThis is a great question. The reason the target application is not possible with other methods (e.g., text similarity) is that not all Hubble images come with descriptive captions, like those used for fine-tuning the CLIP model (i.e., our observations are a small subset of all Hubble observations). Large survey-style campaigns image different parts of the sky without target-specific captions, which is the setting where the current method would be useful to query for phenomena of interest in large unlabeled datasets.\n\n**[...] is there some way to see how alinged the astronomical data with the CLIP model's domain? It is unclear whether the observed performance improvements are due to fine-tuning (from a generalist vision model) or if the original CLIP model had prior exposure to similar astronomical data, enabling rapid adaptation.**\n\nWhile this is difficult to check for the proprietary CLIP model used in our study, it should be possible to check for similarity of image embeddings in public datasets like LAION-5B. Unfortunately, these datasets are currently (May 2024) inaccessible due to safety concerns. However, given the domain-specialized nature of our data, it is highly unlikely that images of this kind were included in the CLIP trained set. The poor performance (Fig. 2 right) and lack of diversity of cosine similarities (Fig. 3 left) in the base model also indicate this. The fact that training from scratch does not match the performance of using the pre-trained model further indicates that a generalist backbone is necessary for performance. \n\n**Q:The paper extensively discusses the use of LLM-generated abstracts, though this approach does not yield superior results. Can the authors delve deeper into why this might be?**\n\nThe [data-processing inequality](https://en.wikipedia.org/wiki/Data_processing_inequality) implies that the summarization process is necessarily lossy; the fact that the raw abstracts perform quite well implies that the CLIP training procedure is able to extract relevant information from the abstracts dataset, even though it is noisy. This capability was not a-priori expected, and is a central message of the study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study investigates the application of image foundation models to astronomical observations, highlighting potential benefits for astrophysics.\n\nVarious training strategies are explored, including fine-tuning, using adapters, and training from scratch.\n\nThe authors also apply the method of Willard & Louf (2023), utilizing LLM-generated abstract summaries as training texts.", "reasons_to_reject": "The paper does not include any baseline comparisons. Questions arise such as: 1), Are there existing tools capable of performing similar tasks? 2) Assuming each observation has an abstract, could description/image retrieval be achieved by calculating text similarities (tfidf or neural network based), treating abstracts as a form of image representation to potentially establish an upper bound?\n\nThe exploration of different CLIP variants like OpenCLIP, and other similar models like SigLIP, could provide insight into the impact of varying backbone foundation models.", "questions_to_authors": "Given that the CLIP model is trained on a diverse but also unreleased dataset, is there some way to see how alinged the astronomical data with the CLIP model's domain? It is unclear whether the observed performance improvements are due to fine-tuning (from a generalist vision model) or if the original CLIP model had prior exposure to similar astronomical data, enabling rapid adaptation.\n\nThe paper extensively discusses the use of LLM-generated abstracts, though this approach does not yield superior results. Can the authors delve deeper into why this might be?", "extra_scores": {}}, {"review_id": "oLgQsCnXCl", "reviewer": "Reviewer_1Yks", "summary": "The paper presents an astronomical dataset of image-text pairs for finetuning CLIP. The dataset consists of 32k training and 3k validation images (paired with text). The dataset is constructed by taking abstracts of (funded!) proposals for the Hubble Space Telescope. The paper tries summarizing these abstracts with LLMs. Both raw and summaries work comparably in loss and retrieval performance on the validation split. The images corresponding to an abstract (up to 20) are found by querying an API.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**There is no technical novelty in data construction or modeling. The dataset is constructed by downloading abstracts and matching with images. The model is standard CLIP. The experiments on the constructed validation set are as expected.**\n\n**Only a loose connection with LLMs. The paper is primarily about CLIP.**\n\nWe frame our study as an application of language models to a previously-unexplored field with domain-informed data curation. We note that the COLM call mentions, “LMs on diverse modalities and novel applications [...] with extra encouragement for less studied modalities or applications.”\n\nRegarding the paper primarily using CLIP and having a loose connection with LLMs, we again note that call for papers mentions, “We consider the term \"language model\" in the broadest way.”, and is thus not restricted to LLMs specifically. \n\nOverall, we believe that an LM application in astronomy, which is an understudied field in the context of LMs with nevertheless high potential, is in-scope for COLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A cool dataset constructed from public resources. The dataset and the finetuned model can be useful for applications in astronomy.", "reasons_to_reject": "- There is no technical novelty in data construction or modeling. The dataset is constructed by downloading abstracts and matching with images. The model is standard CLIP. The experiments on the constructed validation set are as expected.\n- Only a loose connection with LLMs. The paper is primarily about CLIP.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vOMVNJzLR6", "reviewer": "Reviewer_Uywh", "summary": "**Quality:** The authors compile a dataset of textual proposals for astronomical observations written by astronomers so that they can perform these observations with the Hubble Space Telescope,  and the ensuing images observed by telescope. Using this dataset, the authors explore for the first time the original and interesting use case of image and text retrieval in the astronomy domain. The authors fine-tune CLIP using proposal abstracts and related observation images following CLIP’s contrastive learning approach. The authors also experiment with optional summarization of abstracts via LLM-guided summarization into a required format, prompting Mixtral to generate the summaries. The experiments performed are adequate and seem sound with improved results going beyond the zero-shot CLIP model and other baselines the authors compare to. \n\n**Clarity:** The paper is mostly clear; however, certain methodological choices and domain-specific terminology related to astronomy should be explained in more detail to clarify the task and the inputs. \n\n**Originality:** The application of pretrained models in the astronomy domain for images and texts is interesting and novel.\n\n**Significance:** The main contribution is the investigation of the use of pretrained models in the domain of astronomy, revealing multimodal use cases in sciences that can benefit from the advancements in pretrained models. I think the outcomes would be of significance to many other scientific domains.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[...] did the authors have a separate test set?**\n\nDue to practical limitations associated with the small size of the fine-tuning dataset, we did not use different datasets for validation and testing, deeming this sufficient for a proof-of-principle exposition.\n\n**The images are cropped to the 20% of the area of the original image, would this somehow get rid of informative parts in an image?**\n\nThe images are cropped to 20% on the fly during training, as a form of data augmentation. Since each image is seen ~20 times over training, each time cropped differently, the model can learn to leverage a positive association signal when this is present. The association signal may also not be localized to a particular part of the image.\n\n**Why is the random expectation of retrieval accuracy 10%? In 3.3, the formula has the k% of the number of images, but the explanation mentions top k% of captions [...]**\n\nThe random retrieval accuracy is 10% because, without a learned association between modalities, the model will correctly retrieve the correct item in the top-k%, k% of the time by random chance.\n\nAs mentioned in Sec. 3.3, the retrieval evaluation metric when averaged over the test set is symmetric over the image and text modalities; i.e., due to commutativity of the cosine similarity, we can either average over images in the test set while looking at top-k% captions, or vice-versa. In Sec. 3.3, for concreteness we describe text retrieval (“top k% of captions”, “averaged over the images”). Fig. 2 right therefore does not correspond to a particular retrieval task.\n\n**[...] I was not sure what the authors meant by “it is challenging to discern meaningful, strong associations”**\n\nWhile occasional associations are apparent, this is very much the exception rather than the norm for the base model (as can be seen from subsequent retrieved images); however, we agree that the language here should be softened.\n\n**[...] the authors claim that the fine-tuned model “shows strikingly different behavior’’, which, as a non-expert for the astronomy domain, was not clear to me.**\n\nThe difference in behavior can be seen from the retrieved images – in Tab. 4 all the returned images for the fine-tuned model correspond to Jupiter and SN1987A, as appropriate. This is not the case for the base model, in Tab. 3. While domain expertise may be helpful in assessing relevance, one can click through to the Proposal IDs of the retrieved images, confirming their relation to the queries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The application of pretrained models to the astronomical domain is original, showing the applicability and usefulness of pretrained models in the sciences, especially on data that does not correspond to the large bulk of the pretraining datasets, originating from a specific distribution.", "reasons_to_reject": "The paper is written mostly well, but I have some doubts regarding the methodological choices (no test set, cropping images, interpretation of examples; see my questions below), which would need more motivation and clarification in the main paper. Additionally, some domain-specific terminology needs to be explained, especially earlier in the paper. For instance, what is meant by ‘a successful abstract proposal’? This was explained halfway through the paper, which I think should be introduced earlier.", "questions_to_authors": "1. It seems that the evaluation and the analysis was done on the validation set, did the authors have a separate test set?\n\n2. The images are cropped to the 20% of the area of the original image, would this somehow get rid of informative parts in an image?\n\n3. Why is the random expectation of retrieval accuracy 10%? In 3.3, the formula has the k% of the number of images, but the explanation mentions top k% of captions, which I found confusing. It would also be better to clarify which retrieval task the graph on the right of Figure 2 corresponds to. \n\n4. In Table 3, the query is ‘Jupiter’ and the first retrieved image is of Jupiter, so I was not sure what the authors meant by ‘it is challenging to discern meaningful, strong associations’. Additionally, the authors claim that the fine-tuned model ‘shows strikingly different behavior’, which, as a non-expert for the astronomy domain, was not clear to me. Is it also because the authors used shorter/simpler queries as compared to the fine-tuning data?\n\n5. The paper could benefit from having an excerpt of the prompts and schema samples in the main part, which I found informative.", "extra_scores": {}}, {"review_id": "IkJHnvTE6k", "reviewer": "Reviewer_kBem", "summary": "The paper deals with finetuning of the CLIP model for the astronomical data, using free-form proposals for the Hubble telescope as an textual input and images obtained for each proposal as an image input. The results show that finetuning results in meaningful image and text embeddings that allow for the semantic image search using textual input.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**I understand that it is appealing to call a model \"PAPER CLIP\" but this name is misleading**\n\nWe note that we do not call the _model_ PAPERCLIP; instead, as emphasized in the abstract and elsewhere, PAPERCLIP refers to the _method_ of using (processed) proposal abstracts to inform observation-text associations for astronomical images, which in this specific case we apply to observations from the Hubble Space Telescope.\n\n**The dataset used in this study is an exception in astronomical data, since images have meaningful textual counterparts.**\n**The most crucial question is whether it is possible to use the model trained on the Hubble proposal dataset to label/search some other available images.**\n\nThe dataset includes diverse images from Hubble, with different filters, targets, and post-processings, taken over a ~30 year timespan. Our method is intended to be applied to observations from a specific telescope, and we do not expect it to immediately generalize to others without further fine-tuning. We emphasize that not all Hubble images have associated text i.e., not all are the downstream result of targeted proposals, as those used in fine-tuning here (the data used is a subset of Hubble observations). Our method is therefore intended to be applied to unlabeled sets of observations. In the paper, in order to be able to compare with a ground truth, we tested our method on observations which do have associated abstracts.\n\n**[...] why image-text association should be noisy [...] InfoNCE allows for multiple positive pairs, so it is possible to train model taking all image-text correspondences into account simultaneously.**\n\nAlthough the version of InfoNCE used in the paper allows for a batch of $N$ positive pairs, each element of the pair has $N-1$ negatively paired elements; a single data point cannot have multiple positive “anchors”. Therefore, since multiple images can correspond to an abstract, when training with minibatches it is possible that some of the negative pairs will actually be positively associated (e.g., if two abstracts in a batch are the same, but the corresponding images are different). Through the results in the paper, we empirically show that, despite this, the model is able to leverage the signal between positively associated pairs and “beat down” the noise associated with negatively pairing image-text samples that may nevertheless be associated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper deals with interesting data, and the results are promising.\n\nThe paper uses meaningful evaluation and baselines.\n\nThe paper is well written and structured, easy to follow and understand.", "reasons_to_reject": "The dataset used in this study is an exception in astronomical data, since images have meaningful textual counterparts. The dataset is also very small, comparing to vast massives of available astronomical data. So, the most crucial question is whether it is possible to use the model trained on the Hubble proposal dataset to label/search some other available images. It would be nice to see at least qualitative evaluation in that regard. \n\nI understand that it is appealing to call a model \"PAPER CLIP\" but this name is misleading: First, this abbreviation is not very informatie, it does not even mention anything astronomical. Second, and most importantly, the paper does not present any new model, it only fine tune the existing one. I would call the paper \"Fine-tuning CLIP for matching astronomical images and texts\", or something like this, without introducing any additional abbreviations.\n\nI am not sure training with random batches is actually needed. Just random baseline and non-tuned CLIP would be enough in my view.", "questions_to_authors": "I do not understand why image-text association should be noisy, as it is mentioned in the first paragraph in 3.2. InfoNCE allows for multiple positive pairs, so it is possible to train model taking all image-text correspondences into account simultaneously.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are generally in agreement that the paper’s primary contribution is that it applies LLMs (or, more precisely, a VLM) to the interesting and novel domain of astronomical data. The concerns raised are primarily that 1) the method is not overwhelmingly novel (i.e., it is just finetuning CLIP) and 2) there are some questionable methodological choices, specifically a lack of good baselines or designated test sets.

There was a lively discussion between authors and reviewers, which at least partially addressed the above concerns by providing some additional experiments which compared CLIP to the fine-tuned CLIP and to tf-idf on a retrieval task, and which justified the experimental design choices as being the result of data limitations. 

Taking everything into account, I believe this paper is worth including in COLM. It is primarily an engineering paper, but it represents the type of important engineering work that can be difficult to publish in traditional single-discipline conferences, and which will be instrumental in if/how LLMs impact the (scientific) world more broadly. This, as I understand it, is in line with the mission of COLM. I think it is valuable to highlight proofs-of-concept for adapting LLMs/VLMs to new domains, and that this can serve as an exemplar and/or a starting point for similar efforts in astronomy, and in other domain-specific LLM adaptations. 

The reviewers provided very valuable critiques and suggestions during the discussion. I appreciate the authors’ good responses during discussion, and the authors should consider it necessary that the paper is updated to reflect the outcomes of this discussion prior to publishing.

[At least one review was discounted during the decision process due to quality]
Decision: Accept