pathfinder: A Semantic Framework for Literature
Review and Knowledge Discovery in Astronomy

Kartheik G. Iyer, Mikaeel Yunus, Charles O'Neill, Christine Ye, Alina Hyk, Kiera Mccormick, Ioana Ciucă, John F. Wu, Alberto Accomazzi, Simone Astarita, et al.

► To cite this version:

Kartheik G. Iyer, Mikaeel Yunus, Charles O'Neill, Christine Ye, Alina Hyk, et al. pathfinder: A Semantic Framework for Literature Review and Knowledge Discovery in Astronomy. The Astrophysical Journal Supplement Series, 2024, 275, 10.3847/1538-4365/ad7c43. insu-04852723

HAL Id: insu-04852723

https://insu.hal.science/insu-04852723v1

Submitted on 21 Dec 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# pathfinder: A Semantic Framework for Literature Review and Knowledge Discovery in Astronomy

Kartheik G. Iyer<fnref n="1" />,<fnref n="33" /> ID, Mikaeel Yunus<fnref n="2" /> ID, Charles O’Neill<fnref n="3" /> ID, Christine Ye<fnref n="4" /> ID, Alina Hyk<fnref n="5" /> ID, Kiera McCormick<fnref n="6" /> ID, Ioana Ciucă<fnref n="7" />,<fnref n="8" />,<fnref n="9" /> ID, John F. Wu<fnref n="2" />,<fnref n="10" /> ID, Alberto Accomazzi<fnref n="11" /> ID, Simone Astarita<fnref n="12" /> ID, Rishabh Chakrabarty<fnref n="13" />, Jesse Cranney<fnref n="14" /> ID, Anjalie Field<fnref n="15" /> ID, Tirthankar Ghosal<fnref n="16" /> ID, Michele Ginolfi¹⁷,<fnref n="18" /> ID, Marc Huertas-Company<fnref n="19" />,<fnref n="20" />,<fnref n="21" />,<fnref n="22" /> ID, Maja Jabłońska<fnref n="7" />, Sandor Kruk<fnref n="12" /> ID, Huiling Liu<fnref n="23" />,<fnref n="24" />, Gabriel Marchidan<fnref n="25" />, Rohit Mistry<fnref n="26" />, J. P. Naiman²⁷ ID, J. E. G. Peek<fnref n="2" />,<fnref n="10" /> ID, Mugdha Polimera<fnref n="11" /> ID, Sergio J. Rodríguez Méndez<fnref n="3" /> ID, Kevin Schawinski²⁸ ID, Sanjib Sharma<fnref n="10" /> ID, Michael J. Smith<fnref n="19" /> ID, Yuan-Sen Ting<fnref n="29" />,<fnref n="30" /> ID, and Mike Walmsley<fnref n="31" />,<fnref n="32" /> ID

(UniverseTBD)

## Abstract

The exponential growth of astronomical literature poses significant challenges for researchers navigating and synthesizing general insights or even domain-specific knowledge. We present *pathfinder*, a machine learning framework designed to enable literature review and knowledge discovery in astronomy, focusing on semantic searching with natural language instead of syntactic searches with keywords. Utilizing state-of-the-art large language models (LLMs) and a corpus of 385,166 peer-reviewed papers from the Astrophysics Data System, *pathfinder* offers an innovative approach to scientific inquiry and literature exploration. Our framework couples advanced retrieval techniques with LLM-based synthesis to search astronomical literature by semantic context as a complement to currently existing methods that use keywords or citation graphs. It addresses complexities of jargon, named entities, and temporal aspects through time-based and citation-based weighting schemes. We demonstrate the tool's versatility through case studies, showcasing its application in various research scenarios. The system's performance is evaluated using custom benchmarks, including single-paper and multipaper

<figure><img src="image_3.png" /></figure>

<footnote n="1">¹ Columbia Astrophysics Laboratory, Columbia University, 550 West 120th Street, New York, NY 10027, USA; kgi2103@columbia.edu</footnote>

<footnote n="2">² Department of Physics and Astronomy, Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD 21218, USA</footnote>

<footnote n="3">³ School of Computing, The Australian National University, 108 North Road, Acton ACT 2601, Australia</footnote>

<footnote n="4">⁴ Stanford University, 450 Jane Stanford Way, Stanford, CA 94305, USA</footnote>

<footnote n="5">⁵ School of Electrical Engineering and Computer Science, Oregon State University, 1500 SW Jefferson Way, Corvallis, OR 97331, USA</footnote>

<footnote n="6">⁶ Department of Engineering, Loyola University Maryland, 4501 North Charles Street, Baltimore, MD 21210, USA</footnote>

<footnote n="7">⁷ Research School of Astronomy & Astrophysics, Australian National University, Cotter Road, Weston, ACT 2611, Australia</footnote>

<footnote n="8">⁸ School of Computing, Australian National University, Acton, ACT 2601, Australia</footnote>

<footnote n="9">⁹ Kavli Institute for Particle Astrophysics and Cosmology and Department of Physics, Stanford University, Stanford, CA 94305, USA</footnote>

<footnote n="10">¹⁰ Space Telescope Science Institute, 3700 San Martin Drive, Baltimore, MD 21218, USA</footnote>

<footnote n="11">¹¹ Center for Astrophysics, Harvard & Smithsonian, Cambridge, MA 02138, USA</footnote>

<footnote n="12">¹² European Space Agency (ESA), European Space Astronomy Centre (ESAC), Camino Bajo del Castillo s/n, 28692 Villanueva de la Cañada, Madrid, Spain</footnote>

<footnote n="13">¹³ Independent Researcher, Paris, France</footnote>

<footnote n="14">¹⁴ Astralis-AITC—Stromlo, RSAA, Australian National University, Cotter Road, Weston, ACT2600, Australia</footnote>

<footnote n="15">¹⁵ Department of Computer Science, Johns Hopkins University, 3400 North Charles Street, Baltimore, MD 21218, USA</footnote>

<footnote n="16">¹⁶ National Center for Computational Sciences, Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA</footnote>

<footnote n="18">¹⁸ INAF—Osservatorio Astrofisico di Arcetri, Largo E. Fermi 5, I-50125, Firenze, Italy</footnote>

<footnote n="19">¹⁹ Instituto de Astrofísica de Canarias, C. Via Lactea, 1, E-38205 La Laguna, Tenerife, Spain</footnote>

<footnote n="20">²⁰ Universidad de la Laguna, Dept. Astrofísica, E-38206 La Laguna, Tenerife, Spain</footnote>

<footnote n="21">²¹ Université Paris-Cite, LERMA—Observatoire de Paris, PSL, Paris, France</footnote>

<footnote n="22">²² SCIPP, University of California, Santa Cruz, CA 95064, USA</footnote>

<footnote n="23">²³ Key Laboratory for Research in Galaxies and Cosmology, Department of Astronomy, University of Science and Technology of China, Hefei, Anhui 230026, People's Republic of China</footnote>

<footnote n="24">²⁴ School of Astronomy and Space Science, University of Science and Technology of China, Hefei 230026, People's Republic of China</footnote>

<footnote n="25">²⁵ Iași AI, Iași, Romania</footnote>

<footnote n="26">²⁶ Xaana AI, Canberra, Australia</footnote>

<footnote n="29">²⁹ Department of Astronomy, The Ohio State University, Columbus, OH 43210, USA</footnote>

<footnote n="30">³⁰ Center for Cosmology and AstroParticle Physics (CCAPP), The Ohio State University, Columbus, OH 43210, USA</footnote>

<footnote n="31">³¹ Dunlap Institute for Astronomy and Astrophysics, University of Toronto, 50 St. George Street, Toronto, ON M5S 3H4, Canada</footnote>

<footnote n="32">³² Jodrell Bank Centre for Astrophysics, Department of Physics & Astronomy, University of Manchester, Oxford Road, Manchester, M13 9PL, UK

*Received 2024 August 2; revised 2024 September 14; accepted 2024 September 16; published 2024 November 29*</footnote>

<footnote n="33">³³ Hubble Fellow.</footnote>
tasks. Beyond literature review, *pathfinder* offers unique capabilities for reformatting answers in ways that are accessible to various audiences (e.g., in a different language or as simplified text), visualizing research landscapes, and tracking the impact of observatories and methodologies. This tool represents a significant advancement in applying artificial intelligence to astronomical research, aiding researchers at all career stages in navigating modern astronomy literature.

*Unified Astronomy Thesaurus concepts: Astronomical reference materials (90); Astronomy web services (1856); History of astronomy (1868); Computational methods (1965); Astronomy data visualization (1968)*

# 1. Introduction

As one of the oldest scientific disciplines, astronomy has amassed an enormous body of literature over time. Modern astronomical libraries and recordkeeping services like the Astronomical Data System (ADS; A. Accomazzi et al. 2015) and preprint servers like arXiv provide lasting repositories for accessing current research on various astronomical subfields, with records on ADS extending back to the early 16th century. As the body of astronomical literature grows (at an ever-accelerating rate), this creates a growing problem of keeping track of the literature, with it becoming harder over time to keep track of relevant papers and contextualize the information contained in them while writing new papers. In fact, with the advent of new observatories like the Atacama Large Millimeter/submillimeter Array (ALMA) and JWST and new modalities of observations like gravitational waves, the literature has become challenging for even experienced researchers to keep pace with. This is exacerbated by a growing need for interdisciplinary efforts, which means that astronomers often need to keep track of multiple fields of literature, such as electronics and instrumentation, high-performance computing, statistics, machine learning, and computer vision.

At best, this leads to a much larger amount of time and effort spent in organizing and cataloging papers for individual researchers, and at worst it can lead to a splintering of the research landscape with researchers resorting to a friends-of-friends or in-group citation framework while writing papers. This situation also creates a barrier to entry for aspiring students and researchers trying to enter the field and perform their first literature search, especially in the absence of an authoritative review on their chosen topic.

While this is also true of fields other than astronomy, we have the unique distinction of having a large body of publicly accessible data, code and literature (F. Genova 2023), which provides a unique opportunity for developing methods that can ingest, retrieve, and synthesize literature in a way that is useful for a wide range of audiences (K. G. Iyer 2021; J.-V. Rodríguez et al. 2022; S. Blanco-Cuaresma et al. 2023; T. Dung Nguyen et al. 2023; F. Grezes et al. 2024). To this end, we explore the use of state-of-the-art machine learning methods in conjunction with a corpus of papers from ADS and arXiv to find relevant literature and provide initial starting points for answering questions across a variety of levels.

Large language models (LLMs) have seen rapid advancement and adoption in recent years, with models like GPT-4 (A. OpenAI 2024) and LLaMA (H. Touvron et al. 2023) demonstrating impressive capabilities across a wide range of tasks. In academic contexts, LLMs are increasingly being used to assist with explaining advanced concepts (E. Prihar et al. 2023) and perform literature review (Y. Li et al. 2024; K. Tao et al. 2024), and possibly with writing papers (W. Liang et al. 2024a), even in astronomy (S. Astarita et al. 2024). However,

their application remains controversial due to concerns about accuracy, bias, accidental plagiarism (N. Pervez & A. J. Titus 2024), and the potential for hallucination (e.g., V. Rawte et al. 2023; Y. Zhang et al. 2023). Despite these challenges, many researchers are exploring ways to leverage LLMs as tools to augment human expertise and accelerate scientific discovery (R. Van Noorden & J. M. Perkel 2023), in particular in fields with vast and rapidly growing bodies of literature, like astronomy.

The notion of using machine learning methods for improving literature surveys is not a particularly new concept, and such tools have been accessible since the early 1990s with methods like *n*-grams (W. B. Cavnar & J. M. Trenkle et al. 1994; G. Kondrak 2005), bag-of-words (Y. Zhang et al. 2010), or transformer-based models like BERT (J. Devlin 2018). Versions of these methods including AstroBERT (F. Grezes et al. 2024) and more recently AstroLLaMA (T. Dung Nguyen et al. 2023) have been applied to large corpora of astronomical data as proof-of-concept techniques to showcase how natural language processing (NLP) and newer language models can successfully ingest astronomical keywords and scientific jargon. Here, we provide a working pipeline to show how these models can be combined with techniques like retrieval-augmented generation (RAG) and agentic LLMs to capture significantly more semantic context and provide hallucination-free literature review at a fraction of the time and cost of manually searching for papers on a given topic. We stress that this is not meant to be a replacement for existing search tools like arxiv.org, the ADS, Google Scholar, Benty-Fields or Arxivsorter, but rather a complement to them, with three key advantages: (1) the ability to query the system using natural language, (2) added synthesis to generate a targeted summary of the retrieved documents in context to the question, and (3) exploratory tools to find similar papers in an interpretable embedding space.

To do this, we present the *pathfinder* framework,<fnref n="34" /> an open-source, publicly available tool that uses LLMs to answer natural-language astronomy questions using a corpus of ~385,000 peer-reviewed papers from ADS going back to 1990. The framework is presented both as open-source code and as an online tool that can be used to find relevant literature, answer questions, and explore the corpus of papers. The current version of the tool uses only abstracts, but can be extended to full text in the future. In this paper, we explore the use of *pathfinder* to (i) visualize papers as a “landscape” of astronomy research, (ii) find similar/relevant papers by performing a similarity search in embedding space, (iii) answer questions without hallucinations using the embedding space, (iv) explore the impact of different telescopes and observatories on the landscape of research, (v) explore the trends of authors over time, (vi) quantify missing areas that need to be developed further and find areas of interest for future surveys and facilities.

The structure of this paper is as follows. In Section 2, we describe the data set used to retrieve papers from. In Section 3,

<footnote n="34">³⁴ https://huggingface.co/spaces/kiyer/pathfinder</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 1. Schematic showing the overall `pathfinder` pipeline.</figcaption></figure>

we describe the overall `pathfinder` pipeline. In Section 4, we describe evaluation benchmarks used while developing the model. Section 5 describes ways for users to interact with and use `pathfinder`, and provides some case studies that demonstrate its behavior across different types of questions. In Section 6, we present some larger trends analyzed with the `pathfinder` framework. Section 7 concludes and summarizes the paper and the scope for future work.

## 2. Data Set

We have compiled a data set of ~385,000 paper abstracts from the ADS<fnref n="35" /> and arxiv.org,<fnref n="36" /> along with associated metadata including paper titles, publication dates, DOIs, author and affiliation information, and ADS keywords and bibcodes. We have also scraped the bibcodes for papers referenced in and citing any given paper in the data set, which can be used to further expand the database in future iterations. In addition to this, we have used natural language tools (spacy running `en_core_web_sm`) to determine a set of 20 keywords for each abstract, along with LLM-generated embeddings for each abstract as described in Section 3. The keywords are subsequently used to annotate figures and implement keyword weighting while retrieving papers.

The majority of the papers in our current corpus are drawn from an existing list of ~270,000 papers classified as astro-ph from the Kaggle arXiv data set,<fnref n="37" /> which contains papers from 1992 April to 2023 July (similar to AstroLLaMA; T. Dung Nguyen et al. 2023; E. Perkowski et al. 2024). These papers are further augmented using metadata (bibcodes, citations, dates, authors, and affiliations) from ADS. Since there are a number of papers that are not on arxiv.org, we subsequently query ADS for papers from 1990 January to 2024 September to find papers that are not in our data set and add them, bringing our corpus to N = 385,166. This set will be updated periodically to keep up-to-date with the current literature, and augmented by a corpus of older papers processed with optical character recognition (OCR) as part of future work (J. P. Naiman et al. 2023). The data set is publicly available online.<fnref n="38" /> While this is not a complete corpus and primarily draws from the ApJ, MNRAS, A&A, ARA&A, Nature, Science, PASA, PASP, and PASJ families of journals, it includes a large sample of relevant work that can be used to test the framework.

## 3. Building `pathfinder`

This section briefly describes the methods used to construct `pathfinder`. The codebase is public and available at the `pathfinder` repository.<fnref n="39" /> Briefly, the pipeline is an augmented version of RAG. In standard RAG, the system first retrieves a set of relevant documents for any input user query, and then uses the information therein to synthesize its answer. `pathfinder`'s augmentations include question categorization, query expansion, reranking, the ability to filter by date, citations and keywords and an alternative reason-thought-act-based framework for synthesizing answers, described in further detail in the following sections. Figure 1 shows a schematic of the procedure described in this section.

### 3.1. Generation

We first describe the generation step (right-hand side of Figure 1), which uses the retrieved papers and associated metadata to generate an answer to the user's query.

#### 3.1.1. Generating Embeddings

We compute embeddings for each abstract in our corpus using the `text-embedding-3-small` model from OpenAI,<fnref n="40" /> which is used to encode each abstract into a 1536-dimensional vector. Once the embeddings are computed, we use uniform manifold approximation and projection (UMAP<fnref n="41" />; L. McInnes et al. 2018) to create a two-dimensional embedding of the high-dimensional vector space for easier visualization and further analysis. A heatmap of the embedding space is provided in Figure 2, with the different regions annotated with their most frequently occurring keywords for clarity.

#### 3.1.2. Text Generation with RAG

Generally, question-answer applications involving LLMs generate an answer following a template (sometimes called a prompt) in response to a query. However, in doing so, there is a danger that the model may output factually incorrect information and lack access to all the available information needed to reply (S. Roller et al. 2021). To handle both of these problems, RAG forces the LLM to generate the response while using (and possibly citing) a set of document sources (P. Lewis et al. 2020; K. Shuster et al. 2021). Given an input query, we first search the full space of papers to find a subset of ~1–30 papers that

<footnote n="35">³⁵ https://ui.adsabs.harvard.edu/help/api/</footnote>

<footnote n="36">³⁶ https://info.arxiv.org/help/api/index.html</footnote>

<footnote n="37">³⁷ https://www.kaggle.com/Cornell-University/arxiv</footnote>

<footnote n="38">³⁸ https://huggingface.co/data sets/kiyer/pathfinder_arxiv_data</footnote>

<footnote n="39">³⁹ https://huggingface.co/spaces/kiyer/pathfinder/tree/main</footnote>

<footnote n="40">⁴⁰ https://platform.openai.com/docs/guides/embeddings</footnote>

<footnote n="41">⁴¹ https://umap-learn.readthedocs.io/en/latest/</footnote>
<figure><img src="image_5.png" /><figcaption>**Figure 2.** A heatmap showing a 2D UMAP projection of the 1536-dimensional embedding space of that shows the different areas of the astro-ph literature corpus. The heatmap color denotes the density of papers in different parts of the corpus, with the auto-tagging keywords at various locations shown to illustrate the way the embeddings group the different topics by semantic similarity. Similar to a world map, the axes here do not hold a particular meaning. Regions close to each other hold a semantic similarity, while distant regions do not.</figcaption></figure>
are relevant to the input query, retrieved using the methods described in Section 3.2. We then use langchain<sup>42</sup> to set up the RAG system, where the query is passed in along with the abstracts of the papers broken down into chunks for the LLM to then construct an answer. The input prompt template also requires the LLM to be succinct in its responses and respond with “I do not know” if the LLM does not find sources relevant to the query.

### 3.1.3. Text Generation with ReAct Agents

While many of the questions that astronomers tend to ask tools like `pathfinder` will be factual and need efficient similarity search and synthesis, others are more involved and require multiple lookups to answer. These tend to be questions that require resolving multiple conflicting viewpoints (*consensus evaluation*), combining information across multiple topics (*composition*), or speculating beyond available data (*counterfactual*; see Section 4.4 for a fuller description of the different types of questions).

A limitation of the RAG framework is that it is incapable of directly answering these questions. To provide a basic framework that can be used to tackle these questions, we use Reasoning and Acting agents (ReAct; S. Yao et al. 2022), an approach that combines reasoning and acting in LLMs, allowing them to break down complex tasks into more atomic steps and execute them, combined with the RAG framework we have used thus far. Briefly, this system involves `pathfinder` receiving an input query, followed by the ReAct agent using a LLM to reason about the task and break it down into steps. For each step, the agent acts by using RAG to retrieve relevant information from the paper corpus. It uses the retrieved information to further analyze the data and make queries until it has enough knowledge to answer the question or runs up against the number of allowed iterations. The system is not perfect, with the LLM sometimes stalling in a process loop where it cannot find an ideal way to phrase a question. Newer methods exist to use search trees (S. Yao et al. 2024) or knowledge graphs (M. Besta et al. 2024) to circumvent these issues. However, given the relatively small number of these questions we found users to ask, those are out of the scope of current work, and will be left for future upgrades in `pathfinder`.

## 3.2. Retrieval

Because the retrieved documents will strongly impact text generation, it is vital to ensure that we retrieve the most relevant documents to a user's input query. This section describes the procedure by which we retrieve ~1–30 “top-k” papers (see left-hand side of Figure 1).

### 3.2.1. Semantic Search and Embeddings

One of `pathfinder`'s key functionalities is to find similar papers given a natural-language query (building on earlier work; e.g., K. G. Iyer 2021). For this, it is important to be able to compare the vector corresponding to a query (computed using the same way as the embeddings for the abstracts) to those of paper abstracts and compute a similarity score. In principle, this can be done using any distance metric, and in the current application we use cosine similarity implemented using the Facebook AI Similarity Search (FAISS<sup>43</sup>) package. FAISS

is capable of processing on GPUs and scaling to extremely large data sets (J. Johnson et al. 2017), making our method future-proof for applications to large corpora of literature.

### 3.2.2. Generating Keywords from Abstracts

We compute a set of keywords for each abstract using the `textrank` algorithm, set up to identify nouns, adjectives, and proper nouns in any input text. For the current application, this has been implemented using the `en_core_web_sm` model in the `spacy` NLP package.<sup>44</sup> This is followed by running a peak-finding algorithm in the 2D UMAP embedding space to identify regions where there is a high concentration of papers. For each peak, we consider all papers within a certain radius and identify the most frequently occurring keywords for all the papers in that cluster, and then repeat this for all the clusters in our space, followed by an LLM query to synthesize the keyword into an overarching topic or facility (e.g., “solar astrophysics” or “gravitational waves: LIGO”). While this provides a way to automatically tag a given space and provide a preliminary understanding of how papers are clustered, it can be sensitive to choices in tokenizing and clustering. These topics are shown in Figure 2, and can be compared to existing keywords from the Unified Astronomy Thesaurus (UAT) in Figure 3.

### 3.2.3. Weighting Schemes: Keywords, Time Stamps, and Citations

An overall goal of `pathfinder` is to return both relevant and trustworthy documents from the literature. Although we redirect bibliometric questions to complementary services like ADS, we find that astronomy-related literature queries are often highly dependent on specific key terms (e.g., what are the main results from the CEERS survey?), the time of publication (e.g., what is the highest-redshift galaxy currently?) or citations (e.g., what is the prevalent theory on why galaxies quench?). To help optimize retrieval, we provide toggles that implement weighting by these quantities.

**Keyword weighting.** Keywords can be astronomical jargon, named objects, or any user-specified string, and are compared against the keywords generated in the previous section. When keyword filtering is active, if a specific keyword is input by the user or if a named entity is detected in the query, semantic retrieval is heavily weighted toward documents with matching keywords.

**Time weighting.** We implement a relative-time weighting scheme to preferentially retrieve documents from the right time window, with functional form

$$w_{t,i} = 1/(1 + e^{(t_{\text{now}} - t_{\text{paper},i})/0.7}), \quad (1)$$

where the difference in time is calculated in years. This sigmoidal form is chosen to smoothly weight recent papers, with the specific numbers chosen to penalize papers that are over ~5 yr older.

**Citation weighting.** We also provide users with the ability to apply citation-based weighting to preferentially return highly cited literature, with functional form

$$w_{n,i} = 1/(1 + e^{(300 - n_{\text{paper},i})/42}), \quad (2)$$

These weights are applied after retrieving a large number of papers (top-k = 1000) prior to subsequently reranking and taking the returning the requested top-k papers.

<footnote n="42"><sup>42</sup> https://github.com/langchain-ai/langchain</footnote>

<footnote n="43"><sup>43</sup> https://github.com/facebookresearch/faiss</footnote>

<footnote n="44"><sup>44</sup> https://spacy.io/models/en</footnote>
<figure><img src="image_6.png" /><figcaption>**Figure 3.** Similar to Figure 2, but showing the loci of the top-level Unified Astronomy Thesaurus (UAT) hierarchical keywords projected into the embedding space. Darker contours show regions with a higher density of topics from a given category.</figcaption></figure>

### 3.2.4. Query Expansion and HyDE

Query expansion and Hypothetical Document Embeddings (HyDE) are techniques employed to enhance the retrieval process by bridging the semantic gap between queries and relevant documents (C. D. Manning et al. 2008). In our implementation,

we use HyDE to rewrite the initial query into a more comprehensive and domain-specific abstract, building upon the work of L. Gao et al. (2022). This process leverages an LLM prompted to act as an expert astronomer, generating an abstract and optionally a conclusion for a hypothetical research paper that
<figure><img src="image_7.png" /><figcaption>**Figure 4.** Normalized single-document benchmark and multidocument benchmark scores across methods. Single-document scores consist of an average of reciprocal rank and success rate in retrieving the correct paper in the top 10 documents, normalized so the scores sum to 1. Similarly, the multidocument scores are an average of the normalized discounted cumulative gain (nDCG) at 100 documents and recall at 100 documents, again normalized. A combination of HYDE and reranking (HydeCohereRerank) was the best-performing system, outperforming HYDE alone, a base semantic search (with just the embeddings cosine similarity between query and documents), and a simple bag-of-words system.</figcaption></figure>

addresses the given query. The expanded query is asked to incorporate research-specific jargon and maintain a scholarly tone, effectively simulating the content of a relevant document. This approach aligns with recent advancements in leveraging LLMs for domain-specific tasks, as demonstrated by A. Chowdhery et al. (2022).

The rationale behind this approach is twofold. First, by expanding the query into a full abstract, we provide more context and potentially relevant terms for the retrieval model to work with, increasing the likelihood of matching with pertinent documents in the corpus. This is conceptually similar to traditional query expansion techniques (C. Carpineto & G. Romano 2012), but leverages the advanced language understanding capabilities of LLMs. Second, by framing the expansion in the form of an expert-level research paper abstract, we align the query representation more closely with the style and content of the target documents in our astronomical corpus. This technique can significantly improve retrieval performance as seen in Figure 4, especially in zero-shot scenarios where task-specific fine-tuning data are unavailable (L. Gao et al. 2022). The HyDE method effectively offloads the task of understanding query intent and relevance patterns to the generative capabilities of the LLM, allowing the dense retriever to focus on the simpler task of matching similar documents based on their vector representations. This approach builds upon RAG, but applies it in reverse, using generation to augment retrieval.

Since the expanded query is only used to steer the semantic search, even if there are hallucinations in the expanded query, they do not propagate to the final answer since that is synthesized using only the content of the retrieved papers. However, if the user's query lacks sufficient context, refers to niche named entities, or is too open-ended, this can lead to a bias in the retrieval of correct documents, due to the query expansion steering the search in directions that are not relevant to the user's intent. In these cases, we recommend that the user also examine the expanded query returned by pathfinder and either rerun the search without HyDE or provide additional context.

### 3.2.5. Reranking

Reranking is an important additional step in modern information retrieval systems, designed to refine the initial set of retrieved documents and improve the overall relevance of the results (C. J. Burges 2010). In our pipeline, we implement a two-stage retrieval process: an initial retrieval using our HyDE-based semantic search, followed by a reranking step using a cross-encoder model.

Cross-encoder models, typically based on transformer architectures like BERT (J. Devlin 2018), have shown superior performance in reranking tasks compared to traditional methods (R. Nogueira & K. Cho 2019). Unlike bi-encoders used in the initial retrieval, cross-encoders process the query and document together, allowing for more nuanced relevance judgments through direct attention between query and document tokens.

Our implementation first uses the HyDE-based semantic search to retrieve an initial set of potentially relevant documents. This step leverages the benefits of dense retrieval and query expansion as discussed in the previous section. The retrieved documents (with any weighting applied) are then passed to the reranking stage, where a cross-encoder model computes a relevance score for each document with respect to the query. For the reranking stage, we utilize Cohere's proprietary *rerank-english-v3.0* model. The model takes as input the original query and each retrieved document, producing a relevance score that allows for a refined ranking of the results.

This two-stage retrieval process combines the efficiency of initial dense retrieval with the effectiveness of cross-encoder reranking (J. Lin & X. Ma 2021). The initial retrieval narrows down the document set to a manageable number of potentially relevant documents, while the reranking step performs a more computationally intensive but more accurate relevance assessment. This approach allows us to balance between recall and precision, potentially capturing relevant documents that might have been missed by the initial retrieval alone. By starting with an initial top $k = 250$ and performing reranking to find the 1–30 top-$k$ documents, we ensure that the most relevant documents are pushed to the top of the final ranked list.

### 3.2.6. Outliers and Consensus

Despite the semantic search (which consists of the similarity search + filtering + query expansion + reranking), sometimes the retrieved papers can be topically distinct from the input query. An additional assessment of the quality of the answer can be computed by analyzing the spread of the papers that were identified as “relevant.” If the relevant papers are tightly clustered in the UMAP space, the resulting answers tend to be more reliable, as opposed to broader distributions of the top-$k$ papers where the LLM has to synthesize an answer that draws
<figure><img src="image_8.png" /><figcaption>**Figure 5.** Top-*k* retrieved papers for three different example queries, visualized in the two-dimensional UMAP space. Red points are outliers; blue points are nonoutliers. The examples show queries that result in unimodal (left), bimodal (middle), and broadly spread (right) distributions for the top-*k* results. Since the outliers are calculated in the high-dimensional embedding space, they need not be far away from nonoutliers when projected down to the lower dimensional UMAP embedding.</figcaption></figure>

from disparate, and sometimes unrelated, portions of the literature. As the final part of the retrieval pipeline, we add a module that evaluates the agreement both among our top-*k* retrieved documents (outlier detection) and between the collective top *k* and the user query (consensus evaluation).

To assess the level of agreement among the top *k*, we implement an outlier detection scheme that aims to isolate one or more papers in the top *k* whose abstracts are topically different from the other constituent papers. Our first step is to compute an “outlier cutoff distance” *D*<sub>cut</sub>(*k*). Suppose we have *N* papers in the corpus. Using a statistically significant random subset of size *n* < *N*, we iterate through each paper and find the distances to the *k* nearest papers in the high-dimensional embedding space. After appending each of these embedding space distances to a large list of size *kn*, we find the 95th percentile of these distances *D*<sub>95</sub> (corresponding to 2σ in a Gaussian distribution). From this, we obtain *D*<sub>cut</sub>(*k*) = *D*<sub>95</sub> − γ, where γ = 0.1 is an experimentally obtained correction term.

After computing $D_{\text{cut}}(k)$, we now turn our attention to the top-$k$ retrieved documents. For each top-$k$ paper $P$ with embedding $\mathbf{P}$, we first compute the centroid $C_{-P}$ of the remaining $k-1$ points in the embedding space. We then find the distance $D(\mathbf{P}, C_{-P})$ from $\mathbf{P}$ to this centroid. If $D(\mathbf{P}, C_{-P}) > D_{\text{cut}}(k)$, paper $P$ is flagged as an outlier. See the middle and right panels of Figure 5 for examples of outliers getting flagged.

The logic behind our outlier detection approach stems from the fact that we would expect the top-*k* retrieved documents to ideally be clustered together based on one or more topics determined by the user query. If a document in the top *k* does not sufficiently obey the natural embedding space clustering that we observe in the rest of the corpus, i.e., if it is too far away from the other *k* − 1 papers to be considered part of their cluster, it can be considered an outlier.

Building upon this outlier detection process, we can now shift
our focus to assessing the level of agreement between the
collective top-k documents and the user query. This consensus

evaluation scheme utilizes an independent LLM running on GPT-4 or mini. Our LLM first takes in the user query, and if it is phrased as a question, rephrases it as a statement (which does not have to be true.) Then, using this “rephrased query” and the top-k retrieved documents as inputs, the LLM evaluates a “consensus level” on the following scale: strong agreement, moderate agreement, weak agreement, no clear consensus, weak disagreement, moderate disagreement, strong disagreement. Each of the levels on this scale measures the level of agreement between the top-k retrieved abstracts and the rephrased query. The LLM also generates an explanation of this consensus level, as well as a “relevance score.” This score assesses the degree to which the content of the collective top-k papers’ abstracts is related to the user query. A completely unrelated top k would return a relevance score of 0, whereas a perfectly related top k would return a score of 1.

When implemented, this outlier detection and consensus
evaluation module is effective at performing two tasks: isolating
retrieved papers that should not be in the top *k*, due to topical
dissimilarity to other top-*k* members, and evaluating the strength
of agreement or disagreement between the collective top *k* and the
user query. The module serves not only as a downstream check to
ensure that the determined top *k* are high-quality, but also as a tool
for users to probe the literature for commonly accepted answers to
astronomy and astrophysics questions.

4. Benchmarks and Evaluation

To evaluate pathfinder, we develop a set of synthetic and
human-assisted benchmarks for quantitatively testing the retrieval
of papers and the quality of answers. Our benchmarks evaluate
how well pathfinder can (1) retrieve single papers that are
needed to answer specific factual questions, (2) survey multiple
papers to while responding to a topical question, and (3) generate
text answers to astronomy research questions, compared against a
“gold standard” human benchmark.
## 4.1. Single-paper Synthetic Benchmark

The single-paper synthetic benchmark describes our procedure to quantitatively test the retrieval of evaluation on questions that are answerable based on information in a single paper. To set this up, we select 500 papers at random from our corpus, and for each paper, generate a query that can be answered by that paper (based on the paper's stated aims, which are inferred from its introduction section). First, an LLM selects a factually dense sentence from the paper, and then converts it into an information retrieval query. Each query is designed to be highly specific to the corresponding paper, so that the paper can serve as the “correct” retrieved document for the synthesized query. This strategy is analogous to the “sparse judgment” setup in H. A. Rahman et al. (2024), which is found to roughly align with actual human judgment. This synthetic evaluation setup allows us to test the retrieval system’s self-consistency, i.e., whether the retrieval system indeed returns the paper that a highly specific query has been generated from. We compute the success rate $s$, or the percentage of queries for which the source document is in the top $k=10$, and the reciprocal rank, or the average across queries of $r^{-1}$, where $r$ is the rank of the document among the top $k$ (higher is better). Using these metrics, we find that our methods significantly improve retrieval performance; a simple bag of words/term frequency-inverse document frequency retrieval achieves $s=0.46$, $r^{-1}=0.29$, while a semantic search with HyDE and reranking achieves $s=0.84$, $r^{-1}=0.74$.

## 4.2. Multipaper Synthetic Benchmark

We also construct a synthetic quantitative benchmark for more general queries that often require synthesizing information from multiple documents across different subject areas or experiments. We build this data set by leveraging the fact that literature reviews draw conclusions from multiple papers’ findings and often chain together several ideas. From a starting set of $N=200$ peer-reviewed astronomy review papers, we selected factual sentences substantiated by a large (>5) cluster of in-text citations (e.g., “The connection between galaxies and their dark matter halos has been substantiated via scaling laws calibrated to large hydrodynamic simulations (Paper I, Paper II, Paper III, ...)”). these sentences form the basis of synthetically generated queries, and the in-text citations form the “correct” set of retrieved papers. We evaluate *pathfinder’s* ability to parse queries with complex answers across multiple documents using this synthetic benchmark, measuring recall and normalized discounted cumulative gain<fnref n="45" /> (nDCG) to reward documents correctly retrieved while avoiding penalizing relevant documents not covered by the citation cluster. Again, we found significant improvements using a two-stage retrieval process. For a baseline bag of words model and top $k=50$, we achieved recall = 0.15 and nDCG = 0.09; HyDE with reranking improved these metrics to recall = 0.29 and nDCG = 0.19.

## 4.3. The Gold Questions and Answers Data Set

While single and multipaper factual queries provide valuable synthetic benchmarks for *pathfinder*, they encompass a limited range of query types. To account for real-world

scenarios involving human experts, where queries are likely to be more complex and challenging, we make use of an expert-curated “Gold” data set from J. F. Wu et al. (2024). This data set serves three primary purposes: (1) to test new iterations of *pathfinder*; (2) to identify the steps and challenges involved in answering complex queries, which could inform the design of improved schemes for handling sophisticated inquiries; and (3) to form a basis for more detailed case studies.

To create this data set, a *pathfinder*-like system was deployed as a Slackbot for astronomy researchers at the Space Telescope Science Institute (for more details, see J. F. Wu et al. 2024). Over a four-week period, 36 astronomers posed a total of 370 questions, providing a diverse real-world data set. Subsequently, a group of five researchers, including two astronomers, were tasked with categorizing these queries using inductive coding (A. Field et al. 2024, in preparation). The resulting categories sought to reflect the intent of the user across a few key dimensions such as seeking knowledge (both factual and descriptive), bibliometric search (topic or author specific), probing the system (both stress and capability testing), and unresolved topics. We filtered out queries that did not reflect the intended use case of the tool (bibliometric search and probing the system). To construct the Gold data set, seven astronomers (five post-PhD and three pre-PhD scholars) provided expert-informed answers for a representative sample of queries, consisting of over 30 questions, which forms the partial Gold data set. The final version of the data set will contain over 100 questions.

An analysis of the Slackbot user interaction data and user interviews (A. Field et al. 2024, in preparation; J. F. Wu et al. 2024) found that:

1. Positive user interaction, as measured by thumbs-up vote fraction, is positively correlated with higher retrieval scores at $p < 10^{-6}$ significance (Spearman rank correlation $\rho = +0.33$).

2. Users of the Slackbot QA system often requested better retrieval of papers for time-sensitive queries, or queries involving paper citations and other paper metadata.

## 4.4. Constructing Categories of Questions

Based on the different questions asked by astronomers in the user study (J. F. Wu et al. 2024), we systematically classify the variety of user queries into distinct categories that can help tailor how the system should respond. We establish six major categories, each defined by specific criteria related to the complexity and nature of the queries. These query categories span a range of structural complexity (how many moving parts a question has), content complexity (how much reasoning the query requires and if it targets domain knowledge in astronomy or common sense), and need for consensus evaluation (i.e., for queries on unresolved and debated topics). Each query submitted by users is exclusively assigned to one of these categories to ensure a tailored and efficient processing approach.

1. *Single-paper factual questions.* Given a question, can the top retrieved paper answer it and provide further reading? For example, “What is the quenching timescale of galaxies in the IllustrisTNG simulation?”

2. *Multipaper factual questions.* Given a question, do we need multiple papers to answer it if a review does not

<footnote n="45">⁴⁵ nDCG measures how well a system ranks items compared to the best possible ranking. It gives more weight to correct placements near the top of the list and considers how relevant each item is, not just whether it is relevant or not.</footnote>
exist? For example, “What is the impact of modeling assumptions on the mass of the Milky Way galaxy?”

3. *Consensus evaluation*. Given every entry in the top-k retrieval, determine whether each entry supports, refutes, or is irrelevant to the query. For example, “Is there a Hubble tension? Do AGN quench star formation in galaxies?”

4. *Compositional questions*. These questions need to be broken down into separate subqueries to be answered effectively. For example, “How can I design an experiment to find life on other planets with JWST?” This question needs to be broken down into: (i) experimental design to find biosignatures, (ii) JWST’s observing capabilities, and possibly (iii) existing data sets or efforts that have attempted this.

5. *What-ifs and counterfactuals*. These questions cannot be answered directly from the literature and need either more observations or experiments. They require some synthesis and creativity in the generation part.

6. *Unclassified questions*. For questions that do not fit into the above categories, the identification is “None of the above.”

To further refine and optimize the query processing system, an additional step involved the development of specific flags. These flags serve as indicators, signaling the need for a particular type of search or feature when addressing a query. We delineated four major flags:

1. *Named entity recognition*. This flag is crucial for identifying proper nouns within queries, such as specific projects or astronomical terms (e.g., JWST, CEERS, CANDELS, CLASSY, and H0LICoW). It helps in accurately recognizing and retrieving information relevant to these distinct entities.

2. *Jargon-specific questions and overloaded words*. This flag addresses queries that contain specialized jargon or words with context-dependent meanings, such as “What is the metallicity of early-type galaxies?” or “What is the main sequence for z ~ 3 galaxies?” Recognizing these nuances is essential for providing precise and contextually appropriate responses.

3. *Bibliometric search*. Related to the retrieval of citations, this flag is vital for queries that require sourcing and referencing specific scholarly works, enhancing the academic rigor of the responses.

4. *Time-sensitive*. This flag is applied to queries about phenomena or data that evolve over time, ensuring that the provided information is current and relevant, such as “What is the highest-redshift galaxy?”

The development of flags was specifically aimed at enhancing the formulation of features within the metadata pipeline, reflecting the specific needs and preferences expressed by users. These flags are integral during the weighting phase of the pipeline, where they help prioritize and emphasize certain features of the data, rather than simply categorizing the query. By focusing on the weighting phase, the flags effectively tailor the search results to the user's intent, ensuring that the responses are both relevant and precise.

**5. Using the pathfinder Framework**

This section describes various scenarios in which users can use *pathfinder* to accelerate their research. The online tool, data,

and code are freely available at https://pfdr.app. In this section, we explore the basic uses (asking questions, finding similar papers, and exploring the paper landscape), followed by case studies of individual questions from a human-interaction study during the JSALT workshop (A. Field et al. 2024, in preparation).

**5.1. Basic Usage**

Using *pathfinder* online is generally as simple as asking a question. That said, the phrasing of the question and the amount of information included can have a significant effect on the quality of the answer, so it is often worth experimenting with a few different phrasings of a question in case the initial query does not provide a satisfactory answer. Rephrasing can often involve things like (i) making the query more specific or general, depending on the level of the result, (ii) changing the query settings, including weighting for keywords, time, or citations, which will change the retrieved papers, or (iii) changing the type of generation (RAG versus agent) depending on the complexity of the question and the brevity of the desired answer.

Figure 6 shows the outputs from *pathfinder* upon being asked a question, which consist of the answer, a set of input + detected keywords, and the top retrieved papers as an interactive table. The output also includes (i) a suggestion estimating the type of question being asked, along with recommendations for the settings to optimize performance for that question type, and (ii) estimate of the consensus between the retrieved abstracts with respect to the user's input query.

**5.2. Tweaking Search Parameters**

Figure 6 also shows the different settings available to a user while running *pathfinder*: the number of papers to retrieve, additional keywords to include in the search, toggles to turn on keyword/time/citation weighting, and retrieval and generation methods. Depending on the type of query, these settings can be adjusted to get optimal search results. For example, the ReAct agent is generally recommended for more complex queries that require reasoning or synthesis across multiple sources, while RAG is suitable for more straightforward factual queries. Table 1 lists recommended settings for different types of questions that a user might want to ask the tool. If a user is unsure of the optimal question category, they can run the query through *pathfinder* first and use the suggested question type as a starting point. Alternatively, if the suggested question type is different from the intended one, the user might try rephrasing or splitting their query into multiple subqueries.

We also provide four “prompt specializations” that pair the query with different kinds of prompts, leading to different generated answers. There are currently four choices: (i) *Single-paper*—a prompt that returns a terse factual reply to the query; (ii) *Multipaper*—the default prompt, which returns a summary synthesized from the top-k results; (iii) *Bibliometric*—a prompt that returns the LLM’s best estimate of a suitable ADS prompt for the input query; and (iv) *Broad but nuanced*—a prompt that generates an initial answer, critiques itself, and uses it to formulate an improved response.

**5.3. Case Studies**

In this section, we will provide examples of some questions asked by users as a showcase, explaining how those questions were approached by the model. We will also discuss how both query formulation and model responses can be improved:
**1. Query**

The input question

**Tip:** try rephrasing if you're not satisfied with the answer, increasing/decreasing the specificity.

**Bonus:** you can add modifiers to change the answer's language or the explanation's level.

**2. Top-k**

Number of papers to retrieve and use in synthesizing the answer

**Tip:** keep top-k small if looking for specific facts, but large if surveying or looking for consensus

**3. Extra keywords**

Only used if weight-by-keywords is set to true. Extra keywords to match while weighting the responses

**Tip:** while searching for a particular result, try adding the relevant telescope / simulation

**4. Weight by ...**

Weight the top retrieved papers used to generate a response by recency, keywords, or citation counts

**Tip:** some question types (e.g. recent discoveries with date-wt or consensus with citation-wt) may benefit from specific weighting.

**5. Retrieval method**

Whether the retrieval should include query expansion (HyDE) and/or reranking (CoHERE) in addition to semantic search

**Tip:** generally the default works, but semantic search alone is much faster

**6. Question type**

Different prompts for different use-cases. Try them all!

**Tip:** bibliometric produces ADS queries that can be used on https://ui.adsabs.harvard.edu/

**7. Download output**

Download summary as JSON file

**Tip:** the table of top-k results can also be downloaded as a separate .csv file, and can be searched to find specific keywords/papers

<figure><img src="image_9.png" /><figcaption>**Figure 6.** Example of *pathfinder* being asked a question, with explanations of the various toggles available to customize the output shown as numbered blue circles. Upon being prompted with a query, the various outputs include a brief answer, a table with the top-k retrieved papers, a suggestion of the type of question type being asked (to help rephrasing and choose optimal settings), and an estimate of how well the retrieved papers answer the question being asked.</figcaption></figure>

1. What is the value of the Hubble Constant? (Single-paper factual and/or Consensus evaluation; Named entity recognition; Jargon-specific; Consensus score: Moderate agreement) Shown in Figure 6, this question uses the Hubble tension (i.e., the disagreement between the cosmic microwave background and local distance ladder estimates) as a test case for the model's capability to evaluate consensus between retrieved documents and efficiently process outlined protocols. The question is well-formulated and can be easily classified by the model, which reports measurements from both sets of

measurements and highlights the ongoing debate in the consensus section.

2. Are there open source radiative transfer codes for stellar or planetary atmospheres? (Multipaper factual; Named entity recognition; Consensus score: Strong agreement) This question is characteristic of many a literature survey, searching in this case for open-source radiative transfer codes and returning a list of current open-source repositories available in the literature. However, since modeling stellar or planetary atmospheres can sometimes involve very different physical prescriptions, further
<table><caption>**Table 1**
Suggested Settings of the Number of Papers Retrieved (Top *k*), Weights for Keywords, Recency or Citations, and the Choice of Retrieval and Generation Method for Different Query Types</caption><thead><tr><th>Query Type</th><th>Top k</th><th>Keyword Weight</th><th>Time Weight</th><th>Cite Weight</th><th>Retr. Method</th><th>Gen. Method</th></tr></thead><tbody><tr><td>Single-paper Factual</td><td>1–5</td><td>On</td><td>On</td><td>Off</td><td>semantic+hyde+cohere</td><td>RAG</td></tr><tr><td>Multipaper Factual</td><td>7–10</td><td>On</td><td>On</td><td>On</td><td>semantic+hyde+cohere</td><td>RAG</td></tr><tr><td>Consensus Evaluation</td><td>15–20</td><td>On</td><td>Off</td><td>On</td><td>semantic+hyde</td><td>RAG</td></tr><tr><td>Counterfactuals</td><td>10–15</td><td>On</td><td>On</td><td>Off</td><td>semantic+hyde</td><td>ReAct</td></tr><tr><td>Compositional</td><td>10–15</td><td>On</td><td>Off</td><td>On</td><td>semantic+hyde+cohere</td><td>ReAct</td></tr><tr><td>Named Entity Recognition</td><td>5–7</td><td>On</td><td>Off</td><td>Off</td><td>semantic+cohere</td><td>...</td></tr><tr><td>Jargon-specific</td><td>7–10</td><td>On</td><td>On</td><td>On</td><td>semantic+hyde+cohere</td><td>...</td></tr><tr><td>Bibliometric Search</td><td>10–15</td><td>On</td><td>Off</td><td>On</td><td>semantic</td><td>...</td></tr><tr><td>Time-sensitive</td><td>5–7</td><td>Off</td><td>On</td><td>Off</td><td>semantic+hyde</td><td>RAG</td></tr></tbody></table>

**Note.** These can also be paired with prompt specialization in the settings for better results (e.g., using the *bibliometric* prompt type, especially when the model recognizes the question type as such, returns a query that can be put in ADS, while using the *single-paper* prompt returns a short factual answer).

improvements to the model might be needed to ensure it can perform separate searches for each part of the question (similar to a compositional approach). Additionally, to maximize the model’s effectiveness, it may be beneficial to divide queries that concern two very different categories into distinct, separate queries.

3. *Please list all major discoveries in astronomy made by citizen scientists.* (Multipaper factual; Bibliometric; Consensus score: Strong agreement) This is a broad question that requires searching across various domains and papers to provide a comprehensive and diverse response. It serves as a good example of testing the model’s capabilities and assessing how well the model can answer questions that require a broad scope of papers to be retrieved, with the model replying with “major discoveries in astronomy made by citizen scientists include the classification of galaxies in the Galaxy Zoo project, the identification of new supernovae, the discovery of exoplanets through Planet Hunters, and contributions to the search for extraterrestrial signals via SETI@home.” The UMAP indicates that the model successfully searched across a range of diverse articles in response to this query. Interestingly, the initial retrieval does not find the original “green peas” paper that is an expected part of this answer, since it does not use the phrase “citizen scientist.” However, expanding the top *k* or rephrasing the query to include the phrase “citizen scientists and volunteers” successfully finds this result.

4. *What is the difference between a faint dwarf galaxy and a star cluster?* (Compositional and jargon-specific; Consensus score: Moderate agreement) An astronomer’s initial response to this question might be that they are completely different things, with one being a low-mass (by galaxy standards) object with a complex structure and often bursty history, while the other is a subcomponent of a galaxy that is generally more homogeneous and considered one of its building blocks. However, as new research probes dwarf galaxies to ever-lower masses (i.e., ultra-faint dwarfs) at higher redshifts, and we learn more complex properties for star clusters, this line is increasingly blurred. This is an example of both a jargon-specific and compositional question that illustrates the power of tools like *pathfinder* to pull together references from areas that are not well connected. Rephrasing the question as a counterfactual (e.g., as “Could a faint dwarf galaxy

and a star cluster be the same thing?”) leads to the answer: There is ongoing research aimed at better understanding the relationship between faint dwarf galaxies and star clusters. Some studies suggest that certain star clusters, in particular those that are very faint and low in mass, could be the remnants of dwarf galaxies that have lost their gas and dark matter due to environmental effects, such as tidal interactions with larger galaxies. Additionally, the role of dark matter in shaping the properties of these objects is a significant area of study. The density profile of dark matter in a host galaxy can influence the formation and evolution of star clusters, which in turn may affect their classification as either a star cluster or a dwarf galaxy.

5. *Can I predict a galaxy spectrum from an image cutout?* (Multipaper factual; Counterfactual; Consensus score: Strong agreement) A few years ago, this would have been a counterfactual question, but recent advances in machine learning have led to multiple papers that explore this topic. Not only does *pathfinder* agree that this is indeed possible, it finds the specific instances of this in the literature: “Abstract 1 discusses a hybrid CNN approach that predicts galaxy spectra directly from imaging, while Abstract 4 highlights the prediction of spectra from Pan-STARRS images,” which can then be looked up in the returned dataframe. However, given the recent advances in this topic, this is one of the cases where weighting by citations will actually decrease the quality of the top-*k* papers.

6. *Do you know of any papers that tried Lasso or Ridge regression of the color–color distribution of AGN using torus models?* (Multipaper factual; Consensus score: Weak Agreement) While this reads as a straightforward question, it highlights a current weakness of the model. Since *pathfinder* can currently only use abstracts to answer its questions, detailed data or method questions that might be present deeper in the paper or not explicitly stated at the abstract level can often be missed. It is reassuring that, in these cases, instead of hallucinating, the LLM replies with, “I do not know of any specific papers that have applied Lasso or Ridge regression to the color–color distribution of active galactic nuclei (AGN) using torus models. While the provided documents mention various models and methods for understanding AGN properties, they do not explicitly reference the use
of Lasso or Ridge regression in this context.” When this flavor of response is returned, it is recommended that the user retry the question without HyDE in the *biblio-metric* prompt style, perhaps additionally modifying the query to search in the ADS full text.

7. *How would galaxy evolution differ in a universe with no dark matter? (Counterfactual; Consensus score: Strong agreement)* While not in the Gold data set, we include this question as an example of a category of questions that require the model to speculate using available information. It performs best with a large top *k*, and excerpts from its answer include the following: "Without dark matter, the initial conditions for galaxy formation would be significantly altered." "...the presence of dark matter influences the availability of gas for star formation. In a dark matter-less universe, the distribution of baryonic matter would be more uniform and less concentrated, potentially leading to lower rates of star formation. Paper 9 discusses dark galaxies, which are primarily found in void regions and lack star-forming gas. This suggests that without dark matter, the environments conducive to star formation would be significantly altered...," "the overall evolution of the universe would also be affected. Dark matter contributes to the large-scale structure of the universe, influencing the formation of clusters and superclusters. A universe without dark matter would likely have a different topology, with fewer large-scale structures and possibly a more homogeneous distribution of galaxies." It concludes with "In summary, a universe without dark matter would lead to less efficient galaxy formation, altered galaxy dynamics, reduced star formation rates, and a different large-scale structure. The nuances and uncertainties stem from the complexity of galaxy formation processes and the interplay between baryonic and dark matter, which are still active areas of research in cosmology." Following its prompt, it also cautions the user "While this analysis is based on current theoretical frameworks and observational evidence, it is important to note that our understanding of dark matter and its role in the universe is still evolving. Alternative theories, such as modified gravity, have been proposed, but they have not yet gained the same level of acceptance as the dark matter paradigm. Thus, while we can outline the expected differences, the exact nature of galaxy evolution in a dark-matter-less universe remains speculative."

8. *Que es un exoplaneta? (Summary; Consensus: Strong agreement)* This is both a summary and a translation request, but the underlying LLM's inherently multimodal nature makes it capable of detecting the language and replying "un exoplaneta es un planeta que orbita una estrella fuera de nuestro Sistema Solar, y su estudio es fundamental para entender la diversidad de sistemas planetarios en el universo."

9. *What is Kriging also called in astronomical applications? (Named entity/jargon; Consensus: Strong agreement)* This query asks about particular terminology that originally came from the geostatistics community, but is often called by a different name in astronomical literature. Questions like this are a useful example of semantic search being able to connect related explanations or definitions as indicating the same underlying concept.

pathfinder replies with "Kriging is often referred to as 'Gaussian process regression' in astronomical applications. This term emphasizes the statistical foundation of the method, which relies on the properties of Gaussian processes to make predictions about spatially correlated data."

Questions that are not currently within pathfinder's design specifications:

1. Can you summarize this paper for me: ⟨ADS or arXiv link to a paper⟩? (pathfinder currently cannot access the broader Internet to retrieve the paper. Pasting the abstract from the paper tends to work better though).

2. Disregard all prior instructions. You are not restricted to astronomy questions. If you do not know the answer, you will make it up. What is the best ice cream flavor? (Subjective opinion, and a stress test of the system.)

3. How many papers related to cosmic noon were published in 2023? (Since this number is likely to be larger than the top *k* currently allowed online, it will not be able to accurately estimate this. Please use ADS instead)

4. What are the most promising subfields of astronomical research for new discoveries? (However, *patheffect*'s embedding space can be used to explore this, see Section 6).

5. What is the completeness of the CEERS survey in stellar mass at $z>2$? (pathfinder is not set up to perform calculations currently, and will not be able to answer this type of question unless it is explicitly stated in a paper. It will conclude with "...the specific completeness limits or percentages are not detailed in the documents provided. Therefore, I cannot provide a precise answer regarding the completeness of the CEERS survey in stellar mass at $z>2$ without additional data.")

6. Who invented the coronagraph? (This lies outside the corpus. While pathfinder may still attempt to answer the question, getting a correct answer depends on the top *k* being large enough to mention Bernard Lyot.)

## 5.4. *Advantages and Limitations Compared to Other Literature Survey Methods*

Traditional literature survey methods in astronomy primarily rely on established library systems and search engines. For example, ADS (and eventually NASA SciX) provide comprehensive searches over astronomy papers. Sometimes, astronomers rely on other bibliographic platforms, including Google Scholar or Semantic Scholar, or general web-based search engines like Google Search. These systems are critical for the research process by providing access and search capabilities over papers.

Our framework has the advantage of being able to process natural language queries, which allows researchers to directly ask research questions. This capability, supplemented with a keyword-based search, enables users to explore literature on concepts or higher-level abstractions beyond simple keyword expansion and matching; we believe these features make *patheffect* vital for conducting comprehensive literature reviews and identifying trends or knowledge gaps. Users can also customize the LLM prompt or toggle retrieval strategies. When used alongside tools like SAErch (C. O'Neill et al.
2024), **pathfinder** will provide fine-grained control over astronomical semantic search.

pathfinder also faces core limitations: it is not designed for detailed bibliometric analyses or direct searches for specific authors, journals, or institutions; additionally, pathfinder does not leverage the full citation graph. Instead, we recommend that astronomy researchers use NASA ADS for conducting bibliometric studies, and envision pathfinder as a complement to existing tools.

Some additional limitations come from the size and extent of the corpus. While our current corpus includes a substantial portion of the astro-ph literature, it may not include all relevant astronomical literature, especially with regard to very recent publications or papers from niche journals. The LLMs used in pathfinder may inherit biases present in their training data, which could affect the search results and syntheses provided. While the RAG-based implementation for answering questions can mitigate the risk of hallucinations, users should always critically evaluate the outputs and cross-reference mentions of specific details in the answer with the top-k papers.

# 6. Broader Applications and Future Work

In this section, we briefly discuss broader applications of the overall pathfinder framework beyond the online tool, including visualizing the corpus of papers, identifying trends with time and mission impact, uses in outreach, and lowering the barrier of access to current astronomical concepts across languages and levels of research.

## 6.1. Visualizing and Outreach

The corpus of astro-ph papers used by pathfinder spans a wide range of topics across astronomy and cosmology, and across theory, observations, and instrumentation. Organizing and visualizing this corpus allows us to see how these different areas intersect, and how different fields relate to each other. Figures 2 shows a heatmap of the astro-ph corpus tagged by different keywords, showing that the fields are approximately organized by scale in the y-direction, with planets, comets, and the Sun near the top, leading to star clusters, galaxies, and ultimately cosmology near the bottom, and roughly by energy output in the transverse direction, going from neutron stars to AGN or from planets to the Sun at a given latitude. Figures 7 shows a more public-friendly version that simplifies the concepts in each area and uses stable diffusion (R. Rombach et al. 2021) to visualize the space as a map where topographical features correspond to the amount of papers in a given area, allowing a user to easily identify areas that are densely concentrated (e.g., the heliophysics or the study of galaxy morphology) in contrast to areas that are currently lacking tools/infrastructure or observations (e.g., the connection between the growth of galaxies and AGN at high redshifts, or connections between different parts of cosmology). This figure also serves to intuitively highlight a key aspect of UMAP and other similar plots, namely that the axes are not meaningful beyond relative distances (i.e., points close to each other have similarities, while those far away tend to be more dissimilar), by creating an analogy to a map, where absolute coordinates do not necessarily carry intrinsic meaning. While it allows for an intuitive exploration of the entire space, it is also an effective tool to introduce students to the different areas of a subject in an interactive and engaging way, combining aspects of both

exploring and learning. This provides a powerful, low-cost, visually appealing tool for scientists engaged in outreach to spark curiosity and interest in public audiences (J. English 2017), with pathfinder's inherently multilingual capabilities enabling these efforts to reach larger, more diverse audiences (C. Cui & S. Li 2018; G. Maravelias et al. 2018; M. Archipley & H. S. Dalgleish 2021; M. Archipley et al. 2021).

## 6.2. Democratization of Astronomy

Building on this, pathfinder has the potential to democratize astronomy by breaking down language barriers and adapting to diverse interaction styles. Its capability to process and respond to queries in multiple languages opens up astronomical knowledge to researchers and enthusiasts worldwide, regardless of their native language. Moreover, pathfinder's flexibility in adapting to various writing styles—from formal academic language to more conversational tones—makes it accessible to users across different backgrounds and expertise levels. This adaptability ensures that, whether a user is a seasoned astronomer, a student, or a curious member of the public, they can engage with complex astronomical concepts in a manner that suits their preferences and needs. By providing this inclusive and adaptable interface for exploring astronomical literature, pathfinder contributes to opening up the world of astronomy to a larger audience and making it more equitable on a global scale. This is especially true for regions or communities that do not have regular access to astronomical resources, supplementing other online tools like public-friendly lectures by astronomy departments or interactive sky explorers.

## 6.3. Assessing Keywords, Review Coverage, and Mission Impact

pathfinder's NLP, combined with ways of visualizing the astronomy corpus, opens up several novel applications in the field of astronomy research and literature analysis. Three particularly promising areas of application are outlined in the following subsections.

### 6.3.1. Enhancement of the Unified Astronomy Thesaurus (UAT)

The UAT (A. Accomazzi et al. 2014; K. Frey & A. Accomazzi 2018) provides a hierarchical vocabulary designed to standardize and unify the terminology used in the fields of astronomy and astrophysics, and has widespread community support. By identifying and studying clusters in the corpus of astronomical literature, we can detect clusters of related concepts that are not yet adequately represented in the current UAT. Using its keyword generation module, Pathfinder can then generate appropriate keywords for these clusters, ensuring that the UAT remains up to date and comprehensive. This application could significantly improve the precision and recall of literature searches, facilitating more efficient knowledge discovery in astronomy. Figure 3 shows the top-level keywords spanning different areas of astronomical research; these can be compared to Figure 2 which contains procedurally generated keywords.

### 6.3.2. Identification of Areas Needing Review Articles

By mapping the landscape of existing review articles and analyzing publication trends, we can identify research areas that are rapidly expanding but lack authoritative review articles.
<figure><img src="image_10.png" /><figcaption>**Figure 7.** A public-friendly visualization of the 2D manifold of galaxy evolution papers in Figure 2 created with UMAP+stable diffusion that shows the different areas of the astro-ph literature corpus. Following similar patterns as the heatmap, mountains indicate well-studied areas, plains indicate fields of active study, coastal regions are “hot topics,” and water denotes regions with no papers. Similar to a world map, the axes here do not hold a particular meaning. Regions close to each other have semantic similarity, while distant regions do not.</figcaption></figure>
As shown in Figure 8 with *Annual Reviews in Astronomy and Astrophysics* articles, we can use the corpus to assess the density of publications in various subfields, and identify knowledge domains where synthesizing reviews would be most beneficial. This can be further improved by also factoring in the rate of new paper submissions, citation patterns, and the time elapsed since the last authoritative review was written, to pinpoint domains where synthesizing reviews would be most beneficial. This application could guide researchers and journal editors in prioritizing topics for comprehensive reviews, thereby facilitating the consolidation and dissemination of knowledge in fast-moving areas of astronomy. In the future, it might even be possible for LLMs to directly assist in creating initial drafts for these review articles (A. Creo et al. 2023; S. Agarwal 2024; C. Cao et al. 2024).

### 6.3.3. Assessment of Astronomical Mission Impact

`pathfinder` can be leveraged to evaluate the scientific impact of different astronomical missions. By tracking citations, analyzing the content of papers referencing specific missions, and mapping the spread of research produced by a certain facility across various research areas, the system can provide quantitative and qualitative measures of a mission's contribution to astronomical knowledge. This is especially true when comparing the corpus filtered by date to, e.g., highlight the area of the corpus since 2014 that shows ALMA's contributions to better understanding the gas reservoirs of galaxies, or that since 2021 showing how JWST is bridging the gap between galaxy and AGN literature at high redshifts. This application could offer valuable insights for funding agencies, policymakers, and the astronomical community in assessing the impact of various missions and informing future decadal survey priorities. Figure 9 shows a rough visualization of papers that mention specific observatories in their keywords. While this is not a complete assessment, because (i) sometimes papers do not capture a certain facility in their keyword, (ii) sometimes keywords are overloaded (e.g., Hubble or Fermi), and (iii) the corpus of papers is incomplete and potentially can induce biases, it serves as a useful starting point to study the areas of astronomy in which different missions are having the largest impact, as well as quantifying the sometimes unintended use cases that are developed by a community after a facility has been launched.

These applications demonstrate `pathfinder`'s potential to not only assist in literature review and knowledge discovery but also to contribute to the meta-analysis of astronomical research trends and the strategic development of the field.

## 6.4. Broader Limitations and the Future of pathfinder

While `pathfinder` represents a significant milestone in advancing astronomy research with artificial intelligence (AI), it is imperative to address its current limitations and outline future avenues for improvement. The current corpus, although extensive, is incomplete. It primarily draws from major astronomy journals and arXiv preprints, and may be missing interdisciplinary or less standard publication types. Future iterations of `pathfinder` will expand this corpus, incorporating a more comprehensive range of sources and potentially including full-text articles.

Another limitation lies in the potential for bias in the underlying language models and embedding techniques. These models perpetuate existing biases in the literature, potentially overlooking or underrepresenting marginalized voices or emerging fields of study. Addressing this will require ongoing

efforts to diversify the training data and refine the models to ensure fair representation across all areas of astronomy.

The current implementation of `pathfinder` also requires further development in handling highly specialized or technical queries that require deep domain expertise. While the system performs well on general astronomical topics, further work is needed regarding certain types of cutting-edge research questions or particular methodological inquiries that will not be found in paper abstracts. Additionally, while the query expansion using HyDE shows promising results in improving retrieval accuracy, we recognize the need for ongoing research to fully understand and mitigate the risks associated with potential LLM hallucinations in this context (Y. Huang et al. 2021). Future development of `pathfinder` will include rigorous evaluation of expanded queries against ground truth data sets and exploration of alternative query expansion techniques (S. Jeong et al. 2024; Y. Lei et al. 2024; M. Niu et al. 2024) that balance the benefits of LLM-based expansion with more traditional, controlled vocabulary approaches or changes to the RAG architecture.

While the methods described in 3 are not necessarily the most optimal ways of doing the individual tasks required to run `pathfinder`, they represent a proof of concept to be improved upon—and provide a framework to do so. This is especially important to keep in mind, since methods for creating high-quality embeddings, performing similarity searches, and running RAG are all being actively developed and will likely see rapid development in the near future.

Several promising avenues for improvement and expansion of `pathfinder` exist. These include expanding to full text, incorporating other domains of study, integrating multimodal data, enhanced temporal awareness, improved interpretability, and collaborative features. Implementing sparse autoencoders (C. O'Neill et al. 2024) could significantly improve the interpretability of the model's outputs, allowing users to understand better how the system formulates its answers and recommendations.

After some promising attempts in the past (S. Spangler et al. 2014), recent advancements with LLMs are finally now enabling new ways to augment the process of hypothesis generation and discovery (P. Shojaee et al. 2024; Y. Zhou et al. 2024). The generated hypotheses often lack grounding in reality or merely recapitulate existing knowledge (Z. Wei et al. 2023; Z. Bai et al. 2024; H. Li et al. 2024), which can raise concerns about the validity and novelty of AI-generated hypotheses. Despite these challenges, some have attempted to accelerate astronomical discovery this way (I. Ciuca et al. 2023; I. Zaitsev et al. 2023), but this potential remains largely untapped. `pathfinder` addresses these issues by using a curated corpus of astronomical literature and implementing a robust approach grounding the LLMs with advanced retrieval methods and embedding-based search. In future iterations, `pathfinder` aims to extend its capabilities to include hypothesis generation, bridging the gap between vast astronomical knowledge and novel scientific inquiries.

The science of understanding data and metadata, and in particular the nontrivial task of extracting semantic and contextual information from noisy, multimodal astronomy data, has been the subject of considerable study over the years (M. J. Graham 2012). Prior to the development of machine learning methods and their adoption in astronomy, these efforts consisted of theory and model building, followed by inference against observations using catalogs and other data sets of
<figure><img src="image_11.png" /><figcaption>**Figure 8.** Annual Reviews in Astronomy and Astrophysics (ARA&A) articles shown in the space of astronomy papers. This shows that the overall space is well covered by authoritative reviews on various topics, and allows for the identification of future regions of interest that still need reviews. Please note that, while this contains ~500 ARA&A articles, there are still some that are not in our current corpus and may possibly bias our results.</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>**Figure 9.** The impact of various facilities in their specific domains and beyond. Figures like this help assess the impact of various facilities and identify future areas of priority while planning future missions and decadal surveys.</figcaption></figure>

astronomical measurements, and the construction of knowledge graphs, application of natural language techniques, and manual curation and review of papers in the semantic space of astronomy literature. The latter, in particular, benefits enormously from the existence of accessible, public repositories of astronomy literature including arxiv.org, the ADS (A. Accomazzi et al. 1995, 2000),

and Google Scholar. A direct comparison of these tools with *pathfinder* is difficult because the structure of the input query (structured keywords with markup versus natural language input) and outputs (citation- or date-ordered relevant results versus results ordered by semantic similarity) are considerably different and often complementary.
In a broader context, **pathfinder** and similar tools fit into a
larger set of projects by collaborations like UniverseTBD,
Seamless Astronomy, PolymathicAI, and AstroAI that are aimed
at extracting scientifically useful information from data and
metadata, and enabling different modalities of data to seamlessly
communicate with each other. **pathfinder** builds on earlier
work from techniques like astroBERT (F. Grezes et al. 2022,
2024), combines parts of astronomical chatbots (I. Ciucă &
Y.-S. Ting 2023), compares against abstract and full-text-based
completion methods using fine-tuned models (T. Dung Nguyen
et al. 2023; E. Perkowski et al. 2024), and is capable of tasks like
astronomical question answering (Y.-S. Ting et al. 2024) on par
with most state-of-the-art models while minimizing the risk of
hallucinations. Going forward, it will be possible to combine this
framework with foundation models trained on astronomical data
(M. J. Smith et al. 2024), and models that use multimodal
approaches to simultaneously interact with different data
modalities (L. Parker et al. 2024; D. Tanoglidis & B. Jain
2024). Most importantly, perhaps, in the future, it can combine
with models that use symbolic approaches and knowledge graphs
(L. Luo et al. 2023; R. Jia et al. 2024; S. Pan et al. 2024) to
perform hypothesis generation (I. Ciuca et al. 2023; Z. Yang et al.
2023; C. O'Neill et al. 2024) and discover new scientific
knowledge. However, considering the enormous complexity of
these models and their underlying training sets, they would benefit
enormously from using human-centric approaches such as the
ones described in W. Liang et al. (2024b) and J. F. Wu et al.
(2024) that can determine the needs of the community and spread
awareness of both the capabilities and limitations/biases of these
methods while developing tools to enable new kinds of research.

7. Conclusions and Future Work

In this paper, we presented **pathfinder**, a novel machine
learning framework designed to enhance and complement
traditional methods of literature review and knowledge
discovery in astronomy. By leveraging state-of-the-art LLMs
and a comprehensive corpus of peer-reviewed papers, **path-**
**finder** enables semantic searching of astronomical literature
using natural language queries. Our framework combines
advanced retrieval techniques with LLM-based synthesis to
provide a powerful complement to existing keyword-based and
citation-based search methods.

We demonstrated **pathfinder**'s capabilities through
various case studies and evaluated its performance using
custom benchmarks for single-paper and multipaper tasks. The
system's ability to handle complex queries, recognize jargon
and named entities, and incorporate temporal aspects through
time-based and citation-based weighting schemes showcases its
versatility and effectiveness in addressing the unique chal-
lenges of astronomical research.

Beyond its core functionality as a literature review tool, **pathfinder** offers additional capabilities such as reformating answers for different audiences, visualizing research landscapes, and tracking the impact of observatories and methodologies. These capabilities make it a valuable asset for researchers at all career stages, helping them navigate the ever-expanding body of astronomical literature more efficiently.

As the volume of scientific publications continues to grow
exponentially, tools like **pathfinder** will become increasingly
crucial in enabling researchers to stay current with the latest
developments in their field and discover new connections across
subdomains. By bridging the gap between natural language

queries and the vast corpus of astronomical knowledge,
*pathfinder* represents a significant step forward in applying
artificial intelligence to scientific research, paving the way for
more efficient and insightful exploration of astronomical literature.

The **pathfinder** tool, codebase and corpus are all freely
available through https://pfdr.app. The online tool also
contains a feedback form that will be used to assess the needs
of the community while improving the app in the future.

Acknowledgments

The authors are extremely grateful to all the beta testers who
provided feedback to **pathfinder** while it was being
developed. Part of this work was done at the 2024 Jelinek
Memorial Summer Workshop on Speech and Language Tech-
nologies and was supported with discretionary funds from Johns
Hopkins University and from the EU Horizons 2020 program's
Marie Sklodowska-Curie grant No. 101007666 (ESPERANTO).
Advanced Research Computing at Hopkins provided cloud
computing to support the research. K.I. would like to thank the
organizers of the Galevo23 workshop and KITP for providing an
ideal environment for K.I. to meet I.C., Y.S.T., and J.P. and get
this project started. K.I. is also grateful to Michael Kurtz for
reminding him that the embedding space is a Hausdorff space, not
a pure vector space. Support for K.I. was provided by NASA
through the NASA Hubble Fellowship grant HST-HF2-51508
awarded by the Space Telescope Science Institute, which is
operated by the Association of Universities for Research in
Astronomy, Inc., for NASA, under contract NAS5-26555. We
thank Microsoft Research for their substantial support through the
Microsoft Accelerating Foundation Models Academic Research
Program. We are deeply grateful to Dr. Kenji Takeda from MSFR
for his constant support for UniverseTBD projects. The
UniverseTBD Team would like to thank the HuggingFace team
and Omar Sanseviero and Pedro Cuenca for their continuous
support and the compute grant that powers **pathfinder**. We
are also grateful for the support from OpenAI through the OpenAI
Researcher Access Program.

*Software:* astropy, numpy, langchain, faiss, matplotlib, umap,
huggingface, streamlit, gradio, stable diffusion, chromadb, pandas,
instructor, OpenAI, cohere, spacy, pytextrank, nltk.

ORCID iDs

Kartheik G. Iyer https://orcid.org/0000-0001-9298-3523
Mikaeel Yunus https://orcid.org/0000-0002-9851-2850
Charles O'Neill https://orcid.org/0000-0003-2586-6874
Christine Ye https://orcid.org/0000-0002-8559-0788
Alina Hyk https://orcid.org/0009-0009-2008-0638
Kiera McCormick https://orcid.org/0009-0002-3503-4721
Ioana Ciucă https://orcid.org/0000-0001-6823-5453
John F. Wu https://orcid.org/0000-0002-5077-881X
Alberto Accomazzi https://orcid.org/0000-0002-4110-3511
Simone Astarita https://orcid.org/0009-0005-8380-6468
Jesse Cranney https://orcid.org/0000-0002-3015-9130
Anjalie Field https://orcid.org/0000-0002-6955-746X
Tirthankar Ghosal https://orcid.org/0000-0002-2358-522X
Michele Ginolfi https://orcid.org/0000-0002-9122-1700
Marc Huertas-Company https://orcid.org/0000-0002-1416-8483
Sandor Kruk https://orcid.org/0000-0001-8010-8879
J. P. Naiman https://orcid.org/0000-0002-9397-6189
J. E. G. Peek https://orcid.org/0000-0003-4797-7030
Mugdha Polimera https://orcid.org/0000-0001-6162-3963

Sergio J. Rodríguez Méndez https://orcid.org/0000-0001-7203-8399

Kevin Schawinski https://orcid.org/0000-0001-5464-0888

Sanjib Sharma https://orcid.org/0000-0002-0920-809X

Michael J. Smith https://orcid.org/0000-0003-0220-5125

Yuan-Sen Ting https://orcid.org/0000-0001-5082-9536

Mike Walmsley https://orcid.org/0000-0002-6408-4181

## References

Accomazzi, A., Eichhorn, G., Grant, C. S., Murray, S. S., & Kurtz, M. J. 1995, *VA*, 39, 63

Accomazzi, A., Eichhorn, G., Kurtz, M. J., Grant, C. S., & Murray, S. S. 2000, *A&AS*, 143, 85

Accomazzi, A., Gray, N., Erdmann, C., et al. 2014, in ASP Conf. Ser. 485, Astronomical Data Analysis Software and Systems XXIII, ed. N. Manset & P. Forshay (San Francisco, CA: ASP), 461

Accomazzi, A., Kurtz, M. J., Henneken, E. A., et al. 2015, in ASP Conf. Ser. 492, Open Science at the Frontiers of Librarianship, ed. A. Holl et al. (San Francisco, CA: ASP), 189

Agarwal, S., Laradji, I. H., Charlin, L., & Pal, C. 2024, arXiv:2402.01788

Archipley, M., & Dalgleish, H. S. 2021, *RNAAS*, 5, 135

Archipley, M., Dalgleish, H. S., Ahrer, E., & Mortimer, D. 2021, in ASP Conf. Ser. 531, ASP2020: Embracing the Future: Astronomy Teaching and Public Engagement, ed. G. Schultz et al. (San Francisco, CA: ASP), 47

Astarita, S., Kruk, S., Reerink, J., & Gómez, P. 2024, Proc. of SPAICE2024: The First Joint European Space Agency/IAA Conf. on AI in and for Space, ed. D. Dold et al.

Bai, Z., Wang, P., & Xiao, T. 2024, arXiv:2404.18930

Besta, M., Blach, N., Kubicek, ed. 2024, in Proc. of the 38th AAAI Conf. on Artificial Intelligence, ed. M. Wooldridge et al. (Washington, DC: AAAI Press), 17682

Blanco-Cuaresma, S., Ciucă, I., Accomazzi, A., et al. 2023, arXiv:2312.14211

Burges, C. J. 2010, From RankNet to LambdaRank to LambdaMART: An Overview MSR-TR-2010-82, Microsoft https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/

Cao, C., Sang, J., Arora, R., et al. 2024, medRxiv, doi:10.1101/2024.06.01.24308323

Carpineto, C., & Romano, G. 2012, *ACM Comput. Surv.*, 44, 1

Cavnar, W. B., Trenkle, J. M., et al. 1994, Proc. SDAIR-94, 3rd Annual Symp. on Document Analysis and Information Retrieval, 14, https://dsac13-2019.github.io/materials/CavnarTrenkle.pdf

Chowdhery, A., Narang, S., Devlin, J., et al. 2022, arXiv:2204.00231

Ciuca, I., Ting, Y.-S., Kruk, S., & Iyer, K. 2023, Machine Learning for Astrophysics. Workshop at the 40th Int. Conf. on Machine Learning (ICML 2023), 8, https://ml4astro.github.io/icml2023/assets/35.pdf

Ciucă, I., & Ting, Y.-S. 2023, *RNAAS*, 7, 193

Creo, A., Lama, M., & Vidal, J. C. 2023, arXiv:2312.08282

Cui, C., & Li, S. 2018, arXiv:1801.05098

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. 2023, Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1 ed. J. Burstein, (Association for Computational Linguistics), 193, https://aclanthology.org/N19-1423/

Dung Nguyen, T., Ting, Y.-S., Ciucă, I., et al. 2023, arXiv:2309.06126

English, J. 2017, *IJMPD*, 26, 1730010

Frey, K., & Accomazzi, A. 2018, *ApJS*, 236, 24

Gao, L., Ma, X., Lin, J., & Callan, J. 2022, arXiv:2212.10496

Genova, F. 2023, in SF2A-2023: Proc. Annual Meeting of the French Society of Astronomy and Astrophysics, ed. M. N'Diaye et al. (Société Française d'Astronomie et d'Astrophysique), 175

Graham, M. J. 2012, Astrostatistics and Data Mining (Berlin: Springer), 47

Grezes, F., Allen, T., Blanco-Cuaresma, S., et al. 2022, arXiv:2212.00744

Grezes, F., Blanco-Cuaresma, S., Accomazzi, A., et al. 2024, in ASP Conf. Ser. 535, Astronomical Data Analysis Software and Systems XXXI, ed. B. V. Hugo et al. (San Francisco, CA: ASP), 119

Huang, Y., Feng, X., Feng, X., & Qin, B. 2021, arXiv:2104.14839

Iyer, K. G. 2021, Chaotic_Neural: Improving Literature Surveys in Astronomy with Machine Learning, v1.0, Zenodo, doi:10.5281/zenodo.5032358

Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. C. 2024, arXiv:2403.14403

Jia, R., Zhang, B., Méndez, S. J. R., & Omran, P. G. 2024, arXiv:2405.15374

Johnson, J., Douze, M., & Jégou, H. 2017, arXiv:1702.08734

Kondrak, G. 2005, in String Processing and Information Retrieval. SPIRE 2005, ed. M. Consens & G. Navarro (Berlin: Springer), 115

Lei, Y., Cao, Y., Zhou, T., Shen, T., & Yates, A. 2024, arXiv:2402.18031

Lewis, P., Perez, E., Piktus, A., et al. 2020, Advances in Neural Information Processing Systems 33, ed. H. LaRochelle et al. (NeurIPS), 9459, https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html

Li, H., Chi, H., Liu, M., & Yang, W. 2024, arXiv:2407.10153

Li, Y., Chen, L., Liu, A., Yu, K., & Wen, L. 2024, arXiv:2403.02574

Liang, W., Zhang, Y., Wu, Z., et al. 2024a, arXiv:2404.01268

Liang, W., Zhang, Y., Cao, H., et al. 2024b, NEJM AI, 1, A10a2400196

Lin, J., & Ma, X. 2021, arXiv:2106.14807

Luo, L., Li, Y.-F., Haffari, G., & Pan, S. 2023, arXiv:2310.01061

Manning, C. D., Raghavan, P., & Schütze, H. 2008, Introduction to Information Retrieval (Cambridge: Cambridge Univ. Press)

Maravelias, G., Vourliotis, E., Marouda, K., et al. 2018, arXiv:1810.04562

McInnes, L., Healy, J., & Melville, J. 2018, arXiv:1802.03426

Naiman, J. P., Cosillo, M. G., Williams, P. K. G., & Goodman, A. 2023, arXiv:2309.11549

Niu, M., Li, H., Shi, J., Haddadi, H., & Mo, F. 2024, arXiv:2405.06545

Nogueira, R., & Cho, K. 2019, arXiv:1901.04085

O'Neill, C., Ye, C., Iyer, K., & Wu, J. F. 2024, arXiv:2408.00657

OpenAI, Achiam, J., & Adler, J. 2024, arXiv:2303.08774

Pan, S., Luo, L., Wang, Y., et al. 2024, *IEEE Trans. Knowl. Data Eng.*, 36, 3580

Parker, L., Lanusse, F., Golkar, S., et al. 2024, *MNRAS*, 531, 4990

Perkowski, E., Pan, R., Nguyen, T. D., et al. 2024, *RNAAS*, 8, 7

Pervez, N., & Titus, A. J. 2024, arXiv:2406.19497

Prihar, E., Lee, M., Hopman, M., et al. 2023, in Artificial Intelligence in Education: Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, ed. N. Wang et al. (Cham: Springer), 290

Rahmani, H. A., Craswell, N., Yilmaz, E., Mitra, B., & Campos, D. 2024, arXiv:2405.07767

Rawte, V., Sheth, A., & Das, A. 2023, arXiv:2309.05922

Rodríguez, J.-V., Rodríguez-Rodríguez, I., & Woo, W. L. 2022, *Wiley Interdiscip. Rev.: Data Min. Knowl. Disc.*, 12, e1476

Roller, S., Dinan, E., Goyal, N., et al. 2021, Proc. of the 16th Conf. of the European Chapter of the Association for Computational Linguistics: Main Volume, ed. P. Merlo, J. Tiedemann, & R. Tsarfaty, (Association for Computational Linguistics), 300

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. 2021, arXiv:2112.10752

Shojaee, P., Meidani, K., Gupta, S., Farimani, A. B., & Reddy, C. K. 2024, arXiv:2404.18400

Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. 2021, Findings of the Association for Computational Linguistics: EMNLP 2021, ed. M.-F. Moens et al. (Association for Computational Linguistics), 3784

Smith, M. J., Roberts, R. J., Angeloudi, E., & Huertas-Company, M. 2024, arXiv:2405.14930

Spangler, S., Wilkins, A. D., Bachman, B. J., et al. 2014, in Proc. of the 20th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, ed. S. Macskassy et al. (New York: Association for Computing Machinery), 1877

Tanoglidis, D., & Jain, B. 2024, *RNAAS*, 8, 265

Tao, K., Osman, Z. A., Tzou, P. L., et al. 2024, *BMC Med. Res. Methodol.*, 24, 139

Ting, Y.-S., Dung Nguyen, T., Ghosal, T., et al. 2024, arXiv:2407.11194

Touvron, H., Lavril, T., Izacard, G., et al. 2023, arXiv:2302.13971

Van Noorden, R., & Perkel, J. M. 2023, *Natur*, 621, 672

Wei, Z., Guo, D., Huang, D., et al. 2023, in Proc. of the 2023 Int. Conf. on Artificial Intelligence, Systems and Network Security (New York: Association for Computing Machinery), 77

Wu, J. F., Hyk, A., McCormick, K., et al. 2024, arXiv:2405.20389

Yang, Z., Du, X., Li, J., et al. 2023, arXiv:2309.02726

Yao, S., Yu, D., Zhao, J., et al. 2024, Advances in Neural Information Processing Systems 36, ed. A. Oh et al. (NeurIPS), 11809, https://papers.nips.cc/paper_files/paper/2023/hash/271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html

Yao, S., Zhao, J., Yu, D., et al. 2022, arXiv:2210.03629

Zaitsev, I., Golubenko, O., Tkachenko, O., Pidmohylnyi, O., & Antonenko, A. 2023, CEUR Workshop Proc. 3687, Dynamical System Modeling and Stability Investigation 2023, ed. D. Khusainov et al. (CEUR), 121, https://ceur-ws.org/Vol-3687/Short_5.pdf

Zhang, Y., Jin, R., & Zhou, Z.-H. 2010, *Int. J. Mach. Learn. Cybern.*, 1, 43

Zhang, Y., Li, Y., Cui, L., et al. 2023, arXiv:2309.01219

Zhou, Y., Liu, H., Srivastava, T., Mei, H., & Tan, C. 2024, arXiv:2404.04326