From Queries to Criteria: Understanding How Astronomers Evaluate LLMs
Authors: Alina Hyk
Kiera McCormick
Mian Zhong
Ioana Ciucă
Sanjib Sharma
John F Wu
J. E. G. Peek
Kartheik G. Iyer
Ziang Xiao
Anjalie Field
Venue: COLM 2025 (2025)
Keywords: Human-centered computing
AI for Science
LM Evaluation
LLM for Astronomy
TL;DR: We analyze and recommend evaluation criteria grounded in real Human-LLM interactions for exploring scientific literature.

Abstract: There is growing interest in leveraging LLMs to aid in astronomy and other scientific research, but benchmarks for LLM evaluation in general have not kept pace with the increasingly diverse ways that real people evaluate and use these models. In this study, we seek to improve evaluation procedures by building an understanding of how users evaluate LLMs. We focus on a particular use case: an LLM-powered retrieval-augmented generation bot for engaging with astronomical literature, which we deployed via Slack. Our inductive coding of 368 queries to the bot over four weeks and our follow-up interviews with 11 astronomers reveal how humans evaluated this system, including the types of questions asked and the criteria for judging responses. We synthesize our findings into concrete recommendations for building better benchmarks, which we then employ in constructing a sample benchmark for evaluating LLMs for astronomy. Overall, our work offers ways to improve LLM evaluation and ultimately usability, particularly for use in scientific research.

Published as a conference paper at COLM 2025
From Queries to Criteria: Understanding How Astronomers Evaluate LLMs
Alina Hyk", Kiera McCormick 2,3, Mian Zhong', Ioana Ciucă Sanjib Sharma',
John E. Wu J. E. G. Peek3,, Kartheik G. lyer", Ziang Xiao², Anjalie Field
Orreoon State University Loyola University Maryland JJohns Hopkins University *Stanford University "Space Telescope Science Institute 6 Columbia University
Abstract
There is growing interest in leveraging LLMs to aid astronomy and other scientific research, but benchmarks for LLM evaluation in general have not kept pace with the increasingly diverse ways that real people evaluate and use these models in scientific inquiry. In this study, we seek to improve eval- uation procedures by building understanding of how users evaluate LLMs. We focus on a particular use case: an LLM-powered retrieval-augmented generation bot for engaging with astronomical literature, which we de- ployed in the real world. Our inductive coding of 368 queries the bot over four weeks and our follow-up interviews with 11 astronomers reveal how experts evaluated this system, including the types of questions asked and the criteria for judging responses. We synthesize recommendations for building better benchmarks, which we then employ in constructing sample benchmark for evaluating LLMs for astronomy. Overall, our work offers ways to improve LLM evaluation and, ultimately, real-world interaction, particularly for use in scientific research.
findings into
1 Introduction
There is growing interest in the astronomy community in using large language models (LLMs) to accelerate scientific research, as evidenced by tools like AstroLLaMa (Nguyen et al., 2023; Perkowski et al., 2024) and Pathfinder (Iyer et al., 2024), which aim to democra- tize astronomy by enabling global researchers and enthusiasts to search the vast body of astronomical literature, synthesize information, and refine their ideas using Al as source of inspiration (Krenn et al., 2022). However, leveraging this potential remains difficult without principled evaluation frameworks that are grounded real-world user interactions. Existing datasets to evaluate LLMs for science are dominated by closed-form questions, such as multiple-choice questions, which do not reflect the increasingly diverse ways people seek to use I.I.Ms (Sun et al., 2024; Singhal et al., 2023; Guo et al., 2023; Hendrycks et al., 2021). The challenge of evaluating LLMs for astronomy and other scientific fields mirrors current challenges LLM evaluation more generally: while increasingly powerful models perform indistinguishably well on benchmarks, this high performance does not directly translate into usability and reliability, as benchmarks often fail represent realistic user interactions (Bowman& & Dahl, 2021; Raji et al., 2021; Liang et al., 2023; Liao & Xiao, 2023).
In this work, we explore how people use and evaluate an LLM-powered bot for astronomy literature exploration, ultimately translating our analysis into recommendations for building benchmarks that are reflective of the evaluation strategies and criteria that users employ. The system we deploy is retrieval augmented generation (RAG) LLM: user sends query to the bot, the query is compared against a large body of astronomy literature to retrieve the most relevant documents, and the retrieved documents and the user query are passed to an LLM to provide a response with citations the user. Astronomy serves
"qqul contribution

---

Published as a conference paper at COLM 2025
Inductive coding question types
User A Jun 4th 10:28AM
(tAsic astro-ph what's origin multiple populations in globular clusters?
Use-inspired sample benchmark
Ask astro-ph
10.29AM
The origin of multiple stellar populations (MSPs) in globular clusters (GCs) remains complex...no one single scenario fully explains the spectroscopic and photometric complexity observed GCs 10564)
User interviews
Generalizable criteria for benchmark construction
B
User B Jun 4th 10.38
Is this answwe good?
A
4that AM
very generic and not wroon but misses some the hypatheses
We deploy @Ask-astro-pe as a Slack bot and analyze resulting usage through inductive coding of user queries and follow-up interviews. We ultimately synthesize results into user-desired evaluation criteria, which are operationalizable in benchmark datasets.
Figure 1:
as an ideal test domain for investigating users queries to LLMs for scientific research for several reasons. First, the field contains rich open-access literature that spans many decades of research. For example, NASA's Astrophysics Data System (ADS) hosts over 15 million resources encompassing virtually all literature used by astronomers (Accomazzi et al., 2015; 2022), creating a comprehensive knowledge base ideal for LLM applications. Second, short- term direct impacts of research people are rare, reducing the potential harms that AI could cause. Finally, astronomy researchers constitute a well-defined user group, generally clustered around major research centers, which makes accessing potential users straight- forward through collaborations and partnerships. Through our system deployment, we investigate two research questions motivated by our goal of understanding and improving LLM evaluation:
RQ1: What types of questions do astronomers use to evaluate a bot for interacting with academic literature?
RQ2: What criteria do astronomers judge in the bot's responses?
To investigate these questions, as summarized in Figure 1, we built and deployed the Slack bot @Ask-astro-ph and invited astronomers to interact with it in group channel via direct messages; over a span of four weeks, we logged user queries, emoji reactions, and user replies to the bot's responses. We then used an inductive coding approach to analyze the resulting 368 queries and conducted further follow-up interviews with users. All users who interacted with the bot consented to make their de-identified interactions fully public part of this study, offering valuable data for further investigation.
Finally, we synthesize our analysis into specific recommendations on how this work can be leveraged to design human-centered evaluations of LLMs, and we release a sample gold benchmark for evaluating LLMs for astronomy?. Our sample benchmark consists of 40 question-answer pairs, where astronomers have written free-form answers with high- quality citations response to real user queries. Thus, our sample data set valuable for evaluating information retrieval as well as LLM response generation.
Although our work focuses on astronomy, we anticipate that our findings on LLM use and evaluation will generalize to other fields, particularly other observational sciences and semi-verifiable domains. Our overall approach of deriving evaluation criteria and ultimately constructing a benchmark grounded in real user testing is also easily repeatable, e.g., our rstronomy-focussee study serves as an initial low-risk proving ground, as opposed to disciplines like seismology epidemiology, wherein deploying and evaluating L.LMs
Intpss///huggingffte.oo http:///wwwnggggeeeeooe/eiinii/a
2

---

Published as a conference paper at COLM 2025
is riskier and more ethically fraught endeavor. We ultimately aim to improve alignment between user evaluations and benchmark datasets, with the long-term goal of broadly improving the usability of LLMs for science.
Related Work
Most progress in developing LLMs for scientific research has taken the form of benchmark datasets, which offer ways to evaluate model performance for related tasks, like medical question answering (Singhal et al., 2023) or mathematical problem solving (Hendrycks et al., 2021). Given the difficulty of creating benchmark datasets for scientific evaluation, which requires high level of expertise, many datasets use automated construction methods, including extracting entities and relations from structured knowledge resources (Pampari et al., 2018; Pappas et al., 2018; 2020). However, these benchmark datasets often fail to reflect realistic cases (Raji et al., 2021) and do not always correlate with human judgments (Tang et al., 2023; Fabbri et al., 2021), particularly for scientific inquiry which involves nuanced interpretation and domain-specific knowledge. For example, in fask set-ups similar to ours, Gao et al. (2023) and Asai et al. (2024) evaluate LLMs along dimensions like fluency and citation correctness, but they do not justify if these metrics are important to users.
Some work has conducted more realistic evaluations of LLMs through user studies, most commonly for creative writing tasks, like scripts (Mirowski et al., 2023) or novels and fiction (Yang et al., 2022; Calderwood et al., 2020). A few studies have also considered research tasks. For example, Wang et al. (2024) investigate how researchers interact with LLMs, but they assign participants specific tasks rather than allowing open-ended exploration. In contrast, Zhao et al. (2024) take more open-ended approach study LLM use, but their initial work contains limited analysis and does not focus on scientific research. The absence of studies investigating how scientists naturally evaluate and use LLMs creates a significant gap in our understanding of effective evaluation frameworks for scientific applications.
While more reflective of real-world usage than curated benchmarks, user evaluations can be challenging to reproduce and do not facilitate efficient development, they are too time-consuming and labor-intensive to conduct for every small change in a model. The non-determinism of LLMs can make reproducibility challenging even with static bench- marks (Blackwell et al., 2024), though improved output consistency is generally attainable through techniques like greedy decoding or lower temperature values (Renze, 2024; Patel et al., 2024; Song et al., 2024). Scientific domains like astronomy face additional challenges, including the need integrate numerical data with textual knowledge and handle spe- cialized terminology (Manakul et al., 2023), as well as mitigating hallucination risks which could compromise research reliability (Li et al., 2023; Zhang, 2023).
Overall, the continued utility of benchmarks for standardized evaluation, combined with their failure to reflect realistic use cases of LLMs for scientific research, motivates our work. We take an open-ended exploratory approach to identifying how users evaluate LLM-based systems in astronomy, and we translate our analysis into principled criteria to guide future evaluations in scientific research contexts.
3 Methodology
3.1 System Deployment and Data Collection
We constructed an LLM-powered RAG system titled @Ask-astro ph as a bot on Slack. When a user queried the bot, it first retrieved the top 5 most relevant papers with the highest cosine similarity using was based on a dataset of 300,000 arXiv astrophysics papers posted on or before July 2023 (Perkowski et al., 2024). ArXiv serves strong data source as surveys have shown 93-94% of physicists and astronomers use arXiv with 100% usage in astrophysics (Narayana & Bhandi, 2022). The abstracts, conclusion sections, and metadata (arXiv IDs and years) of the retrieved papers were concatenated with prompt (Appendix A) and the initial user query and passed to go to produce an answer for the user.
preesmall-en--e v1.5 embeddings (Xiao et al., 2024). The retrieval
3

---

Published as a conference paper at COLM 2025
We deployed @Ask-astro-ph on the internal Slack workspace for the Space Telescope Science Institute (STScI) and recruited study participants to interact with the bot. STScI conducts astronomy research and operates space telescopes for NASA Astrophysics, petessssssssssseeee employing 150 science staff who spend 0% to 50% of their time doing research. Participants electronically signed a consent document and fill out demographic questionnaire (Appendix B). As exemplified in Figure/ 1, they were invited to a private Slack group channel, where their posts were sent as queries to -stt--stto-phh To provide feedback, users could click on pre-populated thumbs-up/ /ddbss /own emojis and comment on the bot's replies to any user's 's queries. We collected data from May 29, 2024 to June 24, 2024, during which time 368 total queries were sent to the bot. 35 out of 43 participants who signed up sent at least one query.
3.2 Inductive Data Coding and Follow-up Interviews
We analyzed the question types of the queries via inductive coding, First, four authors from astronomy (2), NLP (1), and psychology (1) independently coded 20-50 queries. We emphasized labeling the type of knowledge or evaluation solicited by the user, rather than how a model might answer the question. As users also sometimes repeated the same question to the bot with different exact phrasing, annotators also labeled if a question was a re-ask of a previous question. The four annotators then reconciled and discussed the produced labels and co-coded 100 queries, with multiple iterations of revising the scheme to finalize a scheme and reach consensus. The remaining 267 queries were first coded independently, with 2 3 annotators per query, where 2 2 of the annotators were astronomers. Krippendorff' 's alpha for the independent coding was 0.51. The sources of disagreement were often driven by the diversity of backgrounds of the annotators: for example, astronomers better distinguished which types of queries should be considered unresolved topics in astronomy, whereas other annotators were more familiar with stress- testing strategies. The final label is based on a majority vote of annotators, and in cases of disagreement, we discussed options until agreement is reached. These contested queries were labeled as such with notes on the reasons for their final label to support replicability.
We conducted follow-up semi-structured interviews with 11 study participants out of 19 invitations to further understand their evaluation practice. We selected interviewees who (1) sent the most number of queries, (2) provided the most reply comments, (3) received the most thumbs up from the answers to their queries and (4) likewise but received thumbs d 3 Despite some refusals, our final set had at least 2 people from the top 5 for each category. Each interview was conducted by 1 of authors: they began with a fixed script, but were permitted to deviate as needed. During the interview, the interviewer encouraged the interviewee to talk through a few examples of their queries. All interviews were recorded and transcribed. Four authors independently coded 1-2 interviews and consolidated identi- fied themes to determine an initial coding scheme. They co-coded 3 interviews to establish mutual understanding of the scheme and conducted several iterations of independently coding the remaining interviews, updating the scheme, and re-coding interviews. Overall our study has similar number of participants as related studies focused on user behavior and follows standard practices for data analysis (Lazar et al., 2017; Liao et al., 2020; Zhou et al., 2022; Mirowski et al., 2023; Zhou et al., 2022; Feng et al., 2025).
4 Results
4.1 RQ1: What types of questions do scientists use to evaluate a bot for interacting with academic literature?
Overall, our study reveals variety of question types, which go beyond factual questions typically included in benchmark datasets. Our analysis also shows that although the bot was designed for a specific use case (i.e,, interacting with academic literature) and specific
'In practice, criteria (1) and (2) identified active users, while criteria (3) and (4) selected for users who only sent a few queries, ie. the user with the highest percentage of negatively rated queries is someone who sent 1-2 queries that all received thumbs down, whereas users who sent many queries typically had more mixed ratings of responses.
4

---

Published as a conference paper at COLM 2025
group (i.e., astronomy researchers), there is significant diversity in the types of queries and interaction patterns among users.
4.1.1 User Queries
From Table 1, the most common ques- tion type* was Knowledge Seeking; spe- cific factual (e.g,, "What is the most
% of thumbs thumbs
queries down up
massive known spiral galaxy in the universe?"), followed by Deep Knowl- edge requiring higher-order reason- ing, such producing, a summary, speculation or synthesis of results
KS: specific factual
33%
28%
31%
Deep Knowledge
21%
25%
39%
BS: topic
14%
49%
8%
across papers (e.g,, "What are the most promising subfields of astro- nomical research for new discover- ies?").
KS: broad descrip-
13%
6%
52%
tion
Stress Testing
7%
27%
27%
Bot Capabilities
5%
16%
28%
Although we focus on user perspec- tives rather than the performance of this particular system, Table re- ports user ratings to each query type for completeness and account for ways users may adjust their evalua- tion strategies based on bot perfor- mance. In general, bot responses were rated more positively than neg- atively, except for queries that in- volved specific bibliometric compo- nents 16% of total queries). The bot was not designed for this use case as existing tools already support this search type (e.g., ADS, Google Scholar).
KS: procedure
2%
11%
56%
BS: specific paper 2%
56%
0%
or author
Unresolved Topic
2%
0%
75%
Table Percent of total queries inductively coded as each question type, and the percent of bot responses to queries of the specified type that had net thumbs up / down ratings. They do not responses had no ratings. "KS" = "Knowledge Seek- ing" and "BS" Bibbliometric Search"
to 100% many
Figure provides deeper insight by visualizing categorized queries for each user sequen- tially.5 Almost all users used a range of question types, with distributions varying greatly by user. Some users first queried with Knowledge Seeking: specific factual type (e.g., 3, 19, 23), whereas others started with harder Unresolved Topic questions (e.g., 12)
Similarly, question types were also well-distributed across users. For example, Stress Testing and Unresolved Topic questions did not occur as a burst of queries from a single user but rather were recurring strategies across multiple users. Moreover, users were often persistent in their querying strategy regardless of the bot's responses: although the bot consistently performed poorly on bibliometric questions, users persisted in asking them (e.g., 13, 17, 26). In the long term, we speculate that users would adapt to the system's strengths, but in a shorter-term deployment, we can surmise that users' queries were more driven by their initial evaluation strategies and use cases than the performance of this specific system. One area where users show initial adaptation is when users tried rephrasing previous question (unfilled shapes in Figure 2), which occurred 22 times. Among re-asked questions where the bot's original response received a rating, in 73% of cases the rating was thumbs-down.
4.1.2 Follow-up Interviews
Follow-up interviews allow us to more deeply probe why users asked these questions. We identified 4 common strategies motivating these queries:
We describe each coding category and an example from the data in Table 4 (Appendix C) In Appendix F, we show the same data temporally instead of sequentially.
5

---

Published as a conference paper at COLM 2025
Brumbs
capatalltes
kSs piocedure
Query bot about things they know out of 11 users referenced querying the bot about things they know, primarily their spe- cific area of expertise. An example is prob- ing the bot for question they knew was answerable from one of their recent publi- cations: "So knew[...] had a paper that was published during this period|... That it did have data available that was on this topic" {P9). In other cases, users did not have specific answer in mind, but were confident in their ability to judge a clearly incorrect answer.
caling thumbs
specifie paper of author
KSt specticc
stress lecting
trosderge
lepi:
dissciption
70000/Aoy
Trying out bot on actual use cases Sur- prisingly, even in this time-limited de- ployment, three users described queries as inspired by actual use cases, typically snowledge-seeking or bibliometric queries. One user mentioned querying unfamiliar concept heard in a discussion group {P13). Another described the motivation behind query as: "This happened because I was ex- plaining the same concept student..... sent some references back to them, and I was like, well, let's see if the bot can do better than do." " (P10)
Figure 2: Question type and rating of bot re- sponse for each user.
Nuanced, ambiguous, or difficult questions Other evaluation strategies included query- ing questions that were nuanced or difficult and may not have specific factual answers. These queries were typically coded as Deep Knowledge or Unresolved Topic, and (P4) described them as motivated by the scientific research process: "that's something really important to me, and something that tried to emphasize when work with students that you know there's is never clear cut answer to anything."
Trying to trick bot probe for expected failures Finally, users attempted to trick the bot. In addition to off-topic queries like favorite food flavor, users also employed more nuanced approaches that targeted astronomy knowledge specifically, "And then I went a little the ridiculous route, trying to see like well, if post a question that is obviously wrong, would it still answer" (P7), or targeted potential failures based on their knowledge of the bot's data sources: "There are the April Fool's articles that are always present there, and there was one about Wolf 359. So wanted to see if it would pick that up." (P22)
4.2 RQ2: What criteria do scientists judge in the bot's responses?
In addressing our second research question, we focus primarily on results from follow-up interviews, as we found user thumbs up/ down ratings insufficiently detailed to understand the desired evaluation criteria. Interviews revealed commonly articulated dimensions:
General correctness of response Unsurprisingly, the primary evaluation criterion that all users prioritized was overall correctness, though this criteria involved multiple sub- dimensions beyond general accuracy of terms and definitions. Users cared about the quality of citations, typically seeking for them to be relevant and recent, with mixed opinions on how the model should weigh citation counts. One user criticized more obscure references: "it would return some obscure reference that maybe had some of the words in the abstract, but was not necessarily a good reference to cite" [P26], while another found the bot identifying previously unseen citations useful: "It would give references that I hadn't tt read before(... Iso I would open those" (P4). [P4] also emphasized the importance of the bot accurately reflecting the retrieved references: "I think the the most important thing to

---

Published as a conference paper at COLM 2025
me for a scientific response with citation is th the content of the response that comes from the citation is accurately relayed[... you can find a relevant citation and then you summarize it wrong, that's in some ways more insidious."
Hedging and caveating of answers Eight users judged the model's abilities to qualify responses focusing on two areas: expressions of confidence and providing reasons for denying user requests. Users noted that the responses were often expressed confidently even when they were wrong: "One concern is when it does go wrong. It's hard to discern that it was wrong. It's very good at coloring things with lot of confidence" (P34). While prior investigations have similarly found that LLMs are prone to overconfidence and unreliable epistemic markers (Zhou et al., 2023; 2024), two of our users {P34, P4) identified this characteristic as a particular concern in the context of scientific research: "overconfidence is bad thing to have researcher" {P4). Regarding the second area, the deployed system was fairly conservative and frequently refused to answer queries, which users found frustrating, particularly because the bot failed to explain refusals: "I cannot answer, was responded a lot, and kind of wish it had a little bit more of a qualifier to it...Can you not answer because it's not an astronomy question? Can you not answer because it's not in your source information?" {122]. This frustration with unexplained denied requests reflects previous findings (Wester et al., 2024).
Response specificity and clarity In addition to whether the bot response contained ap- propriate qualifiers, users often commented on the specificity of the responses, and noted that an answer could be factually correct but not informative: "I'm pretty sure this an- swer is accurate, but doesn't really tell you anything" (P22). Users specifically used the word "generic" to describe these responses: "But that's like very generic, very generic answers" {P10}. Users otherwise generally did not comment on the response style, with few suggesting that the language fluency was good enough not to be noteworthy.
Correct interpretation of user query Another criterion that users evaluated in the model was its ability to correctly interpret the user query, which included inferring from context and understanding user intent. One user offered an example: "But somebody asked what's the typical radius of a 10 solar mass cluster. And an astronomer would immediately know 10 solar mass clusters: this person is talking about galaxy clusters, not about clusters of stars or other kinds-there are lots of uses of the word 'cluster' in astronomy-and the bot did not even reply until you were careful about it" (P9}. Another described the idea more generally in expressing what they liked about a different Al model (Claude): "It was as though it really understood the question better than[ 1the other programs did in terms of understanding what is exactly that this person wants from this question" (P36). Users also described rephrasing queries or attempting to adapt to the query types that the bot responded best to. While from the user perspective, this involved a learning curve in understanding how to phrase queries, from developer perspective, reducing the need for users to carefully phrase queries would be preferred.
Broader Impact Finally, users also considered how such LLM tools might impact on their workflow, astronomy research practices, or society as whole. Most commonly users evaluated whether the bot saved them time or work, e.g., (P9) sought "a quicker start" into literature exploration, and (P12) did not interact with the bot much because, "I realized I wasn't t saving myself any work." Other users praised or looked for elements of discovery or surprise: had a really good time playing with that and discovering papers that had never seen" (P4). One user further considered the impact on the broader astronomy research: "[Does] using these agents make us better researchers make us more honest researchers?" and society as whole: "there's 's creeping feeling of like, what am I doing like? Who is this for? Who is this really benefiting?" (P12). Recent efforts to benchmark LLMs have included criteria intended to capture possible societal impacts, such faimess or toxicity (Liang et al., 2023), though this criteria cannot fully answer questions like "Who is this really benefiting?".
7

---

Published as a conference paper at COLM 2025
5 Building Better Benchmarks: Guidelines and Construction
In this section, we synthesize results from $4.1 and $4.2, showing how inclusion of diverse query types and evaluation metrics are necessary for enabling benchmarks to capture users' desiderata. We further demonstrate the feasibility of implementing these guidelines by constructing a sample benchmark for evaluating LLMs for astronomy driven by our data.
In Table 2, we map the query types identified in 54.1 to the desired evaluation criteria iden- tified in $4.2, offering concrete guidance on the types of queries to include in benchmarks in order capture desired evaluation criteria. Our work suggests that existing benchmarks using closed-form questions (examples: Sun et al. (2024); Guo et al. (2023); Hendrycks et al. (2021)) are insufficient to support the broad range of criteria that users care about in using LLMs for science. This type of question is fundamentally limited to subset of Knowledge Seeking: specific factual and only capable of capturing a narrow range General Correctness of Response.
While the criteria in $4.2 and Table more closely capture what scientists evaluate in models, one difficulty is how to implement them effectively. Asking annotators to rate model responses along these dimensions straightforward, but relying on human annotation for every evaluation cycle does not facilitate efficient development. In the "Metric" column of Table we suggest more automated approaches. For example, epistemic markers like "maybe", "might", or "definitely" are indicators of hedging and caveating (Zhou et al., 2023; 2024), and if retrieved evidence actually supporss the model response is possible to evaluate through natural language inference (entailment) models (Gao et al., 2023), as has been implemented in systems for automated fact-checking (Thorne et al., 2018; Samarinas et al., 2021). We could partially capture all criteria by comparing model outputs ground-truth paragraphs, using metrics such as n-gram overlaps (e.g., ROUGE) and BERTScore (Zhang* et al., 2020), though a single score would not be sufficient for providing nuanced feedback to developers. Asai et al. (2024) use LLM judgments for range of criteria that has some overlap with our proposed dimensions, suggested that LLM judgments may be able to provide more nuanced evaluations.
Evaluation Criteria
Query Type
Metric
C1: Correct Interpretation of User Query
Consistency under query paraphrases
Closeness to human-written response
C2: General Correctness of Response
Closeness to human-written response
Correctness of Terms/ Terms/Definntions, Defimitions Quality of Retrieved Citations Good Match of Response and Citation
Exact match of closed-form answers
Comparison with human-identified citations
Natural language inference/ /entailment
C3: Hedging and Caveat of Answers
Level of Model Uncertainty Explanation Requests Denial
Epistemic markers (e.g.. "maybe", "definitely") Closeness to human-written response
C4: Response Specificity and Clarity
Closeness to human written response
Broader Impact
N/A
User studies
KS: specific factual Deep Knowledge BS: topic
Note
KS: broad description @ Stress Testing Bot Capabilities
KS: procedure
BS: specific paper or author Unresolved Topic
Table 2: Recommended query types for capturing each user-desired evaluation criteria, as well as proposed metrics for facilitating fully automated evaluation.
While Table 2 offers concrete ways to capture evaluation criteria through inclusion of suitable query types and automated metrics, we do not identify ways to measure Broader Impact through benchmark datasets. This criteria is better captured through user studies and pilot deployments than static datasets.
5.1 A Sample Benchmark for Astronomy LLM Evaluation
We enact these recommendations in constructing a sample benchmark dataset consisting of 40 real user-L.LM queries drawn from our primary dataset and gold answers with references
8

---

Published as a conference paper at COLM 2025
written by astronomers." 6 To our knowledge, this benchmark is the first evaluation dataset for astronomy that combines real user queries with expert-curated answers from astronomers.
To develop the benchmark, 7 astronomers read through the user queries and selected questions suited to their expertise. They then wrote an answer to each question and searched the literature to provide supporting references, typically 2 to 5 arXiv papers. Each question required about 30 minutes to answer, making the process time-consuming. One astronomer then read through all of the queries and answers to check for correctness. The chosen queries trace the frequency distribution of our primary dataset: approximately 55% are Knowledge Seeking: specific factual, 22% are Deep Knowledge, and 17% are Krowwledge Seeking; broad description. Moreover, our benchmark retains the original format of the questions instead of standardizing or reformulating them, as this better reflects the patterns of interaction in the real world. Although this careful curation limits our initial dataset size, establishes a pathway for creating high-quality question-and -nd--sswer pairs to evaluate LLMs in astronomy. The inclusion of free-form text responses and references allows for evaluating both generation and retrieval components for the proposed criteria on the quality and clarity of the response as well as the quality of retrieved citations.
We validate the quality of this benchmark and its usability for automated evaluation by comparing scores over the benchmark with users' thumbs up and down ratings. More response
specifically, for each query, we pass the human-written response and the resginal generated by @Ask-astro-ph to gpt-4o-miní and prompt the model to generate a score between O and 1 reflecting the closeness of the two responses, with 1 meaning that the responses exactly match (Appendix F). Pearson's correlation between these LLM-generated scores and the thumbs-up huumbs-up/dower /ownn counts from the primary dataset is 0.8239. This strong correlation suggests that automated evaluation using our sample benchmark is highly indicative of user satisfaction.
Discussion and Future Work
While we focus primarily on synthesizing individual user evaluations into automated benchmarks, our study further reveals alternative avenues for user-centric evaluations, As discussed in $3, we deployed the bot in a Slack space where users could interact with group channel or through direct message. While some of the goals and outcomes of creating a shared channel were to encourage engagement, the channel also unexpectedly revealed ways users co-evaluated the bot. Users commented on each other's 's queries, offering direct advice on how to phrase queries or opinions about the bot's accuracy. Emoji reactions also reveal co-evaluation: out of 345 total recorded reactions, in 101 (29%) instances, the reaction was made by different person than the initial query (typically the original querier had reacted as well). Similarly, in follow-up interviews, all participants except one referenced reading other users' queries without the interviewer prompting them. These interactions suggest collaborative practices could be potentially powerful tools for evaluating LLM- powered bots, which support several research initiatives that have considered the role of collaboration in data labeling (Muller et al., 2021; Kuo et al., 2024), and the social nature of Slack (Avula et al., 2018; 2019; 2022; Wang et al., 2022).
Our study further reveals high variance in how users interacted with the bot (Figure 2). While we do not find notable differences in how people of different genders interacted with the both, we do see differences by year of PhD completion, which is correlated with age: more recent graduates tended to ask more stress-testing and bot capabilities questions. Differences by year of degree completion age are also evident in how users rated the bot: the bot's responses to queries from younger more recent graduates were rated significantly more positively (Appendix D). These results emphasize that diverse groups people are needed to design evaluation schemes to ensure widespread usability, even within relatively small community like astronomy researchers. Benchmark datasets are typically constructed
'We provide the entire 40 questions-and-amee answerss benchmark as supplementary material. In future work, we plan to expand the dataset by obtaining expert-curated answers to a more significant subset of the remaining 325 questions, aiming for 100 for our second release, and we also intend to write paraphrases of queries and answers to support more variance in style.
9

---

Published as a conference paper at COLM 2025
by annotators who label data independently; they could be constructed collaboratively, for example, through participatory design workshops, which have been used to integrate broader perspectives into Aí development in other ways (Brown et al., 2019; Katell et al., 2020).
Overall, the rapid development and adoption of LLMs, as well the documented interest in leveraging them to advance scientific research, underscores an urgent need for improved evaluation schemes that are holistic and user-centric in this domain. Our work offers specific recommendations on how to improve current evaluation practices, and we expect many of our results to generalize from astronomy to other domains. We ultimately aim broaden the usability of LLM-powered tools and in-turn, participation in scientific research.
Ethics Statement
IRB Review All protocols, including consent and procedure for de-identified data sharing, were approved by the Institutional Review Board (IRB) at Johns Hopkins University.
Research Positionality We acknowledge that our academic backgrounds have influenced our perspectives on this topic. The authors of this work include 5 astronomers, 3 of whom are employed by the institute where the study took place. The emphasis on astronomy in this study reflects the interests and expertise of these authors, particularly in understanding the potential applications of LLMs in astronomy research. The remaining authors have backgrounds in NLP, HCI, and psychology.
The primary limitation of our work is its focus on small-scale group in a specific domain. The astronomers we recruited were from a single institute, are currently based in the U.S. (although not all astronomers have PhDs, as opposed to students or amateur astronomers. Further, users opted-in to participate in the study, suggesting our user group may not capture the viewpoints of people without any interest (positive or negative) in Al tools. As STSc employs a broad range of astronomers with diverse backgrounds and areas of scientific expertise, we do expect our population to be representative within these criteria. Nevertheless, in order to achieve the goals of broadening the usability of AI tools, future work needs to proactively engage with broader set of users, including incentivizing participation from less interested ones.
Limitations
originally from the U.S.), and already
Our study also limited in that it focused on evaluating one LLM-powered bot for specific use case. People may use expanded evaluation strategies with different tool, and their de- sired criteria may shift over longer-term use of a system integrated their workflow. While we anticipate that many of the findings generalize beyond the specific astronomers who interacted with the tool and beyond astronomical research to other disciplines, especially those in semi-verifiable domains and/or observational sciences, cannot conclusively assume generalizability.
Acknowledgments
This work was conducted in part through JSALT summer workshop hosted by the Center for Language and Speech Processing at Johns Hopkins University. KM thanks the Space Astronomy Summer Program (SASP) hosted by STScI. The authors thank STSeI staff who participated in this study.
References
Alberto Accomazzi, Michael J. Kurtz, Edwin A. Henneken, Roman Chyla, James Luker, Carolyn S. Grant, Donna M. Thompson, Alexandra Holachek, Rahul Dave, and Stephen S. Murray. ADS: The next generation search platform. Open Science at the Frontiers of Librarianship, 492:189, 2015.
10

---

Published as a conference paper at COLM 2025
Alberto Accomazzi, Edwin A. Henneken, Carolyn S. Grant, Donna M. Thompson, Matthew R. Templeton, Jennifer Koch, Sergi Blanco-Cuaresma, Roman Chyla, Stephen McDonald, Golnaz Shapurian, Timothy W. Hostetler, Kelly E. Lockhart, Nemanja Marti- nović, Shinyi Chen, Felix Grezes, and Michael J. Kurtz. Content of the future in the ADS. Bulletin of the AAS, 54(2), apr 27 2022. URL https://bast aas. org http//u//2000 20mpb/22222222.
Akari Asai, Jacquelin He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D' a'arcy, et al. Openscholar: Synthesizing scientific literature with rugmened Ims. arXiv preprint ar arXi::4411.14199 2024.
Sandeep Avula, Gordon Chadwick, Jaime Arguello, and Robert Capra. SearchBots: User engagement with ChatBots during collaborative search. In Proceedings of the 2018 Con- ference on Human Information Interaction & Retrieval, CHIIR '18, pp. 52-61, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450349253, doi: NL.1145/1176349317763
URL http:::/doiooorg////
Sandeep Avula, Jaime Arguello, Robert Capra, Jordan Dodson, Yuhui Huang, and Filip Radlinski. Embedding search into conversational platform to support collaborative search. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, CHIR '19, pp. 15-23, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450360258. doi: 10.1 10.1145/3295550.2298888 3295750. 3298928.
URL
1145/
Sandeep Avula, Bogeum Choi, and Jaime Arguello. The effects of system initiative during conversational collaborative search. Proc. ACM Hum.-Comping Interact., 6(CSCW1), April 2022. doi: 10.1145/351299 URL https: //doi. http:///ooooo////////// 10. 1145/3512913.
Robert E. Blackwell, Jon Barry, and Anthony G. Cohn. Towards reproducible 11m evaluation: Quantifying uncertainty in Ilm benchmark scores, 2024. URL https: org/abs/ 2410.03492.
Samuel R. Bowman and George Dahl. What will it take to fix benchmarking in natural language understanding? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4843-4855, Online, June 2021. Association for Computational Linguistics. doi: 00.18633/v1/202 org/2021 naacl-mainnnnnnaa 385/.
maacl-main.385. URL
Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein,, Andrew Tobin, and Rhema Vaithianathan. Toward algorithmic accountability in public services: A qualitative study of affected community perspectives on algorithmic decision-making services. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI '19, pp. 1-12, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450359702. doi: 01145/329060553000 3300271.
child welfare
URL https: //tp:::///oiiioe org../...111111 1145/329655/
Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B. Chilton. How Novelists Use Generative Language Models: An Exploratory User Study. In Werner Geyer, Yasaman Khazaeni, and Michal Shmueli-Scheuer (eds.), joint Procezdings of the Workshops on Human- Al Co-Creation with Generative Models and User-Aware Conversational Agents, volume 2848 of CEUR Workshop Proceedings, Cagliari, Italy, March 2020. CEUR. URL https: //ce/s-//s//// org/Vol-2844 WHWHHAAG-----NEEEEEE
Alexander Fabbri, Wojciech Kryšcinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. dummEval Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409 2021. doi: 10.1162 10.1162/tacl_a httl/aacclaa0007 URL https://aclanthhi
org/2021 tacl-1.24 1.24.
K. J. Kevin Feng, Q Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy X. Zhang, and David W. McDonald, Canvil: Designerly adaptation for llm-powered user experiences. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI '55
11

---

Published as a conference paper at COLM 2025
New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400713341. doi: 09.1145/370659837711131 URL https://doi org/ 10. 10111115355555 37 31.31
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 6465-6488, Singapore, December 2023. Association for Computational Linguistics. doi: (1116633/112223emmm/t maa.:eemmmllmmmm399.... URL https: petps://aclaatthoogge org/ emnlp-main. 398/.
Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. What can large language models do in chem- istry? A comprehensive benchmark on eight tasks. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //openreview. net/
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solv- ing with the math dataset. In Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol- ume 1, 2021. URL https///ttttttteeeennnneesseeeee neuriipnnnnnnnrissssseee.eeieiip cc/paper. round2.
Kartheik G. Iyer, Mikaeel Yunus, Charles O'Neill, Christine Ye, Alina Hyk, Kiera McCormick, Ioana Ciucă, John F. Wu, Alberto Accomazzi, Simone Astarita, Rishabh Chakrabarty, Jesse Cranney, Anjalie Field, Tirthankar Ghosal, Michele Ginolfi, Marc Huertas-Company Maja Jabtońska, Sandor Kruk, Huiling Liu, Gabriel Marchidan, Rohit Mistry, J. P. Naiman, E. G. Peek, Mugdha Polimera, Sergio J. Rodríguez Méndez, Kevin Schawinski, Sanjib Sharma, Michael J. Smith, Yuan-Sen Ting, Mike Walmsley, and (UniverselBBD) pathfinder: A semantic framework for literature review and knowledge discovery in astronomy. The Astrophysical Journal Supplement Series, 275 nov 2024. doi: 0..384//15384330 add7 /ttp doi.doo..ggo.oo.oooo.o orwoorg//// /0/ 3847 1538-1188801 4365/ad7c43.
URL https:
Michael Katell, Meg Young, Dharma Dailey, Bernease Herman, Vivian Guetler, Aaron Tam, Corinne Bintz, Daniella Raz, and P. M. Krafft. Toward situated interventions for algorithmic equity: lessons from the field. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT '20 pp. 45-55, New York, NY, USA, 2020, Association for Computing Machinery. ISBN 9781450369367. doi: 10.1145/3351095.3322225 URL https://doi org/10////// 10. 0145/3351199 3372874.
Mario Krenn, Robert Pollice Si Yue Guo, Matteo Aldeghi, Alba Cervera-Lierta, Pas- cal Friederich, Gabriel dos Passos Gomes, Florian Häse, Adrian Jinich, AkshatKumar Nigam, Zhenpeng Yao, and Alán Aspuru-Guzik. On scientific understanding with artificial intelligence. Nature Reviews Physics, 4(12):761-769 December 2022. ISSN 2522-5820. doi: 10. 10.1038/s444 00088//42254-0220005111 142254-022-00518-1
URL https: /ttttp::/www.... /www. nature, com/articles/
Tzu-Sheng Kuo, Aaron Lee Halfaker, Zirui Cheng, Jiwoo Kim, Meng-Hsin Wu, Tongshuang Wu, Kenneth Holstein, and Haiyi Zhu. Wikibench: Community-driven data curation for Al evaluation Wikipedia. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 24, New York, NY, USA, 2024. Association for Computing URL https://doi. org/
Machinery. ISBN 9798400703300. doi: 10. 10.1155/361390/ 3642278.
Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. Chapter 8 interviews and focus groups. In Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser (eds.), Research Methods in Human Computer Interaction (Second Edition), pp. 187-228. Morgan Kaufmann, Boston, second edition edition, 2017. ISBN 978-0-12-80539904. doi: https: //dbhoog/11..001//77744-----8--3 octemee//sttaee////////771111119
URL wttt://wwwcieeeeeeretttttcc com/
www.
12

---

Published as a conference paper at COLM 2025
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, Pp. 6449-6464, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 /00.////...emmmmmaa..... //aclanthology org/2023. emnlp-main mealp--mai..390 397/.
URL https:
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Bin- hang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: /ttp:///vee....////////r net mtt//forrummtttt1444211 Featured Certification, Expert Certification.
Q Vera Liao and Ziang Xiao. Rethinking model evaluation narrowing the socio-technical gap. arxiv preprint ar arXiv:2306.03100 2023.
Q. Vera Liao, Daniel Gruen, and Sarah Miller. Questioning the ai: Informing design practices for explainable ai user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, pp. 1-15, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450367080. doi: 10.1145/3313831.522 33376590 URL https://doi. org/10. 1115/3313311111.. 3376590.
Potsawee Manakul, Adian Liusie, and Mark Gales. SelfCheckGPT: Zero-resource black- box hallucination detection for generative large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 / www.002.emmllmmmmi...... //aclanthology org/2023. mamlp-main..55 557/
URL https:
Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. Co-writing screen- plays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23 New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450942255. doi: 10.1145/3544548.. 4558455255.33125. URL https: //doi. org/ 10. 1145/3544545 3581 225.
Michael Muller, Christine T. Wolf, Josh Andres, Michael Desmond, Narendra Nath Joshi, Zahra Ashktorab, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Evelyn Duesterwald, and Casey Dugan. Designing ground truth and the social life of labels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978145038096.. doi: 10.1145/ 3411764.3445402. URL /ttps:://oii org/10. 1145/3411764 3445402.
Nagaraj Mulukunte Narayana and M.K. Bhandi. Arxiv in the Open Access Era: its usage and impact on physics researchers. Bulletin of the AAS, 54(2), apr 27 2022. https://baas.aad wtgggpub/202222001666 2022n22016.
Tuan Dung Nguyen, Yuan-Sen Ting, Ioana Ciuca, Charles O'Neill, Ze-Chang Sun, Maja Jablońska, Sandor Kruk, Ernest Perkowski, Jack Miller, Jason Jason Jingsh Li, Josh Peek, Kartheik Iyer, Tomasz Rozanski, Pranav Khetarpal, Sharaf Zaman, David Brodrick, Ser- gio J. Rodriguez Mendez, Thang Bui, Alyssa Goodman, Alberto Accomazzi, Jill Naiman, Jesse Cranney, Kevin Schawinski, and Roberta Raileanu. AstroLLaMA: Towards spe- cialized foundation models in astronomy. In Tirthankar Ghosal, Felix Grezes, Thomas
13

---

Published as a conference paper at COLM 2025
Allen, Kelly Lockhart, Alberto Accomazzi, and Sergi ings of the Second Workshop on Information Extraction from Scientific Publications, PP. 49- 55, Bali, Indonesia, November 2023. Association for Computational Linguistics. doi: 10.18653/ /v1 wt1/2223.wessp-1..
(eds.), Proceed-
URL attps://aclantthon
org/ 2023. wiesp-1.71
Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. emrQA: A large corpus for question answering on electronic medical records. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, PP. 2357-2368, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: (118653/v1/D1e https://accaantooooo..or//////
URL
Dimitris Pappas, Ion Androutsopoulos, and Haris Papageorgion. BioRead: A new dataset for biomedical reading comprehension. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL BOtLhttss///clanthhott 1ttpo//oog...o//11111144
Dimitris Pappas, Petros Stavropoulos, lon Androutsopoulos, and Ryan McDonald. BioMRC: A dataset for biomedical machine reading comprehension. In Dina Demner-Fushman,
Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii (eds.), Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pp. 140-149, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1
URL
https://aclanthoolog/ org/2020. bionlp-1.15.
Dhavalkumar Patel, Prem Timsina, Ganesh Raut, Robert Freeman, Matthew Levin, Girish Nadkarni, Benjamin Glicksberg, and Eyal Klang. Exploring temperature effects on large language models across various clinical tasks. medRxiv, pp. 2024-07, 2024.
Ernest Perkowski, Rui Pan, Tuan Dung Nguyen, Yuan-Sen Ting, Sandor Kruk, Tong Zhang, Charlie O'Neill, Maja Jablonska, Zechang Sun, Michael J. Smith, Huiling Liu, Kevin Schawinski, Kartheik Iyer, Ioana Ciucă, and UniverseTBD. Astrollama-chat Scaling astrollama with conversational and diverse datasets. Research Notes of the AAS, 8(1):7, jan 2024. doi: 10.3847/2515-511 adlabe. URL https: http//xxxooioooo org/.e..///.. 3847 / ad1abe.
Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. Ai and the everything in the whole wide world benchmark. Jn Vanschoren and Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Bench- marks, volume 1, 2021. URL https://dttttststeeeeeemmaass-re paper_
neurips. cc/ pdf.
Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7346-7356, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 11863/vv1222 findings-emmlp.433 432. URL https:tt//// httpps://aclanthoolpg org/ 2024. findings-emnly 432/.
Chris Samarinas, Wynne Hsu, and Mong Li Lee. Improving evidence retrieval for automated explainable fact-checking. In Avi Sil and Xi Victoria Lin (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies: Demonstrations, pp. 84-91, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1 20211.aaall-dmmom.. Siwianttology..rr/228.. naacl-demos 10.
URL https:
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Steihen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev,
14

---

Published as a conference paper at COLM 2025
Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. Nature, 620 (7972):172-800, August 2023, ISSN 1476-4687. doi: 10.1038/
Evaluation of lims should not ignore non-determinism, 2024. URL 44113-6-222220022112222 URL abs/2407. 10457.
https://www. nature. www.articles/s4111111 586- 0tte:///4441180000-26226999 023-06291-2.
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy:
Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocaí Chen, Lu Chen, and Kai Yu. Scieval: a multi-level large language model evaluation benchmark for scientific research. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourleenth Symposium on Educational Advances Artificial Intelligence, AAAT24/IAAI / AAAT Press, 2024. ISBN 978-1-57735-887 doi: 00.1609/aaaixx3817..888 URL https: //doi.org/11 1609669///00 2009/aaai.. 292117.28772.
Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G. Nestor, Ali Soroush, Pierre A. Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F. Rousseau, Chunhua Weng, and Yifan Peng. Evaluating large language models on medical evidence summarization. upj Digital Medicine, 6(1):1-8, August 2023. ISSN 2398-6352. doi: 10.1088/$11446-02-0088888 URL https://www.. nature. con//articles//41766600000 0ates//47746-02--08888
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 809-88 819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653 /v1 1 URL https: httt:::s//acaanhhollogg/ hww.ooloogy.o.gg/eeeeee org/N18-10744
Dakuo Wang, Haoyu Wang, Mo Yu, Zahra Ashktorab, and Ming Tan. Group chat ecology in enterprise instant messaging: How employees collaborate through multi-user chat channels on slack. Proc. ACM Hum.-Comput. Interact., 6(CSCW1), April 2022. doi: 10.1145/ 3512941. URL https://dot org/10. 1145/3512941.
Jiyao Wang, Haolong Hu, Zuyuan Wang, Song Yan, Youyu Sheng, and Dengbo He. Eval- uating large language models on academic literature understanding and review: An empirical study among early-stage scholars. In Proceedings of the CHI Conference Human Factors in Computing Systems, CHI '4, New York, NY, USA, 2024, Association for Computing Machinery. ISBN 9798400703300. doi: 10.1145/3613900 04445/6613944.3419999 URL https:///oi org/10. /145/31111113 3904 3641917.
Joel Wester, Tim Schrills, Henning Pohl, and Niels van Berkel. "as an ai language model, i cannot": Investigating I1m denials of user requests. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI '24 New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703300. doi: 10.1145/36139044.642222 URL https://doi. org/ 00 391.3361994444411339
John Wu, Alina Hyk, Kiera McCormick, Christine Ye, Simone Astarita, Elina Baral, Jo Ciuca, Jesse Cranney, Anjalie Field, Kartheik Iyer, Philipp Koehn, Jenn Kotler, Sandor Kruk, Michelle Ntampaka, Charles O'Neill, Joshua E. G. Peek, Sanjib Sharma, and Mikaeel Yunus. Designing an evaluation framework for large language models in astronomy research, 2024. URL https: / www.i..vvv..........a........... atsp//bs/22405 20389.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 24, pp. 641-649, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704314 doi: 10.1145
10.1145/366677.3657777 //////666777777777777 URL https: /ttp::://doiooroo org/18. 1145/
1626772.3657878
15

---

Published as a conference paper at COLM 2025
Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and Ray LC. AI as active writer: Interaction strategies with generated text in human-Al collaborative fiction writing. In Alison Smith-Renner and Ofra Amir (eds.), Joint Proceedings of the IUI 2022 Workshops: APEx-UI, HAI-GEN, HEALTHI, HUMANIZE, TExS5, SOCIALIZE, volume 3124 of CEUR Workshop Proceedings, pp. 56-65, Virtual Event, Helsinki, March 2022. CEUR. URL https::/ceuurws.orrgg/ wtt.://rr//ooo11/////// (ee--33124HHpppeee #paper6.
Chen Zhang. User-controlled knowledge fusion in large language models: Balancing creativity and hallucination, 2023. URL httLhttss////riiv.ooo /as//bbss/22 16139.
Tianyi Zhang*, Varsha Kishore", Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating text generation with BERT. In International Conference Learning Representa- fions, 2020. URL httpt::/ wwwpreerreeee..eeeffruueee
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wild- Chat: 1M ChatGPT interaction logs in the wild. In The Twelfth International Conference Learning Representations, 2024. URL / net www..e//oorrmm1118888888/722
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, and Alexan- dra Olteanu. Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 314-324 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/ /vl 2022.naacl-main.24 URL https: /aclaaaaaallooyyyy...ooo org//022....2/ 2022. naacl-main. 24/.
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: How ex- pressions of uncertainty and overconfidence affect language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, Pp. 5506-5524, Singapore, December 2023. Associ- URL
ation for Computational Linguistics. doi: 10.18653 111553/11/// http2222..mmmmmmmam https://aclanthol..
org/2023. emnlp-main 335.
Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. Relying the unreliable: The impact of language models' reluctance to express uncertainty. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3623-3643, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL http::://alaathllogg org/2024. acl -long.n..... 198.
16

---

Published as a conference paper at COLM 2025
A Appendix: System Description Details
Our RAG system follows the framework outlined Wu et al. (2024). In early trials, we included the full-text paper context, but this frequently resulted in incorrect citations; the RAG-based Slack bot would often focus on references in a paper's introduction or the related works section, rather than the actual results from the paper. Therefore, we fetch just the abstracts and conclusions from each arXiv paper (which is extracted and cleaned via the arXiv LaTeX source code). We also insert the metadata for each paper into the prompt string. We only include the five most semantically similar papers (determined via cosine similarity) so as not to exceed the 128,000-token context length of the model popt-44-2024-000
In the code listing below, we provide the full prompt template used by the RAG-based chat bot. The full prompt concatenates the context string (context.str), which contains the arXiv ID, paper year, abstract, and conclusions section. After providing the instructions and the context string, we append the query. This prompt template resulted from multiple iterations from our team's 's initial tests.
You are an astronomy research assistant that can access arXiv paper chunks as context answer user queries. Some of the context may contain LaTeX code, but please answer the query in natural language like a professional research astronomer. Match the level of specificity or generality as the query. ALWAYS cite ALL relevant papers using EXACTLY the citation style in the context, in parentheses:
2364 1406 2364>) The current year is 2024. Unless directed otherwise, prioritize MORE RECENT results ts based on the paper YEAR. Answer in 100 words or fewer. the query not related to astronomy in any way, or if none of the papers can help you answer this, then say 'I cannot answer'.
The arXiv astro-ph papers context string are below:
(context_str)
Query: [query_str)
Answer:
B Appendix: Distribution of Participants Across Demographic Categories
Gender
Race
Year PhD Completion
Man Woman
25
Asian
<2000
5
10
Hispanic/LatinX
2000-2000
9
White
26
2010 - 2015
8
White; Hispanic/Latinx
2011
11
Career Track
Age
Astronomer
15
<34
Postdoc/ Fostdoc/Fellow Fellow
3
35-44
12
Scientist Self-identified Researcher
7
45-54
10
8
55-64
8
>64
2
Table 3: Counts of answers selected on an optional self-reported demographic survey provided to all participants. Not all participants answered every question.
17

---

Published as a conference paper at COLM 2025
C Appendix: Descriptions and Examples for Query Types
Description
Example
Knowledge seeking: specific factual
Questions about very specific pieces of information, such as characteristics, facts, parameters of specific objects, phe- nomena, or processes.
What is the most massive known spi- ral galaxy in the universe?
Deep knowl- edge (Including opinion or spec- ulation)
Research-focused inquiries focusing on brainstorming, summarization, or even speculation and opinions after tensive information searches. Such queries often require deep analysis and multi-level reasoning. Examples are queries requesting detailed explanations or clarifications on specific patterns or phenomena, comparisons evalua- tions between two or more entities or performance within a specific context.
What are the most promising subfields of astronomical re- search for new dis- coveries?
Bibliometric search: topic
Questions that search for references on a specific matter, topic, option piece, or phenomenon. Rather than broader portraiture analysis, the question is seeking specific and highly relevant references.
Please list the most recent papers multiple popula- tions in Globular clusters.
Knowledge seeking: broad description or common sense
Questions that have distinct and correct answers, but an- swers are broad and descriptive, such as paragraph about phenomena and processes
What physical characteristics indi- cate the transition between a brown dwarf and low mass star?
Stress testing
Questions that are unrelated to the astronomy, intentionally misleading, or generally try to bypass guardrails
Why are the cores of all rocky planets made of Silicone?
Bot capabilities
Questions that seek information the bot's capabilities, features, scope of knowledge, or limitations. Questions are not directly related to astronomy or information retrieval, but rather seek information on how to interact with the bot and what its characteristics and capabilities are.
What sort of ques- tions can you answer? Provide your response in the form of bulleted list.
Knowledge seeking: proce- dure
Questions that seek an answer in terms of instructions, descriptions the steps or flow of specific task, a broader picture, or recommendations on how given procedure should be performed how can be interpreted, analyzed, and applied.
How can com- pressed sensing be used for image deconvolution radio interferome- try?
Bibliometric search: specific paper or author
Questions that request details, analysis, or otherwise refer to specific paper or author (often by providing the name of the author, title of the paper, or link to the paper).
Can you list 3 papers published by <redacted:
name>?
Unresolved topic
Questions about topics that are controversial, don't have a single clear answer, are generally unsolved.
Does TRAPPIST-Ib
has an atmosphere?
Table 4: Descriptions and examples for query types as determined through inductive coding. Query types are ordered for consistency with Table 1.
18

---

Published as a conference paper at COLM 2025
D Appendix: Variations in Question Type Frequencies
In Figure and Figure 4, we show bar charts on the frequency of question types by the gender and the year of PhD completion of the participants.
Gender
Man
Woman
Biblommtric
Bibliometrcc
capabilities
Knowledge
Knowledge
Knowledge
testing
Unresolved
search:
knowledge
seeking: broad
seeking: procedure
seeking:
topic
specific paper
topic
description
speciicc
author
factual
Figure 3: Frequency of question types by gender for men (24) and women (10), as self- reported on an optional demographic survey. The Y axis denotes the percent of questions asked by people of the specified gender that were coded as that type. We report percentages rather than raw counts due data imbalance. Distributions are remarkably similar.
Nittometicc swarch specific poow auttor
knowledge sooking: speciie faclual
bot capabi Ries
bisfometic swardh lopic
stress lesting
deep knowlddge (nnluuing opinion speculation)
kncolusigge sweking disasiption
ueeesiieed
Incwledge sooking: procedure
Count
<2000
2010-2015
Year of PhD Completion
Figure 4: Frequency of question types by the year of PhD completion, self-reported optional demographic survey. The number of participants in each category is reported in Appendix B. There is variance in frequencies of question types. For example, stress testing questions were asked more frequently by people who completed their PhDs after 2010.
Figure shows a bar chart question types split by gender. The y-axis depicts the per- centages of questions for each question type (assorted into inductive codes). The gender-disaggregated question type frequencies are generally similar, with small differences only in frequencies of Stress Testing and Unresolved Topic questions.
Figure shows a bar chart of question types split by year of PhD completion. The x-axis labels specifically < 2000, 2000 2009, 2010 2015, and 2015+. There are more Stress
19

---

Published as a conference paper at COLM 2025
Proportional Distribution Thumbs Up and Down by PhD Pho_yearr
100%
Percantage
75%
53.3%
54.5%
62.35
50%
73.3%
25%
46.7%
46 10.0%
47.7%
<2000
2000-2009
2010-2015
2015
PhD PhD_ycar
Thumbs
thumbs down
Jumbs up
Proportional Distribution Thumbs Down Age
Percentage
75%
38.3%
43.3%
47 0%
75. 0%.0%
50%
103.0%
79.1%
25%
68.7%
53.0%
25.0%
38.4%
18-24 years old
25-34 years old
35-44 years old
45-54 years old
55-64 years old
65-74 years old
75 years or older
Age
Thumbs
thumbs_down down
chumbs_up
Figure Distribution of how users rated the chat bot separated by year of PhD completion and age. Younger users seemed to rate the bot more positively than older users.
Testing and Bot Capabilities questions in the 2010-2015 and 2015+ groups. Knowledge-seeking questions are generally evenly distributed.
Figure 5 shows the distribution of how users rated @Ask-astro-ph by year of PhD completion and by age. Users who graduated from the PhD more recently (015-) users (e.g., < 45 years old) rated the bot more positively.
2015+) and younger
E Appendix: Temporal Evolution of User Queries
Figure shows the distribution user queries by the ID for each user and the date of the query. The x-axis shows the ID for each user and the Y-axis shows the general date that queries were asked to the bot. The colors of the points are related to each different question type seen in the legend to the right. Each individual user asked variety of questions throughout the time period.
Appendix: Sample Benchmark Application
A large issue in studying LLMs that has arisen recently is the fact that it is quite difficult to understand how to properly evaluate the quality of responses. How can researchers quantitatively score nuanced answers to complex topics with multiple correct debated results? Using the gold benchmark dataset, we were able to compare responses from the chat bot with the expert-curated answers. We used separate LLM to score the similarities of each chat bot response against the gold benchmark answer, and returned a relevance score between 0 (irrelevant to the gold sample) and 1 (perfectly matching the gold sample).
20

---

Published as a conference paper at COLM 2025
Open Codting coregryy
June
Calegory
Quey Tinesammo
hélienzti search spcc paper or outor
ballc nesionetl search aap2
June 1
sed capatilaas
deop lnowled trnck alin opt spnc. lateer)
knowledge seeling: broad desortatin
knowlcdge scoling: procod
krswkkkge wodking, liginn sppolfis lastaal
losling unvvousbee
May
User
Figure 6: Frequency of question type over time for each user. Each user asks variety of question types throughout their interactions and there is large variance by user in the distribution and ordering of question types.
An example usage of this gold benchmark dataset in this study was making judgments about the quality of two different chat bot system responses. After initially concluded collecting user data over the course of four weeks, we internally created and tested a second upgraded chat bot with an improved retrieval system, prompts, and other adjustments. The new system is described in (lyer et al., 2024). Using the gold benchmark dataset, we compared responses from the initial and upgraded chat bots. The relevance scores provided by the LLM judge provided with accurate and useful numbers that were able to be plotted and analyzed. On average, the updated chat bot responses scored 0.2-0.3 points higher than the original chat bot system, allowing us to be able to quantify the quality of LLM-generated responses without needing to be an astrophysics expert.
The prompt that was used to generate these relevance scores is included below. The model used was gpt-4o-mini.
Task: Evaluate the relevance and quality of a response compared to a gold standard response.
Original Query: (query_str}
Gold Standard Response: {gold_ response) Response to Evaluate: (response}
Instructions:
1. Compare the content and meaning of the two responses.
2. Consider the following aspects:
Accuracy: How factually correct is the response?
Completeness: Does it cover all key points from the gold standard?
Relevance: How well does it address the original query?
21

---

Published as a conference paper at COLM 2025
Coherence: Is the response well-structured and logical?
Conciseness: Is the information presented efficiently?
3. Assign relevance score from 0 to 1, where:
1.0: Perfect match in content and quality
0.8-0.9: Excellent, with minor differences
00.6-07: Good, captures main points but misses some details
-..-005: Fair, partially relevant but significant gaps
-
0.2-0.3: Poor, major inaccuracies or omissions
0.0-0.11 Completely irrelevant or incorrect
Provide your relevance score as float between 0 and 1.
22


Reviews:
[{"review_id": "yPmaVcNh5W", "reviewer": "Reviewer_g8n8", "summary": "The authors present an exploratory study into how expert users of a domain-specific search engine query, interact and assess the RAG-based system that has been deployed. The two underlying research questions aim to answer (a) what types of questions users (in this case astronomers) submit to the system and (b) how users actually judge the quality of the system. Topically this is a great fit as the work addresses a range of key topics spelled out in the Call for Papers.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is not just well-written but the authors also provide a detailed appendix that aims at making the work as reproducible as possible which is a major strength. \n* Reproducibility and replicability have also been taken into account in the Methodology section. Despite the shortcomings (see below) the authors have done a great job here.\n* The discussion of limitations is compelling (but the page limit has been violated by placing it on an extra page beyond the imposed 9-page limit).\n* The created benchmark will be an interesting asset for the community (although it is very small overall).", "reasons_to_reject": "* The work feels more like a pilot study that should be followed up by a more systematic investigation. All the investigations and findings are very interesting but it does not become clear how generalisable they are which is due to a number of reasons including (a) a very small number of queries, (b) concerns around the representativeness of the setup (including dataset, users, questions etc.) and (c) the lack of a plausible baseline which would allow a comparison between the proposed architecture and how users currently conduct their research. Among other things such baseline would provide a very important alignment between what the RAG-based system might offer vs. what a standard (strong) baseline offers already. \n* From a community perspective, how representative are the recruited users for the community as a whole (or for a specific downstream use case)? \n* There is a lack of contextualisation with related studies in Section 3.2 but also in parts of Section 4. What other studies have been conducted (in other communities and contexts) that can be compared to this work. Clearly inductive coding is a method that has been used widely. Pros and cons of different alternatives should be discussed, and the results should be compared and contrasted with previous studies. Just to pick an example: Krippendorff’s alpha is very low making generalisable insights difficult. How does that compare to other studies? Section 4.2 is better in this respect in that some related work is being referenced.", "questions_to_authors": "* More of a suggestion than a question: I get the impression that the work resembles a specific use case of “professional search” which is distinctly different from other types of search (such as web search). I would contextualise this as part of the related work (a good recent reference could be the chapter on Professional Search by Suzan Verberne in the recently published IR book by Alonso and Baeza-Yates: https://dl.acm.org/doi/10.1145/3674127.3674141) \n* Is there some evidence that the arXiv collection is representative for what astronomers use as their first point of call? After all these are papers that have not been peer-reviewed.", "extra_scores": {}}, {"review_id": "oLVAPFNDiF", "reviewer": "Reviewer_aurw", "summary": "This paper presents an interesting study on how researchers (in this particular case, astronomers) use LLMs for their research activities. The authors deployed a GPT-4o-based slackbot and recorded the interactions between researchers and the bot. Behind the bot is a RAG system that is connected to a database of astronomy papers. The authors focused on answering two questions: in a realistic research pipeline, (1) what types of questions people ask and (2) what criteria people use to judge the bot. Answering these questions involves careful manual annotations of the questions and extensive interviews with the participants. Based on the findings, the authors made some recommendations on how to build a benchmark with realistic and indicative criteria, and collected some expert-annotated answers to form a high-quality astronomy datasets for researcher-LLM interactions.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study is interesting and valuable in multiple aspects:\n\n(1) The data it provides (the collected questions, question type annotations, participants' reactions to LLMs' answers, interviews with participants, expert-annotated answers) is extremely valuable for LLM developers, evaluation researchers, and domain experts who intent to use LLMs.\n\n(2) The data quality is very high: the authors described in details how they resolve annotation conflicts and ensure expert-written answers are high-quality. The description of the whole experiment pipeline (slackbot deployment) is also very clear.\n\n(3) The discussion around \"good criteria\" is interesting: instead of just arguing certain perspectives are important (which a lot of prior work has done), the authors interviewed participants and summarized their opinions (as real users who tried out the system). It is certainly very inspiring for future evaluation frameworks.", "reasons_to_reject": "(1) Lack of discussion on existing (and future) evaluation mechanisms. There have been several works on this type of citation-based question answering LLM evaluation, such as [Gao et al., 2023](https://arxiv.org/abs/2305.14627) and [Asai et al., 2024](https://arxiv.org/abs/2411.14199). They discussed a similar application scenario and also detailed automatic evaluation metrics that could reflect user perspectives. The discussion on criteria in this paper is interesting, but not concrete. \n\n(2) Limited scope: the paper mainly focuses on Astronomy. The interview is limited to 11 participants and the final dataset only has 42 question-answer pair.\n\nThat being said, I still believe the paper presents some very interesting observations and data to the LLM community, and the above shouldn't become reasons to reject.", "questions_to_authors": "Please see reasons to reject. One question: why use gpt-4o-mini for the human/LLM judgement correlation experiment? Why not use a more powerful model?", "extra_scores": {}}, {"review_id": "TScGPFm0Hh", "reviewer": "Reviewer_6qMG", "summary": "**Paper summary:** The paper presents the results of an empirical study where astronomers were given access to retrieval-based LLM summaries of user queries. Specifically, users were able to query a bot through Slack, which would retrieve relevant papers using similarity with pre-computed embeddings of a fixed set of papers. Relevant papers were then included in the context of a proprietary LLM to generate responses to user queries. The paper studies (1) what kinds of queries the users used, and (2) how they evaluated the responses, based on post-study interviews.\n\n**Pros:** The paper is clear and well-written, using a good mix of tables and figures for presentation. I also appreciate the focus on a domain science -- astronomy -- at a time when domain scientists are trying to understand how to best leverage LLMs for research. I think this is the main dimension along which the paper spikes.\n\n**Cons:** Methodologically, the study described in paper is somewhat weak, to the extent that it's unclear what conclusions to actually take away. As the authors do acknowledge, the sample size of the study is quite limited, with the final response evaluation criteria based on 11 interviews at the specific institution the study was conducted in. It is unclear how the participants themselves approached the bot during the study -- e.g., as an object to stress-test (as indicated by some queries), or as a tool for serious research, confounded by the fact that they were being \"watched\". The tools used in the study themselves are somewhat outdated -- using RAG embeddings of a small set of papers up to July 2023. In an era where LLMs can conduct large-scale searches not limited to a fixed set of papers, it is unclear whether the study actually reflects real-world use cases. Finally, the authors introduce a \"golden\" benchmark for evaluating LLM responses to astronomy queries. However, no details on the benchmark are included in the main content of the paper, or how in practice it can be used as an evaluation tool for LLMs, so I am not considering it in this review.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clear and well-written paper, with good presentation;\n- Application to a quantitative domain science -- astronomy -- which is well-suited for many LLM use cases.", "reasons_to_reject": "- The study is based on only 11 interviews from a single institution, severely limiting generalizability of findings to the broader astronomy community or other scientific domains. With only 35 users generating 367 queries over 4 weeks, the dataset is too small to draw robust conclusions about query patterns or evaluation criteria for astronomical research;\n- The study doesn't adequately control for how participants approached the task -- whether they were genuinely using the bot for research or merely stress-testing it due to being observed, which confounds the interpretation of results;\n- Outdated stack: The system uses pre-computed embeddings from papers up to July 2023 with fixed paper sets, which doesn't reflect current LLM capabilities for dynamic, large-scale literature search. Given this, it's unclear whether it reflects practical real-world use cases;\n- While the paper introduces a \"golden\" benchmark for evaluation, it provides no implementation details or demonstration of practical utility in the main paper.", "questions_to_authors": "- Given concerns about generalizability, the tech stack (fixed set of papers, no up-to-date knowledge), and user intent, to what extent would you expect the study to accurately reflect modern and forward-looking uses cases of LLMs in astronomy?\n- How did you control for the so-called Hawthorne effect (i.e., participants modifying behavior because they were being observed)? Can you distinguish between users who were genuinely using the bot for research versus those who were stress-testing it because they were participating in a study?\n- Benchmark details: how can the \"golden\" benchmark mentioned in the paper be used practically, e.g. to evaluate current general-purpose LLMs?\n- Are there variations across career stage that may be relevant in the study? (I would think this is particularly interesting to understand adoption patterns)", "extra_scores": {}}, {"review_id": "033tKVoNJZ", "reviewer": "Reviewer_Sraf", "summary": "This paper conducts an in-depth study into the interactions between astronomers and LLM bots. The bot is hosted on a slack channel where the users (astronomy PhDs) can query the bot and evaluate the response via emoji reactions and comments. The bot is a simple RAG pipeline that retrieves paper abstracts from arXiv and use GPT-4o to generate the answer and related citations.\nThe main focus of the paper is to gain a better understanding of what types of questions do astronomers ask and how do they judge responses. Thus, the authors conducted in-depth analysis into the types of user queries, categorizing them into concrete types (e.g., knowledge seeking, stress testing, etc.).\nThen, the authors conducts user studies that reveals how experts use and evaluate the responses. These qualitative results lead to recommendations to the community on how to build better evaluation settings.\nFinally, the collected queries are used to construct a new benchmark.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well motivated—it studies the important setting of how domain experts may interact and use the latest LLM systems. The paper chooses a suitable setting for studying this. \n- The qualitative analyses are extensive and insightful given the difficulty of getting a large number of domain experts. Despite being limited to the field of astronomy, it is likely that many of the takeaways and insights can be applied to many other fields. In particular, the categorization of the queries and different aspects of how experts evaluate the response can inform future work.\n- The analyses and details over the breakdown across different demographics are insightful—the difference between PhDs with varying number of years of experience motivates for diverse evaluation. \n- The resulting artifact—the sample benchmark—can be useful for evaluating LLMs, though further validation and additional evaluation framework may be necessary.", "reasons_to_reject": "- Although the proposed sample benchmark can be useful to the community, it could benefit from further validation of the evaluation framework. For example, it may be more insightful to break the evaluation into multiple aspects as listed in the Table 2 (e.g., correctness of citations vs. matching between responses and citation) as opposed to putting everything into just one number. \n- The main metric for evaluating the RAG system is via the thumb up and thumb down reactions to the responses, but it is not clear how reliable these reactions are. For example, what are the agreements between different users on the same response? From the later analysis, it’s clear that the quality of the responses cannot be condensed into one of two reactions. Further expert validation could be insightful.\n- The evaluation is limited to one specific system. While it is understandably difficult to evaluate multiple systems due to the human effort, it would be incredibly useful to have some comparisons between different systems; for example, one could provide two responses and ask the user to judge which one is between. \n\nMinor comments:\n- Missing related work for discussions: \n    - OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs (Asai et al., 2024)\n- Appendix A: “128,000-character context length of the model” should be “128,000-tokens”", "questions_to_authors": "- What is the Year of PhD completion breakdown between 2016-2020 and >2020?\n- What are the demographics and qualifications of the 7 astronomers who wrote the answers in the benchmark?\n- Are the expert-written answers in the sample benchmark cross-validated?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates how astronomers interact with and access a RAG system built on a domain-specific retrieval component. The topic is a strong fit for COLM, and I believe both the study and the collected dataset will be highly valuable to the community. I consider the paper worth publishing at COLM and recommend its acceptance.

I do not see any major concerns that would justify rejecting the paper:
* The study is relatively small in scale (11 interviews, 35 users generating 367 questions), but I acknowledge the challenge of expanding the scope given the need for deep domain expertise.
* The paper could benefit from a discussion on how the evaluation criteria might be automated, along with references to relevant work, though this can reasonably be left for future work.

I encourage the authors to address these points carefully in the final version.
Decision: Accept