Title: Can LLMs replace Neil deGrasse Tyson? Evaluating the Reliability of LLMs as Science Communicators
Benchmark: SCiPS-QA
Summary: This paper focuses on evaluating the reliability of current LLMs as science communicators. They introduce a dataset, SCiPS-QA, comprising 742 Yes/No queries embedded in complex
scientific concepts, along with a benchmarking suite that evaluates LLMs for correctness and consistency across various criteria. They also benchmark three proprietary LLMs from the OpenAI GPT family and 13 open-access LLMs from the Meta Llama-2, Llama-3, and Mistral families.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Reliability of LLMs as Science Communicators
Phenomenon defined by authors: No
Phenomenon definition: Can existing LLMs answer scientific reasoning questions successfully and faithfully that require understanding the nuances of scientific knowledge?
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: A binary (yes/No) classification task where the model is asked to answer a scientific question.
Task item definition: A question in science 
Task source: Not explained 
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 742
Dataset metadata available: Yes
Dataset metadata detail: topic, date
Sampling method: Unknown
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean and standard deviation
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: General Science / 
New taxonomy: Specific Application / Physical Sciences