PRobELM: Plausibility Ranking Evaluation for Language Models
Authors: Moy Yuan
Eric Chamoun
Rami Aly
Chenxi Whitehouse
Andreas Vlachos
Venue: COLM (2024)
Keywords: plausibility, benchmark, evaluation
TL;DR: This paper introduces PRobELM (Plausibility Ranking Evaluation for Language Models), a benchmark designed to assess language models' ability to discern more plausible from less plausible scenarios through their parametric knowledge.

Abstract: This paper introduces PRobELM (Plausibility Ranking Evaluation for Language Models), a benchmark designed to assess language models' ability to discern more plausible from less plausible scenarios through their parametric knowledge. While benchmarks such as TruthfulQA emphasise factual accuracy or truthfulness, and others such as COPA explore plausible scenarios without explicitly incorporating world knowledge, PRobELM seeks to bridge this gap by evaluating models' capabilities to prioritise plausible scenarios that leverage world knowledge over less plausible alternatives. This design allows us to assess the potential of language models for downstream use cases such as literature-based discovery where the focus is on identifying information that is likely but not yet known. Our benchmark is constructed from a dataset curated from Wikidata edit histories, tailored to align the temporal bounds of the training data for the evaluated models. PRobELM facilitates the evaluation of language models across multiple prompting types, including statement, text completion, and question-answering. Experiments with 10 models of various sizes and architectures on the relationship between model scales, training recency, and plausibility performance, reveal that factual accuracy does not directly correlate with plausibility performance and that up-to-date training data enhances plausibility assessment across different model architectures.

Published as a conference paper at COLM 2024
PROBELM: Plausibility Ranking Evaluation for Language Models
Zhangdie Yuan, Eric Chamoun', Rami Aly', Chenxi Whitehouse", Andreas Vlachos Department of Computer Science and Technology University of Cambridge {zy317, ec806, rmya2, av308 @eamm. ac. uk, chenxi wwwossseee cam.
Abstract
This paper introduces PROBELM (Plausibility Ranking Evaluation for Language Models), a benchmark designed to assess language models' abil- ity to discern more plausible from less plausible scenarios through their parametric knowledge. While benchmarks such as TruthfulQA emphasise factual accuracy truthfulness, and others such COPA explore plausible scenarios without explicitly incorporating world knowledge, PROBELM seeks to bridge this gap by evaluating models' capabilities to prioritise plausible scenarios that leverage world knowledge over less plausible alter- natives. This design allows us assess the potential of language models for downstream use cases such iiterature-base discovery where the focus is on identifying information that is likely but not yet known. Our benchmark is constructed from a dataset curated from Wikidata edit his- tories, tailored to align the temporal bounds of the training data for the evaluated models. PROBELM facilitates the evaluation of language models across multiple prompting types, including statement, text completion, and question-answering Experiments with 10 models of various sizes and architectures on the relationship between model scales, training recency, and plausibility performance, reveal that factual accuracy does not directly correlate with plausibility performance and that up-to-date training data enhances plausibility assessment across different model architectures.
Introduction
Recent advancements in large language models (LLMs) have significantly enhanced their performance across wide range of NLP tasks. Alongside these developments, various benchmarks and datasets are introduced effectively assess the capabilities of LLMs, particularly in terms of knowledge and reasoning (Roemmele et al., 2011 Wang et al.) 2018 Clark et al. 2018 Zellers al. 2019 Wang et al.] Sakaguchi al. 2021: Lin et al. 2022; Whitehouse et al. 2023 Li et al. 2023). However, these benchmarks often
focus predominantly on evaluating factual accuracy or reasoning abilities without explicitly incorporating broader world knowledge.
For instance, benchmarks like TruthfulQA (Lin et al. 2022 are designed to assess the truthfulness factual correctness of LLMs, evaluating their ability to retrieve and apply information encoded during training, including tasks like mathematical induction. However, they do not explicitly address LLMs' capacity to discern plausibility in scenarios where strict factual accuracy might not be directly applicable. On the other hand, datasets such as COРA (Choice of Plausible Alternatives) (Roemmele et al. 2011) evaluate models through causal reasoning tasks, asking the model to choose the more plausible scenarios from the two options for a given premise. For example, given a premise "I tipped the bottle", scenario "The liquid in the bottle poured out" is more plausible than "The liquid in the bottle froze". While COPA extends evaluation beyond mere factual accuracy by introducing
Equal Contribution. More senior authors are listed towards the end of the author list.
Our dataset and code are available at https::///// htwggtttub..et wtt.lcccoo/caaeeee http:/.....a.ncee/eeeaeaaaaa http://aandeee/oREE.l
1

---

Published as a conference paper at COLM 2024
Knowledge tearnt from pre-training
Inference with no additional context
Marcelo Júnior is a Brazilian professional footballer.
Marcelo is Brazilian
Factual
He has been Real Madrid player since the January transfer window in 2007.
Scenarios
More plausible
Marcelo Spanish
Article requires that natural-born citizens Ibero- American countries must have legally resided Spain for 2 years to apply for Spanish nationality.
Marcelo is Irish
Ibero-America includes Brazil.
Marcelo Is Chinese
Less plausible
Figure 1: This figure illustrates the utility of plausibility evaluation in language model outputs: model, trained on data up until 25 July 2011, ranks "Marcelo is Spanish" as the most plausible among non-factual scenarios. This judgement is based on the model's parametric knowledge of Marcelo's career with Spanish club, his residency in Spain and the Spanish naturalisation law. The plausibility of this scenario is confirmed as Marcelo was granted Spanish citizenship on 26 July 2011.
plausible alternative scenarios, it operates within the constraints of an artificially constructed dataset, potentially oversimplifying evaluation by limiting tasks to binary choices. Notably, fine-tuned state-of-the-ar pi such as PaLM and PaLM 2 have shown near-perfect performance COPA (Chowdhery al. 2023 Anil et al. 2023), indicating necessity for more complex and challenging benchmarks that better capture the subtleties of world knowledge and plausibility.
Furthermore, in certain domains, combining broad world knowledge and assessing plau- sibility is crucial. One such domain is literature-based knowledge discovery, particularly within the biomedical field, which can be accelerated by guiding experimental efforts lever- aging existing literature (Gopalakrishnan et al. 2019), requiring the capability of deriving educated guesses that may not be immediately verifiable. In this context, the ability of LLMs to navigate through and infer plausible scenarios from world knowledge becomes invaluable. Yet, as SCIMON (Wang et al. (2023) demonstrates, while LLMs can generate hypotheses, they often lack technical depth and novelty (Wang et al. 2023 contribution to specialised fields.
hindering their
To address the limitation in plausibility evaluation, we introduce PROBELM, a novel bench- mark designed to directly assess the plausibility inference capabilities of LLMs. PROBELM utilises scenarios collected from real-world data through Wikidata, comprising two pri- mary components: new facts sourced from Wikidata that were unknown to models due to the timeframe of their training data, and set of automatically generated less plausible scenarios, and asks models rank scenarios from most least plausible. For instance, consider a scenario where model is asked to determine person's (Marcelo) nationality based on provided facts about their football career and immigration laws, as illustrated in Figure 1 Provided with the available information in the training data of a model such as "Marcelo is Brazilian footballer" and "He has played for a Spanish club since 2007", along with pertinent legal information regarding immigration, like Tibero-Amerin citizens must have lived in Spain for 2 years to apply for citizenship" and TTbero-Amerts includes Brazil", the model is expected to rank "Spanish" as the most plausible nationality, reflecting its ability to reason with encoded parametric knowledge.
natural-bon
We evaluate PROBELM with 10 language models varying in architecture, parameter scale, and the recency of their world knowledge updates. To ensure alignment in the pre-training data sourced from Wikipedia, we select open-source models that disclose the timestamp of the utilised Wikipedia dump in their training, including GPT-2 [Rafford et al.) 2019), BLOOM (Scao al. 2023), LLaMA (Touvron et al. 2023), Pythia (Biderman et al. 2023 and
2

---

Published as a conference paper at COLM 2024
OLMo (Groeneveld et al. 2024), spanning parameter counts from 14 million to 7 billion and training data from 2017 to 2023. Our empirical results reveal the following key observations:
Models with higher performance on factual accuracy do not necessarily excel in plausibility, indicating a divergence between factuality and plausibility capabilities.
Larger models generally exhibit better plausibility inference, but this trend varies across model families.
Models trained on less recent data can sometimes outperform counterparts that use more contemporary datasets, demonstrating that model architecture and training methodologies also influence plausibility inference, independent of model size.
A notable trend suggests that the greater the temporal gap between a model's training data cut-off and the date of the evaluation set, the poorer the model performs on PROBELM, underscoring the importance of up-to-date knowledge in evaluating plausibility.
2
Related Work
With the rise more powerful language models in the form of LLMs, sophisticated reason- ing benchmarks such as the Open LLM Leaderboard have been proposed. The benchmark incorporates tasks such as HellaSwag (Zellers et al. 2019), which evaluates commonsense reasoning within physically situated contexts. Similarly, COPA confronts models with com- monsense scenarios. Yet, these benchmarks do not task the model to incorporate knowledge about the world state into its prediction. On the flip side, benchmarks such TruthfulQA (Lin et al. 2022), FactScore (Min et al., 2023), and HaluEval (Li et al. 2023 Yu et al. 2023
measure the factuality of model's response exclusively. However, previous benchmarks rarely address the complex interplay between world knowledge and plausibility.
Factual errors in an LLM's output are also referred to hallucinations (Xu et al. 2024) and occur when models generate content that is unfaithful to the provided input, established context, or factual correctness. Jii et al. (2023) categorises these into iiput-conflictnng context- conflicting, and fuct-conflicting hallucinations, with the latter category being particularly problematic due to its contradiction with known facts.
While all scenarios in the PROBELM benchmark are technically fact-conflicting halluci- nations (as they are not yet facts of the world the LLM has seen), the crucial point that under existing world knowledge, non-factual scenarios are not all equally plausible. Instead, PROBELM probes a model's ability to make educated guesses about plausible futures to gain insights into the capabilities of LLMs to infer within the limits of their knowledge beyond recalling facts it has already seen.
The importance of benchmark that measures the interplay between world knowledge and plausibility in the context of hallucinations underscored by recent work. Hao et al. (2023) repurpose LLMs both world model and reasoning agent for planning while Gruver et al. 2023) show that LLMs are time series forecasters, generating plausible continuations for existing data. Despite these positive signals, Mündler al. (2024) observes a signifi- cant presence of self-contradictory outputs in LLMs, concretely in 17.7% of all sentences produced by ChatGPT, indicating inconsistent use of world knowledge. This finding is supported by Ye et al. 20224), who observe that larger-scale LLMs demonstrate greater uncertainty compared their smaller counterparts. Finally Kim et al. (2023), highlighted LLMs' difficulties in maintaining consistent reasoning through the FANToM framework.
3 PROBELM Construction
PROBELM is inspired by acknowledging the middle ground between absolute fact and pure fiction, a domain abundant with feasible scenarios. These scenarios, grounded in logic yet not strictly confined to facts, crucial for applications such as knowledge discovery. Plausible inferences are valuable in exploring existing literature and uncovering potential connections or hypotheses that may spark meaningful scientific investigation, even if not
3

---

Published as a conference paper at COLM 2024
immediately verifiable. PROBELM seeks to address significant gap in contemporary evaluation paradigms, establishing a novel benchmark for assessing the proficiency of LLMs in navigating the complexities of real-world knowledge application.
In the illustrative scenario provided in Figure 1 a model trained with data available until the 25th of July, 2011, is asked to evaluate the plausibility of various nationalities for the Brazilian footballer, Marcelo. This task requires the model to engage in deductive reasoning, utilissing its accumulated knowledge to prioritise potential nationalities based on their likelihood. Specifically, the model is expected to integrate the information pertaining to Marcelo's s eligibility for Spanish citizenship, thereby identifying "Spanish" as plausible nationality. This example highlights the PROBELMMss focus on models' ability to perform deductive reasoning and to apply world knowledge.
PROBELM assesses language models' capacity to determine plausibility through three different structured prompts: statements, text completions, and direct inquiries or question answering (see Table 8 in Appendix D for examples).
Models are presented with declarative sentences that depict different scenarios such as "Marcelo is [possible scenario]", where possible scenario could be indicative of national- ities, for instance, Spanish, Brazilian, or Chinese. The model asked to calculate perplexity scores for each sentence, enabling the ranking of statements based on plausibility, with lower perplexity scores indicating higher likelihood.
Statements:
Text Completion: In this prompt format, models are given sentences structured to require completion, such as "Fill in the blank: Marcelo has citizenship of -.. Answer: [possible scenariol"". The models are then tasked with calculating perplexity scores for different possible scenarios, ranking them from most to least plausible based on perplexity.
Here, models presented with direct questions, such "What citizenship(s) does Marcelo have? Answer: [possible scenario)." " where a possible scenario could be Spanish, Irish, or Chinese, similarly the previous examples. The model evaluates the perplexity of each question-answer pair, arranging answers by plausibility according to calculated perplexities.
Question Answering:
The primary aim across these varied formats is not to generate novel text but to compare perplexity scores for the given scenarios. This methodology seeks to measure a model's proficiency in identifying the most plausible scenario among alternatives, leveraging the context embedded within each prompt type. Such an approach crucial for assessing model's capability to evaluate the likelihood or credibility of different scenarios based on the specific context.
3.1 Most Plausible Scenarios
Evaluating language models on PROBELM is challenging due to its subjective nature and susceptibility to biases, including ethical concerns. To address these complexities, our methodology incorporates novel strategy grounded in probabilistic reasoning, with an emphasis on Bayesian inference principles. Bayesian inference posits that the likelihood of a hypothesis evolves in response to new evidence. Accordingly, our approach assumes that the most immediate future events, as sequentially recorded in successive updates of Wikidata, represent the most plausible scenarios compared to our current knowledge base. This assumption rests on the motion that the near future inherently more predictable and grounded in the current state of the world, thereby providing robust benchmark for evaluating the plausibility of language model predictions.
In practice, the first step of our methodology is to determine the recency of each language model's training data, e.g., OLMo's dataset extends up to March 2023. Subsequently, we identify the most plausible scenarios by analysing Wikidata revisions immediately following this cutoff (e-g., in OLMo's case, between March 2023 and the subsequent update) This approach aims to capture recent facts and developments that were not part of the model's knowledge during training. By focusing on Wikidata changes after the training dataset's
We employ TemporalWiki fJang et al. 2022
generate the edits between two Wikidata dumps.
4

---

Published as a conference paper at COLM 2024
timeframe, we pinpoint instances that, though are now confirmed facts or events, represent highly plausible scenarios to the models due to their temporal alignment with the immediate future. This method ensures the scenarios selected for evaluation are both relevant and challenging, providing a comprehensive test of the models' capacity to utilise the world knowledge available at the time of their last update. Consequently, this creates dynamic and adaptive benchmark that accesses the models' plausibility inference abilities.
3.2 Less Plausible Scenarios
Within the PROBELM framework, the construction of less plausible scenarios leverages a statistical technique grounded in examining entity co-occurrences within Wikidata. By systematically altering the entities and their attributes in given plausible maparro-while maintaining the relationship unchanged-we assess the plausibility of these variations according to their occurrence frequency throughout the dataset. This assessment extends to a wide spectrum of entity associations, beyond isolated instances, ensuring a comprehensive evaluation of plausibility across varied scenarios.
Below is an example how we assess the plausibility of different scenarios using Wikidata co-occurrences. Take the case of Marcelo, who is known to be Brazilian, and evaluate the plausibility of the triple <Marcelo, has citizenship, Spanish?>. We assess the likelihood of the "Brazilian" and "Spanish" attributes being linked under the "has citizenship" relation by calculating their co-occurrence within Wikidata when Marcelo is replaced with different entities. Given the rules facilitating Spanish naturalisation for Brazilians (Figure 1), these attributes are likely to co-occur much more frequently than "Brazilian" with "Chinese" for instance, due to China's policy disallowing dual citizenship. Thus, the plausibility of "Chinese" citizenship in this context is diminished. These co-occurrence scores facilitate the construction of plausibility hierarchy based on statistical evidence from real-world data.
By maintaining the logical structure of relations such as "has citizenship" and manipulating the entities and their associated attributes, we generate scenarios from plausible to less plausible. Our approach not only leverages the rich relational data in Wikidata but also mirrors the complexity of real-world knowledge and its application, providing a robust framework for the plausibility assessment of language model outputs. The detailed algo- rithm for generating these scenarios, including the consideration of entities with complex attributes like dual citizenship, is elaborated in Appendix B
3.3
Quality Control
To ensure the generated scenarios are both accurate and relevant, we implement several quality control measures to address potential biases and inaccuracies in the dataset:
Paraphrase Discrimination: To address the presence of repetitive entities in Wikidata, we
employ FuzzyWuzzy 3 tool that calculates sequence differences using the Levenshtein Distance. This method helps in identifying and filtering out paraphrases, thereby reducing redundancy and ensuring the dataset reflects a diverse range of entities. By distinguishing between similar yet distinct entities, FuzzyWuzzy enhances the precision of our dataset.
We manually review Wikidata edits to exclude those not representing event changes. This involves creating whitelist of relations representing events, such as "place of death", ensuring the dataset emphasises eventful modifications, and improving the relevance of our scenarios for plausibility assessment. The full whitelist of relations is presented in Amppendix D
Manual Filtering of Non-event Edits:
To avoid over-generic scenarios, manually filter out general queries derived from the most plausible triples, prioritising queries that yield more specific and insightful scenarios. In the example of <Marcelo, has citizenship, Spanish>, we consider the query on "Which citizenship does Marcelo have?" to be more specific than "Who else Spanish citizen?"
Manual Filtering of General Queries:
5

---

Published as a conference paper at COLM 2024
MODEL (SIZE)
Pre-training Data Up To Evaluation Timestamps
Number of Scenarios
GPT-2 (124M, 1.5B)
December 2017
Dec Jan 2018
9,328
BLOOM (560M, 7B)
December 2021
Dec 2021 Jaa 2022
8,569
LLAMA (7B)
August 2022
Nov 2023 Dec 2023
5,280
PYTHIA (14M, 160M, 2.8B)
March 2020
Oct 2020 Nov 2020
5,104
OLMo (IB, 7B)
March 2023
Nov 2023 Dec 2023
5,280
Table 1: Evaluated models and their corresponding data timelines the PROBELM bench- mark. This table highlights the diversity of model sizes, their training data recency, and the specific timestamps chosen for evaluating their plausibility inference capabilities.
Template Design for Natural Language Conversion: We manually create templates for each relation to convert triples into consistent and natural sentences. This is essential for preserving scenario coherence and readability, facilitating accurate plausibility inference by language models. Customising templates for each relation ensures grammatical correctness and contextual appropriateness, thus improving the dataset's interpretability. The list of the templates used is also illustrated in Appendix D
3.4 Comparison with Other Datasets
Our dataset demands leveraging extensive world knowledge, making the task more complex and relevant for knowledge discovery compared datasets like COPA which focus on commonsense reasoning without extensive world knowledge. For example, the scenario "I tipped bottle" leading to "The liquid poured out" is self-contained. In contrast, scenarios like the Brazilian footballer Marcelo being plausibly Spanish or Portuguese require extensive background knowledge. Our dataset acknowledges the existence of multiple plausible events for given scenario and includes ten less plausible scenarios to ensure challenging evaluation environment, reducing the likelihood of models performing well by random guessing. Negative sampling is designed around event relations, with all 71 relations and 213 templates detailed in Appendix D ensuring a diverse and representative set of scenarios.
Also, note that our concept of plausibility extends beyond identifying surprising events. It targets scenarios that, while not present in the training data of language models LLMs, have subsequently become facts. For instance, the proposition "the Earth is round" would have been highly plausible based on existing knowledge, even though it was not widely accepted in ancient times. This highlights the importance plausibility distinguishing it from predicting future events. Additionally, while individual timestamps in Wikidata might suggest Markovian approach, our model does not strictly adhere to this. The world knowledge and historical events encoded in LLMs are inherently non-Markovian, providing a richer context for evaluating plausibility. This broader perspective crucial for practical and comprehensive plausibility assessments.
knowledge discovery,
As the dataset timeframe-speccfic regarding the models used for evaluation, we include the statistical details in the following section.
4 PROBELM as Plausibility Benchmark
We evaluate PROBELM on 10 models with different sizes and architecture, and the time- frame of training. To ensure alignment in the pre-training data sourced from Wikipedia, we select open-source models that disclose the timestamp of the utilised Wikipedia dump in their training, resulting in models including GPT-2, BLOOM, LLaMA, Pythia, and OLMo. In the following, we detail our experimental setup and the evaluation metrics.
4.1 Experimental Setup
We first collect data points from Wikidata that align the timeframe a given model (outlined in Table 1), following the steps detailed in section 3, For each most plausible scenario,

---

Published as a conference paper at COLM 2024
always add 10 less plausible alternatives, resulting in total of 11 scenarios for models to rank. To ensure our evaluation is comprehensive yet with the practical constraints of time and computational resources, sampled 126,000 scenarios (6,000 most plausible scenarios and 120,000 less plausible scenarios) from each timestamp, leading to four distinct timeframe-specife evaluation datasets. This sampling covered the five models under study, with OLMo and LLaMA models sharing the same timestamp. Details of the models and the corresponding timestamps are included in Appendix A
Next, we follow the quality control measurements as detailed in inssuseection 3.3 to filter our data points. The final number of samples for each timestamp varies, as shown Table 1
For each set of one most plausible scenario alongside its 10 less plausible alternatives, we probe the models using the three types of prompts described in section 3 employing zero-shot approach. We calculate the perplexity for these 11 scenarios for each prompt type and then rank them based on their p scores.
Note that PROBELM is designed to capture unknown or unrecorded events that can be inferred from existing world knowledge, emphasizing knowledge discovery over sequential event forecasting, Effective models should synthesize knowledge plausibly, as illustrated by the example of Marcelo, where evidence should favor plausible over implausible na- tionalities. Our benchmark remains relevant as long as the training data of LLMs predate the evaluation timestamps. For instance, both OLMO and LLama were evaluated using the same benchmark version, with timestamps more recent than their training data. The benchmark construction process is automated, allowing for easy updates with more recent data if needed, ensuring continued relevance for future models.
4.2 Evaluation Metrics
In assessing the performance of language models on the PROBELM benchmark, we utilise set of evaluation metrics designed to capture various aspects of ranking plausibility. Each metric offers a different perspective on the models' ability to discern and prioritise plausible scenarios, providing a comprehensive overview of their capabilities.
Accuracy focuses on the the top-ranked scenario. This metric determines whether the model can identify the most plausible scenario its first choice. However, accuracy does not account for the placement of scenarios beyond the top-ranked ones, making it measure of immediate accuracy rather than a reflection of the overall ranking quality. We note that accuracy here is different from the factual correctness, as none of the ranked scenarios is factual but ranges from most to least plausible.
Mean Reciprocal Rank (MRR) is employed to calculate the average of the reciprocal ranks of the most plausible scenario across all queries. MRR offers insight into the model's i's ability to rank the most plausible scenario highly but does not evaluate the arrangement of other scenarios in the list, emphasising the importance of the top plausible scenario's position within the model's ranking.
Normalised Discounted Cumulative Gain (NDCG) is utilised to assess the quality of the entire ranked list produced by the model. NDCG takes into account the graded relevance of all ranked scenarios, providing a measure of the list's s overall quality and the effectiveness of the model's is ranking across different levels of plausibility. Unlike accuracy and MRR, NDCG does not isolate its evaluation to any single scenario, such as the top-ranked or the most plausible one, but instead evaluates the cumulative relevance of the whole list, rewarding rankings that place more plausible scenarios higher.
In addition, we measure the PLAUSIBILITY score, where we average accuracy, MRR, and NDCG scores across the three different prompt types.
4From a certain perspective, beam search could also be used to obtain more plausible hypotheses, and its use is orthogonal to evaluation with our benchmark. However, maximum posteriori estimates are not always the best way to decode a model, which is why minimum Bayes risk decoding is popular in fields like machine translation. Our method focuses on evaluating the plausibility of scenarios using perplexity, providing a different perspective than beam search.
7

---

Published as a conference paper at COLM 2024
AVG PLAUSIBILITY
Statement
Text Completion
Question Answering
MODEL
ACC.
MRR
NDCG
ACC.
MRR
NDCG
ACC.
MRR
NDCG
OLMo-1B
45.07
29.96
49.92
50.13
22.11
46.47
46.95
42.15
59.09
58.89
OLMo-7B
52.51
31.20
51.62
51.57
44.62
61.87
61.43
45.45
62.60
62.23
BLOOM-560M
30.63
20.58
43.37
42.56
8.78
34.81
34.54
12.54
39.44
39.02
BLOOM-7B
40.74
27.35
49.90
49.50
17.44
44.21
43.39
12.80
41.11
40.29
GPT-2-124M
39.02
29.08
49.08
47.96
18.09
40.84
39.83
26.97
50.12
49.19
GPT-2-1-5B
45.51
29.19
49.81
48.83
25.52
48.92
48.11
41.95
58.92
58.34
LLAMA-7B
40.74
19.20
43.26
43.60
29.96
52.08
51.81
27.69
49.56
49.51
PYTHIA-14M
35.90
20.68
43.15
43.11
17.67
39.48
39.37
23.90
47.90
47.85
PYTHIA-160M
42.45
34.54
54.39
55.03
16.87
39.59
40.13
30.52
55,16
55.84
PYTHIA-2.SB
58.47
43.78
61.17
61.14
43.57
62.77
62.33
53.41
69.13
68.89
Table 2: Evaluation results of language models on PROBELM with corresponding time- frame specific datasets. We report accuracy, MRR, and NDCG scores for three prompt types. AVG indicates average plausibility across all prompts and metrics (average of nine scores). Best performances are in bold, and second-best results are underlined.
Results and Discussion
5.1 Main Results on PROBELM
We present the plausibility score, and accuracy, MRR, and NDCG scores of different prompt types in Table 2 using the timeframe-specific datasets for each model.
The results reveal that while model performance exceeded random chance, overall scores remain low, underscoring the complexity of the plausibility assessment task. For instance, considering a random baseline accuracy of 9.09% when ranking 11 scenarios, most models surpass this baseline across various sizes and prompts. Notably, both BLOOM models marginally outperform the random baseline by approximately points, achieving 12.54% and 12.80% on the question-answering prompt. Specifically, the BLOOM-560M model underperform relative to the random baseline in the text completion prompt with 8.78% ac- curacy. This performance pattern suggests that language models face significant challenges in differentiating between scenarios varying plausibility. It also highlights distinct sepa- ration between models' capabilities in factual accuracy and their ability to infer plausibility. These observations set the stage for our key findings:
Larger models generally outperform smaller ones in plausibility tasks, but performance varies across model families. For example, the ACC. scores across models of varying sizes show 45.07 vs 52.51 for OLMo, 30.63 40.73 for BLOOM, 39.02 45.51 for GPT, and notable increase from 35.90 through 42.45 to 58.47 for Pythia models. However, this effect varies across different model families. Remarkably, the Pythia 2.8B model outperformed all three 7B models by at least 6 percentage points, while the Pythia 160M model surpassed both LLaMA 7B and BLOOM 7B models signifia-ig larger in scale-by percent.
These exceptions underscore that the relationship between model size and plausibility assessment is complex and not linear. Performance variability across different model families underscores the profound impact of architectural differences and training methodologies on model's plausibility inference capabilities, suggesting that these factors are crucial as, not more so than, model size. This complexity in plausibility assessment task performance indicates that mastery in this domain necessitates detailed understanding of the interplay among model architecture, size, and training strategies.
While variations in performance across prompts and metrics are observed, they typically correlate strongly. The most significant gap occur between OLMo-1B's performance the text completion prompt (22.11 ACC.) and the question answering prompt (42.15 ACC.), highlighting some variation in model proficiency across different types of tasks. Despite these variations, the assessment appears to hinge on a consistent underlying mechanism across prompt types. This is further evidenced by the Pearson correlation coefficients be-
8

---

Published as a conference paper at COLM 2024
PROBELM
COРA
ARC
HELLASWAG
TRUTHFULQA
WINOGRANDE
MODEL
FLAUSIBILITY
ACC.
ACC.-NORM
ACC,-NORM
ACC.
ACC.
PYTHIA-2.8B
58.47 (1)
79.0 (4)
33.02 (4)
59.30
35.88
(5)
OLMo-7B
52.51 (2)
85.0 (1)
40.36 (2)
75.65
(2)
35.85 (8)
66.38 (2)
GPT-2-1.55
45.51 (3)
76.0
28.50 (6)
50.89
38.53
58.33
OLMo-1B
45.07 (4)
82.0 (3)
31.06 (5)
62.92 (3)
32,94 (10)
59.98 (4)
PYTHLA-160M
42.45 (5)
64.0 (7)
23.21 (8)
28.48
44.43 (2)
49.88 (10)
LLAMA-7B
40.74 (6)
85.0 (1)
44.62 (1)
76.21 (2)
34.08 (9)
70.01 (1)
BLOOM-7B
40.74 (7)
73.0 (6)
33.45 (3)
62.28 (4)
38.89 (5)
64.64 (3)
GPT-2-124M
39.02 (8)
62.0
22.70
31.14
40.69
51.62
PYTHIA-14M
35.90 (9)
53.0 (10)
21.25
(10)
26.12 (10)
50.37
(1)
50.43 (9)
BLOOM-560M
30.63 (10)
61.0 (9)
23.81
(7)
36.92
42.43 (3)
51.38 (8)
Table 3: Language Models Performance PROBELM versus other reasoning datasets. The
ranking of the models is shown in brackets. ARC shows results on the challenge set. Same as before, plausibility shows the average score of all prompt types and metrics (ACC., MRR, and NDCG). Models are ordered based on the plausibility rank on PROBELM.
tween the performances various prompts, which were notably high: 0.8281 for statement versus text completion, 0.8672 for statement versus question answering, and 0,8976 for text completion versus question answering. Such strong correlations underscore that, despite the prompt-specific performance disparities, the models are evaluated based on a core aspect of their capability of ranking plausibility. Similarly, the Pearson correlation score between MRR and NDCG is 0.9982, indicating consistent plausibility measurements across metrics.
5.2 Comparison with Other Benchmarks
To comprehensively understand the different coverage between PROBELM and other LLM benchmarks, follow www.aaaaaiiohaarnssssa to evaluate zero-shot performance COPA, TruthfulQA (mc2, ie., multiple options can be correct), ARC (challenge set) (Clark et 2018), HellaSwag (Zellers al. 2019), and Winogrande (Sakaguchi et al. 2021). Detault metrics (accuracy or normalised accuracy) are applied to each dataset and the results are included in (Talle
PROBELM shows distinct model performance compared to other reasoning benchmarks. Models that perform well on tasks emphasising factual accuracy do not always maintain their performance levels on plausibility assessments. In some cases, the performance rankings are inversely related. A notable example of this phenomenon observed in the performance disparity between PROBELM and TruthfulQA. Specifically, Pythia-14M, which exhibits the best performance in TruthfulQA, positions near the bottom (9th place) of the PROBELM ranking. This variance underscores the distinct nature of plausibility as an evaluation criterion, demanding separate capabilities beyond those required for tasks focused on factual accuracy. The development of PROBELM thus contributes to the broader framework for evaluating language models by offering different angle to assess their capability to understand and generate plausible content.
5.3 Temporal Effect of PROBELM
We further evaluate models against datasets from 2017, 2020, 2021, and 2023 timestamps to investigate how the recency of training data affects the plausibility performance. The outcomes of this analysis are presented in Figure and detailed in Appeeddxx C Remarkably, the relative rankings of models prove relatively consistent across these varied timeframes, affirming the robustness of our evaluation approach. We observe that, from the 2020 to 2023 timestamps, models trained closer to or after the evaluation dates typically exhibit enhanced performance, underscoring the benefits of leveraging recent data.

---

Published as a conference paper at COLM 2024
OLMo-7B
BLOOM-7B
GPT-2-1.5B
LLAMA-7B
PYTHIA-2.88
Plausibility
50
30
20
2017.12
2020.10
202112
2023,11
Figure 2: PLAUSIBILITY scores of models on the timeframe specific datasets. We illustrate one model per family. Vertical dotted lines represent the training timeframe of each model.
An anomaly is detected in the 2017 dataset, where all models show a decrease in per- formance, suggesting it is an outlier. An analysis reveals that the "occupation" relation disproportionately influences the 2017 dataset, constituting 14.3% of all scenarios compared to less than 0.5% in the other datasets. This discrepancy indicates a significant impact of the "occupation" relation on model performance. Excluding the "occupation" relation from the evaluation on the 2017 dataset results in either similar or higher scores across models, shown in Appendix CJ For instance, evaluating the Pythia-160M model without scenarios involving the "occupation" relation leads to a notable increase in plausibility scores from 39.69 to 42.73. This outcome indicates the models' challenges with certain relations.
6 Conclusion
In this work, we introduce PROBELM, a benchmark designed to evaluate language models' ability to discern plausible scenarios, filling critical gap left by existing benchmarks focused on factual accuracy or plausibility without world knowledge integration. Our comprehensive evaluation of 10 models of varying sizes and architectures against this benchmark reveals key insights including factual accuracy does not directly correlate with plausibility discernment; larger models do not uniformly outperform smaller ones across different families; the recency of training data, while impactful, is not the sole determinant of plausibility performance; and models tend to struggle with plausibility assessment as the gap between their training data and the evaluation set widens. These findings highlight the complexity of plausibility inference and the need for advanced modelling techniques that can better address the challenges of real-world knowledge application. Our dataset and code are publicly available, to facilitate the community for further exploration in enhancing language models' plausibility inference capabilities.
References
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM Technical Report. arxiv preprint arX10:2305.10403, 2023. URL https:///axxv www.aaiiii.oooo//bb..asss
10403
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In International Conference Machine Learning, pp. 2397-2430. URL https://procceedings wwreeeddiinnsml.......m. pttpss//v22//bbdderaaa2aaa/////0
PMLR, 2023.
pdf
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Re-
10

---

Published as a conference paper at COLM 2024
search, 24(240):1-113, 2023. /tt.tt/tt///////.rr///o////////a pdff
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think You Have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. Xiv preprint ar Kio:1100.00457, 2018. URL httpp:://axii..orrr abs/1803.05433
Vishrawas Gopalaksia,,, Kishlay Jha, Wei Jin, and Aidong Zhang. A Survey on Literature Based Discovery Approaches in Biomedical Domain. Journal of Biomedical Informatics, 93:103141, 2019. ISSN 1532-0464. doi:
http://di..rrg11111111//////b
URL
19300590
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLMo: Ac- celerating the Science of Language Models. arXiv preprint ar X10::2402200888,, 2024. URL
https://arxiv. wttp:/xwvvog/bbbb///eeeo 00880000
Nate Gruver, Marc Anton Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large Language Models Are Zero-Shot Time Series Forecasters. In Thirty-seventh Conference Neural Infor- mation Processing Systems, 2023. httphhttt//::rrreeeee wwwprerrreieeeeee/oruummmmmmm
ZK1
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with Language Model is Planning with World Model. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, Pp. 81 8154-8173, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1 wttp///222....mmmmiii..... 1mm.ppmminn.00
URL https
//aclanthology.orgg//
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6237-6250, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Liciiionnnniiiggiing Linguistics. doi: 1t/55/3//120222emm/// https://aclanthooooyy.rr//////// www.::222.mmmnpmmmammm.
URL
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):1-38 2023. wtt/tttss://///// httph:h/tp//tt acm. org/doi/ 10.1145/3571339
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. FANTOM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 14397-14413, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/vb 1023.emnlp-main.89 UtLhhttp:://acll
orgg 2023. emm.pp.mllppmmmm emnlp-main 890
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jun Yun Nie, and Ri-Rong Wen. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6449-6464, Singapore, December 2023. Association 2tt0://111222...mmmmmiiia
for Computational Linguistics. doi: bogllanthology.orgg/2e00 www.222....meppmmmmmmmmmm sminp--miin 397
URL https
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/vv /v11/ /11/2022.acllooggg 229. URL https: http://aaaaaantoll...oooo hww.aaanthoooyyo.rg//2222 002/
acl- 229
11

---

Published as a conference paper at COLM 2024
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076- 12076-1210,, Singapore, December 2023. Association for Computational Linguistics. doi: 213666//11202..emlmmmm/aa //aclanthology. org 2023. emnlp-main.74 741
URL ht
Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictrry Hallu- cinations Large Language Models: Evaluation, Detection and Mitigation. In The Twelfth International Conference on Learning Representations, 2024. URL httpp::/ net/
forum?id=Em@SO1 1X2f
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAl blog, 1(8):9, 2019.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of Plausible alternatives: An Evaluation of Commonsense Causal Reasoning. In 2011 AAAI Spring Symposium Series, 2011. URL
1888881118188811111 1-PB.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An Adversarial Winograd Schema Challenge at Scale. Communications of the ACM, 64(9):
99-106, 2021. URL https::////d oww. /t0.//001111100 1144/3474388
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. BLOOM: 176B- Parameter Open-Access Multilingual Language Model. working paper or preprint, November 2023. URLhhttps://inr
science/acc -033e/0012200
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13977 2023. URL
13971
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bow- man. GLUB: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018, Association for Computational Linguistics. doi: 2.11653/v1/W133
URL https: htttps://aalaattoolggy www.:orggg/////.. 5446
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. Advances in neural information processing sys- tems, 32, 2019. URL http:://wppeeeeeiisssrereeeeeres 0496ffA444ee/fa66666666669222222
pdf
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. SciMON: Scientific Inspiration Machines Optimized for Novelty. arv preprint arXivo2305.14259, 2023, URL https
//arxiv.org/abs/2355. 14259.
Chenxi Whitehouse, Clara Vania, Alham Fikri Aji, Christos Christodoulopoulos, and Andrea Pierleoni. WeblE: Faithful and Robust Information Extraction on the Web. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7734-7755, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: acl-long.422 URL /tttps://aclaaneei http///aaaaaaaaaa...eeco wwwphhhoolgg.ooggg/ooooooo wwt.oogg///2/ 20tg./2233.. ..088.a.c... acl-aclllll 1ongllloon... 428
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is Inevitable: An Innate Limitation of Large Language Models. Xiv preprint ar xrXiv22401.11817 2024. URL https! //axxiv.org/abs/2401.11
12

---

Published as a conference paper at COLM 2024
Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking LLMs via Uncertainty Quantification. 12794
arXiv preprint xxio:2201.11779, 2024.
Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. KoLA: Carefully Benchmarking World Knowledge of Large Language Models. arXiv preprint ar arXio:2306.99296 2023. URL
https://arxiv. www.vvorrg/aaae 2006.////...... 69296
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can Machine Really Finish Your Sentence? In Anna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computa- fional Linguistics, pp. 4791-4800, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v1 /v/1111141 URL htwp::ttss//aatt https::/aallanthhllggy// htwpaaatttoooooooroo/P.... org/P19-1472
A Models Used in Our Experiments
We briefly introduce below the models and the timestamp of the corresponding Wikidata used for our experiments.
GPT-2 124M and 1.5B [Radford et al. 2019) As early iterations in the GPT series, these models offer a perspective on plausibility inference with training data frozen in December 2017. This provides basis for evaluating how well models can reason about plausibility without the benefit of recent information. The closest corresponding timestamp we found is from December 2017 to January 2018.
BLOOM 560M and 7B (Scao et al. 2023) Representing the BLOOM series' commitment to openness and scalability, these models, with data up to December 2021, allow us to explore how differences in model size impact their capacity to discern plausible scenarios without having been exposed to subsequent updates. The closest corresponding timestamp we found is from December 2021 to January 2022.
(Touvron et al., 2023 LLaMA is recognised for its robust performance across a broad range of NLP tasks. Its training data, current of August 2022, positions it uniquely for evaluation with the PROBELM benchmark, specifically designed to assess the model's ability to infer plausibility based on its training up to that point. The closest corresponding timestamp we found is from December 2023 to January 2023.
LLaMA 7B
Pythia 14M, 160M, and 2.8B [Biderman et al. 2023 The Pythia models extend our exploration into the realms of modern architecture and model scalability, spanning the spectrum from an ultra-compact 14M to robust 2.8B parameter setup. This inclusion allows us to assess plausibility inference capabilities across a broad range of model sizes, particularly emphasising the performance of smaller models like the Pythia 14M. The evaluation of Pythia models leverages their training data up to March 2020. The closest corresponding timestamp we found from October 2020 to November 2020.
OLMo 1B and 7B (Grenneell et al. 2024) The newest models in our evaluation, OLMo's training data extends to March 2023. Their assessment against the PROBELM benchmark tests their ability to gauge plausibility based on comprehensive and current dataset, without incorporating the very latest updates. The closest corresponding timestamp we found is from December 2023 to January 2023.
B Algorithm for Generating Less Plausible Scenarios
To systematically generate less plausible scenarios for the PROBELM benchmark, we use the algorithm shown in Algorithm[]] which leverages the statistical distributions of entity co-occurrences within Wikidata. This method ensures that the generated scenarios, while being less plausible, still maintain logical coherence and relevance to the given context.
13

---

Published as a conference paper at COLM 2024
Algorithm 1 Generating Less Plausible Scenarios
Require: A triple from Wikidata (s,
Ensure: Ranked list of less plausible triples L
remains constant for all operations
P {p (s, (s,subclass_of _of, p) E Wikidata)
S f empty list to hold siblings of
for all entities e sharing parent do i f then
Append e to end if
end for
Oall f- empty list hold objects for all siblings for all sib E $ do
Osib f all objects for triples (sib,r,*) r,*)
Append az to Oal
end for Initialize empty map Freq for all obj Out do
if obj in Freg then Freq [obj] + Freglobj] +
else
Freqlobi] < end if
end for
+ sort(Freq, key Freq.ges reverse) Repeat process with fixed to find subjects similar to and generate another ranked list L
C Additional Results on Time-Specific Datasets
We show the plausibility performance for models evaluated on PROBELM with different timestamps in Table 4 (2017), Table (2020), Table (2021), and Table (2023).
AVG
Statement
Text Completion
Question Answering
MODEL
PLAUSIBILITY
ACC.
MRR
NDCG
ACC.
MRR
NDCG
ACC.
MRR
NDCG
OLMo-1B
46.64 (49.91)
30.52
48.90
47.57
26.75
49.72
48.78
44.28
62.05
61.21
OLMo-7B
54.68 (57.72)
33.41
52.70
51.76
46.73
64.68
64.37
47.84
65.86
64.76
BLOOM-560M
37.95 (37.94)
23.20
46.05
45.35
20.76
43.82
43.40
24.31
47.67
47.01
BLOOM-7B
42.48 (43.02)
29.86
50.01
49.39
28.86
52.22
51.78
23.31
48.72
48.18
GPT-2-124M
39.02 (42.83)
29.08
49.08
47.96
18.09
40.84
39.83
26.97
50.12
49.19
GPT-2-1.5B
45.51 (50.16)
29.19
49.81
48.83
25.52
48.92
48.11
41.95
58.92
58.34
LLAMA-7B
49.96 (48.38)
30.52
51.70
51.04
40.95
61.01
60.37
37.63
58.57
49.96
PYTHIA-14M
34.03 (37.77)
20.98
41.50
40.30
16.09
37.52
36.31
22.97
45.94
44.65
PYTHIA-160M
39.69 (42.73)
30.30
50.27
49.51
15.54
37.08
36.15
30.97
53.75
53.62
PYTHIA-2.SB
50.85 (54.73)
34.30
53.08
51.98
36.51
58.44
58.37
42.51
61.69
60.80
Table 4: Evaluation Results of Language Models on PROBELM Using the Same Evaluation Dataset of December 2017 to January 2018. Plausibility Scores in Parenthesis Are Obtained Without Evaluating the "Occupation" Relation.
D More details on PROBELM
An example of the prompts used to evaluate the models is shown in Table 8
14

---

Published as a conference paper at COLM 2024
AVG PLAUSIBILITY
Statement
Text Completion
Question Answering
MODEL
ACC.
MRR
NDCG
ACC.
MRR
NDCG
ACC.
MRR
NDCG
OLMo-1B
53.59
35.74
54.94
55.41
32.93
56.40
57.20
52.61
68.19
69.11
OLMo-7B
58.09
39.36
57.53
57.25
49.60
67.22
66.31
50.40
67.43
67.07
BLOOM-560M
40.93
31.12
53.08
53.17
17.87
44.27
44.40
23.89
50.21
50.23
BLOOM-7B
47.03
39.16
58.70
59.34
28.71
54.87
55.23
23.69
51.60
51.81
GPT-2-124M
42.96
32.73
52.78
52.99
18.47
43.58
43.72
30.72
55.54
56.11
GPT-2-1.5B
52.36
37.35
57.67
57.74
26.70
52.22
52.36
52.81
68.37
68.61
LLAMA-7B
47.14
25,30
48.46
48.69
35.74
57.89
58.25
34.33
57.61
57.95
PYTHIA-14M
35.89
20.68
43.15
43.11
17.67
39.48
39.37
23.90
47.90
47.85
PYTHIA-160M
42.45
34.54
54.39
55.03
16.87
39.59
40.13
30.52
55.16
55.84
PYTHIA-2.8B
58.55
43.78
61.17
61.14
43.57
62.77
62.33
53.41
69.13
68.89
Table 5: Evaluation Results of Language Models on PROBELM Using the Same Evaluation Dataset of October to November 2020.
AVG
Statement
Text Completion
Question Answering
MODEL
PLAUSIBILITY
ACC.
MRR
NDCG
ACC.
MRR
NDCG
ACC.
MRR
NDCG
OLMo-1B
50.22
38.52
56.12
55.17
28.73
51.99
51.24
45.92
63.21
62.12
OLMo-7B
54.66
38,02
56.87
55.83
44.29
62.85
61.73
45.92
63.81
62.63
BLOOM-560M
30.63
20.58
43.37
42.56
8.78
34.81
34.54
12.54
39.44
39.02
BLOOM-7B
36.22
27.35
49.90
49.50
17.44
44.21
43.39
12.80
41.11
GPT-2-124M
38.92
32.25
51.89
51.22
14.81
39.64
39.38
24.22
48.82
48.25
GPT-2-1-5B
46.68
35.51
56.02
54.81
19.70
45.68
45.06
43.66
60.86
59.86
LLAMA-7B
48.14
28,48
51.28
50.22
35.88
57.62
56.14
38.14
58.60
56.86
PYTHIA-14M
35.16
20.08
42.90
43.15
15.55
38.66
39.38
22.08
47.06
47.69
PYTHIA-160M
36.66
31.37
53.11
52.37
11.92
35.31
34.82
PYTHIA-2.SB
52.93
41.53
58.64
57.78
37.64
57.46
56.60
44.54
61.86
60.93
Table 6: Evaluation Results of Language Models on PROBELM Using the Same Evaluation Dataset of December 2021 to January 2022.
We show the relations included in PROBELM and different types of prompts in Table Table 10 Table 11
15

---

Published as a conference paper at COLM 2024
AVG PLAUSIBILITY
Statement
Text Completion
Question Answering
MODEL
ACC.
MRR
ACC.
MRR
NDCG
OLMo-1B
45.07
29.96
49.92
50.13
22.11
46.47
46.95
42.15
59.09
58.89
OLMo-7B
52.51
31.20
51.62
51.57
44.62
61.87
61.43
45.45
62.60
62.23
BLOOM-560M
32.07
22.93
46.51
46.49
36.55
37.04
10.33
40.03
BLOOM-7B
35.37
30.79
51.56
51.34
13.42
42.81
42.44
8.47
38.91
38.58
GPT-2-124M
35.06
23.97
46.22
46.72
10.95
36.35
36.79
20.45
46.87
47.18
GPT-2-1-5B
44.84
29.75
51.70
50.92
20.25
46.21
46.05
41.12
59.18
58.59
LLAMA-7B
40.77
19.42
43.26
43.60
29.96
52.08
51.81
27.69
49.56
49.51
PYTHIA-14M
28.89
12.40
37.21
37.74
32.76
33.77
14.67
41,02
41.14
PYTHIA-160M
36.34
26.86
49.87
50.18
10.95
34.04
34.99
22.52
48.60
49.05
PYTHIA-2.8B 52.46
36.16
55.55
55.34
36.98
58.42
58.26
46.07
63.45
62.86
Table 7: Evaluation Results of Language Models on PROBELM Using the Same Evaluation Dataset November to December 2023.
Prompt Type
Examples
STATEMENT
"Marcelo is Spanish.", "Marcelo is Irish.", ", "Marcelo is Chinese." "
TEXT COMPLETION
"Fill the blank: Marcelo has citizenship
Answer: Spanish." "
QUESTION ANSWERING
"Question: What citizenship(s) does Marcelo have? Answer: Spanish." "
Table 8: Illustrative Examples of Prompt Types Used in PROBELM.
16

---

Published as a conference paper at COLM 2024
PID RELATION
TEMPLATE FOR STATEMENT
P5096
member of the crew of
<subject>is member the crew of cobject>.
P122
basic form of government stepparent
The basic form of government of <subject>is <object>. <subject>is the stepparent of <objects.
P3448
P1479
has contributing factor
cobject>>ss contributing factor <subject>. <subject>iss the discoverer or inventor of <object>.
P61
discoverer or inventor
P3320
board member
<subjecl>sery as board member of cobject>.
P7779
member of military unit editur
<subject>is member the military unit cobject>.
P98
<subject>-is the editor of <object>. < nominated for <object>. <subject>iss the presenter of <object>. ssabject-rrppace <ubject>.
P1411
nominated for
P371
presenter
P1365
replaces
P488
chairperson
<ubject>>ervvs as the chairperson of <object>.
P27
country of citizenship
<subject> country of citizenship is <objects.
P20
place of death
subjects>ss s place of death is <object>.
P1344
participant
> a participant in <object>. <subject> replaced by <object>. <subject>speaks the following languages: cobjects.
P1366
by
P1412
languages spoken written or signed
P276
location
The location of <subject>is cobject>.
P407
language of work or name
The language of <subject>'s 's work or name is <object>. <subject> the position <object>.
P39
position held
P1532
country for sport
<subject>-eprrsentt <object>in sports competitions. <subject>iss the unmarried partner of <object>. <subject>is member the sports team <object>.
P451
unmarried partner
P54
member sports team
PS00
notable work
<subject>'s notable work includes <object>. <subject>'s residence is in <object>. <subject>iss located the administrative territorial entity <object>>
P551
residence
P131
located the administrative ter-
ritorial entity
P106
occupation
<subject>'s 's occupation is <object>.
P69
educaled
<subject> educated at cobject>.
P509
cause of death
<subject> 's cause of death was <object>. <subject>is member of the political party <objects.
P102
member of political party
P19
place of birth
<subject>> place of birth <object>. <subject>'s home venue is <object>. <object>applies to the jurisdiction of <subject>,
P115
home venue
P1001
applies to jurisdiction
P840
narrative location
<subject>is the narrative location of cobject>. <subject>is employed by <objects.
P108
employer
P57
director
<subject>>ss the director of <object>.
P2416
sports discipline competed
the sports discipline <object>.
P400
platform
<subject>is is available on the platform <object>.
P1433
published
<subject>was published in <object>>.
P1056
product material produced
<subject>-roducess <object>os product material.
P9071
character type
<subject>is characlerized as <object>type. <subject>is member of the parliamentary group <object>. <subject>> 's work location is <object>.
P4100
parliamentary group
1937
work location
P1066
student of
<subject>iis student of cobject>.
P1535
used by
cobject> used by <subject>.
P6
head of govermment
<subject>>ss the head of govemment of cobject>.
P2283
use
<subject>>is is used for <object>> <subject>> 's academic major is <object>. <subject>iss affiliated with <objects. csubject=achiewe a victory in cobject>. <subject> involved the conflict <object>.
P812
academic major
P1416
affiliation
P2522
victory
P607
conflict
P749
parent organization
is part of the parent organization <object>.
P2283
uses
<subject>uses <object>>.
P802
student
<subject>-is student cobjects. <subject>'s 's place of burial is <object>. <subject= was married at <object>> <subject>>s the head coach of <object>>
P119
place of burial
P2842
place of marriage
P286
head coach
P2541
operating area
<subject> 's operating area is <object>. <subject>is present in the work <ohject>. <subject>is interested in <object>. <object:>s conferred by <subject>. <subject>is the musical conductor of <object>. <subject> elected in <object>. <subject> served during the parliamentary term cobjects.
P1441
present in work
P2650
interested
P1027
conferred by
P3300
musical conductor
P2715
elected in
P2937
parliamentary term
P1399
convicted of
<subjecl>was convicted of <object>. <ubjject>iis used for the work <object>. subbjett 's manner of death was <object>. <ubjeetl>was detained <objects. <subject> was the successful candidate in <object>.
P1686
for work
P1196
manner of death
P2632
place detention
1991
successful candidate
P2443
slage reached
csubject--eecchet the slage <object>. <subject>-as written for <object>.
P6872
has written for
Table 9: Relations and templates for statement prompts used in PROBELM.
17

---

Published as a conference paper at COLM 2024
PiD RELATION
TEMPLATE FOR TEXT COMPLETION
P50%
member of the crew of
Fill the blank: <subject>is member of the
Answer: <object>
P122
basic form of government
Fill in the blank: The basic form of dovernment of <subject>is _.. Answer: <object>
P3448
stepparent
Fill the blank: <subject>is the
of _._ Answer: <object>
P1479
has contributing factor
Fill in the blank: <ohject>is a contributing factor to Answer: <subject>>
P61
discoverer inventur
Fill in the blank: _._ the discoverer inventor of <ubject>. Answer: <subject>>
P3520
board member
Fill in the blanke <subject> serves as a board member
Answer: <object>
17779
member of military unit editor
Fill in the blank: <subject>is member of the military unit Answer: <object>>
P98
in the blank:
is the editor of cobjects. Answer: <subject>
P1411
nominated for
Fill in the blank
was nominated for <object>> Answer: <subject>>
P371
presenter
in the blank -.. is the presenter of <toject>. Answer: <subject>>
P1365
replaces
Fill in the blank <subject>
Answer: <object>
P488
chairperson
in the blank: -._ serves as chairperson of <object>. Answer: <subject>
P27
country of citizenship
Fill in the blank <subject>'s country of citizenship is Answer: <object>>
P20
place of death
the blank: <subject>'s 's place of death -- Answer: <object>>
P1344
participant in
Fill in the blank: was participant in <objects. Answer: <subject>>
P1366
replaced by
the blank was replaced by - Answer: <ubject> Fill in the blank: <subject> the following languages:
P1412
languages spoken written
Answer: <object>
signed
P276
location
Fill in the blank: The location of <subject>is
Answer: <object>>
P407
language of work or name
Fill
blank: The language of <subjects's work or name -.. Answer: <object>
P39
position held
Fill in the blank: <subject~holds the position of Answer: <object>
P1532
country for sport
Fill the blank: -.. represents <object>in sports competitions. Answer: <subject>>
P451
unmarried partner
Fill in the blank: <subject>is the unmarried partner of... Answer: <object>>
P54
member sports team
Fill the blank: <subject>iss member of the sports team -.. Answer: <object>>
PS00
notable work
Fill in the blank: <subject>'s notable work includes - Answer: <object>
P551
residence
Fill the blank: <subject>'s residence is
Answer: <object> Fill in the blank: <snbject>is located in the administrative territorial entity
P131
located in the administra- tive territoraa entity
Answer: <ubject>>
P106
occupation educated at cause death
Fill in the blanke <subject>'s 's occupation is
Ansser: <object>
P69
Fill in the blanke <subject> educated at Answer: <ubject>>
P509
Fill in the blanke <subject>'s 's cause of death was
Answer: cobjects
P102
member of political party
Fill in the blank: <subject>>1s member of the political party Answer: <object>>
P19
place of birth
Fill in the blanke <subject>'s place of birth is .... Answer: <object>>
P115
home venue
Fill in the blank <subject>'s home venue is - Answer: <object>
P1001
applies to jurisdiction
the blank: <object>applles to the jurisdiction of .... Answer: <subject>
PS40
narrative location
Fill in the blank <subject>is set in the narrative Jocation
Answer <object>
P108
employer
the blank: <subject>is employed by -.- Answer: <object>
P57
director
Fill in the blank: .......... the director of <objects. Answer: <subject>>
P2416
sports discipline com-
the blank: esubjeet>compett in the sports discipline of ..... Answer: <object>>
peted in
P400
platform
Fill the blank isubject>is available on the platform ... Auswer: cobject>
P1433
published in
Fill in the blank: <subject>was published in Answer: <object>
P1056
product material pro-
Fill the blank: <subyettspoduuee ... as product or material. Answer: <ubject>
duced
P9071
character type
Fill the blank: <subject>is characterized - type. Answer <object>>
P4100
parliamentary group
Fill in the blank: <subject>is a member of the parllamentary group Answer:
<object>>
P937
work location
Fill in the blank: <subject>'s 's work location is is_.. Answer: <object>
P1066
of
Fill the blank: <subject>is student of Answer: cobject>
P1535
used by
Fill in the blank: <ohject>is used by _ Answer: <subject>>>>
P6
head of government
Fill in the blank:
the head of government of <ubject>. Answer <subjects
P2283
use
in the blank: <subject>>s used for Answer: <ohject>
PS12
academic major
Fill in the blanks <subject>'s 's academic major
Answer: cobject>
P1416
affiliation
inll the blanke <subject>is affillated with Answer: cobjects
P2522
victory
Fill in the blank <subject>achieved a victory in -.. Answer: <object>>
P607
conflict
Fill the blanke <subject>was involved in the conflict Answer: <object>
P749
parent organization
Fill in the blank: <subject>>is a part of the parent organization
Answer <object>
P2283
uses
the blank: <subject>usss -.. Answer: <object>
PS02
student
Fill in the blank <subject>>ss a student at at... Answer: <object>
P119
place of burial
the blank: <subject>'' 's place of burial is -__ Answer: cobject>
P2842
place of marriage
Fill in the blank: <subject married at
Answer: <objects>
P286
head coach
the head coach of <object>. Answer: <subject>> Fill in the blank: <subject>'s 's operating anaa is
the blank:
P2541
uperating area
Answer: <object>
P1441
present work
the blanke <subject>is present the work Answer: <object>
P2650
interested in
Fill in the blank: <subject>is interested in Answer: <object>
P1027
conferred by
Fill the blank cobject>is conferred by Answer: <subjects
P3300
musical conductor
Fill in the blank ... is the musical conductor of cobject> Answer: <subject>>
P2715
elected in
Fill the blank <subject> was elected cobjects. Answer: <ubject>
P2937
parliamentary term
Fill in the blank: <subject>served during the parliamentary term Answer:
<object>>
P1399
convicted of
Fill in the blank: <subject> was convicted of
Answer: <object>
P1686
for work
Fill the blank: <subject>>s used for the work Answer: <object>
P1196
manner of drath
Fill in the blank: <subject>>s 's manner of death was... Answer: <abjects
P2632
place of detention
Fill in the blanks <subject> detained at _.. Answer. <objecct>
1991
succesfull candidate
Fill in the blank: <subject>was was the successful condidate in
Answer: <abject>
P2443
stage reached
Fill in the blanks reachhel the stage _.. Answer: cobjects
P6872
has writlen for
Fill in the blank <subject>has written for Answer: <object>
Table 10: Relations and templates for text completion prompts used in PROBELM.
18

---

Published as a conference paper at COLM 2024
PiD RELATION
TEMPLATE FOR QUESTION ANSWERING
P50%
member of the crew of
Question: <subject>-s member of the crew of what? Answer: <object>.
P122
basic form of government
Question: What is the basic form of government of <subject>? Answer: <object>.
P3448
stepparent
Question: isubject>is the stepparent of who? Answer: <ubject>.
P1479
has contributing factor
Question: <object>is a contributing factor to What? Answer: <subject>.
P61
discoverer inventur
Question: Who is the discoverer inventur of <ubjects? Answer: <subject>>
P3520
board member
Question: <subject> servees as board member of what? Answer <objects.
17779
member of military unit
Question: isubject>is member of the military unit of what? Answer <objects.
P98
editor
Question: Who is the editor of <abject>? Answer: <subject>.
P1411
nominated for
Question: Who was nominated for <object>? Answer: <subject>.
P371
presenter
Question: Who the presenter of <ubject>? Answer: csubjects.
P1365
replaces
Question: Whom did <subject>replace? Answer: <object>.
P488
chairperson
Question: Who serves as the chairperson of <object>? Answer: <subject>.
P27
country of citizenship
Question: What is <subject>'s 's country of citizenship? Answer, <object>.
P20
place death
Question: Where is <subject> S place of death? Answer: <object>.
P1344
participant in
Question: Who was participant in <object>? Answer: <subject>.
P1366
replaced by
Question: What was <subject by? Answer: cobject>.
P1412
languages spoken written signed
Question: What language does <subjertspeakks
Answer: <object>.
P276
location
Question: Where is the location of <subject>? Answer: cobject>.
P407
language of wurk or name
Question: What is the language of <subjects's work or name? Answer cubjects. Dosttoondess Question: What position does ssubjecthhold" Answer: <object>.
P39
position held
P1532
country for sport
Question: Who represents <object>in sports competition?? Answer: <subject>.
P451
unmarried partner
Question: Who is the unmarried partner of <subject>? Answer: <object>.
P54
member sports team
Question: <subject>is member of which sports team? Answer: <object>.
PS00
notable work residence
Question: What is a notable work of <subject>? Answer: <object>.
P551
Question: Where is the residence of <subject>? Answer: <object>. Question: In which administrative territorial entity is <subject> Answer: <ubject>.
P131
located in the administra- tive territoria entity
P106
occupation educated at cause death
Question: What is the occupation of <subjects? Answer: <object>.
P69
Question: Where <subjstteeducated? Answer, <object>.
P509
Question: What was the cause of death of <subject>? Answer: <object>.
P102
member of political party
Question: Which political party is <subject>a member of? Answer: <object>.
P19
place of birth
Question: Where is the place of birth of <subject>? Answer: <object>.
P115
home venue
Question: What is the home venue of <subject>? Answer: <object>.
P1001
applies to jurisdiction
Question: which jurisdiction does cobject>apply? Answer: <subject>.
PS40
narrative location
Question: What is the narrative location of <subject>? Answer: <object>.
P108
employer
Question: Who the employer <subject>? Answer: cobject>.
P57
director
Question: Who is the director of cobject>? Answer: <subject>. Question: which sports discipline does <subject>-oomptt? Answer: <object>.
P2416
sports discipline com-
peted in
P400
platform
Question: On which platform csubject Answer: cobjects. Question: Where was suujectt Answer: <object>. Question: What product or
P1433
published in
P1056
product material pro-
does <subject>pooduue? Answer: <ubject>>
duced
P9071
character type
Question: What type of character mubjeets-haraacterize as? Answer: <object>> Question: Which parliamentary group is <subject= member of? Answer: <object>.
P4100
parliamentary group
P937
work location
Question: What is the work location <subject>? Answer <object>.
P1066
otudent
Question: Who is <subject>a student of? Answer: <object>.
P1535
used by
Question: Who uses <object>? Answer: <subject>.
P6
head of government
Question: Who is the head of govermeent of <ohject>? Answer: <subjects.
P2283
Question: What is useb for? Answer: <object>.
P812
academic major
Question: What is <subject>>' 's academic Answer <object>.
P1416
affiliation
Question: What is <subjects with? Answer: <object>.
P2522
victory
Question: In which <objectdid <subject=achieve a victory? Answer <object>.
P607
conflict
Question: In which conflict was innovve--ivolved? Answer: <object>.
P749
parent organization
Question: What parent organization is <subject>a part of? Answer: cobject>,
P2283
uses
Question: What does usubject=use? Answer: <object>.
P502
student
Question: which institution <subject>a student? Answer: cobject>.
P119
place of burial
Question: Where is ssubject-buried Answer: <object>>
P2842
place marriage
Question: Where was <subject> married? Answer: <object>.
P286
head coach
Question: Who is the head cuach of <ubject>? Answer: <subject>.
P2541
operating area
Question: What is the operating area of <subject>? Answer: cobject>.
P1441
present in work
Question: In which work is <subject>pnsent Answer <ubject>.
P2650
interested
Question: What is <subject>
in? Answer: cobjects.
P1027
conferred by
Question: Who conferred cobject>? Answer: <subject>. Question: Who is the musical conductor of <object>? Answer: <subjects.
P3300
musical conductor
P2715
elected in
Question: what
<subject elected? Answer: <objects.
P2937
parlimenntrry term
Question: During which parliamentary term did <subject<ervve? Answer <objects.
P1399
convicted of
Question: What
sssubject-cooviict of? Answer: <object>.
P1686
for work
Question: What is <subject-used for terms of work? Answer: <object>.
P11%
of death place of detention
Question: What was the manner of death <subject>? Answer <object>.
P2632
Question: Where was <suhjettsdenined? Answer: <object>.
1991
successful candidate
Question: In which election selection <subject> the succesfull candidate?
Answer: <objects.
P2443
stage reached has writlen for
Question What stage did <subject Answer: <object>.
P6872
Question: For whom or what has <subjectswwitte?? Answer: <object>.
Table 11: Relations and templates for question answering prompts used in PROBELM.
19


Reviews:
[{"review_id": "Kk57fnfQso", "reviewer": "Reviewer_PJpF", "summary": "This paper introduces a Plausibility Ranking Evaluation for Language Models, a benchmark designed to assess their ability\nto discern more plausible from less plausible scenarios through their parametric knowledge. In this benchmark, the LMs need to make predictions by either finishing a statement, completing the text, or completing the answer, and the authors use perplexity to rank all the candidates in the dataset to evaluate each LM's performance. Experiments are conducted on ten open source (relatively) small-size LMs to test the performance on this benchmark and show that factual accuracy (on another dataset) does not directly correlate with plausibility performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thanks for the review.\n\nGround Truth Ranking:\n\nWe use information from Wikidata added after the training cut-off of the LLMs. This ensures our benchmark evaluates their ability to infer new world knowledge beyond co-occurring words. Our analysis shows that the most frequently co-occurring entity is not always the most plausible entity (>~83% of the cases), illustrating that plausibility often requires reasoning beyond mere co-occurrence.\n\nGenerating Ranking Candidates in Real Applications:\n\nOur benchmark can assess the plausibility of selected outcomes but not an exhaustive list. Extending our benchmark to rank a full list of outcomes is a promising future direction and we plan to achieve this with human annotation. This will enhance our benchmark in real-world applications, such as knowledge discovery, where identifying the most plausible hypotheses from a broader set is crucial.\n\nDifference from Beam Search:\n\nBeam search could indeed be used to obtain more plausible hypotheses, and its use is orthogonal to evaluation with our benchmark. However, maximum a posteriori estimates are not always the best way to decode a model, which is why minimum Bayes risk decoding is popular in fields like machine translation. Our method focuses on evaluating the plausibility of scenarios using perplexity, providing a different perspective than beam search.\n\nDeeper Analysis on Key Observations:\n\nOur finding that higher performance on factual accuracy does not necessarily correlate with better plausibility performance can be attributed to the distinct nature of these tasks. Factual accuracy tests a model's ability to recall and verify known facts, while plausibility requires the model to infer and rank scenarios that might not be explicitly present in its training data but are logical extensions of its knowledge. For example, Marcelo being Brazilian is factual, but Marcelo being Spanish is plausible due to his history and Spanish naturalization laws.\n\nThe drop in performance with the 2017 dataset is due to the specific characteristics of the data used. The \"occupation\" relation disproportionately influenced the performance. When “occupation” involves uncommon terms like \"valets de chambre,\" the perplexity is very low across all models, making it the top choice regardless of the subject entity. This occurs because such specific terms might have been less frequently represented in the training data, causing models to assign lower perplexity to these terms when they appear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Precisely evaluating plausibility is essential in many applications, such as knowledge discovery.\n2. The authors conduct experiments on many LMs.", "reasons_to_reject": "1. First, I have doubts about the ground truth ranking in this dataset. If we use entity co-occurrences to make the ranking plausible, then basically, it's all about statistics but not reasoning. Additionally, LMs themselves are trained with loss that are highly correlated with \"token\" co-occurrence, and hence, I don't think it's a substantial ground truth set with this construction setting.\n2. Second, it would be more appreciated if the authors could provide solutions for generating all the \"ranking candidates\" in real applications. For example, if conducting knowledge discovery, how would this plausibility benchmark help us get and rank all plausible candidates? If we still rely on LMs to generate the results and use the perplexity to rank, what would be the difference between it and beam search?\n3. I think deeper analysis should be conducted, especially on the observations the paper found -- is there any reason and insights on \"higher performance on factual accuracy do not necessarily excel in plausibility\" ; Any deeper analysis on the drop of 2017 performance drop?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oyYgweuQtW", "reviewer": "Reviewer_K7jZ", "summary": "This paper proposes PRobELM, a new benchmark suite for plausibility ranking evaluation for language models. \\\nPlausibility ranking, as the research problem the paper studies, is important. The paper also provides a systematic approach to automatically curate test cases (authors name it as \"scenario\"). \\\nThe evaluation section of the paper is its strongest part; it is very extensive, covering ten models and different configurations of the testing. \\\nThe biggest weakness for me is that there appears to be a gap between the paper's realization of plausibility and the actual definition of plausibility (check weakness section).\n\nI am now giving a neutral score (round(5.5) = 6). I look forward to authors' responses and will raise my scores if questions clarified.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thanks for your review! We hope to clarify all your questions:\n\nRealization of Plausibility:\n\nThe approach we take is not directly related to the surprising nature. We consider the most plausible scenarios as immediate next events that, while not recorded in the LLMs’ training data, have since become facts. Imagine traveling back to ancient times when people believed the Earth was flat; the scenario \"the Earth is round\" should be most plausible based on your knowledge, though it would seem very surprising to them. This shows the importance of plausibility in knowledge discovery, distinct from forecasting future events. We acknowledge that our writing in that section might have been confusing as we talked about \"near future events.\" We will rephrase this to clarify.\n\nMarkovian:\n\nOur model doesn't adopt a strictly Markovian approach. While individual timestamps in Wikidata might suggest a single-step view, the nature of world knowledge and historical events captured in LLMs is inherently non-Markovian. This broader context allows for more practical and comprehensive evaluation of plausibility, reflecting real-world applications better.\n\nBayesian Graph Structure:\n\nExploring the graph structure of a Bayesian model to create new evaluation cases is a promising future direction. Yet, our current focus is on scenarios that have already occurred but not in the training data.\n\nComparison with COPA:\n\nCOPA focuses on commonsense reasoning without requiring knowledge about the world. An example in COPA, \"I tipped a bottle\" leading to \"The liquid poured out\" is self-contained and doesn't challenge the model’s world knowledge. In contrast, our benchmark requires models to leverage extensive world knowledge, making the task more complex and relevant for knowledge discovery.\n\nHyperparameter for Less Plausible Events:\n\nRegarding why we chose 10 less plausible scenarios, we aimed to make it reasonably difficult for models to score well with random guesses. We can explore with different numbers of less plausible scenarios in future versions of the paper to provide a comprehensive evaluation.\n\nNegative Sampling:\n\nNegative sampling is centered around event relations, and all 71 relations and 213 templates are detailed in the appendix D (the relations are manually selected in the QC process). This ensures a diverse and representative set of scenarios for evaluation. The full list and distribution of event relations are provided to maintain transparency and reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reason to accept:\n- The problem this paper tackles is important. There are many insightful intuitions in the paper, such as Bayesian inference as an approach to plausibility modeling.\n- The evaluation benchmark is promising, especially concerning the temporal evolution of world facts and a language model, which is trained before (or after) a fact has become established. It is particularly useful for end users to evaluate models on recent facts to assess their reliability in a specific (new) domain and scenario.\n- As mentioned in the summary, the evaluation of the paper is very comprehensive. It is beneficial to have three different evaluation settings (i.e., statement, text completion, and question answering). The consistency of models' rankings is also verified across different task settings and temporal changes.", "reasons_to_reject": "Reason to reject:\n\nI have some reservations about the paper's realization of plausibility in the dataset.\n\nThe definition of plausibility is \"the quality of seeming likely to be true, or possible to believe\" ([Cambridge Dictionary](https://dictionary.cambridge.org/dictionary/english/plausibility)).\n\nThe paper's realization in Section 3.1 (most plausible scenario) adopts a Bayesian view, considering that a future event is the most plausible given the current world state. This current realization is more suited to temporal sequence prediction rather than plausibility prediction. This is because a future event could be very surprising (e.g., Marcelo might score a goal in the 90th minute of a football match, which is very surprising (NOT plausible) given all world states in the previous 89 minutes). Can the authors substantiate how the current implementation actually models plausibility, which can also be modeling surprising events?\n\nIt is interesting to view all events $ (e_1, e_2, ..., e_n )$ in a Bayesian graph model. If I am not mistaken, the authors are now adopting a chaining view. Can the authors explain whether it is Markovian or not (i.e., future events depend only on the previous one step)? Will being Markovian or not make any difference to the empirical evaluation?\n\nI also encourage the authors to explore the graph structure of the Bayesian graph. Is it possible that a new plausibility evaluation case can be created by $P(e_3 | e_2, e_4)$? This means that an intermediate event ($e_3$ in this case) can also be tested as long as it is not shown in the training corpus and we are certain that observed events $(e_2, e_4)$ are shown. If such a graph structure can be realized, can we make the dataset larger?", "questions_to_authors": "Questions for the authors:\n\nMy major questions are raised in the Weakness section. Please help to address them.\n\nOther questions:\n- Can you better articulate or visualize the differences between your dataset and COPA?\n\n- How was the hyperparameter for the 11 less plausible events determined?\n\n- I am not very clear on the negative sampling part (Sec 3.2). Could you articulate or visualize this better? Are these samples centered around the event relations? If so, how many event relations are there in your dataset, and what is their distribution? Do different event relations undergo different processing?\n\n- I am keen to know whether your evaluation is Markovian. P(e_2 | e_1) is different from P(e_3 | e_2, e_1). I think it is interesting to find the difference between only looking at one previous observed event and looking at all observations.", "extra_scores": {}}, {"review_id": "v3qBiS6WiX", "reviewer": "Reviewer_9vFJ", "summary": "This paper introduces a new benchmark called PRobELM, which aims at testing the LLMs capacity of recognizing plausibility of events and scenarios. PRobELM is created from facts extracted from WikiData, using data points that were added at a later timestamp than the training data of the models under evaluation, and then a set of alternative, less plausible facts are generated for each of them and LLMs are asked to rank them from the most to the least plausible.\n\nThe authors evaluate the new benchmark with 10 different LLMs of different families and sizes, using the perplexity scores for each set of scenarios, and found that plausibility inference is a challenging task even for models that excel in identifying factual accuracy. Even if large models generally perform better, model architecture seems to have a more important effect of the results. with the Pythia family achieving strong scores across metrics and settings.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Error Analysis with Surface Form Manipulation:\n\nThank you for this insightful suggestion. We agree that examining the robustness of models' judgments through surface form manipulations, such as paraphrasing and syntactic transformations, would add significant value to our analysis. We are committed to incorporating this additional error analysis into the next version of our paper to better understand the models' robustness and variability in plausibility judgments.\n\nRelated Work:\n\nWe appreciate the reference to the recent preprint by Kauf et al. (2024) on evaluating LLMs from a psycholinguistic perspective. We find the parallels in their findings and our results particularly intriguing, especially the observation that LLM-derived log-likelihoods can approach human-level distinction for possible versus impossible events but fall short on subtle distinctions. We will include a discussion of this related work in our next version to provide a more comprehensive context for our findings and to highlight relevant research in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ important and timely benchmark to evaluate LLMs capacity to discriminate the plausibility of events and scenarios. The dataset seems to be well conceived and several quality control measures have been implemented in the creation process\n+ extensive evaluation with different models, prompt types and careful control of the data contamination issue, thanks to inclusion of Wikidata facts that the LLMs could not have been exposed to, given the timeframe of their training", "reasons_to_reject": "None, for what I can see.", "questions_to_authors": "- I was wondering if you could add an additional error analysis where you manipulate the surface form of the sentences corresponding to a set of scenarios (e.g. one way could be to try to paraphrase them via synonymy replacement, or using some syntactic transformations) and check to what extent the models' judgements are robust to such variations. \n- a recent preprint you might be interested in, evaluating LLMs from a psycholinguistic point of view:\nKauf, C., Chersoni, E., Lenci, A., Fedorenko, E., & Ivanova, A. A. (2024). Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models. arXiv preprint arXiv:2403.14859.\nAs an interesting point in common with your results, I see the fact that LLMs-derived log-likelihood can get you close to human-level for the distinction possible vs. impossible events, but they fall short of humans for more subtle distinctions between events with similar degrees of plausibility.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a benchmark construction method, and specific instantiations, for assessing the ability of language models to identify more or less plausible scenarios. The reviewers were initially mildly positive and after the rebuttal and discussion phase became more strongly positive. 

The reviewers found the paper's contribution to evaluating plausibility assessment to be important and timely. They appreciated the extensive experiments and the quality controls implemented in the benchmark. 

The discussion period raised two important points: 
1. how relevant this benchmark is for future models, given that it's built with snapshots of Wikipedia edits. The authors provided sufficient answers here. I would strongly suggest adding a thoughtful discussion of this in the paper, as well as a concrete procedure in the released code base. 
2. how to define plausibility and what the benchmark is testing for vs other prior datasets (discussion with Reviewer K7jZ). The discussion was insightful and again, the authors should integrate it into the paper. 

Given the discussion and trusting the authors edit the paper accordingly, I recommend acceptance and believe this is a useful contribution to the community.
Decision: Accept