PRobELM: Plausibility Ranking Evaluation for Language Models

Authors: zhangdie  yuan
eric  chamoun
rami  aly
chenxi  whitehouse
andreas  vlachos
marcelo  júnior
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 22:54:19

Abstract: This paper introduces PRobELM (Plausibility Ranking Evaluation for Language Models), a benchmark designed to assess language models' ability to discern more plausible from less plausible scenarios through their parametric knowledge. While benchmarks such as TruthfulQA emphasise factual accuracy or truthfulness, and others such as COPA explore plausible scenarios without explicitly incorporating world knowledge, PRobELM seeks to bridge this gap by evaluating models' capabilities to prioritise plausible scenarios that leverage world knowledge over less plausible alternatives. This design allows us to assess the potential of language models for downstream use cases such as literature-based discovery where the focus is on identifying information that is likely but not yet known. Our benchmark is constructed from a dataset curated from Wikidata edit histories, tailored to align the temporal bounds of the training data for the evaluated models. PRobELM facilitates the evaluation of language models across multiple prompting types, including statement, text completion, and question-answering. Experiments with 10 models of various sizes and architectures on the relationship between model scales, training recency, and plausibility performance, reveal that factual accuracy does not directly correlate with plausibility performance and that up-to-date training data enhances plausibility assessment across different model architectures.

Main paper: 
 introduction: 
 Recent advancements in large language models (LLMs) have significantly enhanced their performance across a wide range of NLP tasks. Alongside these developments, various benchmarks and datasets are introduced to effectively assess the capabilities of LLMs, particularly in terms of knowledge and reasoning (Roemmele et al., 2011; Wang et al., 2018; Clark et al., 2018; Zellers et al., 2019; Wang et al., 2019; Sakaguchi et al., 2021; Lin et al., 2022; Whitehouse et al., 2023; Li et al., 2023) . However, these benchmarks often focus predominantly on evaluating factual accuracy or reasoning abilities without explicitly incorporating broader world knowledge.For instance, benchmarks like TruthfulQA (Lin et al., 2022) are designed to assess the truthfulness or factual correctness of LLMs, evaluating their ability to retrieve and apply information encoded during training, including tasks like mathematical induction. However, they do not explicitly address LLMs' capacity to discern plausibility in scenarios where strict factual accuracy might not be directly applicable. On the other hand, datasets such as COPA (Choice of Plausible Alternatives) (Roemmele et al., 2011) evaluate models through causal reasoning tasks, asking the model to choose the more plausible scenarios from the two options for a given premise. For example, given a premise "I tipped the bottle", scenario "The liquid in the bottle poured out" is more plausible than "The liquid in the bottle froze". While COPA extends evaluation beyond mere factual accuracy by introducing Figure 1 : This figure illustrates the utility of plausibility evaluation in language model outputs: a model, trained on data up until 25 July 2011, ranks "Marcelo is Spanish" as the most plausible among non-factual scenarios. This judgement is based on the model's parametric knowledge of Marcelo's career with a Spanish club, his residency in Spain and the Spanish naturalisation law. The plausibility of this scenario is confirmed as Marcelo was granted Spanish citizenship on 26 July 2011.plausible alternative scenarios, it operates within the constraints of an artificially constructed dataset, potentially oversimplifying evaluation by limiting tasks to binary choices. Notably, fine-tuned state-of-the-art models such as PaLM and PaLM 2 have shown near-perfect performance on COPA (Chowdhery et al., 2023; Anil et al., 2023) , indicating a necessity for more complex and challenging benchmarks that better capture the subtleties of world knowledge and plausibility.Furthermore, in certain domains, combining broad world knowledge and assessing plausibility is crucial. One such domain is literature-based knowledge discovery, particularly within the biomedical field, which can be accelerated by guiding experimental efforts leveraging existing literature (Gopalakrishnan et al., 2019) , requiring the capability of deriving educated guesses that may not be immediately verifiable. In this context, the ability of LLMs to navigate through and infer plausible scenarios from world knowledge becomes invaluable. Yet, as SCIMON (Wang et al., 2023) demonstrates, while LLMs can generate hypotheses, they often lack technical depth and novelty (Wang et al., 2023) , hindering their contribution to specialised fields.To address the limitation in plausibility evaluation, we introduce PROBELM, a novel benchmark designed to directly assess the plausibility inference capabilities of LLMs. PROBELM utilises scenarios collected from real-world data through Wikidata, comprising two primary components: new facts sourced from Wikidata that were unknown to models due to the timeframe of their training data, and a set of automatically generated less plausible scenarios, and asks models to rank scenarios from most to least plausible. For instance, consider a scenario where a model is asked to determine a person's (Marcelo) nationality based on provided facts about their football career and immigration laws, as illustrated in Figure 1 . Provided with the available information in the training data of a model such as "Marcelo is a Brazilian footballer" and "He has played for a Spanish club since 2007", along with pertinent legal information regarding immigration, like "Ibero-American natural-born citizens must have lived in Spain for 2 years to apply for citizenship" and "Ibero-America includes Brazil", the model is expected to rank "Spanish" as the most plausible nationality, reflecting its ability to reason with encoded parametric knowledge.We evaluate PROBELM with 10 language models varying in architecture, parameter scale, and the recency of their world knowledge updates. To ensure alignment in the pre-training data sourced from Wikipedia, we select open-source models that disclose the timestamp of the utilised Wikipedia dump in their training, including GPT-2 (Radford et al., 2019) , BLOOM (Scao et al., 2023) , LLaMA (Touvron et al., 2023) , Pythia (Biderman et al., 2023) , and OLMo (Groeneveld et al., 2024) , spanning parameter counts from 14 million to 7 billion and training data from 2017 to 2023. Our empirical results reveal the following key observations:• Models with higher performance on factual accuracy do not necessarily excel in plausibility, indicating a divergence between factuality and plausibility capabilities. • Larger models generally exhibit better plausibility inference, but this trend varies across model families. • Models trained on less recent data can sometimes outperform counterparts that use more contemporary datasets, demonstrating that model architecture and training methodologies also influence plausibility inference, independent of model size. • A notable trend suggests that the greater the temporal gap between a model's training data cut-off and the date of the evaluation set, the poorer the model performs on PROBELM, underscoring the importance of up-to-date knowledge in evaluating plausibility. 
related work: 
 With the rise of more powerful language models in the form of LLMs, sophisticated reasoning benchmarks such as the Open LLM Leaderboard have been proposed. The benchmark incorporates tasks such as HellaSwag (Zellers et al., 2019) , which evaluates commonsense reasoning within physically situated contexts. Similarly, COPA confronts models with commonsense scenarios. Yet, these benchmarks do not task the model to incorporate knowledge about the world state into its prediction. On the flip side, benchmarks such as TruthfulQA (Lin et al., 2022) , FactScore (Min et al., 2023) , and HaluEval (Li et al., 2023; Yu et al., 2023) measure the factuality of a model's response exclusively. However, previous benchmarks rarely address the complex interplay between world knowledge and plausibility.Factual errors in an LLM's output are also referred to as hallucinations (Xu et al., 2024) and occur when models generate content that is unfaithful to the provided input, established context, or factual correctness. Ji et al. (2023) categorises these into input-conflicting, contextconflicting, and fact-conflicting hallucinations, with the latter category being particularly problematic due to its contradiction with known facts.While all scenarios in the PROBELM benchmark are technically fact-conflicting hallucinations (as they are not yet facts of the world the LLM has seen), the crucial point is that under existing world knowledge, non-factual scenarios are not all equally plausible. Instead, PROBELM probes a model's ability to make educated guesses about plausible futures to gain insights into the capabilities of LLMs to infer within the limits of their knowledge beyond recalling facts it has already seen.The importance of a benchmark that measures the interplay between world knowledge and plausibility in the context of hallucinations is underscored by recent work.  
probelm construction: 
 PROBELM is inspired by acknowledging the middle ground between absolute fact and pure fiction, a domain abundant with feasible scenarios. These scenarios, grounded in logic yet not strictly confined to facts, are crucial for applications such as knowledge discovery.Plausible inferences are valuable in exploring existing literature and uncovering potential connections or hypotheses that may spark meaningful scientific investigation, even if not immediately verifiable. PROBELM seeks to address a significant gap in contemporary evaluation paradigms, establishing a novel benchmark for assessing the proficiency of LLMs in navigating the complexities of real-world knowledge application.In the illustrative scenario provided in Figure 1 , a model trained with data available until the 25th of July, 2011, is asked to evaluate the plausibility of various nationalities for the Brazilian footballer, Marcelo. This task requires the model to engage in deductive reasoning, utilising its accumulated knowledge to prioritise potential nationalities based on their likelihood. Specifically, the model is expected to integrate the information pertaining to Marcelo's eligibility for Spanish citizenship, thereby identifying "Spanish" as a plausible nationality. This example highlights the PROBELM's focus on models' ability to perform deductive reasoning and to apply world knowledge.PROBELM assesses language models' capacity to determine plausibility through three different structured prompts: statements, text completions, and direct inquiries or question answering (see The primary aim across these varied formats is not to generate novel text but to compare perplexity scores for the given scenarios. This methodology seeks to measure a model's proficiency in identifying the most plausible scenario among alternatives, leveraging the context embedded within each prompt type. Such an approach is crucial for assessing a model's capability to evaluate the likelihood or credibility of different scenarios based on the specific context.Evaluating language models on PROBELM is challenging due to its subjective nature and susceptibility to biases, including ethical concerns. To address these complexities, our methodology incorporates a novel strategy grounded in probabilistic reasoning, with an emphasis on Bayesian inference principles. Bayesian inference posits that the likelihood of a hypothesis evolves in response to new evidence. Accordingly, our approach assumes that the most immediate future events, as sequentially recorded in successive updates of Wikidata, represent the most plausible scenarios compared to our current knowledge base.This assumption rests on the motion that the near future is inherently more predictable and grounded in the current state of the world, thereby providing a robust benchmark for evaluating the plausibility of language model predictions.In practice, the first step of our methodology is to determine the recency of each language model's training data, e.g., OLMo's dataset extends up to March 2023. Subsequently, we identify the most plausible scenarios by analysing Wikidata revisions immediately following this cutoff (e.g., in OLMo's case, between March 2023 and the subsequent update). 2 This approach aims to capture recent facts and developments that were not part of the model's knowledge during training. By focusing on Wikidata changes after the training dataset's timeframe, we pinpoint instances that, though are now confirmed facts or events, represent highly plausible scenarios to the models due to their temporal alignment with the immediate future. This method ensures the scenarios selected for evaluation are both relevant and challenging, providing a comprehensive test of the models' capacity to utilise the world knowledge available at the time of their last update. Consequently, this creates a dynamic and adaptive benchmark that accesses the models' plausibility inference abilities.Within the PROBELM framework, the construction of less plausible scenarios leverages a statistical technique grounded in examining entity co-occurrences within Wikidata. By systematically altering the entities and their attributes in a given plausible scenario-while maintaining the relationship unchanged-we assess the plausibility of these variations according to their occurrence frequency throughout the dataset. This assessment extends to a wide spectrum of entity associations, beyond isolated instances, ensuring a comprehensive evaluation of plausibility across varied scenarios.Below is an example of how we assess the plausibility of different scenarios using Wikidata co-occurrences. Take the case of Marcelo, who is known to be Brazilian, and evaluate the plausibility of the triple <Marcelo, has citizenship, Spanish>. We assess the likelihood of the "Brazilian" and "Spanish" attributes being linked under the "has citizenship" relation by calculating their co-occurrence within Wikidata when Marcelo is replaced with different entities. Given the rules facilitating Spanish naturalisation for Brazilians (Figure 1 ), these attributes are likely to co-occur much more frequently than "Brazilian" with "Chinese" for instance, due to China's policy disallowing dual citizenship. Thus, the plausibility of "Chinese" citizenship in this context is diminished. These co-occurrence scores facilitate the construction of a plausibility hierarchy based on statistical evidence from real-world data.By maintaining the logical structure of relations such as "has citizenship" and manipulating the entities and their associated attributes, we generate scenarios from plausible to less plausible. Our approach not only leverages the rich relational data in Wikidata but also mirrors the complexity of real-world knowledge and its application, providing a robust framework for the plausibility assessment of language model outputs. The detailed algorithm for generating these scenarios, including the consideration of entities with complex attributes like dual citizenship, is elaborated in Appendix B.To ensure the generated scenarios are both accurate and relevant, we implement several quality control measures to address potential biases and inaccuracies in the dataset:Paraphrase Discrimination: To address the presence of repetitive entities in Wikidata, we employ FuzzyWuzzy3 , a tool that calculates sequence differences using the Levenshtein Distance. This method helps in identifying and filtering out paraphrases, thereby reducing redundancy and ensuring the dataset reflects a diverse range of entities. By distinguishing between similar yet distinct entities, FuzzyWuzzy enhances the precision of our dataset.We manually review Wikidata edits to exclude those not representing event changes. This involves creating a whitelist of relations representing events, such as "place of death", ensuring the dataset emphasises eventful modifications, and improving the relevance of our scenarios for plausibility assessment. The full whitelist of relations is presented in Appendix D.To avoid over-generic scenarios, we manually filter out general queries derived from the most plausible triples, prioritising queries that yield more specific and insightful scenarios. In the example of <Marcelo, has citizenship, Spanish>, we consider the query on "Which citizenship does Marcelo have?" to be more specific than "Who else is a Spanish citizen?". Table 1 : Evaluated models and their corresponding data timelines in the PROBELM benchmark. This table highlights the diversity of model sizes, their training data recency, and the specific timestamps chosen for evaluating their plausibility inference capabilities.We manually create templates for each relation to convert triples into consistent and natural sentences. This is essential for preserving scenario coherence and readability, facilitating accurate plausibility inference by language models. Customising templates for each relation ensures grammatical correctness and contextual appropriateness, thus improving the dataset's interpretability. The list of the templates used is also illustrated in Appendix D.Our dataset demands leveraging extensive world knowledge, making the task more complex and relevant for knowledge discovery compared to datasets like COPA which focus on commonsense reasoning without extensive world knowledge. For example, the scenario "I tipped a bottle" leading to "The liquid poured out" is self-contained. In contrast, scenarios like the Brazilian footballer Marcelo being plausibly Spanish or Portuguese require extensive background knowledge. Our dataset acknowledges the existence of multiple plausible events for a given scenario and includes ten less plausible scenarios to ensure a challenging evaluation environment, reducing the likelihood of models performing well by random guessing. Negative sampling is designed around event relations, with all 71 relations and 213 templates detailed in Appendix D, ensuring a diverse and representative set of scenarios.Also, note that our concept of plausibility extends beyond identifying surprising events. It targets scenarios that, while not present in the training data of language models LLMs, have subsequently become facts. For instance, the proposition "the Earth is round" would have been highly plausible based on existing knowledge, even though it was not widely accepted in ancient times. This highlights the importance of plausibility in knowledge discovery, distinguishing it from predicting future events. Additionally, while individual timestamps in Wikidata might suggest a Markovian approach, our model does not strictly adhere to this. The world knowledge and historical events encoded in LLMs are inherently non-Markovian, providing a richer context for evaluating plausibility. This broader perspective is crucial for practical and comprehensive plausibility assessments.As the dataset is timeframe-specific regarding the models used for evaluation, we include the statistical details in the following section. 
probelm as a plausibility benchmark: 
 We evaluate PROBELM on 10 models with different sizes and architecture, and the timeframe of training. To ensure alignment in the pre-training data sourced from Wikipedia, we select open-source models that disclose the timestamp of the utilised Wikipedia dump in their training, resulting in models including GPT-2, BLOOM, LLaMA, Pythia, and OLMo.In the following, we detail our experimental setup and the evaluation metrics.We first collect data points from Wikidata that align the timeframe of a given model (outlined in Table 1 ), following the steps detailed in section 3. For each most plausible scenario, we always add 10 less plausible alternatives, resulting in a total of 11 scenarios for models to rank. To ensure our evaluation is comprehensive yet with the practical constraints of time and computational resources, we sampled 126,000 scenarios (6,000 most plausible scenarios and 120,000 less plausible scenarios) from each timestamp, leading to four distinct timeframe-specific evaluation datasets. This sampling covered the five models under study, with OLMo and LLaMA models sharing the same timestamp. Details of the models and the corresponding timestamps are included in Appendix A.Next, we follow the quality control measurements as detailed in subsection 3.3 to filter our data points. The final number of samples for each timestamp varies, as shown in Table 1 .For each set of one most plausible scenario alongside its 10 less plausible alternatives, we probe the models using the three types of prompts described in section 3, employing a zero-shot approach. We calculate the perplexity for these 11 scenarios for each prompt type and then rank them based on their perplexity4 scores.Note that PROBELM is designed to capture unknown or unrecorded events that can be inferred from existing world knowledge, emphasizing knowledge discovery over sequential event forecasting. Effective models should synthesize knowledge plausibly, as illustrated by the example of Marcelo, where evidence should favor plausible over implausible nationalities. Our benchmark remains relevant as long as the training data of LLMs predate the evaluation timestamps. For instance, both OLMO and LLama were evaluated using the same benchmark version, with timestamps more recent than their training data. The benchmark construction process is automated, allowing for easy updates with more recent data if needed, ensuring continued relevance for future models.In assessing the performance of language models on the PROBELM benchmark, we utilise a set of evaluation metrics designed to capture various aspects of ranking plausibility. Each metric offers a different perspective on the models' ability to discern and prioritise plausible scenarios, providing a comprehensive overview of their capabilities.Accuracy focuses on the the top-ranked scenario. This metric determines whether the model can identify the most plausible scenario as its first choice. However, accuracy does not account for the placement of scenarios beyond the top-ranked ones, making it a measure of immediate accuracy rather than a reflection of the overall ranking quality. We note that accuracy here is different from the factual correctness, as none of the ranked scenarios is factual but ranges from most to least plausible.Mean Reciprocal Rank (MRR) is employed to calculate the average of the reciprocal ranks of the most plausible scenario across all queries. MRR offers insight into the model's ability to rank the most plausible scenario highly but does not evaluate the arrangement of other scenarios in the list, emphasising the importance of the top plausible scenario's position within the model's ranking.Normalised Discounted Cumulative Gain (NDCG) is utilised to assess the quality of the entire ranked list produced by the model. NDCG takes into account the graded relevance of all ranked scenarios, providing a measure of the list's overall quality and the effectiveness of the model's ranking across different levels of plausibility. Unlike accuracy and MRR, NDCG does not isolate its evaluation to any single scenario, such as the top-ranked or the most plausible one, but instead evaluates the cumulative relevance of the whole list, rewarding rankings that place more plausible scenarios higher.In addition, we measure the PLAUSIBILITY score, where we average accuracy, MRR, and NDCG scores across the three different prompt types. Table 2 : Evaluation results of language models on PROBELM with corresponding timeframe specific datasets. We report accuracy, MRR, and NDCG scores for three prompt types. AVG indicates average plausibility across all prompts and metrics (average of nine scores).Best performances are in bold, and second-best results are underlined. 
main results on probelm: 
 We present the plausibility score, and accuracy, MRR, and NDCG scores of different prompt types in Table 2 , using the timeframe-specific datasets for each model.The results reveal that while model performance exceeded random chance, overall scores remain low, underscoring the complexity of the plausibility assessment task. For instance, considering a random baseline accuracy of 9.09% when ranking 11 scenarios, most models surpass this baseline across various sizes and prompts. Notably, both BLOOM models marginally outperform the random baseline by approximately 3 points, achieving 12.54% and 12.80% on the question-answering prompt. Specifically, the BLOOM-560M model underperform relative to the random baseline in the text completion prompt with 8.78% accuracy. This performance pattern suggests that language models face significant challenges in differentiating between scenarios of varying plausibility. It also highlights a distinct separation between models' capabilities in factual accuracy and their ability to infer plausibility. These observations set the stage for our key findings:Larger models generally outperform smaller ones in plausibility tasks, but performance varies across model families. For example, the tween the performances on various prompts, which were notably high: 0.8281 for statement versus text completion, 0.8672 for statement versus question answering, and 0.8976 for text completion versus question answering. Such strong correlations underscore that, despite the prompt-specific performance disparities, the models are evaluated based on a core aspect of their capability of ranking plausibility. Similarly, the Pearson correlation score between MRR and NDCG is 0.9982, indicating consistent plausibility measurements across metrics.To comprehensively understand the different coverage between PROBELM and other LLM benchmarks, we follow lm-evaluation-harness5 to evaluate zero-shot performance on COPA, TruthfulQA (mc2, i.e., multiple options can be correct), ARC (challenge set) (Clark et al., 2018) , HellaSwag (Zellers et al., 2019) , and Winogrande (Sakaguchi et al., 2021) . Default metrics (accuracy or normalised accuracy) are applied to each dataset and the results are included in Table 3 .Models that perform well on tasks emphasising factual accuracy do not always maintain their performance levels on plausibility assessments. In some cases, the performance rankings are inversely related. A notable example of this phenomenon is observed in the performance disparity between PROBELM and TruthfulQA. Specifically, Pythia-14M, which exhibits the best performance in TruthfulQA, positions near the bottom (9th place) of the PROBELM ranking. This variance underscores the distinct nature of plausibility as an evaluation criterion, demanding separate capabilities beyond those required for tasks focused on factual accuracy. The development of PROBELM thus contributes to the broader framework for evaluating language models by offering a different angle to assess their capability to understand and generate plausible content.We further evaluate models against datasets from 2017, 2020, 2021, and 2023 timestamps to investigate how the recency of training data affects the plausibility performance. The outcomes of this analysis are presented in Figure 2 and detailed in Appendix C. Remarkably, the relative rankings of models prove relatively consistent across these varied timeframes, affirming the robustness of our evaluation approach. We observe that, from the 2020 to 2023 timestamps, models trained closer to or after the evaluation dates typically exhibit enhanced performance, underscoring the benefits of leveraging recent data. An anomaly is detected in the 2017 dataset, where all models show a decrease in performance, suggesting it is an outlier. An analysis reveals that the "occupation" relation disproportionately influences the 2017 dataset, constituting 14.3% of all scenarios compared to less than 0.5% in the other datasets. This discrepancy indicates a significant impact of the "occupation" relation on model performance. Excluding the "occupation" relation from the evaluation on the 2017 dataset results in either similar or higher scores across models, shown in Appendix C. For instance, evaluating the Pythia-160M model without scenarios involving the "occupation" relation leads to a notable increase in plausibility scores from 39.69 to 42.73. This outcome indicates the models' challenges with certain relations. 
conclusion: 
 In this work, we introduce PROBELM, a benchmark designed to evaluate language models' ability to discern plausible scenarios, filling a critical gap left by existing benchmarks focused on factual accuracy or plausibility without world knowledge integration. Our comprehensive evaluation of 10 models of varying sizes and architectures against this benchmark reveals key insights including factual accuracy does not directly correlate with plausibility discernment; larger models do not uniformly outperform smaller ones across different families; the recency of training data, while impactful, is not the sole determinant of plausibility performance; and models tend to struggle with plausibility assessment as the gap between their training data and the evaluation set widens. These findings highlight the complexity of plausibility inference and the need for advanced modelling techniques that can better address the challenges of real-world knowledge application. Our dataset and code are publicly available, to facilitate the community for further exploration in enhancing language models' plausibility inference capabilities. 
 
 Appendix: We briefly introduce below the models and the timestamp of the corresponding Wikidata used for our experiments.GPT-2 124M and 1.5B (Radford et al., 2019) As early iterations in the GPT series, these models offer a perspective on plausibility inference with training data frozen in December 2017. This provides a basis for evaluating how well models can reason about plausibility without the benefit of recent information. The closest corresponding timestamp we found is from December 2017 to January 2018.BLOOM 560M and 7B (Scao et al., 2023) Representing the BLOOM series' commitment to openness and scalability, these models, with data up to December 2021, allow us to explore how differences in model size impact their capacity to discern plausible scenarios without having been exposed to subsequent updates. The closest corresponding timestamp we found is from December 2021 to January 2022.LLaMA 7B (Touvron et al., 2023) LLaMA is recognised for its robust performance across a broad range of NLP tasks. Its training data, current as of August 2022, positions it uniquely for evaluation with the PROBELM benchmark, specifically designed to assess the model's ability to infer plausibility based on its training up to that point. The closest corresponding timestamp we found is from December 2023 to January 2023.Pythia 14M, 160M, and 2.8B (Biderman et al., 2023) The Pythia models extend our exploration into the realms of modern architecture and model scalability, spanning the spectrum from an ultra-compact 14M to a robust 2.8B parameter setup. This inclusion allows us to assess plausibility inference capabilities across a broad range of model sizes, particularly emphasising the performance of smaller models like the Pythia 14M. The evaluation of Pythia models leverages their training data up to March 2020. The closest corresponding timestamp we found is from October 2020 to November 2020.OLMo 1B and 7B (Groeneveld et al., 2024) The newest models in our evaluation, OLMo's training data extends to March 2023. Their assessment against the PROBELM benchmark tests their ability to gauge plausibility based on a comprehensive and current dataset, without incorporating the very latest updates. The closest corresponding timestamp we found is from December 2023 to January 2023.To systematically generate less plausible scenarios for the PROBELM benchmark, we use the algorithm shown in Algorithm 1, which leverages the statistical distributions of entity co-occurrences within Wikidata. This method ensures that the generated scenarios, while being less plausible, still maintain logical coherence and relevance to the given context. An example of the prompts used to evaluate the models is shown in Table 8 . We employ TemporalWiki(Jang et al., 2022) to generate the edits between two Wikidata dumps.https://pypi.org/project/fuzzywuzzy/From a certain perspective, beam search could also be used to obtain more plausible hypotheses, and its use is orthogonal to evaluation with our benchmark. However, maximum a posteriori estimates are not always the best way to decode a model, which is why minimum Bayes risk decoding is popular in fields like machine translation. Our method focuses on evaluating the plausibility of scenarios using perplexity, providing a different perspective than beam search.https://github.com/EleutherAI/lm-evaluation-harness 


Decision: This paper proposes a benchmark construction method, and specific instantiations, for assessing the ability of language models to identify more or less plausible scenarios. The reviewers were initially mildly positive and after the rebuttal and discussion phase became more strongly positive. 

The reviewers found the paper's contribution to evaluating plausibility assessment to be important and timely. They appreciated the extensive experiments and the quality controls implemented in the benchmark. 

The discussion period raised two important points: 
1. how relevant this benchmark is for future models, given that it's built with snapshots of Wikipedia edits. The authors provided sufficient answers here. I would strongly suggest adding a thoughtful discussion of this in the paper, as well as a concrete procedure in the released code base. 
2. how to define plausibility and what the benchmark is testing for vs other prior datasets (discussion with Reviewer K7jZ). The discussion was insightful and again, the authors should integrate it into the paper. 

Given the discussion and trusting the authors edit the paper accordingly, I recommend acceptance and believe this is a useful contribution to the community.

Reviews: review: paper_summary: paper_summary: This paper introduces a Plausibility Ranking Evaluation for Language Models, a benchmark designed to assess their ability
to discern more plausible from less plausible scenarios through their parametric knowledge. In this benchmark, the LMs need to make predictions by either finishing a statement, completing the text, or completing the answer, and the authors use perplexity to rank all the candidates in the dataset to evaluate each LM's performance. Experiments are conducted on ten open source (relatively) small-size LMs to test the performance on this benchmark and show that factual accuracy (on another dataset) does not directly correlate with plausibility performance.
strength_weakness: strength_weakness: 1. First, I have doubts about the ground truth ranking in this dataset. If we use entity co-occurrences to make the ranking plausible, then basically, it's all about statistics but not reasoning. Additionally, LMs themselves are trained with loss that are highly correlated with "token" co-occurrence, and hence, I don't think it's a substantial ground truth set with this construction setting.
2. Second, it would be more appreciated if the authors could provide solutions for generating all the "ranking candidates" in real applications. For example, if conducting knowledge discovery, how would this plausibility benchmark help us get and rank all plausible candidates? If we still rely on LMs to generate the results and use the perplexity to rank, what would be the difference between it and beam search?
3. I think deeper analysis should be conducted, especially on the observations the paper found -- is there any reason and insights on "higher performance on factual accuracy do not necessarily excel in plausibility" ; Any deeper analysis on the drop of 2017 performance drop?
paper_summary: paper_summary: This paper proposes PRobELM, a new benchmark suite for plausibility ranking evaluation for language models. \
Plausibility ranking, as the research problem the paper studies, is important. The paper also provides a systematic approach to automatically curate test cases (authors name it as "scenario"). \
The evaluation section of the paper is its strongest part; it is very extensive, covering ten models and different configurations of the testing. \
The biggest weakness for me is that there appears to be a gap between the paper's realization of plausibility and the actual definition of plausibility (check weakness section).

I am now giving a neutral score (round(5.5) = 6). I look forward to authors' responses and will raise my scores if questions clarified.
strength_weakness: strength_weakness: Reason to reject:

I have some reservations about the paper's realization of plausibility in the dataset.

The definition of plausibility is "the quality of seeming likely to be true, or possible to believe" ([Cambridge Dictionary](https://dictionary.cambridge.org/dictionary/english/plausibility)).

The paper's realization in Section 3.1 (most plausible scenario) adopts a Bayesian view, considering that a future event is the most plausible given the current world state. This current realization is more suited to temporal sequence prediction rather than plausibility prediction. This is because a future event could be very surprising (e.g., Marcelo might score a goal in the 90th minute of a football match, which is very surprising (NOT plausible) given all world states in the previous 89 minutes). Can the authors substantiate how the current implementation actually models plausibility, which can also be modeling surprising events?

It is interesting to view all events $ (e_1, e_2, ..., e_n )$ in a Bayesian graph model. If I am not mistaken, the authors are now adopting a chaining view. Can the authors explain whether it is Markovian or not (i.e., future events depend only on the previous one step)? Will being Markovian or not make any difference to the empirical evaluation?

I also encourage the authors to explore the graph structure of the Bayesian graph. Is it possible that a new plausibility evaluation case can be created by $P(e_3 | e_2, e_4)$? This means that an intermediate event ($e_3$ in this case) can also be tested as long as it is not shown in the training corpus and we are certain that observed events $(e_2, e_4)$ are shown. If such a graph structure can be realized, can we make the dataset larger?
questions: questions: Questions for the authors:

My major questions are raised in the Weakness section. Please help to address them.

Other questions:
- Can you better articulate or visualize the differences between your dataset and COPA?

- How was the hyperparameter for the 11 less plausible events determined?

- I am not very clear on the negative sampling part (Sec 3.2). Could you articulate or visualize this better? Are these samples centered around the event relations? If so, how many event relations are there in your dataset, and what is their distribution? Do different event relations undergo different processing?

- I am keen to know whether your evaluation is Markovian. P(e_2 | e_1) is different from P(e_3 | e_2, e_1). I think it is interesting to find the difference between only looking at one previous observed event and looking at all observations.
paper_summary: paper_summary: This paper introduces a new benchmark called PRobELM, which aims at testing the LLMs capacity of recognizing plausibility of events and scenarios. PRobELM is created from facts extracted from WikiData, using data points that were added at a later timestamp than the training data of the models under evaluation, and then a set of alternative, less plausible facts are generated for each of them and LLMs are asked to rank them from the most to the least plausible.

The authors evaluate the new benchmark with 10 different LLMs of different families and sizes, using the perplexity scores for each set of scenarios, and found that plausibility inference is a challenging task even for models that excel in identifying factual accuracy. Even if large models generally perform better, model architecture seems to have a more important effect of the results. with the Pythia family achieving strong scores across metrics and settings.
strength_weakness: strength_weakness: None, for what I can see.
questions: questions: - I was wondering if you could add an additional error analysis where you manipulate the surface form of the sentences corresponding to a set of scenarios (e.g. one way could be to try to paraphrase them via synonymy replacement, or using some syntactic transformations) and check to what extent the models' judgements are robust to such variations. 
- a recent preprint you might be interested in, evaluating LLMs from a psycholinguistic point of view:
Kauf, C., Chersoni, E., Lenci, A., Fedorenko, E., & Ivanova, A. A. (2024). Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models. arXiv preprint arXiv:2403.14859.
As an interesting point in common with your results, I see the fact that LLMs-derived log-likelihood can get you close to human-level for the distinction possible vs. impossible events, but they fall short of humans for more subtle distinctions between events with similar degrees of plausibility.

Comments: title: Several questions are clarified
Additional rebuttal
Reply to 3/3
Reply to 2/3
Reply to 1/3
Response from Reviewer
One final question  (thread 3/N, N=3)
Additional rebuttal
Other questions
ACK your response
Response from Reviewer
comment: Hi authors:

Thank you for your fast responses. I am sympathetic with you about your arguments about your work is about known VS unknown (in the parametric knowledge of a LM). I believe this is one key difference against prior works. I am raising my scores to 7 given such arguments. 

I still have reservations about the modelling part. Current approach is black box and it does not tell what is a LM actually modelling.  As you wrote, and I agree, that the skills is to synthesise its knowledge in a plausible manner. I believe it should be defined in a formal way -- what knowledge and in which manner (e.g. deductive or inductive reasoning?). 

Best, K7jZ
Thanks again for your review and additional question.

Sorry for the confusion. In the response we meant that: using our 71 relations, extending our dataset to a future version of Wikidata for future models is fully automated, as the quality control (sec 3.3) is relation specific not timestamp specific. Given that we have done the quality control, the rest of the data construction is fully automated therefore extending is also fully automated.
Q: Can you help us differentiate your work against conditional QA/NLI or situated QA/NLI. For example, this paper [1] appears to adopt a pretty similar formalisation as you do in your paper (e.g. Figure 1 in this paper).

A: There are at least two differences we can think of between ProbELM and SituatedQA: 1) SituatedQA is about known facts, e.g. in the example of Figure 1 in the paper all of the answers are factual, just at different points in time/geographical space. However, something can be plausible but not yet known as a fact, and this is the focus of ProbELM. 2) The (factual) answers to the SituatedQA change over time, but this is not the case in ProbELM; the shape of the earth will not change, but it is not known at the time.

Q: Following the example of "earth is round", can we say your work can be seen as considering the truthfulness of a proposition conditioned on one additional fact (e.g. "ship will return to the same point after sailing for months").

A: It can happen like this, but It would not always be the case. If someone did this experiment and the ship did not return to the same point (due to wind or any other reason), the earth being round would remain a plausible hypothesis.

Q: If above is true, the questions are (1) how would you differentiate your work against previous conditional / situated QA/NLI studies? (2) Do you think inferring such hidden / additional facts is the crux for solving your task? What can we expect from a model if it can perform very well on your task?

A: As said, the above is not always true. Having said this, a good model would perform well in our task if it is able to synthesize its knowledge in a plausible manner. As stated in the example in figure 1 of our paper, the evidence about the footballer Marcelo, if combined correctly, should make the Spanish nationality more plausible than Irish.
Q: Given there are many possibilities about the facts of the world, do you think your dataset have a good coverage about the next plausible event? For example, "the earth is round" is the case where there is only one plausible event. Will there be cases where multiple plausible events exist? What will be the coverage of your dataset on it? 

A: Yes there will be cases where there are multiple plausible events. Take the Brazilian footballer Marcelo again, it is plausible that he is Spanish (he actually is, but not in training data) due to the fact he played in Spain for a number of years and the immigration policy and let’s say it is also plausible that he is Portuguese because his parents are Portuguese. In this case, our dataset would also place Portuguese near the top of the rankings because there are many Brazilians that are also Portugueses, thus the co-occurrences between these two nationalities must be frequent. Yet you are right we only evaluated the most plausible event (in this case it would be Spanish) and ignored the second plausible event (Portuguese). Extending our evaluation to multiple plausible events would be a promising future direction.

Q: Following the discussion on the multiple plausible events. What will be the distribution of the N (number of) multiple plausible events? Will it be a normal distribution?

A: It depends on the relation, if the relation is mutually exclusive (such as place of death), then the distribution is more concentrated whereas other relations (such as place visited) can have many plausible events. Therefore the distribution should be different from relation to relation.

Q:What properties should a model possess to perform well on your task of plausibility prediction? In another word, what are we actually modelling? Are we modelling knowledge and facts? Are we modelling a capacity for sequential prediction?

A: PRobELM is trying to capture the unknown/unrecorded/unthought events (not known facts at the time) that can be inferred from current world knowledge, especially in the context of knowledge discovery. PRobELM is not about forecasting the future sequential events, which would be uncertain (not guaranteed to happen).
Thanks again for your review and additional questions

Q: Based on your explanation, can we say that the plausible events given the current states must be seen in written corpora? If people haven't discovered the earth is round (and write them down in corpora), what will be the plausible event?

A: No it doesn’t need to be written in corpora to be a plausible event. And such a plausible event, the earth is round, although not discovered, should be possible to infer from other knowledge such as natural phenomena that cannot be explained if the earth is not round. And better LLMs should be better able to handle this.
Hi Authors,

Thank you for your explanation. However, from the paper, especially Section 3.3, I don't see the dataset creation is fully automated. Can you elaborate more about this?

Thank you.
## Final question (thread 3/N, N=3) ##

Hi Authors:

I have another question. Can you help us differentiate your work against conditional QA/NLI or situated QA/NLI. For example, this paper [1] appears to adopt a pretty similar formalisation as you do in your paper (e.g. Figure 1 in this paper). 

Following the example of "earth is round", can we say your work can be seen as considering the truthfulness of a proposition conditioned on one additional fact (e.g. "ship will return to the same point after sailing for months"). 

If above is true, the questions are (1) how would you differentiate your work against previous conditional / situated QA/NLI studies? (2) Do you think inferring such hidden / additional facts is the crux for solving your task? What can we expect from a model if it can perform very well on your task?

[1] SITUATEDQA: Incorporating Extra-Linguistic Contexts into QA. EMNLP 2021. https://aclanthology.org/2021.emnlp-main.586.pdf

Best, K7jZ
Ground Truth Ranking:

We will report more detailed statistics in the next version.

Future LLMs:

Benchmark is built for each LLM: Your concern is reasonable, however our benchmark can be reused as long as the training data of the LLM is not more recent than the evaluation time stamps of the benchmark. See in Table 1, that OLMO and LLama are evaluated on the same version of the benchmark, since the timestamps considered are more recent than the training data. Furthermore, we can always remove evaluation scenarios as more recent training data is used. Finally, the benchmark construction process is fully automated, so even if we needed to create new scenarios for later timestamps than the ones we considered in this version, it would only require running our code on more recent versions of wikidata.
Hi authors, 

Sorry for my late reply!

If your bandwidth allows, I have following questions:

- Given there are many possibilities about the facts of the world, do you think your dataset have a good coverage about the next plausible event? For example, "the earth is round" is the case where there is only one plausible event. Will there be cases where multiple plausible events exist? What will be the coverage of your dataset on it? 
- Following the discussion on the multiple plausible events. What will be the distribution of the N (number of) multiple plausible events? Will it be a normal distribution?
- What properties should a model possess to perform well on your task of plausibility prediction? In another word, what are we actually modelling? Are we modelling knowledge and facts? Are we modelling a capacity for sequential prediction? 

I hope these questions can benefit your own understanding of the topic and help AC/colleague reviewers decide. 

Best, K7jZ
Hi authors, 

Thanks for your response. Thanks for the clarification for the first point. I would raise the score from 5.5 (neutral) to 6 as slightly positive for your paper. 

Based on your explanation, can we say that the plausible events given the current states must be seen in written corpora? If people haven't discovered the earth is round (and write them down in corpora), what will be the plausible event?
Thank you for the explanation. I get most of the rebuttal ideas. For the "Ground Truth Ranking" concern, I would expect a more thorough analysis to be presented in the later version of the paper.

Another concern I have for this paper is that this paper is built for each LM and uses the corresponding cutoff time to generate the data. How do you expect this benchmark to be long-lasting for future research and development on LM?

If this additional concern can be answered, I'll further raise my evaluation.
Thanks you for your response! I confirm my (positive) judgement of this work.