Every Answer Matters: Evaluating Commonsense with Probabilistic Measures (2024)

Abstract: Large language models have demonstrated impressive performance on commonsense tasks; however, these tasks are often posed as multiple-choice questions, allowing models to exploit systematic biases (Li et al., 2022). Commonsense is also inherently probabilistic with multiple correct answers. The purpose of “boiling water” could be making tea, cooking but also could be killing germs. Existing tasks do not capture the probabilistic nature of common sense. To this end, we present commonsense frame completion (CFC), a new generative task that evaluates common sense via multiple openended generations 1. We also propose a method of probabilistic evaluation that strongly correlates with human judgments. Humans drastically outperform strong language model baselines on our dataset, indicating this approach is both a challenging and useful evaluation of machine common sense.

heading: 1 Introduction
text: Most existing commonsense evaluations utilize multiple-choice question-answering (MCQA) tasks (Talmor et al., 2019; Sap et al., 2019a; Huang et al., 2019; Bhagavatula et al., 2020; Feng et al., 2022). This format offers a limited view of common sense. MCQA tasks simplify the problem with unrealistically small answer sets, and making the options challenging is difficult (Zellers et al., 2018, 2019). More crucially, common sense is implicit – understanding assumptions that are unspoken precisely because they are common knowledge. MCQA, by evaluating common sense explicitly, fails to capture a model’s ability to utilize this knowledge in unprompted, generative contexts. MCQA is also fundamentally at-odds with the fact
*Now at Google DeepMind, Eightfold.ai, Thorn, and Meta respectively.
1Dataset and PROBEVAL available at: https://github. com/qxc101/PROBEVAL_CFC/
that common sense is inherently probabilistic, and should be evaluated as such.
In order to avoid the issues in MCQA, many recent benchmarks have proposed generative commonsense evaluations. (Lin et al., 2020; Chen et al., 2023). While generative evaluation avoids the difficulty of generating hard negatives, it does not reflect the fact that there are often multiple correct answers, especially for commonsense questions. For example, in the phrase "they boiled the water", we can infer using our common sense that the most likely reason for this action is cooking or making tea. However, people in areas with limited clean water access may view this as a way to remove germs and ensure it’s safe to drink. This aspect is, unfortunately, frequently overlooked during the benchmark creation process. To ensure that the model can serve diverse populations, it is important to gather multiple responses. By focusing on collecting implicit information from larger popu-
493
lation, this provides a more accurate evaluation of common sense required in real-world settings.
In this setting, it is crucial to address the commonsense questions with multiple correct answers. While some previous works have proposed a clustering-and-ranking evaluation (Boratko* et al., 2020), we demonstrate that such an approach can lack nuance. Instead, we embrace the probabilistic nature of common sense and evaluate the model’s ability to capture a probability distribution over answer clusters sampled from the population. This ensures that the model not only captures common sense, but does so in a calibrated way.
To this end, we propose commonsense frame completion (CFC), a task focused on inferring missing information in a context sentence. This emphasizes the implicit nature of common sense in a given context and is tightly connected to downstream applications, such as home assistants, where inferring such information about a user’s query is key. In CFC, the questions are generated by identifying missing information from a given context sentence about daily scenarios. For each context-question pair, we collected a large number of diverse answers from human annotators.
In order to evaluate multiple responses effectively, we additionally propose a new evaluation method. Responses from different individuals often vary, and common sense is typically defined as the knowledge shared by nearly all people. To make every answer count, we consider the annotators’ answers from a probabilistic viewpoint and evaluate models using probabilistic measures (Moss, 2018; Pavese, 2020; Chater et al., 2006). Specifically, given raw string answers, we cluster them and consider the categorical distribution over these clusters based on the number of answers contained in each. We propose automated clustering and alignment mechanisms which allow models to be evaluated directly by comparing the KL divergence between distributions. The task format and evaluation metric are shown in Figure 1. We evaluate the proposed metric with two datasets, CFC and ProtoQA (Boratko* et al., 2020), and show high correlations with human judgments for both datasets.
Finally, we report multiple LLMs’ performance on CFC measured by the proposed probabilistic metrics. We identify a large performance gap between existing large models and humans, indicating the limitations of current LLMs.

heading: 2 Related Work
text: Commonsense Evaluation Creating commonsense benchmarks to evaluate model performance is a long-standing research topic (Sakaguchi et al., 2020; Lin et al., 2020; Sap et al., 2019b; Zhou et al., 2020). However, most benchmarks are created using a multiple-choice selection paradigm, which is simpler to evaluate but misaligned with the realworld use-case of commonsense knowledge, and most egregiously ignores the existence of multiple correct answers. We are not the first ones to gather multiple human answers to facilitate robust evaluations, however. Aydin et al. (2014) and Boratko* et al. (2020) also collected multiple human responses for each question to get aggregated human ground-truth answer sets.
Our work differs from these due to our emphasis on commonsense as implicit and probabilistic. We don’t treat each answer equally; rather, we aim to match the answer distribution given by human responses. For this purpose, we propose a novel probabilistic evaluation for open-ended generation tasks with multiple correct answers. A similar probabilistic evaluation was studied from a language model generation point of view (Pillutla et al., 2021). They proposed a KL-based evaluation to measure language model generations, while our focus is on the implicit answer distribution.
Among the previous work, our proposed dataset CFC is most similar to ProtoQA (Boratko* et al., 2020) as they share a similar task format (one question, multiple answers). However, there are several key differences. CFC is designed to uncover implicit commonsense information in various contexts, making it a more generalized variant of ProtoQA. Consequently, the question format in CFC is significantly more diverse than that of ProtoQA. In addition, the number of answers collected for CFC is rigorously justified through the Neyman-Pearson lemma, guaranteeing a sufficient and representative sample.
Commonsense as Probabilistic Knowledge In most knowledge evaluation benchmarks, commonsense knowledge is defined as absolute facts (Bian et al., 2023; Chen et al., 2023). We relax this absolute intersection between human knowledge using a probabilistic approach. The probabilistic notion of knowledge is well supported. Moss (2018) stated human beliefs or credences, inherently probabilistic, should be regarded as legitimate forms of knowledge. The growing trend towards building
probabilistic models in cognitive studies reinforces the idea of human cognition and memory functioning as probabilistic processors (Chater et al., 2006). This collective body of work supports the view that common sense serves as probabilistic assumption instead of definite judgements in human mind.

heading: 3 CFC Task Description
text: In this section we motivate and describe the task of "commonsense frame completion" (CFC). We aim to create a task which evaluates implicit common sense with multiple correct answers. Given a direction such as “put the water on the burner to boil”, it is common sense which allows us to understand that the water is likely in a kettle and not simply dumped on the burner. Unlike factual question answering tasks, there is no single correct answer. In this example, the water could be placed in a “kettle”, “pot”, “cup”, or “glass”, although the former answers are more probable.
It is necessary for any machine learning model which claims to capture common sense to correctly predict all the possible answers and have some sense of the distribution over the implicit information. To assess a model’s ability in this regard, we view the context sentence as a structured semantic frame, identify a missing slot, and ask the model to provide a distribution of potential slot fillers.

heading: 4 Dataset Creation and Analysis
text: We now describe the dataset creation process of CFC. We first need to collect reasonable context sentences which contain natural element of common sense. CommonGen (Lin et al., 2020) is a commonsense dataset which contains many short sentences describing basic information about daily life, and so we choose this dataset as the source for potential context sentences.
Given a short sentence, we then identify implicit information. To this end, we perform semantic parsing on the sentence, and identify missing slots. We use AMR (Banarescu et al., 2013) for semantic parsing based on its ability to provide a rich representation of the sentence with a pre-defined fixed schema for the predicate roles. If a predicate is found, AMR parsing will match it to a schema and fill in the values for any identified slots. Any slots marked with amr-unkown indicate potential items of missing information, enabling us to obtain human annotations for the missing slot values.
We uniformly sampled 63,788 sentences from the CommonGen dev dataset, and parsed them using the AMR parser from Cai and Lam (2020), generating 228,170 pairs of context questions with missing slots. From this, we randomly sampled 101 (sentence, missing slot) pairs for crowd workers to annotate, such that we had a balanced distribution of missing slot types, as detailed in Section 4.2. We present the context sentence and missing slot to crowdworkers, who were also provided with training examples and descriptions of the meaning of each slot type (see Table 1). The number of answers is chosen such that the resulting answer distribution is stable (see Section 4.2). Each element of the raw dataset therefore includes a context sentence, missing slot value, and a collection of slot fillers.

heading: 4.1 Probability Distribution
text: In an open-ended task where multiple humans are asked to provide answers as raw strings of text there are a multitude of answers which may essentially capture the same underlying idea. Ultimately we are not interested in the different variations of the surface form, but rather in capturing the essence of the underlying concept. In the boiling water example, we may want to treat "kettle" and "teapot" as
though they were representative of the same general concept. As originally proposed in Boratko* et al. (2020), we consider clustering the responses, converting a set of answer strings into a categorical distribution over answer clusters, where the probability of obtaining an answer from a given cluster is proportional to the number of answer strings contained within it. We explore both manual clustering and automated clustering methods (see Section 5).

heading: 4.2 Analysis
text: Number of Answers The number of potential slot fillers might be very large, and we want to ensure we sample enough to approximate the true distribution over answer concepts. An essential question is how many samples are enough to approximate the true distribution with reasonable error rate? This is a classic problem in statistics, for which the Neyman-Pearson lemma proves that the uniformly most powerful test is to consider the KL divergence DKL(g∥f) = ∑ x g(x) log g(x) f(x) where g is the empirical distribution and f is the true distribution (Harremoës and Tusnády, 2012). The recent work from Mardia et al. (2020) showed that this can be bounded by the following equation
P(DKL(gn,k∥f) ≥ ϵ) ≤ e−nϵ [ 3c1 c2 k−2∑
i=0
Ki−1( e √ n 2π )i ]
where c1 and c2 are constant values, n is the number of samples, and k is the number of categories in the categorical distribution.
In our setting, we manually clustered 50 questions, and found that the number of categories is not more than 8. To get a bound on the number of answers we should collect, we set ϵ = 0.2, k = 8, and solve e−nϵ [ 3c1 c2 ∑k−2 i=0 Ki−1( e √ n 2π ) i ] for n. Figure 2 (a) shows the value of this bound on the y-axis for increasing numbers of samples n on the x-axis. As we can see from the graph, for 100 samples, the error rate is less than 0.05, allowing us to approximate the true answer distribution with 95% confidence if there are fewer than 8 categories in the categorical distribution.
Question Types We collected 101 (context, missing slot) pairs, and obtained 100 slot fillers for each from crowdworkers, resulting in 10,100 annotations overall. The data collection page we used on Amazon MTurk is shown in Fig 72. We create
2The annotators are paid 0.15 per answer, and they are all anonymous English speakers who are based in the US.
a dev set with 55 examples and a test set with 46 examples. The distribution of missing slot types are shown in Figure 2 (b). Each question type is associated with a different type of commonsense reasoning, e.g time represents temporal commonsense reasoning.

heading: 5 Probabilistic Evaluation
text: In this section, we detail the method of evaluating multiple correct answers. As we relaxed commonsense to be probabilistic knowledge, a rigorous probabilistic evaluation is required; however the task is presented (both to humans and models) as a generative question answering task. Therefore, we need a way to compare two large sets of answer strings. We will start by how human evaluators may compare the similarity of there sets of answers and then describe the various ways by which this process can be automated.

heading: 5.1 General Framework
text: Given a question, the ground truth answer set G and the model generated answers H, the goal is to evaluate the similarity between these two answer sets. This is a difficult task even for a human, especially if the answer sets are large and diverse, however, we cares more about concepts being captured rather than unique surface forms. So we start with clustering the answer strings in G to form meaningful concept level clusters.3 We could then match the answers in H to the proposed groundtruth clusters in G. Upon having the clusters, we could define categorical probability distributions over the clusters, Pg and Ph, where the probability assigned to a given cluster is proportional to the
3When clustering, a new category "wrong" is added to the answer set to account for the wrong answers for a question. These will then be discarded prior to model evaluation.
number of answer strings assigned to it.4 Finally, the similarity between G and H can be inferred by calculating the KL divergence of the two distributions, DKL(P̂g||P̂h). The overall evaluation framework is depicted in Figure 3.

heading: 5.2 Automatic Evaluation
text: Based on the general framework, we propose an automatic metric, PROBEVAL. The key steps are: 1. Embed ground-truth answers from G into vector space. 2. Automatically cluster the embeddings to obtain ground-truth clusters of G. 3. Match elements of H to clusters of G by assignment function score. Each step presents a number of options, which we detail in the following sections.
Embedding We first embed the discrete word tokens in G and H as word vectors. We experimented with various word embedding models, both without context(Word2Vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014) and FastText (Bojanowski et al., 2017)) and with context (BERT (Devlin et al., 2019), and RoBERTa (Liu et al., 2019)) We found FastText to perform best, and use it for all
4To eliminate zero probabilities, we use Laplace smoothing on all categories before calculating the probabilities, — adding one dummy answer to all categories.
future embedding components. For answers with multiple tokens, we use the average of the FastText embeddings to represent those answer.
Clustering Given the vector representation of the word answers, we experimented with various clustering algorithms including X-means (Pelleg et al., 2000), G-means (Zhao et al., 2008) and hierarchical agglomerative clustering (HAC) (Murtagh and Legendre, 2014). We used the implementation from pyclustering (Novikov, 2019). The parameters used by these clustering algorithms are treated as hyper-parameters and are tuned based on the correlation score as we discuss in section 5.3.
Matching Given the predicted answers, we aim to match the answers to one or multiple ground truth answer clusters. This was also a requirement for ProtoQA (Boratko* et al., 2020), so we leverage the best-performing WordNet matching function from there. We also have embeddings for our answers, we consider embedding-based similarity matching functions. We train a Gaussian regression model for each cluster in the ground-truth answers. The regression takes one answer representation as input, and output is the label of whether the answer belongs to one particular cluster. We also experimented with cosine similarity as a alternative for Gaussin regression model. If an answer matches with multiple clusters we divide the weight evenly among all matching clusters.

heading: 5.3 Validation of the CFC Evaluator
text: In order to validate PROBEVAL’s performance, we compared it with the human evaluation results on two generative datasets, ProtoQA (Boratko* et al., 2020) and CFC. A robust automatic evaluation method should align well with human judgment when given different model predicted answers.
We started by taking a linear combination of the ground-truth distribution P̂g and a uniform distribution P̂u to create diverse distributions that interpolates between the ideal ground truth answers to random noise, details in Appendix A.1.1. However, arguably, the most important area to assess the quality of the evaluator is around answers which are likely to be returned from a model. We thus extend the above distribution by taking a linear combination of the answer distributions of a given baseline model P̂h, the ground-truth distribution P̂g, and a uniform distribution P̂u, with most of the weight assigned to the answers from a baseline model. This method predominantly features
model-generated answers and is defined by the equation: p = zP̂h + w′1P̂g + w ′ 2P̂u, where w ′ 1 and w′2 are coefficients obtained from uniform distributions, emphasizing the blend of model insights with controlled randomness. The intuition behind this equation is that we want to define a new distribution p , which is a weighted combination of distributions P̂h, P̂g and a uniform distribution P̂u. We want these weights to be independent but still place most of the weight on the predicted distribution hatPh. We achieve this by first sampling z in (0.5, 1), and then sampling two other values w1 and w2 in (0, 1). We then computer w′1 = w′1(1−z) w1+w2 , and w′2 = w′2(1−z) w1+w2
. The coefficients are normalized in this process so that z + w′1 + w ′ 2 = 1.

heading: 5.3.1 Dataset
text: We experimented with two datasets, the first one being ProtoQA (Boratko* et al., 2020). Both ProtoQA and CFC dev sets have 100 ground-truth answers and 30 additional human responses that were collected to measure human performance. For each question, in addition to the 130 human responses, we also use the 300 generated answers from the fine-tuned GPT2 model. All of these answers are annotated by expert annotators with cluster matching to the ground-truth clusters (details in Section 5.3.2). We use the GPT2 answers as the prediction set, H. We sample 50 answer sets for each question from H and G according to the sampling procedure mentioned above. ProtoQA dev has 102 questions and CFC dev has 55 questions.

heading: 5.3.2 Gold Evaluator Annotation
text: An essential component in validation is to get human annotations for both clustering and matching. We requested two expert annotators to cluster 100 crowd-worker answers into 8 to 10 clusters independently as ground truth clusters. The innerannotator agreement reached 0.76 when measured by BLANC (Recasens and Hovy, 2011). For model prediction matching in CFC, we take one of the baseline model predictions and employ GPT4 as silver annotation for matching. Two human annotators then verified the GPT4 matching, and reached 0.94 BLANC agreement. For ProtoQA, we used the human-annotated cluster and matching results in the paper (Boratko* et al., 2020)
We use automatic clustering and matching to get DKL(P̂g||P̂h). We can also evaluate the KL for manual clustering and matching, as all answers in ProtoQA have been annotated by human experts
with clusters and assignments. After getting the human and automatic KL values for various sampled answer sets, we use the Spearman correlation coefficients across questions to measure the alignment between automatic and human evaluation.

heading: 5.3.3 Experiment Setup
text: Baseline To compare PROBEVAL with other metrics, we assess the correlation between ProtoQA evaluator and human annotation. ProtoQA Evaluator evaluates multiple answer output as well, and it evaluates model predictions via ranked list of answers. The higher the score it, the better model prediction is. To accommodate ProtoQA score, The correlation is measured between 1 - ProtoQA scores (MaxAnswer@10) and KL divergence with gold clustering/matching annotations.

heading: 5.3.4 Results
text: Our findings, detailed in Table 2 and 3 5, indicate strong correlations between human evaluation and PROBEVAL. Table 2 illustrates the correlation when using WordNet as the matching function with various automatic clustering algorithms. PROBEVAL demonstrates a stronger correlation with human scores when compared to the baseline, ProtoQA evaluator. Table 3 presents the results by using other matching functions that are not implemented in the ProtoQA evaluator in combination with different clustering algorithms. The result showed that whenever humans are involved in the evaluation process, whether in clustering or match-
5Scores are averaged cross 10 runs for setting with human clustering.
ing, the correlation is consistently higher. However, it is worth noting that a fully automatic evaluator with HAC as the clustering algorithm and WordNet as the matching function achieves around 0.7 correlation score that is nearly as good as humans.
Efficiency Analysis We also report the running time of PROBEVAL. When using pre-trained word embeddings to calculate the similarity score, PROBEVAL requires approximately 0.319 seconds per question. When using WordNet as the similarity score, PROBEVAL takes about 49.47 seconds per question. The longer evaluation time associated with WordNet is a direct result of its multistep process. This process becomes particularly time-consuming when the model predictions are long, as it involves the tokenization of the sentence followed by matching each token to ground truth tokens. Our main goal, however, was to come up with an automated evaluation to allow for efficient exploration, and our proposed approach is still much more efficient than human evaluation, which could take up to 10 minutes to perform clustering alone for a single question. Furthermore, human evaluation is expensive and not always feasible. Therefore, PROBEVAL offers an efficient and automated metric for the probabilistic evaluation of diverse answers.

heading: 5.4 Qualitative Analysis

heading: 5.4.1 Visualization
text: In order to qualitatively understand the correlation score gap between ProtoQA evaluator and PROBEVAL shown in table 2, we plotted KL values when
using human clustering and wordnet matching function, shown in figure 4. It is evident from figures (b) and (d) that PROBEVAL exhibits a clear positive correlation with the gold KL score. In contrast, the ProtoQA score tends to be a horizontal line, showing minimal correlation with the gold scores. We hypothesize that ProtoQA is less sensitive to subtle changes in the answer distribution, and we will verify in the next section.

heading: 5.4.2 Prediction Error Types
text: We designed three sampling techniques to mimic common errors in model predictions when the model is tasked to predict multiple correct answers with accurate probability calibration. These sampling methods aim to reveal the performance difference between evaluators when given varying error types ranging from easy to hard to identify. Example samplings can be seen in Figure 5. A list of answer examples are shown in section A.2 in the Appendix.
Missing Answers The first type of error is one of the most common and easily made errors when predicting multiple correct answers, i.e missing
one or more correct answers in prediction. In the implementation, we intentionally delete a random number of categorical probabilities of the ground truth distribution, Pg, from Ph.
Wrong Ranking If the model did not miss any correct answers, we ask the next question of whether it gives the correct ranking. For wrong ranking sampling, the probabilities of categories of the ground truth distribution, PG, are randomly switched so that the answer ranking is wrong.
Wrong Score The last error type assumes the model predicts all correct answers with the correct ranking, however, the scores are not well calibrated. This could be the hardest task but can also be extremely important for high steak domains where the model needs to be well calibrated. For this sampling, the categorical ranking is kept the same but the categorical probabilities of the ground truth distribution, PG, are varied to a random degree.
The results are shown in Table 4. For missing answers, both evaluators achieved correlation scores over 0.7, indicating a consistent alignment with human scores. However, PROBEVAL significantly outperforms the ProtoQA evaluator when the prediction includes incorrectly ranked responses. This demonstrates that the primary discrepancy between the ProtoQA and CFC evaluators arises from the ProtoQA evaluator’s inaccurate judgment when the prediction includes wrong rankings. A particularly notable finding emerges from the wrong score errors, given this error type is extremely hard to iden-
tify, even for humans, PROBEVAL achieves positive correlation scores, while the ProtoQA evaluator exhibits nearly zero correlation. This performance gap can be attributed to the ProtoQA max-10’s limitation of considering only the first 10 correct answers. In contrast, PROBEVAL considers all answers and is able to capture these finer changes, resulting in its ability to evaluate more nuanced differences between model predictions.

heading: 6 CFC Results
text: Given the high correlation of PROBEVAL with human gold KL scores, we employ PROBEVAL in evaluating CFC model performance. All the evaluator parameter are tuned on CFC dev data, then fix the parameters to report results on CFC test data.
Baseline Models In order to generate different answers for the same prompt, we use Nucleus Sampling (Holtzman et al., 2019). We generate 200 sampled answers from the GPT2Large model and 100 answers for the GPT2XL model for each question and treat them as the model prediction. We experimented with temperatures from 0.1 to 1.0, and chose the model parameters with the best dev performance, then reported the test performance.
We conducted experiments using various large language models, employing Hugging Face’s PyTorch GPT2 Large and XL models (Wolf et al., 2019; Radford et al., 2019) and OpenAI’s API for versions 3.5-turbo and 4 (Bian et al., 2023; OpenAI, 2023). Our tests spanned zero-shot, one-shot, and fine-tuning scenarios using the ProtoQA dataset.
In one-shot experiments, we reformatted CFC questions as "[Q]: context, question, [A]" and in-
cluded a sample Q&A pair from the CFC dev set to familiarize the model with the format. For finetuning, we used the ProtoQA pre-trained model and also trained GPT2 Large with a similar task format for 3 epochs on an Nvidia M40 GPU, denoted as GPT2-L FT in our results.
Human Performance In order to get a human performance on this task, we collected 30 additional human responses and evaluated them the same way as a model prediction.
Discussion As shown in Table 5, the best performing model are large models, i.e GPT3.5/GPT4, or fine-tuned GPT2 model. Large models showed significant performance improvement compared to other models, even without fine-tuning data. However, all the model performances still have a large gap compared to human. This indicates the proposed benchmark combined with the probabilistic measurement PROBEVAL is able to identify the performance gap between LLMs and humans, and leaves us ample space to improve the model.

heading: 7 Conclusion
text: In this paper, we assert that commonsense is an implicit probability distribution over missing information, and propose a dataset that aims to evaluate commonsense in this setting via a generative question answering task; moreover, we embrace the probabilistic nature of commonsense knowledge in both the dataset creation and the metric design. We propose a probabilistic automatic evaluation PROBEVAL for evaluating answer distributions that is highly correlated to human judgment. Using this metric, we observe that model performance on our new dataset is significantly worse than human performance, indicating that the task is sufficiently challenging. In the future, we aim to further extend the size of the dataset, both in number of instances as well as answer length.

heading: 8 Limitation
text: We acknowledge that our collected answers are not nearly as perfect for populations around the world and coule be biased towards populations from certain regions, in this case, English speakers in the US. But we argue this framework is one step in the right direction, and we leave the collection for broader cultures for future work.
The collected data also only included a limited scope of commonsense. We believe that extend-
ing the CFC to other commonsense domains, like temporal and social understanding, is important. For example, the social outcome of a particular event could vary significantly depending on culture, and each outcome is valid. The size of the proposed dataset is also limited; however, considering the number of annotations we collected for each question, the size is decent. We also proposed a probabilistic measurement PROBEVAL, which is needed in the era of LLMs to identify model limitations.
We also acknowledge that the proposed evaluator PROBEVAL could be susceptible to adversarial attacks due to the automation and flexibility nature of the evaluation module. It is possible to have a model that achieves high scores but performs poorly for the task. We will explore the combination of symbolic and neural methods to increase the robustness of the evaluator in future work.

heading: Acknowledgments
text: We thank the NLP group at the University of Pittsburgh for their feedback on the early draft of the paper. We thank Joey Hou at Pitt for his help in data annotation. We also thank the anonymous reviewers for their feedback to help us improve the paper. Lorraine was partially supported by the DARPA MCS program at the beginning stage of the work. Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the sponsor.

heading: A Appendix
text: A.1 Validation of the CFC Evaluator
A.1.1 Diverse Sampling We employed a naive sampling stragegy called diverse sampling to emulate the a noisy scenario where the prediction set is a linear combination of the ground-truth distribution P̂g and a uniform
distribution P̂u. The formulation of diverse sampling is p = αP̂g + (1 − α)P̂u where α is drawn from a uniform distribution between 0 and 1. The comparison result between CFC evaluator and protoQA evaluator using diverse sampling are shown in Table 6 and comparison of CFC evaluator under different automatic clustering and matching are in Table 7.
A.2 Examples of sampling methods
Question : A helicopter is being used to tackle the disaster. Who will use helicopters to tackle the disaster?
Ground Truth Answers A selection of ground truth answers and their probability are:
• Cluster 1: “pilot”, p (C1) = 0.5
• Cluster 2: "disaster management team", p(C2) = 0.3
• Cluster 3: "people" p(C3) = 0.2.
In terms of probabilistic distribution, p(cluster 1) > p(cluster 2) > p(cluster 3) for the ground truth.
Wrong Score Sampling The distribution could be:
• Cluster 1: “pilot”, p(C1) = 0.7
• Cluster 2: "disaster management team", p(C2) = 0.2
• Cluster 3: "people", p(C3) = 0.1
The ranking of distribution is kept the same but the actual probabilities of the clusters are much different than that of the ground truth clusters.
Wrong Ranking Sampling The distribution could be:
• Cluster 1: “pilot”, p(C1) = 0.2
• Cluster 2: "disaster management team", p(C2) = 0.3
• Cluster 3: "people" p(C3) = 0.5
The ranking of distribution is now cluster 3 > cluster 2 > cluster 1 which is different from the ground truth ranking.
missing answer sampling The distribution could be:
• Cluster 1: “pilot”, p(C1)=0.7
• Cluster 3: "people", p(C3)=0.3.
Note that Cluster 2 (disaster management team) is missing from the sampled distribution.

heading: Future Work
text: But we argue this framework is one step in the right direction, and we leave the collection for broader cultures for future work. The collected data also only included a limited scope of commonsense. We believe that extend-
ing the CFC to other commonsense domains, like temporal and social understanding, is important. For example, the social outcome of a particular event could vary significantly depending on culture, and each outcome is valid. The size of the proposed dataset is also limited; however, considering the number of annotations we collected for each question, the size is decent. We also proposed a probabilistic measurement PROBEVAL, which is needed in the era of LLMs to identify model limitations. We also acknowledge that the proposed evaluator PROBEVAL could be susceptible to adversarial attacks due to the automation and flexibility nature of the evaluation module. It is possible to have a model that achieves high scores but performs poorly for the task. We will explore the combination of symbolic and neural methods to increase the robustness of the evaluator in future work