ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Sentences
Authors: Yuxin Wang
Xiaomeng Zhu
Weimin Lyu
Saeed Hassanpour
Soroush Vosoughi
Venue: ICLR 2025 Spotlight (2025)
Keywords: implicit language
pragmatics
learnable metric
text evaluation
automatic evaluation
explicit language
Primary area: datasets and benchmarks

Abstract: Handling implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users. Despite its importance, the absence of a metric for accurately measuring the implicitness of language significantly constrains the depth of analysis possible in evaluating models' comprehension capabilities. This paper addresses this gap by developing a scalar metric that quantifies the implicitness level of language without relying on external references. Drawing on principles from traditional linguistics, we define "implicitness" as the divergence between semantic meaning and pragmatic interpretation. To operationalize this definition, we introduce ImpScore, a reference-free metric formulated through an interpretable regression model. This model is trained using pairwise contrastive learning on a specially curated dataset consisting of (*implicit sentence*, *explicit sentence*) pairs. We validate ImpScore through a user study that compares its assessments with human evaluations on out-of-distribution data, demonstrating its accuracy and strong correlation with human judgments. Additionally, we apply ImpScore to hate speech detection datasets, illustrating its utility and highlighting significant limitations in current large language models' ability to understand highly implicit content. Our metric is publicly available at https://github.com/audreycs/ImpScore.

Published as a conference paper at ICLR 2025
IMPSCORE: A LEARNABLE METRIC FOR QUANTIFY- ING THE IMPLICITNESS LEVEL OF SENTENCES
Yuxin Wang Xiaomeng Zhu" Weimin Lyu Saeed Hassanpour!
Soroush Vosoughi"
"Department of Computer Science, Dartmouth College
*Department of Linguistics, Yale University
DDpaarment of Computer Science, Stony Brook University
$ #Deparmeent of Biomedical Data Science, Dartmouth College
ABSTRACT
Handling implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users. Despite its importance, the absence of metric for accurately measuring the implicimesss of language significantly constrains the depth of analysis possible in evaluating models' comprehension capabilities. This paper addresses this gap by developing scalar metric that quantifies the implicitness level of language without relying external references. Drawing principles from traditional linguistics, we define "implicitness" as the divergence between semantic meaning and pragmatic interpretation. To operationalize this definition, introduce IMPSCORE, a reference-fiee free metric formulated through an interpretable regression model. This model trained using pairwise contrastive learning on specially curated dataset consisting of (implicit sentence, explicit sentence) pairs. We validate IMPSCORE through a user study that compares its assessments with human evaluations on out- out-o--distriutton data, demonstrating its accuracy and strong correlation with human judgments. Additionally, we apply IMPSCORe hate speech detection datasets, illustrating its utility and highlighting significant limitations in current large language models' ability to understand highly implicit content. Our metric is publicly available at https://gihub..oo//aut httph//....oouuueee////psoo/oo//
INSTRUCTION
Implicit expression of opinions or attitudes is pervasive interpersonal communication, presenting unique challenges and opportunities for Natural Language Processing (NLP). As large language mod- els (LLMs) have scaled in parameter size, their capabilities in understanding implicit information have improved significantly, impacting tasks like automatic content moderation and AI safety (Hartvigsen et al., 2022; Kim et al., 2022; Wen et al., 2023).
Despite these advancements, critical foundational limitations remain, particularly in the nuanced understanding and measurement of implicit language. Current benchmarks often employ subjective criteria for collecting implicit data, which complicates the assessment of data quality and undermines robust evaluation (Jahan & Oussalah, 2023). Furthermore, the prevalent binary labeling of data as ei- ther implicit or explicit does not align well with the complex ways humans use and perceive language, obscuring the true depth of LLMs' understanding capabilities (Anwar et al., 2024). Addressing these challenges requires a more refined, granular metric for implicitness that can evaluate language on continuum rather than in binary terms.
This paper introduces a novel, reference-fres implicitness of any English sentence. We define "implicitness" as the divergence between the semantic meaning and the pragmatic interpretation of sentence, where we use "semantic" to describe the literal meaning of sentence, which is function of the literal meaning of the words and phrases that compose the sentence and its syntactic structure (Partee, 1984). We use "pragmatic" to describe the intended meaning that can be inferred from context and usage. This definition is inspired by foundational work in theoretical linguistics (Grice, 1975; Carston, 2009). For instance, the sentence "I' finding bit hard concentrate" typically communicates difficulty in concentration
IMeric-IMPSCORE -to quantitatively assess the

---

Published as a conference paper at ICLR 2025
Sentences
implicitness level?
They bothh soon be compleinnn rigiiiiiggg about bodl Food
pragmatically dose
Food
[ ve had better mede domaneree
The food here terrible
better cisewhrere
complaiiing
I'm finding it bit hand concuntrots
You
loudly
You ane tellinn too louckyy
Ein finging a hard concontrute
They seam be feil tilking about differmnt thing.
pragmatically nott doose"
I've had better neels dicodeere.]
I'm Finding bit hand concantrate.
An illustration of how humans typically perceive sentences with different implicitness levels. Sen- tences directly expressing their pragmatics are generally more explicit (indicated by blue arrows), and it is easier to distinguish the implicitness levels between sentences with close pragmaties (expressing similar meaning).
Figure 1:
but can pragmatically imply a complaint about the loudness of an addressee's speech, demonstrating a significant semantic-pragmanl divergence. In contrast, "You are talking too loudly" is more explicit, as the spoken content directly conveys the intended meaning. These are illustrated in Fig.
Basing this definition, we designed an interpretable regression model and trained reference-free learnable metric. IMPSCORE, to measure implicitness scores for English sentences. As shown in Fig. 2, for each input sentence, IMPSCORE first uses a text encoder to generate an embedding, then maps its semantic and pragmatic features into two independent diw-dimensional spaces. The implicitness score is calculated by transforming these features into the same space and applying distance metric to measure their divergence. Since our goal is to measure the implicitness of an arbitrary sentence, not just among sentences of the same intended meaning, IMPSCORE includes an additional indicator to measure pragmatic distance between sentences to aid the interpretation of their implicitness scores. This is illustrated in Fig. 1, where is more difficult to compare implicimess level between sentences with dissimilar pragmatics (red box). IMPSCORE is trained via pairwise contrastive learning, where we collected and processed 112, 580 (implicit sentence, explicit sentence) pairs from various data sources, half of which are pragmatically close (i.e., positive pairs) and half are pragmatically distinct (i.e., negative pairs). IMPSCORE is tasked to generate higher implicitness scores for implicit sentences in each pair and smaller pragmatic distances for positive pairs.
IMPSCORE'S training achieved a high accuracy rate, exceeding 95%, in both discerning levels of implicimess between sentences and differentiating degrees of pragmatic closeness among sentence pairs. To evaluate the effectiveness of our trained IMPSCORE on ouitt--distribution (OOD) data, we implemented user study designed to rank implicitess levels and measure pragmatic closeness. The results from this study indicated that IMPSCORE's predictions achieve reasonable accuracy and show positive correlation with human judgments. Notably, IMPSCORE's performance consistently aligns with the Law of Large Numbers, demonstrating that as the number of observations increases, the average of the results becomes more accurate and stable across diverse datasets.
Further exploring the utility of IMPSCORF, we applied it in the context of hate speech detection across two main applications: (1) was utilized evaluate popular datasets, analyzing their levels of implicitness and pragmatic diversity. Our quantitative analysis confirms the model's capability to effectively discern varying degrees of implicitness. (2) We assessed the performance of three advanced language models-GPTT-4- data exhibiting different levels of implicitness. The findings revealed a notable degradation their performance as the implicitness level intensified, highlighting a critical bottleneck in these models' ability to comprehend highly implicit sentences, a challenge often obscured by their generally robust performances standard hate speech detection benchmarks.
1-B--...--8---stttte and OpenAl AI Moderation- aggiinst
The lightweight nature of IMPSCORE, coupled with its capability for rapid inference, enables efficient and effective quantitative comparisons across extensive text corpora. These attributes position IMPSCORE as a preferable alternative to more costly LLM evaluators, which often raise concerns regarding their operational transparency, faithfulness, and computational efficiency.
The key contributions of this paper are:
Introduction of a novel metric, IMPSCORE, which offers refined interpretation of "implicitness." effectively quantifying this aspect of language in a learnable and operational manner.
Construction of a comprehensive dataset comprising 112, 580 (implicit sentence, explicit sen- tence) pairs from diverse sources, tailored specifically to train IMPSCORE.
Execution of a user study aimed at validating IMPSCORE's performance on OOD data, which confirmed the model's adeptness at accurately predicting sentence implicitness levels.
2

---

Published as a conference paper at ICLR 2025
pragmaticss difference
test embeedding
Sie delivered a hoartfel, performance.
AAA
Space Transfomation
Pragmatic Projection
Learning Objectives
Siee song her beart out.
Text Encoder
pragmatiic space
m & Priss
regative sampling
Sie song out.
nepetov
Semantic Projection
1 Temmpleted the project ahead the doodling.
implicimess difference
semantic space
semantic space
Figure 2: An overview of the training process of IMPSCORE. The sentence in blue is an implicit sentence, while the sentence in green and the sentence in red explicit sentences in the positive and negative pairs, respectively. Colored A and markers denote the feature embedding points of sentences in corresponding colors. 1ı, 1z. and Is denote the implicitness scores these sentences. APt and AP denote the pragmatic distances of the positive and negative pairs, respectively. 7/1 and ?22 are model hyperparameters.
Application of IMPSCORE in the realm of hate speech detection, revealing significant insights into the limitations of contemporary language models in processing highly implicit language.
RELATED WORK
Automatic NLP Metrics on Text Features Many rule-based and machine learning -baedd NLP metrics have been proposed to evaluate text quality and features, in either reference- based reference- free manner. Representative works include those in machine translation (Papineni et al., 2002; Zhang al., 2020), coherence and cohesion analysis (Barzilay & Lapata, 2008; Pradhan et al., 201 10. and readability assessment (Flesch, 1948; Lee & Vajjala, 2022). Recently, there has been work delving into quantifying unique text characteristics, such as simplicity (Cripwell al., 2023), politeness (Fu et al., 2020), and factual precision (Min et al., 2023). However, to the best of our knowledge, there is no existing automatic metric on evaluating implicitness of text. Directly applying other metrics is also ill-suited to address this problem.
I Linguistic Measures of Implicitness Theoretical linguistics has long examined how exactly meaning is conveyed through phenomena such presupposition (Beaver, 1997) and implicature (Grice, 1975), without providing a quantitative measure of the implicitness of sentences. Among the few relevant works, Vallauri & Masia (2014), proposed a system of quantification indexes to measure implicit causality bias, focusing on how specific verbs influence causal attributions by calculating bias scores for each verb. Garassino et al. (2022) qualitatively assessed participants abilities detect and interpret different types of implicit content. However, neither study addressed calculating implicitness scores for entire sentences. Additionally, their results heavily rely human annotations, making them difficult to reproduce automatically.
FRAMEWORK OF IMPSCORE
Fig. illustrates the overall training process of IMPSCORE, which employs pairwise contrastive learning and computes an implicimess score I for each sentence. Each step is detailed below.
3.1 POSITIVE AND NEGATIVE PAIRS PREPARATION
The input to IMPSCORE consists a positive pair and corresponding negative pair. The positive pair contains an implicit sentence and pragmatically close explicit sentence 82, while the negative pair uses the same sj but replaces $2 with a pragmatically distant sentence s3. In this design, $1 serves as anchor, ensuring pragmatically close-far far relationship between the two pairs for IMPSCORE to leam. The methodology for constructing these pairs is detailed in $4.
3.2 PAIRWISE LEARNING OF IMPLICITNESS
After inputting the prepared positive pair (81, 82) and negative pair (81, 83) into IMPSCORE, IMP- SCORE generates text embeddings e e2, and eg E Rd for 81, 82, and 83 respectively using a Text Encoder fe. Here, d is the embedding dimension and 0 denotes the parameters of Text Encoder:
e (1,233 f0(8(12233)-
3

---

Published as a conference paper at ICLR 2025
Then, IMPSCORE extracts the semantic features and pragmatic features from the text embeddings. We use two projection matrices to map text embeddings into a pragmatic and a semantic space independently
Pragmatic Projection matrix W, R and Semantic Projection matrix W, Raxi I is the dimension of semantic and pragmatic features which much smaller than d. The pragmatic feature h" and semantic feature h for $1, $2, and 83 are derived via below multiplications:
" (1.2.3) e(1.2,3) W p
h (1,2.3 - e(1.2,3 W
(2)
To measure the implicitness of each sentence, which is interpreted as the divergence between its pragmatics and semantics, we then put the two features in the same space. We simply transform the pragmatic features into the semantic space using a Space Transformation matrix W, e Rix (other transformation approaches are also explored in $5.3.):
h (1.233 h (1223) W..
(3)
where hij hi, há E R are transformed pragmatic features of $1, $2, and 83 in semantic space. Then, the level of implicitness I of each sentence is calculated as the Cosine distance of the transformed semantic features and original semantic features. Employing Cosine distance as the metric ensures that the implicitness the model parameters.
remains within standardized range of [0, 2), regardless of changes in
[122333 = 1 CosineSin (12,3)))
The objective for this pairwise learning of implicitness is to maximize the implicitness level of 81+ and minimize the levels $2 and s3. We define the implicitness loss Limp as below, which adopts Pairwise Margin-Bass Based Ranking Loss:
(8, max(0, 72 (7 T2)),
(5)
Limp (81, 83) max(0, 71 - (71 Ia)),
where Vi is hyperparameter with range [0, 2). By minimizing the two losses, the model is encouraged to compute implicitness level for 81 at least "/1 higher than that of #2 and 83.
3.3 PAIRWISE LEARNING OF PRAGMATICS
As outlined in the introduction, we also want to learn an indicator of the pragmatic distance of sentences. Thus, we design another learning objective. Specifically, we extract the pragmatic features from sentences 81, 82, and 83 and employ the Euclidean distance to measure their pragmatic distances, denoted as APt for positive pairs and AP for negative pairs:
AP+ = || b
AP = |b h7ll2.
We aim to ensure that sentences in positive pairs exhibit smaller pragmatic distances compared to those in negative pairs. Similar to the implicitness loss, we design a pragmatic loss, Eprag, where 72 is a hyperparameter with range [0, oc).
Lerag (81, 82, 83) max (AP - AP*)).
3.4 INTEGRATION AND MODEL EMBODIMENT
The final loss function is a weighted sum of Cimp and Lprag, where a, a predefined factor ranging from [0, oc), dictates the relative importance assigned the learning of pragmatics.
imt ((31, $2), (81, ss)) Limp($1, 82) + Cimm((11 8s) + a Lprag (8,,82, 82)
82,
(8)
The overall learning objective Te is to minimize Lfinat over all the training set D in mini-batch approach, where θ (0, Wp. W., s, W.) all the parameters IMPSCORE:
Je argming
Σ
Liinnt ((11, $2), (81, 83)).
For model implementation, we embody the Text Encoder with Sentence-BERt (Reimers & Gurevych, 2019) with version a -mpnet-- connet-bbasee -base-aeeeeee ! We prioritize Sentence-BHER over other text encoders, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), because Sentence-BBer Dirence--ERTT/ is
model htm!

---

Published as a conference paper at ICLR 2025
specifically finetuned to derive semantically meaningful sentence embeddings for sentence pairs, which we believe is beneficial for our task. The maximum imput sequence length of Sentence- 384 and its output dimension is 768. We do not normalize the output embeddings. For the weights of matrices Wp. p. Ws, and W, they are initialized with Xavier Initialization (Glorot & Bengio, 2010):
is
W We Ul- ve Vo
W, ~ ~1 va
La
Vett va+tll
V2
4 TRAINING DATA CONSTRUCTION
To train IMPSCORE, we require dataset of (implicit sentence, explicit sentence) pairs. However, the best of our knowledge, no available datasets meet this criteria. Thus, we collected and processed the training data ourselves. We explored range of existing datasets that indicate implicitness across various NLP tasks, including Textual Entailment/Nattral Language Inference, Implicit Hate Speech Detection, Sentiment Analysis, Irony and Sarcasm Detection, and Discourse Relation. Details on dataset construction and processing are provided in Appendix A.2.
Specifically, for implicit hate speech datasets, some already provide explicit explanations for implicit samples, which can be directly used to construct (implicit, explicit) pairs. For others, we prompted GPT-3.5 with implicit hate speech to generate corresponding explicit ones. Appendix A.1 presents the GPT prompting instructions. This approach was also applied sentiment analysis datasets. Textual Entailment datasets, which feature sentence pairs (premise, hypothesis) with entailment relationship "premise hypothesis", can be useful as implicit sentences often entail explicit ones. However, despite the entailment relationship between premise and hypothesis, not all of them necessarily demonstrate different implicitness levels. For example, in the pair ("I drove my car yesterday." ", "I have a car.") telling which sentence is more implicit is challenging without additional context and knowing their intention, as both sentences appear explicit. We also present a quality assessment of the constructed data from a linguistic expert in Appendix A.2. In total, we constructed 56, 290 pairs from 13 datasets.
Positive and Negative Pairs Generation We treated all 56, 290 pairs constructed above as positive pairs, where both sentences are pragmatically aligned. Negative pairs were created by replacing the explicit sentence in a positive pair with the explicit sentence from another randomly chosen positive pair within the same dataset. This inner-dataset replacement avoids inconsistencies in implicit-cxplici relationships that could arise from the different labeling standards across datasets. Tab. presents the statistics of the training data. Examples of positive and negative pairs are in Tab. 7 in Appendix A.2. We also provide samples of our training data in the Supplementary Material.
Table 1:
Statistics of training data. The average length is calculated as the number of characters in a sentence.
The number in parentheses is the standard deviation.
# Positive Pairs
# Negative Pairs
Avg. Length (Implicit Sentences) 109.5 (+70.7)
Avg. Length (Explicit Sentences)
56, 290
56, 290
66.94 (+-45.5
5 TRAINING OF IMPSCORE
Recall the inference process of IMPSCORE: each imput point is in format of {(s1, s2), ss)} where $1 is an implicit sentence, and 82 and $3 are explicit sentences. ($1, 82) denotes the positive pair and (si, $3) for the negative pair. During inference, IMPSCORE computes implicitness scores for 81, 82, and $3, meanwhile, calculates the pragmatic distances for (81, 82) and (81, 83).
5.1 EXPERIMENT SETUPS
Evaluation Metric We record two results of IMPSCORE's performance during test: 1) Impliciteess Accuracy the percentage of IMPSCORE successfully predicting higher implicitness scores for the implicit sentences across all pairs, regardless of whether they are positive or negative. 2) Pragmatics Accuracy - the percentage of IMPSCORE successfully predicting smaller pragmatic distances for positive pairs compared to their corresponding negative pairs.
Hyperparameter Setting We set the implicitness margin 71 to 0.5, the pragmatic distance margin 72 to 0.7, and a to 1.0. The dimension of two spaces is 64. Adam optimizer is used for gradient descent. The data randomly split into training, validation, and test sets an 8 ratio. We train for 30 epochs, validating after each epoch, and save the model with the best validation performance on Implicitness Accuracy. The parameter size of IMPSCORE is about 105 MB.
5

---

Published as a conference paper at ICLR 2025
5.2 TRAINING PERFORMANCE
Fig. 3 presents the performance of IMPSCORE on test set. The table in left panel shows the Implicitness Accuracy and Pragmatics Accuracy, along with the average implicitness scores for sentences (T and fe and the average pragmatic distances for pairs (AP+ and AP-). IMPSCORE achieves over 95% accuracy in both metrics. The center violin plot illustrates the distribution of implicitness scores of implicit sentences (blue), and for explicit sentences in positive pairs (green) and negative pairs (red). IMPSCORE effectively distinguishes between implicit and explicit ones, demonstrating distributive bipolarity. The gaps between their average scores are larger than V1 = 0.5. The right violin plot presents the distribution of pragmatic distances of positive pairs (green) and negative pairs (red). IMPSCORE effectively captures closer pragmatics in positive pairs. Although the output embeddings of IMPSCORE were not normalized, the results suggest that the range of pragmatic distances remains constrained within narrow section. This is influenced by the properties of embeddings produced by Sentence-BERT. A breakdown of IMPSCORE's 's testing performance on different data sources is in Appendix A.
5.3 ABLATION STUDY AND HYPERPARAMETER SENSITIVITY
In this subsection, we examine the effects of different model design and hyperparameter settings. For model design, we focus on two aspects: distance metric and space transformation direction. We vary the metrics for calculating implicitness scores and pragmatic distances in (Cosine, Euclidean] ) distances, resulting in four combinations. Besides, we change the space transformation direction in 1) transform pragmatic features to semantic space (p~>s;; 2) transform semantic features to pragmatic space (s->p); and 3) transform both pragmatic features and semantic features to third space (p.s-thirdd) using as the follows. W and W are two independent matrices, and h and h are the transformed pragmatic and semantic features in the third space.
41,2,33 = 1/ [1233) (112333) h tp (1223) = h. (12233) W h (1233)
h (12333) W?
(10)
This leads to 3 scenarios. Totally, we have 3 x 4 = 12 variations of IMPSCORE. For each variant, we train it three times using the same setting in Sec. 5.1 and calculate the average Implicitness and Pragmatics Accuracy. Their results are summarized in Tab. 2. We observed that using the Cosine metric for calculating implicitness scores yields steadily better results than using Euclidcan. However, changes on the metric for calculating pragmatic distances and the space transformation direction do not significantly affect the model's 's performance.
To assess the impact of hyperparameters, we varied y from (0.1, 0.7, 1.0, 1.5), "/2 from (0.5, 0.7, 1.0, 1.5, 2.0}. and a from {0.5, 1.0, 1.5, 2.0}. We use the original model design of IMPSCORE (i.e. Cosine Euclidean and p->s). For each hyperparameter setting, we run IMPSCORE 3 times and report the average Implicitness Accuracy. We fixed a and plotted 3D graphs of 71 and 72 on Implicitness Accuracy, smoothing the surface with interpolated points. The result when a 1.0 is presented in Fig. 4, and results of a {0.5, 1.5, 2.0} are in Appendix A.4. We observe that ? = 0.5 yields the best performance compared to other values. However, IMPSCORE did not show obvious fluctuations to variations in 7/ and within the given ranges. We think the insensitivity
Distribution of implicimees
Distribution Pragmatic Distance
Implicit Acc.
Pragmatic Acc.
0.952
0.962
Average imp
Average (rrp
AP+
1.296
0.514
Average Apt
Average AP"
0.685
1.675
is regatine
0.0
0.5
1.0
1.5
2.0
0.0
2.0 Pragmatic Distance
1.0
Implicitness Score
Figure 3: Left panel: Detailed results of IMPSCORE on the test set. time and daap denote the implicitness score of implicit sentence and explicit sentence (higher indicates more implicit), and APt and AP denote the pragmatic distance of positive pair and negative pair (higher indicates pragmatically farther). Center panel: The distribution of implicitness scores of implicit sentence, explicit sentence in positive pair, and explicit sentence in negative pair. Right panel: The distribution pragmatic distances of positive pairs and negative pairs.

---

Published as a conference paper at ICLR 2025
Table 2:
Ablation study results of different model variations of IMPSCORE. The number in the left of in- dicates the Implicitness Accuracy, and the right number indicates the Pragmatics Accuracy. Best Implicitness Accuracy are highlighted in bold. IMPSCORE robust to different Space Transformation methods.
0350
Accumay
0,00
Accuracy (imp/prag)
Space Transformation
Metric (impx ppaag)
p-28
S-tp
Cusine x Euclidean
00.953/09622
0.947/0.962
0.946610.93
Cosine xCosine
0.952/0.973 0.953/0,9617
0.941700.971
12 14
Figure 4: Hyperparameter sensitivity of 71 and "22 on Implicitness Accuracy when a = 1.0. Highest point is marked in red dotted box.
Faclidean x Cosine
0.92770.964
0.928/0.974
0.929/0.974
Eaclidean x Enclidean 0.926/0.963
0.927/0.962 0.924/0.963
to 72 is due to the lack of normalization in the output embeddings and the use of Euclidean distance for calculating pragmatic distance, which enables the model generate wide range of scores to optimize the loss function effectively.
Experiments in the following sections all utilized trained metric, which computes implicitness score 7 for each input sentence and an additional pragmatic distance A for inputted sentence pairs.
6 CORRELATION WITH HUMAN JUDGMENTS ON OOD DATA
In this section, we applied the trained IMPSCORE and conducted user study on out-of-distribution (OOD) data to examine the generalization ability of it.
6.1 OOD DATA PREPARATION
We prompted GPT-4 with 20 different topics to generate 4 sentences with varying implicitness levels for each topic (most explicit, explicit, implicit, and most implicit). The sentences in each topic formed a group (i.e., G.). We, along with linguist, then manually refined and verified the sentences each group to ensure they manifest distinct levels of implicitness, allowing us to assign accurate gold ranks (ranked from most explicit to most implicit). Here, we show the user study procedure and results for 10 topics in Tab. 3. Results on the remaining 10 topics are in Appendix A.13. The detailed sentences for each group in Tab. are in Appendix A.3.
6.2 USER STUDY DESIGN
We conducted two tasks in the user study. The first task, Ranking. is to rank four sentences within a given topic by their levels of implicitness. consists of 10 topics (G₁ G1o Tab. 3). The second task, Choice, is to select the sentence pragmatically closest to reference sentence from three options. We provided a reference sentence labeled as "cxplicit" in implicitness level. The three options included one sentence from the same topic group labeled as "most implicit," and two sentences also labeled as "most implicit" but from different topic groups. The Choice task also contains 10 questions. See Tab. 9 in details. To avoid fatigue of participant and ensure user study quality, we divide these questions into Set and Set 2², where each set contains 5 Ranking questions and 5 Choice questions. We built user study website on Gorilla and recruited 10 English speakers on Prolific as participants We divide and assign 5 participants to each set of tasks, so each question has 5 human participants. For testing IMPSCORE, in Ranking questions, we compute implicitness scores for 4 sentences and rank them accordingly. In Choice questions, we input each option sentence along with the reference sentence into IMPSCORE and calculate their pragmatic distance. The option with smallest pragmatic distance are chosen as the answer. We report Kendall's ' Tau (r) and Spearman's 's Rho (p) two metrics for comparing the correlation two rankings - in Ranking task, and report Accuracy in Choice task. Kendall's Tau focuses pairwise concordance, while Spearman's 's Rho measures the monotonic relationship between rankings. Both 7 and are in range of [-1, 1]. where indicates perfect positive correlation, -1 perfect negative correlation, and 0 no correlation. The calculation of Kendall's 's Tau is in Eq. 11 in Appendix A.3. Screenshots of the user study questions are in Appendix A.10.
2 Please see the topic design, user study platform, and user participant recruitment details in Appendix A.5. Set I consists of the first 5 Ranking questions in Tab. 8 and the first 5 Choice questions in Tab. 9, while Set 2 consists of the rest.
7

---

Published as a conference paper at ICLR 2025
Table 3: 10 topic groups of OOD data in user study. Design of these topics is in Appendix A.5.
Table Inter- puter-patiicipntt Kendall's Tau (7) correlation results in Ranking task for user study.
Group
Topic
Human1
1:22
6.00
9:40
G1
Ending Relationship Critiquing colleague's 's work Dealing with a rehellious child Giving the boss feedback about her behavior Giving an unfavorable performance review of ballet
Humen2
1.23
1.00
6.00
G2
Human3
Set
Ga
Human4
947
0.99
0.90
Human5
Gs
Impšccre-
Human1
0.67
0.73
1.0
8:60
G.
Asking roommate
out Handing off a difficult project to a colleugue
Human2
1.39
0.57
835
Gr
Human3
Disliking John's personality
Humand-
S
372
0.33
Go
Decline a friend's 's party invittiion
HumanS-
0.75
6:47
1.00
1000
Gío
Remind a roommate to clean the kitchen
ImpScore
Human1
Human2
Human3
Humand
Human5
Imps
Human1
Humanz
Humans
Humand
Human5
ImpScore
Table 5: Results of participants in Ranking and Choice tasks compared with gold ranks/answers in user study. T and stand Kendall's ' Tau and Spearman's 's Rho. Numbers are the average performance of each participant on 5 questions of each task. f indicates that higher results are better.
Tasks
Human
Human 2
Human
Human 4
Human 5
IMPSCORE
Avg. rf
Avg. p² Avg. 71
Avg. p
Avg. +f
Avg. pf
Avg. yf
Avg. p²
Avg. 7 Avg. p
Avg. rf
Avg. p²
Ranking (Set 1)
0.53
0.64
0.93
0.96
0.93
0.96
0.73
0.80
0.93
0.96
0.73
0.80
Ranking (Set 2)
0.80
0.88
0.73
0.72
0.93
0.96
0.67
0.80
1.00
1.00
0.80
0.88
Accuracy
Accuracy
Accuracy
Accuracy
Accuracy
Accuraey
Choice (Set 1)
0.80
0.80
0.80
0.80
0.80
0.80
Choice (Set 2)
1.00
1.00
0.80
0.60
1.00
1.00
6.3 OOD TEST RESULTS
The test results of human participants and IMPSCORE on OOD data are in Tab. 4 and Tab. 5, where Tab. 5 details the т, p. and Accuracy in comparison to gold labels both tasks; Tab. 4 shows the inter- piher-participantt 7 on the Ranking task. Their predictions on each question of both tasks are in Appendix A.3. We observe in Tab. 5 that IMPSCORE achieves satisfactory performances both Ranking and Choice tasks compared to the gold labels. Although it does not achieve 100% correctness match the top performance among human participants, there is no notable performance gap between IMPSCORE and human participant group. The inter- pitrrppatiiiiaant 7 results in Tab. 4 show that IMPSCORE has good correlation with human judgments, with no significant difference from human-human correlation results. Interestingly, sometimes IMPSCORE predicts the same wrong answer most human participant, suggested Qs of Choice task in Tab. 9. Results on p correlation is presented in Fig. 9. Additionally, we have two exciting observation from IMPSCORE's results:
(m) sentences in each Ranking question correctly, which can be observed from the results in Tab. 8. Though, IMPSCORE sometimes misranks sentences with minimal differences in implicitness, it performs perfectly when distinguishing between sentences with significant implicitness gap.
Observation 1: IMPSCORE ranks all "most explicit" and "most implicit"
Observation 2: IMPSCORE demonstrates consistent and reliable predictions over collections of data. For Ranking task, we averaged the implicitness scores of sentences across 10 questions by their implicitness level. The results are presented in Tab. 10. The average scores distinctly reflect the consistent variance across these four levels of sentences. This illustrates that IMPŠCORE's performance aligns with the Law of Large Numbers, where despite occasional errors in individual predictions, the average results across many cases converge to the expectation.
6.4 EFFECTIVENESS OF PRAGMATIC AND SEMANTIC EMBEDDINGS
In this subsection, we analyzed the effectiveness of the latent pragmatic and semantic embeddings (circles and triangles in Tab. 2) of OOD data in this user study. We retrieved the learned pragmatic and semantic embeddings, and utilized t-SNE to visualize their distributions in 2-D spaces, as displayed in Appendix A.6. The pragmatic distribution in Fig. 11 shows that sentences within the same topic group (expressing the similar meaning) tend to cluster closely, suggesting the effectivvneess of their pragmatic embeddings. The semantic distribution in Fig. 12 shows a more dispersed pattern among sentences within the same group. This can be attributed to the diverse semantics used in sentences when expressing the same meaning, which we further illustrate with case study in Fig. 13. We
8

---

Published as a conference paper at ICLR 2025
1.0
Michne-greecddd
Tta
12-
Litent Levidiaon
6.5
RouGen
Davidaor
Advernettl
Pragmark Dietance
26
Figure 5: Average implicitness scores with standard deviations of different hate speech datasets.
Figure 6: The pragmatic distances of sentences within cach datasct.
selected sentences from topic groups analyze their semantic similarities and differences, explaining why some sentences are positioned farther apart than others.
7
APPLICATIONS WITH IMPSCORE
7.1
IMPLICITNESS LEVELS OF HATF SPEECH BENCHMARKS
The first application is evaluating the implicitness level of different Hate Speech datasets. We selected popular datasets as follows. Note that some already have implicit speech labels.
ToxiGen (Hartvigsen et al., 2022y: A large-scale machine-generatee dataset for hate speech detection. It contains 125, 672 hate speeches which are all claimed implicit.
Latent Hatred (EISherief et al., 2021): dataset for understanding implicit hate speech. contains 8, 189 Twitter posts, among which 7, 100 are labeled as implicit hate speech.
DGHSD (Vidgen et al., 2021): A dynamically generated dataset to improve online hate detection. It contains 22, 175 hate speeches, among which 3, 439 are implicit ones.
Adversarial (Ocampo et al., 2023): A syncretic dataset collecting messages from multiple sources. It contains 31, 830 hate speeches, among which 24, 823 are implicit ones.
SBIC (Sap et al., 2020): A dataset crawling toxic posts from multiple websites like Reddit and Twitter: It contains 24, 019 offensive samples in the training.
Davidson (Davidson et al., 2017): dataset crawled from Twitter using lexicon matching. contains 20, 620 samples that contain hate speech or offensive language.
We processed all their samples through IMPSCORE to infer implicitness scores. For implicit samples already labeled, we recorded their scores separately. Fig. shows the average scores and standard deviations. Implicit samples exhibit higher implicitness levels compared to the corresponding entire datasets. We can see that the Adversarial dataset has the highest implicitness. The Davidson dataset, collected using a prompting GPT for implicit hate speech detection, also has low implicitness, potentially indicating the limited ability of LLMs generate language with implicitness comparable to human levels. Detailed score distributions for each dataset are shown in Fig. in Appendix A.7.
shows very low implicitness level. ToxiGen, generated by
We also examined the pragmatic diversity in each dataset. We randomly sampled 2, 000 distinet sentence pairs from each dataset and calculated their pragmatic distances. The distributions of these distances shown in Fig. 6. The Davidson dataset exhibits notably lower pragmatic distance compared to others, indicating the least diversity in sentence pragmatics.
These findings demonstrate how IMPSCORE can be used compare the properties of different datasets, offering valuable insights for selecting appropriate benchmarks in experimental studies.
7.2 LANGUAGE MODELS CAPABILITIES ON DIFFERENT IMPLICITNESS LEVELS
In the second application, we aim to explore LLMs' understanding capabilities language with different levels of implicitness. We selected two state-of-the-art LLMs: GPT-4-Turbo and Llama- 3. -.--Sut-nttruct, along with a content moderation tool - OpenAl Moderation". We chose three mentioned datasets: ToxiGen, DGHSD, and Adversarial, as our test sets. The first two demonstrate a wide spectrum of implicitness, while Adversarial predominantly contains samples with higher levels of implicitnsss, as depicted in Fig. 14 in Appendix. We divided their samples into subsets based on their implicitness scores, ranging from [0, 0.25) to [1.75, 2/. with each interval spanning
hhttps://platfae oow.aaaraooopooo..oo.o..moaaa.a. openai www/dccc///c/ccccccccccccccccccc des htte:/gguddmmmmmeaaiiiiio /omd/mateeerooooommmmoommo

---

Published as a conference paper at ICLR 2025
100%
Datection Accurncy
OpenAl Moderation
Turbo
Llama-3.1-88 1-8----st--tttt
80%
70%
60%
50%
40%
30%
20%
ToxiGen
10%
[0.00, 0.25)
[0.25, 0.50)
[0.50, 0.75)
[0.75. 1.00)
[1.00, 1.25)
[1.25, 1,50)
[1.50, L.75)
[1075,
Implicitness Score
100%
Delection Accuracy
90%
80%
70%
60%
50%
40%
OpenAT Moderation
GPT.4 Tumb
30%
20%
DGHSD
10%
0%
[0.00,
0.25)
[0.25, 0.50)
[0.50, 0.75)
[0.75, 1.00)
[1.00, 1.25) Implicimees Score
[1.25, 1,50)
[1.50, L.75)
[1.75,
Figure 7: Hate speech detection accuracy of LLMs sentence samples with different levels of implicitness. Blue columns indicate the distribution of samples among different ranges of implicitness.
0.25. The task is binary classification where present hate speech to LLMs and ask them to determine whether it is problematic in zero-shot prompting scenario. The prompt format is presented in Tab. 15 in Appendix A.8. For OpenAl Moderation, applied a lenient criterion by checking if it flags content as potentially harmful, without necessarily categorizing it as hate speech. We report Accuracy, the proportion of test samples that LLMs successfully identify as problematic.
The detection accuracy, displayed in Fig. 7 for ToxiGen and DGHSD and Fig. 16 in Appendix for Adversarial A.9, reveal that detection accuracy declines as the implicitness level of sentences increases. In ToxiGen and DGHSD, accuracy for the three LLMs drops consistently and they have very low detection success rates in high implicitness ranges. Adversarial, GPT-4-Turbo and Llama- 3.1-8B-Instrue show performance improvement on range [1.75, 2]. We attribute this to the samples in this range being very similar, reducing diversity and potentially biasing the results, as demonstrated in Appendix A.9. Overall, the results accord with our expectation, underscoring the effectiveness of IMPSCORE. The clear downward trend in performance highlights a significant limitation in LLMs' understanding of highly implicit sentences. This problem is statistically undiscovered in the past and also overlooked by the high average benchmark performances (Jahan Oussalah, 2023).
Additionally, applied IMPSCORE to assess the generation capabilities of different LLMs, as detailed in Appendix A.15.
8 CONCLUSION AND DISCUSSION
In this paper, we addressed the problem of quantifying the implicitness level of language. By providing an interpretable definition of "implicitness," " we developed a metric capable of computing implicitness scores for sentences. IMPSCORE'S performance on out-of-distribution test data demonstrates satisfactory accuracy and alignment with human judgments, which highlights its generalizability.
However, this research has limitations that suggest directions for future work. First, while the dataset comprising 112,55 580 sentence pairs, expanding it could improve IMPSCORE 's accuracy and generalizability. To do so, beyond expanding the search for existing datasets, leveraging LLMs to generate synthetic data while maintaining a coherent speech style is a promising approach. Moreover, incorporating broader range of implicitness types and multilingual data would enhance IMPSCORE 's versatility. Second, IMPSCORE 's evaluation, though comprehensive, could benefit from further user studies and wider range of inventory questions. Third, as the dataset grows, employing more advanced text encoders like SimCSE (Gao et al., 2021) may further improve performance. To this end, we encourage the research community to explore and refine IMPSCORE 's capabilities.
Looking forward, IMPSCORE has broad potential in NLP. It could assess implicitness in machine- generated text, aiding both retrospective analysis and model training. Specifically, IMpSCORE could serve as tool in reinforcement learning setups, providing a reward signal that enhances language models' ability to handle nuanced linguistic constructs. Additionally, IMPSCORE'S capacity for screening complex, -ighimmlliittness data could significantly benefit tasks in intent detection, social awareness, and theory of mind analyses, among others.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research was supported part by a grant from the US National Library of Medicine (R0ILM013833).
REFERENCES
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Chenyu Zhang, Ruiqi Zhong, Sean O hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung. Lilian Ed- wards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwani, Yoshua Bengio, Danqi Chen, Philip Torr, Samuel Albanie, Tegan Maharaj, Jakob Nicolaus Foerster, Florian Tramèr, He He, Atoosa Kasirzadch, Yejin Choi, and David Krueger. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https:// /ttp:p/:////eeeeieww..... www.:/ettteeteeeemeee ret.:///ooomddddoT-o0o5
Regina Barzilay and Mirella Lapata. Modeling local coherence: An entity-based approach. Computational Linguistics, 1-33,, 2008. doi: 10. 10.11622/o0i::0088.444 3.... URL https:
//aclanthology org/J08-1001..
David lan Beaver. Presupposition. In Handbook of logic and language, pp. 939-1008. Elsevier, 1997.
Robyn Carston. The explicit/implici distinction in pragmatics and the limits of explicit communica-
tion. International review of pragmatics, 1(1):35 02: 2009.
Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. RA3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7976-7986, July 2020. doi: 10.18653/v1/2020.aee mttp:///i.12200..mmma..........
Liam Cripwell, Joël Legrand, and Claire Gardent. Simplicity level estimate (sle): A learned reference- less metric for sentence simplification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12053-12059, December 2023. doi: 10. 18653v1/v
2023.emnlp-main. 739..
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber, Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM '17, pp. 512-515, 2017.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, June 2019, doi: 18.18653/v
N19- 1423.
Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. Latent hatred: A benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 345-363, 2021. doi: 1122653/1/2021.111 10220202..mppmmaim.22...
Rudolph Flesch. A new readability yardstick. Journal of applied psychology, 32(3):221, 1948.
Liye Fu, Susan Fussell, and Cristian Danescu-Niculedn Niculescu-Mizii. Facilitating the communication of politeness through fine-grained paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 5127-5140. Association for Computational Linguis- tics, November 2020. doi: 10. 1/2553/112020.mmm/l man.nn......11........
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In FraiieFranccine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language
11

---

Published as a conference paper at ICLR 2025
Online and Punta Cana, Dominican Republie, November 2021. Association for Computational Linguistics. doi: 10. 10.18653 -emnlp-main.5555555 main.552. URL
Processing, pp.
https::/aclanthol.oe .rgg 202122221. emnip-main. 552.
Davide Garassino, Nicola Brocca, and Viviana Masia. Is implicit communication quantifiable?
corpus-based analysis of british and italian political tweets. Journal of Pragmatics, 194:9-22, June
2022. doi: 10.101 2tt////ppagma2000000000
Elizabeth Jasmi George and Radhika Mamidi. Conversational implicatures in english dialogue: Annotated dataset. Procedia Computer Science, 171:2316 2323, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pp. 249-256, March 2010.
Herbert Paul Grice. Logic and conversation. Syntax and semantics, 3:43-58, 1975.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale -egnerateee dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp- 3309-3326, May 2022. doi: 10.1 1
Md Saroar Jahan and Mourad Oussalah. A systematic review of hate speech automatic detection using natural language processing. Neurocomputing, 546: 2023.
Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. Are natural language inference models imppressive? learning implicature and presupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 8690-8705, 2020. doi: 10.18653/ v1/2020.acl- ww1.200.l-mmai..70. 768...768..
Youngwook Kim, Shinwoo Park, and Yo-Sub Han. Generalizable implicit hate speech detection using contrastive learning. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 6667-6679, October 2022. URL https: /tLtsp/s///cllatttoooooog wrg//0222
col 1.nin.1.55 579.
James Kohnen. Crucial conversations: Tools for talking when stakes are high, 2008.
Justin Lee and Sowmya Vajjala. A neural pairwise ranking model for readability assessment. In
Findings of the Association for Computational Linguistics: ACL 2022, pp. 3802-3813, May 2022. doi: 102553//11022.... http://ww/222...nnnnnn/lllllllle findings-aall.300.
Zhengyan Li, Yicheng Zou, Chong Zhang, Qi Zhang, and Zhongyu Wei. Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. November 2021. doi: 10.18653/v1/200 1/202/111221.l..2 w/2.:1.emnllpp----e. m2in.22.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, July 2019.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Tyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual preci- sion in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076-12100 Association for Computational Linguistics, December 2023. doi: 118865331/2222 http:/22..mmlpmmaaae main.741. 7.1.
Nicolas Ocampo, Elena Cabrio, and Serena Villata. Playing the part of the sharp bully: Generat- ing adversarial examples for implicit hate speech detection. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 2772, 2023. doi: 10.18653/w1/2023 findings-acl.
173.
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, July 2002. doi: 10.3115/107330331777133
12

---

Published as a conference paper at ICLR 2025
Barbara Partee. Compositionality, Varieties of Formal Semantics/Foris, 1984.
Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pp. 1-27,
June 2011. URL https: httpa/caaanntoooooooooooo /tt.ttss///aaaantoooooo
org/ //11--111 1901.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo. Aravind Joshi, and Bonnie Webber. The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), May 2008.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IICNLP). pp. 3982-3992, November 2019. doi: 10.1 1211886//v 1/tp//DDD11//// 1410.
Omid Rohanian, Shiva Taslimipoor, Richard Evans, and Ruslan Mitkov. Wiv semeval-201 2018 task 3: Dissecting tweets in search of irony. In Proceedings of The 12th International Workshop on Semantic Evaluation, pp. 553-559, 2018.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah Smith, and Yejin Choi. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 5477-5490, July 2020. doi:
10.18653/1/2000.aac
Damien Sileo. tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation. arXiv preprint arXiv:2301.0540 055488, 2023. URL https:
arxiv.org/ abs /20ll 05948.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai. Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Silco, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth. C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang. Erkut Erdem, Ernie Chang, Ethan Chi, Ethan Dyer. Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhski,, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumeed Francesca Happé, Francois Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Loper Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason
13

---

Published as a conference paper at ICLR 2025
Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochamee
Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Dliveros-Colón, Luke Metz, Lürfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung. Paul Pu Liang. Paul Vicol, Pegah Alipoommllabsshi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank. Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjue Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo- Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu. Xudong Shen, Yadollah Yaghoobzadeh. Yair Lakretz, Yangqiu Song. Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao. Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang. Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https::////. http:s::///opeeeeeeeeeeeeeeeeeee wwwp:eewwwwwwweeefrreeeeeee www.iiddduyyyyyyce
Sara Tonelli and Elena Cabrio. Hunting for entailing pairs in the penn discourse treebank. In Proceedings of COLING 2012, pp. 2653-2668, 2012.
Edoardo Lombardi Vallauri and Viviana Masia. Implicitness impact: Measuring texts. Journal of Pragmatics, 61:161-184, 2014. doi:
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational I Linguistics and the 11 International Joint
14

---

Published as a conference paper at ICLR 2025
Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1667-1682, 2021. doi: 10.18653/v 1/2021 w..22...acloooooo acl-long. 132.
Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, and Minlie Huang. Unveiling the implicit toxicity in large language models. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 1322-1338, December 2023. doi: 10.18653/v1/ 112.eemlp-main...
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL
https: https:://penreee //ppnneeiiewweeeee//e/ee/e..eeee nnt/
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 GPT PROMPTS FOR POSITIVE PAIR GENERATION
role: system
content (ToxiGEN): Generate a more explicit sentence that the given hate speech implies. For example, "it is not good idea be trustful of foreigners." "foreigners are not trustful." Don't put any prefix like "Explicit version:" in the output.
content (DGHSD); Generate more explicit sentence that the given hate speech implies. For example, "it is not a good idea be trustful of foreigners." - "foreigners are not trustful." Don't put any prefix like "Explicit version:" in the output.
content (Adversarial): Generate more explicit sentence that the given hate speech implies. For example, "it is not a good idea to be trustful of foreigners." " + "foreigners are not trustful." Don't 't put any prefix like "Explicit version:" in the output.
content (Implicatures): In the given conversation, what does the response imply? Please directly generate an explicit version of the response. Don't put "An explicit version:" or <Explicit Response>:" in the output.
content (LUDWIG): In the given conversation between and B, what does B's response imply? Please directly generate an explicit version of B's response. Don't put "B:" or "An explicit version:" such kinds of prefix in the output.
Generate a more explicit sentence the given sentence implies. Don't put any prefix like "Explicit version:" in the output.
content (ABSA):
content (SemEval): The given sentence contains irony sarcasm. Please generate more explicit version of the sentence expressing the actual meaning. Don't put any prefix like "Explicit version:" in the output.
content (Snarks): The given sentence is sarcastic. Please generate an explicit version of the sentence expressing the actual meaning. Don't put any prefix like "Explicit version:" in the output.
role: user
content: <Inserted Input Implicit Sentence>>
Figure 8: Prompts for generating positive pairs using GPTs from different data sourees.
A.2 TRAINING DATA DETAILS
We constructed training data from existing public data sources over various NLP research topics. These sources include ToxiGEN (Hartvigsen et al., 2022), Latent Hatred (EISherief et al., 2021). semmnically-Geeerateedd Saieee-peeeh---aaeeeee (DGHSD) (Vidgen et al., 2021), Adversarial-Implici. Hate-Speech (Adversarial) (Ocampo et al., 2023), SBIC (Sap et al., 2020) from the topic of hate speech detection; Implicatures (Sileo, 2023), ImpPres (Jeretic et al., 2020), LUDWIG (George & Mamidi, 2020) from the topic of natural language inference; ABSA (Li et al., 2021) from the topic of Sentiment Analysis; 2emEval-2018 Task 3 dataset (Rohanian et al., 2018) from the topic of Irony Detection; Snarks (Srivastava al., 2023) and (Chakrabryyy al., 2020) from the topic of Sarcasm Detection; and PDTB (Prasad et al., 2008; Tonelli & Cabrio, 2012) from the discourse relation study. Tab. 6 summarizes the processing details.
16

---

Published as a conference paper at ICLR 2025
Table 6: Processing details of our training data construction. The column # Pos. Pairs represents the number positive (implicit, explicit) training pairs constructed from each data source. The column Generation describes the method used to construct these pairs, either through prompts to GPT models or by using original labels annotated in the data sources.
Data Source
Pos. Pairs
Generation
Processing Description
Implicit Hate Speech Detection
Each hate speech instance ToxiGEN is rated for "toxic level" (1.0 5.0) and "intent level" (1.U 5.0). selected sentences with toxic levels from 2.0
ToxiGEN
648
GPT-3,5
intent levels below implicit, those with both levels above as explicit. prompted GPT with implicit sentences generate positive pairs.
This dataset contains Twitter posts labeled "implicit hatred" or "ex- plicit hatred". implicit posts are further explained clarify what they imply. We pair these implicit posts with their explanations obtain positive pairs.
Latent Hatred
12, 036
Original
Some hate spozch are labeled "implicit". prompt with implicit sentences to generate positive pairs.
DGIISD
3, 415
GPT-3.5
Some hate speech labeled as "implicit" and some is labeled as "ex- plicif". We prompt with implicit sentences to generate positive pairs.
Adversarial
25, 193
GPT-3.5
Some sentence samples are explained to clarify why they are implicitly biased. We pair them with their explanations to create positive pairs.
SBIC
316
Original
Natural Language Inference / Textual Entailment
Each sample dialogue containing context and an original response. We prompt GPT with
The original response is implicit the dialogue generate explicit response. We then pair the implicit sentences and explict ones to get positive pairs.
Implicatures
955
GPT-3.5
following three files of the implicature part gradable_aaject.ie jsonn,,, jsonl. gradable, bataablleerbbbbb verb. isonl, moda 5. where each data point consists of sentence and its implicature. corresponding to implicit sentence and a explicit sentence respectively.
the IMPPRES dataset:
IMPPRES
300
Original
As a part of BIG-bench, LUDWIG dataset of conversational impli- catures where each utterance (question) associated with conversa- tional response (implicit answer) and its implicature (explicit answer, i.c. yes/no). We append questions with implicit answers to create im- plicit sentences and questions with explicit answers to create explicit sentences.
LUDWIG
GPT-3.5
Sentiment Analysis
Each sample independent sentence, and labeled "im- plicit sentiment" "non-implicit sentiment". We prompl GPT with implicit ones and get positive pairs.
ABSA
933
GPT-3.5
Irony Sareasm Detection
Some tweet posts are labeled as ironic. Wc prompt GPT with implicit ones and get positive pairs. We found that GPT-3.5 models under- perform in understanding irony, so we switch
SemEval
1,316
GPT-4
instead.
This datascts contains sentence pairs which are lexical similar. In cach puir, one is a sarcasm and the other is not. We prompt GPT with sarcasm ones and get positive pairs.
Snarks
180
GPT4
This dataset contains sarcasi sentences with which we prompt GPT to obtain positive pairs.
R²
Original
Discourse Relation
Each sample hypothesis pair (Text + Hypothesis), which been manually annotated as entailing entailing. We further them (implicit, explicit) positive pairs.
PDTB
Original
Total
56, 290
SSpecifically, GPT:3.5-turho-0125 and GPT-4o models.
17

---

Published as a conference paper at ICLR 2025
Our constructed dataset has also been assessed by a linguistic expert. 20 sentence pairs (10 positive and 10 negative) were randomly sampled from each source for expert annotation, and an integer from 1 to 5 is given to each dataset using the following metrie: For a given sampled sentence pair, it is considered appropriate if (1) it is a positive pair where the two sentences are the same in the pragmatic dimension and follows the (implicit sentence, explicit sentence) pattern (2) it is negative pair where the two sentences are pragmatically distinct but also follow the (implicit sentence, explicir sentence) pattern. A fitness integer ranging from 1 to 5 is assigned to each source based on the percentage of sampled sentences that fit our design goal: 1 < 20%; 2 20% I < 40%; 40% S s < 60%; 60% S < 80%; 2 80%. The quality assessments, along with examples of positive and negative pairs, are presented in Tab. 7.
Table 7: Quality assessment and examples of positive and negative pairs our constructed training data, Note that some examples are highly toxic and harmful. A linguistic expert rated samples from each source dataset on a scale from 1 to 5.
Data Source
Quality Rating Positive and Negative Pair Example (implicit sentence, explicit sentence)
Implicit Hate Speech Detection
("the British were not being fair when they played cards with the native americans", "Native Americans are not capable of playing cards fairly.") Neg. ("the British were not being fair when they played cards with the native ameri- cans", "if youre not white youre not american.")
Pos:
ToxiGEN
5/5
('if you white in that state watch yourself !", "non- white people are dangerous.") Neg: you you are white that state watch yourself !", "Minorities are inferior to whites.")
Pos:
Latent Hatred
5/5
Pos: ("I am pretty glad was born white", "White people are superior." Neg: ("I am pretty glad I was born white". "Most of us aren't yellow fever idiots though and respect your entire culture. But asian men automatically assume we don't ' like them and we just want take their women and have respect for anything. That's not truc all, we just don't respect them.")
DGHSD
5/5
done for you ?", "Feminism has not benefited you in any way.") Neg: ("what has feminism ever done for you ?", "the creation story is proof that the native americans were no civilized socicty in fact one the most savage peoples to ever walk the carth.")
Pos: ("what has feminism
Adversarial
5/5
Pos: ("What runs in every black family? The dad", "black men always leave their families.") Neg: ("What runs in every black family? The dad",, "makes it souns if jewish people not part of the white race.")
SBIC
5/5
Natural Language Inference / Textual Entailment
Pos: ("- Do you expect Sheila to he late for the party tonight? - Is the pope Catholic?", ". Do you expect Sheila to be late for the party tonight? Yes, Sheila is always late for everything.") Neg: ("- Do you expect Sheila to be late for the party tonight? Is the pope Catholic??. ". Are you going keep your part time job the next semest at the biology laboratory? You do not plan to keep your part- tirt-time job at the biology laboratory for the next semester.")
Implicatures
4/5
Pos: ("The ladders are fine.", "The ladders are not great.") Neg: ("The ladders are fine.", "Kristin did not manage to confuse Lisa.")
IMPPRES
5/5
Pos: (*- Do you like my new outfit? You shouldn't 't be allowed to buy clothes.", ". Do you like my new outfit? That outfit does not look good on you.") Neg: (*- Do you like my
LUDWIG
5/5
outfit? You shouldn't be allowed to buy clothes.". "- Are you going to talk to Mark? need to talk to Mark and address something that has been bothering me.")
Sentiment Analysis
Pos: ("I didn't go there for food can't comment." "Since did not go that place specifically to eat, I not have any feedback or opinions on the food available there.") Neg: ("I didn't go there for food so can't comment.", "One drawback, I wish the keys were backlit.")
ABSA
4/5
Irony/ Sarcasm Detection
18

---

Published as a conference paper at ICLR 2025
("Sweet United Nations video. Just in time for Christmas.", "The United Nations released video, but it seems poorly timed or irrelevant for the Christmas season.") Neg: ("Sweet United Nations video. Just in time for Christmas.", "THEY BE should ALLOW more REFUGEES among them will be potential.")
Pos:
SemEval
5/5
("Nobody can be successful without having inherited all their money first",, "People can achieve success through their own hard work and determination, without needing to inherit moncy.") Neg: ("Nobody can be successful without having inherited all their money first.", "The best way defeat stronger/larger and better regimented force is by fighting on your terms.")
Pos:
Snarks
5/5
good mood today. She just old cranky.". "Mom a bad mood today.") Neg: ("Mom in a good mood today. She is just old and cranky.", "Such an annoying start for my morning.")
Pos: ("Mom
R
5/5
Discourse Relation
Pos: ("Look how much is moving around.", "The ventilation great.") Neg: ("Look how much air moving around.". "Investors are buying stocks that have predictable earnings.")
PDTB
4/5
A.3 USER STUDY QUESTIONS AND RESULTS
Tab. presents the detailed questions in the Ranking task and ranks answered by human participants, IMPSCORE, and the gold ranks. Tab. 9 presents the detailed questions in the Choice task and answers of human participants, IMPSCORE, and gold answers. Colored boxes are associated with specific sentence options in each question.
Table Details 10 questions Ranking task of the user study, along with answers from human participants, IMPSCORE, and the gold rank for each question.
Q / Group Gi
Topic: Ending Relationship
I've made my mind to end this relationship. It's clear as day that we need to go our
separate ways.
It scems me that association has reached its natural conclusion.
Our recent challenges have made think about our future. Do you think we're really right for each other?
I've been reflecting on happiness and fulfillment, both for myself and people care
about. What your thoughts where stand?
Rankings
Human1
Human2
Human3
Human4
Human5
IMPSCORE
Gold Rank
most explicit
explicit
implicit
most implicit
Q2 Group Ga Topic: Critiquing colleague's work
This report has several inaccuracies and the analysis lacks depth. We need to address these errors immediately.
There few points your report that aren't aligning with data standards. We should
revise these sections for accuracy.
Your report offers some interesting insights, but wonder if a deeper data review could enhance
our conclusions.
Your report has sparked interesting thoughts about our project. Maybe could review the
data together to explore other potential directions?
Rankings
Human1
Human2
Human3
Human4
HumanS
IMPSCORE
Gold Rank
most explicit
explicit
implicit
most implicit
19

---

Published as a conference paper at ICLR 2025
Qa / Group Ga Topic: Dealing with rebellious child
You need to follow the rules of our house there will be consequences.
We must understand, my child, that respecting our household rules is crucial.
Do you think could talk about your recent behavior and how fits with our family
expectations?
Do you think we're the same page about how things should be home?
Rankings
Human1
Human2
Human3
Human4
Human5
IMPSCORE
Gold Rank
most explicit
explicit
implicit
most implicit
Q4 Group G4
Topic: Giving the boss feedback about her behavior
Your recent behavior has negatively affected our team's dynamics and needs to be addressed immediately.
I've noticed some actions from you that could be impacting team morale negatively.
think exploring different approaches thoughts?
management might henefit all. What are your
Have you considered the impact of our leadership styles the team's atmosphere?
Rankings
Human1
Human2
Human3
Human4
Human5
IMPSCORE
Gold Rank
most explicit
explicit
implicit
most implicit
Qs / Group G3 Topic: Giving unfavorable performance review of ballet
The performance did not meet the expected standards of technique and expression,
Several sections require substantial revisions ensure they fully capture the artistic intent and emotional depth required.
The ballet might benefit from additional practice in precision and synchrony.
The performance hinted at classical roots, yet further rehearsals could enhance its clarity
and cohesion.
Rankings
Human1
Human2
Human3
Human4
Human5
IMPSCORE
Gold Rank
most explicit
explicit
implicit
most implicit
Qo Group Go
Topic: Asking roommate move out
You must find new place and out by the end of this month. you to find a new place.
Our living arrangement isn't working anymore.
Should we consider other living situations to ease tensions?
Maybe exploring other housing options could benefit us both?
Rankings
Human1
Human2
Human3
Human4
Human5
ImPScore
Gold Rank
most explicit
explicit
implicit
most implicit
Q7 / Group G7 Topic: Handing off difficult project to colleague
This project has become too difficult for me. I think your skills are better suited for it. Can take it over?
I'm struggling with this project and believe you have the right skills manage it. Would you be willing to take over?
You have a flair managing complex challenges. This project is proving tough me. Interested in taking the reins?
I've seen how well handle tricky projects. There's one my desk that seems right up your alley. Fancy giving it a whirl?
Rankings
Human1
Human2
Human3
Human4
Human5
IMPSCORE
Gold Rank
20

---

Published as a conference paper at ICLR 2025
most explicit
explicit
implicit
most implicit
Qs / Group Gs
Topic: Disliking John's personality
find it challenging get along with John because of his behavior.
John's behavior often conflicts with my expectations.
How do you perceive John's 's approach to things? sometimes find it doesn't quite match my own style.
John certainly has distinctive approach, doesn't 't he?
Rankings
Humanl
Human2
Human3
Human4
HumanS
IMPSCORE
Gold Rank
most explicit
explicit
implicit
most implicit
Qo / Group Go Topic:
Declining friend's party invitation
approciate the invitation, but I don't enjoy parties at all, so won't be attending this one. I'm generally not big fan of parties, so think I'll pass on this one. That' Il definitely be blast! I'll prohably take rain check this time and meet up with you
another day.
tend prefer quieter evenings. How about we get together for more low-key gathering soon?
Rankings
Human
Human2
Human3
Human4
Human5
IMPSCORE
Gold Rank
most explicit
explicit
implicit
most implicit
Q10 Group G10
Topic: Reminding roommate to clean the kitchen
You need to clean the kitchen every time you use it. It's not fair for others to take care of it. clean up after yourself. Maintaining cleaner kitchen could really benefit both of us. What do you think about making
I've noticed you often leave the kitchen messy. Please make
an extra effort?
Hey, how about we both try to keep the kitchen spotless after we use it? It would make cooking much nicer!
Rankings
Human1
Human2
Human3
Human4
Human5
IMPSCORE
Gold Rank
most explicit
explicit
implicit
most implicit
Table 9: Details of the 10 questions in Choice task of the user study, along with answers from human participants, IMPR, and the gold answer for each question.
Q Reference sentence: noticed few unusual behaviors during the exam that I'd like understand better.
Exams can overwhelming, and sometimes they might prompt actions wouldn' 't normally consider.
Perhaps the new team member would thrive with a bit more mentoring on our specific methods.
I've seen how well you handle tricky projects. There's 's one on my desk that seems right up your alley.
Fancy giving it whirl?
Answer
Humanl
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest
21

---

Published as a conference paper at ICLR 2025
Q2 Reference sentence: Hey, the scent of your dinner is pretty powerful. What do you think about eating it outside?
Your dinner smells intense! Maybe it would be more enjoyahle if you had it outside in the fresh air?
Do you think we're re on the same page about how things should be at home?
You really savor your noodles loudly! It's interesting, as it's quite the opposite my culture, where we
cat quictly.
Answer
Humanl
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Q3 Reference sentence: You have a flair for managing complex challenges. This project is proving tough for me. Interested in taking the reins?
I've seen how well you handle tricky projects. There's one on my desk that seems right up your alley.
Fancy giving it a whirl?
Have you considered the impact of our leadership styles on the team's atmosphere?
Reflecting on my optimal working conditions, I'm curious if less oversight might enhance my output.
Answer
Humanl
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Qj Reference sentence: It's a bit tight for me financially these days.
Mixing money and friendship often blurs the lines.
John certainly has a distinctive approach, docsn't he?
How do you think you could boost your contribution to team?
Answer
Human1
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Qs Reference sentence: Could we perhaps look other options that might align more closely with our
expectations?
Is there a chance we could turn over a new leaf with a different model or terms? Hey, how about we both try to keep the kitchen spotless after we use it? It would make cooking so much nicer!
Your report has sparked interesting thoughts about our project. Maybe we could review the data together to explore other potential directions?
Answer
Humanl
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Qa Reference sentence:
That restaurant is not exactly my favorite could we look at some other options?
That restaurant is curious choice. Perhaps we could check out a few other places as well?
Your dinner smells intense! Maybe it would be more enjoyable if you had it outside in the fresh air?
You really savor your noodles loudly! It's interesting, as it's quite the opposite my culture, where
we eat quietly.
Answer
Human1
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Q7 Reference sentence: I've been thinking about how I work best with more freedom in handling
tasks.
Reflecting on my optimal working conditions, I'm curious if less oversight might enhance my output,
Could we discuss our financial arrangement? It's time to sort out a plan.
Hey, how about we both try to keep the kitchen spotless after we use it? would make cooking so much nicer!
Answer
Humanl
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Q: Reference sentence:
Do you think we could try new approach where handle more things by
ourselves?
We value your advice, but we're keen on handling things ourselves to learn and grow. Perhaps the new team member would thrive with a bit more mentoring on our specific methods. How do you think you could boost your contribution our team?
22

---

Published as a conference paper at ICLR 2025
Answer
Humanl
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Qa Reference sentence: Have you tried the deodorants? They ro great for staying fresh all day.
just found some amazing products for staying fresh longer. I think you might like them too.
Do you think we're the same page about how things should be home? Hey, how about both try keep the kitchen spotless after it? It would make cooking much nicer!
Answer
Humanl
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Q10 Reference sentence: Do you think we could talk about your recent hehavior and how it fits with our family expectations?
Do you think we're same page about things should home?
The performance hinted at classical roots, yet further rehearsals could enhance its clarity and cohesion.
Maybe exploring other housing options could benefit us both?
Answer
Humanl
Human2
Human3
Human4
Human5
IMPSCORE
Gold Answer
Closest one
Human1
1.00
0.65
0.65
0:44
0.60
0:40
Human2
1.00
3.00
8.
0:72
Human3
Sct
Human4
0.44
0.84
0.84
1.00
76
0:48
Human5
76
1.00
0.76
ImpScore-
0.72
5.72
0.76
1:00
Human1-
1.00
0.76
0:84
0:84
0.88
0.72
Human2
Human3
0.84
1.00
0.76
Human4-
Set 2
0.84
0.68
0.76
1.50
0.80
Human5
ImpScore-
0.22
0:50
0.32
0.60
0.88
1.00
Human1
Human2
Human3
Humand
Human5
ImpScore
Human1
Humanz
Human3
Human4
Humans
ImpScare
Figure 9: Inter-participant Spearman's 's Rho (p) correlation results in Ranking task for user study.
Table 10: Implicitness scores computed by IMPSCORE for 4 sentences in 10 Ranking task questions. The last row indicates the average scores across questions of 4 levels sentences. Higher scores indicate higher implicitness levels.
Implicitness score of each sentence computed by IMPSCORE
Topic Group
Most explicit sentence
Explicit sentence
Implicit sentence
Most implicit sentence
G:
0.91
0.96
1.10
1.55
Ga
0.94
0.96
1.10
1.18
Ga
0.90
0.87
1.52
G4
0.44
0.67
0.57
0.97
Gs
0.22
0.72
0.88
0.83
Go
0.93
0.94
1.50
1.36
G7
0.53
0.89
0.86
1.30
Ga
0.49
0.33
1.04
1.40
Go
0.67
1.40
1.57
1.73
G10
0.90
0.91
1.13
1.84
Average implicitness
0.69
0.84
1.06
1.37
23

---

Published as a conference paper at ICLR 2025
The equation for computing Kendall's Tau when measuring the correlation of two ranks:
(Number concordant pairs) (Number of discordant pairs) (number of pairs)
(II)
A.4
HYPERPARAMETER SENSITIVITY
Please the results in Fig. 10.
Impliommss Accuracy
Impliciooss Accunccy
0.5
1.5
02 1
1344
135
1832
139
323
Impl
Figure 10: Hyperparameter sensitivity of " and yy Implicitness Accuracy when (0.5, 1.5,2.0). 2.0). The highest point is marked in red dotted box.
A.5 USER STUDY DESIGN
OOD Data Topics We adopted most of the topics from the book Crucial conversations: tools for talking when stakes high (Kohnen, 2008), section "Some Common Crucial Conversations" in Chapter One. This book introduces how to apply different expressions important conversations, which we think is a very useful resource. Other topics were designed by ourselves. The details are shown in Tab. 11. We also asked linguist to assess the quality of sentences under each group in Tab. 11. The quality of the test examples was evaluated on two dimensions: pragmatics and implicitness. The pragmatics rating (0 to 4) reflects the level of pragmatic agreement among the sentences. If all four sentences convey the same meaning, the group receives score of If three of the sentences share the same meaning, the group is given a score of 3, and so on. In terms of the implicitness rating, since (t) = 6 pairs can be created from the four sentences in group, the implicitness rating will range from 0 to 6, measuring the extent which the four sentences show progressively higher level of implicitness.
Platforms and Participant Recruitment We used Gorilla (https: htattps://app.goiilll gorilla sc/) to design the interface for the user study and Prolific (https: /www//wwwww.c www.:wwwppo...oo www.ppooliie. fi c.c.f.//c) for recruit- ing participants. All participants were native and proficient speakers of English, hold bachelor's degree, have conducted more than 10 surveys Prolific, and have history of 100% approval rate.
User Study Quality Assurance We provided participants with examples of both the Ranking and Choice tasks before they began the survey, showcasing the tasks and guiding them on how to properly answer the questions. Their progress was monitored at every step to ensure no abnormal responses submitted. Before recruiting participants, with the project, to pilot the study. While the two pilot participants did not report any questions or concerns when completing the experiment, they noted that the judgments were highly subjective.
asked two lab members, who were unfamiliar
24

---

Published as a conference paper at ICLR 2025
Table 11: The source of each OOD topic group either borrowed from or inspired by the book Crucial conversations: tools for talking when stakes are high or developed by us. Detailed sentences under each group. after refinement and verification, are shown in Tab.8. For each group, linguist provided a assessment of the pragmatics quality (rated from 0 to 4) and implicitness level quality (rated from O to 6) of its sentences.
Topic Group
Source
Pragmatics Rating
Implicitness Rating
C. Ending Relationship
Book
4
6
G2 Critiquing colleague's work
Book
4
6
Cs: Dealing with rebellious child
Book
4
6
G2 Giving the feedhack about behavior
G: Giving an unfavorable performance review of ballet
Book
4
Go Asking roommate to move out
Book
4
6
Cr: Handing off a difficult project to colleague
Original
4
6
G Disliking John's personality
Book
4
6
Go: Decline a friend's 's party invitation
Original
4
6
G1o. Remind roommate to clean the kitchen
Book
4
6
A.6 EMBEDDING VISUALIZATION AND CASE STUDY ON OOD DATA
Fig. 11 displays a 2-D visualization of the pragmatic embeddings for sentences from the ten topic groups in user study data using t-SNE. The clusters of sentences within the same topic group are evident, highlighting the effectiveness of the pragmatic embeddings in grouping similar thematic content. Fig. 12 presents the distribution of their semantic embeddings. Although points marked with the same colors generally tend to cluster together, some are noticeably distant from their corresponding groups. We analyze this is attributed to the fact that while sentences within the same topic may be pragmatically aligned, their varied expressive styles contribute to broader semantic distribution. For better understanding, case study of the semantic embeddings from two topic groups is provided in Fig. 13.
group
Erdiig Relttirrahp
Critizsing collaagec's
40
Acking a roommute to move
20
friend's
Ramind rommmntt dieanthe letonen
0-
Handing off dirioutt project leve
erplich
-00
cip
imp kit
moc: Implct
-40
-60
-40
-20
20
40
60
Figure 11: t-SNE Visualization of pragmatic Embeddings of the OOD Data in the User Study. Points with the same color belong to the same topic group, and different markers indicate varying levels of implicitness.
25

---

Published as a conference paper at ICLR 2025
75
Eeding Relationship
Critisting collapurke
Ackingg room mate mase
50
Draling wen robe lous dhil
wiing the boss feodback sbout her tehawlo
Giiing anurtwerraee perforance nev baiee
Daikinng jotn peronnaity
Dec me ritends party mi mibaoon
Remind roummats dean the kitches
Handing aff difiuut project leajue
lewel
most explic
expleet
25-
implet
rrpict
Figure 12: 1-SNE Visualization of semantic emheddings of the OOD data in the User Study.
75
75
meee new peill I' odbato caz
50
sedied lave
Shoudd conake obst
25
uas is Bliil alle yaunul.
Marazigg dlate Sacher What dkacabus caker eus affeet
-25
-25
smangeeed lifh ficla p0rd
Eovataciio cane
-50-
-50
Vaybe replaring inoelssinnggtinn buk??
-75
-75-
100
-40
-20
20
40 60
80
100 -40
-20
40
60
80
Figure 13: Case studies on the distribution semantic emheddings for two topic groups: Remind roommate to clean the kitchen (left figure) and Asking a roonmate to move oud (right figure). The corresponding sentence for each point is annotated alongside. The text pieces highlighted within the sentences are the parts conveying similar semantics, explaining why some sentences cluster together while others do not.
A.7 IMPLICITNESS SCORE DISTRIBUTION OF HATE SPEECH DATASETS
2,00
1.75
1.50
Implctneess Scare
1.25
1.00
0.75
Adversanal (imp)
Latest (a.))
Latent Hatred dimp)
0.50
Adversrial
0.25
Davidon
DGiiSD
DGGUS
SBiC
0.00
Figure 14: Distribution of implicitness scores computed by IMPSCORE of each hate speech detection dataset.
26

---

Published as a conference paper at ICLR 2025
PROMPTS FOR HATE SPEECH DETECTION A.8
role:
system
content: Please indicate whether the following text is toxic, harmful, otherwise inappropriate by simply answering "yes" or "no."
role: user
content: <Inserted Hate Speech Sentence
Figure 15:
Prompts for using GPr-4-Turbo and Llama-3. 3.1-aBBBBBBBBBBB for Hate Speech Detection. We set the maximum output token of Llama-3. 1-8B-Instruct to 3. All other hyperparameters are set to default.
A.9 SAMPLES AND HATE SPEECH DETECTION ACCURACY ON ADVERSARIAL. DATASET
The hate speech detection result on Adversarial dataset shown in Fig. 16. The implicitness range [1.75, of Adversarial dataset is filled with very similar sentences shown in Tab. 12.
Table 12: Examples of very similar sentences in the implicitness range [1.75, 2] in Adversarial dataset.
middle east is a war zone why do we clearly have to admit a failing cultural into ours ?
middle east is zone, why do apparently have admit failing cultural into ours ?
middle east is a war zone , why do we surely have to admit a failing cultural into ours
middle east is a war zone why do we doubtlessly have to admit a failing cultural into ours ?
middlc cast is a war zone why do presumably have to admit failing cultural into ours ?
middle is a war zone, , why do indeed have admit failing cultural
?
middle east is a war zone, why do we probably have to admit a failing cultural into ours
woman should undoubtedly be aloud to wear what she wants it is a little thing in the western world we like to call freedom.
what she wants, little thing the western woman should doubtlessly be aloud world we like to call freedom.
woman should seemingly be aloud wear what she wants, is little thing in the western world like call freedom.
a woman should likely be aloud to wear what she wants, it is a little thing in the western world we like to call freedom.
a woman should clearly be aloud to wear what she wants, it little thing in the western world we like to call freedom.
woman should absolutely be aloud to wear what she wants it is little thing in the western world we like to call freedom.
Delection Accuracy
90%
OpenAl Moderation. - 4PT-4-Turbu
Llarna-3. 1-BBBBBB Instructttt
80%
70%
60%
50%
40%
30%
20%
Adversarial
10%
[0.00, 0.25) [0:25, 0.50)
[0.50, 0.75)
[0.75, 1.00)
[1.00, 1.25)
[122, 1.50)
[1,50,1.7
[1.75. 2.00]
Impliciness Score
Figure 16: Hate speech detection accuracy of LLMs on sentence samples with different levels of implicitness. Blue columns indicate the distribution of samples among different ranges of implicitness.
A.10 USER STUDY QUESTIONNAIRE
Fig. 17 shows a screenshot of a question in the Ranking task, and Fig. 18 shows a screenshot of question in the Choice task.
27

---

Published as a conference paper at ICLR 2025
[1/5] Please Rank the Following Statements by Their Level of Implicitness (from explicit to implicit)
Topic: Ending Relationship
Most explicit Least implicit
I've made up my mind to end this relationship. It's clear as day that we need to go our separate ways.
:: It seems to me that our association has reached its natural conclusion.
:: I've been reflecting lot an happiness and fulfillment, both myself and the people care about. What are your thoughts on where we stand?
Our recent challenges have made me think about our future. Do you think we're really right for each other?
Most implicit
(Drag the options to enable continuing)
Next O
Figure 17: Screenshot of Ranking task question in study questionnaire.
[1/5] Please indicate the statement that is pragmatically closest to the given one.
Given Statement: noticed few unusual behavlors during the exam that like understand better.
Perhaps the new team member would thrive with bit more mentoring on our specific methods.
how well you handle tricky projects. There's my that right up your alley. Fancy giving it whir??
Exams be overwhelming, and sometimes they might prompt actions wouldn't normally consider.
Next
Figure 18: Screenshot of Choice task question study questionnaire.
A.11 TRAINING PERFORMANCES BREAKDOWN
Table 13 includes the performance breakdown of IMPSCORE in different data sources in the test set during training.
28

---

Published as a conference paper at ICLR 2025
Table 13: Breakdown IMPSCORE's performance on different data sources in test set during training.
Source of Test Data
Source Topics
(implicit, explicit) Pair
Correct Prediction
Accuracy
DGHSID
Implicit Hate Speech Detection
302
245
0.811
Latent Hatred
Implicit Hate Speech Detection
1166
1156
0.991
ToxiGEN
Implicit Hate Speech Delection
0.830
Adversarial
Implicit Hate Speech Detection
2460
0.977
SBIC
Implicit Hate Speech Detection
900
896
0.996
Implicatures
Natural Language Inference
89
68
0.764
ImpPres
Natural Language Inference
12
12
1.000
LUDWIG
Natural Language Inference
69
55
0.797
ABSA
Sentiment Analysis
136
112
0.824
SemEval
Irony Sarcasm Delection
132
108
0.818
Snarks
Irony Sarcasm Detection
16
10
0.625
R-3
Irony Sareasm Detection
14
12
0.857
Irony/ Sarcasm Detection
4
0.667
A.12 CROSS-LINGUAL PERFORMANCE OF IMPSCORE
conducted an experiment to evaluate IMPSCORE 's performance on three non-English languages: French, German, and Chinese. Specifically, we first utilized Google Translate" to translate the 10 group topics from the user study in $6 into these won-English languages. Subsequently, we prompted ChatGPT in each non-English language to generate four sentences per topic, varying in levels of implicitness. Here are the prompt formats:
1. French: Pour sujet Mettre fin à une relation >, pourriez-vous générer phrases différents niveaux d' l'mpliiit? du plus explicite au plus implicite.
2. German:
Formulieren Sie zum Thema ,,Beziehung beenden" bitte 4 Sätze mit unterschiedlichen Implizitheiisgraden, vom explizitesten bis zum implizitesten.
3. Chinese:
1 - " # A
#1 (M
The generated sentences these languages were then translated back to English using Google Translate. We used Google Translate instead of directly using ChatGPT for translation to minimize inherent cross-language alignment biases in ChatGPT, ensuring more independence of each language. Then, IMPSCORE rank the four sentences within each topic and compare the predictions with the gold rankings. The average results are summarized below. Detailed scores for each group visible in the Tab. 14.
These findings indicate that IMPSCORE demonstrates a moderate positive correlation with gold rankings these translated sentences. However, its accuracy diminishes compared the assessment on original English sentences in our paper (avg. = 0.76 and = 0.84), particularly for language that is linguistically distant from English (i.e., Chinese).
This reduction in accuracy may stem from loss of subtle implicitness nuances in original languages during translation into English, suggesting potential limitations using translations as proxy for cross-lingual evaluation of implicitness.
Table 14: IMPSCORE's performance on translated text of non-English sentences.
Non Mon-English Languages
Avg. over groups
Avg. P over groups
French
0.63
0.74
German
0.70 0.43
0.63
Chinese
0.58
"nttps://transslaee.gooo
29

---

Published as a conference paper at ICLR 2025
A.13 ADDITIONAL USER STUDY RESULTS
We conducted another 10 groups OOD test data for user study, following the same procedure detailed the paper $6. Below are the topics:
G1: Reject an auto dealerships offer
G2: Talking coworker about specific hygiene problem
G3: A student cheating on the exam
G4: Asking friend to repay loan
G5: Complain about micromanaging
Gg: Unpleasant food smell
Gy: Criticizing a colleague's 's work
Gg: Eating with big sound
Gg: Having bad memory about hometown
G1g: Talking to team member who isn't keep commitments
We divided the topics into two sets (Set 1: topics G1-G5; Set 2: topics Gg-G10) and recruited five human participants to rank their implicitness. The results of their predictions compared to the gold ranks are summarized in Tab. 15. The correlation plots between human participants and IMPSCORE are in Fig. 19 and Fig. 20.
The results indicate generally high accuracy of IMPSCORE with the gold ranks and concordant with human participants, which align with our conclusion in line 406-409 in the paper. set I, IMPSCORE slightly underperforms the average human ranking, and in set 2, IMPSCORE slightly surpasses the average human ranking.
Table 15: Performances of IMPSCORE and human participants compared to the gold ranks on additional 10 data groups
Human
Human
Human 3
Human 4
Human 5
IMPSCORE
Set
Avg. 7"
Avg. p"
Avg. 7"
Avg. p²
Avg. 7
Avg. p²
Avg.
Avg. n²
Avg. 7!
Avg. p²
Avg. r!
Avg. p
0.93
0.96
0.87
0.88
0.87
0.92
0.93
0.96
0.87
0.92
0.67
0.72
0.93
0.96
0.93
0.96
Human1 -mm
6.00
Humant
-3:6
8.32
9.86
053
0.02
3.00
Human2-Jim
Human33 8:55
895
3.00
8.53
5.55
3.81
Humand
6.64
8.66
1.86
054
3.66
3.02
Human4-J1
HumanS- -2
0.00
3.44
Humand
0.32
3.86
189
3.82
Humani-
1.08
035
13
0:31
Humant
1-8
036
3.92
Human3.
0.7
033
139
0:51
Humand
0.00
036
L97
022
882
Humans-
68B
687
647
Humand:
0.8
692
632
ImpSicore
Humani
Humanz
Humand
Humand
Humard
Himani Human2
3
Aumand
Humard
Imp Scont
Human]
Humanz
Humart
Humani
Humant
imptScore
1umant
Humanz 3
Humast
Humand
Impdcore
Figure 19; 7 correlation values between partici- pants.
Figure 20: correlation values between partici- pants.
A.14 QUALITATIVE ANALYSIS OF IMPSCORE'S PERFORMANCE
We evaluated the IMPSCORE's performance the in-distribution test set on the following features.
A.14.1
THE LENGTH OF THE SENTENCES
For 5, 630 pairs of (implicit sentence, explicit sentence) in the test data, we analyzed sentence length by calculating the average number of space-separated tokens per sentence. We then plotted the
30

---

Published as a conference paper at ICLR 2025
distribution of sentence lengths alongside the corresponding numbers of correctly predicted cases. The result plot is in Fig. 21, which shows no significant decline in accuracy for longer sentences.
400
200
150
100
40
60
80
100
120
340
Average Sentence Length of Pair
Figure 21: IMPSCORE's prediction performance on different length of sentence pairs.
A. 14.2 SENTIMENT
We evaluated the sentiment of each sentence in the pairs using the pre-trained sentiment classifi- cation model cardi fftlp/twitter-roolt rta-bbbbbrt----eeeeeeeeee iment-lated Sentiments were categorized into (positive, neutral, negative), and each pair was grouped into one of eight combinations (e.g,, negative-positive, neutral-neutral).
The result plot is in Tab. 22. The "negative-positive group exhibits the lowest performance, likely due to a bias introduced by small sample size. Otherwise, the results do not reveal a clear underperformance for any specific sentiment group. Additionally, we observed that negative sentiment dominates the dataset- a trend that aligns with our expectations, as implicit sentences often arise in sensitive negative contexts.
2750
Total Sentonce
Correctly Predicted Pairs
2500
2250
2000
Pair Count
1750
98.0%
750
500
250
91.3%
75.9%
pos posineg negisse nog-neg nog neu neu goz Implicit Sentence Sentiment Explicit Sentence Sentiment
Figure 22: IMPSCORE's prediction performance on different sentiment sentence pairs.
A.14.3 TSNE FOR CORRECT/INCORRECT PREDICTIONS
We also analyzed the semantic distribution of correctly predicted pairs and incorrectly predicted pairs. To do so, concatenated the implicit and explicit sentences in each pair, generated embeddings using Sentence-BEi
and applied t-SNE for dimensionality reduction to visualize the results in a 2D plot. The visualization is in Fig. 23.
The result reveals that while both correct and incorrect predictions are broadly distributed, certain clusters of incorrect predictions emerge (highlighted Box 1 and Box 2 in the plot). On closer
31

---

Published as a conference paper at ICLR 2025
examination, found that the incorrect predictions in Box predominantly pertain digital devices, such as "computers." "CDs," "laptops," and "RAM." This suggests a potential challenge for IMPSCORE in handling domain-speciin or technical terminology. Box 2 contains pairs where distinguishing the implicitness levels between sentences proved exceptionally difficult. For example: Will that make everything alright? Thank you., Will that make everything alright? Thank you for your concern.)
Sentence Tranforrmr Embedding Distribution
Coneect Predictors
Incoriett Medictiers
-30
20
Figure Embedding distributions of IMPSCORE's correctly and incorrectly predicted sentence pairs.
A.15 APPLYING IMPSCORE TO THE GENERATED SENTENCES OF LLMS
We zero-shot prompted seven LLMs (GPT-3.5, GPT-4o, 1.ama-3.2-Instrree Mistral-7B, Mistral- Nemo, Claude-haiku and Claude-sonnet) with 10 topics same with the ones in the user study and asked them to generate four sentences with different implicitness levels. We used the prompt "Given the topic [topic], please generale four sentences about it, ranging from Output each sentence on new line." and set the temperature to each model's default value.
explicit most implicit.
For each rum, we calculate the average implicitness scores of sentences over 10 groups in each implicitness level (most explicit, explicit, implicit, most implicit). We conducted runs and plot the mean and standard deviation of the average scores. The result is in Tab. 24.
The results demonstrate that, according to IMPSCORE's evaluation, all the models are generally capable of producing sentences with distinguishable implicitness levels. Among them, GPT-4o exhibited the best performance in generating sentences with the most clearly distinguishable levels of implicitness. Additionally, GPT-4o and Claude-sonnet showed the strongest ability to generate highly implicit sentences, aligning with our expectations given their advanced capabilities.
Mean Implicitness Values (5 runs over 10 groups) with Standard Deviations
Libe
4PT44
1.0
Matral7B matrut
Implicitness scores
0.8
0.4
0.2
Most explici
Explicit sentence
Implicit sentence
Most implicit sentence
Figure 24: IMPSCORE's scores on the generated sentences seven LLMs.
32


Reviews:
[{"review_id": "ABmEFZ6UgI", "reviewer": "Reviewer_czyP", "summary": "This paper introduces IMPSCORE, a novel metric for quantifying the implicitness level of language by measuring the divergence between semantic meaning and pragmatic interpretation. The authors develop an interpretable regression model trained through pairwise contrastive learning on a newly created large dataset of 112,580 sentence pairs. The metric processes sentences using a text encoder to generate embeddings, maps semantic and pragmatic features into separate spaces, and calculates implicitness scores based on the divergence between these features after space transformation.\n\nExperimental results validate the reliability of IMPSCORE. The authors also conduct a user study comparing the metric's assessments with human evaluations on out-of-distribution data, showing strong correlation with human judgments. They also demonstrate practical applications by analyzing implicitness levels in hate speech detection datasets and evaluating performance of several large language models across different implicitness levels, revealing that model performance degrades as implicitness increases.", "questions": "- I'm curious if you could run some experiments to test the metric's performance on other languages? An easy (though, not perfect) way would be to look at translated sentence pairs across multiple languages to assess its language independence and potential biases?\n- I'd love to see some qualitative error analysis for the metric. Are there any patterns that emerge in the sentences on which IMPSCORE isn't accurate?\n- Is IMPSCORE just as reliable on longer sentences as on shorter ones?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well motivated and presents a clear operational definition of implicitness and develops a concrete methodology for measuring it. The paper is well written and easy to follow. \n- Experimental results show strong support for the IMPSCORE as a reliable metric. The authors further validate it through careful ablation studies, user studies, and practical applications in analyzing both datasets and model performance.\n- The training dataset is substantial and diverse (and is a contribution in itself), drawing from multiple domains and types of implicit language, while the evaluation includes both in-distribution and out-of-distribution testing.", "weaknesses": "- The metric is developed and tested only for English language content. It is not clear how this would translate to other languages or whether this metric will work in cross-lingual settings.\n- While well-designed, the user study involves only 10 participants evaluating 10 questions each, which is a relatively small sample size for validating a metric intended for broad use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yB4WKmRVnv", "reviewer": "Reviewer_keRt", "summary": "This paper introduces ImpScore, a reference-free metric designed to quantify the level of implicitness in human language. The model for ImpScore is trained via contrastive learning by calculating the divergence between the semantic (literal) meaning and the pragmatic (contextual) interpretation of a sentence. They collected a training data with 112,580 pairs. Through extensive experiments, ImpScore demonstrates strong correlation with human judgments. They also show the utility in evaluating hate speech detection datasets, revealing limitations in current large language models' ability to handle implicit content.", "questions": "* How does the performance of ImpScore vary across different types of implicit language (e.g., indirect requests, cultural idioms, irony)? Are there specific types that it struggles with?\n* Have you examined any potential biases and quality of the synthetic data?\n* Can you compare with existing methods or other baselines for implicitness? For example, the methods mentioned in Related Work (e.g. Garassino et al. (2022) and Vallauri & Masia (2014))?\n* I would like to see if you can implement ImpScore to the LLMs' abilities to generate implicit expressions. Similar to the data creation in Section 6.1, you can prompt different LLMs to generate 4 sentences with varying implicitness levels. Then, you calculate the ImpScore for these generations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a rarely addressed problem, quantifying implicitness in language. It offers a new approach to measuring implicitness that moves beyond binary or purely lexical classifications.\n2. ImpScore is a reference-free metric, which distinguishes it from previous metrics that often rely on external references or manual annotations.", "weaknesses": "1. Limited scope of implicit languages. The paper’s dataset includes synthetic data generated by GPT-3.5 and focuses primarily on specific types of implicit language (e.g., hate speech). However, ImpScore’s generalizability across broader types of implicit expressions (such as indirect requests or cultural idioms) remains uninvestigated. I wonder if the authors have evaluated their model on other types of implicit expressions, such as irony or casual dialogues.\n2. While the paper performs some ablation studies, it primarily uses Sentence-BERT as the embedding model. Exploring alternative text encoders could provide a more comprehensive understanding of ImpScore’s design choices. They may also utilize some contrastive learning-based models, such as SimCSE. \n3. I am also concerned about their data synthetic approach. The reliance on GPT-3.5 for generating explicit counterparts of implicit sentences could introduce unintended biases, particularly if the model generates stereotypical or simplified responses (we can also see the average length of explicit sentences is much shorter than implicit ones). I wonder if the authors verified the quality of the generated explicit sentences.\n4. I also expect some model comparisons. For example, they can use their training data as a binary classification task and compare the classification accuracy between the simple classifier and their proposed model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yih7u3WFH8", "reviewer": "Reviewer_bQJm", "summary": "This paper introduces a scalar metric to compute the implicitness score for sentences known as IMPSCORE. They also contribute a dataset of 56,290 pairs of implicit and explicit sentences used to train the IMPSCORE model. The metrics reliability is studied through a user study on OOD data. Moreover, the authors apply IMPSCORE on various hate speech benchmarks and also compare the performance of three language models - gpt-4-turbo, llama-3.1-8b-instruct and OpenAI Moderation - on these datasets with varying levels of implicitness.", "questions": "1. Why is the same hyper-parameter $\\gamma_1$ used for both $L_{imp} (s_1,s_2)$ and $L_{imp} (s_1,s_3)$?\n2. Is the data split into train, validation and test sets stratified i.e. does it maintain the ratio of positive and negative pairs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper tries to address an important problem of understanding the implicitness level of language through a novel framework to measure it based on the divergence between semantic meaning and pragmatic interpretation of a sentence. It also introduces a novel dataset consisting of diverse samples (validated from a linguistic expert) to train the model used in computing the implicitness score.\n2. Overall, the study is comprehensive. Authors justify their choice for the final setting by performing sufficient ablations in terms of the design choices and hyper-parameters. They also conduct a user study to understand correlation of the metric with human judgement.\n3. The paper is technically sound and is well written in general.", "weaknesses": "1. The number of sentences used for human judgment correlation (40 - 4 sentences across 10 topics) are not sufficient enough to examine the generalization ability of the IMPSCORE metric. Expanding this set is necessary to improve the reliability of the metric.\n2. Even though some justification is provided to measure the pragmatic distance (in the Introduction section), it is not entirely clear on why it is important. The ablation experiments also does not seem to affect the pragmatic distances, further casting doubts on its significance. Additional explanation for this sub-metric would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)