ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Sentences

Authors: Yuxin Wang, Xiaomeng Zhu, Weimin Lyu, Saeed Hassanpour, Soroush Vosoughi
Primary area: datasets and benchmarks
Keywords: implicit language, pragmatics, learnable metric, text evaluation, automatic evaluation, explicit language

Abstract:
Handling implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users. Despite its importance, the absence of a metric for accurately measuring the implicitness of language significantly constrains the depth of analysis possible in evaluating models' comprehension capabilities. This paper addresses this gap by developing a scalar metric that quantifies the implicitness level of language without relying on external references. Drawing on principles from traditional linguistics, we define "implicitness" as the divergence between semantic meaning and pragmatic interpretation. To operationalize this definition, we introduce ImpScore, a reference-free metric formulated through an interpretable regression model. This model is trained using pairwise contrastive learning on a specially curated dataset consisting of (*implicit sentence*, *explicit sentence*) pairs. We validate ImpScore through a user study that compares its assessments with human evaluations on out-of-distribution data, demonstrating its accuracy and strong correlation with human judgments. Additionally, we apply ImpScore to hate speech detection datasets, illustrating its utility and highlighting significant limitations in current large language models' ability to understand highly implicit content. Our metric is publicly available at https://github.com/audreycs/ImpScore.

Decision: accept (spotlight)
Reviewer scores: 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730512003159

thread_type:
official_review

<thread_object>
creator: reviewer_bqjm

<content>
summary:
This paper introduces a scalar metric to compute the implicitness score for sentences known as IMPSCORE. They also contribute a dataset of 56,290 pairs of implicit and explicit sentences used to train the IMPSCORE model. The metrics reliability is studied through a user study on OOD data. Moreover, the authors apply IMPSCORE on various hate speech benchmarks and also compare the performance of three language models - gpt-4-turbo, llama-3.1-8b-instruct and OpenAI Moderation - on these datasets with varying levels of implicitness.

strengths:
1. The paper tries to address an important problem of understanding the implicitness level of language through a novel framework to measure it based on the divergence between semantic meaning and pragmatic interpretation of a sentence. It also introduces a novel dataset consisting of diverse samples (validated from a linguistic expert) to train the model used in computing the implicitness score.
2. Overall, the study is comprehensive. Authors justify their choice for the final setting by performing sufficient ablations in terms of the design choices and hyper-parameters. They also conduct a user study to understand correlation of the metric with human judgement.
3. The paper is technically sound and is well written in general.

weaknesses:
1. The number of sentences used for human judgment correlation (40 - 4 sentences across 10 topics) are not sufficient enough to examine the generalization ability of the IMPSCORE metric. Expanding this set is necessary to improve the reliability of the metric.
2. Even though some justification is provided to measure the pragmatic distance (in the Introduction section), it is not entirely clear on why it is important. The ablation experiments also does not seem to affect the pragmatic distances, further casting doubts on its significance. Additional explanation for this sub-metric would be beneficial.

questions:
1. Why is the same hyper-parameter $\gamma_1$ used for both $L_{imp} (s_1,s_2)$ and $L_{imp} (s_1,s_3)$?
2. Is the data split into train, validation and test sets stratified i.e. does it maintain the ratio of positive and negative pairs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590017125

thread_type:
official_review

<thread_object>
creator: reviewer_kert

<content>
summary:
This paper introduces ImpScore, a reference-free metric designed to quantify the level of implicitness in human language. The model for ImpScore is trained via contrastive learning by calculating the divergence between the semantic (literal) meaning and the pragmatic (contextual) interpretation of a sentence. They collected a training data with 112,580 pairs. Through extensive experiments, ImpScore demonstrates strong correlation with human judgments. They also show the utility in evaluating hate speech detection datasets, revealing limitations in current large language models' ability to handle implicit content.

strengths:
1. The paper tackles a rarely addressed problem, quantifying implicitness in language. It offers a new approach to measuring implicitness that moves beyond binary or purely lexical classifications.
2. ImpScore is a reference-free metric, which distinguishes it from previous metrics that often rely on external references or manual annotations.

weaknesses:
1. Limited scope of implicit languages. The paper’s dataset includes synthetic data generated by GPT-3.5 and focuses primarily on specific types of implicit language (e.g., hate speech). However, ImpScore’s generalizability across broader types of implicit expressions (such as indirect requests or cultural idioms) remains uninvestigated. I wonder if the authors have evaluated their model on other types of implicit expressions, such as irony or casual dialogues.
2. While the paper performs some ablation studies, it primarily uses Sentence-BERT as the embedding model. Exploring alternative text encoders could provide a more comprehensive understanding of ImpScore’s design choices. They may also utilize some contrastive learning-based models, such as SimCSE. 
3. I am also concerned about their data synthetic approach. The reliance on GPT-3.5 for generating explicit counterparts of implicit sentences could introduce unintended biases, particularly if the model generates stereotypical or simplified responses (we can also see the average length of explicit sentences is much shorter than implicit ones). I wonder if the authors verified the quality of the generated explicit sentences.
4. I also expect some model comparisons. For example, they can use their training data as a binary classification task and compare the classification accuracy between the simple classifier and their proposed model.

questions:
* How does the performance of ImpScore vary across different types of implicit language (e.g., indirect requests, cultural idioms, irony)? Are there specific types that it struggles with?
* Have you examined any potential biases and quality of the synthetic data?
* Can you compare with existing methods or other baselines for implicitness? For example, the methods mentioned in Related Work (e.g. Garassino et al. (2022) and Vallauri & Masia (2014))?
* I would like to see if you can implement ImpScore to the LLMs' abilities to generate implicit expressions. Similar to the data creation in Section 6.1, you can prompt different LLMs to generate 4 sentences with varying implicitness levels. Then, you calculate the ImpScore for these generations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675221006

thread_type:
official_review

<thread_object>
creator: reviewer_czyp

<content>
summary:
This paper introduces IMPSCORE, a novel metric for quantifying the implicitness level of language by measuring the divergence between semantic meaning and pragmatic interpretation. The authors develop an interpretable regression model trained through pairwise contrastive learning on a newly created large dataset of 112,580 sentence pairs. The metric processes sentences using a text encoder to generate embeddings, maps semantic and pragmatic features into separate spaces, and calculates implicitness scores based on the divergence between these features after space transformation.

Experimental results validate the reliability of IMPSCORE. The authors also conduct a user study comparing the metric's assessments with human evaluations on out-of-distribution data, showing strong correlation with human judgments. They also demonstrate practical applications by analyzing implicitness levels in hate speech detection datasets and evaluating performance of several large language models across different implicitness levels, revealing that model performance degrades as implicitness increases.

strengths:
- The paper is well motivated and presents a clear operational definition of implicitness and develops a concrete methodology for measuring it. The paper is well written and easy to follow. 
- Experimental results show strong support for the IMPSCORE as a reliable metric. The authors further validate it through careful ablation studies, user studies, and practical applications in analyzing both datasets and model performance.
- The training dataset is substantial and diverse (and is a contribution in itself), drawing from multiple domains and types of implicit language, while the evaluation includes both in-distribution and out-of-distribution testing.

weaknesses:
- The metric is developed and tested only for English language content. It is not clear how this would translate to other languages or whether this metric will work in cross-lingual settings.
- While well-designed, the user study involves only 10 participants evaluating 10 questions each, which is a relatively small sample size for validating a metric intended for broad use.

questions:
- I'm curious if you could run some experiments to test the metric's performance on other languages? An easy (though, not perfect) way would be to look at translated sentence pairs across multiple languages to assess its language independence and potential biases?
- I'd love to see some qualitative error analysis for the metric. Are there any patterns that emerge in the sentences on which IMPSCORE isn't accurate?
- Is IMPSCORE just as reliable on longer sentences as on shorter ones?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732642995262

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We just uploaded a new revision of our paper where we corrected a number of format errors in table and figure captions.

-----

We sincerely thank all reviewers for their thoughtful and detailed feedback on our work. Reviewers highlighted the significance of the problem addressed in this paper—quantifying the level of implicitness in language—describing it as "an important problem" (bQJm) and "rarely addressed" (keRt) in existing literature. They also commended our novel approach to measuring implicitness as "operationally clear and concrete" (czyP). Moreover, two reviewers (czyP, bQJm) commended the robustness and comprehensiveness of our methodology, particularly noting the sufficiency of our experiments, which include ablation studies, user studies, and practical applications.

The constructed training dataset was similarly praised for its significant and diverse nature, with reviewers describing it as “a novel dataset consisting of diverse samples” (bQJm) and “substantial and diverse” (czyP), underscoring its value as an additional contribution to the field. Additionally, reviewer (keRt) highlighted the reference-free nature of ImpScore, emphasizing how it "distinguishes it from previous metrics."

-----

We have replied to reviewers' every question and concern and conducted additional experiments to incorporate their useful feedback. Key efforts include:
1. *More User Study (reviewer czyP and bQJm)*: We followed reviewers' suggestions and conducted an additional user study with 10 new out-of-distribution (OOD) topics. The results are consistent with our conclusions in the original manuscript.
2. *Breakdown on ImpScore’s performance (reviewer czyP and keRt)*: we conducted further analysis on ImpScore’s performance on different implicitness types and sentences with different features such as length and sentiments. 
3. *Additional Application of ImpScore (reviewer keRt)*: We applied ImpScore to the generated text of 7 LLMs (GPT-3.5, GPT-4o, Llama-3.2-Instruct, Mistral-7B, Mistral-Nemo, Claude-haiku, and Claude-sonnet) and presented the comparison on their implicitness levels.

We have expanded our Appendix and extended our discussion in the Conclusion and Future Work section to incorporate these changes and reviewers' feedback. 

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work introduces ImpScore, a new metric for measuring the implicitness of language in text. Specifically, ImpScore measures the difference between the surface semantic meaning and the expected pragmatic interpretation of a given text by training a regression model on a set of (embeddings of) sentence pairs. To train this model, the paper also proposes a large dataset of these sentence pairs that map between implicit and explicit phrasings of a sentence's meaning. 

The experimental results show that the ImpScore model is very accurate on the proposed dataset (i.e., good at distinguishing between implicit and explicit sentences) and correlates well with human judgments. The experiments also consider practical settings for measuring the implicitness of language with hate-speech detection by LLMs (where models perfrom worse at detecting hate-speech at high levels of implicitness).

Strengths:
- Proposes a clear, operational definition of implicitness and provides a large dataset with examples of implicit and explicit sentence pairs following this definition (czyPs, bQJm).  
- The paper is well-written and well-motivated (czyP, bQJm).
- The experimental results indicate that this is a reliable metric for measuring implicitness, based on agreement with human judgments and ablations. The paper also suggests how this score can be used in practical settings by ablating the effect of implicitness on hate-speech detection (czyPs).

Weaknesses:
There are no major weaknesses, in my opinion. Minor weaknesses include:
- The use of synthetic data during dataset creation (keRt), though the paper and author response indicate that significant effort was taken to verify the automatically generated data that went into the new dataset.
- It is unclear if the dataset and model will be released with the paper. I recommend that the authors release these artifacts, which will greatly strengthen the contribution of this work.

Other weaknesses, such as the limited scope of the metric (czyPs, keRt) and small sample size for user study (zyPs, bQJm), were addressed in the author response. I appreciate the translation-based multilingual evaluation here and would love to see follow-up work in this area with annotated data in other languages.