Natural Language Decompositions of Implicit Content Enable Better Text Representations
Authors: Alexander Hoyle
Rupak Sarkar
Pranav Goel
Philip Resnik
Venue: EMNLP 2023 Main (2023)
Keywords: computational social science
implicit
decompositions
decompose
propositions
generation
political science
sentence embeddings
clustering
TL;DR: We use LLMs to infer the explicit and implicit propositions communicated by natural language utterances, then use them in downstream applications.

Abstract: When people interpret text, they rely on inferences that go beyond the observed language itself. Inspired by this observation, we introduce a method for the analysis of text that takes implicitly communicated content explicitly into account. We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed, then validate the plausibility of the generated content via human judgments. Incorporating these explicit representations of implicit content proves useful in multiple problem settings that involve the human interpretation of utterances: assessing the similarity of arguments, making sense of a body of opinion data, and modeling legislative behavior. Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP and particularly its applications to social science.

Natural Language Decompositions of Implicit Content
Enable Better Text Representations
Alexander Hoyle*
Rupak Sarkar*
Pranav Goel
Philip Resnik
Computer Science
Computer Science
Computer Science
UMIACS, Linguistics
University of Maryland
{hoyle, rupak, (cs. umd. edu,
resni edu
Abstract
Far in embedding space
When people interpret text, they rely on infer- ences that go beyond the observed language introduce self. Inspired by this observation, a method for the analysis of text that takes im- plicitly communicated content explicitly into account. We use large language model to produce sets of propositions that are inferen- tially related the text that has been observed, then validate the plausibility of the generated content via human judgments. Incorporating these explicit representations of implicit con- tent proves useful in multiple problem settings that involve human interpretation of utter- ances: assessing the similarity of arguments, making sense a body of opinion data, and modeling legislative hehavior. Our results sug- gest that modeling the meanings behind ob- served language, rather than the literal text alone, is valuable direction for NLP and par- ticularly its applications to social science."
I'll roamned and rambled and followed my footstaps. To the sparkling
Federal lands and wallers not extraction. Publiic lands are national tressures that should be protected for generations, auctioned incustryys highest bidders.
her diamond deserts, all. around me, voice was sounding. This land was made and me!
d 0.84
Happy #NationelParkWWer
Lat's get out from behind our screens and enjoy this beautiful land home!
Closer in embedding space
40.=0.45 d..- 0.77
Natural Park Week is
Federal lands and waters should protected fron fossil fuel extraction.
tine to appreciate
natural wonders.
Preserving natural resaurces generations is impurtant.
Exploring nature is beneficial.
Figure 1: Example showing a pair of tweets from legis- lators along with meleled propositions. Em- beddings over observed tweets have high cosine dis- tance, while embeddings over (different types of) propo- sitions place them closer each other (see $ 5).
1 Introduction
The meaning and import of an utterance often underdeternined by the utterance itself. Human interpretation involves making inferences based on the utterance understand what communicates (Bach, 1994; Hobbs et al., 1993). For the disci- plines and applications that are concerned with making sense of large amounts of text data, hu- man interpretation of each individual utterance is intractable. Some NLP methods are designed to fa- cilitate human interpretation of text by aggregating lexical data; for example, dictionaries map words constructs (e.g., Pennebaker al., 2001), and topic models discover interpretable categories in a form of automated qualitative content analysis (Grimmer and Stewart, 2013; Hoyle et al., 2022). Much of the time, though, these techniques oper- ate over surface forms alone, limiting their ability
to capture implicit content that facilitates human interpretation. While contextual embeddings are a step in this direction, these representations remain dominated by lexical content (Zhang et al., 2019).
In this work, we introduce a framework for the interpretation of text data at scale that takes im- plicitly communicated content more explicitly into account. Specifically, we generate sets of straight- forward propositions that are inferentially related to the original texts. We refer to these as inferential decompositions because they break the interpre- tations of utterances into smaller units. Broadly speaking, we follow Bach (2004, p. 476) in distin- guishing "information encoded in what is uttered" from extralinguistic information. Rather than being logical entailments, these generations are plausi- ble entailments of the kind found in discussions of textual entailment (Dagan et al., 2009) and implica- ture (Davis, 2019). This idea relates to decomposi- tional semantics (White et al., 2020), but eschews
Equal contribution. 'Code and data availahle at g git. cww/coom///aooo /hhho//h infernttia decomposoossiioeo-o-oo tions

---

linguistically-moote mottivated annotations in favor of more open-ended structure that facilitates direct interpretation in downstream applications.
Human utterances communicate propcaitions that may or may not be explicit in the literal meaning of the utterance. For each utterance, state the implicit and explicit propositions communicated by that ut- terance in a brief list. Implicit propositions may be inferences about the subject of the utterance or about the perspective of its author. All generated proposi- tions should be short, independent,, and written in direct speech and simple sentences.
We perform this process using large language model (LLM), specifying practitioner protocol for crafting exemplars that capture explicit and im- plicit propositions based on utterances sampled from the corpus of interest, thereby guiding the language model to do the same ($ 2). In designing our approach, we observe that the inherent sparse- ness of text data often makes it useful to represent text using ower-dimensional representations. Ac- cordingly, our notion of decomposition (and the generation process) encourages propositions that contain simple language, making them both easier interpret and more amenable standard em- bedding techniques. We observe that the viability of this technique rests on models' ability to reli- ably generate real-world associations (Petroni et al., 2019; Jiang et al., 2020; Patel and Pavlick, 2022),
# #
INPUT: { (uttteracee }
OUTPUT: { iinference 1 inference 2 I...
Figure 2: A condensed version of our prompt to models.
tion of this sentence in context involves deriving the speaker's communicative intent from the ex- pression itself, together with an implicit universe of propositions outside the utterance, w-world knowledge, hypotheses about the speaker's 's beliefs, and more. In this case, some elements of U might be factual background knowledge such as "The U.S. shares a border with Mexico" that is not communi- cated by e itself. Other elements might include im- plicitly communicated propositions such as "immi- gration should be limited". Propositions from this latter category. consisting of relevant plausible en- tailments from the utterance, we denote as R C U.?
well as their capacity to follow instructions and mirror the linguistic patterns of provided exemplars (Brown et al., 2020; Liu et al., 2022).
We situate our approach within the text-as-date framework: the "systematic analysis of large- scale text collections" that can help address substantive problems within a particular discipline (Grimmer and Stewart, 2013). First, we validate our method with human annotations, verifying that generated decompositions plausible and easy read, and we also show that embeddings of these inferential decompositions can be used improve correlation with human judgments of argument similarity ($ 3). We then turn to two illustrations the technique's utility, both drawn from real-world, substantive problems in computational social science. The first problem involves making sense of the space of public opinion, facilitating human interpretation of set of comments responding to the US Food and Drug Administration's plans to authorize COVID- 19 vaccine for children ($ 4). The second involves the question of how likely two legislators are to vote together based on their tweets ($ 5).
We are motivated by the idea that, if human in- terpretation includes the identification of plausible entailments R. based on the expressed automated text analysis can also benefit from such inferences, particularly in scenarios where understanding text goes beyond "who did what to whom". The core of our approach is to take expression and explicitly represent, language, a body of propositions that are related inferentially it.
Operationally, our method is as follows:
1. For the target dataset, randomly sample a small number of items (e.g., tweets).
2. Craft explicit and implicit propositions rele- vant to the items (inferential decompositions expressed language) following the instruc- tions in appendix A.2 to form exemplars. Prompt a large language model with our in- structions (fig. 2) and these exemplars.
4. Confirm that a random sample of the gener- ated decompositions are plausible ($ 3).²
2 The Method and its Rationale
5. Use the decompositions in the target task.
The key idea in our approach is to go beyond the observable text to explicitly represent and use the kinds of implicit content that people use when in- terpreting text in context.
*Cr. Bach's characterization pragmatic information as "relevant to hearer's determination of what the speaker is communicating ... generated by, or least made relevant by, the act of uttering it". Similarly, interpretation as abduction (Hobbs et al., 1993) is an inferential process that starts from the utterance.
Consider the sentence Build the wall!. Follow- ing Bender and Koller (2020), human interpreta-
'While this step is not necessary, we recommend validating generations, particularly in sensitive use cases.

---

Decompositons Implicit
Paraphrase Explicit
Baseline
Explicit
Both
60%
L
Implicit
Examples
Implicit
Arg Facet
54.01
59.98
58.84
59.79
40%
BWS Arg.
53.74
55.32
54.63
55.09 .9
UKP Aspect
50.04
52.86
53.40
53.39
20%
Explicit
Twitte-STTS 69.05
72.69
69.92
71.92
0%
SICK-R 80.59
81.39
92.92
80.00
Definitely Probably
May or Probably Definitely not
STS-B
83.42
81,27
77.59
81.76
may Plausibility
Table 1: Measured against human similarity judgments, similarities computed using inferential decompositions generally better than similarities computed using the observable text (Spearman p). Improvements from the baseline are underlined.
Figure Human-judged plausibility of inferentially- related propositions for different inference types. The vast majority of inferred propositions are plausible.
To ground these propositions in a context, we develop domain-dependee user guidelines (step 2) for exemplar creation, limiting the focus to infer- ences about the utterance topic and its speaker (and explicit content, appendix A.2).*
in some circumstances (Maynez et al., 2020; Cao et al., 2022; Ji et al., 2023), but can also exhibit high factuality relative to prior methods (Goyal et al., 2023). This leads to the question: Do lan- guage models reliably produce plausible explicit & implicit propositions?
Continuing with the example e of Build the wall!, the method might generate the augmented represen- tation R- = {A US border wall will reduce border crossings, Illegal immigation should be stopped, ...) as plausible inferences about the speaker's perspective (an actual example appears in fig. 1). Because the propositions are expressed in simple language, they are casier to represent with standard embedding techniques: when using K K-means clus- tering to over the embedded representations, the clusters are far more readable and distinct than baselines ($ 4). Hence, this approach facilitates interpretation of text data at scale.5
Second, since similarity over embedded text un- derpins both the text analysis ($ 4) and downstream application ($ 5) that validate our approach, we measure the correlation of embedding and ground- truth similarities for several standard semantic- textual-similane tasks. Assuming that human sim- ilarity judgments make use of both explicit and implicit inferences, also ask whether includ- ing such information in sentence representations improve automated estimates of similarity.
Generation of decompositions. We generate in- ferential decompositions for datasets across a di- verse set of domains. Our method should effec- tively encode stance, often an implicit property of text, so we adopt three argument similarity datasets (Argument Facets from Misra et al. 2016; BWS from Thakur et al. 2020- 10; and UKP from Reimers et al. 2019). As a reference point, we also select several standard STS tasks from the Massive Text Embedding Benchmark, which are evaluated for their observed semantic similarity (MTEB, Muennighoff et al., 2022). 6 We also use the datascts underpinning our analyses in sections 4 and 5: public commentary on FDA authorization of COVID- 19 vaccines, and tweets from US senators. use the instruc-
3 Generation Validity
Text analysis methods used in the computational social sciences have known issues with validity. The interpretation of unsupervised models is par- ticularly fraught-leading to potentially incorrect inferences about the data under study (Baden et al., 2021). We validate the two components of our approach: the quality of the generations and the similarity of their embeddings.
First, if we want to use the generated inferen- tial decompositions in downstream applications, is important that they are reasonable inferences from the original utterance. Large language mod- els are known hallucinate incorrect information
To generate decompositions, tions and exemplars in appendix A.4, further di- viding the exemplars into explicit and implicit
'In principle, alternative guidelines meeting other desider- ala are possible, and can validated with same process. "SScll"" refers to datasct sizes that render human analysis too costly, and is constrained only by computational budget.
"For the Twitter dataset (Xu et al., 2015), we use the origi- nal 5-scale Likert similarity, not the binary scores in MTEB.

---

categories, as determined by our guidelines (ap- pendix A.2). See fig. 1 for an illustration. For the human annotation, we sample 15 examples each from STS-B B (Cer et al., 2017), BWS, Twitter-STS (Xu et al., 2015), and our two analysis datasets. The language model is text sentt-dvinni---e (INSTRUCTGPT, Ouyang et al., 2022), and em- beddings use all -mnnt- base v2 (Reimers and Gurevych, 2019). Throughout this work, use nucleus sampling with p 0.95 (Holtzman et al., 2019) and a temperature of 1.
Method
Silhouettet
CH²
DB
Comments
0.052
247
3.41
Sentences
0.042
219
3.74
Decompositions (ours)
0.090
329
3.03
Comments
0.035
172
3.28
25
Sentences
0.035
152
3.64
Decompositions (ours)
0.096
239
2.80
Comments
0.029
104
3.26
50
Sentences
0.042
93
3.51
Decompositions (ours)
0.114
153
2.73
Table Intrinsic metries of clustering quality. On a random subsample of 10k comments, sentences, and decompositions, the intrinsic metrics rank our model higher both for fixed number of clusters (holded) and across clusters (underlined). CH is the Calinski- Harabasz Index and DB is Davics-Bouldin
Human annotation of plausiblity. In answering the first question, a set of 80 crowdworkers anno- tated the extent to which a decomposition is rea- sonable given an utterance- feeeranceefrom" "I Definitely" to "5 Definitely not"-and and whether it adds new information to that utterance (full instructions in ap- pendix 1.5.1, recruitment details in appendix A.5). We majority-code both answers, breaking ties for the plausibility scores with the rounded mean.
on the argument similarity datasets over the embed- ding baseline (table 1), where pairs are annotated for the similarity of their position and explanation with respect to a particular topic (e.g., supporting minimum wage increase by invoking inflation). In this task, explicit decompositions resemble the implicit ones-annotators give similar proportions of "probably reasonable" scores to both types." 8
In the vast majority of cases (85%-93%), the generated decompositions are at least "probably" reasonable (fig. 3). As expected, the plausibility of implicit inferences tends to be less definite than either a paraphrase baseline or explicit inferences- but this also speaks to their utility, as they convey additional information not the text. Indeed, im- plicit inferences add new information 40% of the time, compared to 7% for explicit inferences (and 15% for paraphrases). As further validation to sup- port the analyses in $ 5, professor of political science annotated the implicit decompositions of observable legislative tweets, finding 12 of 15 to be at least "probably reasonable," two ambiguous, and one "probably not reasonable".
On the conversational Twitter-STS dataset, the method also shows improvement, likely due to the colloquial and contextualized nature of the origi- nal utterances." Unsurprisingly, on standard STS benchmarks, the implicit method fares worse, likely because it over gver-generalizes from specific instances that reduce precision, even they remain correct: "A person is mixing a pot of rice." + "The person is preparing food." Indeed, our method is intended to create such generalizations to assist in interpretabil- ity at scale, not to improve STS tasks. 10
Inferential Decompositions Help Theme Discovery
Semantic Textual Similarity. Here, we measure whether our method can improve automated mea- surements of semantic sextual similarity. For each example in each of the STS datasets, we form a set S, - fs,, $1, 2. | consisting of the orig- inal utterance and n decompositions. As baseline, we computed the cosine similarity comparisons be- tween embeddings of the original sentences 8i, Sj. obtained using a aal1-pponn hampmettbase-ee Pairwise comparisons for expanded representations S,, Sy.
Since the augmented representations we are cre- ating go beyond the observed text to inferentially related propositions, we expect it to be useful problem settings where observable text is the "tip of the iceberg" - intuitively, problems where it is par- ticularly important to consider not only what was said, but what is behind what was said. Specifically,
were scored by concatenating the embedding for s; with the mean of the embeddings for the st...
FFrr example, from a 38-word utterance, explicir: "The minimum wage should be higher than S7.25", implicit: "The current minimum wage insufficient"
Our method substantially improves correlation
'Even the "explicit" setting generates "Chris Kelly has died" from the original "RIP To Mac daddy Chris Kelly". "Alterig the prompt to support STS by instcad generating paraphrases leads to state-of-the tf-th-art results, appendix A.3.
'Approaches designed to measure the similarity of sets of vectors gave similar results(c.g.. Zhelezniak et al., 2020)

---

Evaluation Time
Membership Score
Relatedness Judgement
4
60
0.75
40
0.50
2-
0.25
0
Comments
Sentences
Decompositions
Comments
Sentences
Decompositions
Comments
Sentences
Decompositions
Figure 4: Human evaluation of clustering outputs. Clusters decompositions (our method) take significantly less time to review and are more distinctive from one another. Relatedness scores are high for the observed comments, but significantly worse membership identification scores reveal this to be a spurious result owed to the topical homogeneity of the dataset (all comments are about COVID vaccines). All differences are sigmificant at p 0.05 except membership scores between comments and sentences and evaluation times for sentences and decompositions.
we ask whether the representation of comments' explicit & implicit propositions lead to improved discovery of themes in corpus of public opinion. contentious is- fits that description: expressions of opinion are generated from a more complex interplay of personal values and background beliefs about the world. This is a substantive real-world problem; in the US, federal agencies are required to solicit and review public comments to inform policy.
Dataset. We randomly sampled 10k responses from a set of about 130k responses to request for comments by the FDA regarding child vaccine authorization." 12 Our dataset contains often-lengthy comments expressing overlapping opinions. collo- quial language, false beliefs or assumptions about the content or efficacy the vaccine, and a general attitude of vaccine hesitancy (see the "Comment" column of table 10 for examples).
Understanding public opinion
We generate 27,848 unique inferential decompositions from the observable comments at an average of 2.7 per comment. We use K-means clustering to identify categories of opinion, vary- ing K. Specifically, two authors created 31 ex- emplars from seven original comments from the dataset that exhibit mixture of implicit proposi- tion types (uable 10). In addition clustering the observed comments themselves as baseline, as a second baseline we split each comment into its overt component sentences and cluster the full set of sentences. This results in 10k comments, 45k sentences. and 27k decompositions.
Method.
Our approach is related efforts showing that intermediate text representations are useful for in- terpretive work the computational social sciences and digital humanities, where they can be aggre- gated to help uncover high-level themes and narra- tives in text collections (Bamman and Smith, 2015; Ash al., 2022). In a similar vein, cluster in- ferential decompositions of utterances that express opinion to uncover latent structure analogous the discovery of narratives in prior work. We analyze a corpus of public comments to the U.S Food and Drug Administration (FDA) concerning the emer- gency authorization of COVID-19 vaccinations in children in terms of content and goals, our ap- plication resembles the latent argument extraction of Pacheco et al. (2022), who, building on content analysis by Wawrzuta et al. (2021), clustered tweets relating COVID-19 19 to facilitate effective anno- tation by a group of human experts. In our case, we not only discover valuable latent categories, but we are able to assign naturalistic category labels automatically in an unsupervised way."!
Automated Evaluation. We lack ground truth labels for which documents belong to which clus- ter,
first turn to intrinsic metrics of clus- ter evaluation: the silhouette (Rousseeuw, 1987),
12 Progulations oons gow.goooooooooooooo 1088-0001 Comments are public and users can elect to post anonymously. We obtained permission from the agency to use these data. We will not directly release the data out of caution, because the original authors did not explicitly consent to redistribution, but we refer interested researchers to https: / /ww.. regulation ions. gov/bull kdown oad and Pampell (2022). Note that some comments can contain upsetting language, which we communicated to annotators.
"In preliminary experiments, topic model outputs were of
mixed quality.

---

Source
Cluster 1
Cluster
Cluster
Desompasttio Clusters
The vaccine he harmful children. Children are vulnerable to the long-term effects of the vaceine.
Natural immanity is better than vaccine- indaced immumity.
The government attempting control citicens.
Natural immunity important for chil- dien develop.
American rights are being croded.
Vaceines are under trials their side effects are unknown.
Thee government should investigate use of to protect our children. The US values freadom and choice.
Natural immunity is more effective.
vaccime has been properly lested. There is not enough data on the vaccine's long effects.
immunity
lightng
covid vimss
People should develop immunities stay healthy.
Natural immunity should he reflected in science.
Expeit Narrativss
The vaccine not properly tested, was
Natural methods of protection are better
Lack of trast the government
Wawrzata et al. (2021)
developed too quickly
than the vaccine
Crondworker Label
Long-term vaceine worries
Natural Immunity
Control by government
Table 3: For public commentary datasets about COVID-19 19, clusters of inferential decompositions (our approach, top row) align with arguments discovered independently by Wawrzuta et al. (2021) (middle row). The overlap is strong despite the commentary coming from different platforms (Government website & Facebook) and countries (US & Poland). In addition, outside of the exemplars passed to the LLM (table 10), our approach is also entirely unsupervised. In the bottom row, we show illustrative label for each cluster from a crowdworker.
(Caliński and Harabasz, 1974),
that comment clusters are less distinct. Moreover, the comprehension time for comments is signifi- cantly longer than for sentences and decomposi- tions (Evaluation Time in Fig 4), taking over 50% longer to read. On the other hand, clusters of decompositions strike a balance: they obtain mod- erately strong relatedness scores, can be understood the quickest, and are highly distinct.
and Borris-Bouudii (Davies and Bouldin, 1979) scores; roughly speaking, these variously measure the compactness and distinctiveness of clusters. Since metrics can be sensitive to the quantity of data in a corpus (even if operating over the same content), we subsample the sentence and decompo- sition sets to have the same size the comments (10k). 13 Clusters of decompositions dramatically outperform clusters of comments and sentences across all metrics for each cluster size-in fact, independent of cluster size, the best scores are ob- tained by decomposition clusters (Table 2).
Convergent Validity. Although further explo- ration is necessary, we find that our crowdworker- provided labels can uncover themes discovered from classical expert content analysis (table 3). For example, two crowdworkers assign labels contain- ing the text "natural immunity" to the cluster in table thi aligns with the theme NATURAL IM- MUNITY IS EFFECTIVE discovered in Pacheco et al. (2023) (through a process requiring more human effort) and a similar narrative in Wawrzuta et al. (2021). Meanwhile, this concept does not appear anywhere in the crowdworker labels for the base- line clusters of sentences or comments.
Human Evaluation. Performance on intrinsic metrics does not necessarily translate to usefulness, we also evaluate the cluster quality with hu- man evaluation. After visual inspection, we set K 15. For a given cluster, we show an annotator four related documents and ask for a free-text label describing the cluster and a 1-5 scale on perceived "relatedness". We further perform a membership identification task: an annotator is shown an unre- lated distractor and held-out document from the cluster, and asked to select the document that "best fits" the original set of four. Participant information and other survey details are appendix A.S.
Decompositions support analyses of legislator behavior
Having established our method's ss ability to facilitate human interpretation of text data ($ 3 and $ 4), we now examine the usefulness of generated inferen- tial decompasitions in a very different downstream application where the relevant analysis of text is likely to involve more than just overt language con- tent. Here, we model legislator behavior using their speech (here, tweets), asking: does the similarity between legislators' propositions help explain the similarity of their voting behavior?
Results are shown Fig 4. While comment clus- ters receive a higher relatedness score, this is likely due to the inherent topical coherence of the dataset: there are often several elements of similarity be- tween any two comments. A lower score in the membership identification task, however, indicates
13 Results are similar for the silhouette and Davies-Bouildin scores without subsampling: the Calinski-Harabasz Harabasz is better for the sentences.

---

Legend
> Supreme Court should be saved from threats to repraductive rights
Democrat
Republican
Kavaneughi's confirmation jespardize repraductive rights
> Roe Wade affirms woman's right choose
> Roe should upheld
Elective aborttons after 20 weeks should be prohibited
Access abortion should protected
Abortion after 20 weeks is unethical
The US should align abortion laws with other countries
Figure 5: t-SNE (van der Maaten and Hinton, 2008) visualization of the embedding space of implicit inferred decompositions found the "Abortion" topic from legislative tweets. * and are two clusters selected from 10 clusters obtained using K X-manss; * (59% Democrat) talks about the role of judiciary in reproductive rights, while the * (73% Republican) talks about banning late stage abortion. Our method leads to more compact (better Silhouette, CH, and DB scores compared tweets) and easier interpret clusters that help with narrative discovery,
Traditional theories of homophily (McPherson al., 2001) suggest that shared properties (e.g.. electoral geography, Clark and Caro, 2013) in- crease the likelihood that two legislators vote the same way - and many research questions have centered around such co-voting (Ringe et al., 2013; Peng et al., 2016; Wojcik, 2018). In preliminary experiments found when modeling co-voting, text similarity between legislator was valuable predictor of co-voting (Goel, 2023). Here posit that such modeling can further benefit by going beyond surface text to capture inferentially related propositions related to viewpoints.
tween a pair of legislators i, using mixed effects regression model, controlling for the random ef- fects of both actors under consideration. The co- vote rate the number of times the legislators vote the same way - yea or nay - divided by their total votes in common within a legislative session.
Az X.g x
E log
3o + a æ.s
(1)
8. are regression cofficients, and a; by model random effects for legislators i.j.
Ziy n-dimensional feature vector, where each element similarity score that captures a type of relationship between legislators and j. While these features have traditionally included state membership, party affiliation, or Twitter con- nections (Wojeik, 2018) 8) or joint press releases (Des- marais et al., 2015); we consider the language sim- ilarity between pairs of legislators based on our proposed method.
Consider Figure 1. While the observable tweets themselves are not similar in their surface form, they express similar views regarding the impor- tance of the preservation of nature, something that would be clear to a human reader interpreting them. This can captured by inferential decompositions that reveal authors' viewpoints toward the issue.
Dataset. The goal is represent each legislator using their language in such a way that we can measure their similarity to other legislators-our informal hypothesis of the data-generating process is that latent ideology drives both vote and speech behavior. Specifically, we follow Vafa et al. (2020) by using their tweets; data span the 115th 117 sessions of the US Senate (2017-2021).*
We operationalize our method for this purpose by creating exemplars that contain inferences about the utterance topic and perspective (table 12 in appendix). This guides the language model toward domain-relevant facets of similarity between the texts, and therefore the authors of the texts, that may not be apparent from the surface form.
Model Setup. For the task of modeling co-vote behavior, we extend the framework introduced by Ringe et al. (2013) to incorporate individuals' lan- guage into the model. At high level, we oper- ationalize legislator homophily by measuring the similarities of their embedded speech Twitter. Fol- lowing Ringe et al. (2013) and Wojeik (2018), we model the log odds ratio the co-voting rate be-
We further suppose that ideological differences are most evident when conditioned on partic- ular issue, such as "the environment"; Bateman et al. (2017) note that aggregated mea- sures like ideal points mask important variation across issues. To this end, we first train topic
fact,
"Tweets from github. com/ccoo/aaaa lex itel/ congresstweets, votes from voteview. com/ com/about. about.

---

model on Twitter data to group legislator utter- ances into broad issues. 15 Two authors indepen- dently labeled the topics they deemed most in- dicative of ideology, based on the top words and documents from the topic-word (3(+) and topic- document (o(k) distributions." 16 Then, for each selected topic k and legislator I, we select the top five tweets U,(R) = { {uil .......)),, " filtering out those with estimated low probability for the topic, 0 0.5. Finally, we generate a flexi- ble number of inferential decompositions for the tweets R((A) using GPT-3.5." !7 We use two prompts each containing six different exemplars to increase the diversity of inferences. The collections of tweets and decompositions are then embedded with Transfomrrs (all-mpna) from Reimers and Gurevych, 2019).
115th
116th
117th
3 f
MAET
MAEJ
Sim. U
10.64
11.86
744.81
17.36
981.56
Sim. R 12.69
17.31
736.21
23.0
964.86
Sim. U 6.02
4.40
11.09
+
741.59
935.45
Sim. R 7.01
12.67
10.82
!.i
Table 4: Explanatory power (.) and predictive capacity (MAE) of text similarity measures in mixed effects model of co-vote behavior for 3 sessions the US Senate. Coefficients are large and significantly above zero (p < 0.001). For the mean absolute error (MAE), we multiply the predicted co-vote agreement error with the total number of votes in common for the legislator pair.
row 2 in table 4) has higher regression coefficient (3) than the corresponding coefficient for similar- ity in utterances (U; row 1). This also holds for a model that uses both similarity measures in the regression (row 3).
Language Similarity. To form a text-based sim- ilarity measure between legislators and j. first compute the pairwise cosine similarity be- tween the two legislators' sets of text embeddings, Scos (U!4) U/(()) (or R. for the decompositions). The pairs of similarities must be further aggregated to single per-topic measure; we find the 10th per- centile works well in practice (taking maximum similarity can understate differences, whereas the mean or median overstates them, likely due to fi- nite sample bias; see Demszky et al. 2019). This process creates two sets of per-topic similarities for (k) (k)
Additionally, we compare the predictive capac- ity of the two similarity measurements in two sce- narios: we train the model on data from the 115th Senate to predict co-vote for the 116th and train on data from the 115th and 116 sessions to predict co-vote for the 117 Using similarity in decom- positions leads to a lower MAE between predicted and actual co-vote agreement than using similarity in utterances; both similarity measures together fur- ther reduce the error for the 117th Senate (table 4),
each (i. j) -egiislatt pair, Si (u), Sig (r).
Examples of utterances and their decompositions in fig. 6 help contextualize these results. For the left-hand example, the method infers shared im- plicit cropositionn"Prrsiidoo that underlies two tweets with little observed text in common. However, the method can occasionally overstate similarities between utterances (and thus, between legislators): while the decompositions in the right-hand example are valid inferences. they are also overly general ("voting is important").
Results. Similarities based on inferential decom- positions, (r)) both help explain the variance in co-vote decisions as well as help predict co- voting agreement over and above similarities based the observed utterances (tweets) alone, Siy(u) (table 4) Specifically, for the regression mod- els, across three Senate sessions, the mixed effects model that uses similarity in decompositions (R;
Trump weak"-
Text was processed using the toolkit from Hoyle et al. (2021), and modeled with collapsed Gibbs-LDA (Griffiths and Steyvers, 2004) implemented in MALLET (McCallum, 2002). "E.g. top words "border. crisis, biden, immigration" correspond to the politically-charged issue of immigra- tion; the more benign "tune. live, watch. discuss" cov- ers tweets advertising a media appearance to followers. Both annotators initially agreed on 92% of labels; dis- agreements were resolved via discussion. The final set of 33 topic-words be found in github. com/ ahoho/ peeerettal-ddecmppssttii.
Our approach further uncovers the narratives that emerge among legislators from different parties around a particular issue. In discussions around abortion and reproductive health, our decompo- sitions capture finc- fine-grained viewpoints about the role of supreme court and judiciary, and the con- tentious debate around late stage abortions (Fig. 5). Clustering over implicit decompositions reveal finer opinion-spacee that succinetly capture author viewpoints towards facets of a particular issue.
17 "Results table 4 are extremely similar when using Alpaca-7B, suggesting the findings are robust.
We use memhership in the same political party a control variable in cach of the mixed effects models.

---

Correct behavior
Incorrect behavior
close embedding Space
As Joe Biden said: Trump may believe mouthing the
Under ordinary times, Mike Pompeo would confirmed over whelmingly. allow senators to vote on this nominee the floor is the right thing to happen.
Looks like Putin has gotten so used to Trump's obsequious servility that little Biden straight talk
Elections are not only about the present moment. They are also about who we are as people and who want the future.
words LAW ORDER makes him strong. But Trump's fallure call on his supporters to stop acting an armed militia shows how weak he is [-]
0.85
0.00
President Trump is
President Trump is weak
Voting important
Voting is important
Figure 6: Pairs of legislative tweets (green) and associated decompositions (orange). Here, we show instances where embeddings of the decompositions are closer than embeddings the original tweets. The example on the left shows the method working intended, whereas the example on the right undesired behavior. Although the method generates multiple decompositions per tweet, we only show the two closest. In appendix A.1, we discuss instances where the decompositions more distant than the tweets.
6 Related Work
2020), but instead to surface the explicit or implicit propositions inferrable from utterance.
As computational text analysis method that op- erates over reduced representations of text data, our work is similar that of Ash et al. (2022) and Bamman and Smith (2015). Closest to our clustering effort ($ 4), Ernst et al. (2022) extract and cluster paraphrastic propositions to generate summaries. However, they constrain themselves to the observed content alone, limiting their ability to capture context-dependent meaning.
Our work also bears a similarity to Opitz and Frank (2022), who increase the interpretability of sentence embeddings using an AMR graph for a sentence. However, AMR graphs are also tied to the information present in an utterance's 's surface form (and, moreover, is unclear whether AMR parsers can accommodate noisier, naturalistic text). LLMs have been used to generate new informa- tion ad-hoc in other settings, for example by aug- menting queries (Mao et al., 2020) or creating ad- ditional subquestions in question-answering (Chen et al., 2022). In contemporaneous work, Ravfogel et al. (2023) use an LLM to generate abstract de- scriptions of text to facilitate retrieval. In Gabriel et al. (2022), the authors model writer intent from a headline with LLMs. Becker et al. (2021) gener- ate implicit knowledge that conceptually connects contiguous sentences in longer body of text.
Our method continues the relaxation of formal semantic representations NLP, a process exem- plified by the Decompositional Semantics Initia- tive (White et al., 2016), which aims to create an intuitive semantic annotation framework robust to variations in naturalistic text data. The gener- ated inferential decompositions in our work are and open-ended, structured by a user-defined schema in the form of exemplars. In this way, we also follow a path forged by natural language inference (Bowman et al., 2015; Chen et al., 2017), which has increasingly relaxed seman- tic formalisms when creating datasets. In future work, we plan to relate both the exemplars and outputs to formal categories and investigate their utility in downstream lasks.
7 Conclusion
Our method of inferential decompositions is use- ful for text-as-data applications. First, we uncover high-level narratives in public commentary, which are often not expressed in surface forms. Second, we show that, by considering alternative represen- tations of legislators' speech, we can better explain their joint voting behavior. More broadly, treating implicit content as a first-class citizen in NLP- capability enabled via generation in large language models- the potential transform the way we approach problems that depend on understanding what is behind people's utterances, rather than just the content the utterances themselves.
Our methods for generating decompositions are distinct from extracting commonsense knowledge from e.g. through templates (Tandon et al., 2014), free-form generation from LLMs Bossclut al. (2019), or script induction (Schank and Abel- son, 1975) (which can also use LLMs, Sancheti and Rudinger 2022). Our decompasitions are not designed to produce general commonsense knowl- edge such as 'rain makes roads slippery" (Sap et al.,

---

8 Limitations
cate for manual verification of a sample of outputs in sensitive contexts (step 4 of our protocol 2).
Our validity checks Section reveal that while most decompositions are deemed reasonable by hu- mans, some are not (Fig. 3.). It remains to be stud- ied the extent to which implausible generations are affecting with the results, or if they are introducing harmful propositions not present in the original text. In future work, we will explore whether known po- litical biases of language models (Santurkar et al., 2023) affect our results. Although an open-source model (specifically Alpaca-7B Taori et al. 2023) produces similar results to those reported, our main experiments primarily use models released by Ope- nAI, which may lead to potential reproducibility issues. All our analyses and experiments focus utterances in the English language, which could limit the generalizability of our method. Relatedly, our experiments are also specific to the US socio- cultural context and rely on models that are known to be Western-centric (Palta and Rudinger, 2023). The embeddings could be made more sensitive to the particular use case. In future work, we plan additionally fine-tune the embeddings that they are more sensitive to the particular use case (e.g., establishing argument similarity, Behrendt and Harmeling, 2021)).
10 Acknowledgements
We thank Sweta Agrawal for her abundant support and feedback on earlier drafts and Justin Pottle for assistance with data quality annotation. We also appreciate productive discussions with both Rachel Rudinger and Maharshi Gor. Jordan Boyd-Graber and Chenglei Si gave excellent notes during our internal paper clinic, and we thank our anonymous reviewers for their very helpful comments. This work was supported in part by the National Science Foundation under award #2008761, the Food and Drug Administration (FDA) of the U.S. Depart- ment of Health and Human Services (HHS) part of financial assistance award (Center of Excel- lence in Regulatory Science and Innovation cooper- ative agreement U01FD005946 with the University of Maryland), and a grant from the University Maryland Social Data Science Center. Any opin- ions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily represent the official views of, nor an endorsement, by any sponsor by the U.S. government.
9 Ethics Statement
References
The work is in line with the ACL Ethics Policy. Models, datasets, and evaluation methodologies used are detailed throughout the text and appendix. The human evaluation protocol was approved by an institutional review board. No identifying informa- tion about participants was retained and they pro- vided their informed consent. We paid per survey based on estimated completion times to be above the local minimum wage (appendix A.5). Partici- pants were paid even if they failed attention checks. All the datasets were used with the appropriate or requested access as required. We acknowledge that we are using large language models, which are sus- ceptible to generating potentially harmful content. Generally, the potential for misuse of this method is not greater than that of the large lan- guage models used to support it. In theory, it is possible that practitioner could draw incorrect conclusions about underlying data if the language model produces a large number of incorrect state- ments. To the extent that those conclusions inform downstream decisions, there could be a potential for negative outcomes. For this reason, we advo-
Elliott Ash, Germain Gauthier, and Philine Widmer. 2022. Relatio: Text semantics capture political and economic narratives.
Kent Bach. 1994. Conversational impliciture. page 284.
Kent Bach. 2004. Pragmatics and the philosophy of language. The handbook of pragmatics, 463:487.
Christian Baden, Christian Pipal, Martijn Schoonvelde, and Mariken A.C.G. van der Velden. 2021. Three gaps computational text analysis methods for cial sciences: A research agenda. Communication Methods and Measures, 16:1 18.
David Bamman and Noah A. Smith. 2015. Open extrac- tion of fine-grained political statements. In Proceed- ings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 76-85, Lisbon, Portugal. Association for Computational Linguistics.
David A. Bateman, Joshua D. Clinton, and John S. Lap- inski. 2017. A house divided? roll calls, polarization, and policy differences in the u.s. house, 1877-2011 I. American Journal of Political Science, 61(3):698 714.
Maria Becker, Siting Liang, and Anette Frank. 2021. Reconstructing implicit knowledge with language

---

models. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 11-24, Online. Association for Computational Linguistics.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1657-1668. Vancouver, Canada. Association for Computational Linguisties.
Maike Behrendt and Stefan Harmeling. 2021. Argue- bert: How to improve bert embeddings for measuring the similarity of arguments. In Conference on Natu- ral Language Processing.
Jennifer Hayes Clark and Veronica Caro. 2013. Multi- member districts and the substantive representation of women: An analysis of legislative cosponsorship networks, Politics & Gender, 9(1):1-30.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5185-5198, Online. Association for Computational Linguistics.
Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Ratio- nal, evaluation and approaches. Natural Language Engineering, 15:i - xvii.
Antoine Bosselut, Hamnah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4762-4779, Flo- rence, Italy. Association for Computational Linguis- tics.
David L. Davies and Donald W. Bouldin. 1979. A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI- 1:224-227.
Wayne Davis. 2019. Implicature. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, Fall 2019 edition. Metaphysics Research Lab, Stanford University.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference, In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Juraf- sky. 2019. Analyzing polarization in social media: Method and application to tweets on 21 mass shoot- ing.. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putcational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 2970- 3005, Minneapolis, Minnesota. Association for Com- putational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Bruce A Desmarais, Vincent G Moscardelli, Brian F Schaffner, and Michael S Kowal. 2015, Mcasuring legislative collaboration: The senate press events net- work. Social Networks, 40:43-54.
Tadeusz Caliński and Joachim Harabasz. 1974, A den- drite method for cluster analysis. Communications in Statistics-hheory and Methods, 3:1-27.
Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-equivalers semantics for translation eval- uation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 162-171, Montréal, Canada. Association for Computational Linguistics.
Meng Cao. Yue Dong, and Jackie Cheung. 2022. Hal- lucinated but factual! inspecting the factuality of hallucinations in abstractive summarization. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 3340-333355 3354, Dublin, Ireland. Associa- tion for Computational Linguistics.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-20177 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 11th International Workshop on Semantic Evaluation (SemEval-2011) pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Ori Ernst, Avi Caciularu, Ori Shapira, Ramakanth Pa- sunuru, Mohit Bansal, Jacob Goldberger, and Ido Dagan. 2022. Proposition- clustering for multi- document summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1765-1779, Seat- tle. United States. Association for Computational Linguistics.
Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022. Generating literal and implied sub- questions to fact-check complex claims. In Confer- ence on Empirical Methods in Natural Language Processing.
Saadia Gabriel, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi. 2022. Misinfo reaction frames: Reasoning

---

about readers' reactions to news headlines. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguisties (Volume I: Long Papers), pages 3108-3127, Dublin, Ireland. Associa- tion for Computational Linguistics.
Nitin Madnani, Necip Fazil Ayan, Philip Resnik, and Bonnie Donr 2007. Using paraphrases for parameter tuning in statistical machine translation. In Proceed- ings of the Second Workshop on Statistical Machine Trunslation, pages 120-127, Prague, Czech Republic. Association for Computational Linguisties.
Pranav Goel. 2023. Analyzing Communicative Choices to Understand Their Motivations, Context-bute Variation, and Social Consequences. Ph.D. thesis, University of Maryland.
Yuning Mao, Pengcheng He., Xiaodong Liu, Yelong Shen, Jianfeng Gao. Jiawei Han, and Weizhu Chen. 2020. cnneration-augmeent domain question answering. In Annual Meeting of the Association for Computational Linguistics.
retricval for open-
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2023. News summarization and evaluation in the era of gpt-3.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919 On- line. Association for Computational Linguistics.
Thomas L. Griffiths and Mark Steyvers. 2004. Find- ing scientific topics. Proceedings of the National Academy of Sciences of the United States of America, 101:5228- 5235.
Andrew Kachites McCallum, 2002. Machine learning with MALLET. http:/ hstp://mat /ma leet..... cs. umass. edu.
Justin Grimmer and Brandon M Stewart. 2013. Text data: The promise and pitfalls of automatic con- tent analysis methods for political texts. Political Analysis, 21:267 - 297.
Miller McPherson, Lynn Smith-Lovin, and James M Cook. 2001. Birds of a feather: Homophily in social networks. Annual review of sociology. 27(1):415- 444.
Jerry R Hobbs, Mark E Stickel, Douglas E Appelt, and Paul Martin, 1993. Interpretation as abduction. Arri- ficial intelligence, 63(1 (222-)))9-14..
Amita Misra, Brian Fecke, and Marilyn Walker. 2016 6. Mcasuring the similarity of sentential arguments in dialogue. In Proceedings of the 17th Annual Meeting of the Special Inierest Group on Discourse and Dia- logue, pages 276-287, Los Angeles. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text de- generation. In International Conference on Learning Representations (ICLR), volume abs/1904.0977, 904,09751.
Alexander Hoyle, Pranav Goel, Andrew Hian-Cheon Cheong, Denis Peskov, Jordan Bayd-Grabrr, and Philip Resnik, 1.21. Is automated topic evaluation broken? the incoherence of coherence. In NeurlPS.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma. Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-tS: Scalable sentence encoders from pre- trained text-to-text models. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pages 1864-1874, Dublin, Ireland. Association for Computational Linguistics.
Alexander Miserlis Hoyle, Rupak Sarkar, Pranav Goel, and Philip Resnik. 2022. Are neural topic models broken? In Findings of the Association for Computa- fional Linguistics: EMNLP 2022, pages 5321-5344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12): 1-38.
Juri Opitz and Anette Frank. 2022. SBERT studies meaning representations: Decomposing sentence em- beddings into explainable semantic features. In Pro- ceedings of the 2nd Conference of the Asia-Pacifie Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference Natural Language Processing (Volume 1: Long Pupers). pages 625-638, Online only. Association for Computational Linguistics.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423 438.
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama. Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller. Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Alisa Liu, Swabha Swayamdipta. Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and Al collabora- tion for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826-6847, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.

---

Maria Leonor Pacheco, Tunazzina Islam, Lyle Ungar, Ming Yin, and Dan Goldwasser. 2022. Interactively uncovering latent arguments in social media plat- forms: A case study on the covid-19 vaccine debate. In Proceedings of the Fourth Workshop on Data Sci- ence with Human-ingg Mamman-in-the-Loug (Language Advances), pages 94-111, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joinl Conference on Natu- ral Language Processing (EMNLP-IJCNT) pages 3982-3992, Hong Kong. China. Association for Com- putational Linguistics.
Maria Leonor Pacheco, Tunazzina Islam, Lyle Ungar, Ming Yin, and Dan Goldwasser. 2023, Interactive concept learning for uncovering latent themes in large text collections.
Nils Reimers, Benjamin Schiller. Tilman Beck, Jo- hannes Daxenberger. Christian Stab, and Iryna Gurevych. 2019. Classification and clustering of arguments with contextualized word embeddings. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 567- 578, Florence, Italy. Association for Computational Linguistics.
Shramay Palta and Rachel Rudinger. 2023. FORK: A bite-sized test set for probing culinary cultural biases commonsense reasoning models. In Findings of the Association for Computational Linguisties: ACL 2023. pages 9952-9962, Toronto, Canada. Associa- tion for Computational Linguisties.
Nils Ringe, Jennifer Nicoll Victor, and Justin Gross. 2013. Keeping your friends close and your enemies closer? information networks in legislative politics. British Journal of Political Science, 43(3):01--62,
Monica Pampell. 2022. Get bulk download of public comments from regulations.gov in under 15 minutes! YouTube video https: / /www.w youtube, com/ watcc27v=ruBBxIB0QQI accessed October 19, 2023.
Peter J. Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics. 20:53-65.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Abhilasha Sancheti and Rachel Rudinger. 2022. What do large language models learn about scripts? In Proceedings of the 1Ith Joint Conference on Lexical and Computational Semantics, pages 1 1-11, Seattle, Washington. Association for Computational Linguis- tics.
Shibani Santurkar, Essn Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang. and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? ArXiv, abs/2303. 17548.
Roma Patel and Elizabeth-Jane Pavlick. 2022. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representa- tions.
Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020. Commonsense reason- ing for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 27-33, Online. Association for Computational Lin- guistics.
Tai-Quan Peng, Mengehen Liu, Yingeai Wu, and Shixia Liu. 2016. Follower-followe network, communica- tion networks, and vote agreement of the us members of congress. Communication Research, 43(7):996- 1024.
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwe 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001.
Roger C. Schank and Robert P. Abelson. 1975. Scripts, plans, and knowledge. In Proceedings of the 4th In- ternational Joint Conference on Artificial Intelligence Volume 1, IJCAI'75, page 151-157, San Francisco, CA. USA. Morgan Kaufmann Publishers Inc.
Fabio Petroni, Tim Rocktäschel, Sebastian Ricdel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Niket Tandon, Gerard Melo, and Gerhard Weikum. 2014. Acquiring comparative commonsense knowl- edge from the web. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 28(1).
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stan- ford alpaca: An instruction-folloiinng llama model. https:/ hetr//ggittbbb ccm/tatsu-lab/ stanford paca.
Shauli Ravfogel, Valentina Pyatkin, Amir DN Cohen, Avshalom Manevich, and Yoav Goldberg. 2023. Re- trieving texts based on abstract descriptions.

---

Nandan Thakur, Johannes Daxenberger, and Iryna Gurevych. 2020-10. Bws argument similarity corpus.
A
Appendix
A.1 Further Qualitative Analysis of Legislative Tweets
Keyon Vafa, Suresh Naidu, and David Blei. 2020. Text- based ideal points. In Proceedings the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5345-5357, Online. Association for Computational Linguistics.
In Section 5, we showed that our method can place utterances with distant observed content closely through decompositions. In this section, we pro- vide further illustrative examples where small ut- terance distances yield larger decomposition dis- tances (Fig 7). As before, we show examples of "correct" behavior (where the method works as in- tended) as well as failures (where it does not). In the left column of Fig 7, tweet pair that dis- cusses similar topic (Coronavirus) has relatively similar embeddings even though they communicate considerably different content; appropriately, the generated decompositions are more distant. At the same time, although the two utterances in the right communicate very similar content, the generated decompositions are nonetheless further in embed- ding space a problem is exacerbated by the open referent "they". In future iterations of the method, we plan to regularize outputs to avoid such issues.
Laurens der Maaten and Geoffrey Hinton. 2008. Visualizing data using I-sne, Journal of Machine Learning Research, 9(66):557--260
Dominik Wawrzuta, Mariusz Jaworski, Joanna Gotlib, and Mariusz Panczyk. 2021. What arguments against covid-19 19 vaccines tent analysis comments. Vaccines, 9.
facebook in poland: Con-
Aaron Steven White, Drew Reisinger, Keisuke Sak- aguchi, Tim Vieira, Sheng Zhang. Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Uni- versal decompositional semantics Universal De- pendencies. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing. pages 1713-1723, Austin, Texas. Association for Computational Linguistics.
Aaron Steven White, Elias Stengel-Eskin, Siddharth Vashishtha, Venkata Subrahmanyan Govindarajan, Dee Ann Reisinger, Tim Vieira, Keisuke Sakaguchi, Sheng Zhang, Francis Ferraro, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2020. The universal decompositional semantics dataset and de- comp toolkit. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 5698-5707, Marseille, France. European Language Resources Association.
A.2 Decomposition Exemplar Creation Meta-Prompt
Below we present a condensed version of human in- structions for the structured creation of exemplars. Human utterances communicate proposi- tions that may may not be explicit in the literal meaning of the utterance. Your goal to make brief list of propositions that are implicitly or explicitly conveyed by the meaning of an utterance.
Stefan Wojeik. 2018. Do birds of feather vote together. or is it peer influence? Political Research Quarterly, 71(1):75-87.
Wei Xu, Chris Gallison-Burche and Bill Dolan. 2015. SemEval-2015 task 1: Paraphrase and semantic sim- ilarity in Twitter (PIT). In Proceedings of the 9th International Workshop on Semantic Evaluation (Se- mEval 2015). pages 1-11, Denver, Colorado. Associ- ation for Computational Linguistics.
All the propositions you include should be short, independent, and written in direct speech and simple sentences. If possible, try to keep propositions to single clause con- sisting of subject, predicate, and ob- ject (don't worry too much about sticking this format: noun phrases and prepositional phrases are acceptable). It may be helpful to "break up" propositions as necessary, and you should disambiguate unclear referents when possible ("vaccine" + "COVID vaccine").
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pages 1298-122 1308, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
By implicitly conveyed propositions, we mean propositions that are plausibly rea- sonably inferred, even if they were not neces- sarily intended be conveyed by the utter- ance. Shorter utterances will typically com- municate fewer propositions. Longer texts may communicate several; we ask you to pre-
Vitalii Zhelezniak, Aleksandar Savkov, and Nils Ham- merla. 2020. Estimating mutual information between dense word embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8361-8371, Online. Association for Computational Linguistics.

---

Correct behavior
Incorrect behavior
Right now, our focus must be evacuating Americans, coalition partners, and refugees from Afghanistan. Their lives are at risk because their support mission, and we must everything can ensure their safe evacuation.
Close - for embedding space
The most urgent and important responsibiiity we have is ensuring every American every Afghan who aided our efforts safely evacuated from Afghanistan. No American or ally America will be left behind.
by China's government to cover-up coronavirus and its spread aren't just negligence they criminal nature. China must be held accountable. [..]
One major lesson we have learned from the #COVID19 crisis that pharmaceutical and medical supply are overly dependent foreign
0.54
0.15
I
I
1 Protecting American citizens 15 essential
0.71
0.35
Those varal cristnals
Me nore donestic nedicine production
ensure their safety
Figure 7: Pains of legislative tweets (green) and associate decompositions (orange). Mirroring Fig fig. 6, we show instances where embeddings the decomposition are farther than embeddings of the original tweets, The example the left shows the method working intended, whereas the example on the right shows undesired behaviour. Although the method generates multiple decompositions per tweet, we only show the two closest We use the Sentence-Tranee Transformer model all -mpnet -aseeet--aseee v2 to compute embeddings; generations from Alpaca-7B.
fer writing propositions that are most tral the meaning that's being communi- cated. With the above mind, limit your- self to five propositions per utterance. Write diverse propositions that are minimally re- dundant with one another. Propositions can
Sentence- T5
Paraphrases Alpaca
GPT-3
SICK-R
79.98
81.46
80.49
STS-B
83.93
85.49
86.28
STS12
79.02
76.97
79.18
STS13
88.80
88.44
89.18
fall into several categories.
STS14
84.33
84.18
85.36
STS15
88.89
89.28
89.28
Explicit propositions [required] To generate propositions in this category, rephrase elements of the utterance's explicit meaning as one or more simple propositions communicated by the utterance. It may be the case that almost changes neces- sary, and you will merely write a paraphrase. Including world knowledge acceptable.
STSI6
85.31
85.15
85.23
STS17
88.91
89.94
90.29
Table 5: Spearman's p for STS benchmarks using a saraphrasc-basee variant of method. Sentence-TS embeds texts in each pair with sentence-t5-4 (Ni et al., 2022), Paraphrases concatenates averaged em- beddings of additional paraphrases generated with zero- shot Alpaca-7B or GPT-3. Improvements over Sentence- T5 are underlined. This demonstrates that our method consistently improve STS correlations of arbitrary baseline embeddings.
Inferences about utterance subject [optional]. Make infer- ences from the utterance about the subject it talking about. These are nontrivial but commonsense implications that can be reasonably and directly inferred from the utterance.
express value judgments (not necessarily true). These should be written as implied statements. Rather than mentioning the author explicitly by writing something like "the author author/speaker believes/ tetes/hhiiksseeeeee X", just write the X.
Inferences about utterance perspective [optional] Often, the author of an utterance is, intentionally not, conveying information about their perspectives or preferences. Write down inferences from the utterance that consistent with the author's perspective (as you understand it). These propositions sometimes take the form of modals ("should", "must"), or are general statements that may
A.3 Adapting the method for semantic textual similarity tasks
Our proposed method decomposes utterances into related propositions, which does not markedly im- prove standard STS tasks that typically rely on the explicit content in text (see lower half of table I).

---

However, an alternative approach that instead generates multiple expressions of the same mean- ing does improve sentence embedding performance. A single sentence is only one way of expressing a meaning, and in many settings, there is value in considering alternative ways of commumicating that same meaning. For example, the BLEU score for machine translation evaluation (Papineni et al., 2002) works more effectively with multiple refer- ence translations (Madnani et al., 2007). Dreyer and Mareu (2012) take this observation a step fur- ther by using packed representations encode ex- ponentially large numbers variations given an original sentence.
A.5 Survey Details
Inferential Decomposition Annotation. 80 flu- ent English speakers in the US and UK with at least a high school diploma (or equivalent) annotate random sample of 15 utterance decompositoioooiiiio pairs, recruited via Prolific fprolific co). We paid 2.10 USD/survey, which take median 10 minutes.
Clustering Annotation. We recruited 20 fluent English speakers in the US and UK with at least a high school diploma via Prolific. After instruction with two artificially high- and low-quality clusters to help calibrate scores, participants reviewed random sample of ten clusters from the pool of 45. We paid 3.50 USD/survey, median completion time was 17 min.
breaning-equivalee
Here, we show that improvements sentence representations can obtained by expanding a sentence's form with multiple text represen- tations restating the same content. Specif- ically, represent every sentence s; by consisting of
Total compensation to annotators was 257 USD; we targeted 14 USD/hour.
set S; the original utterance and paraphrases. As baseline, we computed the cosine similarity comparisons between embeddings of the original sentences 8, 8j, obtained with the state-of-the- art Sentence-T5 (Ni et al., 2022). Pairwise comparisons for expanded representations S,, Sj. were scored by concatenating the embedding for 8; with the mean of the embeddings for the Si,,, [e(s;); Σn e) Three paraphrases per input were generated with both a 7B-parameter Alpaca model (Taori et al., 2023) and the OpenAl text cext-davinci--ee -03 (derived from
81,,,
A.5.1 Instructions and Examples Provided to Survey Participants for Human Annotation and Evaluation
We present the instructions and examples provided to human annotators or survey participants, in order to validate the quality of the generations (S 3), fig. and fig. 9 respectively.
We present the instructions and examples pro- vided to human annotators or survey participants, in order to evaluate the quality of clustering of- fered by our approach ($ 4), in fig. 10 and fig. 11 respectively.
?) using a O-shot prompt: "Paraphrase the following text. An###An Text:
[input]An Paraphrase: [output]"
Table summarizes results on STS tasks from the Massive Text Embedding Benchmark (MTEB, Muennighoff et al., 2022). Our method improves over the Sentence-TS alone in all but one in- 20
A.4 Prompts and Exemplars
We present our prompts in table and our exem- plars tables 7 to 12.
"FFor all experiments we use the model sentence-t5-x- directionally similar results were observed for the lightweight all all-mpnet aal--pmet-base-eee
20 "'Given the modularity of our approach, we expect that for instances where we there un absolute improvement over the Sentence-TS T5 baseline, substituting the state-of-the-an emhed- ding model would further improve results.

---

Prompt
propesitions Exemplars Prompe
Paraphrases
Paraphrase the tollowing text.
4+
Text: <input>
Paraphrase: <output>
FDA Comments
Bunon utterances contain propositions that may or may not be explicit in literal neaning of utterance. Civer ulterance, staze the propositions of that uttersace in a brief 1is.. All generated proposit ions should be short, independent, and writter direst apeach and simple sentencas. propoastion consists of subjezt, vers, and an object.
6
public consenta the FDA website concerning the ccvid vaecine.
Usterance: <input>
Propositions: soutput>
Explicit
Bunan ullerances commuiccate propositions, Por each utterance, atase explicit propossticns communicated by that utterance in brief list. All generated propositicns should be short, independent, written in Tf possible, write proposit itins with a aubject, verb, ard object. <citaset_dcseriptiise #+1
speech simple sentences.
Implicit
Honan utterences communcaate propositiens that may nct be explicit in the literal neaning of the utterance. Por each usterance, staze the implicit pcoposition commuicated by that utterance in briat 1ist. Implicit propositions may be inferences about perapect ivv of ita author. A11 generated propos lons should independent, and writtan in direct speeca simple sensences. if possible, write propositions with a subject, verh, and object. slataset_deaerigg iori II
7
subject
uscerance or
Buunn utterances communccate propositions that nay or may not explicit in the literal meaning of the utterance. For utterance, commuiicated by that utterance in a br brie" list. Trplisit propesitions be inferenees about the suoject the usterancs or about the perspective of its author. All generated propesitions should be short, independent, and speech ard simple sent encea. possiblle write propositions with subject, vert, and object. castasot_deser.ptiicnn
implicit
direct
Table 6: Prompt templates used for obtaining the decompositions. The FDA Comments and their generations used in 4, and the generations legislative tweets were used in 5, We used six exemplars along with the prompts in both these cases. When generating from Alpaca-7B, we alter these templates according to their format.

---

Academic qualifications were less important in previous policies
The anii-defamation league influential organization
Infereneus about Utterrncee Subject [Optional)
Yukos Oil is important the Russian econony
Switzerandd made deal with semitic
Palestinian and Israe li forces are in conflict
woman preparing food beverage
Disabled people can swim long distances
The baby is happy
The anti defamation league accused Switzerland funding leroorism April 2008
Somenne who swam the English Channel has a wheelchair Mikhail Khordarkowky 's arrest led a fall in Russian stocks
Mikhail Khordorkovky is accused of froud tax evasion
Palesti nian and Israeli forces, clashed in the West Bank
new policy emplasizes academic qualifications
Propositions: Explicit [Required]
juiie from lemon
The baby laughs and cnawls Thieves steal wheelchair
woman
The new policy gives greatest weight to grades, test scores and a student's high school curriculum. the deladeffaation leegue took out full-page adverti sments in swiss and international new spapers earlier in april 2008 accusing wwizerrlnnd funding terrorism through the deal. A woman squezzing lemon.
Russian stocks fell after the arrest last Saturday of Mikhail Khodorkovsky chief executive of Yukos Oil, on charges fraud and evasion.
Palestinians clash security forces Bank
Thieves steal Channel swimmer's 3 whoelchair
Utterance
The baby is laughing and crawling ling.
Source
STS-12
STS-B
STS-B
STS-B
STS-B
STS-B
STS-12
Table 7: Exemplars for inferential decomposition for the source type of Newswirc/image captions. We sample n
exemplars from this set form prompt, per Table 6.

---

Inferenees about Utterance Perspective [Optional]
Abortion rights have been longstonding
The state should be held accounaable when it does wrong
Religious people illogical things
Feminist arguments are inconsistent
The death penalty appeals base instinets
The justice system is imperfect
The death penaliy should be stopped now
Religion irrational
Inferences about Utterance Subject [Optional]
The minimum wago is enforeed by law The minimum wage applics broadly
The law protects the right to abortion
Some virtual currencies can be converted into fiat money.
Virual currencies can be useful.
The implementation death penulty flawed
The death penalty allows wrongful deaths
Accusations sexism are indiscriminate
Roe Wade decided in January 1973
Propositions: Explicit [Required]
The death penalty is form revenge
Online stores can necepp and exchange
Employyes must mott cortain criteria
The death penalty puts innocent people to death
The death penalty needs to be put on hold while is being fixod
The death penalty is irreversible
Roe Wade dee lired abortion constitutional right
People who believe in a god came up with something insane
to receive the minimum wage.
There winning against accusations sexism
NXT is virtal currency
NXT.
[Tppíi: death penalty] If we don't stop the denth penalty now, we will put innocent people to death while we try to fix the problems.
[Topic: death penalty) the use the death penalty is an act of revenge which cannot undone, so should state be put on trial for putting to death an innocent person?
[Topic: minimum wago) After 90 consecutive days of employment or the employee reaches
Supreme Court that declured abortion constitutional right. decided January 1973.
[TTppi: cryptocurencyl The new initintive will allow merchants to accept NXT and other virtual currencies as payment on their online stores and eusily exchunge for fiat money.
20 cars age. whichever comes first. the employec must receive the current foderal minimum wage or the state minimum wage, whichever is higher.
(Toppi: leminsst movement Does it involve women? It's sexist. Does it not involve women? It's woxist No winning.
come with insanity? Oh wait, they believe Sky-God. #fyilive
]opiie: abortion] Roe Wade,
Uitterance
[Topic: athelsn) How
UKP Sentential
Source
SemEval16
UKP Aspeet
Arg Mining
SemEval16
AFS
BWS
AFS
Table 8: Exemplars for inferential decomposition for the source type of argument//sannce datasets. We sample n
exemplars from this set to form prompt, per Table 6.

---

Inferences about Utterance Perspcttive (Optional)
The Lion King is an emotiooal movic
The Eagles are a hated Leain
Mafast 's death is sad
Inferences about Uiterance Subject [Optional)
The Boston Marahhon bombers were apprehended
Roy Nelson and Kongo were in a combit sport
Kevin Durant played Carlos Delinoo in basketball
Terorists boubed Boston Marathon
Lewandowsks is an important player
Russel Westbrook injured his knee
Injuring key player will help Real Madrid win
Zach Eatz a Footbaal player
Mufisa dies in the Lion King
Real Madrid trying to injure Lewandowski
Boston Marahon Bomberss planned allack Time Square, NYC
The Philodel phiia Eagles dralted Zach Ertz
Russell Westbrook will recieve kinee surger
Propesitions Explleit [Requtred]
Carlus Delfino dunked on Kevin Durant
is cmotional when Mufasa dies
Roy Nelson knocked out Kongo
Emotiionlly can't handle when Mufasa dies
Doston Manithon Bumber planmed to turget Timess Square
Russell Westbrook getting knee surgery
Cartos Delíino got cookies dipped Durant's head
Real Madrid out to kill lewandowski
Zuch Harz, taken by the damn Eagles
Roy Nelson just smoked Kongo
Utterance
Table 9: Exemplars for inferential decomposition for Twitter-PC. We sample n exemplars from this set to form a prompt, per Table 6.

---

Inferences about Utterance Purs Purspective [Optional]
Forcing the covid vncciine limits freedom
People should choooe whether they get vaccinated.
There may be comspiracy limit information about vaccine side effects.
Infereneus about Utteranee Subject [Optional]
Health pro ssiionass don't reparrt adveme effects of vaocines.
The covid vaccine causes blood closs.
Covid vaceines do prevent covid.
Vaccine mandates violates the rights of citizens.
The covid vaccine approval process is unscientific.
The government is illegaly forcing the covid vaccine on people..
The covid vaccine causes scrious health problems
Proposition: Explicit [Required]
Vacciine mandaies viulate bodily autonomy.
Children contrnct coronavitus
The covid vaceine kills people.
Covid
Bodily autonony is everyone right and extremely important. everyone should have the right to choose whether or not they want to have a vaecine. It should not be mandated for any profession children at any level of education as violates their rights.
lideiity in entire VACCINE process, deaned scientilie. There SO many inconsistencics from the onset how this vaccine going
half of those know now have health issues after their shot a unable issues function in their daily routines. know health professionals. that when asked if they were sahmitting VAERS info, the ruply is We simply do not have the time or tried Have you ever tried to input Viers into? It IMPOSSIBLES LE!!
Stop illeglly forcing the clot shot ento the citizens and their children. This is wrong and you are uking away our freedom. The Covid vaccine is killing people.
Kids don get the lake coved flu and then paccines don work if you can still gat Refer to Colin Powell
appears to me there is
Utterance
From stand covid.
Table 10: Part one of the exemplars for inferential decomposition for FDA comments. We sample n exemplars from this set form prompt, per Table 6. The other exemplars for FDA comments are provided in Table 11.

---

Inferences about Utterance Perspuctive (Optional
People prometing COVID vaccines are acting Like Nazis
Children rohust illness thin athals
Inferences about Utterance Subject (Optional)
The government people to
The use of covid vacciaes in children a human rights violation.
Children are not susceptible to complications from covid
The proger protocal to approve vaccinas was not followed.
Propositions: Explicit [Required)
Covid is traamblc with other medications.
Those mundating the will be held
Covid nonex iseet children.
Young children immunee
The covid vaceine experimentll
covid vaceine dangerons.
from ex peeimmenall saceines. The proper
Our children need to be protected
protoce not been followed children should pigs and pot it risk. See the Naremberg trials you will be held uccountalle
guinen
Please do oot offer vaccinations for kids They have benstiful immuue systems keep them healthy and fight off vieuses and bacteria.
Do not force childrn to take a dangerous vaccine for sickmess that is nearly nonexistent that age and is casily trcotable bic with other medications.
Utterance
Table 11: Part two of the exemplars for inferential decomposition for FDA comments. We sample n exemplars from this set form prompt, per Table 6. The other exemplars for FDA comments provided in Table 10.

---

Inferences about Utterance Perspeetive [Optional]
Black Americars deserve better treatment by policc
Police should not be violent without couse
People a affected with Long Covid deserve treatment
Foreign interference in US elections wrong
The instittion of policing has flows
Womans' rights should be supported
Police perform importan function in society
Percomal data uvers should he protected
Donald Trump is ant-inmigrant
Illegaa immigrants should uncuted humanely
Inferences about Utterance Subjeet [Optional]
Rossia will be prevented from purchosing election advertisemenss
Russia interfered in the 2016 presidemtal election
Qualified immunity in need refoem
George Floyd's death was unjust
Wyonming supports gander equality
Fltcehonk is not committed to user privacy
Tech compunies profit users" personal data
DACA receipients should recieve ammesty
People who immigrated children rightfal citizens
Police sometimes violate Americans constitutiona rights
Cornavinus can have adverse long-term effects
The Honest Ads Act will strengthen procecions against foreign snneigneeeetinnng election interfeencce
There is an urgent need to help DACA recipients
Propoditions: Explicit [Required]
Mark Zuckerberg makes insincere apologies
Facebook trying to avoid accomntailiiy
Qualitied immuniyy reform should Kocus on formalizing police police hehavior
Many Americuns are batting the long term effects Coronwinus
Rules police use-o force noed be codified
Theating palients with Chrunic Fatigne Syndrome requires funding
A police officer killed George Floyd
Support for Dreamers bipartisan
Wyooming was fiet state recognize right vate
Our nation is hurting. Clorree Floyd's death was horrific jusiice must he served. A single of violence of the hands mn officer is one loe many. George Fleyd deserved beservedeettee beter. All black Americans do. Inleed. all Americans do.
The #oonest Ads stengthen protections against foreign interferenee our election. No more election ads paid fhr in mubles.
Happy Wonning Day! Today, our great Equality Stote selehrates years of being first to officially rsognize women's inberent digtt to vote and hold hold office.
"More "More-apologe apologies from Mark 7 hickerrergg won't fix Facebook. We need accountability and action - not vague commitments to better while continuing to prolit off of users"
Wreneerreamess is as urgent a task as ever. Presidens Trump Tramppreatee created this crisis, and be should stop tanking bipartisun congressional efforts solee it We
Qualified inmuntty reform should have its focus professiomaizzing police de mentssmentss institutionaliiing best police precticss when it cornes
force, protecting corstitutional rights American citizens Survivors of the coconaviros show symptoms of MECTS, debilkuainn and chromic illees that already impacts 2.5 million Americans. am fightinto to secure the funding needed to treat this discass and give paticats they nesd.
to these kids to keep them in the country they ever known as home.
Utterance
Finding pemmanen sulution to
personall data.
Table 12: Exemplars for inferential recomposition of legislative tweets, used in the covote prediction task in
Section 5. We sample n exemplars from this set to form prompt, per Table 6.

---

Introduction
In this survey, you will be answering simple questions about written content from different sources, like news articles and social media posts.
Instructions
You will review two pieces of written content and will answer two questions about the relationship between them. We call these pieces of writing Text 1 and Text 2.
You will answer how reasonable it to conclude that someone who said the first text would also say the second text. In some cases, Text 1 may be expressing opinion or belief. You should answer this question based on what the person who said Text 1 would say (not necessarily what you think).
Sometimes, Text may be factual, like news story. Here, you should answer whether Text plausibly true based Text 1.
2. You will answer whether Text adds new information to Text or changes the information Text 1. By information, mean something that not explicitly said the Text 1. paraphrase does not add new information.
The answer may not always be obvious, so use your best judgment.
Figure 8: Instructions used in the survey for human validation of the generations discussed in $ 3.

---

Examples
Text 1:
There is teal teapot in the kitchen.
Text 2: The kitchen contains teal teapot.
Answer: If someone said Text 1, then it is definitely reasonable that they would also say Text 2. Text
does not add new information to Text 1.
Text 1: The cat meowed in the kitchen before its usual dinnertime.
Text 2:
The cat was hungry.
Answer: someone said Text 1, then probably reasonable conclude that they would also say Text
Text adds new information to Text
Text 1: I'm really hungry right now!
Text 2: I just ate big meal.
Answer: If someone said Text 1, then it is probably not reasonable to conclude that they would also say
Text 2. Text 2 adds new information to Text 1.
Text 1: The plandemic is massive attempt place global power into dictatorship based
complete lie!
Text 2: COVID a hoax
Answer: someone said Text 1, then it is probably reasonable conclude that they would also say Text
2. Text 2 also adds new information to Text 1.
Text 1:
The arraignment of former President Donald Trump has concluded. Trump pleaded guilty 37 charges related to alleged mishandling of classified documents.
Text 2: Donald Trump pleaded not guilty to charges.
Answer: If someone said Text 1, then it is definitely reasonable to conclude that they would also say Text
does not add new information Text
Text 1: Barack Obama was elected president in November 2016.
Text 2: Barack Obama was elected president in May 1900.
Answer: If someone said Text 1, then it is definitely not reasonable conclude that they would also say Text 2. Text 2 adds new information to Text 1.
Figure 9: Examples used the survey for human validation of the generations as discussed in $

---

Introduction
this survey, you will be answering questions about small collections of comments. All comments this survey related to COVID-19 vaccines in children.
As background, the FDA approved vaccination for children ages 5-11 in 2021. During this approval process, FDA sought public commentary. You will reading excerpts of that commentary.
Note that much of the commentary makes reference to known misinformation, falsehoods, and unsubstantiated claims. You should not take the commentary at face value!
Instructions
You will review short lists of commentary, then answer a few questions about how related the Items are each list. Everything related COVID vaccination children, so you are interested what makes them related other than that fact.
First, you will words that succinctly describe the collection. Try your best to write descriptive label.
collection and write a label what they have common. label can ane
Then you will answer how related the collection is, other than the fact that they are all about COVID vaccination. The scale from Very loosely related Very closely related.
Finally, you will read two more documents, and select the one you think best belongs with the others.
Figure 10: Instructions used in the survey for human evaluation of clustering quality as discussed in $ 4.

---

Examples
Here, will show two examples of comment collections that all relate the economy.
Closely-related Collection
1. Its profiteering, not inflation. Inflation is the term they use to cloak the reality and make you think no one is causing this responsible. Its profiteering.
2. A lack of antitrust enforcement is the main driver because it allows companies to raise prices without consequences from competitors or the government.
3. The greed at the corporate level is getting out control. Not sure how long they expect to able to do this, how much they think they can things too expensive for consumers, all the while they're taking in obscene salaries.
make their employees' lives excruciatingly difficult, and
4. Inflation happening, and caused by many economical factors. However is being exasperated by corporate greed, inflation just the mask.
These comments very closely related. descriptive label might "inflation and corporate greed", because the comments discuss corporations raising prices to increase profits.
Loosely-related collection
1. Ban all cryptocurrency entirely and get all of the functional benefits by issuing a central bank digital currency instead.
2. The US is already one of the most economically diverse, least trade dependent nations with wide variety of secure supply lines.
All can say that recently noticed that ONE THING splurge splurgED on (Clif bars they're my husbands favorite and he has one for breakfast every day) have gone nearly $1.50 in 4 months.
I only apply to jobs that have the salary listed now because inevitably if it's not listed, it's terrible salary. I'm thankful that states are passing salary transparency laws that require companies to post the pay range job postings.
These comments are all very loosely related. It is hard to think of more descriptive label than something general like "the economy".
Figure 11: Examples used in the survey for human evaluation of clustering quality as discussed in $ 4.


Reviews:
[{"review_id": "h246r4CMty", "reviewer": "Reviewer_xCQB", "summary": "", "questions": "- A: Is step 4 of the method (line 156-157) intended to take place in practical use cases? And how high is the cost associated with this step?\n- B: What do you mean by the term \"scale\" in your setup and title? There is no discussion in the paper so far.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback, and for recognizing our method as a valuable way to incorporate implicit knowledge in computational social science applications. We also appreciate the acknowledgement of our thorough method assessment for two real-world use cases.\n\n**[Validation of Decompositions]** We’ll respond both to your first “reason to reject” and question A. Step 4 of our protocol involves the validation of LM-generated outputs for a sample from the target dataset. Yes, it is not strictly necessary, because we validate outputs across five datasets spanning diverse domains. In practice, we would recommend this validation check for specialized datasets or sensitive use cases. Even in such cases, this step is inexpensive: a sample of 150 annotations are sufficient for a 5% margin of error when estimating the proportion of valid generations (95% CI, assuming a lower-bound of $p=0.85$, as found in this work). This takes an end-user 50 minutes (based on the median crowdworker time of 20s/item), or 37.5 USD for 3 crowdworkers per item @ 15 USD/hour.\n\nThus, this is a relatively low cost optional step, and the method is indeed fully unsupervised if it is skipped. Our main goal was to follow best practices in computational social science, which is to validate unsupervised model outputs, e.g., topic models (see Grimmer & Stewart, 2013). We agree that this was not clearly distinguished in our paper, and we will address this in the writing of the final version. \n\n**[Method details]** We also agree with your second point that too much of the method is in the appendix, and will move key details to the main text (e.g., discussing a relevant example of the prompt instructions).\n\n**[Meaning of “scale”]** Question B: Thank you for pointing out that we were not sufficiently clear with our usage of the term “scale”. We use our method to automatically “interpret” the implicit & explicit propositional content in large amounts of text data, which we in turn aggregate to help the end-user identify broader themes (in Section 4). In this way, it functions similarly to a topic model, which also facilitates interpretation of text corpora. Hence, the method facilitates efficient analysis of large-scale datasets (e.g., over 5,000 documents), with an optional amount of validation involving at most a small subset, as discussed above. In addition, we note that our method significantly improves reading times of individual text items (see “Evaluation Time” in Figure 3), facilitating human engagement with large datasets. We will clarify this in the final version to make the relationship with the title more evident.\n\nWe will make the suggested presentation improvements. We will also correct the term “generation clusters”, which should have read “*decomposition* clusters”.\n\n**References:**\n* Grimmer, Justin, and Brandon M. Stewart. \"Text as data: The promise and pitfalls of automatic content analysis methods for political texts.\" Political analysis 21, no. 3 (2013): 267-297.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This approach appears to be a valuable method for incorporating implicit knowledge.\n- A thorough assessment of the method's effectiveness is conducted, both in an intrinsical and extrinsical way via well-motivated use cases.", "reasons_to_reject": "-  The proposed method saves manual annotation effort when creating the inferential decompositions. Afterwards, however, these are manually checked for reasonableness and novelty (step 4, line 156-157). It remains unclear how high this additional effort is, with which costs the application of this methodology is associated overall and whether this results in a serious limitation for the application. Or is this step of intrinsic validation only exemplary in the context of this work and omitted in practical application? (This would also fit with Figure 4: \"our approach is also entirely unsupervised\"). This point needs to be clarified and discussed.\n- While the analysis and discussion is very detailed, the explanation of the method itself is very brief and can rather be found in the appendix. More detail is needed here in the main body.", "questions_to_authors": "- A: Is step 4 of the method (line 156-157) intended to take place in practical use cases? And how high is the cost associated with this step?\n- B: What do you mean by the term \"scale\" in your setup and title? There is no discussion in the paper so far.", "paper_topic_and_main_contributions": "This paper presents a method that captures not only the information explicitly encoded in a text unit, but also the associated knowledge that a person relies on when understanding a text. The goal is to support machine learning tasks that rely on interpretation, such as many tasks in computational social science, through datasets with enriched knowledge. Instead of using human annotators to generate the so-called inferential decompositions, the method proposes to use LLMs and then manually validate the generated data for reasonableness and whether it contains new information not explicitly included in the original text unit. Experiments prove the usefulness of the method in two practical use case scenarios: e-rulemaking and legislator behavior. It shows that inferential decompositions lead to a better, more compact and easier to interpret clustering (thematically, opinions).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Typos, please check lines 173, 204-205 (missing commas)\n- A variety of tables and figures are used in this work. This is somewhat cluttered, especially since the Tables/Figures are often not found on the page where they are referenced.\n- If tables are referenced in the appendix, this should be made clear (e.g., see Table 11 in Appendix A.x).\n- The references have to be checked, sometimes page numbers are missing.\n- Figure 3: What are “generation clusters”?"}}, {"review_id": "YVHNaivOgp", "reviewer": "Reviewer_5xdd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for noting that our work is an effective application of LLMs to problems in social sciences, and for viewing our work and discussion as valuable for the NLP community. \n\n**[Commonsense knowledge]** We agree that our inferential decomposition method relates to commonsense knowledge extraction. In our related work section (lines 532-538) we discuss some prior work more specifically focused on commonsense inference. If accepted, we will add relevant references to LLMs’ capacity for commonsense extraction (e.g., Trinh and Le 2018; Bosselut et al. 2019; Yang et al. 2020; Bian et al. 2023), as well as traditional methods like script induction (e.g., Weber et al. 2020) and template-based approaches (e.g., Tandon et al. 2014). These are important precursors to our work. At the same time, we emphasize that our decompositions are not constrained to commonsense knowledge alone.\n\n**[Qualitative Analyses]** We agree that qualitative analyses are important.  We qualitatively examined our method in an intrinsic way: in Table 3, we show that crowdworker-assigned labels of our method’s clusters match narrative themes discovered by experts in prior work; meanwhile, none of the crowdworker labels for baselines recover those themes (Lines 387-391). In Figure 4, we provide examples showing that clusters based on our method reveal opinions at a high level of granularity.  Regarding downstream tasks, in our sentence-similarity experiments (Table 1), we found our method improves correlations with human judgments of Tweet similarity, but does not perform as well on SICK-R (which uses image/video captions). In an unreported experiment, two authors independently scored a sample of 100 LLM-generated paraphrases from the Twitter-STS and SICK-R datasets, and found that LLMs often inject implicit text for Twitter (16%) but never for SICK-R (significant at p<0.05). This led us to conclude that implicit text is important when evaluating the similarity of social language, which informs our CSS experiments in Sections 4 and 5. An example of how our inferential decomposition method can place two tweets closer in embedding space is given in Figure 1.\n\nWe will contextualize this discussion in terms of qualitative performance analysis if accepted; specifically, we will include examples where legislator similarities based on our method better explain voting similarity than the baseline.\n\n**[Meaning of “scale”]** We appreciate your noting that our use of the term “scale” was not clear. Repeating our response to Reviewer 1, we use our method to automatically “interpret” the implicit & explicit propositional content in large amounts of text data, which we in turn aggregate to help the end-user identify broader themes (in Section 4). In this way, it functions similarly to a topic model, which also facilitates interpretation of text corpora. Hence, the method facilitates efficient analysis of large-scale datasets (e.g., over 5,000 documents). In addition, we note that our method significantly improves reading times of individual text items (see “Evaluation Time” in Figure 3), facilitating human engagement with large datasets. We will clarify this in the final version to make the relationship with the title more evident.\n\n**[Reproducibility]** All datasets are public, our appendices include all prompts and exemplars, and code was provided with the supplementary materials (which we will release on acceptance). With regard to human participation in our experimentation, we will also release the human data if accepted, and detailed human annotation instructions are provided in the Appendix.\n\n\n**References:**\n* Acquiring Comparative Commonsense Knowledge from the Web. (Tandon et al., AAAI 2014).\n* A Simple Method for Commonsense Reasoning. (Trinh and Le., arxiv 2018)\n* COMET: Commonsense Transformers for Automatic Knowledge Graph Construction (Bosselut et al., ACL 2019)\n* Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models (Yang et al., COLING 2020)\n* Causal Inference of Script Knowledge (Weber et al., EMNLP 2020)\n* ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models. (Bian et al., arxiv 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It is interesting to see LLMs are effectively applied to problems in social sciences. The discussion on the importance of these tasks is valuable for the NLP community.", "reasons_to_reject": "- The proposed method reminds me of traditional methods for extracting common sense knowledge (like script knowledge, causal relations, etc.), while their relationship is not discussed. Similar ideas have been discussed repeatedly in the history and explaining the essential differences from such previous ideas should be necessary.\n- No qualitative analysis of the proposed method is given. As an NLP perspective, it is important to investigate why the proposed method increases performance.\n- It is not clear what \"interpretation at scale\" means. The title should describe the main content of the paper properly.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method for generating implicitly related propositions of an observed text using large language models.\nA prompt is designed to make large language models output texts that describe propositions behind the input text.\nThis method is evaluated by human evaluation, content clustering, and predicting legislative behaviors.\nThe style of this paper is not like a typical NLP paper, which mainly discusses technological ideas, but is more focused on the problem and task settings in the context of social science.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aWxPotrnhy", "reviewer": "Reviewer_V5xX", "summary": "", "questions": "Question 1: Regarding operationalization: The proportions are formulated following instructions from the appendix, and these examples are prompted to the LLM together with the instructions. An example of these instruction in the paper would suffice. The appendix should contain material to support the paper, but is no substitute to extra pages missing. Main content to present the work in a clear manner should be in the main paper (1st eight pages).\n\nQuestion 2: I cannot follow well the break between the formula in L 137 and the one in L 140. I understand the rationale behind the formula in L 137 (formula numbering makes communication easier), but I don’t understand why it is presented, when the MI between the intension and relevant information-enriched expression is not calculated, and instead inferred propositions are requested via the LLM. How do both formulas relate to each other?\n\nQuestion 3: L 169: Figure 1 is not the same examples as the one in the text. The reference here is confusing.\n\nQuestion 4: L 252: You calculate the similarity between the original sentences? What is the relation between to sentences? Based on what are two sentences selected? Is it any two sentences?\n\nQuestion 5: L 259: Correlation or similarity?\n\nQuestion 6: L 331: Are the two authors you mention, authors from the paper? 31 examples refer to one comment or all comments?\n\nQuestion 7: L 448: Please introduce beta_0 and beta_k.\n\nQuestion 8: L 480: Why do you calculate similarity between two legislators? What happens next? What do the two sets of similarities regard to?\n\nQuestion 9: L 510: Figure 4 is difficult to read. Is this an embeddings dimension reduction and a clustering visualization in one figure? I did not find an explanation on the parameter details of this figure.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We are very pleased that you believe this work fills an important but understudied niche, and that it addresses the key problem of automatically analyzing social communication by relying on implicit information. We are also glad that you find merits in our motivation and the design of our approach. We also particularly appreciate your constructive feedback on the presentation of our experimentation.\n\nIt appears that your concerns revolve primarily around the way we presented the experimentation, and the division of material between the main body and the appendix, not methodological soundness *per se*. As discussed in our response to Reviewer 1, we will expand on the method (e.g., prompt instructions) in the main text. We will also update the experimental sections according to your suggestions:\n* We will put the research questions at the beginning of each section:\n    * Section 4 (Method validation):\n        * Do language models reliably produce plausible explicit & implicit propositions (as judged by human annotators)?\n        * Under the assumption that human similarity judgments make use of both explicit and implicit information, does including such  information in sentence representations improve automated estimates of similarity?\n    * Section 5 (Analysis of public commentary): \n        * Does the representation of comments’ explicit & implicit content lead to improved discovery of themes in a corpus of public opinion (as judged by human annotators)?\n    * Section 6 (Analysis of legislative behavior):\n        * Does the automated similarity between legislators’ explicit & implicit propositions help explain the similarity of their voting behavior? \n* We will explain key hyperparameters & datasets at the beginning of each section, and justifications for our decisions where they do not already exist (as you note, many are included in the references)\n* We will clarify the differences between the similarities used:\n    * In Section 4, we will emphasize that similarities are computed over pairs of sentences from existing semantic textual similarity (STS) datasets\n    * In Section 6, we will move the details of the legislator similarity computation from the appendix to the main text. Similarities are computed over two legislators, who produce multiple tweets, and therefore we must aggregate over all sets of tweets.\n\n**[Responses to questions]** \n* Q1: As discussed, we will illustrate instructions in the main paper, rather than in the appendix.\n* Q2: We appreciate your pointing out this lack of clarity and will update the explanation in the final version (and number the equations!).\n    * As you note, our goal here is to build intuition, and it does not influence our method. We want to show that, over sampled enriched / augmented expressions $R$ from the LLM (equation on L140), the mutual information between intent and these augmented expressions (LHS of equation on L135) increases relative to the MI between the intent and original expression alone. That is, $\\text{I}(E, R ; I ) > \\text{I}(E ; I)$. The identity on L135 shows that this is the case, assuming $\\text{I}(R; I | E) > 0$. Hence, we do not need to calculate it directly: we illustrate that by sampling from an LLM, this MI increases, as desired.\n* Q3: Figure 1 contains an actual, real-world example, whereas the text contains an illustrative example to help with explanation. We can make the two the same in the final version.\n* Q4: This section applies our approach to the standard semantic textual similarity (STS) task (e.g., Cer et al. 2017). Sentence pairs come from existing STS and argument similarity datasets, introduced on lines 204-205 and 209.\n    * Each dataset consists of $n$ sentence pairs, and annotators score each pair based on the semantic similarity. We follow prior work and construct distributed representations (i.e., embeddings) of each sentence in a pair, then compute an automated similarity (here, cosine).  So “I like NLP” / “I love NLP” might receive a human score of 4.5 out of 5, and the cosine similarity of the embeddings might be 0.9\n* Q5: This answer follows from Q4. We compute the spearman correlation between embedding similarities and human-annotated similarities across all sentence pairs in each dataset (Table 1). The higher correlation implies that embeddings based on our method are more aligned with human similarity judgments.\n* Q6: Yes, the exemplars are created by two of this paper’s authors following a consensus process. Multiple propositions are produced for 7 original comments, totalling 31 (comment, proposition) pairs; we will make this explicit in the final version.\n* Q7: The $\\beta$ are learned parameters of the regression model, we will introduce them right after the equation. Thank you for pointing this out.\n* Q8: We can include details of the similarity calculation in the main text, instead of the appendix.\n    * Prior work computes similarities between legislators to help explain their joint voting behavior, and we follow this experimental setup (see references on lines 407-419). We measure the embedding similarity between legislators using different representations of their online speech. Specifically, we compare similarities based on their observed tweets (a baseline) to those based on generated propositions (our proposed method). Our hypothesis is that legislators whose speeches contain semantically similar propositional content will have similar voting behavior. We find evidence supporting this hypothesis.\n* Q9: Yes, the graph is representing both a dimension reduction of the embeddings as well as a k-means clustering of those embeddings ($K=10$). Two clusters are circled, which correspond to Republican/conservative and Democratic/liberal framings of the topic of abortion. We will elaborate on the hyperparameters and the method used to generate this figure, and refine it to be more straightforward.\n\n**[Reproducibility]** We recognize this is somewhat subjective, but as noted to reviewer 2, all datasets are public, our appendices include all prompts and exemplars, and code was provided with the supplementary materials (which we will release on acceptance). With regard to human participation in our experimentation, we will also release the human data if accepted, and detailed human annotation instructions are provided in the Appendix.\n\n**References:**\n* SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (Cer et al., SemEval 2017)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Positive: I enjoyed the motivation and overall idea to overcome the gap between intention and expression. I really think that this is a very important niche and improving/developing supportive, automated ways to analyze communication in any social field is really an underestimated task in our society. I also see some merits in the overall design and approach to make use of LLM prompting to collect extra world knowledge.", "reasons_to_reject": "Problems: I understand, you operationalize your study by clustering and measuring embeddings similarities of your augmented data. This way you can explain the results more closely. However, I feel a bit lost in the way you design the experiments. A better structuring of the work can help the reading. When you explain the setup of an experiment, give structured details on the Research Question, the data and its format, the technique you use, the parameters for the algorithm and an explanation of the parameter values selection. You often refer to the literature, which is good, but give to few details on the technology you use. Often similarities are calculated, but these are not justified well and nor are they properly referred to (“do not let your appendix speak for you”).", "questions_to_authors": "Question 1: Regarding operationalization: The proportions are formulated following instructions from the appendix, and these examples are prompted to the LLM together with the instructions. An example of these instruction in the paper would suffice. The appendix should contain material to support the paper, but is no substitute to extra pages missing. Main content to present the work in a clear manner should be in the main paper (1st eight pages).\n\nQuestion 2: I cannot follow well the break between the formula in L 137 and the one in L 140. I understand the rationale behind the formula in L 137 (formula numbering makes communication easier), but I don’t understand why it is presented, when the MI between the intension and relevant information-enriched expression is not calculated, and instead inferred propositions are requested via the LLM. How do both formulas relate to each other?\n\nQuestion 3: L 169: Figure 1 is not the same examples as the one in the text. The reference here is confusing.\n\nQuestion 4: L 252: You calculate the similarity between the original sentences? What is the relation between to sentences? Based on what are two sentences selected? Is it any two sentences?\n\nQuestion 5: L 259: Correlation or similarity?\n\nQuestion 6: L 331: Are the two authors you mention, authors from the paper? 31 examples refer to one comment or all comments?\n\nQuestion 7: L 448: Please introduce beta_0 and beta_k.\n\nQuestion 8: L 480: Why do you calculate similarity between two legislators? What happens next? What do the two sets of similarities regard to?\n\nQuestion 9: L 510: Figure 4 is difficult to read. Is this an embeddings dimension reduction and a clustering visualization in one figure? I did not find an explanation on the parameter details of this figure.", "paper_topic_and_main_contributions": "Content: The paper describes a technique to text analysis by formulating (using LLM’s) propositions that contain more explicit knowledge beyond the implicit meaning of a certain utterance. The idea behind is that the intention of an utterance is not necessarily 100% visible in a certain expression (formulation), hence, extra (world) knowledge is helpful to improve text analysis at scale. The technique is tested against a crowd-souring approach, and two downstream tasks (clustering comments on one controverse topic, and legislator “similarity-based prediction”).", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "see Questions"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for the “interpretation” of text at scale. The authors propose using LLMs to generate propositions which are “inferentially related” to an original text (see AC suggestions at the bottom). They propose that these propositions provide “explicit representations” of the “implicit” interpretations that come with reading textual information. The authors validate generated propositions with a simple crowd experiment, and test the utility of generated propositions for clustering public comments and predicting legislator voting patterns.

Reviewers offered a largely positive assessment of the soundness and excitement of this work, especially after discussing with authors during the rebuttal period. All soundness and excitement scores were 3 or higher (following discussion). Reviewer xCQB described the paper as “a valuable method for incorporating implicit knowledge” and suggested that the authors provided a “thorough assessment of the method's effectiveness.“ Reviewer 5xdd noted that it was interesting to see how LLMs could be “effectively applied to problems in social sciences” and reviewer V5xX liked that it supported “automated ways to analyze communication” in a social field.

However, reviewers did offer some negative comments about the paper. The AC has summarized negative comments below, in order of what the AC believes to be the most and least important to acceptance/rejection decisions.

Reviewer 5xdd noted a lack of comparison with “traditional methods for extracting common sense knowledge.” In the AC’s opinion, this does seem like a limitation of the work. But this limitation should be placed in a broader context. Overall the authors did a good job grounding their submission in recent work from NLP and CSS, so this shortcoming does not seem core to the underlying soundness or excitement of the submission.  

- Note: in the discussion, the authors replied by suggesting that “commonsense knowledge” and “inferentially related” knowledge are not equivalent. They included references to work on LLMs for commonsense reasoning. But some of the nuances between inferentially-related reasoning and commonsense reasoning did not come through on OpenReview.  See the AC’s suggestion about “inferentially related” at the end of this review. 

Reviewer xCQB wondered if the method required human validation, which might limit the utility of the work. The authors replied that human validation was a “low cost optional step” best applied in “sensitive” use cases. Figure 2 seems to show that (at least for these experiments) human plausibility judgments seem to validate the LLM. Thus it seems like in the future you could use this method with less crowdworker validation (in some settings). So this does not seem like a major shortcoming of the work.

Reviewer 5xdd asked for qualitative analysis of the method, especially regarding cases where the method does not work. The authors replied with explanations of the ways in which their paper does incorporate qualitative analysis, and promised to contextualize the paper with examples upon acceptance. They also promised to analyze cases where “the method does not perform as well relative to the baseline.” Reviewer 5xdd replied to say that the authors’ “responses make sense to me mostly” which seems to largely address this issue, provided the authors make the requested changes.

Some of the reviews critiqued the style of the submission. These things might be changed during future revisions, and do not seem central to acceptance decisions. The requested stylistic changes are as follows. (Note: the AC agrees with reviewer 5xdd about the title.)
- Both reviewer xCQB reviewer V5xX noted that the authors should add more detail about the method in the main body of the paper. The authors agreed that this stylistic change was necessary during discussion. This is a stylistic issue that could be addressed during revision.
- Reviewer 5xdd pointed out that the title of the paper may not reflect its content. They noted that the paper should have a more concrete and descriptive title that accurately describes the body of the work.

While this metareview largely focuses on negative feedback, in total, it seems like the reviewers offered a largely positive assessment of the paper, and that the limitations they brought up appeared to be (1) addressed during the rebuttal period or (2) not central to the soundness or excitement of the work.

Suggestions from AC: It was hard to find the precise definition of “inferentially related” in this paper. The related work section helped explain some of the ideas behind the approach, but a more explicit statement of what “inferentially related” means would help the paper, even if the “formal semantic representation” (L525) of this phrase is itself “relaxed” (L535). Also, the AC agrees with 5xdd that the paper should have a more clear and descriptive title.
Decision: Accept-Main