SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings
Authors: Kang Liu
Venue: ICLR 2024 poster (2024)
Keywords: Representation Learning
Natural Language Processing
Contrastive Learning
Set Operation
Querying Framework
Sentence Embedding
Deep Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce SetCSE, an innovative information retrieval framework which employs sets to represent complex semantics and incorporates well-defined operations for structured information querying within the provided context.

Abstract: Taking inspiration from Set Theory, we introduce SetCSE, an innovative information retrieval framework. SetCSE employs sets to represent complex semantics and incorporates well-defined operations for structured information querying under the provided context. Within this framework, we introduce an inter-set contrastive learning objective to enhance comprehension of sentence embedding models concerning the given semantics. Furthermore, we present a suite of operations, including SetCSE intersection, difference, and operation series, that leverage sentence embeddings of the enhanced model for complex sentence retrieval tasks. Throughout this paper, we demonstrate that SetCSE adheres to the conventions of human language expressions regarding compounded semantics, provides a significant enhancement in the discriminatory capability of underlying sentence embedding models, and enables numerous information retrieval tasks involving convoluted and intricate prompts which cannot be achieved using existing querying methods.

Published as a conference paper at ICLR 2024
SETCSE: SET OPERATIONS USING CONTRASTIVE LEARNING OF SENTENCE EMBEDDINGS
Kang Liu Independent Researcher kangl fu@umi ch. edu
ABSTRACT
Taking inspiration from Set Theory, we introduce SetCSE, innovative information retrieval framework. SetCSE employs sets to represent complex semantics and incorporates well-defined operations for structured information querying under the provided context. Within this framework, we introduce an inter-set contrastive learning objective to enhance comprchension of sentence embedding models concerning the given semantics. Furthermore, we present a suite of operations, including SerCSE intersection, difference, and operation series, that leverage sentence embeddings of the enhanced model for complex sentence retrieval tasks. Throughout this paper, we demonstrate that SetCSE adheres the conventions of human language expressions regarding compounded semantics, provides significant enhancement in the discriminatory capability of underlying sentence embedding models, and enables numerous information retrieval tasks involving convoluted and intricate prompts which cannot be achieved using existing querying methods.
INTRODUCTION
Recent advancements in universal sentence embedding models (Lin et al., 2017: Chen et al., 2018;
Reimers & Gurevych, 2019; Feng al., 2020; Wang & Kuo, 2020; Gao et al., 2021; Chuang et al., 2022; Zhang et al., 2022; Muennighoff, 2022; Jiang et al., 2022) have greatly improved natural language information retrieval tasks like semantic search, fuzzy querying, and question answering (Yang et al., 2019; Shao et al., 2019; Bonial et al., 2020; Esteva et al., 2020; Sen et al., 2020). Notably, these models and solutions have been primarily designed and evaluated on the basis of single- sentence queries, prompts, or instructions. However, both within the domain of linguistic studies and everyday communication, the expression and definition complex or intricate semantics frequently entail the use of multiple examples and sentences "collectively" (Kreidler, 1998; Harel & Rumpe, 2004; Riemer, 2010). In order to express these semantics a natural and comprehensive way, and search information for in straightforward manner based the provided context, propose Set Operations using Contrastive I Learning of Sentence Embeddings (SeCCSE), a novel query framework inspired by Set Theory (Cantor, 1874; Johnson-Laird, 2004). Within this framework, each set of sentences is presented to represent a semantic. The proposed inter-set contrastive learning empowers language models better differentiate provided semantics. Furthermore, the well-defined SetCSE operations provide simple syntax to query information structurally based on those sets of sentences.
SetA Faviranmune
Set Earniag Calk
BnBrD
fA
UnBnDD
Swcial
Set
SupaiiE
D
Anabe:
a soggl wark ar sffo. adter tene deaeriiieiei andhiieees
Governanee
SatE Dasge
Step Inter-set Contrastive Leaming
Step Complex Query by SetCSE Operations
Figure 1: The illustration of inter-set contrastive learning and SetCSE query framework.
As illustrated Figure 1, SetCSE framework contains two major steps, the first is to fine-tune sentence embedding models based inter-set contrastive learning objective, and the other is to

---

Published as a conference paper at ICLR 2024
retrieve sentences using SetCSE operations. The inter-set contrastive learning aims to reinforce underlying models to learn contextual information and differentiate between different semantics conveyed by sets. An in-depth introduction of this novel learning objective can be found in Section 3. The SetCSE operations contain SetCSE intersection, SerCSE. difference, and SetCSE operation series (as shown in Figure Step 2), where the first enable the "selection" and "deselection" of sentences based on single criteria, and the serial operations allow for extracting sentences following complex queries. The definitions and properties of SctCSE operations can be found in Section 4.
Besides the illustration of this framework, Figure also provides an example showing how SeCCSE can be leveraged to analyze S&P 500 companies stance on Environmental, Social, and Governance (ESG) issues through their public earning calls, which can play an important role in company growth forecasting (Utz, 2019; Hong et al., 2022). The concepts of ESG are hard to convey in single sentences, which creates difficulties for extracting related information using existing sentence retrieval method. However, utilizing SetSS framework, one can easily express those concepts in sets of sentences, and find information related to "using technology to solve social issues, while neglecting its potential negative impact" in simple syntax. More details on this example can be found in Section
This paper presents SetCSE in detail. Particularly, we highlight the major contributions as follows:
The employment of sets to represent complex semantics in alignment with the intuition and conventions of human language expressions regarding compounded semanties.
2. Extensive evaluations reveal that SctCSE enhances language model semantic comprehension by approximately 30% on average.
3. Numerous real-world applications illustrate that the well-defined SetCSE framework enables complex information retrieval tasks that cannot be achieved using existing search methods.
2 RELATED WORK
Set theory for word representations. In Computational Semantics, set theory is used to model lexical semanties of words and phrases (Blackburn & Bos, 2003; Fox, 2010). An example of this is the WordNet Synset (Fellbaum, 1998; Bird et al., 2009), where the word dog is component of the synset {dog. domestic dog, Canis familiaris) }. Formal Semantics (Cann, 1993; Partee et al., 2012) employs sets to systematically represent linguistic expressions. Furthermore, researchers have explored the use of set-theoretic operations on word embeddings to interpret the relationships between words and enhance embedding qualities (Zhelezniak et al., 2019; Bhat al., 2020; Dasgupta et al., 2021). More details on the aforementioned and comparison with our work are included in Appendix A.
Sentence embedding models and contrastive learning. The sentence embedding problem is extensively studied in the area of nautral language processing (Kiros et al., 2015; Hill et al., 2016; Conneau al., 2017; Logeswaran & Lee, 2018; Cer et al., 2018; Reimers & Gurevych, 2019). Recent work has shown that fine-tuning pre-trained language models with contrastive learning objectives achieves state-of-the-ar results without even using labeled data (Srivastava et al., 2014; Giorgi et al., 2020; Yan et al., 2021; Gao et al., 2021; Chuang et al., 2022; Zhang et al., 2022; Mai et al., 2022), where contrastive learning aims to learn meaningful representations by pulling semantically close embeddings together and pushing apart non-close ones (Hadsell et al., 2006; Chen et al., 2020).
3 INTER-SET CONTRASTIVE LEARNING
The learning objective within SetCSE aims to distinguish sentences from different semantics. Thus, we adopt contrastive learning framework in Chen al. (2020), and consider the sentences from different sets as negative pairs. Let hn and h, denote the embedding of sentences m and n, For N number of sets, S,, respectively, and sim(hm, b,,) denote the cosine similarity Il..
h
i 1, 1,..., N, where each St represent a semantic, the inter-set loss Linter is defined as:
N
Linter-set Σ l,. where L, = Σ log Σ
mES;
ngS;
Specifically, é, denotes the inter-set loss of S, with respect to other sets, and T is for temperature setting.
2

---

Published as a conference paper at ICLR 2024
As see, strictly following Equation 1, the number of negative pairs will grow quadratically with N. In practice, we can randomly pick a subset of the combination pairs with certain size to avoid this problem.
Our evaluations find that the above learning objective can effectively tine-tune sentence embedding models to distinguish different semantics. More details the evaluation can be found in Section 5.
4 SETCSE OPERATIONS
In order to define the ScICSE operations, we first quantify "semantic closeness", i.e., semantic similarity, sentence to a set sentences. This closeness measured by the similarity between the sentence embedding to the set embeddings.
Definition The semantic similarity, SIM(x, S), between sentence x and set of sentences S is defined
SIM xz, 5)
1
Σ sim(h,, hx),
JS|
RES
where sentence & represents sentences in S, and h denotes the sentence embedding.
4.1 OPERATION DEFINITIONS
For the sake of readability, we first define the calculation of series of SerCSE intersection and difference, and then derive the simpler case where only single SetCSE intersection or difference operation is involved.
Definition 2. For given series of SetCSE operations An Bi n By N... NMM, the result an ordered set A, denoted (A, 3), where the order relationship is defined as
M
N
M
x 3 y if and only if SIM(2, B,) for all x and y in A. Remark. As one the elements in A by the similarity with sets B, and dissimilarity with Cj. In practice, when using SetCSE querying framework, one rank the sentences in descending order and select the top ones which are semantically close to B; and different from Cj. Another observation is that series of SetCSE operations is invariant to operation orders, other words, we have A AnD B C A ANCO B.
Σ SIM(I, CJ) S
SIM(y, Bd)
SIM(y, Cý), (3)
the ScCCSE operations An B1 n.... By C1 ... rank order
Following Definition 2, SetCSE intersection and difference are given by Lemma and 2, respectively. Lemma 1. The SerCSE intersection A i B equals (A, =), where for all a. A,
x z y and only if SIM(r, B) & SIM(y, B).
(4)
Lemma 2, The SetCSE difference A C equals (A, 3). where for all x. E A,
チ y and only if SIM(æ, C) SIM(y, C*).
(5)
SetCSE intersection or difference does not satisfy the commutative law, in other words, An B 4 B.A, and ANC + C'A.A A. The advantage and limitation of the properties mentioned in Remarks are discussed in Appendix B.
Remark.
4.2 ALGORITHM
Combining Section 3 and the above in Section 4, present the complete algorithm for SetCSE operations in Algorithm 1. As one can see, the algorithm contains mainly two steps, where the first step fine-tune sentence embedding model by minimizing inter-set loss Lintessa and the second one is to rank sentences using order relationship in Definition 2.
5 EVALUATION
In this section, we present the performance evaluation of SetCSE intersection and difference. The evaluation of series of SetCSE operations presented in details in Appendix C.4.
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 SerCSE Operation An B1 n.. n.....B VCI N...B.CCN.....C...
1: Input: Sets of sentences A, BI...., Bv. .i...., Cm. sentence embedding model øb
2: Fine-tune model @ by minimizing Linter-set w.r.t. Bi,.... BN, Ci...., CMs, denote as ø
3: for sentence in do
4: Compute E SIM(z, B.) -ド SIM(x, Cý), where all embeddings are induced by ø
5: end for
6: Form (A, and rank sentences in A in descending order
To cover a diverse range of semantics, we employee the following datasets this section: AG News Title and Description (AGT and AGD) (Zhang et al., 2015), Financial PhraseBank (FPB) (Malo al., 2014). Banking77 (Casanueva et al., 2020), and Facebook Multilingual Task Oriented Dataset (FMTOD) (Schuster et al., 2018).
We consider an extensive list models for generating sentence embeddings, including encoder-only Transformer models such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019b), their fine-tuned versions, such as SimCSE (Gao et al., 2021), I, DifICSE (Chuang et al., 2022). and MCSE (Zhang et al., 2022), which are for sentence embedding problems, Contriever (Izacard et al., 2021), which is for information retrieval; the decoder-only SGPT- model (Muennighoff, 2022) is also included. In addition, conventional techniques such TFIDF, BM25, and DPR (Karpukhin et al., 2020) are considered as well.
5.1 SETCSE INTERSECTION
Suppose labeled dataset S has N distinct semantics, and S; denotes the set of sentences with the i-th semantic. For SetCSE intersection performance evaluation, the experiment is set up as follows: 1. In each S; randomly select nsample of sentences, denoted as Qi, and concatenate remaining sentences in all S,, denoted as U. Regard Q2 's as example sets and U as the evaluation set. For each semantic conduct U n Q; following Algorithm and select the top |S,] Rample sentences. View the i-th semantic as the prediction of the selected sentences and evaluate accuracy and F1 against ground truth.
As a control group, repeat Step while omitting the model fine-tuning Algorithm 1.
Throughout this paper, each experiment is repeated 5 times to minimize effects of randomness. The hyperparameters are selected as ssmmpl 20, 0.05, and train epoch equals 60, which are based on fine-tuning results presented in Section 7 and Appendix C.
News-T
AC Nows-
Banking
Ace
FI
BM25
38.91
41.19
DPR
25.00
25 00
25.00
25 25.00
33.33
38.81
41.30
41.35
38.33
42.87
42.02
$3.40
BFRT
43.83
RoBERTa
40.75
40.78
$4.58
$4.40
$4.69
53.07
75.10
64.49
72.0 72.60
70.84
Existing Model
Set Insersect
SGPT
54.38
34.90
34.977
35
52.83
52.89
37.35
37.84
79,80
78.86
SinCSE-BERT
55.28
55.00
68.07
58
$6.13
53.79
82.69
82.50
90.01
89.93
SimCSE SimCSE-RoBERR RoBEKIa
29.68
49.72
64.87
93.44
93.47
DIFUSE-BERT
49.04
49.95
61.64
61.31
50.88
47.77
83.02
82.87
91.61
91.42
DiNCSE-R.BERT.
46.61
60.05
82.95
MCSE-BERT
29.98
.9.91
68.79
68.14
54,01
50.89
77.35
77.21
93.56
92.49
MCSF-RoRFRTS
46.32
46.29
$7.10
$5.96
53.32
85.80
85.60
04.39
94.30
BERT
70.47
70.32
87.24
71.65
71:01
95.06
95.06
08.00
98
RoBERTs
75.87
75.71
88.30
83.26
73.76
73.09
83.59
83.66
99.39
99.39
Centriever
72.88
72.77
83.97
83.09 99
67.88
67. 50
94.20
94.22
97.03
97.05
SGPT
86.65
Interseut
SeCSE
SimCSE BERT
77.24
73.22
89.48
89.46
33.59
83.44
97.84
97.84
99.63
99.63
SinCCB-RRBERIL
79.57
98.35
DiBCSE BERT
76.43
76.45
78.31
78.30
80.93
80.84
98.24
98.25
99.79
99.79
CHTCSE-R-BERTE
78.02
04
88.63
83.89
83.75
98.49
97.55 89
97.87
MCSE BERT
75.04
63.96
88.77
83.75
84.03
83:97
97.76
97.76
99.53
99.53
78.18
78.21
89
80 23
86.21
86.00 96.08
98.65
08.65
98. 66
08.65
Ave. Improsement.
56%
57%
46%
47%
50%
27%
12%
12%
Table 1: Evaluation results for SerCSE intersection. As illustrated, the average improvements on accuracy and Fl are 39% and 37%, respectively.
4

---

Published as a conference paper at ICLR 2024
YOU
Our noa
Sparts
Sparc
Sillee Soorts
Silleh
Solteh
Sillich
Sporss
Sparts
GERT
EMCSE-DEAT
DIPCSEC GERT
NCSE-CERT
SGPT-125M
Bustneer Softee
Busineed
Rusl SoiTich
Saftee
Softech
Settach
ta
Spara
Spari
Rest Softeet
Rudi wes
Ruil
Buc Settcch
Softec
Scttch
SmCCEERRuBeTt
DIFCSE RoBERT
NCSE ReBeRTa
Contriever
Figure 2: The t-SNE plots of sentence embeddings induced by existing language models and the SetCSE fine-tuned ones for AGT dataset. As illustrated, the model awareness of different semantics are significantly improved.
portice
postion
peative
portio
repatre
restral requrtive
rogalivs
ropatre
SHICSE-BERT
CHICSE
SCPT.LESM
postive restral
pos't
tive
posthie nestral
postive
pastis
SinCS6 RussiTu
DITICE RUSERTa
MCSE RUBERD
Conminsnn
Figure 3: The I-SNE plots of sentence embeddings induced by existing language models and the SeICSE fine-tuned ones for FPB dataset. As illustrated, the model awareness of different semantics are significantly improved.
The detailed experiment results can be found in Table 1, where "SetCSE Intersection" and "Existing Model Set Intersection" represent results in Step 2 and 3, respectively. To illustrate the performance in a more intuitive manner, we include the t-SNE (Van der Maaten & Hinton, 2008) plots of the sentence embeddings, as shown in Figures 2 and 3 (refer to Section C for (-SNE plots of AGD, Banking77 and FMTOD datasets). As one can see, on average, the SetCSE framework improves performance of intersection by 38%, indicating a significant increase on semantic awareness. Moreover, the encoder- mnceder-based models perform better than the decoder-based SGPT. This phenomenon and potential future works discussed in detail in Appendix C.3.
5

---

Published as a conference paper at ICLR 2024
5.2 SETCSE DIFFERENCE
Suppose a labeled dataset S has N distinct semantics, and S; denotes the set of sentences with the i-th semantic. Similar to Section 5.1, the evaluation on SetCSE difference set up as follows:
1. In each S;, randomly select neampl of sentences, denoted by Qi and concatenate remaining sentences in all S,, denoted as U.
For each semantic i, conduct U QQ following Algorithm 1, and select the top (1 Psample) sentences, which are supposed have different semantics than i. Label the selected sentences as "not i", relabel ground truth semantics other than i to "not 1" as well. Evaluate prediction accuracy and Fl against relabeled ground truth.
3. As a control group, repeat Step 2 while omitting the model fine-tuning in Algorithm 1.
Results for above can be found Table Similar to Section 5.1, we also observed significant accuracy and F1 improvements. Specifically, the average improvement across all experiments is 18%.
AG News:T
AG News-D
FFB
Banking?7
FMTOD
P1
F-
Acc
FI
BM25
57.47
60.77
71.74
DPR
$6.50
54.45
56.98
54.76
66.67
58.06
68.70
68.06
71.67
68.45
TFLD
23.76
40.13
51.33
32.37
38.58
ВEKT
79,05
70.00
85.42
ReRERT:
20.35
58.
77.15
67.22
77.90
69 46
75.10
64.40
86.41
80.44
Existing Model
Sei Diflenence
Contriever
77.07
79.55
87.75
SGPT
67.85
54.87
67.85
$4.86
76.71
67.3 14
69..3
56.84
89.95
85.38
SinCSE-BERT
77.64
67.87
84.04
76.77
78.06
71.14
91.92
88.07
90.30
86.21
SimCSE RoBERIa
74,84
64.09
77.40
67.68
83,05
76.57
92.59
89,05
96.71
95.11
DINCSE-BERT
74.97
64.27
80.80
72.30
75.44
68. 34
91.84
87,95
95.80
93.79
DiOCSE
MCSE- BERT
74.72
63.96
K3.24
75.68
78.46
71.23
88.43
83,03
90.66
95.05
MCSE-RoRERTA
73.07
61.70
77.75
78.87
71.30
92.65
89.13
97.10
95.68
BEXT
87.39
92.92
94,91
78.
97.22
99.42
99.1
RoBLKII
$4.36
80.85
99.55
Contriever
85.03
79.56
92.95
89.61
83.50
76.27
95.11
92.78
98.94
98.42
SGPT
58.44
SaCSE
Difference
SimCSE BERT
68.62
$3.30
94.74
92.20
91.80
87.90 $7.99
99.04
98.56
99.81
99.72
GINCSE-ROBERT.
89.78
$4.98
92.74
99.29
09:72
09. 08.58
DiBCSE BERT
85:22
$2.72
94,
92.13
90.67
86.07
99.04
98.55
99.90
99.85
DiTTSE-ROREIT)
80.01
87
94.31
88
99.
98.72
98.95
98.43
MCSE-BERT
88:33
90.89
98.94
999.76
99.64
MCSF-RoRERT
89.86
$5.09
94. 36
91.65
93.19
90.00
99.14
98.72
99.63
99.45
Ave. Ingrovement
19%
31%
15%
21.95
6%
8%
Table 2: Evaluation results for SetCSE difference. As illustrated, the average improvements on accuracy and F1 are 14% and %, respectively.
As mentioned, the evaluation of SerCSE series of operations can be found Appendiex C.4. The extensive evaluations indicate that SetCSE significantly enhances model discriminatory capabilities, and yields positive results SerCSE intersection and SetCSE difference operations.
6 APPLICATION
As mentioned, SetCSE offers two significant advantages in information retrieval. One is its ability to effectively represent involved and sophisticated semantics, while the other is its capability to extract information associated with these semantics following complicated prompts. The former is achieved by expressing semantics with sets of sentences phrases, and the latter is enabled by series of SetCSE intersection and difference operations. For instances, operation An B1 .... n By Ci ... essentially means "to distinguish the difference between Bı. BN, CI. .OM, and to find sentences in A that contains semantics in B, while different from semantics C, 's."
In this section, we showcase in detail these advantages through three important natural language processing tasks, namely, complex and intricate semantic search, data annotation through active learning, and topic discovery. The datasets considered cover various domains, including financial analysis, legal service, and social media analysis. For more use cases and examples that are enabled by SetCSE, refer to Appendix D.

---

Published as a conference paper at ICLR 2024
6.1 COMPLEX AND INTRICATE SEMANTIC SEARCH
In many real-world information retrieval tasks, there is a need to search for sentences with or without specific semantics that are hard to convey in single phrases or sentences. In these cases, existing querying methods based on single-sentence prompt are of limited use. By employing SerCSE, one can readily represent those semantics. Furthermore, SetCSE also supports expressing convoluted prompts via its operations and simple syntax. These advantages are illustrated through the following financial analysis example.
In recent years, there has been an increasing interest in leveraging a company's 's Environmental, Social, and Governance (ESG) stance to forecast its growth and sustainability (Utz, 2019; Hong et al., 2022). Brokerage firms and mutual fund companies have even begun offering financial products such as Exchange-Tradee Traded Funds (ETFs) that adhere to companies" ESG investment strategies (Kanuri, 2020; Rompotis, 2022) (more details ESG are included in Appendix D.3). Notably, there is no definitive taxonomy for the term (CFA Institute, 2023), and lists of key topics are often used to illustrate these concepts (refer to Table 3a).
The intricate nature of ESG concepts makes it challenging to analyze ESG information through publicly available textual data, e.g., S&P 500 earnings calls (Qin & Yang. 2019). However, within the SetCSE framework, these concepts can be readily represented by their example topics. Combined with several other semantics, one can effortlessly extract company earnings calls related convoluted concepts such as "using technology to solve Social issues, while neglecting its potential negative impact," and "investing in Environmental development projects," through simple operations. Table 3 provides a detailed presentation of the corresponding SetCSE operations and results.
Set A-
Envirommental
{climate change, carbon emission reduction, water pollution, air pollution, renewable energy)
Set B Social
{diversity inclusion, community relations, customer satislaction, fair wages, data security)
Set C Governance
fothical practices, transparent accounting, business integrity. risk management, compliance
Sei D) New Tech
(machine learning, artificial intelligence, robolics, generative model, neutral netwarks)
Set E Danger
[personal privacy breach, wrongful disclosure, pose threat, misinformation, unemployment)
Set F- Invest
(strategic investement. growth investment, strategic plan, invest, investment)
(a) Examplc topics for defining ESG (Investopedia, 2023) and example phrases for other semantics.
Operation: X n BOBDDV /xdind sentence about "use tech to influence social issues poaseively".
Results:
We're now using machine learning in most of our integrity work to keep our community safe.
But we know we also have a responsibility to deliver these fundamental technical and advances to fulfill the promise of bringing people closer together.
Our data and technology combined with specialized consulting experience help organization transition to digital future while ensuring their workforce thrives.
And you'll 'll see us integrating advances in machine learning so that eustomers can get better satisfaction
(b) Search for sentences related to to"using technology to solve Social issues, while neglecting its potential negative impact," utilizing a serial of three SetCSE operations.
Operation: X n F
/sfind sentence about Winvest in environmental development
Results:
We also continue to make progress on the $1.5 billion of undefined renewahle prejects, which are included in our capital forecast.
To that end, our growth initiatives beyond the projects under construction have been focused investments in natural gas and renewable projects with long term.
I would note that the $9.7 billion plan includes the natural gas storage well as the UP generation investment that I just discussed.
(c) Search for sentences related to "investing in Environmental development projects," via simple SetCSE syntax.
Table 3: Demonstration of complex and intricate semantics search using SetCSE serial operations, through the example of analyzing S&P 500 company ESG stance leveraging carning calls transcripts.
7

---

Published as a conference paper at ICLR 2024
6.2 DATA ANNOTATION AND ACTIVE LEARNING
Suppose building a classification model from scratch, and only an unlabeled dataset is present. Denote the unlabeled dataset as X, and each class as 1, .N. One quick solution is to use SetCSE as a filter to extract sentences that are semantically close to example set S; for each i, and then conduct a through human annotation, where the filtering conducted using X n S,. More interestingly, SetCSE supports uncertainty labeling in active learning framework (Settles, 2009; Gui et al., 2020), where the unlabeled items near a decision boundary between two classes i and j can be found using X i S.j.
We use Law Stack Exchange (LSE) (Li et al., 2022) dataset to validate the above data annotation strategy. Categories in this dataset include "copyright", "criminal law", "contract law". etc. Table 4a presents the sentences selected based on similarity with the example sets, whereas Table 4b shows the sentences that are on the decision boundaries between "copyright" and "criminal law". The latter indeed are difficult to categorize at first glance, hence labeling those items following the active learning framework would definitely increase efficiency in data annotation.
Operation: n
/+find seatences related to "capyricht"./
Results:
Who owns a copyright on scanned work?
How does copyright on Recipes work?
Is OCRed text automatically copyright?
Operation: X n Sa
/+tind related "criminal Lax"+/
Results:
Giving Someone Money Because of Criminal Act?
What are techniques used in law robustly incentivize people tell the truth?
Canada how long can person under investigation?
(a) Extract sentences close to "copyright" "criminal law" categories for further human annotation.
Operation: X n S1 0 S₂ /*Eind sentences decision boundary ot "copyyighe"" and "criminal lax
Results:
there any country or state where the intellectual author of homicide has twice or more the penalty than the physical author?
Would police in the US have any alternative for handling a confiscated computer with a hidden partition?
Is there criminal database for my city Calgary
Audio fingerprinting legal issues
Is reading obscene written material online illegal in the UK?
(b) Following uncertainty labeling strategy in active learning framework, find sentences the decision boundary between "copyright" and "criminal law" categories with the help of SetCSE serial operations.
Table 4: Demonstration of LSE dataset annotation and active learning utilizing SetCSE
6.3
NEW TOPIC DISCOVERY
The task of new topic discovery (Blei & Lafferty, 2006; AlSumait et al., 2008: Chen et al., 2019) emerges when dataset of interest is evolving over time. This can include tasks such as monitoring customer product reviews, collecting feedback for currently airing TV series, or identifying trending public perception of specific stocks, among others. Suppose we have a unlabeled dataset X, and N identified topics, the SetCSE operation for new topic extraction would be X XVT, stands for the set of example sentences for topic i.
AN, where Tj
We use the Twitter Stance Evaluation datasets (Barbieri et al., 2020; Mohammad et al., 2018: Barbieri et al., 2018; Van Hee et al., 2018; 8; Basile et al., 2019; Zampieri et al., 2019; Rosenthal et al., 2017; Mohammad et al., 2016) to illustrate the new topic discovery application. Specifically, we select "abortion", "etheism". and "feminist" as the existing topics, denoted as Ti. Tg. and T,, and use X TH NT, \ T, to find sentences with new topics. In the created evaluation dataset, the only other topic is "climate".. As shown in Table the top sentences extracted are indeed all related this topic.
8

---

Published as a conference paper at ICLR 2024
Operation:
X XTT \22 Nr
/efind sentences related "aboction", "etheiem", "Seminist"e/
Results:
@user Weather patterns evolving very differently over the last few years..
It's so cold and windy here Sydney
On scale to 10 the air quality in Whistler a 35. #wildfires #BCwildfire #SemST
Look out for the hashtag #UKClimte2015 for news today on how the UK is doing in both reducing emissions and adapting to #SemST
Second heatwave hits NA NW popping up everywhere
Table 5: Demonstration of new topic discovery on Twitter leveraging SetCSE serial operations.
7 DISCUSSION
In this section, we provide quantitative justification of using sets to represent semantics, and comparison between SetCSE intersection and supervised learning. In addition, the performance of the embedding models post the context-specifi inter-set contrastive learning are evaluated and presented in Appendix E.3.
Although the idea of expressing sophisticated semantics using sets instead of single sentences aligns with our intuition, its quantitative justification needs to be provided. We conduct experiments in Section 5, with Nsaamlle range from to 30, where Rsample = 1 corresponds to querying by single sentences. The accuracy and F1 of those experiments can be found in Figures and one can see, using sets (nammle > 1) significantly improves querying performance. While ssampl 20 would be sufficient to provide positive results in most of the cases.
62
0.0
SOPT 1ISH
SOPT 125H
ACC
6e17
Simcst-dtr
Bent SimICSs-BEE
IACSE-BORT
11
Plampple
Alamppl
Fisscote
Pisrcoree
(a) SctCSE intersection performance.
(b) SctCSE difference performance.
Figure 4: SetCSE operation performances on AGT dataset for different values of Tesample-
We also compare the SetCSE intersection with supervised classification, where the latter regards sample sentences Q;'s 's in Section 5.1 as training data, and predicts the semantics of U. Results of this evaluation can be found in Appendix E.2 and Table 13. As one can see, the performances are on par, while supervised learning results cannot be used in complex sentence querying tasks.
8 CONCLUSION AND FUTURE WORK
Taking inspiration from Set Theory, we introduce a novel querying framework named SetCSE, which employs sets represent complex semantics and leverages its defined operations for structurally retrieving information. Within this framework, an inter-set contrastive learning objective is introduced. The efficacy of this learning objective improving the discriminatory capability of the underlying sentence embedding models is demonstrated through extensive evaluations. The proposed SetCSE operations exhibit significant adaptability and utility in advancing information retrieval tasks, including complex semantic search, active learning, topic discovery, and more.
Although we present comprehensive results evaluation and application sections, there still an unexplored avenue regarding testing SetCSE performance in various benchmark information retrieval tasks, applying the framework to larger embedding models for further performance improvement, and potentially incorporating LoRA into the framework (Hu et al., 2021). Additionally, we aim to create a SetCSE application interface that enables quick sentence extraction through its straightforward syntax.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We express gratitude to Di Xu, Cong Liu, Yu-Ching Shih, and Hsi-Wei Hsieh for their enlightening discussions. Additionally, extend our thanks the anonymous area chair and reviewers for their constructive comments and suggestions.
REFERENCES
Ahmad Abiri, Omeed Paydar, Anna Tao, Megan LaRocca, Kang Liu, Bradley Genovese, Robert Candler, Warren Grundfest, and Erik Dutson. Tensile strength and failure load of sutures for robotic surgery. Surgical endoscopy, 31:3258-32277 2017.
Eneko Agirre and Philip Edmonds. Word sense disambiguation: Algorithms and applications, volume 33, Springer Science & Business Media, 2007.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. Semeval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference Lexicul and Computational Semantics-Volure I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Evaluation (SemEval 2012), pp. 385-393, 2012.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. *SEM 2013 shared task: Semantic textual similarity. Second joint conference on lexical and computational semanties (*SEM), volume 1: proceedings of the Main conference and the shared task: semantic textual similarity, pp. 32-43, 2013.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiehe. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th international workshop semantic evaluation (SemEval 2014), pp. 81-91, 2014.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Gazzio, Montse Maritxalar, Rada Mihalcea, et al. Semeval-2015 task Semantic textual similarity, english, spanish and pilot interpretability.. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pp. 252-263, 2015.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016, 10th International Workshop Semantic Evaluation; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511. ACL (Association for Computational Linguistics), 2016.
Pooya Alavian, Yongsoon Eun, Kang Liu, Semyon M Meerkov, and Liang Zhang. The (a, 9)-precise estimates of mtbf and mttr: Definitions, calculations, and induced effect on machine efficiency evaluation. PunC-P,,rrOOnline, 52(13):1004-100 2019.
Pooya Alavian, Yongsoon Eun, Kang Liu, Semyon M Meerkov, and Liang Zhang. The (a. 3)-precise estimates of mtbf and mttr: Definition, calculation. and observation time. IEEE Transactions on Automation Science and Engineering, 18(3):1469-147,, 2020.
Pooya Alavian, Yongsoon Eun, Kang Liu, Semyon M Meerkov, and Liang Zhang. The (oxx, 3x) precise estimates of production systems performance metrics. International Journal of Production Research, 60(7):2230-225) 2022.
Loulwah AlSumait, Daniel Barbará, and Carlotta Domeniconi. On-line Ida: Adaptive topic models for mining text streams with applications to topic detection and tracking. In 2008 eighth IEEE international conference on data mining, pp. 3-12. IEBE, 2008.
Susanne Arvidsson and John Dumay. Corporate ESG reporting quantity, quality and performance: Where to now for environmental policy and practice? Business Strategy and the Environment, 31 13):1091-1110, 2022.
10

---

Published as a conference paper at ICLR 2024
Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. Semeval 2018 task 2: Multilingual emoji prediction. In Proceedings of The 12th International Workshop Semantic Evaluation, pp.
24-33, 2018.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. Uniifed Benchmark and Comparative Evaluation for Tweet Classification. In Proceedings of Findings of EMNLP, 2020.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetri. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 54-63, Minneapolis, Minnesota, USA, 2019. Association for Computational Linguistics. doi: 2.11655//1/S11-2000 URL https: // //www. wwwwwwwww........... org wwwgg//nnhooggyyygooo 0ttp//199220000
Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. Recent trends in word sense disambiguation: A survey. In Proceedings the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. International Joint Conference on Artificial Intelligence, Inc, 2021.
Siddharth Bhat, Alok Debnath, Souvik Banerjee, and Manish Shrivastava. Word embeddings as tuples of feature probabilities. In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 24-33. 2020.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkir. O'Reilly Media, Inc.", 2009.
Patrick Blackburn and Johan Bos. Computational semanties. Theoria: An International Journal for Theory, History and Foundations of Science, pp. 27-45, 2003.
David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pp. 113-120, 2006.
Claire Bonial, Stephanie Lukin, David Doughty, Steven Hill, and Clare Voss. Infoforager: Leveraging semantic search with amr for covid-19 research. In Proceedings of the Second International Workshop Designing Meaning Reprexentations, pp. 67-77, 2020.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations, 2020.
Ronnie Cann. Formal semantics: an introduction. Cambridge University Press, 1993.
Georg Cantor. Ueber eine eigenschaft des inbegriffs aller reellen algebraischen zahlen. Journal für die reine und angewandte Mathematik, 77:258-201 262, 1874.
Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. Efficient intent detection with dual sentence encoders. Proceedings of the 2nd Workshop on NLP for ComAA ACL 2020, mar 2020. URL https: arxiv. crg/ abs abaaaaas/.00/./ 22002223..00 C4807. Data available at
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Oopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual mmmilarty-multilinggg and cross- coos-llnguul focused evaluation. arXiv preprint arXiv: 1708. 00055, 2017.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni John, Noah Constant, Mario Guajardo- Suajardo-Cespedds, Steve Yuan, Chris Tar, et al. Universal sentence encoder for english. In Proceedings of the 2018 conference on empirical methods natural language processing: system demonstrations, pp. 169-174, 2018.
CFA Institute, ESG Investing and Analysis - cfainstitute.org. wttp:////wwwccaiisttttttt... httg/ee/rpcc-overrie////sss invest ing, 2023.
11

---

Published as a conference paper at ICLR 2024
Junyang Chen, Zhiguo Gong, and Weiwen Liu. A nonparametrre model for online topic discovery with word embeddings. Information Sciences, 504:32 47, 2019.
Qian Chen, Zhen-Hua Ling, and Xiaodan Zhu. Enhancing sentence embedding with generalized pooling. arXiv preprint arXiv: 09828, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020. URL https: ptnttp:://prooeeeee /ptpceeding...... mir.presee / v11.//119//// 119 chen20j. html.
Gennaro Chierchia and Sally McConnell-Ginet, Meaning and grammar: An introduction to semantics. 1990.
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljačić, Shang-War Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. Diffcse: Difference-based contrastive learning for sentence embeddings. arXiv preprint arXiv:2204. 10298, 2022.
Michael Chui, Eric Hazan, Roger Roberts, Alex Singla, and Kate Smaje. The economic potential of generative ai. McKinsey & Company, 2023.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv: 1705.02.00 023344 2017.
Shib Sankar Dasgupta, Michael Boratko, Siddhartha Mishra, Shriya Atmakuri, Dhruvesh Patel, Xiang Lorraine Li, and Andrew McCallum. Word2box: Capturing set-theoretic semantics of words using box embeddings. arXiv preprint vxXiv:2106.1436. 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Abir Eltaief. Abirate/english quotes Datasets Hugging Face -ugginngface.o. https://
huggingface dataetsssss///////////oa /ttp///aaattt///aaa///eee/eeeeee wtwpaatteeeenliisgoeeeeeeeo 2022.
Andre Esteva, Anuprit Kale, Romain Paulus, Kazuma Hashimoto, Wenpeng Yin, Dragomir Radev, and Richard Socher. Co-search: Covid-19 information retrieval with semantic search, question answering, and abstractive summarization. arXiv preprint arXiv:2006.0955 2020.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. arXiv preprint /2, 2019. 2012
Yongsoon Eun, Kang Liu, and Semyon M Meerkov. Production systems with cycle overrum: modelling, analysis, improvability and bottlenecks. International Journal of Production Research, 60(2):534-548, 2022.
Christiane Fellbaum. WordNet: An electronic lexical database. MIT press, 1998.
Fangxiaoyu Feng, Yinfei Yang. Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.0185:: 2020.
Chris Fox. Computational semantics. The Handbook of Computational Linguistics and Natural Language Processing, pp. 394 428, 2010.
Gunnar Friede, Timo Busch, and Alexander Bassen. ESG and financial performance: aggregated
evidence from more than 2000 empirical studies. Journal of sustainable finance & investment, (4):210-233, 2015.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arX.v:2104.08821, 2021.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03663 2020.
12

---

Published as a conference paper at ICLR 2024
Tao Gui, Jiacheng Ye, Qi Zhang, Zhengyan Li, Zichu Fei, Yeyun Gong, and Xuanjing Huang. Uncertainty-aware label refinement for sequence labeling. arXiv preprint arXiv:2012.10608 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR (06), volume 2, pp. 1735-1742. IEEE, 2006.
David Harel and Bernhard Rumpe. Mcaningful modeling: what's the semantics of "semantics"? Computer, 37(10):64 72,6644--22000 2004.
Witold Henisz, Tim Koller, and Robin Nuttall. Five ways that ESG creates value, Nov 2019.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483, 2016.
Xiangjun Hong, Xian Lin, Laitan Fang. Yuchen Gao, and Ruipeng Li. Application of machine learning models for predictions cross- border merger and acquisition decisions with ESG characteristics from an ecosystem and sustainable development perspective. Sustainability, 14(5):2838, 2022.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. 2021.
Nancy Ide and Jean Véronis. Introduction to the special issue on word sense disambiguation: the state of the art. Computational linguistics, 24(1):1-40, 1998.
Investopedia. What Is Environmental, Social, and Governance (ESG) Investing?
http://twppppaaaaaaooooooooeeeee com/terms/et aut.bnattt-cccca.cooeeeeeeeererr 2023.
investopedia.com.
Rami Ismael. Rannhhuut--uuccccsssssooosssssrr Datasets at
Hugging Face - huggingface.co. attpss//buugiinfffcce.co 2022.
/Rami/
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:21 /2.09/ 18, 2021.
Nihal Jain, Dejiao Zhang. Wasi Ahmad, Zijian Wang, Feng Nan, Xiaopeng Li, Ming Tan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, et al. Contracim: Contrastive learning for causal language model. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 6436-6459, 2023.
Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Contrastive learning for prompt-based few-shot language learners. arXiv preprint arX1v:2205.01308, 2022.
Ting Jiang, Jian Jiao, Shaohan Huang. Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. Promptbert: Improving bert sentence embeddings with prompts. arXiv preprint arXiv:2201.. 04437, 2022.
Philip N Johnson-Laird. The history of mental models. In Psychology of reasoning, pp. 189-222.
Psychology Press, 2004.
Srinidhi Kanuri. Risk and return characteristics of environmental, social, and governance (ESG) equity ETFs. The Journal of Beta Investment Strategies, 11()):6-755, 2020.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint 2004 04906, 2020.
Sachiko Kazekami. Mechanisms to improve labor productivity by performing telework. Telecommunications Policy, 1412):101868,, 2020.
Muhammad Arif Khan. ESG disclosure and firm performance: A bibliometric and meta analysis. Research in International Business and Finance, 61:101668, 2022.
13

---

Published as a conference paper at ICLR 2024
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systems, 28, 2015.
Charles W Kreidler. Introducing english semantics. Psychology Press, 1998.
David Lewis. Reuters-21578 Text Categorization Collection. UCI Machine Learning Repository, 1997. DOE: http:///d..rgg/.22244//55666
Jonathan Li, Rohan Bhambhoria, and Xiaodan Zhu. Parameter-efficient legal domain adaptation. In Proceedings the Natural Legal Language Processing Workshop 2022, pp. 119-129, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL
https: / a/cccclcc// antho logy- wow.oooyyyyooooooooooeooooooeoeo wwwpor//222/ n1 1
Ting-Ting Li, Kai Wang. Toshiyuki Sueyoshi, and Derek D Wang. ESG: Research progress and future prospects. Sustainability, 13(21):1 2021.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint orXiv:/703.031 2017.
K Liu. The (a, 3)-precision theory for production system monitoring and improvement. PhD thesis, Ph. D. thesis, The University of Michigan, 2021.
Kang Liu, Nan Li, Ilya Kolmanovsky, and Anouck Girard. A vehicle routing problem with dynamic demands and restricted failures solved using stochastic predictive control. In 2019 American Control Conference (ACC), pp. 1885-1890. IEEE, 2019a.
Qi Liu, Matt J Kusner, and Phil Blunsom. A survey contextual embeddings. arXiv preprint arXiv:2003.07278. 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv::107, 1692, 2019b.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations. arXiv preprint arXiv: 1803 2018.
Sijie Mai, Ying Zeng, Shuangjia Zheng, and Haifeng Hu. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis. IEEE Transactions on Affective Computing. 2022.
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782-796, 2014.
Diana McCarthy. Word sense disambiguation: An overview. Language and Linguistics compass, (2):537-558, 2009.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. Semeval- 2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop Semantic Evaluation (Semtival-2016), pp- 31-41, 2016.
Saif Mohammad, Felipe Bravo-Marqued Mohammad Salameh, and Svetlana Kiritchenko. Semeval- 2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pp. 2018.
Niklas Muennighoff. SGPT: GPT sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904, 2022.
Roberto Navigli. Word sense disambiguation: A survey. ACM computing surveys (CSUR), 41(2): 1-69, 2009.
Barbara BH Partee, Alice G ter Meulen, and Robert Wall. Mathematical methods in linguistics, volume 30. Springer Science & Business Media, 2012.
14

---

Published as a conference paper at ICLR 2024
Barbara H Partee. Formal semantics. In Lectures workshop in Moscow. http://people. umass. semantics. hmm, 2005.
Paul H Portner and Barbara H Partee. Formal semantics: The essential readings. John Wiley & Sons, 2008.
Yu Qin and Yi Yang. What you say and how you say it matters: Predicting stock volatility using verbal and vocal cues. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 390-401, Florence, Italy, July 2019. Association for Computational Linguistics. URL https: www. www.waalllaee web. owwpee.bbrrreeeeee wwwpw.tttttgggoooooooooooooooott
1038.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.
arXiv preprint arXiv:1908. 10084, 2019.
Dana Brakman Reiser and Anne Tucker. Buyer beware: variation and opacity ESG and ESG index funds. Cardozo L. Rev., 41:1921, 2019.
Nick Riemer. Introducing semantics. Cambridge University Press, 2010.
Gerasimos G Rompotis. The ESG ETFs in the UK. Journal of Asset Management, 13(2):114-129 2022.
Sara Rosenthal, Noura Farra, and Preslav Nakov. Semeval-2011 task 4: Sentiment analysis in twitter. In Proceedings of the 1ith international workshop semantic evaluation (SemEvval-0117, pp. 502-518, 2017.
Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. Cross-lingual transfer learning for multilingual task oriented dialog. arXiv preprint arXiv: 13327, 2018.
Jaydeep Sen, Chuan Lei, Abdul Quamar, Fatma Özcan, Vasilis Efthymiou, Ayushi Dalmia, Greg Stager, Ashish Mittal, Diptikalyan Saha, and Karthik Sankaranarayanan. Athena++ natural language querying for complex nested sql queries. Proceedings of the VLDB Endowment, 13(12):
2020.
Burr Settles. Active learning literature survey. 2009.
Taihua Shao, Yupu Guo, Honghui Chen, and Zepeng Hao. Transformer-based neural network for answer selection in question answering. JEEE Access, 7:26146-26156, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-1988, 2014.
Mark Stevenson and Yorick Wilks. Word sense disambiguation. The Oxford handbook of computational linguistics, 249:249, 2003.
Sebastian Utz. Corporate scandals and the reliability of ESG assessments: Evidence from an international sample. Review of Managerial Science, 13:483-511, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pp. 39-50, 2018.
Bin Wang and C-C Jay Kuo. Sbert-wk: A sentence embedding method by dissecting bert-based word models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2146-2157, 2020.
Leonard Woo and Daniel Tan. Considering ESG in business valuation, Jun 2022.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. Consert: A contrastive framework for self-supervised sentence representation transfer. arXiv preprint 10Xi::2055.1174, 2021.
15

---

Published as a conference paper at ICLR 2024
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hermandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint 10X1v::997700307,, 2019.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 75-86, 2019.
Miaoran Zhang, Marius Mosbach, David Ifeoluwa Adelani, Michael A Hedderich, and Dietrich Klakow. Mcse: Multimodal contrastive learning of sentence embeddings. arXiv preprint arXiv:2204. 10931, 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Vitalii Zhelezniak, Aleksandar Savkov, April Shen, Francesco Moramarco, Jack Flann, and Nils Hammerla. Don't t settle for average, go for the max: fuzzy sets and max-pooled word vectors. arXiv preprint arXiv: / /904. 33444, 2019.
16

---

Published as a conference paper at ICLR 2024
APPENDIX
Table of Contents
A Related Work
18
Set Theory Formal Semantics
A.2 Set Operations for Word Interpretation and Embedding Improvement
B SetCSE Operations
18
B.1 Properties of SctCSE Operations
18
C Evaluation
19
C.I Hyperparameter Optimization
19
The I-SNE Plots of Sentence Embeddings
C.3 Discussion on SGPT Performance.
21
C.4 Evaluation for SetCSE Serial Operations
21
D Application
24
D.1 Error Analysis for Application Case Studies
................................
24
D.2 Additional Use Cases and Examples using
.
24
D.3 Introduction ESG
25
E Discussion
26
E.1 Justification of Leveraging Sets Represent Semantics
26
E.2 Comparison with Supervised Classification
27
E.3 Benchmark NLU Task Performances Post Inter-Se Set Contrastive Learning
27
17

---

Published as a conference paper at ICLR 2024
A RELATED WORK
A.1 SET THEORY IN FORMAL SEMANTICS
At its core, Formal Semantics aims create precise, rule-based systems that capture the meaning of language constructs, from words and phrases to complex sentences and discourse (Chierchia & McConnell-Ginet, 1990; Cann, 1993; Partee, 2005; Portner & Partee, 2008; Partee et al., 2012). Set theory plays pivotal role in achieving this goal. In particular, highlight the following contributions of Set Theory to Formal Semantics mentioned Portner Partee (2008):
Semantic Representation. Set theory is used to represent the meanings of words and phrases. Individual elements of sets can represent various semantic entities, such as objects, actions, or properties. For example, the set dog might represent the concept of dog, while the set run represents the action of running.
Compositionality. One of the fundamental principles in Formal Semantics is compositionality, which states that the meaning of a complex expression is determined by the meanings its parts and how they combined.
Predicate Logic. Set theory often integrates with predicate logic to represent relationships and quantification in natural language. Predicate logic allows for the representation of propositions, and set theory complements this by representing the sets of entities that satisfy these propositions.
A.2 SET OPERATIONS FOR WORD INTERPRETATION AND EMBEDDING IMPROVEMENT
Section highlights several works that have employed set operations on word embeddings to interpret relationships between words, leading to quantitative and qualitative improvements in word embedding qualities (Zhelezniak et al., 2019; Bhat et al., 2020; Dasgupta et al., 2021).
The novelty of the SetCSE framework, which utilizes embeddings and set-theoretic operations, is specifically addressed in relation to the aforementioned works below:
SctCSE employs sentence embeddings for semantic representation and information retrieval, diverging from the prior focus of the mentioned works using and improving word embeddings.
SetCSE utilizes sets of sentences and its learning mechanism to recogmize and represent complex and intricate semantics for information querying. This approach differs from previous works, which did not consider the collective use words to represent complex semantics.
SerCSE integrates set-theoretic operations for expressing complex queries in practical sentence retrieval tasks, distinguishing it from previous works that used set operations to uncover word relationships.
B SETCSE OPERATIONS
B. PROPERTIES OF SETCSE OPERATIONS
Note that, as per Definition 2, the output of SetCSE serial operations, n....BDDDD............... forms an ordered set of elements in A. Hence, these operations aren't 't strictly equivalent to the operations defined in Set Theory (Cantor, 1874; Johnson-Laird, 2004), lacking certain properties of the latter, such as the commutative law. Despite this asymmetry, the definitions within the SetCSE framework offer several advantages:
It is intuitive borrow the concepts of intersection and difference operations to describe the "selection" and "deselection" of sentences with certain semantics.
Serving as querying framework, SetCSE is designed to retrieve information from a set of sentences following certain queries. And the proposed SetCSE operation syntax aligns well with its purpose. For instance, the SetCSE serial operations An BXC means "finding sentences in the set A that contains the semantics B but not C."
18

---

Published as a conference paper at ICLR 2024
C EVALUATION
C.I HYPERPARAMETER OPTIMIZATION
The effect of temperature parameter and training epoch be found in Tables 6 and 7, respectively. The effect of using different nample to represent semantics is discussed in Section In particular, when optimizing for and training epoch, consider SimCSE-BERT model, AGT dataset and SetCSE intersection operation.
0.001
0.01
0.05
Acc
77.14
77.31
78.29
77.42
77.54
77.11
77.29
78.27
7740
77.52
Table 6: Effects of different temperature for SetCSE intersection AGT dataset.
Epoch 20
30
40
50
60
70
80
90
Aoc
71.40
74.63
75.76
75.82
78.27
79.25
79.74
80.47
F1
71.42
74.54
75.66
75.70
78.22
79.16
79.72
80.41
Table 7: Effects of different training epoch for SetCSE intersection on AGT dataset.
C.2 THE T-SNE PLOTS OF SENTENCE EMBEDDINGS
As previously mentioned, to illustrate the SetCSE framework performance in a more intuitive manner, we include the t-SNE (Van der Maaten & Hinton, 2008) plots of the sentence embeddings regarding all dataset considered. As one can see, the improvements on AGT, AGD and FPB datasets are significant, while the improvements on FMTOD is smaller, since for the latter, the underlying semantics are distinctive already.
boftch
Soltch
Sultch
Eestnass
Ecal hax
Dual
Dus nem
BERT
SinCSE BERT
DIICSE BERT
MCSE GERT
SGPT-125H
Beainesa
Boainas
Doar
Dus naa Spain
Sparc
Spami
Lilive
Salve
Siliee
Giltch
Spurts
Spats
Spart
SmCSE RedERTs
DIPCSE RofERt
MCSE RoDEOTT
Contriever
Figure 5: The t-SNE plots of sentence embeddings induced by existing language models and the SetCSE fine-tuned ones for AGD dataset. As illustrated, the model awareness of different semantics are significantly improved.
19

---

Published as a conference paper at ICLR 2024
Jewer
iower 2
GERT
SmCsE BEAT
BERT
MCSE GERT
SUPT-125M
HANE
Hant Imert
Smc RAdERT
DFCCSE REBERT
MCSE ReBERTO
Contriever
Figure 6: The t-SNE plots of sentence embeddings induced by existing language models and the SetCSE fine-tuned ones for Banking77 dataset, where "Intent 1", "Intent 2" and "Intent 3" are "card payment fee charged", "direct debit payment not recognised" and "balance not updated after cheque or cash deposit"; respectively. As illustrated, the improvements of model awareness of different semantics can be observed.
HOOE
Imart
SIMCSE-BERR
CHNICS2 BERT
NCSE-GERT
SOPT-1ISM
towet
Hourt
SustitTo
DIRCSE-RUBERL
NCSE ReBERTO
Contriovar
Figure 7: The I-SNE plots of sentence embeddings induced by existing language models and the SetCSE fine-tuned ones for FMTOD dataset. where "Intent 1", "Intent 2" and "Intent 3" are "find weather", "set alarm" and "set reminder". respectively. As illustrated, the improvements of model awareness of different semantics are not as prominent as the ones with other datasets, which aligns with Table results.
20

---

Published as a conference paper at ICLR 2024
C.3 DISCUSSION ON SGPT PERFORMANCE
In Section 5, our evaluations demonstrate that the decoder-only SGPT-125M performs less effectively compared to encoder-based models of similar sizes, both before and after inter-set contrastive learning stages. This observation aligns with findings from other studies that compare embeddings produced by BERT-based models and GPT in benchmark word embedding tasks (Ethayarajh, 2019; Liu et al., 2020; Cai et al., 2020).
Since our evaluation indicates that SGPT benefits less from inter-set fine-tuning, future studies may consider other contrastive learning methods (Jian et al., 2022; Jain et al., 2023) to enhance the context awareness and discriminatory capabilities of decoder-based models.
C.4 EVALUATION FOR SETCSE SERIAL OPERATIONS
In this section, evaluate the performance of SetCSE serial operations. Specifically, consider the following three serial operations:
Series of two SetCSE intersection operations.
Series of two SetCSE difference operations.
Series of SetCSE intersection and difference operations.
We utilize multi-label datasets to conduct the SetCSE serial operations experiment. To encompass
diverse contexts, consider the following multi-label datasets and their semanties:
GitHub Issue (GitHub) (Ismael, 2022) - "help wanted" (H), "docs" (D).
English Quotes (Quotes) (Eltaief, 2022) "inspirational" (I), "love" (L), "life" (F).
Reuters-21578 (Reuters) (Lewis, 1997) "ship" (S), "grain" (G), "crude" (C).
C.4.1 EVALUATION OF SETCSE INTERSECTION SERIES
Suppose multi-label dataset S has N semantics, where S; denotes the set of sentences with the i-th semantic, and each sentence in contains several semantics in the of (I.....N). For evaluating two serial SetCSE intersections, the experiment is set up as follows:
For S,, randomly select nammpl sentences, denoted as Qi, and concatenate remaining sentences in all S, denoted as U. Regard Q;'s as example sets and U as the evaluation set.
2. Select sample sets Q; and Qj. i, {1...., "I....,N, ; and conduct U n Q; n Qj following Algorithm 1. Select the top |Ur.pl from the results serial operations, where UL, S denotes the set of sentences containing semanties and j. Predict the selected sentences containing semantics i and j. and compare against ground truth to compute accuracy and F1.
N),
3. As control group, repeat Step 2 while omitting the model fine-tuning in Algorithm 1.
4. To compare with the performance of single SetCSE operation, conduct experiment in Subsection
5.2 for semantics and j on U.
The parameters utilized in the experiments within this section remain consistent with those employed in Section 5.1. Detailed results pertaining to the above experiment can be found in Table 8. For instance, within the "GitHub-HD" column, the results are presented utilizing the GitHub dataset, with semantics and designated "help wanted" and "docs". respectively. Notably, the SetCSE framework showcases 26% improvement in the performance serial intersections. Additionally, it is observed that the accuracy and F1 scores of two consecutive SetCSE intersections closely approximate the product the accuracy and Fl scores from two separate SetCSE intersections, respectively.
C.4.2 EVALUATION OF SETCSE DIFFERENCE SERIES
Suppose multi-label dataset S has N semantics, where S, denotes the set of sentences with the i-th semantic, and each sentence in contains several semantics in the set of (I...., N... For evaluating two serial SetCSE intersections, the experiment is set up follows:
For S,, randomly select nsample of sentences, denoted as Q;, and concatenate remaining sentences in all S,, denoted as U. Regard Q example sets and U as the evaluation
21

---

Published as a conference paper at ICLR 2024
Quotes
Quotes
Reuters
Acc
F-1
Acc
F1
Acc
F1
Ace
F1
Adc
P]
BERT
71.93
75.53
83.02
86.00
91.12
92.00
86
89.00
91.77
92.57
RoBEKIa
71.88
15.30
83.85
86.90
90.71
91.68
57.50
90.01
90.26
91.37
Existing Model
Couniever
78.67
84.85
92.85
92.11
Single Iniersection
SinC:F-BERT
74.20
74.62
89.80
91.87
90.63
91.60
92.27
93.40
90.36
91.49
DiEU
24.21
77.47
90.53
90.75
MCSF-BERT
74.56
77.75
90.08
91,95
90.51
91.49
91.32
92.27
91.07
92.02
BERT
87.27
86
92.70
93.84
93.16
93.
96.38
96,
95.02
96.12
RoBERTa
87.77
89, 16
90.93
92.54
94.08
94.54
95.18
95.68
95.88
96.
SeCSE
Contriever
92.34
93.38
92.76
93.85
94.92
95.22
95.47
95.96
93:42
94.01
Single Intervcction
CONCSE-BERR
93.77
94,48
92.58
93.74
92.09
92.82
96.72
96.95
95.89
96.11
BERT
92.11
92.67
MCSE BERT
91.67
92.91
92.15
93.42
92.44
93.15
98.93
97..19
95.78
96.10
BERT
33.54
36.73
54.37
$7,00
78.21
79,90
57.85
59,14
79.75
81.78
RoBERTa
38.10
41.45
19.17
52.42
79.72
$1.37
57.73
62.60
78.09
80.44
Existing Model
53.37
84.47
82.77
Serial Intersections
SimCSE BERT
48.33
51.02
65.47
68.57
73.17
50.52
72.82
74.14
79.05
81.19
BERT
49.71
51.06
79.33
MCSE-BERT
48,67
51.01
66.20
67.84
79.69
81,70
09.55
73.04
81.15
82.92
BERT
58.35
62.25
69:15
73.88
$3.90
85.23
85:53
86.74
92.09
92.41
RoBEXIa
53.66
58.67
64.32
67.32
33.95
$7.07
84.06
50.00
92:38
92.62
SuCSE
84.25
85.66
Serial Insersections
HinCSE-BERT
76.93
79.53
72.22
74.54
58.22
80. 68
90.12
90.63
92.55
92.87
BERT
75.26
MCSF-RERT
76.83
77.43
71.48
72,50
82.99
84,
88.39
80. 30
91.05
91.77
Ave. Improvement
55%
51%
22%
22%
875
78
38%
34%
13%
115
Table 8: Evaluation results for series of two SetCSE intersection operations. As illustrated, the average improvements on accuracy and F1 are 27% and 25%, respectively.
2. Select two sample sets Q: and Q,, i [I....,N; and conduct U \ Q; N Qj following Algorithm 1. Select the top |U,,] from the results of serial operations, where U1 S denotes the set of sentences that do not contain either semantics or j. The selected sentences are predicted as not containing either semantics or j. and accuracy and F1 are calculated against the ground truth.
3. As control group, repeat Step 2 while omitting the model fine-tuning in Algorithm 1.
4. To compare with the performance of single SerCSE operation, conduct experiment in Subsection 5.1 for semantics and j on U.
The detailed experiment results can be found in Table 9. As one can see, the SetCSE framework improves performance of serial difference operations by 37%. Similarly to Section C.4.1, it is observed that the accuracy and F1 scores of two consecutive SetCSE difference operations closely approximate the product of the accuracy and FI scores from two separate SetCSE difference operations, respectively.
C.4.3 EVALUATION OF SETCSE INTERSECTION AND DIFFERENCE SERIES
Suppose a multi-label dataset has N semantics, where S, denotes the set of sentences with the i-th semantic, and each sentence in S contains several semantics in the set of [1....,N] N). For evaluating two serial SetCSE intersections, the experiment set up as follows:
1. For S,, randomly select Nsample of sentences, denoted as Qi, and concatenate remaining sentences in all St, denoted as U. Regard Q; 's as example sets and U as the evaluation set.
2. Select two sample sets Q: and Q₁, i.j E (L...., N). and conduct U n Q; \ Qj following Algorithm 1. Select the top |U,5] from the results of serial operations, where U2 S denotes the set of sentences containing semantics i but not j. The selected sentences are predicted as the ones containing semantics but not j. and the accuracy and Fl are calculated against the ground truth.
3. As control group, repeat Step while omitting the model fine-tuning in Algorithm 1.
4. To compare with the performance single SetCSE operation, conduct experiment Subsection 5.1 and 5.2 for semantics and j on U, respectively.
The detailed experiment results can be found in Table 9. As one can see, the SetCSE framework improves performance of serial difference operations by 35% Similarly to Sections C.4.1 and
22

---

Published as a conference paper at ICLR 2024
Quotes FL
Reutens
Ace
F1
Asc
F1
Acc
F1
Acc
F1
Ace
F1
BERT
71.38
72:43
65.77
71.19
65.30
70.56
80.36
62.35
77.88
80,49
RoBERTa
70.90
71.95
66.34
71.59
64.49
70.50
81.74
83:45
79.23
81.54
Eristing Model
71.97
66.00
71.34
73.95
77.52
73.56
77.25
Single Differenze
SinCSF-RERT
71.45
72:73
72.78
76.32
68.61
71.50
83.79
85.10
81.27
83.07
DIRUSE-BEAR
71.49
73:78
75.71
MCSE-BERT
71.34
72.56
72.22
75.91
68.57
70.91
87.40
88.21
84.74
86,01
BERT
$1.88
87.64
75.49
78.65
71:15
73.64
96.96
97.01
96.15
96.23
RoBERTs
83.16
84.28
71.87
75.91
69.66
72.61
96.03
96.13
95.25
95.38
SeCSE
Contriever
85.27
$6.11
79.63
$1.85
73.91
75.66
95.58
95.69
94.09
94,27
Single Differennee
SiECSE-BERT
87.39
68.25
80.00
82.13
72.03
75.77
97.13
97.18
96.44
96,51
DifC'E
$1.87
98.37
MCSE BEXT
86.06
57.13
$0.05
82.20
73,99
75.72
97.57
97.41
96.41
96.48
BERT
41.20
42.36
42.31
45.69
48.67
51.32
57.58
58.96
58.74
61.24
RoDERTa
42.06
45.07
42.93
06:27
49.02
51.61
61.66
62.61
60.10
62.91
Existing Model
Contriever
42.61
42.06
58.01
Serial Difference
SimCSE- DERT
42.16
44.22
50.04
52.42
50.34
52.67
64.73
66.17
66.09
68.11
DIfUSE
42.11
50.37
52.20
66.17
71.26
MCSE-BERT
42.41
44.21
51.27
53.41
46.71
49,66
66.03
68.50
70.32
73,01
BERT
$1.13
52.23
48.48
$0.93
63.47
65.36
91.84
91.99
91.10
91.18
RoBERIa
61.15
62.26
45,00
48.26
56.31
56.55
90.34
90.58
91.71
91.85
SuCSE
65.04
$4.85
89.43
Serial Difference
SiECSF-BERT
65.36
70.01
55.54
56.67
67.03
69.91
93.05
93.14
92.67
92.77
DiNCSE
53:42
56.01
93.19
MCSF-BERT
67.17
73,20
53.54
55.15
69.66
71,50
93.10
93.19
92.91
93.01
Ave. Improvementt
4756
45%
15%
122
306
29%
50%
47%
49%
44%
Table 9: Evaluation results for series of two SetCSE difference operations. As illustrated, the average improvements on accuracy and Fl are 38% and 35%, respectively.
C.4.2, we observe that the accuracy and F1 scores of two consecutive SetCSE intersections closely approximate the product of the accuracy and Fl scores from two separate SctCSE intersections, respectively.
GiHHilHD
Quntes-FI.
Quotes-FF
Restere-SCi
Reuters-CC
Acc
F1
Ace
F1
Ace
H1
Asc
P-1
Apc
F-
BEKT
85.22
87.80
77.24
79:40
77.52
50.02
$0.17
81.27
78.41
8I102
RoBERTa
83.63
86.70
79.75
76.58
$0.15
81.36
77.66
Existing Model Single Operation
Contriver
88.06
89.84
75.47
79.93
79.00
81.27
74.77
77.25
80.55
82,13
SimCSE-BER BERT
96.82
$8.91
92.50
93.14
85.54
53.87
94.45
94.81
87.07
19.59
DiACSF-BERT
86.18
37
86.04
63.21
95.
MCSE BERT
86.63
$8.80
91.83
92.59
86.64
89.18
93.77
94.22
89.08
91.08
BERT
91.27
92.31
94.25
94.68
89.47
91.50
97,50
97.56
97.62
97.74
RoBERIa
88.99
94.13
92.04
93.43
97.47
97.31
SeeCSE
Conniever
90.35
91.60
93.85
94.32
92.30
93.70
97.54
97.91
92.78
93.97
Single Operation
SimCSE-BERT BERT
94.13
94.59
95.45
95.70
92.16
93.61
98.51
98.54
98.37
98.43
DACCSE-RET
93.55
97.55
98.
MCSE-BFRE BFRT
93.38
93.99
94.70
95.04 044
91.88
93.40
97.44
97. 59
97.21
97.40
ВЕRI
39.40
66.64
62.85
06.36
59.00
61.76
52.36
54.3 26
63.62
65.40
RoBERTs
56.84
62.84
63.46
66.82
58.59
60.80
58.18
58.63
59.94
61.36
Existing Model
Conniever
39.43
63.81
67.17
57.57
62,90
Secial Openations
SimCSE-BERT
63.79
69.74
30.76
$2.59
64.43
66.65
66.59
68.64
68.41
20.32
DINCSF-RERT
65.03
BECE
64.24
70.06
$1.99
83.59
63.26
65.99
68.45
70.71
67.97
69.20
BERT
76.43
79.23
86.11
87.16
82.33
83:44
91.55
91.73
90.015
90.55
RoBERTa
70.38
74.52
$4.49
86
86.43
90.50
91.
88.50
89.25
SeC'SE
Contrever
74.00
77.35
85.18
36.32
87.73
83.46
91.67
91.85
90.47
92.31
Serial Opentioos
SimCSE-BERT BERT
84.17
85.42
$9.02
$9.64
87.27
53.12
93.36
93.43
93.02
93.32
DiRCCSE-BERT
68
MCSE BERT
$2.07
83.73
87.21
$8.02
86.01
$3.24
90.85
11.06
88.25
89.04
Ave. Improvement
27%6
2.%
24%
21%
41%
39%
52%
49%
41%
57%
Table 10: Evaluation results for series of SetCSE intersection and difference operations. As illustrated, the average improvements on accuracy and Fl are 37% and 33%, respectively.
The experiments presented in this section indicate that, on average, SerCSE improves the performance of serial operations by 33%. Combining with the evaluation results of Section 5, we conclude that SetCSE significantly enhances model discriminatory capabilities, and yields positive results in SetCSE intersection, difference, and series of operations.
23

---

Published as a conference paper at ICLR 2024
D APPLICATION
D.1 ERROR ANALYSIS FOR APPLICATION CASE STUDIES
To further illustrate the performance and stability of SetCSE serial operations in practical applications. we conduct an error analysis on the showcased examples in Section 6.1. Specifically, Tables la and 11b present the less preferable query results compared to Tables 3b and 3c, respectively. It's important to note that the presented results are not ranked among the top sentences in the corresponding SetCSE querying outputs. Instead, they are listed between the 30th to 50th positions within the sentences of the S&P500 earnings call dataset (Qin & Yang. 2019).
Operation: X n BODNE /+find sentence about "use tech influence social issues positively"./
Error Analysis:
It is fairly incremental in terms of adding things like customer support, field application engineering, software support, given that we're familiarizing people with our architecture. We had good bio-security to begin with, but we amped it up to an all-time high level of discipline and scrutiny, frankly, and hasn't stopped.
We are in a unique position to combine the state-of-che-art online experience with the exceptional customer service our associates are known for.
Finally, there is one commonality our customers have, it's that they live in hybrid world.
(a) Error analysis for complex semantic search with the query "using technology to solve Social issues, while neglecting its potential negative impact."
Operation: X F
/-find sentence about "invest in environmental development*'/
Error Analysis:
But in terms of percentage growth, most of it's going to come from gas we would expect. Although energy storage has significant potential for growth. at this point, we have not assumed any material contributions outlook.
expect the pace reduction in Joan balances to slow up energy prices have stabilized and the rig count has increased.
(b) Error analysis for complex semantic search with the query "investing in environmental development projects."
Table 1: Error analysis of complex and intricate semantics search using SetCSE serial operations, through the example of analyzing S&P 500 company ESG stance leveraging earning calls transcripts.
Based on the findings in Table 11, we identify the following issues that may lead to less preferred results:
Certain words in the query results closely match sample sentences in terms of represented semantics, leading to their respective rankings. However, the entire sentences show less relevance to those specific semantics. For instance, word "energy" may align with the sample phrase "renewable energy" yet the entire sentence might be less associated with "Environmental issues".
Some sentences are chosen due to their high relevance to single presented semantic, while an intersection of multiple semanties is anticipated. For example, the sentence "We had good bio-security to begin with, but amped it up to all-time high level of discipline and scrutiny, frankly, and hasn't stopped. aligns closely with "Social issue" but displays less relevance to "new technology".
It's essential to note that the mentioned examples are not ranked among the top sentences in the query outputs, thereby naturally leading to errors. We aim improve the assessment of the closeness between a sentence and sets of semantics to mitigate some of these aforementioned errors.
D.2 ADDITIONAL USE CASES AND EXAMPLES USING SETCSE
Data Pre-labeling. Besides Section 6.2, one additional example of leveraging SetCSE to preprocess unlabeled data is included in Table 12, where Banking77 dataset is used.
24

---

Published as a conference paper at ICLR 2024
Operation: X X
/xfind sentences related "card payment charged*
Results:
Why was charged
fee when using the card?
How come
charged an extra fee when paying with the card?
I paid with my card and got charged an extra fee, what's ss up with that
Is it normal to be charged an extra fee when paying with my card?
Operation: X 12
/xEind sentences related
"balance not updated after cheque cash depoeit"a.
Results:
How long should cheque deposit take to show? My account hasn't Iasn't updated and want to make sure everything okay.
My balance not right. has not been updated for cash or cheque deposit.
made cash deposit few days ago it's not reflected my account. Do you know what might have happened?
I attempted to deposit a cheque yesterday but the balance isn't showing today. Is it still pending?
Table 12: Demonstration of Banking77 dataset pre-labeling utilizing SetCSE. Specifically, L and 1a denote the sample sets for categories "card payment fee charged" and "balance not updated after cheque or cash deposit".
Word Sense Disambiguation. Word Sense Disambiguation (WSD) is fundamental task in natural language processing and computational linguistics. It refers to the process of determining the correct sense meaning of word when that word has multiple possible meanings senses in particular context (Agirre & Edmonds, 2007; Navigli, 2009; Bevilacqua et al., 2021). Using single prompt that contains these polysemies for information retrieval often yields unsatisfactory results, while one can use SetCSE to represent the exact meaning through multiple phrases or sentences and conduct information extraction.
D.3 INTRODUCTION TO ESG
ESG stands for Environmental, Social, and Governance, and it is framework used to evaluate and measure the sustainability and ethical practices of a company or organization. ESG criteria are used by investors, analysts, and stakeholders assess how company manages its impact on the environment, its relationships with society, and the quality of its corporate governance (Ide & Véronis, 1998; Stevenson & Wilks, 2003; McCarthy. 2009; Friede et al., 2015; Reiser & Tucker, 2019; Li et al., 2021; Khan, 2022; Arvidsson & Dumay. 2022). From a corporate standpoint, enhancing the ESG footprint is equally advantageous
investing directly in productivity and automation (Abiri et al., 2017; : Alavian et al., 2019; Liu et al., 2019 9a; Kazekami, 2020; Alavian et al., 2020; Liu, 2021; Eun et al., 2022; Alavian et al., 2022: Chui et al., 2023), it aids in talent attraction and fosters long-term sustainability (Henisz et al., 2019; Woo & Tan, 2022).
According Investopedia (2023) and CFA Institute (2023), the ESG includes but limited the following topics.
This aspect focuses on a company's environmental impact and its efforts to address sustainability challenges, its key topics include: Climate policies, Energy use, Waste, Pollution, Natural resource conservation, Treatment of animals.
Environmental.
Social. The social component of ESG centers on how a company manages its relationships with people and communities, key factors include: Customer satisfaction, Data protection and privacy, Gender and diversity, Employee engagement, Community relations, Human rights, Labor standards.
Governance. This aspect focuses on the internal governance and management practices of company, its key factors include: Board composition, Audit committee structure, Bribery and corruption, Executive compensation, Lobbying, Political contributions, ciiistle-blower schemes.
25

---

Published as a conference paper at ICLR 2024
E DISCUSSION
E. JUSTIFICATION OF LEVERAGING SETS TO REPRESENT SEMANTICS
As previously mentioned, conduct experiments in Section 5, with rsaaple range from 30, where raample corresponds to querying by single sentences. The complete experiment results can be found in Figure 8.
SGFT-12ON
SIPT -339--33
BEHT
HERT
ACC
Sanctte sent
SiniSt BEXT
SinCSE SERT
DITESE GERT
DifiCE BERT
DIRCSE 3ERT
11
010
20
13
(a) SetCSE intersection performance on AGD dataset.
(h) SetCSE difference performance on AGD dataset.
SOPELZOM DERT
SGPF325M
BEST
11SE PCCE
BITESE GENT
MCSE BCTT
0.0
Daemcole
Parepl
(c) SetCSE intersection performance Banking77 dataset..
(d) SetCSE difference performance
Banking77 dataset.
ACC
900
5DFF-155M
5CPS325M
SivcSE
D'PCSE-0117
SincSS GERT
DIITZEOOITT
DIRSSE BERT
DINCSE BEAT
0.5-
NCSE-0217
MICS 8O1T
Pgamplee
(c) SctCS intersection performance on FPB
(f) SctCSE diffcrence performance on FPB dataset.
datasct.
S0F1-120M
SIPI 2235 DERT
DEAT
SirCSS DE
SenCzE SEIT D1RT
RE38-0011
1.5
085
GERT
Auurett
SetCSE intersection performance
(h) SerCSE difference performance FMTOD datasct.
FMTOD dataset.
Figure 8: SetCSE operation performances on AGD, Banking77, FPB, and FMTOD datasets for different values of sample-
26

---

Published as a conference paper at ICLR 2024
E.2 COMPARISON WITH SUPERVISED CLASSIFICATION
We also compare the SetCSE intersection performance with supervised classification, which regards the sample sentences are training sets, and predicting the class of each queried sentence. For comparing the two mechanisms, we aslo control the training epochs as the same.
The detailed results for the same evaluation datasets considered in Section 5 are listed in Table 13. As one can see, the results are on par with the one for SetCSE intersection, while supervised classification can not be used for querying semantically different sentences and conducting subsequent querying tasks.
News-T
AG News-D
FPB
Banking77
FMTOD
Acc
Acc
Acc
Acc
Acc
BERT
70.37
68.43
86.05
85.89
71.04
71.56
93.92
93.74
98.49
98.49
RoBERTa
75.54
75.56
88.60
88.64
75.34 15.34
75.58
83.29
83.29
99.05
99.05
92.35
37.55
22
SinCSE-BERT
77.77
77.72
86.14
86.17
82.52
82.53
98.63
98.54
99.31
99.31
SinCSE
SinCSE-RoBERTE RoBERTa
78.01
78.01
87.88
87.73
85,19
85.16
98.41
98.41
97.63
97.63
DITCSE BERT
74.11
74.05
88.49
88.44
82.6 61
82.66
98.45
98.45
99.66
99.66
DITCSE RoBERTs
78.37
78.33
88. 29
88.27
84. 36
84.27
98.53
98.53
99.14
99.14
MCSE. MERT BERT
72.99
72.38
85. 34
85.
80.22
80.41
97.29
97.16
99.35
99.35
COSE-R
75.94
76.02
88.14
88.13
86.76
86.78
98.26
98.29
99.16
99.16
Table 13: Evaluation results for supervised classification (7ssample 20).
E.3 BENCHMARK NLU TASK PERFORMANCES POST INTER-SERR CONTRASTIVE LEARNING
As mentioned in Section 5 and Appendix C, inter-set contrastive learning significantly enhances the embedding models' awareness of presented semantics. However, our interest also lies in evaluating the model's performance on general Natural Language Understanding (NLU) tasks after this context- specific finettuiig. To this extend, we conduct evaluations on seven standard semantic textual similarity (STS) tasks (Agirre et al., 2012; 2013; 2014; 2015; 2016; Cer et al., 2017).
We first perform inter-set contrastive learning as described in the Section 5.1 experiment using the five considered datasets. Subsequently, we evaluate the model's performance on STS tasks. The model used is SimCSE-BERY and the training aeperpparameters are the same as the ones in Section 5.1. The Spearman's 's correlation results for the STS tasks are presented in Table 14.
STS12
STS13
STS14
STSI5
STS
STS-B
SimCSE-BERT
83.22
81.74
83.09
SimCSE BERT (AGT)
SimCSE-BERT (AGD)
64.57
73.18
74.57
75.58
74.01
79, 14
SimCSE-BERT (FPB)
57.02
65.37
67.17
65.08
68.52
77.26
Sim (Banking77)
66.22
75.67
77.14
80,88
79.41
80.19
SimCSE.BERY (FMTOD)
66.39
72.84
76.82
80.31
78.95
80.12
Ave. Change
-7%
-7%
-6%
-8%
-7%
-4%
Table 14: Model performance on STS tasks post inter-set contrastive learning.
Notably, the application of inter-set contrastive learning exhibits no noteworthy adverse effect on the model's performance in benchmark STS tasks. On average, the utilization of inter-set contrastive learning only minimally diminishes the model's performance across STS tasks by 7% in Spearman's correlation.
27


Reviews:
[{"review_id": "NduXqt9Rba", "reviewer": "Reviewer_3g7o", "summary": "This paper introduces Set Operations using Contrastive Learning of Sentence Embeddings (SetCSE), a framework that incorporates set theory into semantic search to handle complex and multifaceted queries. Recognizing that intricate semantics often arise from clusters of sentences rather than isolated ones, the authors present a method where sets of sentences collectively represent semantics. The inter-set contrastive learning component is designed to fine-tune language models to capture contextual nuances and discern between different semantic sets. The SetCSE operations - intersection, difference, and series - are employed to structure queries effectively, allowing for a granular and nuanced retrieval of sentences from large corpora. An illustrative use case of analyzing S&P 500 companies’ stances on ESG issues demonstrates the framework's practical utility in complex information retrieval scenarios.", "questions": "1. Can the authors discuss any observed limitations or frequent error patterns during SetCSE operations?\n\n2. How does SetCSE scale with the size of the dataset and the complexity of the query semantics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Provides a well-defined and practical framework that enables complex information retrieval tasks which are not possible with current search methodologies.\n\n2. Applies contrastive learning to sentence embeddings in a unique way, emphasizing contextual differentiation between sets of sentences.\n\n3. Offers compelling real-world applications, such as parsing nuanced topics like ESG stances from earnings calls, highlighting the framework’s potential for practical deployment.", "weaknesses": "1. There is no mention of an error analysis which would be beneficial in understanding the limitations of SetCSE in certain scenarios.\n\n2. The applications of complex semantic search, data annotation, and new topic discovery are very cool with the detailed examples, but there is not quantification here or comparison with others with existing set methods from the literature (same with Table 1 and 2 as well). Do you have comparisons with other methods from the literature on this topic?\n\nTypos:\nSection 7 \"DISUCSSION\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OecP3CIS1V", "reviewer": "Reviewer_BQxn", "summary": "The current research piece builds on top of existing sentence embedders based on contrastive learning by defining set operations that can be applied to sentences. Concretely, the concept of semantic similarity between sentences is extended to be defined between individual ones and Sets of sentences, by taking the average similarity. \nTwo new set operations are defined. The operations build upon the order relationship between the elements of the first set. The intersection of A and B is the set of elements in A that are closer to B (formally all the elements in the intersection are more similar to B than any other element not in the intersection). And the difference A minus C, corresponding to all the elements in A that are less similar to C (analogously to intersection, all the elements in the different are *less* similar to C tan the elements not in the difference).\nThe framework can be applied to any language models that measure sentence similarity. Experiments are carried out taking baselines as TDIDF, BERT, RoBERTa, Contriever SimCSE, DiffCSE MCSE and SGPT on the AG News, Financial PhraseBank, Banking77 and MTOD datasets.\nBaselines are compared without and with the contrastive training that makes the model aware of set operations, showing an improvement in their perception of these set operations.\nThe work closes with a use case application. The set operations can be used to search related sentences using a set of sentences as positive or negative filtering criteria.", "questions": "How do the models perform on sentence similarity tasks after applying the SetCSE training?\n\nHow about other general NLU tasks such as sentence classification, sequence tagging, extractive QA or multiple choice QA? \n\nThe largest model where this approach was applied was a RoBERTa-like model. Does the approach escalate to bigger models? \n\nCan it be applied while using LoRA?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The idea of this paper is really clever, simple and straightforward.\nThe motivation is there, provide a LLM with notions of set theory to improve it in terms of search capabilities. \nExperimentation seems reasonable and enough. Proves the point authors want to provid", "weaknesses": "The authors make a good point to show the capabilities brought by the new training regime. However, there is no analysis on capabilities that are lost because of it. Do the models train with this regime underperform on sentence similarity or information retrieval datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t3JC5HiNpK", "reviewer": "Reviewer_fKmY", "summary": "The paper proposes a new information retrieval framework, SetCSE, based on set operation and sentence-level contrastive learning. The whole framework includes two steps: 1. fine-tuning sentence embedding model by minimizing inter-set loss. 2. ranking sentences in the set based on definition 2. The paper then defines two set operations, interactions and differences, based on the sentence-set semantic similarity. Furthermore, the paper conducts experiments for both set intersection and difference. Additionally, the paper shows three real-world applications, including semantic search, data annotation, and new topic discovery. The paper also provides quantitative analysis by comparing SetCSE with supervised learning.", "questions": "How to adapt the proposed method to a situation where sentences are not clustered with semantic meaning or where the clusters do not exist?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper formulates the sentence retrieval problem as a combination of sentence-set similarity and set operations, which is novel for the community. \n2. The paper provides a comprehensive set of experiments, introducing two new settings: set intersection and set differences. It uses multiple baselines to demonstrate the robustness of the framework. The paper also offers sentence embedding visualizations to illustrate the improvement in sentence representations. Additionally, the paper presents detailed hyperparameters and offers quantitative justification for the proposed framework\n3. The paper provides three different downstream applications. Each is paired with background papers and results to show the effectiveness of the proposed methods.", "weaknesses": "1. The paper would benefit from an additional experiment on sentence retrieval, similar to the one in Section 6.1. In this setting, the paper could compare its performance against traditional retrieval-based methods such as DPR and BM25 to better illustrate the model's improvements. Furthermore, in Table 2, certain baseline models do not show significant improvements with SetCSE. For instance, the improvements for SGPT are marginal when compared to other baselines. The paper should include an analysis explaining the variations in improvements among different models\n2. The core concept is to create clusters of sentences with semantic meaning, and this can limit the generalization ability of the proposed framework. In comparison to other baselines, the incorporation of semantic meaning within sets naturally provides additional information for training.\n3. The paper fails to provide code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TDT0xBIy0x", "reviewer": "Reviewer_wrMM", "summary": "This paper proposes a method to learn the sentence embedding from existing pre-trained sentence embedding to distinguish different semantics and define similarity-based set operations, including intersection and difference and their combinations. The paper evaluates the method on artificial data and shows several case studies of applications, including semantic search, data annotation, and topic discovery.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a novel and simple method to fine-tune existing pre-trained embeddings to be fit to set-based operations. \n- The results in the artificial setting show the feasibility of the method for several different sentence embedding\n- The case studies show interesting results that present the method's potential.", "weaknesses": "- Representing set operation with embedding is not novel, but the comparison and discussion compared with existing methods are missing. \n  - Vitalii Zhelezniak, Aleksandar Savkov, April Shen, Francesco Moramarco, Jack Flann, Nils Y. Hammerla Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors. ICLR 2019.\n  - Siddharth Bhat, Alok Debnath, Souvik Banerjee, and Manish Shrivastava. Word Embeddings as Tuples of Feature Probabilities. RepL4NLP. 2020.\n  - Shib Dasgupta, Michael Boratko, Siddhartha Mishra, Shriya Atmakuri, Dhruvesh Patel, Xiang Li, and Andrew McCallum. Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings. ACL2022.\n- The set operation presented in the paper does not satisfy the commutative law, and it orders the elements in the first set. This is not the usual set theory, and the users may be confused if they use the method that supports usual set operations, but the limitations are not discussed in detail.\n- The quantitative evaluation is performed only in artificial settings, and there are only case studies for the application results. It is unclear how the method can be stably used for the application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)