Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations
Authors: James Y. Huang
Wenlin Yao
Kaiqiang Song
Hongming Zhang
Muhao Chen
Dong Yu
Venue: EMNLP 2023 Main (2023)
Keywords: sentence embedding
representation learning

Abstract: Traditional sentence embedding models encode sentences into vector representations to capture useful properties such as the semantic similarity between sentences. However, in addition to similarity, sentence semantics can also be interpreted via compositional operations such as sentence fusion or difference. It is unclear whether the compositional semantics of sentences can be directly reflected as compositional operations in the embedding space. To more effectively bridge the continuous embedding and discrete text spaces, we explore the plausibility of incorporating various compositional properties into the sentence embedding space that allows us to interpret embedding transformations as compositional sentence operations. We propose InterSent, an end-to-end framework for learning interpretable sentence embeddings that supports compositional sentence operations in the embedding space. Our method optimizes operator networks and a bottleneck encoder-decoder model to produce meaningful and interpretable sentence embeddings. Experimental results demonstrate that our method significantly improves the interpretability of sentence embeddings on four textual generation tasks over existing approaches while maintaining strong performance on traditional semantic similarity tasks.

Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operations
James Y. Huang', Wenlin Yao Kaiqiang Song,, Hongming Zhang",
Muhao Chen and Dong Yu'
TUniversity of Southern California; *Tencent Al Lab, Seattle {huangjam, muwaachhe Jesscheeeeuuc...ee (wenlinyao, riversong, hongmzhang, dyu]@global. tencent.com
Abstract
optimize the sentence representation space such that the distance between embeddings reflects the semantic similarity of sentences.
Traditional sentence embedding models encode sentences into vector representations capture useful properties such as the semantic similar- ity between sentences. However, in addition similarity, sentence semantics can also be interpreted via compositional operations such sentence fusion or difference. It is unclear whether the compositional semanties of sen- lences can be directly reflected as composi- tional operations in the embedding space. To more effectively bridge the continuous embed- ding and discrete text spaces, we explore the plausibility of incorporating various composi- tional properties into the sentence embedding space that allows us interpret embedding transformations as compositional sentence op- erations. We propose INTERSENT, an end-to- end framework for learning interpretable sen- tence embeddings that supports compositional sentence operations in embedding space. Our method optimizes operator networks and bottleneck encoder-decoder model to produce meaningful and interpretable sentence embed- dings. Experimental results demonstrate that our method significantly improves the inter- pretability of sentence embeddings four tex- tual generation tasks over existing approaches while maintaining strong performance on tradi- tional semantic similarity tasks.!
While the similarity structure of sentence em- bedding models an important aspect of the inter- sentence relationship, contrastive learning meth- ods do not provide a direct way of interpreting the information encoded in the sentence embedding. Despite the existence of probes for individual lin- guistic properties (Conneau al., 2018), it is still unclear whether the embedding fully captures the semantics of the original sentence necessary for reconstruction. Morcover, sentence semantics not only can be interpreted by their similarity but also via sentence operations such as fusion, difference and compression. While these operations of sen- tence semantics have been previously studied indi- vidually sequence-to-seequecce generation tasks (Geva et al., 2019; Botha al., 2018; Filippova and Altun, 2013; Rush al., 2015), remains an open research question whether these operations can be directly captured as operations in the sentence em- bedding space. We argue that the ability to interpret and manipulate the encoded semantics is an impor- tant aspect of interpretable sentence embeddings, which can bridge the continuous embedding space and the discrete text space. Particularly, this abil- ity benefits concrete tasks such as multi-hop search and reasoning (Khattab al., 2021), instruction fol- lowing (Andreas and Klein, 2015), compositional generation (Qiu et al., 2022), and summarization (Brook Weiss et al., 2022).
1 Introduction
Learning universal sentence embeddings is crucial to wide range of NLP problems, they can provide out-of-the-box solution for various im- portant tasks, such as semantic retrieval (Gillick et al., 2018), clustering (Hadifar et al., 2019), and question answering (Nakov et al., 2016). Recently, contrastive learning has been shown to be an ef- fective training paradigm for learning sentence em- beddings (Giorgi et al., 2021; Yan et al., 2021; Gao et al., 2021; Chuang et al., 2022). These methods
In this work, we propose INTERSENT, an end- to-end framework for learning interpretable and ef- fective sentence embeddings that supports compo- sitional sentence operations. Our method combines both generative and contrastive objectives to learn well-structured embedding space that satisfies useful properties for both utility and interpretabil- ity. Specifically, together with an encoder-decoder model, we train several small operator networks on
'Code is available at htop::://gihhtt wtt.ub..co//////// wwt.cooo///i/a/ com/jyhuang36/ InterSent

---

Text Space
Output Sentence
Target Sentence
Sentence Decoder
Sentence Embedding Space
Contrastive
Embedding
Embedding
Embedding
Operator
Embedding
Sentence Encoder
Text Space
Operation
Sentence
Sentence
Target Sentence
Figure I: Overview of our proposed framework IENT for learning interpretable sentence embeddings. Given pair of input sentences (or single input sentence for compression and reconstruction), INTERSENT encodes the input sentence(s) into sentence embedding (s), which are fed into an operator network to compute the output embedding for decoding. During training, INTERSENT ensures alignment between the output and target in both the embedding space and text space via contrastive and generative objectives respectively.
casily available weak supervision data to capture different sentence operations. Sentence embed- dings learned by our model not only preserve the ability to express semantic similarity but also sup- port various sentence operations (such as sentence fusion, difference, and compression) for interpret- ing compositional semantics.
tence embedding space and provide a detailed de- scription of our proposed framework INTER SENT.
2.1 Problem Definition
Our notion of interpretable sentence representa- tions centers around the ability to interpret em- bedding vectors and simple transformations de- fined over them in the embedding space, as human- comprehensible sentences and sentence operations in the text space. In other words, our goal is to establish
Our contributions are three-fold. First, we pro- pose INTERSENT, an interpretable sentence em- bedding model that establishes a mapping between the continuous embedding space and the discrete text space by connecting transformations on em- beddings and compositional operations texts. Second, our method significantly improves the in- terpretability of sentence embeddings on four tex- tual generation tasks. Third, we demonstrate that interpretable sentence embeddings learned by our method still maintain strong performance on tradi- tional semantic similarity and text retrieval tasks.
a mapping between embedding vectors and sen- tences, which allows to both encode sentences into vectors and decode vectors into sentences;
a mapping between certain simple transforma- tions over vectors and certain sentence opera- tions, which allows us to manipulate sentence semantics in the embedding space.
2 Method
In this work, we explore the plausibility of sup- porting several common sentence operations that are previously studied as individual sequence-to- sequence generation tasks (Geva al., 2019; Botha et al., 2018; Filippova and Altun, 2013; Rush al., 2015). These include the following compositional operations:
Fig. illustrates the overall architecture of INTERSENT. Our method, INTERSENT, optimizes both contrastive objective (in the continuous space) and generative objective (in the discrete text space) jointly during training. this section, we define the notion of interpretability for a sen-

---

Given the embeddings of two sentences, an embedding of their fusion, which contains information from both sentences, can be inferred.
Sentence Fusion:
simple arithmetic operations, such as addition and subtraction of embedding vectors. We empirically show that defining sentence operations with sim- ple arithmetics leads to inferior performance on downstream tasks (see $4.2 for more details). comparison, MLP transformations achieve good balance between simplicity and flexibility to fit dif- ferent sentence operations. All operator networks take sentence embedding (for compression), concatenated pair of sentence embeddings (for fu- sion and difference), and compute new sentence embedding as the target. For the compression op- erator, we use a small intermediate dimension size to limit the information flow and encourage the model only to preserve essential information in the compressed embeddings.
Sentence Difference: The embedding of the dif- ference between two sentences can be inferred from their individual embeddings. To avoid am- biguity, we restrict this definition to only cases where the first sentence contains the whole infor- mation of the second sentence. other words, sentence difference is essentially defined as an inverse operation of sentence fusion.
In addition, consider the following compres- sion operation, as well sentence reconstruction that help interpret the meaning of any sentence embedding vector.
2.3 Bottleneck Model
Sentence Compression: Given the embedding of sentence, we seek to infer the embedding of the compression or summarization of this sentence.
Our model uses Transformer- bassed language mod- els encoders and decoders for sentence embed- dings. Unlike in the typical encoder-decoder archi- tecture for sequence generation, where the decoder has access to the contextualized representations of all tokens, the encoder INTERSENT only outputs single vector as the representation for cach in- put sentence. Following previous work (Gao et al., 2021; Chuang et al., 2022), we take the represen- tation of the [CLS] token from the encoder the sentence embedding. This information bottleneck forces the model to produce meaningful sentence embeddings such that the decoder can reconstruct the semantics given the embedding vectors alone. It is worth noting that the encoder and decoder are shared across all sentence operations, which forces the model to capture the operations in the embedding space, rather than learning task-specific encoders and decoders for each operation.
Sentence Reconstruction: The content of the orig- inal sentence can be recovered from its sentence embedding. This property serves the founda- tion for interpretability, it allows to under- stand the semantics of any sentence embedding vector, including those computed by applying the sentence operations (in the form vector transformations) in the embedding space.
In other words, we aim to learn a sentence en- coder Enc, a sentence decoder Dec, and sentence operator functions fius. faitt, Scomp that satisfies the following properties:
Enc(si (1 82) = fin. (in((En((ss), Enc(s2) where sentence 81 @ 82 is the fusion of sentence $1 and $2-
Enc(s1 e 82) x fain(E (ann(Enc(s1). Enc(s2)) where sentence $1 e 82 is the difference of sentence $1 and 82.
2.4 Training Objective
INTERSENT combines contrastive and genera- tive objectives to learn interpretable sentence em- beddings that captures both semantic similarity and sentence operations. Specifically, we train INTERSENT to maximize alignment between out- puts and targets in both the embedding space and text space. This means that the output embedding computed by operator function should be close to the target embedding, and the target sentence be decoded from the output embedding. The first objective is realized by optimizing contrastive loss with bn-batch negatives. For the i-th training instance, let v; and v, denote the output embed-
Enc(3) R feomp( (Enc())) where sentence & is a compression of sentence s.
s z Dec(Enc(s)) where s' and are a pair of sentences expressing the same semantics.
2.2 Sentence Operator
We two-layer MLPs fit operator functions over embeddings, which can be trained together with the rest of the model components an end- to-end manner. Some compositional sentence op- erations may be alternatively approximated with

---

ding (computed by the encoder fene and an oper- ator function from the input sentence(s)) and the
3.2 Implementation
We train the encoder and decoder with weights initialized from RoBERTa² (Liu et al., 2019) and BART (Lewis al., 2020), respectively. This hy- brid setting allows us to utilize the high-quality sentence-level representations from RoBERTa while taking advantage of the generative capability of BART. In experiments, we also adopt another two encoder-decoder model backbones (i.c., T5 (Raffel et al., 2020) and BERT+BERT), but they perform slightly worse than RoBERT&+BART (see Appx. $C for more details). The loss balancing fac- tor is set to 0.01, as the contrastive loss converges much faster than the generative loss. We train the model on the combination of the five datasets for five epochs. More details about hyperparameters can be found in Appx. $B.
target embedding (computed by encoding the tar- get sentence directly). The contrastive objective for (v;, vt)) is given by
sim(ya.
Li,con = log
(1)
Σj=1 Ji
where N is the mini-batch size, and r is the softmax temperature. To ensure the target sentence can be decoded from the output embedding, we also optimize the following conditional generative loss:
17:1
Ligeen
Σ
log p(t,,,//Ti,sa vi). (2)
where T, denotes the target sequence for the i-th training instance in the batch. Both losses are com- bined with a balancing factor at:
3.3 Baseline
We compare our model with previous sentence embedding models, well unsupervised and contrastive baselines trained on the same datasets as INTERSENT. All models, including INTERSENT, use RoBERTa-base as the sentence encoder. RoBERTa-cls and RoBERTa-avg are sentence embeddings directly extracted via the [CLS] token and average pooling from un- finetuned RoBERTa model. SRoBERTa (Reimers and Gurevych, 2019) trains sentence embed- ding model on supervised NLI pairs by optimizing a cross-entropy loss to predict NLI labels. De- CLUTR (Giorgi et al., 2021) is an unsupervised contrastive sentence embedding model trained on sampled positive and negative pairs from raw text. CSE (Gao et al., 2021) instead uses different dropout masks applied to the same sentence data augmentation. DifECSE (Chuang al., 2022) intro- duces an additional replaced token detection objec- tive such that the embeddings are sensitive to token replacement. Since previous methods are trained on different datasets, we additionally include unsu- pervised and supervised contrastive models trained on our data for direct comparison. The Unsuper- vised Contrastive baseline is trained similarly to SimCSE, by breaking down all sentence pairs and triplets in our datasets into individual sentences. For (weakly) Supervised Contrastive baseline, we concatenate the two shorter sentences in the fusion and difference datasets and then use sentence
L;- = Li,oon + a.iigen-
(3)
3 Experiments
In this section, we evaluate the interpretability of INTERSENT and recent sentence embedding mod- els on four generative sentence operation tasks. Then, conduct experiments on traditional zero- shot semantic textual similarity benchmarks. Fi- nally, we compare our model with previous meth- ods on zero-shot sentence retrieval tasks.
3.1 Data
To learn the sentence operations we described pre- viously, INTERSENT is trained on the combination five weak supervision datasets for sentence fu- sion, difference, compression, and reconstruction. DiscoFuse (Geva et al., 2019) is sentence fusion dataset constructed with heuristic rules that com- bines two sentences into one. WikiSplit (Botha al., 2018) is spli--and-rehrrase dataset extracted from Wikipedia edits, which we use for the sen- tence difference task. For compression, we use the Google (Filippova and Altun, 2013) and Giga- word (Napoles et al., 2012) sentence compression datasets. Both of these datasets are collected by pairing the news headline with the first sentence use ParaNMT (Wict-
of the article. Finally, ing and Gimpel, 2018) for sentence reconstruction, which contains paraphrase pairs generated from back-translation. It is worth noting that all of these datasets are constructed automatically. More de- tails of the datasets can be found Appx. SA.
We follow previous work and use RoBERTa-hase model (110M parameters) to make the results comparable.

---

Fusion R-2
Difference
Comp. (Google)
Comp. (Gigaword)
Avg. R-2
Model
R-1
R-L
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-L
RoBERTa-cls
42.8
16.9
35.1
30.9
10.4
27.0
27.7
10.6
25.8
30.4
11.5
27.7
32.9
12.4
28.9
RoBERTa-avg
68.7
43.1
58.5
54.4
29.0
46.3
43.6
23.8
41.2
37.4
16.4
34.1
51.0
28.1
45.0
SRoBERTa
49.0
21.9
39.0
16.2
33.2
37.8
35.5
36.0
15.5
32.7
40.5
18.2
35.1
DeCLUTR
73.4
46.7
61.6
56.1
30.1
47.3
50.3
30.1
47.9
39.6
17.9
36.1
54.9
31.2
48.2
SimCSE
53.2
24.3
42.0
36.1
13.6
30.8
38.4
18.2
35.9
35.0
31.5
40.7
17.6
35.1
DifICSE
57.8
28.7
40.3
16.9
34.2
21.0
39.0
36.4
32.8
44.1
20.5
38.0
Eacoders trained data
Unsup, Contr.
$7.9
28.6
45.8
38.9
15.7
33.1
41.4
20.7
38.7
35.9
15.0
32.4
43.5
20.0
37.5
Sup. Contr.
57.7
29.4
46.9
36.9
14.4
31.7
50.3
28.7
47.5
41.6
19.2
37.9
46.6
22.9
41.0
InterSent 88.7
71.9
82.2
73.0
48.4
64.3
69.6
50.9
66.3
48.0
24.7
43.8
69.8
51.5
64.2
Table 1: Model performance four textual generation tasks for interpretability evaluation. Unsup. Contr. and Sup. Contr: represents Unsupervised and Supervised Contrastive baselines respectively. We report ROUGE-1/2/L scores.
Model
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
RoBERTa-cls
16.67
45.57
30.36
55.08
56.99
38.82
61.90
43.63
RoBERTa-avg
32.11
56.33
45.22
61.35
61.98
55,49
62.03
53.49
SRoBERTa
71.54
72.49
70.80
78.74
73.69
77.77
74.46
74.21
DeCLUTR*
52.41
75.19
65.52
77.12
78.63
72.41
69.99
SimCSEt
70.16
81.77
73.24
81.36
80.65
80.22
68.56
76.57
DifICSE
70.05
83,43
75.49
82.81
82.12
82.38
71.19
78.21
Eacoders trained on our data
Unsupervised Contrastive 69.11 1
82.05
74.99
82.65
81.00
81.20
69.56
77.22
Supervised Contrastive
72.46
81.42
77.18
84.08
79.68
82,95
74.82
78.94
InterSent
70.97
81.03
75.30
83.18
79.60
80.60
69.45
77.16
Table 2: Model performance on Semantic Textual Similarity (STS) tasks. We report Spearman's 's correlation on all tasks. 1: results taken from (Reimers and Gurevych, 2019). t: results taken from (Gao et al., 2021). c: results taken from (Chuang et al., 2022).
Model
MRR@10
recall
operator networks and decoders on top of their trained encoders, making the model architecture identical across all models. For all baseline models, we take the sentence embeddings encoded by these sentence encoders, and optimize the added operator networks and decoder during training. This setting allows us examine if existing sentence embed- dings already support sentence operations and con- tain sufficient information for reconstruction. By comparing contrastive baselines with our method trained on the same data, we can also have a better understanding of how much fine-tuning sentence encoders (along with the rest of the model) on both generative and contrastive objectives can benefit the interpretability of the learned embedding space. We report ROUGE-1/2/L scores (Lin, 2004).
RoBERTa-cls
56.53
65.43
RoBERTa-avg
54.87
64.89
SRoBERTa
71.13
79.81
DeCLUTR
75.84
87.02
SimCSE
79.78
89.32
DifICSE
79.49
89.53
Encoders trained on our data
Unsupervised Contrastive 79.34
89.11
Supervised Contrastive InterSent
79.67
89.71
80.30
89.94
Table 3: Model performance the zero-shot QQP sentence retrieval task. We report both Mean Reciprocal Rank 10 (MRR@) 10) recall@ 10.
pairs as weak supervision pairs.
3.4 Interpretability
Results. As shown in Tab. 1, our method signif- icantly outperforms all baselines across four tence operation tasks. Without fine-tuning, average pooling, which aggregates all token representations, unsurprisingly outperforms CLS pooling by a large margin. Among previous sentence embedding mod-
Setup. We first compare the interpretability of sentence embedding space on generative sentence operation tasks including fusion, difference and compression. Since none of the baseline models include a decoder for sentence generation, we stack

---

els, DeCLUTR, which incorporates a masked lan- guage modeling objective, has the best overall inter- pretability performance. While contrastive learning on our data indeed helps the model adapt to our datasets, there still exists a large gap between the su- pervised contrastive baseline and our INTERSENT. This demonstrates that simply training sentence en- coders with contrastive objective, in previous sentence embedding models, is not sufficient to create an interpretable sentence embedding space. Jointly optimizing both contrastive and generative objectives encourages sentence encoders to pre- serve sufficient token-level information to better support sentence operations and reconstruction.
both Mean Reciprocal Rank @ 10 (MRR @ 10) and recall 10 metries.
Results. As shown in Tab. 3, INTERSENT achieves the best performance on the sentence retrieval task. Notably, INTERSENT outperforms the super- vised contrastive baseline trained on the same data, which shows that adding interpretability properties can benefit modeling semantic similarity. Com- bined with the significant improvement in embed- ding interpretability and strong STS performance, demonstrate that INTERSENT learns an inter- pretable sentence representation space that supports various sentence operations while preserving the ability to capture semantic similarity.
3.5 Semantic Textual Similarity
Analysis
Setup. In addition to interpretability, we also in- vestigate if INTERSENT preserves the ability to capture semantic similarity. Following previous work, we evaluate our model on the semantic tex- tual similarity (STS) tasks, including STS 2012- 2016 (Agirre et al., 2016), STS Benchmark (Cer et al., 2017), and secceeelass (Marelli et al., 2014). The goal of these tasks is to estimate the semantic similarity between two sentences by com- puting the cosine similarity of their sentence em- beddings. All models are evaluated under the zero- shot setting without training on any STS data, We report Spearman's correlation for all tasks.
To provide better understanding of INTERSENT, we investigate how INTERSENT handles longer text, and present an ablation study on the effect of individual loss functions, and choice of pre- trained language models for encoders and decoders. Then, we analyze the operator functions learned by INTERSENT through detailed case study.
4.1 Passage Retrieval
The goal of passage retrieval is to retrieve the most semantically relevant passage given the query sen- tence, whereof the query and passages are of dif- ferent granularities. Sentence embedding models generally do not perform well on passage retrieval tasks due to their asymmetric nature. Addition- ally, passages are generally much longer than the query and contain multiple sentences, making it challenging for sentence embedding model capture their semantics the same way as it does for single sentences (Muennighoff et al., 2022). To investigate how well sentence embedding models handle longer text, we evaluate passage retrieval performance NaturalQuestions (Kwiatkowski et al., 2019) and MSMARCO (Nguyen et al., 2016) datasets, under the zero-shot setting without train- ing on any passage retrieval data.
Results. As shown in Tab. 2, incorporating addi- tional properties that support sentence generation leads to a slight performance decrease on the STS tasks compared to the supervised contrastive base- line. We also observe that the gap between unsuper- vised and supervised contrastive basclines trained on our data is relatively small, as the weak super- vision data we use inherently contain some noise. Nevertheless, INTERSENT'S performance on STS is still strong enough match the unsupervised contrastive baseline trained on the same data.
3.6 Sentence Retrieval
Setup. One important application of sentence em- beddings is sentence retrieval, where the goal is to retrieve the most semantically relevant sentence given the query sentence. We conduct sentence retrieval experiments on the QQP dataset, which is originally designed for paraphrase identification. We follow the data splits used in BEIR (Thakur et al., 2021) and report zero-shot performance on the test set that contains 10,000 queries. We use
As shown in Tab. 4, INTERSENT achieves the best performance on both passage retrieval tasks. We can also clear performance gap between INTERSENT and baselines trained on the same data. This demonstrates that modeling compositional manties between sentences helps the model better capture the semantics of longer text, and preserves necessary information for retrieval.

---

Model NQ
MSMARCO
Operator
Fusion
Difference
STS
SimCSE DifICSE
41.58 40.52
35.55
Arithmetic
59.02
62.28
77.46
30.64
MLP
82.19
64.34
77.16
Encoders trained on our data
Table 6: Model performance on sentence fusion, differ- ence and STS trained with different choice of operators. We report ROUGE-L score on interpretability tasks and the average Spearman's correlation on STS tasks.
Unsupervised Contrastive 43.10 42.90
35.13
Supervised Contrastive
32.56
InterSent
49.64
37.87
Table 4: Model performance zero-shot passage re- trieval tasks. We report recall @ 100 on both Natu- ralQuestions and MSMARCO datasets.
operators with simple arithmetics leads to signifi- cant decrease in generation performance, especially on sentence fusion. This demonstrates that, while simple arithmetics themselves are easier under- stand, they do not accurately capture the nature of sentence operations in the embedding space.
Model
Interpret.
STS
INTERSENT
64.18
77.16
Generative-only
65.64
64.11
Contrastive-only
78.29
Table 5: Model performance on the interpretability and STS tasks trained with different training objectives. We report the average ROUGE-L score on interpretability tasks and the average Spearman's correlation on STS tasks.
4.3 Case Study
To better understand the characteristies of oper- ator functions learned by INTERSENT, and how they interact with each other, we conduct case study on multi-step sentence operations enabled by INTERSENT. We present a few representative amples in Tab. 7, which covers basic operations of our method (fusion, difference, and compression), as well as compound ones that combine two basic operations. All sentence operations we demon- strate are carried out in the sentence embedding space, and the output sentence is decoded from the final embedding calculated by the operators.
4.2 Ablation Study
Effect of Training Objectives. We conduct an ablation study on the role of contrastive and gen- erative objectives by comparing the model perfor- mance of INTERSENT with generative-only and contrastive-only baselines. Both of these two base- lines are optimized using only one of the training objectives. For the contrastive-only baseline, we only report the STS performance since the decoder is not aligned to produce any meaningful output given the sentence embedding.
As shown in Tab. 7, INTERSENT can generate coherent output sentences that follow the individual sentence operations we apply to the embeddings. Moreover, we observe that the fusion and differ- ence operators learned by our method indeed repre- sent inverse operations on sentences, as shown by the output of two compound operations: difference after fusion and fusion after difference. Our model does not directly enforce this property. Instead, it emerges result of the joint training sentence fusion and difference tasks. Operators learned by INTERSENT can also be combined many other ways, and we demonstrate two examples of com- pound operations supported by INTERSENT: multi- sentence fusion that fuses more than two sentences, and compression after fusion, which compresses the combined information from two sentences. As shown in Tab. 7, INTERSENT generates reason- able outputs for these compound operations even though they are not directly optimized during train- ing. This demonstrates the potential of our inter- pretable sentence embedding space in which we can represent more complex and diverse sentence
As shown in Tab. 5, the combination of con- trastive and generative objectives is crucial to sup- porting sentence operations while maintaining the ability to capture semantic similarity. Without generative objective, is impossible to examine the content being encoded the sentence embeddings. On the other hand, the generative-only bascline only improves slightly on generative tasks at the cost of a significant performance drop on STS tasks, SENT achieves desirable balance between interpretability and semantic similarity.
Choice of Operators. We investigate the effect of using simple arithmetics instead of MLPs oper- ators by simply computing addition and subtrac- tion for sentence fusion and difference respectively. The compression operator remains be trainable MLPs for both models. As shown in Tab. 6, both models have similar STS performance, but defining

---

Operation
Sentence
(A) They wanted to do more than just straight news.
(B) They hired comedians who were talented vocalists.
Fusion
(A Wanting do than just straight news, they hired comedians who were talented
vocalists.
Difference after Fusion
((A é
A) They wanted hire talented comedians who were more vocal.
(A) The first edition of the dictionary was printed in 1940, but soon became of print in 1958.
(B) The first edition of the dictionary was printed in 1940.
Differenee
(A e B) However, it soon became out of print in 1958.
Fusion after Difference
(B B (A B)) The first edition of the dictionary was printed in 1940, but soon became of
print in 1958.
(A) Nearly one million people have been left without water in South Africa's 's northern region
because of disastrous drought, the regional Water Affairs acting director said Wednesday.
Compression
(A) One million people left without water in drought-hit South Africa.
(A) Due financial difficulties, the airline filed for bankruptey.
(B) The airline suspended scheduled passenger services.
(C) The airline planned to lease its fleet other airlines.
Multi-sentence Fusion
(AA B) C) However, due to financial difficulties, the airline suspended scheduled flights for bankruptey, and the airline planned to lease its flect to other airlines.
(A) The Commonwealthh sumuni opened in Sri Lanka with heavy security on Friday.
(B) The summit was attended by heads of state or their representatives from 53 member nations.
Compression after Fusion
(A The high security summit was convened by leaders of the Commonwealth.
Table Examples of sentence operations supported by INTERSENT. Following the notation we defined in $2.1, we use θ and to denote sentence fusion, difference and compression respectively.
operations by combining basic operators.
lcarning, purely generative methods do not directly optimize the similarity between embeddings, and generally do not outperform contrastive methods on semantic similarity tasks.
5 Related Work
Sentence Embedding. Following the distribu- tional hypothesis of semantics (Harris, 1954), early unsupervised sentence embedding methods (Kiros al., 2015; Hill al., 2016: Logeswaran and Lee, 2018) extend the idea of word embedding models (e-g., word2vec (Mikolov et al., 2013)) by predict- ing surrounding sentences based on the given sen- tence. Supervised methods (Conneau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019) utilize human-annotated data, mostly premise- hypothesis pairs from natural language inference, improve the quality of sentence embedding fur- ther. Recently, contrastive learning has emerged as a widely used learning paradigm for sentence embeddings (Giorgi et al., 2021; Yan et al., 2021; Gao et al., 2021; Chuang et al., 2022). These meth- ods learn a well-structured representation space by explicitly bringing sentences with similar se- mantics (or augmented versions of the same sen- tence) closer. Meanwhile, several works have also explored generative modeling of sentence embed- dings with denoising or masked language modeling objectives (Giorgi al., 2021; Wang et al., 2021; Huang et al., 2021; Gao et al., 2021; Chuang et al., 2022; Wu and Zhao, 2022). Unlike contrastive
Representation Interpretabiliit. Prior works have studied the interpretability of text representa- tions and their operations from various perspectives. Early works on word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Arora et al., 2016; Ethayarajh et al., 2019) have demonstrated compo- sitional properties of word embedding spaces that allow us to interpret simple arithmetic operations semantic analogies. Similar properties have been studied in the context of sentence embeddings (Zhu and de Melo, 2020). Previous works have also in- vestigated the compositionality of word, and phrase embeddings from pre-trained language models (Yu and Ettinger, 2020; Hupkes et al., 2020; Dankers et al., 2022; Liu and Neubig, 2022). Another impor- tant aspect of interpretability is whether the original information can be recovered from its embedding alone. While this generative property has been used as an auxiliary objective improve sentence em- bedding models (Wang et al., 2021; Huang et al., 2021; Gao et al., 2021; Chuang et al., 2022; Wu and Zhao, 2022), the quality of the generated text is rarely in the interest of these methods.

---

6 Conclusion
Sanjeev Arora, Yuanzhi Li. Yingyu Liang, Tengyu Ma. and Andrej Risteski. 2016. A latent variable model approach to pmi-hased word embeddings. Transac- rions of the Association for Computational Linguis- lics, 4:385-399.
In this work, present INTERSENT, an end- to-end framework for learning interpretable sen- tence embeddings that support sentence opera- tions, including fusion, difference, compression, and reconstruction. INTERSENT combines both contrastive and generative objectives to optimize operator networks together with a bottleneck encoder-decooder model. Experimental results show that INTERSENT significantly improves the inter- pretability of sentence embeddings four textual generation tasks. Moreover, we demonstrate that our interpretable sentence embedding space pre- serves the ability to capture semantic similarity, and even improves performance on retrieval tasks.
Jan Botha, Manaal Faruqui. John Alex, Jason Baldridge, and Dipanjan Das. 2018. Learning to split and rephrase from wikipedia edit history. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Lunguage Processing, pages 732-737.
Daniela Brook Weiss, Paul Roit, Ori Ernst, and Ido Dagan. 2022. Extending multi-text sentence fusion resources via pyramid annotations. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1854-188 184-1866 1860, Seattle, United States. Association for Computational Linguistics.
Acknowledgement
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval- 2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1 11th International Workshop on Semantic Evalu- ation (SemExal-2017) 2017), pages 1-14.
We appreciate the reviewers for their insightful comments and suggestions. James Huang and Muhao Chen were supported by the NSF Grants IIS 2105329 and ITE 2333736.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajando-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english, In Proceedings of the 2018 conference on empiri- cal methods in natural language processing: system demonstrations, pages 169-174
Limitations
First, as sentences can be transformed and com- bined in much more diverse and complex ways such as multi-sentence intersection (Brook Weiss et al., 2022), the list of sentence operations we study in this work is not exhaustive. Additional con- straints, such as the inverse relationship between fusion and difference, may also be introduced to directly enforce the consistency of operators. Sec- ond, all training datasets we use are generated au- tomatically thus, they inevitably contain noise. In this regard, our method shares the same limitations as the broad class of weakly supervised methods where training data are automatically generated.
Yung-Sune Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. Diffese: Difference-basee based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Alexis Conneau, Douwe Kiela, Holger Sehwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 670-680.
References
Alexis Conneau, Germán Kruszewski. Guillaume Lam- ple, Loie Barrault, and Marco Baroni. 2018. What you can cram into single $ &!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136.
Eneko Agirre, Carmen Banca, Daniel Cer, Mona Diab, Aitor Gonzälez-Agiree, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. Semeval-2016 task I: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop Semantic Evaluation (SemEval-2016), pages 497-511.
Verna Dankers, Christopher Lucas, and Ivan Titov, 2022. Can transformer be too compositional? analysing id- iom processing in neural machine translation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 3608-3626.
Jacob Andreas and Dan Klein. 2015. Alignment-basse compositional semantics for instruction following. In Proceedings of the 2015 Conference Empiri- cal Methods in Natural Language Processing, pages l 1165-1174.

---

Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Towards understanding linear word analogies. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 3253- 3262.
Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. Baleen: Robust hoo reasoning at scale via condensed retrieval. Advances in Neural Infor- mation Processing Systems, 34:27670-27682.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. Advances in neural information processing systems, 28.
Katja Filippova and Yasemin Altun. 2013. Overcom- ing the lack of parallel data in sentence compression. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1481-1491.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epsteen, Mllia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics. 7:452- 466.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 0894-6910.
Mor Geva, Eric Malmi, Idan Szpektor, and Jonathan Berant. 2019. Discofuse: A large-scale dataset for discourse-based sentence fusion. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4445-3455.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising pouuuencett-sequenee pre-training for nat- ural language generation. translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018 End-to-end retrieval in continuous space. arXiv preprini Arxiv:181 1a0::111100000000 1.08008.
Chin- Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 81.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. Declutr: Deep contrastive learning for unsuper- vised textual representations.
Emmy Liu and Graham Neubig. 2022. Are represen- tations built from the ground up? empirical ex- amination of local composition in language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (FMNIPP).
Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Vol- une 1: Long Papers), pages 879-895.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1907:1907.1192
Amir Hadifar, Lucas Sterckx, Thomas Demeester, and Chris Develder. 2019. A self-training approach for short text clustering. In Proceedings of the 4th Workshop on Representation Learning for NLP pages 194-199.
Zellig S Harris. 1954. Distributional structure. Word, 146-12...
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence represen- tations. In International Conference on Learning Representations.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1367-1377.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014. A sick cure for the evaluation of composi- tional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) 14), pages 216- 223.
James Y Huang, Kuan- Hao Huang, and Kai-Wei Chang. 2021. Disentangling semantics and syntax in sen- tence embeddings with pre-trained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1372-1379.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. Advances neural information processing systems. 26.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Eliaa Bruni. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Rexearch, 67:757-795
Niklas Muennighoff. Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Mieb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316

---

Preslav Nakov, Lluis Marquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Frei- hat, Jim Glass, and Bilal Randeree. 2016. Semeval- 2016 task 3: Community question answering. In Pro- ceedings of the 10th International Workshop on Se- muntic Evaluation (SemEval-2016), pages 525-545.
John Wieting and Kevin Gimpel. 2018. Paranmt-50m Pushing the limits of paraphrastic sentence embed- dings with millions of machine translations, In Pro- ceedings the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 45 1-462.
Courtney Napoles, Matthew R Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceed- ings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extrac- tion (AKBC-WEKEX) pages 95-100.
Bohong Wu and Hai Zhao. 2022. Sentence represen- tation learning with generative objective rather than contrastive objective. In Proceedings of the 2022 Conference Empirical Methods in Natural Lan- guage Processing.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: human generated machine read- ing comprehension dataset. In CoCo@ NIPs.
Yuanmeng Yan, Rumei Li. Sirui Wang. Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: con- trastive framework for self-supervised sentence repre- sentation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pages 5065-5075.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Lang Yu and Allyson Ettinger. 2020. Assessing phrasal representation and composition in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 4896-4907
Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. 2022. Improving compositional generalization with latent structure and data augmentation. In Proceed- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4341-4362, Seattle, United States, Association for Computational Linguistics.
Xunjie Zhu and Gerard de Melo. 2020. Sentence analo- gies: Linguistic regularities in sentence embeddings. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3389-3400
A Dataset
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res.. 21(140):1-67.
All training datasets we use in this work are pub- licly available except Gigaword which is licensed under LDC. We use them in accordance with their license and intended use. We use the balanced Wikipedia por- tion of Discofuse dataset, which consists of 1.490.803/45,957444.55 train/dev/test respectively. WikiSplit dataset consists of 989. 0599944455,0005,0000 for train/dev/test respectively. Google dataset consists of 200,000/10,000 for train/test respectively. For Gigaword, we filter out headline pairs with fewer than four overlapping tokens after removing stopwords. The resulting dataset consists of 3,535,01 1/190,034//78,999 for train/dev/test respectively. ParaNMT dataset consists of 5,370,128 paraphrase pairs and we use the entire dataset for training.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert. Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pages 3982-3992.
instances for
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021, Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. Thirty-fifih Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
B Hyperparameter
Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. Tsdae: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning. Findings of the Association for Compu- lational Linguistics: EMNLP 2021, pages 67 1688.
All experiments are conducted on NVIDIA V100 GPUs. Model training takes roughly 15 hours to complete on an 8-GPU machine. INTERSENT uses RoBERTa-base and BART-base as the encoder and

---

decoder respectively, which has roughly 200 mil- lion parameters total. During training, apply linear learning rate schedule with linear warmup on the first 5% the data. All inputs are truncated to maximum of 64 tokens. We finetune the en- coder and decoder with learning rate of 5e-6 and le-4 respectively, All operator networks use ReLU the activation function. The intermediate dimen- sion size of the compression operator is set to 384. We tune all hyperparameters on the STS-B and sentence generation dev sets.
Choice of Pretrained Language Models
We compare the dev set performance of different combinations of pretrained language models coders and decodes for INTERSENT in Tab. 8. We observe that the pair of RoBERTa and BART as the encoder and decoder achieves a good balance between generation and semantic similarity tasks.
Model
Interpret.
STS-B
BERT BERT
36.18
78.75
T5 RoBERTa BART
60.02
79.37
55.18
81.22
Table 8: Model performance on the dev set of inter- pretability tasks and STS-B. We report the average ROUGE-I score on interpretability tasks and the av- erage Spearman's correlation STS-B.


Reviews:
[{"review_id": "jImKZXoiMZ", "reviewer": "Reviewer_eJVG", "summary": "", "questions": "See the weakness part, correct me if I made some mistakes.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Freezing Baseline Encoder\nAs we discussed in section 3.4, the goal of this setting is to demonstrate to what extent we can improve from existing sentence embeddings on the sentence operation tasks without sacrificing downstream performance on semantic similarity tasks. This objective has not been studied in previous works. We have made our best effort to conduct a fair comparison with baselines by using an identical model architecture (RoBERTa encoder + operator + decoder), and training contrastive baselines on our data. Combined with our proposed operator networks and bottleneck structure, further fine-tuning the baseline encoders on generative objectives would essentially turn these baselines into InterSent itself.\n\n### Clarifications of Unsupervised/Supervised Contrastive Baselines\nThe unsupervised/supervised contrastive baselines are similar to unsupervised/supervised SimCSE, trained on our dataset instead. Note that, for the supervised baseline, we create weak supervision sentence pairs from the fusion and difference datasets by concatenating the two shorter sentences, as opposed to the original supervised SimCSE that uses human-written NLI pairs.\n\n### Goal of InterSent\nThe goal of this work is very different from previous sentence embedding methods. In addition to learning sentence embeddings that reflect semantic similarity via some distance measure, we aim to improve the interpretability of the sentence representation by supporting sentence operations including fusion, difference, compression, and reconstruction. We have shown in Table 1 that these desirable properties do not come with previous methods. Hence, it is necessary to introduce new model components and datasets to accomplish this goal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing in this paper is clear and engaging, and easy to follow.\n\n2. The results obtained in this study exhibit great strength, particularly in the context of 4 interpretable-related tasks. The improvements achieved are truly impressive.\n\n3. The inclusion of extensive analysis and ablation studies greatly enhances the reliability of this paper.", "reasons_to_reject": "1. In the experimental results section, the author evaluates the baseline models by equipping them with the same operator and decoder as the proposed InterSent model, and only optimizing the operator and decoder networks. While the author's motivation is to show that the sentence representation extracted by the InterSent encoder performs better than the baselines trained with different objectives, it may be unfair to the baselines for the following reasons:\n\n- The encoder of InterSent is jointly optimized, while all the baseline models are frozen(Correct me if I am wrong for this part). Since the evaluation is conducted on a generation task, regardless of the objectives equipped to the framework, of course the models that freeze the encoder will perform poorly compared to models that optimize both the encoder and decoder. This aspect weakens the persuasiveness of the results.\n\n- The model structure and training objectives are usually complementary. The specific bottleneck + operator + decoder structure in  InterSent may not necessarily be generalizable or suitable for other baselines.\n\n2. In lines 310-322, it would be beneficial to provide more details about the supervised contrastive and unsupervised contrastive baselines. This section is somewhat confusing, as it is unclear whether the unsupervised contrastive baseline refers to unsp. SimCSE and the supervised contrastive baseline refers to sup. SimCSE, both trained on the proposed compositional dataset.\n\n3. A critical disadvantage of this work is that all the baselines only consider the encoders to achieve their goals, so all the optimizating strategies only focus on their encoders. However, in InterSent, additional decoder, operator network, and operation-related data are required. ", "questions_to_authors": "See the weakness part, correct me if I made some mistakes.", "paper_topic_and_main_contributions": "In this paper, the author claims that in addition to sentence similarity, compositional operations such as fusion and difference can also represent sentence semantics. Therefore, the paper proposes InterSent, an end-to-end framework for learning sentence representations that supports these compositional operations in embeddings. Specifically, the paper introduces a contrastive objective and a generative objective to ensure that the model captures the compositional operations while preserving the ability to compare similarity. \n\nUnlike traditional Seq2Seq models that attend the decoding tokens to all encoded tokens, InterSent adopts a bottlenet Model that attends the decoding tokens to a single representation extracted by an encoder. This forces the encoded sentence representation to cover as much information as possible. The contrastive objective then pushes the encoded embedding close to the embedding of the sentence processed by the compositional operations. Additionally, the generative objective ensures that the target embedding can be decoded by the sentence embedding. \n\nThe proposed InterSent is evaluated on various tasks. The results demonstrate that InterSent outperforms all the baselines by a significant margin. The paper also includes extensive analysis and ablation studies to further validate the effectiveness of InterSent.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iRWBJZUG1o", "reviewer": "Reviewer_43Hp", "summary": "", "questions": "A.\tIn L268, the authors claim they conduct reconstruction evaluation using ParaNMT. But I don't find relevant results. Do I miss something?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Applications of InterSent\nThe tasks of sentence fusion/difference are more than just simple concatenation/deletion, as they usually involve adding connectives (which requires understanding of inter-sentence relations), manipulating clauses, coreference resolution, paraphrasing, etc. In addition to interpretability, InterSent also has value in practice, especially in scenarios where we need to manipulate sentence embeddings (e.g. retrieving summary, searching information related to multiple input sentences), as the resulting vector from a sentence operation can be computed very efficiently without the need to reconstruct all input sentences.\n\n### Evaluation Datasets\nWhile the evaluation datasets are automatically constructed to avoid expensive human annotations,  we would also like to point out that these are standard evaluation datasets used in previous works on sentence operations.\n\n### Question 1\nWe only use the ParaNMT as a noisy training set for sentence reconstruction, and do not directly evaluate sentence reconstruction as a downstream task since no sentence operator is associated with this task. However, the quality of sentence reconstruction is reflected in other sentence operation tasks as shown in Table 1, since all of them require reconstruction from sentence embeddings as the final step.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work pushes the sentence embedding further than semantic similarity. It inspires a new direction using sentence embedding to solve generative tasks.\n* The advantage of InterSent over previous sentence embedding methods on interpretability for the defined three operations is significant.\n* The case study is very useful. The results that fusion and difference are mutually inversed operations are impressive.\n* The paper is well-organized and easy to follow.\n", "reasons_to_reject": "* The definition of fusion and difference is simple in this paper. In the case study, I find the fusion operator almost just concatenates the two sentences, while the difference just removes sentence B. I do not see its value in practice, since for fusion, one can simply reconstruct the two sentences respectively and concatenate them. \n* The evaluation datasets are constructed automatically. Hence, it is unclear how InterSent performs in real natural language scenarios.\n* Though interesting, I still have no idea about the application of InterSent and don’t understand its relationship with the tasks listed in L68-L72. It would be much better to further elaborate this point.\n", "questions_to_authors": "A.\tIn L268, the authors claim they conduct reconstruction evaluation using ParaNMT. But I don't find relevant results. Do I miss something?", "paper_topic_and_main_contributions": "This paper proposes a novel sentence embedding InterSent both capturing semantic similarity and sentence operation. When applying fusion, difference, and compression operators to textual sentences, one can directly obtain the resulting sentence by manipulating embeddings of the input sentences and conducting reconstruction. For this purpose, the authors parameterize the sentence operation in embedding space by an MLP. During training, the input and output sentences are first fed into the encoder to get their embedding. Then they align the operated embedding with the output embedding by contrastive learning and use a decoder to reconstruct the output sentence from the operated embedding. Extensive experiments demonstrate InterSent significantly outperforms previous embedding methods on operation interpretability while keeping comparative on semantic similarity tasks.", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2aowlSRJzH", "reviewer": "Reviewer_KAiQ", "summary": "", "questions": "This is not exactly a question, but a more detailed qualitative analysis of successes and failure would be enlightening.\nyou provide essentially successful examples, analysis of failures would be an interesting add-on to this paper .", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Question 1\nWe appreciate this suggestion, and will accordingly add more examples in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper relies on a simple idea \n- The idea is well motivated by the experiments\n- The results are state of the art and sometimes provide a very strong improvement\n- The results are reported on a wide range of datasets\n", "reasons_to_reject": "- nothing specific", "questions_to_authors": "This is not exactly a question, but a more detailed qualitative analysis of successes and failure would be enlightening.\nyou provide essentially successful examples, analysis of failures would be an interesting add-on to this paper .", "paper_topic_and_main_contributions": "The paper describes a method for computing sentence embeddings\nthat aims to be interpretable and that supports compositional operations.\n\nthe operations at stake in the paper are \n - sentence fusion\n - sentence difference\n - sentence compression\n - sentence reconstruction\n\nIn practice the authors encode sentence with a transformer encoder (BERT) and take  the [CLS] token embedding \n  as sentence embedding. They compose sentence embeddings for 2 sentences by taking both embeddings and passing them through a two layer MLP.\n\nThe loss is contrastive like in previous work but they also add a generative objective that is shown to be highly relevant for interpretability and system performance.\n\nThe paper carries a large number of experiments on benchmark datasets and rely on strong encoder and decoders : BART,ROBERTA and T5. On interpretability datasets the authors report strong improvements over the baselines and strong performances on semantic textual similarity and on sentence retrieval and passage retrieval.\n\nThe discussion of the paper motivates the relevance of using two objectives and of using an MLP for composition rather than explicit arithmetic operators. A qualitative illustration of the system output is also provided\n\n\n\n\n\n\n\n ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a sentence embedding technique that considers the operability and compositionality of sentences, computes semantic similarity, and offers easy interpretability. The research direction is intriguing, the writing is clear, and the experiments are extensive. We hope that the camera-ready version will address areas the reviewers found unclear or lacking, ensuring the manuscript is valuable for a wide range of readers.
Decision: Accept-Main