Unifying Discrete and Continuous Representations for Unsupervised Paraphrase Generation
Authors: Mingfeng Xue
Dayiheng Liu
Wenqiang Lei
Jie Fu
Jian Lan
Mei Li
Baosong Yang
Jun Xie
Yidan Zhang
Dezhong Peng
Jiancheng Lv
Venue: EMNLP 2023 Main (2023)
Keywords: unsupervised paraphrase generation
discrete variables
VQ-VAE
entity
TL;DR: An unsupervised paraphrase generation method that encodes the sentence meaning and the entities with discrete and continuous variables to generate structurally dissimilar paraphrases with accurate entities.

Abstract: Unsupervised paraphrase generation is a challenging task that benefits a variety of downstream NLP applications.
Current unsupervised methods for paraphrase generation typically employ round-trip translation or denoising, which require translation corpus and result in paraphrases overly similar to the original sentences in surface structure.
Most of these methods lack explicit control over the similarity between the original and generated sentences, and the entities are also less correctly kept.
To obviate the reliance on translation data and prompt greater variations in surface structure, we propose a self-supervised pseudo-data construction method that generates diverse pseudo-paraphrases in distinct surface structures for a given sentence.
To control the similarity and generate accurate entities, we propose an unsupervised paraphrasing model that encodes the sentence meaning and the entities with discrete and continuous variables, respectively.
The similarity can be controlled by sampling discrete variables and the entities are kept substantially accurate due to the specific modeling of entities using continuous variables.
Experimental results on two benchmark datasets demonstrate the advantages of our pseudo-data construction method compared to round-trip translation, and the superiority of our paraphrasing model over the state-of-the-art unsupervised methods.

Unifying Discrete and Continuous Representations for Unsupervised Paraphrase Generation
Mingfeng Xue Dayiheng Liu" Wenqiang Lei*
Jie Fu Jian Lan
Mei Li
Baosong Yang
Jun Xie Yidan Zhang Dezhong Peng Jiancheng Lv
College of Computer Science, Sichuan University
Engineering Research Center of Machine Learning and Industry Intelligence mi Lingfengxue@stur scu. edu.cn onsinuriségmall com
Abstract
2017: Buck et al., 2018: Zhang al., 2022), doc- ument modeling (Zhang et al., 2021), and oth- ers. Supervised paraphrase generation methods re- quire large-scale, manually annotated paraphrase datasets, which are labor-intensive to build. This makes unsupervised paraphrasing emerging re- search direction.
Unsupervised paraphrase generation is chal- lenging task that benefits variety of down- stream NLP applications. Current unsuper- vised methods for paraphrase generation typ- ically employ round-trip translation or denois- ing, which require translation corpus and sult in paraphrases overly similar to the orig- inal sentences in surface structure. Most of these methods lack explicit control over the similarity between the original and generated sentences, and the entities also less cor- rectly kept. To obviate the reliance on transla- tion data and prompt greater variations sur- face structure, propose self-supervised pseudo-data construction method that gen- erates diverse geedo-paraphrasse in distinct surface structures for given sentence. To con- trol the similarity and generate accurate enti- ties, we propose an unsupervised paraphrasing model that encodes the sentence meaning and the entities with discrete and continuous vari- ables, respectively. The similarity can be trolled by sampling discrete variables and the entities are kept substantially accurate due to the specific modeling of entities using contin- uous variables. Experimental results on two benchmark datasets demonstrate the advan- tages of our pseudo-data construction method compared to round-trip trip translation, and the su- periority of paraphrasing model over the state-of-the of-tte-arttt unsupervised methods.
Existing unsupervised paraphrase generation methods primarily employ round-trip transla- tion (Mallinson et al., 2017; Wieting and Gimpel, 2018; Guo et al., 2021) or denoising (Hegde and Patil, 2020; Guo et al., 2021; Niu et al., 2021), which leads to four flaws. Firstly, round-trip translation requires extra translation corpus. Sec- ondly, the generated paraphrases similar to the original sentences in surface structure, faithful round-trip translation and denoising tend to main- tain the syntax, sentence pattern, and tense of the original sentence. Thirdly, most existing methods cannot explicitly control the similarity of the gen- erated paraphrases to the original sentences. Ex- plicitly controlling the similarity allows for greater variations in surface structure and the generation of diverse paraphrases meets the requirements of different downstream tasks. Lastly, the existing methods all ignore the importance of generating accurate entities, while entity errors usually lead to serious factual errors (e.g. paraphrasing "Jobs created Apple" to "Gates created Apple").
To produce paraphrases with great variation in surface structure without translation data, pro- pose a self-supervised pseudo-data construction method. Also, enable explicit control of simi- larity and accurate generation of entities, we pro- pose a paraphrasing model which unifies discrete and continuous variables to encode sentences.'
1 Introduction
The task of paraphrase generation aims to re- construct a sentence in different surface struc- ture (grammar and words) while preserving its underlying semantics. It plays essential role in many downstream tasks like text summariza- tion (Cao et al., 2017; Zhao al., 2018), ma- chine translation (Zhou et al., 2019; Thompson and Post, 2020), question answering (Dong et al.,
The pseudo-data constructor is trained on non- parallel sentences in self-supervised manner, eliminating the need for translation data. From
'The discrete and continuous variables here are hoth dense representations in neural networks.
*Coreesponding author.

---

paraptiraes
Score, which performs an evaluation of the preci- sion and recall of the entities and correlates sig- nificantly better with human evaluation than the existing metrics on the entity.
Tom wishes to buy Mervodes Benz
likes buy Meroades Benz.
[PEC] wishes to buy [PRO].
[PER] Tom
[PER] likes buy [PRO].
[PRO] Mercedes Benz
Discrete Vartables
Contimions
Varlables
2
Related Works
Tom wants Mersedes Benz.
Unsupervised paraphrase generation methods of- ten employ translation or denoising. Mallinson et al. (2017) and Guo et al. (2021) employ a straightforward round-trip translation approach. Guo et al. (2021) propose hybrid decoder taking the hidden states of set-to-sequence model (Vinyals al., 2016) and a round-trip trans- lation model. Hu et al. (2019a) and Hu et al. (2019b) expand translation datasets into para- phrase datasets with lexically constrained decod- ing (Hokamp and Liu, 2017; Post and Vilar, 2018). Principled Paraphrasing introduces an adversarial reconstruction decoder to encourage the removal of structural information from the original sen- tence, followed by a translation task to ensure the retention of semanties (Ormazabal et al., 2022). These methods rely on translation datasets and of- ten produce paraphrases that resemble the origi- nal sentences. As for denoising, Liu et al. (2020) model paraphrase generation as an optimization problem and design sophisticated objective func- tion to enhance semantic preservation, expression diversity, and fluency of paraphrases, failing ex- plicitly control the similarity generate diverse paraphrases. Niu et al. (2021) propose Dynamic Blocking to enforce a surface structure dissimilar from the input, which easily leads to key phrases being blocked. Roy and Grangier (2019) pro- pose to learn paraphrasing models from unlabeled monolingual corpus with VQ-VAE (van den Oord et al., 2017). Yet discrete variables are expres- sively less powerful than continuous variables (Liu et al., 2021), and is difficult to encode large quantities of entities with a limited number of dis- crete variables in a codebook.
onginal sentence
Figure 1: The paraphrasing model unifies discrete and continuous variables to represent the complete seman- ties. The "[PER]" and "[PRO]" mean the category "per- son" and "product" of the entities, respectively.
the candidates generated by the constructor, se- mantically similar and structurally dissimilar para- phrases are selected to train the paraphrasing model, resulting in greater variations in surface structure. The paraphrasing model is inspired by the ideas of applying VQ-VAE to unsupervised paraphrasing (Roy and Grangier, 2019) and the separate modeling of the sentence meaning and the entities (Gu et al., 2016). The model employs discrete variables to encode the sentence mean- ing and continuous variables to model the entities. As an example, the sentence "Tom wants to buy a Mercedes Benz" could be split into the meaning "someone wants to buy a product" and the enti- ties "someone: Tom, product: Mercedes Benz" as illustrated Figure 1. The similarity can be con- trolled by sampling discrete variables at different distances from the original representation. The en- tities are kept substantially accurate with the spe- cific modeling using continuous variables.
We evaluate our method two benchmark datasets, WikiAnswers (Fader et al., 2013) and QQP?. The automatic evaluations indicate that our pseudo-data construction method surpasses round- trip translation and encourages more varied sur- face structures. The automatic and human evalu- ations demonstrate that our paraphrase generation method outperforms the state-of-the unsuper- vised methods semantic preservation, surface structure variation, and entity retention.
We present self-supervised Pseudo-data Con- structor that generates paraphrase-lis data for training the paraphrasing model, which eliminates the need for the translation corpus. And pro- pose a paraphrasing model generate diverse paraphrases with controlled similarity through sampling discrete variables and generate accurate entities through specific modeling entities with continuous variables.
Besides, commonly used metries such as BLEU (Papineni et al., 2002) and Self-BLEU do not perform a fair evaluation of entity accuracy. To more effectively evaluate the accuracy of gener- ated entities, we introduce a metric called Entity-
hhttps: //www. kaggle. com/ com/datasets/quoraa question- waatinn-ppiis--ttaeeea dataset.

---

D Train Constructor
BERT Store
Self BLEU
2) Consrrust training data for Paraphrosing Manke
wants buy
He wants Tom to buy a eccedes Benz
0,55
0.58
x
[PRO] shit [PER] wants buy:
[PER] Tem
Meccedes Benz
Tom wants to buy Meccaales Beez
1,00
1,00
[PRO] Meroodes Denz
oriinnll sentenre
Mercedes Pauz what Tom wants buy.
0,87
0.43
pivot sumtenne
original entities
Infermnce
A [PRO] what [PFR) wants huy. [sepl] [PER] Tom (PRO) Menoedes Bens. cuncalenated ixpul
Paeudo-Dela Constructor
Switss is what Lee wants to bug. ganembted parppirase
wents, Tom, buy, Mercedes Benz
Pamplimiig Model
Tom, wants, buy
Tom, wants, Merrades Herze
Truin Paruphnusing Model
Merrodes Benzi
Merrodas Denz, Tom, wants, bug shaffled keyworts
[5] [PER] wants to buy a [PRO). [eepl [PER] Len [PRO) Switcsh
keywords
Paruplmsing Mudel
Tom wants bug Miercedes Benz original sontence
scants to Swatcch
Tom wants buy Morredas Benz onginal semtrnoe
Impor
Figure 2: Overview of our approach. The dashed lines with arrow indicate inferencing.
3 Approach
input within the training. We filter the candidates using BERTScore (Zhang et al., 2020) and Self- BLEU because not all keywords orders yield valid paraphrases. Then each selected candidate and the original sentence form an input-target pair.
We describe the proposed approach in this section. Besides, there are some efforts that show similari- ties to our work, we give detailed comparisons be- tween them and our method in Appendix A.
To separate modeling the sentence meaning and entities for explicitly controlling the similarity without destroying entities, we further split the meaning and the entities of the input sentence. We replace the entities in the input sentence with the entity categories (c.g. replace "Mercedes Benz" with "[PRO]") to obtain the entity-replaced pivot sentence and concatenate several special tokens, the pivot sentence, special separator, and the cor- rect entities from the target sentence to serve as the input to the paraphrasing model.²
3.1 Overall Training Procedure
The overview of our approach is depicted in Fig- ure 2. The training is conducted in three steps.
1) Train pseudo-data constructor: Traditional construction methods which rely on round-trip translation introduce two lim- itations: the need for the translation corpus and domain gaps between the translation corpus and the paraphrase corpus. To remove these limita- tions, we train pseudo-data constructor in self- supervised manner on the non-parallel paraphrase data. As shown in step I of Figure 2, we remove the stop words from a sentence and split the re- maining words into words and noun chunks as the input keywords. The constructor is trained by re- constructing the original sentence with ordered in- put keywords.
3) Train paraphrasing model. The paraphras- ing model is able to produce paraphrases with lexi- cal variations and accurate entities. The model uti- lizes discrete variables to encode sentence mean- ings and stores these variables codebook. The close variables in the codebook tend to express dif- ferent words with similar semantics and we sam- plc the discrete variables to bring lexical variation. The entities are specifically modeled with contin- uous variables, which ensures the accuracy of the generated entities. During training, the paraphras- ing model takes the concatenated sequence con- structed in step as input and outputs the original sentence. We train the paraphrasing model in a standard sequence-to-sequeece manner.
2) Construet training data for paraphrasing model. As depicted in step 2 of Figure 2, we generate paraphrase candidates with varied word orders using the constructor and filter the candi- dates for semantically similar but structurally dis- similar pseudo-paraphrasn paraphrases to train the paraphras- ing model. This encourages variations in word or- der while preserving semantics. Specifically, shuffle the keywords of sentence multiple times and feed them into the constructor. The construc- tor outputs several candidates corresponding to the orders of the keywords, as the order of the key- words in the target sentence is consistent with the
'In this study, we use Spacy (https://spag) io, v3.4, "en core core_weheehheeeeeeeeee web trf" model) as the NER tool.

---

output
Arran) Arom Aprny Retercsato Auanz
sentence I. The cross attention the encoder out- put allows the model to automatically learn when to focus on discrete variables that represent the sentence meaning and when to focus on contin- uous variables that represent the entities.
codebookk
drup
keep contimaous
pivat sontence
I of original entities
(44.Ag)
Upreat Ra Apesol
Piraan Prom A
Self-A llention Fead Forward
3.3 Inference Step
In inference, given a sentence x; the entities a² are recognized from z using a NER tool and replaced with entity categories to form the pivot sentence x The special tokens, the pivot sentence, the separator token, and the correct entities sequence are concatenated, similar to the training step 2, form the input è = {s; ; [sse];; a²) Different from training. each h quantized to h3 by sam- pling a vector from the codebook according to the probability:
[ER] wants buy [PRO]. [sep) [PER] Tom [PRO] Mertedes Benz
compalemabed
Input
Encoder architecture of the paraphrasing model. "s;" denotes special token be quantized, in this figure take two special tokens as example.
Figure 3:
3.2 Paraphrasing Model
Given an original sentence x = [wi, w2,..., wi) and its paraphrase generated by the pseudo-data constructor, we construct the input as described in step 2 in Section 3.1. Following Roy and Grangier (2019), we quantize the representation of special tokens added at the beginning of the input to en- code the sentence meaning. We denote the special tokens with = {$1, }m . the entity-replaced pivot sentence with y the special separator with [sep]. and the entities æ with at = The input to the paraphrasing model would be: -
p(h)) Softmax(-"") ".
where d [d1, d2.....dk) means the distances of h3 to each vector c, in the codebook, d; I - =ill: and r is the sampling temperature that we set manually to control the similarity of the generated and original sentences. As the temper- ature increases, it becomes more likely to sample discrete variables that are farther away from h leading greater dissimilarity. Finally, the dis- crete and continuous variables are fed into the de- coder to generate the paraphrase autoregressively.
The paraphrasing model is an encoder-decoder architecture that features a codebook containing vectors. As shown in Figure 3, the encoder is a transformer encoder with a quantization pro- cess that converts continuous variables into dis- crete variables. The input å is fed into N-layer self-attention blocks, and the last block outputs the hidden states h {hi; ..., he h2 hhlspl, ha²) of each token. The output h {hi..... h.,) of the special tokens are quantized as:
4 Experiments
4.1 Datasets
We evaluate our approach on two benchmark datasets, WikiAnswers (Fader et al., 2013) and QQP'. The WikiAnswers dataset contains 18,000k question-paraphe paraphrase pairs scraped from the WikiAnswers website. The QQP dataset com- prises over 400k lines of potential question du- plicate pairs. We adopt the unsupervised setting as Guo al. (2021). For WikiAnswers, we ran- domly select 500k non-parallel sentences, 3k par- allel paraphrases, and 20k parallel paraphrases for training, validation, and testing. For QQP, we ran- domly select 400k non-parallel sentences, 3k par- allel paraphrases, and 20k parallel paraphrases for training, validation, and testing. The statistics of the processed datasets are in Appendix B. In the
h1 ck. where & arg min [i - q: 1|
where ai denotes the i-th vector in the code- book. The hidden states of the pivot sen- tence y are dropped, while the hidden states ....,
he = (hq""..., Ag of the entities at are re- tained and output as continuous variables with- out being quantized. We concatenate the discrete variables and the continuous variables. i.e. hene (h?...., A/ h4 ...., h..,,""" }. as the final output (also referred to as memory) of the encoder.
A transformer decoder serves as our decoder, which takes enc as input and outputs the original
phttps://wwal kaggle. wwwpggle..cmm/aaaee htm//daaasets/quurrll question wawoooo-ppii-dddaaaa dataset.

---

pseudo-data constructing step, we pick the candi- dates with Self-BLEU below 60 and BERTScore above 85 as pseudo data.
rameters. We use the same pre-trained BART- base checkpoint in the constructor to initialize the paraphrasing model except for the codebook. The codebook contains 32,768 vectors of dimen- sion 768. To initialize these vectors, we train the paraphrasing model in a completely continu- ous style, skipping the quantization, for 4 epochs. Then we feed partial training data into the encoder and cluster the hidden states of the special token into 32,768 classes using k-means to assign val- ues the vectors in the codebook. In quantiza- tion, the hidden states of special tokens are quan- tized separately with the same codebook. Follow- ing van den Oord et al. (2017), the model is trained with straight-hhrough gradient estimation and ex- ponentiated moving averages. The paraphrasing models trained the corresponding datasets with 10 epochs a single 32G Tesla V100 GPU for about 60 minutes per epoch.
4.2 Baselines
We compare our method with four unsupervised paraphrase generation methods, including round- trip translation, RRT (Guo et al., 2021), TA+SS+DB (Niu et al., 2021), and Principled Paraphrasing (Ormazabal al., 2022). While some of the baseline methods employ parallel translation corpus, we consider them as unsuper- vised baselines for comparison since they do not rely on parallel paraphrase data.
Set2seqRRTT trains a set2seq (Vinyals et al., 2016) model and pair of round-trip translation models separately. In generating, a hybrid decoder takes a weighted sum of the hidden states from the set2seq model and the translation model as the probability of the next token. TA+SS+DB applies Dynamic-Blocking to generate paraphrases with no identical n-grams to the input. We note that we prevent the blocking of entities in our repli- cation to generate more accurate entities. Princi- pled Paraphrasing introduces an additional adver- sarial reconstruction decoder to remove much structural information from the original sentence possible. Subsequently, traditional transla- tion loss is employed to preserve the semantic in- formation. This method allows for the control of similarity to the original sentence by adjusting the weights of the adversarial loss and translation loss during training. These aforementioned methods represent the current state-of-the-an in unsuper- vised paraphrase generation. By comparing with these approaches, we aim to demonstrate the periority of our approach in preserving semantics and entities while altering surface structures.
4.4 Evaluation Metrics
4,4.1 Automatic Evaluation
Recent works on paraphrase generation typically take BLEU (Papineni et al., 2002), Self-BLEU, and iBLEU (Sun and Zhou, 2012) as evaluation metrics. Niu et al. (2021) propose BERT-iBLEU which leverages BERTScore and Self-BLEU to measure the semantics and surface structure. Fol- lowing their works, We take BLEU, Self-BLEU, iBLEU, and BERT-iBLEU" the metries.
Existing evaluation metrics fail to perform a fair evaluation of entities. To address this limitation, we propose Entity-Score, an Fl-score-like evalua- tion method that performs a holistic evaluation of the precision and recall of the generated entities. Entity- calculates precision by determining whether the entity in the paraphrase appears in the input, and recall by assessing whether the entity in the input present in the paraphrase. The fi- nal Entity-Scome Score is then computed as the Fl score based on precision and recall. A more detailed de- scription of the Entity-Score is in Appendix C. We adopt Entity-Score as an evaluation metric.
For reference, we also present the results of two supervised methods, Binntfinetune (Lewis et al., 2020) and DNPG (Li et al., 2019), and large lan- guage model, GPT-3.5-turbe in Appendix D.
4.3 Training Details
For the pseudo-data constructor, we finetune BART (Lewis et al., 2020) model from Hugging- face's checkpoint on the non- non-parallel training set. The paraphrasing model contains 267M parame- ters, including a codebook that contains 50M pa-
4.4.2 Human Evaluation
We randomly select 50 examples containing en- tities from QQP, and three annotators are asked to rate the paraphrases generated by our method
'nttps://platfore wwt.:/aatoorrooooniii...mm http://mppppnee..oocc//////eeeee gpt-3-5.
FFllow Niu et al. (2021). we use "roberta- large_ 1o7_no- versioon=0..../0uuttaaaa tranus2_rrnn223.0))) to calculate BERTScore.

---

Method
BLEUt
Self-BLEUL
iBLEUT
BERT-iBLEUT
Entity-Score?*
round-trip translation
27.85
55.66
19.50
75.49
49.67
TA+SS+DB
30.56
47.29
22.78
81.65
67.83
R
33.82
48.76
25.56
80,46
69.91
Principled Paraphrasing"
33.69
55.53
24.77
77.31
56.04
Principled Paraphrasing"
31.25
44.18
23.71
82.01
49.00
Ours(r=5)
38.57
48.52
29.86
81.16
85.10
Ours(7=11)
34.17
35.78
27.18
85.32
81.27
Ours(r=12)
31.59
32.11
25.22
86.15
80.79
Table 1: Evaluation results WikiAnswers dataset. means the sampling temperature in quantization. Principled Paraphrasing" uses À = 0.85 and K' = 0.70. Principled Paraphrasing" uses À 0.80 and K = 0.70. And À and K are ayper-parameters in Ormazabal et al. (2022). The best and second best results are indicated in bold and underlined. respectively.
Method
BLEUt
Self-BLEU,
iBLEUT BERT-iBLEUT
Entity-Score?
round-trip translation
23.25
68.12
14.11
67.56
80.76
TA+SS+DB
14.35
36.19
9.30
84.83
74.80
Ret2se+RRTT
22.41
51.78
14.99
80.44
74.17
Principled Paraphrusing"
24.53
73.39
14.74
63.39
55.04
Principled Paraphrasing'
15.60
35.81
10.46
85.78
34.18
Ours(r=5)
25.58
62.78
16.74
73.51
95.35
Ours(r=10)
23.80
57.82
15.64
76.80
94.58
Ours(r=18)
14.22
33.37
9.46
86.44
91.44
Table 2: Evaluation results on QQP dataset. Principled Paraphrasing" uses À 0.90 and K = 0.70. Principled Paraphrasing @ uses À = 0.73 and K = 0.70.
- and 10) and three strong baselines ac- cording the semantics, structure, and entity scale of to 3, the higher, the better. The seman- ties measures the degree of preservation of seman- tics. The structure evaluates the extent of variation surface structure. The entity measures the ac- curacy and completeness of the generated entities. The details of the human evaluation settings are given in Appendix F.
cate that our proposed method exhibits superior performance in BLEU, iBLEU, and Entity-Scom when 7=11. Furthermore, when 7=12, our method outperforms the baseline methods Belf-BLEU, BERT-iBLEU, and Entity-Score. Similar trends can observed the results displayed in Table
2) Our approach showcases exceptional proficiency in generating accurate entities. Our method consistently achieves significantly higher Entity-Score compared to other methods, and the Entity-Score remains high as the tempera- ture rises, which demonstrates the effectiveness of our method in preserving entities.
4.5 Evaluation Results
4.5.1
Automatic Evaluation Results
We select the models that perform best the dev set and report their performances on the test set. Table I and Table present the results WikiAn- swers and QQP. Several generated examples are provided in Appendix H. The following conclu- sions can be drawn from the results:
3) The similarity between the generated paraphrases and the original sentences in sur- face structure can be explicitly controlled by adjusting the sampling temperature during quantization. At low sampling temperatures, our method maintains good semantics, scoring high in BLEU and Self-BLEU. While at high pling temperatures, our method generates para- phrases with greater variation in surface structure and scores well Belf-BLEU. The detailed eval- uation results on different sampling temperatures can found in Appendix E.
1) Our approach outperforms the state-of- the-art unsupervised methods existing met- ries at appropriate temperatures. To ensure fair and unbiased comparison, we carefully select sampling temperatures that generate paraphrases with comparable BLEU scores to the baseline methods. The results presented in Table 1 indi-

---

Method
BLEUT
Self-BLEU,
iBLEU1
BERT-BLEUU
Sctrry-Scoret
Round-trip Translation
27.85
$5.66
19.50
75.49
49.67
WikiAnswers
Pseudo-data Constructor
32.08
28.20
26.05
88.84
77.12
Round-trip Translation
23.25
68.12
14.11
67.56
80.76
QQP
Pscudo-data Constructor
17.35
34.11
12.20
88.12
85.25
Table 3: Experimental results of different pseudo-data construction methods.
Semantic
Structure
Entity
4.6
Analyses
Se12scq=RTT
2.37
2.03
2.67
4.6.1
Pseudo-data Constructor
TA+SS+DB
2.11
2.26
2.52
To compare the performance our Pscudo-data Constructor and Round-trip Translation in con- structing pseudo-data, we apply the Pseudo-data Constructor and Round-trip Translation directly the test sets of WikiAnswers and QQP datasets. For both the Constructor and Round-trip trans- lation we generate 5 candidates for each sen- tence and then select the best one as the pseudo- paraphrase based on BERT-iBLEU. We employ nucleus sampling (Holtzman et al., 2020) with p=0.95 to generate multiple candidates in transla- tion because the translation models do not feature diverse generation.
Principled"
2.30
1.89
2.06
Principled
1.95
2.12
1.75
Ours(r=5)
2.77
2.10
2.97
Ours(r=10) 2.55
2.41
2.89
Agreement 0.45
0.46
0.87
Table 4: The results of human evaluation. "Principled" means Principled Paraphrasing. Principled uses À = 0.90 and K 0.70. Principled uses À 0.73 and K = 0.70. We use Fleiss' Kappa (Fleiss, 1971) to measure the inter-rater agreement. The scores between 0.40-0.60 mean "moderate agreement", and the scores above 0.80 mean "very good agreement".
As shown in Table 3, the Pseudo-data Construc- outperforms Round-urip Translation in most metrics. Particularly, the Pseudo-date Constructor far outperforms Round-trip Translation on Self- BLEU, which demonstrates that the Pseudo-data Constructor is more likely to generate pseudo- data with greater variation in surface structure. Notably, the accuracy of the entities generated by Round-trip Translation on the two different datasets varies substantially, and our constructor generates entities with higher accuracy on both datasets. This might be related the domain of the training data for the translation models.
Metric
Correlation Value
BLEU
0.257 (p 0.01)
Self-BLEU
0.256 (p 0.01)
iBLEU
0.185 (p < 0.01)
BERT-iBLEU
-0.152 (p < 0.01)
Entity-Score
0.579 (p < 0.01)
Table Spearman's 's correlation between the automatic evaluation and the human evaluation entity.
4.5.2 Human Evaluation Results
4.6.2 Effectiveness of Entity-Score
To verify the effectiveness of Entity-Score, we compute the Spearman's 's correlation between the automatic evaluation scores and the human eval- uation score of the entity. As shown Table 5, Entity-Score correlates clearly better with human evaluation than existing metrics in evaluating gen- erated entities. More details about the correlation given in Appendix G.
Table 4 shows the results of human evaluation on QQP. Our method with 7=10 outperforms other baselines in three metrics and our method with T=5 shows significant superiority over other base- lines in semantics and entities. As the sampling temperature rises, the variation in the paraphrases generated by method increases. Furthermore, method obtains significantly higher scores in entity, which highlights the superiority of our method in generating accurate entities. The results the human evaluation are compatible with the results of the automatic evaluation.
WWe use the translation models on Huggingface's check- point tpt: wtw./guggggcc.eeeeeeceee/aa co/f hfo/faabooo//// wmt 19-en-de https: /ttgs//ggfggg/aacc co/facebook///e/oooooooooooooo/a wmt 19-de-en). "BERT-BBLEU is a composite metric calculated by BERTScore and Self-BLEU, which makes the filtering cri- teria here coincide with Section 3.1.

---

Method
BLEUt
Self-BLEU4
iBLEUT
BERT-iBLEU?
Entity-Scoree
round-trip translation
23.25
68.12
14.11
67.56
80.76
+L.CD
23.39()0114)
71.06(294)
13.95(a.16)
64.85(271)
83.82(e3.06)
Principled Paraphrasing
24.53
73.39
14.74
63.39
55.04
+LCD
25.18
76.24(-2.85)
13.04(40.)
(3.13
72.92
Ours (Discrete)
24.12
59.01
15.81
74.62
60.76
+LCD
(8.78(.5.44)
60.35
(0.87(4.94)
73.04(-1.58)
86.57
+Continuous
25.58(+1446)
62.78(.3.77)
16.74
73.51,-1.11)
95.35(-34.59)
Table 6: Results of methods combined with lexically constrained decoding (L.CD). Discrete in method means that only discrete variables are used in inference. The results in brackets indicate the variation in performance after combining the LCD and continuous variables, with "+" indicating an improvement and indicating a decline.
Model
BLEU
Self-BLEU.
iBLEU
BERT-iBLEU
Entity-Scoref
Ours (r=5)
25.58
62.78
16.74
73.51
95.35
w/o pseudo data
23.36
68.17
14.21
68.95
94.98
w/o entity
23.85
57.45
15.72
74.94
59.55
w/o continuous
24.12
59.01
15.81
74.62
60.67
Table 7: Evaluation results ablation study. The sampling temperature is set to 5. Without pseudo data means we
train the paraphrasing model in a denoising manner without constructing pscudo data. Without entity means that do not feed the entities to the encoder and the decoder only takes the quantized hidden states of special tokens. Without continuous means that we quantize the hidden states of special tokens and entities.
4.6.3
Comparison with Lexically Constrained Decoding
data constructor increases the variation in surface structure as "w/o pseudo data" score higher Self-BLEU. Taking entities as part of the input to the paraphrasing model and keeping the hidden states of entities as continuous variables are cru- cial to generate accurate entities, since "w/o en- tity" and "w/o continuous" drop a lot in Entity- Score. In our approach, various modules play critical roles, cooperating to generate semantically similar, strracc-structuralll distinct paraphrases that contain accurate and complete entities in an unsupervised setting.
Our paraphrasing model encodes entities individu- ally using continuous variables to improve the ac- curacy of the generated entities. Lexically con- strained decoding (LCD) on entities offers the po- tential achieve this as well. On the QQP dataset, we analyze the impact of LCD on our method (only using discrete variables) and two baselines that obtain near BLEU scores with our method, to demonstrate the superiority of continuous en- coding compared to LCD. Table 6 shows the ex- perimental results. It can be that LCD im-
Conclusion
proves the entity accuracy on all methods, but our method still far outperforms the others. In our ap- proach, the use of continuous variables is more fective than LCD, and LCD causes a reduction in the other metrics. The results demonstrate the va- lidity of our use of continuous variables to encode entities individually.
In this paper, propose self-supervised pseudo-data construction method that generates paraphrases with more variation in surface struc- ture than round-trip translation without the need for translation data. We propose a paraphrasing model that employs discrete and continuous vari- ables to represent sentence meaning and entity information respectively for unsupervised para- phrase generation. Our approach achieves state- of-the-art performance on two benchmark para- phrase datasets and far outperforms other meth- ods terms of accuracy in generating entities. In addition, we proposed an evaluation metric called Entity-Score for evaluating the precision and recall
4.6.4 Ablation Study
To better observe what roles the different mod- ules play in our approach, remove one module time from our approach and train paraphras- ing models on the non non-parallel training set. Ta- ble 7 shows the performance of our approach on QQP after removing different modules. Pseudo-

---

of the generated entities. Entity-Score correlates better with the human evaluation of entity than ex- isting metrics. To the best of our knowledge, this is the first work concerned with ensuring the qual- ity of the generated entities in the paraphrase gen- eration task. The framework encoding text with discrete and continuous variables is promising on many NLP tasks, c.g. summarization and trans- lation. Future work will focus extending our model more general seq2seq framework.
affect our experimental conclusions.
Ethies Statement
We declare that all of the co-authors of this paper are aware of and honor the ACL Code of Ethics. In this paper, propose effective pseudo-data construction method, a paraphrasing model, and an evaluation metric on entities. Our contributions lie in the methodological aspect and do not bring any ethical issues. However, The publicly released datasets and pre-trained models used in this pa- per may contain some negative effects, such as gender and religious bias. We recommend other researchers stay careful when using methods with these datasets and models.
Limitations
To the best of our knowledge, our approach has the following two limitations. First, the paraphras- ing model has great advantages when dealing with sentences containing entities but will reduce to the traditional VQ-VAE when dealing with sentences that do not contain entities. Second, our approach employs additional named entity recognition tool. The performance of the named entity recog- nition tool we use will directly affect the perfor- mance of our approach.
We conduct a human evaluation to rate the gen- erated results, and in doing so, we ensure that no personal information any of the annotators is involved in this work. We reviewed the samples to be rated before the human evaluation to make sure there are negative statements. All the an- notators were informed in advance of the potential risks of offensive statements generated by artificial intelligence.
Regarding the Entity-Score, there are two lim- itations that we are currently aware of. First, like in our upproach, the process of caleulating the Lititt-Score requires the use of named en- tity recognition tool. Thus, the Entity-Score re- sults are directly influenced by the tool. Second, the paraphrase generation tasks, some entities can exist in different forms, such as "$2,000" and "two thousand dollars". Currently, we do not opti- mize the Entity-Score -core calculation process for these kinds of cases.
Acknowledgments
This work is supported by the Fundamental Re- search Funds for the Central Universities under Grant 1082204112364 and the Key Program of the National Science Foundation of China under Grant 6 61830006.
In terms of experiments, we only conduct exper- iments on two English question datasets, WikiAn- swers and QQP. We choose these two datasets because 1) they are two widely used benchmark datasets, 2) they contain large number of enti- ties to validate the performance of our proposed paraphrasing method and Entity-Score, and 3) follow the experimental settings with Niu al. (2021).
References
Christian Buck, Jannis Bulian, Massimiliano Cia- ramita, Wojeiech Gajewski, Andrea Gesmundo, Neil Houlsby, and Wei Wang. 2018. Ask the right ques- tions: Active question reformulation with reinforce- ment learning. In ICLR. OpenReview.net.
Ziqiang Cao. Chuwei Luo, Wenjie Li, and Sujian Li. 2017. Joint copying and restricted generation for paraphrase. In AAAI, pages 3152-3158, AAAI Press.
Regarding the choice of pre-trained model, we only experiment on BART. We focus on the prob- lem of accurate entity and controlled surface struc- ture similarity in paraphrasing tasks, with the key point being the coordination of discrete and con- tinuous variables. We choose BART simply be- cause is a commonly used seq2seq pre-trained model. We believe that using other pre-trained models, such as T5 (Raffel et al., 2020), would not
Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. EMNLP, pages 875-886 Association for Computational Linguistics.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In ACL (1), pages 1608-1618. The As- sociation for Computer Linguistics.

---

Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018, Hierarchical neural story generation. In ACL (1), pages 889-898. Association for Computational Lin- guistics.
Kuan-Hao Huang and Kai-Wei Chang. 2021. Generat- ing syntactically controlled paraphrases without us- ing annotated parallel pairs. In FACI,, pages 1022- 1033. Association for Computational Linguisties.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Ashutosh Kumar, Satwik Bhattamishra, Manik Bhan- dari, and Partha P. Talukdar. 2019. Submodular optimization-hased diverse paraphrasing and its ef- fectiveness in data augmentation. In NAACI-IILT (1), pages 3609-3619. Association for Computa- tional Linguisties.
Yao Fu, Yansong Feng, and John P. Cunningham. 2019. Paraphrase generation with latent bag of words. In NeurlPS, pages 13623-13634.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov,
Tanya Goyal and Greg Durrett. 2020. Neural syntac- tic preordering for controlled paraphrase generation. In ACI, pages 252- Association for Computa- tional Linguisties.
Luke 7ettlemoyer. 2020. BART: denoising sequence-to-sent so-euuenceeeeee pre- training for natural language generation, translation, and comprehension. In ACL, pages 7871-7880 As- sociation for Computational Linguistics.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequenee learning. In ACL (T). The As- sociation for Computer Linguistics.
Zichao Li, Xin Jiang, Lifeng Shang, and Qun Liu. 2019. Decomposable neural paraphrase generation. In ACI. pages 3403-3414 Association for Com- putational Linguistics.
Zilu Guo, Zhongqiang Huang, Kenny Q. Zhu, Guandan Chen, Kaibo Zhang, Boxing Chen, and Fei Huang. 2021. Automatically paraphrasing via sentence re- construction and round-trip translation. In IJCAI, pages 3815-3821. ijcai.org.
Dianbo Liu, Alex Lamb, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael C. Mozer, and Yoshua Bengio. 2021. Discrete- neural communica- tion. In NeurlPS, pages 2109-2121.
Chaitra V. Hegde and Shrikumar Patil. 2020. Unsuper- vised paraphrase generation using pre-trained lan- guage models. CoRR, 057477
Xianggen I Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song. 2020. Unsupervised para- phrasing by simulated annealing. In ACL, pages 302-312. Association for Computational Linguis- tics.
Chris Hokamp and Qun Liu. 2017. Lexically con- strained decoding for sequence generation using grid beam search. In ACL (I), pages 1535-1546. Asso- ciation for Computational Linguistics.
Jonathan Mallinson, Rico Sennrich, and Mirella Lap- ata. 2017. Paraphrasing revisited with neural chine translation. In EACL (1), pages 881-893. As- sociation for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR. OpenReview.net.
Tong Niu, Semih Yavuz, Yingbo Zhou, Nitish Shirish Keskar, Huan Wang, and Caiming Xiong. 2021. Un- supervised paraphrasing with pretrained language models. In EMNLP (1), pages 5136-514 5150. Asso- ciation for Computational Linguistics.
Tom Hosking and Mirella Lapata. 2021. Factorising meaning and form for intent-preserving paraphras- ing. In ACL/JJNLP (I), pages 1405-14188 Associ- ation for Computational Linguistics.
Tom Hosking, Hao Tang, and Mirella Lapata. 2022. Hierarchical sketch induction for paraphrase genera- tion. In ACL (7), pages 2489-2501. Association for Computational Linguistics.
Aitor Ormazabal, Mikel Artetxe, Aitor Soroa, Gorka Labaka, and Encko Agirr,. 2022. Principled para- phrase generation with parallel corpora. In ACL (1), pages 1621-1638. Association for Computational Linguistics.
J. Edward Hu, Rachel Rudinger, Matt Post, and Ben- jamin Van Durme. 2019a. PARABANK: monolin- gual bitext generation and sentential paraphrasing via Texically-coossrrannee neural machine translation. AAAI, pages 6521 6621-6 AAAI Press.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic eval- uation of machine translation. In ACL, pages 311- 318. ACI.
J. Edward Hu, Abhinav Singh, Nils Holzenberger, Matt Post, and Benjamin Van Durme. 2019b. Large- scale, diverse, paraphrastic bitexts via sampling and clustering. CoNLL, pages 44 54, Association for Computational Linguistics.
Matt Post and David Vilar. 2018. Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. In NAACL-HLT, pages 1314-1324. Association for Computational Linguis- tics.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 10::::::114:6..
A Methodological Comparisons
A.1 Wordset for Paraphrasing
Our proposed Pseudo-data Constructor uses key- words as imput and outputs the original sentence during training. In inference, we use the shuf- fled keywords as input which prompt the construc- tor to generate sentences that match the order of the keywords, which makes it possible for each sentence to correspond to multiple possible para- phrases to enrich the diversity of pseudo data. We additionally use BERT-iBLEU to filter data pairs with large semantic offsets too similar struc- tures to ensure the quality of the pseudo-data.
Aurko Roy and David Grangier. 2019. Unsupervised paraphrasing without translation. In ACL (1), pages 6033-6039. Association for Computational Linguis- tics.
Hong Sun and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In ACL (2), pages 38-42. The Association for Com- puter Linguisties.
Brian Thompson and Matt Post. 2020. Automatic ma- chine tramslation evaluation in many languages via zero-shot paraphrasing. In EMNLP (!), pages 90- 121. Association for Computational Linguistics.
There are some works that use the change of the wordset orders to paraphrase. Fu et al. (2019) first predict the neighbors of the words in the origi- nal sentence to form the bag of words then sample and reorganize the words into sentences. Huang and Chang (2021) and Guo et al. (2021) encode the input sentence using transformer encoder without positional encoding to ignore its surface structure. Reconstructing the original sentence using unordered keywords reduces the diversity of paraphrases because the same wordset always yields the same output regardless of the order of the words. Therefore, we train the constructor by preserving the order of keywords and infer- ence by shuffling the keywords obtain diverse paraphrase pseudo-data. Goyal and Durrett (2020) guide the decoder output syntactically meeting sentences by reordering the syntax of the input and appending the new syntactic position informa- tion to the encoder output. This approach requires complex rearrangement and evaluation of the syn- tax tree and uses it as additional input. Therefore, we choose simpler pseudo-data construction pro- of shuffling and filtering.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. In NIPS, pages 6506-6315.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence sequence for sets. In ICLR (Poster).
John Wieting and Kevin Gimpel. 2018. Paranmt-50mr Pushing the limits of paraphrastic sentence embed- dings with millions of machine translations. In ACL (I), pages 451-462, Association for Computational Linguistics.
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021. Poolingformer: Long document modeling with pooling attention. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 12437-12446. PMLR.
Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2022. Adversarial retriever-ranker for dense text retrieval. In ICLR. OpenReview.net.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with BERT. In ICLR. Open- Review.net.
A.2 Discrete and Continuous Variables
Our paraphrasing model uses discrete and contin- uous variables to encode sentence meanings and entities, respectively. Hosking and Lapata (2021); Hosking et al. (2022) use discrete and continuous variables to encode the syntax and semantics of the sentences. In terms of the model structure, both of their models and our paraphrasing model consist of encoder that outputs discrete and continuous variables and decoder that accepts discrete and continuous variables. The differences are a) their methods are supervised, b) their methods require additional templates with the same syntax as the
Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018. Integrating trans- former and paraphrase rules for sentence simplifica- tion. In EMNLP, pages 3164-3173. Association for Computational Linguistics.
Zhong Zhou, Matthias Sperber, and Alexander Waibel. 2019. Paraphrases as foreign languages in multilin- gual neural machine translation. In ACL (2), pages 113-122. Association for Computational Linguis- tics.

---

label as input, c) they train another syntax predic- tor to identify the target syntax while we sample the presentations with different surface structures from discrete codebook, and d) they aim to gen- erate paraphrases of different syntaxes while we emphasize the accuracy of entities in generating paraphrases of different surface structures.
ploys discrete variables encode sentences while we enhance the accuracy of entities with contin- uous variables and b) VQ-VAE trained using a reconstruction objective while we use an addi- tional Pseudo-data Constructor to further enhance the diversity of paraphrases. Principled Paraphras- ing uses an additional adversarial reconstruction decoder to encourage the encoder to remove much structural information from the original sen- tence as possible, and then employs a translation task to motivate the encoder to retain the seman- tics. This approach can control the similarity of the paraphrases by controlling the loss weights of reconstruction and translation. Besides the need for additional parallel translation data, Principled Paraphrasing requires setting the loss weights be- fore training, which means that it requires train- ing multiple models with different hyperparame- ters generate paraphrases with different degrees of similarity. In contrast, our approach controls the similarity by sampling discrete variables in the inference phase, in other words, one trained model can be used to generate diverse paraphrases.
A.3 Lexically Constrained Decoding
We use continuous variables in the paraphrasing model improve the accuracy of the entities and lexically constrained decoding (Hokamp and Liu, 2017) has the potential to accomplish this task, The difference is that our approach meets this con- straint on the input and model side with a soft training objective, while lexically constrained de- coding implements this on the decoding side with a hard occurrence constraint. This hard constraint leads to degradation of the generation quality be- cause the constraint that some tokens must be gen- erated results in the sampling of suboptimal text during the generation process. We compare our approach with the lexically constrained decoding and demonstrate the superiority of our approach in Section 4.6.3.
B The Statistics of the Datasets
A.4 Explicit Control of Similarity
Table 8 shows the statistics of the processed datasets.
We adopt discrete variable sampling to obtain mul- tiple paraphrases of the same sentence, and we control the similarity between the paraphrase and the original sentence in terms of surface struc- ture by adjusting the sampling temperature. Some decoding strategies, such as top-k sampling (Fan et al., 2018), nucleus sampling (Holtzman et al., 2020), and DIPS (Kumar et 2019) support sim- ilar functionality. The difference is that our dis- crete variables encode the meaning and structure of the whole sentence, and sampling on discrete variables implies sampling at the sentence level, while the sampling of the above decoding strate- gies goes the token level. Further, our sampling is performed on the model side (input side for de- coder) and the sampling of decoding strategies is performed on the decoding side, which also means that our method can be used simultaneously with these decoding strategies without conflicts.
Train Val / Test
Avg token
Avg entity
Wiki QQP
500k / 3k / 20k
8.25
0.26
400k / 3k / 20k
11.33
0.44
Table 8: Statistics of WikiAnswers and QQP. The train- ing sets contain non-parallel sentences and the val/test sets contain parallel paraphrase-paie pairs.... "Avg token" means the average number of tokens per sentence and "avg entity" means the average number of entities per sentence.
C Details of Entity-Score
We use N denote the number of input sentences, to denote an entity, A,, A,,, and A% to denote the set of entities the n-th input sentence, in the references of the n-th input sentence, and in the generated paraphrase of the n-th input sentence.
The existing works that can control the sim- ilarity by hyperparameters also include VQ- VAE (Roy and Grangier, 2019) and Principled Paraphrasing (Ormazabal et al., 2022). Our ap- proach draws on the discrete sampling method of VQ- DAE,,, with the difference that a) VQ-VAE em-
In calculating the precision, we consider an en- tity as true positive if the entity in the generated paraphrase appears in the input sentence or refer- ence sentences, otherwise false positive, i.e.

---

Task
Method
BLEUt
Self-BLEU,
IBLECT
BERT-IBLEU"
Entity-
bantttinetune
37.20
37.40
29.74
85.52
70.10
DNPG
41.64
33.26
34.15
WikiAnswers
GPT-3.5-turbo
20,93
15.75
17.26
91.26
66.71
Ours(r=5)
38.57
48.52
29.86
81.16
85.10
hart+finctune
31,85
37.98
24.87
86.29
68.72
DNPG 25.03
45.17
18.01
QQP
GPT-3.5-turbo
9.30
15.77
6.79
92.46
72.78
Ours(r=5)
25.58
62.78
16.74
73.51
95.35
Table 9: Evaluation results on supervised methods and large language model.
find that the paraphrasing quality of GPT-3.5- turbo is much better than other methods in sub- jective ratings. We assume that this because the labels of the paraphrase datasets are still quite in- adequate in terms of diversity and surface structure variation. Also, GPT-335-turbe tends to generate paraphrases that are longer than the original sen- tences, which directly leads to poor performance on BLEU scores.
1, a UA)
TP= Σ Σ
fTr(a); ST(())
0.
otherwiise
0,
(A. UA))
FP Σ Ser(a); [w((
1. otherwise
TP
precision + FP
In calculating the recall, consider an entity false negative if the entity in the imput sentence docs not appear in the generated paraphrase, i.c.
BLEU
80
FN
Σ fin(a); fix(a) 4
Ag otherwise"
40
TP
recall
FN
30
12
14
15
18
70
50
Thereafter, we calculated entity-score with pre- cision and recall like F1 F1-Score, i.e.
Figure 4: BLEU, Self-BLEU and entity- score QQP change with increasing sampling temperature.
ectity-score 2 x precision x recall
precision + recall
BLEU
SB
iB
Bert-iB
ES
Results on Supervised Methods and GPT-3.5-turbo
1
25.87
63.99
16.88
72.62
95.82
5
25.58
62.78
16.74
73.51
95.35
23.80
57.82
15.64
76.80
94.58
When experimenting with GPT-3.5-turbo, simple instruction "Paraphrase the following sentence and keep the entities correct:". Table shows the results on supervised methods and GPT- 3.5-turbo.
12
21.58
52.06
14.22
79.97
93.80
19.14
45.17
12.71
82.97
92.60
16
16.55
38.77
11.02
85.09
91.74
14.22
33.37
9.46
86.44
91.44
20
12.49
29.70
8.27
87.12
90.10
20.25
5.40
88.20
87.93
can be seen that there are still gaps between our method and the supervised methods, espe- cially on the surface structure, as our method scores much higher in Belf-BLEU, Surprisingly, approach achieves the highest Entity-Scores on both datasets, even in the face of powerful large language model. It is worth noting that al- though GPT-3.5-turbo performs poorly BLEU, we analyze some of the generated examples and
Table 10: The experimental results with rising sam- pling temperatures. For friendly look, we use SB, iB, Bert-iB, and ES to denote Self-BLEU. iBLEU, Bert- iBLEU, and Entity-Score, respectively. 7=1 means that we disable the sampling.
In comparison with supervised methods, our ap- proach does not require parallel paraphrase data.

---

Compared to large language models, our approach consumes much fewer compulational resources during training and inference.
demonstrates the consistency among the three an- notators.
G Correlation Between Automatic and Human Evaluation
E Evaluation Results on Different Sampling Temperatures
Table 11 presents the Spearman's correlation be- tween the automatic evaluation metries and the human evaluation scores. Entity-Score correlates better with human evaluation on entity than other automatic metrics. Self-BLEU correlates well on semantics, surface structure, and entity. This is be- cause more word overlap tends to imply higher se- mantic similarity, less structural variation, and a greater possibility of containing the same entities. BERT-iBLEU shows the opposite result to Self- BLEU. This may be caused by Self-BLEU being over-considered in calculating BERT-iBLEU. Un- expectedly, BLEU correlates poorly with human evaluation of semantics, This is due to the fact that paraphrase generation is highly open-ended task. And iBLEU does not seem to be good eval- uation metric either, because iBLEU uses BLEU as the key to evaluate the semanties.
Figure shows the changes in BLEU, Self-BLEU, and Entity-Score on the QQP dataset with rising sampling temperatures in our method. Table 10 presents the detailed experimental results. As the temperature rises, the Entity-Scort still maintains decent although both BLEU and SeIf-BLEU drop substantially, which demonstrates that trol the similarity to large extent while maintain- ing the accuracy of the generated entities.
We observed the generated paraphrases at high (T 50) sampling temperature. There is part of the paraphrases that deviates significantly from the original sentences in semantics although these paraphrases achieve a high average Bert-iBLEU score. This the reason why we select sampling temperatures that yield paraphrases with similar BLEU to the baselines to compare Table and Table 2.
It is not fair enough use single metric (iBLEU or BERT-iBLEU) evaluate the qual- ity of the paraphrases as in previous works. Our method outperforms unsupervised baselines multiple automatic metrics and human evaluation that demonstrates the superiority of our method.
F Human Evaluation Details
We randomly seleet 50 examples containing enti- ties from QQP. Three annotators are asked to rate the paraphrases generated by our method (r = 5 and r = 10) and three strong baselines accord- ing to the semantics, surface structure, and en- tity scale of 1 to 3, the higher the better. The annotators rate the generated paraphrases as bad/normal/good, corresponding to 1/2/3 points respectively. Before the rating, we informed all the annotators of the potential risks caused by the negative statements generated by artificial intelli- gence. As shown in Figure 5, Figure 6, and Fig- ure 7, we present detailed scoring guidelines and examples rated with different scores. The anno- tators were recruited in the authors' labs and they all have relevant English publications in the field of neural networks. Payment for this human eval- uation is made in full by the lab's supervisor based on the workload.
H Generated Examples
Table 12 shows several examples generated using our method and the unsupervised baselines. Our approach generates paraphrases with greater vari- ations in surface structure, which include more word order variations (sample 3-6), lexical vari- ations (sample 3-6), and introduced extensions (sample 1 and 6). At different sampling tempera- tures, our method generates paraphrases with dif- ferent surface structural similarities the origi- nal text, while the semantics are well preserved. When the input contains entities, our method gen- erates more accurate entities than the other base- lines (sample 1-5).
We Fleiss' Kappa (Fleiss, 1971) to mea- sure the inter-rater agreement and the results 0.45 (moderate agreement), 0.46 (moderate agree- ment), and 0.87 (very good agreement) seman- tics, structure, and entity. The agreement score

---

Semantics
Surface Structure
Entity
BLEU
0.174 (p < 0.01)
-0.124 (p = 0.03)
0.257 (p < 0.01)
Self-BLEU
0,401 (p < 0.01)
-0.386 (p < 0.01)
0.256 (p 0.01)
iBLEU
0.035 (p = 0.31)
-0.012 (p 0.83)
0.185 (p < 0.01)
BERT-iBLEU
-0.326 (p 0.01)
0.377 (p 0.01)
-0.152 (p < 0.01)
Entity-Score
0.278 (p 0.01)
-0.027 (p 0.65)
0.579 (p 0.01)
Table 11: Spearman's 's correlation between the automatic evaluation metrics and the human evaluation scores.
Human Evaluation on Semantics
We glad that help with the human evaluation. One thing note is that the sentences to be rated are generated by artificial intelligence. Althaugh we have done a preliminary review of these sentences, they may contain offensive information. If you find any sentences with offensive statements (racial bias, gender bias, religious bias, etc.), please contact and will immediately replace the samples at issue.
Paraphrase Generation aims generate sentence that the same semantic meaning as the input with different need to rate the paraphrases according to whether it contains the same semantics the input.
structure (grammar,
pattern, words). this evaluation,
the sentence and the input contain different semantics, please select [bad].
If sentence and input contain similar semantics, please select [normal].
If the sentence and the input contain the exact same semantics, please select [good].
Here are some examples:
input: what you think about this movie?
s1: do you like Jackie Chen? [bad]
s2: do you like this movie? [normal]
s3: what your opinion about this movie? [good]
what you think af the mavie? [good]
1. on an average, much or marks are required get selected in indian telecommunication service through ies?
bad
normal
good
how much rank and marks required to get selected in indian ies through telccommunication service an average?
Figure 5: The scoring page for human evaluation on the semantics.

---

Human Evaluation on Structure
are glad that you help us with human evaluation. One thing to note is the sentences rated generated artificial intelligence. Although have done preliminary review lhese sentences, they may siill contain offensive information. you find any sentences with offensive statements (racial bias, gender bias, religious bias, etc.), please contact us and we will immediately replace samples at issue.
Paraphrase Generation aims to generate sentence that has the same somantic mcaning as the input but with a different sentence structure (grammar, sentence pattern, and words). In this evaluation, we need you to rate the paraphrases according how much differs from the input the surface structure.
the sentence and the input are the same, please select [bad].
sentence and input in just words similar in grammar, please select [normal].
If the sentence and the input differ greatly grammar and pattern, please select [good].
Here are some examples:
input: what you think about movie?
s1; do you like Jackie Chen? [good]
s2: do you like this movie? [good]
s3: what is your opinion about movie? [normal]
what you think this movie? [bad]
on average, much or marks are required to gel selected in indian telecommunication service through ies?
bad
normal
good
how much rank and marks
required to get selected in
indian les through telecommunication service on an average?
Figure The scoring page for human evaluation on the surface structure.

---

Human Evaluation on Entity
evaluation. One thing rated generated by artificial intelligence. Although have done preliminary review of these sentences, they may atill contain offensive information. If you find any sentences with offensive statements (racial bias, gender bias, religious bias, etc.), please contact we will immediately replace the samples at issue.
We glad that you help with
that the sentences
Paraphrase Generation aims to generate sentence that has the same semantic meaning as the input but a different sentence structure (grammar, sentence pattern, and words). this evaluation, we need you the paraphrases according whether the contain accurate entities input.
sentence contains no entiries or wrong entities, please select [bad].
part the entities the input, please select [normal]. If the sentence contains all the entities in the input. please select [good].
Here are some examples:
input: you think about movie Titanic and Leonardo DiCaprio?
s1: do you like Jackie Chen? [bad]
s2: do you like this movie? [bad]
s3: what your opinion about Titanic? [normal]
what you about Leonardo DiCaprio Titanic? [good]
on an average, much or marks are required to get selected in indian
telecommunication service through ies?
bad
normal
good
how much rank and marks
required to get selected in indian ies through telecomiuniiation service an average?
Figure 7: The scoring page for human evaluation the entity.

---

Original
what are some possible solutions if forgot my icloud password?
Set2seq+RTT
what are the possible solutions if forget my password?
TA+SS+DB
what are the possible solutions forget my iphone icloud password?
Principled
what are some possible solutions forgot my i[fr] Quelle password?
Principled
what solutions can be found if have lost my imple password?
Ours(T 5)
forgot my icloud password, what are some solutions?
Ours(T 10)
forgot my icloud password. what are the solutions to this?
Original
will the nintendo switch become successful?
Set2seq
is the ncero switch successful?
TA+SS+DB
will nintendo's the switch become successful?
Principled
will the nintendo switch become successful?
Principled
will the nintendo switch
be successful?
Ours(r = 5)
will the nintendo switch
ever become successful?
Ours(T 10)
will nintendo switch ever become successful?
Original
which is the best book with which can prepare for gre exams ..?
Sel2se+RRTT
what are the best books to prepare for the greco examination
TA+SS+DB
which book is best for the gre ? with which books which is the best book with which can prepare for gre exams ?
prepare for exams..
Principled
Principled
what's the best book for to prepare for my exams?
Ours(r 5)
how do i prepare for gre exams ..which is the best book for it?
Ours(T 10)
which is the best book for preparation of gre exams ..?
Original
was the suicide squad movie good in your opinion?
Set2seq+RTT
do you think the film of the suicide squad is good?
TA+SS+DB
was suicide squad good movie in your opinion? was the suicide squad movie good in your opinion?
Principled
Principled
do you think the movie about the suicidequad is a good one?
Ours(T 5)
is the suicide squad movie any good in your opinion?
Ours(T 10)
what is your opinion on the movie suicide squad ?
Original
how many days does it take a pan card to arrive after applying?
Set2seq+RTT
how many days after applying for pane card
TA+SS+DB
how many days does pan card take to arrived in india after apply?
Principled
how many days does it take pan card to arrive after applying?
Principled
how many days does take to arrive after the application to receive a card?
Ours(T = 5)
when applying for pan card how many days does it take to arrive?
Ours(r = 10)
when applying for pan card how many days does take for the pan card to arrive?
Original
how should stop being insecure?
Set2seqRRTT
how do i stop being insecure?
TA+SS+DB
how should stopped being insecure?
Principled
how should stop being insecure?
Principled
how can stop being insecure?
Ours(r = 5)
how can stop being insecure?
Ours(T = 10)
what can i do to stop being insecure about myself?
Table 12: examples generated using our method two baselines. The entities original sentences are indicated in blue.. The correct/wrong entities in generated paraphrases are indicated in green red


Reviews:
[{"review_id": "HwP7tovMuq", "reviewer": "Reviewer_PEo6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments about our work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-", "reasons_to_reject": "-", "questions_to_authors": "", "paper_topic_and_main_contributions": "-", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8RCwuPw9zX", "reviewer": "Reviewer_PrBd", "summary": "", "questions": "- How does the method handle cases where entities are close in distributional space? I'm thinking of a case like \"President of the USA\" and \"Barack Obama\", which might be very close in distributional space since they co-occur frequently, but where one cannot replace \"President\" with \"Obama\" in all cases. \n- Why did you choose BART instead of T5? It seems to me like the setting in Fig. 3 bears a strong resemblance to the T5 pre-training/inference regime and the use of extra tokens to bind variables. Have you tried your method with T5? \n- Maybe I'm misunderstanding the baselines, but have you tried running the pseudo-data constructor on its own as a paraphrase model? How much better is the existing model (especially at entities) than the constructor alone? \n- what was the basis for choosing the number of vectors in the codebook? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are excited that you like our work and now we offer the following responses to your questions.\n\n$\\text{\\color{blue}Q1: How does the method handle close entities like \"President\" and \"Obama\"?}$\n\nA1: We encode entities using continuous variables, and we do not apply sampling and replacement on continuous variables during training and inference, so the close entities, like \"President\" and \"Obama\", will not be intentionally replaced.\n\n$\\text{\\color{blue}Q2: Why did you choose BART instead of T5?}$\n\nA2: We focus on the problem of accurate entity and controlled surface structure similarity in paraphrasing tasks, with the key point being the coordination of discrete and continuous variables. We choose BART simply because it is a commonly used seq2seq pre-trained model. We believe that using other pre-trained models, such as T5, would not affect our experimental conclusions.\n\n$\\text{\\color{blue}Q3: How about running the pseudo-data constructor on its own as a paraphrase model?}$\n\nA3: We do not treat the constructor as a baseline and consider it as one of the essential modules of our method. We recognize that the constructor can be used on its own as a paraphrasing model. So we compare the constructor with another pseudo-data construction method, round-trip translation, in 4.6.1. If we directly compare our method with the constructor, our method significantly outperforms the constructor in syntactic preservation and entity accuracy.\n\n$\\text{\\color{blue}Q4: What was the basis for choosing the number of vectors in the codebook?}$\n\nA4: There is no consensus on the size of the codebook from previous works (Roy and Grangier, 2019; Hosking and Lapata, 2021), we explore multiple sizes and subsequently identify the most optimal one based on its performance.\n\nIn addition, we will add the reference and optimize the notation as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Experimental results. The results of the method are strong and meet the goals set out in the introduction. The authors also include relevant ablations. The human evaluation makes me more confident in the quality of the results. \n\n2. Relevant baselines. The authors compare against relevant baselines. Especially relevant is the comparison/inclusion of lexically-constrained decoding, which can be combined with their method. \n\n3. Well-written and presented. For the most part, I was able to understand the paper immediately. The model and analysis is presented in a generally well-organized and well-written way. The related work section is well-written and situates the method well. I especially liked the presentation of differences in Table 5. ", "reasons_to_reject": "1. Extension of past work. As far as I can see, the model here is an improvement on/extension of the VQ-VAE from Roy and ABC (2019). \nHowever, this is not a major weakness, as the authors make targeted modifications for entity faithfulness and also use pre-trained models, which makes the method more relevant to current NLP efforts. \n\n", "questions_to_authors": "- How does the method handle cases where entities are close in distributional space? I'm thinking of a case like \"President of the USA\" and \"Barack Obama\", which might be very close in distributional space since they co-occur frequently, but where one cannot replace \"President\" with \"Obama\" in all cases. \n- Why did you choose BART instead of T5? It seems to me like the setting in Fig. 3 bears a strong resemblance to the T5 pre-training/inference regime and the use of extra tokens to bind variables. Have you tried your method with T5? \n- Maybe I'm misunderstanding the baselines, but have you tried running the pseudo-data constructor on its own as a paraphrase model? How much better is the existing model (especially at entities) than the constructor alone? \n- what was the basis for choosing the number of vectors in the codebook? ", "paper_topic_and_main_contributions": "This paper describes a method for unsupervised paraphrasing that does not rely on translation corpora and is capable of producing diverse paraphrases. \nUnlike past work on unsupervised paraphrasing, the method is explicitly focused on maintaining the faithfulness of entity mentions. \nThe authors propose a VQ-VAE architecture trained on data extracted by pseudo-labelling.\nThe central idea of the VQ-VAE approach is that the sentence syntax can be handled in a discrete and quantized space, allowing the authors to sample high variance sentences, while the entity mentions are handled separately. \nBy separating the entities from the rest of the sentence, the method allows for greater diversity while still maintaining entity faithfulness. \n\nThe authors evaluate their method using automatic and human evaluation. \nIn addition to standard automatic metrics, the authors introduce a new metric which focuses specifically on entity F1. \nThe proposed method outperforms past work and is competitive with supervised methods in some cases. \nThe summary of contributions is\n1. a VQ-VAE method for paraphrasing which maintains entity mentions\n2. a pseudo-labelling method for creating paraphrase data \n3. a metric for scoring entity mention faithfulness ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- L473 should cite Holtzman et al. (2020)\n", "Typos_Grammar_Style_and_Presentation_Improvements": "- I really don't like the notation $y^{-e}$, $e^{x}$, etc.. I was immediately confused as $e$ already has a common use in equations, especially exponentials like $e^x$. \n"}, "extra_text": {}}, {"review_id": "fyhsnIFHEg", "reviewer": "Reviewer_Lw6x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review the paper. We apologize if some parts of the manuscript were not comprehensible enough and led to misinterpretations. Hopefully, our response will help to alleviate any concerns you may have.\n\n$\\text{\\color{blue}Reason 1: It is unclear how the pseudo-data constructor is trained with the shuffled keywords.}$\n\nAnswer 1: As stated in lines 181-183, \"the constructor is trained by reconstructing the original sentence with ordered input keywords\". We do not shuffle the keywords in training constructor. This information is further illustrated in step 1 of Figure 2.\n\n$\\text{\\color{blue}Reason 2: It is unclear if the generated sentences (the candidates) are grammatically incorrect.}$\n\nAnswer 2: As stated in lines 197-200, \"we filter the candidates using BERTScore (Zhang et al., 2020) and Self-BLEU because not all keywords orders yield valid paraphrases\". We have acknowledged the potential grammatical incorrectness and have included a screening step specifically aimed at identifying and mitigating this problem. We also visualize this phenomenon and the corresponding screening process in step 2 of Figure 2.\n\n$\\text{\\color{blue}Reason 3: It is unclear if the entities can be output from the same model.}$\n\nAnswer 3: The entities can not be output from our paraphrasing model. Specifically, we employ Spacy to recognize the entities, as duly stated in line 212. You raised an interesting concern, as we mentioned in the Limitations (lines 569-573), the reliance on an external tool can be considered as one of the limitations of our proposed method. However, this does not affect the effectiveness of our method and the experimental conclusions.\n\n$\\text{\\color{blue}Reason 4: The existing back-translation dataset e.g., ParaNMT, should be evaluated.}$\n\nAnswer 4:  Following Niu et al.(2021), we demonstrate the effectiveness of our method on two commonly used datasets, WikiAnswers and QQP. We also explain the reasons for our choice of datasets in lines 585-593. ParaNMT and other back-translation datasets do not exhibit properties that are not available in WikiAnswers and QQP and are relevant to the goals of our approach. So we believe that our method will achieve promising results on these datasets as well. We will try to add the ParaNMT results in the modified version.\n\n$\\text{\\color{blue}Reason 5: ChatGPT should also be used to compare the performance.}$\n\nAnswer 5: We provide experimental results and analysis of ChatGPT in Appendix D, with notes in the Section Baselines in lines 343-346. However, we must consider that ChatGPT is a large-scale language model, and we have limited information about its training data and architecture, even we cannot be sure whether there is any test data leakage. Therefore, it would be unfair to compare our method directly with ChatGPT. Nonetheless, we include the experimental results in the appendix as a reference and provide corresponding analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel approach of unsupervised paraphrase generation.\n- The results showed that the proposed approach is promising and performs better than existing state-of-the-art unsupervised paraphrase generation approaches.\n- Propose a novel metric, Entity-Score, to evaluate the precision and recall of the entities.\n", "reasons_to_reject": "- Some parts of the approaches are unclear. It is unclear how the pseudo-data constructor is trained with the shuffled keywords. In the paper, it's just one small paragraph that is used to describe this process. It is also unclear if the generated sentences (the candidates) are grammatically incorrect, since the word shuffling may result in grammatical incorrect sentences.\n- Entities need to be created during inferencing. It is unclear if the entities can be output from the same model.\n- The existing back-translation dataset e.g., ParaNMT, which is a very popular unsupervised paraphrasing dataset, should be evaluated to see how the proposed approach really performs with it. ChatGPT, should also be used to compare the performance, since it is also some-what of an unsupervised setting", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an approach for unsupervised paraphrase generation through unifying discrete and continuous representation. Unsupervised paraphrase generation is important due to the labor intensive effort in annotation in the supervised setting. Existing approaches in unsupervised paraphrase generation rely on back-translation or denoising, which suffer from multiple limitations - a) back-translation results in additional corpus of another language, b) the generated paraphrases of the original unsupervised approaches are very similar to the original sentences, c) existing unsupervised approaches could not control the syntactical similarity of the generated paraphrases and d) the generation of correct entities in the paraphrases are ignored. This paper is intended to tackle these limitations. This paper proposes a self-supervised pseudo-data construction approach (trained on non-parallel corpora) and a paraphrasing model that unifies both discrete and continuous variables (discrete variables are used for encoding the sentence meaning and the continuous variables are used to model the entities) to encode sentences. The datasets used are WikiAnswers and Quora. Results show that the pseudo-data construction approach has more varied syntactical paraphrases than back-translation and the overall experimental results are promising. The paper also introduces a new metric, Entity-Score, to evaluate the precision and recall of the entities.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a unique approach to unsupervised paraphrase generation by blending discrete and continuous representation. This method stands out from earlier unsupervised techniques, aiming to produce diverse syntactical paraphrases without the need for translation corpora while maintaining entity accuracy. At its heart, the methodology utilizes a VQ-VAE architecture combined with a self-supervised pseudo-data construction approach trained on non-parallel corpora. For assessment purposes, the research employed the WikiAnswers and Quora datasets and introduced the Entity-Score metric to measure the precision and recall of entities.
Regarding the topic and contributions, both reviewers concur that the paper presents a fresh perspective on unsupervised paraphrase generation, with a particular emphasis on maintaining entity integrity. They both underscore the innovative use of a VQ-VAE architecture and acknowledge the introduction of the Entity-Score metric as a significant advancement.
In terms of strengths, there's a unanimous sentiment about the promising nature of the experimental results. Both reviewers laud the novel approach and express appreciation for the Entity-Score metric, particularly highlighting the thorough evaluation bolstered by human assessments.
On the flip side, clarity appears to be an issue. The reviewers both identify areas in the paper, such as the methodology of the pseudo-data constructor and the treatment of entities, that could benefit from more explicit elaboration. They recommend additional evaluations using renowned datasets or models to further validate the approach.
Additionally, while Reviewer 3 praises the clarity and structure of the paper, Reviewer 2, though not explicitly critical, refrains from complimenting its writing style.
Overall, despite recognizing the paper's strengths and contributions, the reviewers have different areas of focus and depth in their critiques.

NB.: The assessment based on the two reviews should suffice for now. Decisions can still be made from those reviews, or if deemed necessary, another reviewer could be approached to provide further insights at a later time.
Decision: Accept-Main