A Quality-based Syntactic Template Retriever for Syntactically-Controlled Paraphrase Generation
Authors: Xue Zhang
Songming Zhang
Yunlong Liang
Yufeng Chen
Jian Liu
Wenjuan Han
Jinan Xu
Venue: EMNLP 2023 Main (2023)
Keywords: Paraphrase generation
Syntactic template retrievers
Mutual diversity

Abstract: Existing syntactically-controlled paraphrase generation (SPG) models perform promisingly with human-annotated or well-chosen syntactic templates. However, the difficulty of obtaining such templates actually hinders the practical application of SPG models. For one thing, the prohibitive cost makes it unfeasible to manually design decent templates for every source sentence. For another, the templates automatically retrieved by current heuristic methods are usually unreliable for SPG models to generate qualified paraphrases. To escape this dilemma, we propose a novel Quality-based Syntactic Template Retriever (QSTR) to retrieve templates based on the quality of the to-be-generated paraphrases.  Furthermore, for situations requiring multiple paraphrases for each source sentence, we design a Diverse Templates Search (DTS) algorithm, which can enhance the diversity between paraphrases without sacrificing quality.  Experiments demonstrate that QSTR can significantly surpass existing retrieval methods in generating high-quality paraphrases and even perform comparably with human-annotated templates in terms of reference-free metrics. Additionally, human evaluation and the performance on downstream tasks using our generated paraphrases for data augmentation showcase the potential of our QSTR and DTS algorithm in practical scenarios.

A Quality-based Syntactic Template Retriever for Syntactically-controllle Paraphrase Generation
Xue Zhang, Songming Zhang, Yunlong Liang, Yufeng Chen;
Jian Liu, Wenjuan Han, Jinan Xu Beijing Key Lab of Traffic Data Analysis and Mining,
Beijing Jiaotong University, Beijing, China (xue_zhang, smzn yunlong ang, chenyf, fiienyfjjiannin jianliu, wjhan, jaxu] tu. edu. cn
Abstract
Source sentencee
which book best for object oriented pregramming?
Human-unnota tad template
(ROOT (SBARQ (WHNP (WP 1) (SQ (VBP) (NP (NP) / /PP L.0)
Existing syntactically-contrllled paraphrase generation (SPG) models perform promisingly with human-annotated or well-chosen syntactic templates. However, the difficulty of obtaining such templates actually hinders the practical application of SPG models. For one thing. the prohibitive cost makes it unfeasible to manually design decent templates for every source sen- lence. For another, the templates automatically retrieved by current heuristic methods are usu- ally unreliable for SPG models to generate qual- itied paraphrases. To escape this dilemma, we propose novel Quality-based Syntactic Tem- plate Retriever (QSTR) to retrieve templates based on the quality the to-be-generated para- phrases. Furthermore, for situations requiring multiple paraphrases for each source sentence, we design a Diverse Templates Search (DTS) algorithm, which can enhance the diversity be- tween paraphrases without sacrificing quality. Experiments demonstrate that QSTR can sig- nificantly surpass existing retrieval methods in generating high-quality paraphrases and even perform comparably with human-annotated templates in terms of reference-fiee freee metrics. Additionally, human evaluation and the perfor- mance downstream tasks using our gener- ated paraphrases for data augmentation show- case the potential our QSTR and DTS algo- rithm in practical scenarios.
what are the best books for object oriented programming?
Paraphrase
Semmnntic:: 0.90// Syntaxc 0.071
Unsuitable tempalte
(ROOT (SBARQ (SINV (VP (VBZ JI (NP (NNP) ) (NNP) y) (VP (VB) (PP (ADVP) /SBAR (
does programming theory seem to be the most difficult subject beginmers so why think book is the best for beginners?
Panaphr ase
Semanticc:: 0.53 Syntaax: 0.51]
QSTR retrieved tamplate
(ROOT :SBARG (WHNP (WP ll (5Q (VBZ) ) (NP (NP) 30 L.))
what best book leam object oriented
Paraphrase
pregramming?
Semmnntic:: 0.96 Syntax: 0.17
The generated paraphrases with different tem- plates. "Semantics" and "Syntax" represent the seman- tic similarity with the source sentence and the syntactic distances against the template, respectively. Obviously, an unsuitable template may lead a poor paraphrase.
Figure I:
generation (SPG) is proposed and attracts extensive attention in the research community (lyyer et al., 2018: Chen et al., 2019a; Kumar al., 2020: Sun et al., 2021; Hosking and Lapata, 2021; Yang al., 2021b, 2022a; Huang et al., 2022; Hosking et al., 2022). Different from traditional PG models, SPG models take syntactic templates as additional con- ditions to generate paraphrases conformed with the corresponding syntactic structures, whose forms generally include syntax parse trees' and sentence exemplars. After years of research, SPG models can already generate syntax- contux-conformin and high- quality paraphrases with aanan-tnnttated or well- selected syntactic templates (refer to the first case in Figure 1).
1 Introduction
Paraphrase generation (PG) (Madnani and Dorr, 2010) is to rephrase a sentence into an alternative expression with the same semantics, which has been applied to many downstream tasks, such as question answering (Gan and Ng, 2019) and dia- logue systems (Jolly et al., 2020; Gao et al., 2020; Panda et al., 2021; Liang et al., 2019, 2021, 2022). On this basis, to improve the syntactic diversity of paraphrases, syntactically
However, such promising performance heavily relies on those satisfying templates, whose difficult acquisition in practice largely hinders the applica-
paraphrase YYufeng Chen is the corresponding author.
'In this paper, we use syntax parse trees as templates fol- lowing most previous work (Sun et al., 2021; Yang et al., 2022a; Huang et al., 2022).

---

tion of SPG models. Firstly, manually tailoring templates for every source sentence is practically unfeasible since it is time-consuming and labori- ous. Alternatively, automatically retrieving decent templates is also difficult, and unsuitable templates would induce semantic deviation syntactic errors in the generated paraphrases (refer to the second case in Figure 1). On templates retrieval, current solutions are mostly heuristic methods and assume that suitable templates should satisfy certain con- ditions, e-g., high frequency in the corpus (Tyyer et al., 2018) or high syntactic similarity on the source side (Sun et al., 2021). I). The only exception is Yang et al. (2022a), which utilizes contrastive learning to train a retriever. Nonetheless, it also assumes that suitable template should be well- aligned with the corresponding source sentence. Albeit plausible, the retrieval standards in these methods have no guarantee of the quality of the generated paraphrases, which may lead to the un- stable performance of SPG models in practice.
current retrieval methods on the diversity between multiple paraphrases and meanwhile maintain their high quality. In the end, using templates from QSTR to generate paraphrases for data augmen- tation achieves better results than previous retrieval methods in two text classification tasks, which fur- ther indicates the potential of QSTR in downstream applications.
In summary, the major contributions of this pa- per are as follows?:
We propose a novel Quality-based Syntac- tic Template Retriever, which can retrieve suitable syntactic templates to generate high- quality paraphrases.
To reduce the repetition when retrieving multi- ple templates by current methods, we design a diverse templates search algorithm that can in- crease the mutual diversity between different paraphrases without quality loss.
To address this limitation, we propose a novel Quality-based Syntactic Template Retriever (QSTR) to retrieve templates that can directly im- prove the quality of the paraphrases generated by SPG models. Different from previous methods, given a source sentence and a candidate template, QSTR scores the template by estimating the quality of the bebereeerated paraphrase beforehand. To achieve this, we train QSTR by aligning its output score with the real quality score of the generated paraphrase based on a paraphrase-quality-beee metric, i.e., ParaScore (Shen et al., 2022). With sufficient alignment, templates with higher scores from QSTR would be more probable to produce high-quality paraphrases. Moreover, when general- ing multiple paraphrases for each source sentence, we observe a common problem in QSTR and pre- vious methods that the top-retrieved templates tend to be similar, which may result in similar or even repeated paraphrases. Aiming at this, we further design a Diverse Templates Search (DTS) algo- rithm to enhance the diversity between multiple paraphrases by restricting maximum syntactic sim- ilarity between candidate templates.
The automatic and human evaluation results demonstrate the superiority of our method, and the performance in data augmentation for downstream tasks further prove the applica- tion values of QSTR in practical scenarios.
Related Work
As the syntactically paraphrase genera- tion task is proposed (lyyer al., 2018) and has received increasing attention, previous work mainly focuses on improving the performance the gen- erated paraphrases conforming to the correspond- ing human- templates. More specifically, most of them modify the model structures to bet- ter leverage the syntactic information of templates (Kumar et al., 2020; Yang et al., 2021a, 2022a). Fur- thermore, Sun et al. (2021) and Yang et al. (2022b) generate paraphrases based on the pre-trained lan- guage models, e.g., BART (Lewis et al., 2020), ProphetNet (Qi et al., 2020), and yield better per- formance.
Compared to the concentration on SPG models, only rare methods focus on how to obtain tem- plates for these SPG models in practice. Among them, lyyer et al. (2018) and Huang and Chang (2021) directly use the most frequent templates in the corpus. Sun et al. (2021) select the syntax parse trees of the target sentences in the corpus
Experiments two benchmarks demonstrate that QSTR can retrieve better templates than pre- vious methods, which help existing SPG models generate paraphrases with higher quality. Addition- ally, the automatic and human evaluation showcase that our DTS algorithm can significantly improve
"The code is publicly available at: XZhange0/QSTR

---

P
Painutes bot Product
Maximum values as weight
X
Source Sentence
Encoder-s
Feed Forward
Sentence
SPG
Syntactic Template
Encoder-
t
Feed Forward
Syntacti Template
t
Figure 2: The model architecture and the training process of QSTR. QSTR models the relationship between the source sentence and the template t and maps it into score 8, which denotes quality estimation of the paraphrase to be generated. Then, the training objective is to align the score with the true quality score & of the paraphrase p based the ParaScore metric.
whose source sentences are syntactically similar to the input source sentence. Yang et al. (2022a) retrieve candidate templates based on distance in the embedding space. However, the retrieval stan- dards in these heuristic methods cannot guarantee the quality of generated paraphrases. On the con- trary, our QSTR directly predicts the quality of the paraphrases to generated with the template, and the retrieved templates have greater potential to generate high-quality paraphrases.
h - (h1, 12 kmm) - Enc-t(t,, t..... fm)-
To further extract the semantic and syntactic fea- tures, we add two feed-forward networks FFN-s and FFN-t after the two encoders. Then we can obtain the final sentence embeddings ex {er ....., ......,,,, ( and the final template embeddings e' fef. .....,, e
e FFN-s(h'),
(1)
e' = -FFN-t(h'.
(2)
3 Methodology
To model the mapping relationships between the tokens in sentence and the constituents of a re- lated template, calculate the pairwise dot prod- uet between two kinds of embeddings and obtain the correlation matrix C:
In this section, we first introduce our QSTR ($3.1), which includes the model architecture 1$3.1.1) and the training objective ($3.1.2). Then we design diverse templates search algorithm to improve the mutual diversity between multiple paraphrases ($3.2).
Caxm = e" -ee)"""
(3)
3.1 QSTR: Quality-based Syntactic Template Retriever
where Cj indicates the degree of correlation be- tween the token 2; and the syntactic constituent ty. Then we take the maximum value of each row/column in C' as the weight for weighted av- eraging sentence/template embeddings and obtain their final representations and of:
3.1.1
Model Architecture
As shown in Figure 2, QSTR has two-tower ar- chitecture that contains sentence encoder and a syntactic template encoder. The two encoders are used to encode the source sentence x and the syntactic template respectively. Formally, given a source sentence with tokens x {21, L2,, _n.} and template" with m con- stituents {t1, t,,...,tmh the sentence en- coder Enc-s embeds into sentence embeddings he and the syntactic template encoder Enc-t em- beds into template embeddings h':
max (Cty =c?), ==1
(4)
m
0/
máx (ity * ct).
(5)
m
In the end, we concatenate 2" and of and trans- form it scalar s through linear layer and Sigmoid function:
h" (hi, Ng .... (.)) Enc-s(21,)) (,,, 22,, kn),
Forr example, the template"
(omplate"RROr"" "(ROOT (8 (VS (VP)))
( ")"" is formalized as ("(, 'ROOT", 'C, 'S,, 'C. "".
Sigmoid [f
(6)

---

where a score that represents the matching de- gree between the source sentence and the syntactic template.
Algorithm 1 Diverse Templates Search Algorithm
Input: imput sentence x, the whole template li- brary T {t,, 12...., ,trr]), the number of the retrieved templates d, the syntactic diver- sity threshold |
In nutshell, given a sentence x and a template t, the goal of QSTR to output a quality estimation s for the future paraphrase through modeling the interaction between x and t:
Output: diverse templates set TT 1: Initialize T = Ø as a min heap.
2: for each i C [1, [T]] do
8 = Q))
3: QSTR(xx, t;)
3.1.2 Training Objective
4: if [Tal < d then Ta-push(t,)
During training, aim to align the estimated score s from QSTR with the real quality value of the para- phrase. Towards this end, for each x, we randomly sample some templates from the whole template library T as the candidate template set Tr which also includes the templates of x and the reference y as more competitive candidates. Then, given Ta {t1, ................................ (x). can obtain the prior estimations S₃ = {si, ......, ....,.................... .s} for these tem- plates from QSTR by Eq.(7). At the same time, we can use the SPG model generate the correspond- ing paraphrases Ps (P1: P2..... P&) based on Ta and evaluate their real quality. To acquire the quality scores of these paraphrases, we select the reference-based metric ParaScore.ef (Shen et al., 2022) that has the highest correlation with human evaluation. Formally, given a source sentence and its reference y, the quality value z: of the para- phrase p; can be calculated by:
else if min TED(t, Ta(+)) and 8
STa(O] then Ta.pop(Ta(0)
Ta.push(t;)
9: else
10: continue
11: end if
12: Ta-heap_sorls by sT.l
13: end for
14: return TT
3.2 Diverse Templates Search Algorithm
In practice, the top templates retrieved by exist- ing retrieval methods may have similar features, e.g., syntactic structures, which may lead to repe- titions when generating multiple paraphrases for one source sentence. To improve the mutual diver- sity between multiple paraphrases while maintain- ing their high quality, we design a general diverse templates search (DTS) algorithm as described in Algorithm 1, which can be equipped with existing retrieval methods.
qi = ParaScoreeee (D. x, y),
(8)
where
qi construct the quality set Qe { [l, -2.-....................... ..
Next, we use the Mean Square Error (MSE) loss to quickly align the prior predictions S₃ with the posterior quality Q₁ quantitatively:
Taking QSTR as example, given an input sen- tence x, traverse the whole template library T {t1, .t,t......ttt. and calculate a score s; for each t, (lines 2~33). Then, we maintain a min heap Ta in a size to collect the satisfactory tem- plates t, from T, which have high scores and mean- while diverse syntactic structures between each other (lines 4~). To find these templates, we calculate the Tree Edit Distance (TED) (Zhang and Shasha, 1989) between the template t; and tem- plates in Ta and ensure that the minimum TED value is greater than threshold before append- ing t, to T6 (line 6). After one traversal, the heap T, will contain the final d qualified templates for (11) diversely rephrasing the source sentence x.
Lise MSE(S& Qu).
(9)
Morcover, to better learn the quality ranks among TE. we also calculate a pairwise rank loss Lrank for S₁ according Q
Lrank
Σ max ((8) (7)) 1l llg 20 0), (10)
where 63, = 81 8j. 87 = qi Uj and 11 11 <0 =
when 6", < 0 otherwise
In the end, the overall training objective L con- sists of the above two loss functions:
= Cmee + Arank

---

4 Experimental Settings
4.2 Contrast Methods
Here we introduce all the contrast methods for ob- taining templates, Moreover, we also conduct para- phrase generation with Vicuna-13B' (Chiang et al., 2023) strong baseline.
4.1 Datasets and Implementation Details
Following previous work (Sun al., 2021: Yang et al., 2022a), we conduct our exper- iments on ParaNMT-Small (Chen et al., 2019b) and QQP-Pos (Kumar et al., 2020). Specifi- cally, ParaNMT-Small contains about 500K para- phrase pairs for training, and 1300 manually la- beled (source sentence, exemplar sentence, ref- erence sentence) triples, which are split into 800/500 for the test/dev set. And QQP-Pos con- tains about 140K/3K/3K pairs/trpllestriples for the train/test/dev set. the test/dev set of two datasets, the exemplars are human-annotated sentences with similar syntactic structures as the reference sen- tences but different semantics. The function of exemplars is to provide their syntax parse trees templates that guide SPG models to generate paraphrases syntactically close to references.
Datasets.
The syntax parse tree of the reference sentence used as the template, which can be regarded as the ideal template for the source sentence.
Ref-as- -aaas-Teeplaee..
The syntax parse tree of the exemplar sentence used as the template, which can be seen as the plate.
tem-
Random Template. We We randomly select tem- plate from the template library for SPG.
Freq-R. Following lyyer et al. (2018), we choose the most frequent template in the template library for SPG.
SPG Models. In our experiments, we use two strong SPG models, i.e., AESOP (Sun et al., 2021) and SI-SCP (Yang et al., 2022a), to evaluate the ef- fectiveness of our QSTR. To the best our knowl- edge, AESOP4 has state-of-the-art performance among previous SPG models. Besides, S1-SCP* includes a novel tree transformer to model parent- child and sibling relations in the syntax parse trees and also achieves competitive performance.
Sun et (2021) select the parse tree of the target sentence whose corresponding source sentence has the most similar syntactic structure to the input sentence.
AESOP-R.
SISCP-R.
Yang et al. (2022a) encode the sen- tences and the templates into the same space and retrieve syntactic templates based on the similari- ties between their representations.
We use this large language model (LLM) (Chiang al., 2023) for zero-shot para- phrase generation. Please refer to Appendix A for more details.
Vicuna-13B.
Implementation Details. We use the Stanford CoreNLP toolkit (Manning al., 2014) to obtain the syntax parse trees of the sentences, and we trun- cate all parse trees by height 4 and linearise them following Yang et al. (2022a). Then, we build the template library using the parse trees from the train- ing set of each dataset. The two encoders in QSTR are initialized with Roberta-bass (Liu et al., 2019) for both datasets. We use the scheduled AdamW optimizer with a learning rate of 3c-5 during train- ing. The max length sentences and templates are 64 and 192 respectively. The batch size is set to 32, and the size & of T& is set to 10. We train our QSTR for 10 and 20 epochs on ParaNMT-Small and QQP-Pos respectively. The coefficients A₁ and X in (11 are all set to 1. And the threshold | in the DTS algorithm is set to 0.2.
4.3 Evaluation Metries
Semantic Metries. We use BLEU-R (Papineni et al., 2002) to evaluate literal similarity between generated paraphrases and references. To further measure the semantic similarity, transformer" to encode the sentences into em- beddings and then calculate the cosine similarity between the paraphrase and the source/reference sentence as cos-S/cos-R.
sentence
Syntactic Metric. Following Bandel et al. (2022), we calculate the Tree Edit Distance (TED) (Zhang and Shasha, 1989) between the syntax trees of the paraphrase and the template to reflect how
ftttps:///gituub..om/lluuuu hhttpp:://ghhooocom///sssss//// 'https: /tttp::///tanfordnn hwtpddl............ooeo.
'https://Imsss.orgee http://mssyssooorooo////o/o 00023--0-30-vieumee hnttps:/huuggingaaaee co/ weoteeeeeeceeeeeeeeeeeeeeeeeeeee tr transformers all all-mpnet-base-y base-v2
github. westtubb.o//oreell io/CoreNLP/

---

Templates
BLEU-SI
BLEU-RT
IBLEUT
cos-S7
cos-R
ParaScore frmel
ParaScore
TED1
QQP-Pos
Ideal Templates
Ref-as- Template
22.430
60.260
43.722
0.850
0.914
0.856
0.948
0.096
Exemplar-as us Templaate
21,060
47.270
33.604
0.836
0.881
0.848
0.909
0.132
Available Templates in Practice
Random Template
15.030
10.480
5.378
0.728
0.703
0.794
0.739
0.214
Freq-R (lyyer et ul., 2018)
17.210
16.050
9.398
0.814
0.789
0.821
0.787
0.193
AESOP-R (Sun al., 2021)
24.430
13,230
5.698
0.789
0.742
0.813
0.769
0.236
SISCP-R (Yang al., 2022a)
28.180
18,580
9.228
0.851
0.809
0.837
0.825
0.221
QSTR (ours)
20.260"
22.080'
13.612
0.820'
0.851
0.828
I.MM-bosed paraphrase generation
Vicuna- (Chiang al., 2023)
13,302
5.060
1.388
0.851
0.762
0.861
0.786
ParaNMTTsmall
Ideal Templates
sef-as-Template
8.630
36.710
27.642
Exemplar-as: Template
7.000
Available Templates
Random Template
6.160
7.700
4,928
0.569
0.549
0.761
0.629
0,180
Freq-R (lyyer 2018)
8,740
12.510
8.260
0.671
0.653
0.834
0.726
0.226
AESOP-R R (Sun al., 2021)
7,970
8.770
5.422
0.649
0.613
0.816
0.691
0,185
SISCP-R (Yang al., 2022a)
7.960
10.570
6.864
0.675
0.643
0.837
0.749
0,142
QSTR (ours)
7.530'
13.970'
9.670
0.690'
0.685'
0.860'
0.769
0.123
LLM-based paraphrase generation
Vicuna-138 138 (Chiang et al., 2023)
10,564
5.279
2.110
0.731
0.666
0.861
0.724
Table I:
Performance of paraphrases generated with different kinds of templates using the AESOP model for SPG. Metrics with mean the higher value better, while | means the lower value is better. Results highlighted in bold and underline represent the best and the second-best results respectively. And results with mark statistically better than the most competitive method "SISCP-R" with < 0.05. For all retrieval methods in "Available Templates in Practice", we use the top-1 retrieved template for each source sentence to generate the paraphrase.
much the paraphrase is syntactically conformed with the template. In our experiments, we observe that lower TED values generally indicate that the templates are more suitable for the source sentence.
Results
5.1 Main Results
Table shows the performance of paraphrases with different templates two datasets, using the AE- SOP model for SPG. The results based on the SI- SCP model are presented in Appendix Totally, several conclusions can be drawn from the results:
Diversity Metric. BLEU-S calculates the BLEU scores between the paraphrase and the source sen- tence, whose lower values generally represent the paraphrases are more literally diverse from the source sentences.
(1) Under the reference- based metrics, iee. BLEU- R, cos-R, ParaScore and iBLEU, our QSTR significantly surpasses other baselines, which demonstrates that the templates retrieved by QSTR are closer to the ideal templates.
Comprehensive Metries. Based on BLEU-S and BLEU-R, calculate the iBLEU (Sun and Zhou, 2012) measure the overall quality of paraphrases by iBLEU aBLEU-R (1 BLEU-S, where we set a = 0.8 following Hosking et al. (2022). Additionally, ParaScore (Shen et al., 2022) is the state-of-the-art metric for paraphrase quality evalu- ation, which can comprehensively evaluate seman- tic consistency and expression diversity of para- phrases. Therefore, also use the reference- based and reference-free versions of ParaScore, i.e.. ParaScore,e/ and ParaScore/res: more convinc- ing metrics. Between them, ParaScorefrac free can het- ter reflect the quality of paraphrases when reference sentences are unknown in practical scenarios.
(2) The results of the reference-free metrics (i.e.. BLEU-S, cos-S and ParaScore/red also verify the superiority of QSTR compared to other methods in practical scenarios and QSTR per- form fully comparably with the human-labeled exemplar sentences (Exemplar-as -ermamaatee))).
QSTR also achieves much lower TED value than other retrieval methods. which indicates that the templates from QSTR are more suit- able for the source sentence and the generated paraphrases conform more with the templates.

---

Templates
Rep Rate (%)
M-BLEU
ParaScore
Training Objectives
PCC (%) in Dev
PCC (%) in Test
SISCP-R
14.01
30,70
0.840
QSTR
57.51
57.46
+ DTS (ours)
2.70
22.73
0.839
w/o Lema
56.70 (0.81
56.78 (0.6RL)
QSTR (purs)
13.48
32,90
w/o Lyank
56.53 (0.98L)
56.48 (0.98L)
(ours)
Table 3: Pearson correlation coefficients (PCC) between QSTR predictions and ParaScoreref scores under differ- ent training objectives the dev/test set of the QQP-Pos datasct.
Table The evaluation results the paraphrases gener- ated with the top-10 10 retrieval templates for each source sentence on the QQP-Pos dataset.
Although some methods can achieve lower BLEU-S scores than QSTR (e.g., "Random Tem- plates"), the corresponding cos-S scores are also significantly inferior, which means the generated paraphrases with these templates have poor seman- tic consistency with the source sentences.
Acceptance Rate (%)
Retrieval Methods
Quality
Diversity?
SISCP-R
3.689
3.472
43.07
QSTR (ours)
3.837
3.752
53.47
QSTR DTS (ours)
4.188
3,988
70.67
Table 4: The results of human evaluation.
Furthermore, despite the unfair comparison, also report the results of Vicuna- 13B, which con- ducts zero-shot paraphrasing without the need for templates. Although using a much smaller SPG model, QSTR can yield the closest performance to Vicuna- 13B among template retrieval methods. And the detailed analysis is shown in Appendix A. In conclusion, these results sufficiently demon- strate that our QSTR can provide more suitable templates to generate high-quality paraphrases.
Analysis
6.1 Ablation Study
To verify the effectiveness of the two training ob- jectives for QSTR, we further conduct an ablation study. Specifically, we calculate the Pearson cor- relation coefficient (PCC) between the predicted scores from QSTR and the true quality scores from ParaScoreref. Table 3 presents the results of QSTR when Lise Lrank is removed during training, which show that the correlations decline obviously without Lrank or Lmsee Thus, both objectives ben- efit the training of QSTR and they can be comple- mentary to each other.
5.2 Mutual Diversity of Multiple Paraphrases In this section, we evaluate the effectiveness of our DTS algorithm when generating multiple para- phrases for each source sentence. On the evaluation metries, we first calculate the lenennce-level repeti- tion rate (Rep-Rate) between the paraphrases gen- erated with the top-10 retrieval templates. And we use mudual-BLLR BLUU (M-BLEU) to measure the lit- eral similarity between multiple paraphrases, which averages the corpus-level BLEU scores between different paraphrases. Additionally, we also report the average ParaScore/res free: scores of 10 paraphrases for quality evaluation.
6.2 Human Evaluation
We further conduct the human evaluation on the paraphrases from the three retrieval methods (SISCP-R, QSTR, QSTR+DTS), Specifically, we randomly select 50 source sentences from the QQP- Pos test set and generate 5 paraphrases for each sen- tence using the AESOP model with templates from the three retrieval methods. Next, we let three anno- tators score each paraphrase from two aspects, i.e., the overall quality (I~55) and the diversity against the source sentence (1~55). The detailed guidelines are listed in Appendix C. Besides, we also define a paraphrase can be accepted if its quality score 2 4 and diversity score 2 3 and it is unique among the 5 paraphrases. The final evaluation results in Table 4 show that the paraphrases generated with QSTR have better quality and diversity under human eval- uation. Moreover, the DTS can further promote the quality and diversity of the paraphrases and largely improve the acceptance rate.
Table 2 presents the results of our DTS algo- rithm on two retrieval methods when retrieving 10 templates. The results showcase that equipped with DTS, the values of Rep-Rate and M-BLEU are decreased significantly, which means the DTS algo- rithm can effectively improve the mutual diversity between multiple paraphrases. Moreover, the sta- ble ParaScore scores of these paraphrases prove that the DTS algorithm has little impact on the qual- ity of the paraphrases. To sum up. by combining our QSTR with the DTS algorithm, the SPG mod- cls can generate multiple paraphrases with both high mutual diversity and high quality.

---

source senience; which hook would you recommend to improve english
Random Templates
SISCP-R
1. why did a't english improvee?
which are the best books to ingrove eaglish ? wan na ask someone plens
2. really too late to learn onglish."
what books would you recommend me improve my eaglish?
3. how english good for learning
what books would you recommend ?
4 what is best english book or book? ?
what books would you recommend english? ?
5 what are good english reading or learning book
which are the best english books?
which baok would
best improve english?
which books are the best
improve my english
7
how do actually learn english?
what book would you recommend imgrove english"
8
can unyone help me improve english?? ?
what book would you recommedd me to improve my english?
9
how has one improved his english?
what books should read to improve my english:
10
what would you suggest books improve onglish?
which book should
improve my english
QSTR (ours)
QSTR+ DTS (ours)
1
what is the best book for improving english
what the best book for improving english? ?
2. which is the hest book to improve english?
which are the best books to improve english
3.
some
best hooks improving english?
improve my spoken english?
4 what is the hest book improve my spoken english soon ?
which english books you would recommend improve english
5 what are some good books for improving english? ?
what books should i read to improve my spaken english? ?
6 what are some good books resources to improve english? ? books improve english?
which english book buy to improve my spoken english?
7
which are the which is the best book for improving english? ?
which the best book improve english
8
what english books you would rocommend to improve your pronunciation? ?
9 what are the best books for improving english? ?
9
which english book should read which books or magazines would you recommend me to improve my english
10 what are the best books improve english?
Table 5: Paraphrases generated by the AESOP model with templates retrieved by different methods. The paraphrases in red/blue/black represent that they are terible/repeatedeliibbble
Train Aug
Test Aug
Train & Test Aug
Retrieval Methods
MRPC
QQP
SST-2
SST-2
SST-2
Few-shot baseline
70.750
70,500
86.546
86.546
87.864=
+ Random templates
70.333
69,833
86.216
86.875
88.029
+ Freq
71.833
71.167
85.832
86.985
87.974
+ AESOP-R
71.500
72,000
87.040
87.095
88,138
+ -SSCCP-R
71.583
71.917
86,930
87.150
88.468
+ QSTR (ours)
72.250
72.417
87.864
87.534
88.523
Table 6: Test accuracies of downstream tasks (i.e., MRPC, QQP, and SST-2) after adding paraphrases with different templates respectively to the original baseline for data augmentation. "Train Aug" means generating paraphrases for the training samples as the training corpus. "Test Aug" represents generating paraphrases for test samples and conducting majority voting for the final predictions. "Train & Test Aug" combines the aforementioned two strategies. And results with the same mark I or are from the same model.
6.3 Case Study
fication tasks from GLUE (Wang et al., 2018) evaluation benchmarks. Then, we randomly sam- ple 200 instances from the train set fine-tune dert-base-uncased (Devlin et al., 2019) to ob- tain the baseline classifier as the few-shot baseline. In addition, we utilize the AESOP model with tem- plates from different retrieval methods to generate paraphrases for the train set and the test set respec- tively. Specifically, the augmented data for the train set are used train classifiers together with the original instances. As for the test set, we evalu- ate the augmented data as additional results, getting the majority voting as the final results. Moreover, we combine the aforementioned two strategics as a further attempt. Please refer to Appendix D for more training details.
We list 10 generated paraphrases of different re- trieval methods for the same source sentence in Table 5. Among them, random templates produce the most inferior paraphrases, which shows that current SPG models are very sensitive to different templates. With the templates retrieved by SISCP- R, the paraphrases may be similar to each other and also have some syntactic or semantic errors. In contrast, our QSTR performs better on the quality of the paraphrases and the DTS algorithm further improves the mutual diversity of paraphrases.
6.4 Applications on Downstream Tasks
To further test the performance of QSTR on down- stream tasks, we apply it to augment data for few- shot learning in text classification tasks. Specif- ically, we select SST-2, MRPC, and QQP classi-
The results in Table 6 present that our method brings the highest improvement over the baseline

---

compared to other methods on the three strategies. Specifically, "Train Aug" leads to better perfor- mance with our QSTR but not stably with other methods. "Test Aug" contributes to stable im- provements with all methods. And "Train & Test Aug" further improves the final performance. In conclusion, our QSTR showcases the best perfor- mance under all strategies, which indicates that our method can effectively promote the application values of SPG models on downstream tasks.
References
Elron Bandel, Ranit Aharonov, Michal Shmueli- Scheuer, llya Shnayderman, Noam Slonim, and Liat Ein-Dor. 2022. Quality controlled paraphrase gen- eration. In Proceedings of the 60th Anmual Meet- ing of the Association for Compulational Linguistics (Volume 1: Long Papers), pages 596-609, Dublin, Ireland. Association for Computational Linguistics.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019a. Controllable paraphrase gen- eration with syntactic exemplar. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5972-5984, Florence, Italy. Association for Computational Linguisties
7 Conclusion
In this work, we propose a quality-based template retriever (QSTR) to retrieve decent templates for high-quality SPG. Moreover, we develop diverse templates search (DTS) algorithm reduce the repetitions in multiple paraphrases. Experiments show that the SPG models can generate better para- phrases with the templates retrieved by our QSTR than other retrieval methods and our DTS algo- rithm further increases the mutual diversity of the multiple paraphrases without any loss of quality. Furthermore, the results of the human evaluation and the downstream task also demonstrate that our QSTR and DTS algorithm can retrieve better tem- plates and help SPG models perform more stably practice.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019b. A multi-task approach for disentangling syntax and semantics in sentence rep- resentations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 2453-2464, Minneapolis, Minnesota. Association for Computational Linguistics.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Limitations
Although Parascoreref (Shen et al., 2022)) has been the state-of-the-ard metric for the quality evalua- tion of paraphrases, it is still far from perfect as the supervision signal for QSTR. We will explore better metrics for evaluating the quality of para- phrases to guide the training of QSTR in future work. Moreover, we only utilize Vicuna- 13B for zero-shot paraphrase generation, which leads to an unfair comparison with other methods. In future work, We will try to finetune Vicuna-13B the SPG task and verify the effectiveness of our method with this new backbone.
Wee Chung Gan and Hwee Tou Ng. 2019. Improv- ing the robustness of question answering systems to question paraphrasing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6065-6075, Florence, Italy. Asso- ciation for Computational Linguistics.
Silin Gao, Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. Paraphrase augmented task-oriented dialog genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 639-649, Online. Association for Computational Lin- guistics.
A Acknowledgements
The research work described in this paper has been supported by the National Natural Science Foun- dation of China (NSFC) via grants 61976016 and 61976015 and the Talent Fund of Beijing Jiaotong University via grant 2023XKRC006. The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to im- prove this paper.
Tom Hosking and Mirella Lapata. 2021. Factorising meaning and form for intent-preserving paraphrasing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1418, Online. Association for Computational Linguistics.

---

Tom Hosking, Hao Tang, and Mirella Lapata. 2022. Hi- erarchical sketch induction for paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Pupers), pages 2489-2501, Dublin, Ireland. Association for Computational Linguistics.
Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNIP), pages 5569-5580, Hong Kong, China. As- sociation for Computational Linguistics.
Yunlong Liang, Fandong Meng, Ying Zhang. Yufeng Chen, Jínan Xu, and Jie Zhou. 2021. Infusing multi- source knowledge with heterogeneous graph neural network for emotional conversation generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13343-13352.
Kuan-Hao Huang and Kai-Wei Chang. 2021. Gener- ating syntactically controlled paraphrases without using annotated parallel pairs. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 1022-1033, Online. Association for Computa- tional Linguistics.
Yunlong Liang, Fandong Meng. Ying Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2022. Emotional conversation generation with heterogeneous graph neural network. Artificial Intelligence, 308: 103714, 14.
Kuan-Hap Huang, Varun Iyer, Anoop Kumar, Sriram Venkatapathy, Kai- Wei Chang, and Aram Galstyan. 2022. Unsupervised syntactically controlled para- phrase generation with Abstract Meaning Represen- tations. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 1547-1554, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Nitin Madnani and Bonnie J. Dorr. 2010. Generat- ing phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguisties. 36(3):341-877
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long Papers), pages 1875-1885, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language pro- cessing toolkit. In Proceedings of 52nd Annual Meet- ing of the Association for Computational Linguis- fics: System Demonsirations, pages 55-60, Balti- more, Maryland. Association for Computational Lin- guistics.
Shailza Jolly, Tobias Falke, Caglar Tirkaz, and Daniil Sorokin. 2020. Data-efficient paraphrase generation to bootstrap intent classification and slot labeling for new features in task-oriented dialog systems. In Proceedings of the 28th International Conference on Computational Linguistics: Industry Track, pages 10-20, Online. International Committee Compu- tational Linguistics.
Subhadarshi Panda, Caglar Tirkaz, Tobias Falke, and Patrick Lehnen. 2021. Multilingual paraphrase gener- ation for bootstrapping new features in task-oriented dialog systems. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 30-39. Online. Association for Computa- tional Linguistics.
Ashutosh Kumar, Kabir Ahuja, Raghuram Vadapalli, and Partha Talukdar. 2020. Syntax-guided controlled generation of paraphrases. Transactions of the Asso- ciation for Computational Linguistics, 8:329-345.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
BART: Denoising sequence-to-sequeee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguistics.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofci Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for sequence-oo-Sequeneeee---r-e of the Association for Computational Linguistics: EMNLP 2020, pages 2401-2410 Online. Association for Computational Linguistics.
In Findings
Yunlong Liang. Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2019. A novel aspect-guided deep transition model for aspect based sentiment analysis. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language
Lingfeng Shen, Lemao Liu, Haiyun Jiang, and Shuming Shi. 2022. On the evaluation metrics for paraphrase generation. In Pruceedings of the 2022 Conference

---

on Empirical Methods in Natural Language Process- ing. pages 3178-3190, Abu Dhabi, United Arab Emi-
A
The experiment based on Vicuna-13B
rates. Association for Computational Linguistics.
Vicuna- 13B (Chiang et al., 2023) is large-scale model that trained by fine-tuning LLaMA user- shared conversations collected from ShareGPT. And preliminary evaluation using GPT-4 as a judge shows Vicuna-13B outperforms other models like LLaMA and Stanford Alpaca in more than 90% cases (Chiang et al.. 2023). To explore its per- formance on paraphrase generation, We use the instruction "Please give ten paraphrases of the next sentence in English, the input sentence. These para- phrases should have the same meaning and diverse syntactic structures with the given sentence." obtain the generated paraphrases from Vicuna-13BB The evaluation results on automatic metrics (as shown in Table 1) represent that Vicuna- 13B can generate competitive paraphrases with our QSTR. Additionally, through our observation of specific cases, Vicuna- tends to add more additional in- formation or expands the original sentence length during generating paraphrases. Several cases gen- erated from Vicuna- 13B are shown in Table The advantage of this performance is that the generated paraphrases are more diverse against the source sentence. while the disadvantages are that the sen- tences become more redundant and the syntactic structures are not uncontrollable.
Hong Sun and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In Proceed- ings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 38 42, Jeju Island, Korea. Association for Computational Linguistics.
Jiao Sun, Xuezhe Ma, and Nanyun Peng. 2021, AESOP: Paraphrase generation with adaptive syntactic control. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 5176-5189, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE:
A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP,, pages 353-355, Brussels. Belgium. Association for Com- putational Linguistics.
Erguang Yang, Chenglin Bai, Deyi Xiong, Yujie Zhang, Yao Meng, Jinan Xu. and Yufeng Chen. 2022a. Learning structural information for syntax-controlles paraphrase generation. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2079-2090, Seattle, United States, Association for Computational Linguistics.
Erguang Yang, Mingtong Liu, Deyi Xiong, Yujie Zhang, Yao Meng. Changjian Hu, Jinan Xu, and Yufeng Chen. 2021a. Syntactically- innoomeedd unsupervised paraphrasing with non- con-palllle data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 2594-2604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
B
The Results of SI-SCP
Firstly, to show the applicability of QSTR, the QSTR model used for the SI-SCP model is the same one used for the AESOP model (QSTR- based-on-AESOP), It is trained using the para- phrases generated by the AESOP model, which may leave gap from the paraphrases from the SI-SCP model. Thus, also conduct the more effective approach, using the SI-SCP model generate paraphrases during the training process RQSTR-based-on-S11 -S--CCC. Table shows the per- formance of paraphrases with different templates from QSTR and other baselines on both datasets based on the SI-SCP backbone. The results show- case that our QSTR exhibits significant superiority compared to other baselines, and the -bseed on-SI-SCP has better performance on most met- rics than OSTR-based-on- AESOO.. However, the promising results with SaTTR-based-o--ESS. in- dicate that our QSTR may be directly applied to other SPG models without retraining.
Haoran Yang, Wai Lam, and Piji Li. 2021b. Contrastive representation learning for exemplar-guides para- phrase generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4754-4761, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Haibo Zhang, Xue Zhao, Wenqing Yao, and Boxing Chen. 2022b. GCPG: general framework for controllable paraphrase generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4035 4044, Dublin, Ireland. Association for Computational Linguistics.
Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM Journal on Computing, 18(6): 125-1

---

source sentence: why do some people like cats more than dogs
What the reason behind the preference some individuals cats over dogs?
2 What makes certain people have greater fondness for cats compared to dogs?
3
Why are some people more inclined towards cats rather than dogs?
4
What drives some individuals prefer cats dogs?
5
What is the explanation for the fact that some people prefer cats to dogs?
Why do certain people have a stronger affinity cats compared to dogs?
7
What the reason for the disparity in preferences between people who prefer cats and those who prefer dogs?
8
What sets apart the individuals who prefer cats from those who prefer dogs?
9
Why do people find cats more appealing than dogs?
10
What are the factors that contribute to the variance in preferences for cats and dogs among people?
source sentence: how can send private message through quora"
I
How do I privately message someone on Quora?
2
Can you tell me the process for sending private message Quora?
3
I'm trying to send confidential message on Quora, but I'm not sure how How can you help?
there way send a discreet message on Quora? so, would go about doing
5
How do I go about sending secure messuge Quora?
6
I'm trying to send a message on Quora that only the recipient can see How can do that?
7
Can you explain the sleps send private message Quora?
8
I'm trying to send message on Quora that won't show up in someone's 's feed How can I do that?
How do I send message on Quora that won't be visible anyone elsc?
10
Can you provide me with the procedures to send private message Quora?
Table 7: Paraphrases directly generated by Vicuna-13B.
C
Guidelines for human evaluation
or whether the syntactic structure is the same. And the larger the score, the higher the diversity. The detailed guidelines
The overall quality evaluates paraphrases from the perspectives of grammar correctness and seman- tic consistency with the source sentence, and the larger the score, the higher the quality. The detailed guidelines are as follows.
follows.
"5" means the paraphrase has very different syntax from the source sentence and adopts many new words and phrases.
"5" means the paraphrase is fully grammati- cally correct and completely semantically con- sistent with the source sentence.
"4" means the paraphrase has a new syntax against the source sentence and revises some original words.
"4" means the paraphrase has slight gram- matical error, but still maintains the correct semantics, and can also be considered valu- able paraphrase.
"3" means the paraphrase has new syntax against the source sentence but still adopts original words or phrases.
"2" means the paraphrase has the same expres- sion as the source sentence but adopts few new words phrases.
"3" means the paraphrase has slight gram- matical error and a minor semantic deviation.
"2" means the paraphrase has serious gram- matical error and a major semantic deviation, but is still complete sentence.
"1" means the paraphrase almost identical to the source sentence.
"1" means the paraphrase is not complete sentence is totally irrelevant the source sentence.
D Training details of Downstream Tasks
Since MRPC does not provide the official dev set, randomly sample 1200 instances from the of- ficial test set as the final testing set and use the rest instances as the dev set. QQP adopts the same settings. As for the SST-2, we directly use the
The diversity evaluates whether the paraphrase has diverse expressions compared to the input sen- tence based on whether the words used are the same

---

Templates
BLEU-SI
BLEU-R
IBLEUT
cos-ST
cos-R
ParaScore
ParaScore, T TED1
QQP-Pas
Ideal Templates
Ref-as-Template
23.810
52.760
37,446
0.838
0.888
0.844
0,920
0.136
Exemplar-as-Tepplus
23.160
42,730
29.552
0.829
0.860
0.840
0.888
0.181
Available Templates Practice
Random Template
20,540
15.920
8.628
0.755
0.732
0.802
0.768
0.301
R (yyee al., 2018)
22.120
18.190
10.128
0.795
0.776
0.811
0.790
0.208
AESOP-R (Sun et al., 2021)
28.040
17.190
8.144
0.798
0.761
0.806
0.790
0.323
SISCP-R (Yang et al., 2022a)
29.230
21.050
10.994
0.839
0.805
0.820
0.817
0.284
-TT--based-on-A---- (ours)
21.520
22.540
13.728
0.815
0.804
0.833
0.820
0.220
SSTR-based-nn-I-SCFF (ours)
24.580
22,500
13.084
0.828
0.810
0.850
0.829
0.214
ParrNNMTssaal
Ideal Templates
Ref-as-Template
12.060
26.130
18.492
0.711
0.758
0.837
0.158
Exemplar-as- Temmpllteeeeee
Avoilable Templates in Practice
Random Template
10.110
3.474
0.562
0.748
Freq-R (lyyer al., 2018)
14.030
9,520
0.607
0.786
0.312
AESOP-R (Sun al., 2021)
13.340
9,470
0.664
0.615
0.793
0.231
SISCP-R (Yang et al., 2022aa
17.200
0.807
OSTR-based-on-AASOO (ours)
13,830
8,432
0.694
0.675
0.823
0.768
0.157
DSTR-based-on-SI-SCP (ours)
16.070
12.270
7,002
0.710
0.680
0.833
0.781
0.167
Table Performance paraphrases with different kinds templates based SI-SCP backbone. Metrics with t means the higher value is better, while means the lower value is better. And the bold and underline represent the best and the second best respectively. For all retrieval methods in "Available Templates in Practice", we use the top-1 retrieved template each source sentence generate the paraphrase.
official test set and dev set. All classifiers are ini- tialized with bert-base bart-base-uncased (Devlin et al., 2019). The batch size is 128, 64, and 64 for MRPC, QQP, and SST-2 respectively. The learning rate is 10-4 and the number of training epochs is 20. And we use AdamW W optimizer with weight decay being 10 10-5


Reviews:
[{"review_id": "8fe91Wtwq1", "reviewer": "Reviewer_h28a", "summary": "", "questions": "1. The current objective just takes into account the syntactic signal compatibility implicitly. Would it help to include a syntactic signal objective explicitly while training QSTR?\n    \n2. In downstream tasks, would it be better to use the generations as augmentations or generate multiple instances for test set, evaluate them, get majority voting and display the final result ?\n    \n3. What are the guidelines for human evaluation ?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your constructive comments and suggestions, they are exceedingly helpful for us to improve our paper. We will response to your concerns point-by-point in the following.\n\n---\n\n> **Q1**：Evaluation on downstream task that is similar to original task. Using paraphrase generations from DTS to evaluate paraphrase detection is helpful but not sufficient to show that diverse paraphrases are helpful for downstream tasks. It would be better to show results on non-paraphrastic tasks like sentiment classification, or NLI.\n\n**A1**：We are very grateful for your suggestion, and we have conducted additional experiments on SST-2 (a sentiment classification dataset from GLUE). The results in the table below present that our methods perform better compared to other baselines, which demonstrates that our methods are helpful for downstream tasks. And we will add these experiments to our paper in the new version.\n\n| Retrieval Methods\t| SST-2 (%) |\n| -------------- | :---------------: |\n| Few-shot baseline\t | 86.546 |\n| &emsp;+Random templates\t | 86.216 |\n| &emsp;+Freq-R\t  | 85.832 |\n| &emsp;+AESOP-R\t | 87.040 |\n| &emsp;+SISCP-R\t | 86.930 |\n| &emsp;+QSTR (ours) | \t**87.864** |\n\n---\n\n> **Q2**：Statistical significance testing missing.\n\n**A2**：Thanks very much for your suggestion, and we have added the statistical significance testing, which shows that our QSTR significantly surpasses other methods with $p<0.05$. We will supplement the results in the new version.\n\n---\n\n> **Q3**：The current objective just takes into account the syntactic signal compatibility implicitly. Would it help to include a syntactic signal objective explicitly while training QSTR?\n\n**A3**：In preliminary experiments, we tried to use Tree Edit Distance (TED) between the parse trees of paraphrases and templates as the syntactic supervision signal to train QSTR. However, we failed for the following reasons. \n- For each source sentence, the TED values of the generated paraphrases tend to be similar (or equal to 0) at most times, which makes it difficult to train QSTR.\n- The process of syntax parsing is slower than calculating ParaScore by about 5 times, which largely lengthens the training time.\n- In this paper, our goal is to improve the quality of the generated paraphrases using our retrieved templates, however, we found that lower TED values may not necessarily correspond to higher-quality paraphrases. For example, using the syntax parse tree of the source sentence as the template will have a very low TED value, but it will make the generated paraphrase too similar to the source sentence.\n\nNevertheless, inspired by your question, we will explore the possibility of combining ParaScore and TED as a new objective to retrieve more compatible templates in future work.\n\n---\n\n> **Q4**：In downstream tasks, would it be better to use the generations as augmentations or generate multiple instances for test set, evaluate them, get majority voting and display the final result ?\n\n**A4**：We have compared two methods on the SST-2 dataset, including data augmentation for the training set and the voting strategy you mentioned for the test set. The results in the table below showcase that:\n- Data augmentation for the training set performs slightly better with our QSTR but stably with other methods.\n- Majority voting for the test set leads to stable improvement for all methods.\n- Our QSTR shows the best performance under both settings.\n\n| Retrieval Methods\t| for train (%) | for test (%) |\n| -------------- | :---------------: | :---------------: |\n| Few-shot baseline\t | 86.546 | 86.546 |\n| &emsp;+Random templates\t | 86.216 | 86.875 |\n| &emsp;+Freq-R\t  | 85.832 | 86.985  |\n| &emsp;+AESOP-R\t | 87.040 | 87.095 |\n| &emsp;+SISCP-R\t | 86.930 | 87.150 |\n| &emsp;+QSTR (ours) | \t**87.864** | **87.534** |\n\n---\n\n> **Q5**：What are the guidelines for human evaluation ?\n\n**A5**：In human evaluation, we let three annotators score each paraphrase from two aspects, *i.e.*,  the overall quality (1-5) and the diversity against the source sentence (1-5).\n\n(a) The overall quality evaluates paraphrases from the perspectives of grammar correctness and semantic consistency with the source sentence, and the larger the score, the higher the quality. The detailed guidelines are as follows.\n\n- \"5\" means the paraphrase is fully grammatically correct and completely semantically consistent with the source sentence. \n- \"4\" means the paraphrase has a slight grammatical error, but still maintains the correct semantics, and can also be considered a valuable paraphrase.\n- \"3\" means the paraphrase has a slight grammatical error and a minor semantic deviation.\n- \"2\" means the paraphrase has a serious grammatical error and a major semantic deviation, but is still a complete sentence.\n- \"1\" means the paraphrase is not a complete sentence or is totally irrelevant to the source sentence.\n\n(b) The diversity evaluates whether the paraphrase has diverse expressions compared to the input sentence based on whether the words used are the same or whether the syntactic structure is the same. And the larger the score, the higher the diversity. The detailed guidelines are as follows.\n\n- \"5\" means the paraphrase has a very different syntax from the source sentence and adopts many new words and phrases.\n- \"4\" means the paraphrase has a new syntax against the source sentence and revises some original words.\n- \"3\" means the paraphrase has a new syntax against the source sentence but still adopts original words or phrases.\n- \"2\" means the paraphrase has the same expression as the source sentence but adopts a few new words or phrases.\n- \"1\" means the paraphrase is almost identical to the source sentence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and the method is easy to follow\n    \n2. Comprehensive evaluation and Comparison against competitive baselines, including Vicuna-13B (LLM).", "reasons_to_reject": "1. Evaluation on downstream task that is similar to original task. Using paraphrase generations from DTS to evaluate paraphrase detection is helpful but not sufficient to show that diverse paraphrases are helpful for downstream tasks. It would be better to show results on non-paraphrastic tasks like sentiment classification, or NLI.\n    \n2. Statistical significance testing missing.", "questions_to_authors": "1. The current objective just takes into account the syntactic signal compatibility implicitly. Would it help to include a syntactic signal objective explicitly while training QSTR?\n    \n2. In downstream tasks, would it be better to use the generations as augmentations or generate multiple instances for test set, evaluate them, get majority voting and display the final result ?\n    \n3. What are the guidelines for human evaluation ?", "paper_topic_and_main_contributions": "Prior methods in Syntactic Paraphrase generation focused on generating sentences that conform to a particular syntactic style that was given as an input to the generation models. While those works aimed at solving the task of syntax-guided paraphrase generation via strong semantic and syntactic encoders and decoders, they lacked a concrete method for choosing the target syntactic styles. The paper proposes a syntactic template retriever model to select not only a good set of high quality diverse structures for syntax-guided paraphrase generation but also highly compatible ones.\n\nTo address the issue of lack of good syntactic templates, the paper proposes a method called Quality-based Syntactic Template Retriever (QSTR). QSTR involves an architecture that calculates the interaction between source sentence and syntactic templates, to come up with a compatibility signal that is brought close to paraphrase quality scores of the output sentence using a metric called ParaScore. Two objectives namely mse and rank loss help in achieving the desired result.\n\nAdditionally the paper proposes Diverse Templates Search (DTS) algorithm that prevents mode collapse between retrieved candidates. The paper shows better results than previous heuristic based template selection mechanisms. Additionally the paper reports good results on human evaluation and a downstream classification task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SMTDq3cfDy", "reviewer": "Reviewer_TgL2", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your positive comments and appreciating the contributions of our work. Next, we clarify the reason why templates retrieved from QSTR seem to be less effective when using SISCP as the backbone SPG model. \n\nActually, to show the applicability of QSTR, the QSTR model used for the SISCP model is the same one that used for the AESOP model. It is trained using the paraphrases generated by AESOP model, which may leave a gap from the paraphrases from the SISCP model. Thus, the more effective approach is using SISCP model to generate paraphrases during the training process. \n\nIn the table below, we have complemented the corresponding results with SISCP as the SPG model, which showcases that the QSTR trained using paraphrases generated by SISCP has better performance on most metrics than the original one trained using the paraphrases from AESOP and still exhibits significant superiority compared with other baselines. And we will revise the corresponding results to our paper in the new version.\n\n> **QQP-Pos**\n\n| Templates |  BLEU-S↓| BLEU-R↑ | iBLEU↑ | cos-S↑ | cos-R↑ | $ParaScore_{free}$ ↑| $ParaScore_{ref}$↑ | TED↓ |\n|--------------- | :----: | :----: | :----: | :----: | :----: | :----: |  :----: | :----: | \n | Random Templates  | **20.540**  | 15.920  | 8.628  | 0.755 |  0.732  | 0.802 |  0.768  | 0.301 | \n | Freq-R   | 22.120  | 18.190  | 10.128  | 0.795  | 0.776 |  0.811 |  0.790  | **0.208** | \n | AESOP-R |   28.040  | 17.190  | 8.144  | 0.798 |  0.761 |  0.806 |  0.790 |  0.323 | \n | SISCP-R  |  29.230  | 21.050  | 10.994 |  **0.839** |  0.805  | 0.820 |  0.817  | 0.284 | \n| QSTR-based-on-AESOP (original) |21.520 \t | **22.540** \t | **13.728** \t | 0.815 \t | 0.804 \t | 0.833 \t | 0.820 \t | 0.220  | \n| QSTR-based-on-SISCP (new) | 24.580  | \t22.500 \t | 13.084 \t | 0.828↑ \t | **0.810**↑ \t | **0.850** ↑\t | **0.829**↑\t | 0.214↓  | \n\n---\n\n> **ParaNMT-small**\n\n| Templates |  BLEU-S↓| BLEU-R↑ | iBLEU↑ | cos-S↑ | cos-R↑ | $ParaScore_{free}$ ↑| $ParaScore_{ref}$↑ | TED↓ |\n|--------------- | :----: | :----: | :----: | :----: | :----: | :----: |  :----: | :----: | \n | Random Templates  | **10.110**  | 6.870  | 3.474  | 0.596  | 0.562  | 0.748  | 0.653  | 0.256 | \n | Freq-R   | 14.030  | 9.520  | 4.810  | 0.643  | 0.607 |  0.786 |  0.700  | 0.312 | \n | AESOP-R  |  13.340  | 9.470  | 4.908  | 0.664  | 0.615  | 0.793  | 0.711 |  0.231 | \n | SISCP-R   | 17.200  | 11.810  | 6.008  | **0.723** |  0.676  | 0.807  | 0.779  | 0.190 | \n| QSTR-based-on-AESOP (original) | 13.160  | \t**13.830** \t | **8.432** \t | 0.694 \t | 0.675 \t | 0.823 \t | 0.768 \t | **0.157**  | \n| QSTR-based-on-SISCP (new) | 16.070  | \t12.770 \t | 7.002  | \t0.710↑  | \t**0.680**↑\t | **0.833**↑ \t | **0.781**↑  | 0.167  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper tackles the often overlooked aspect of syntactically controlled paraphrase generation i.e. how to select appropriate templates, since randomly selected templates might not be compatible with a source sentence. The proposed method is intuitive and can be plugged into any SPG model to improve the performance. I think both tackling an important problem and providing a scalable solution makes the work valuable to the community, especially researchers working on paraphrase generation or related problems.\n\n2. I also really appreciate the fact that the authors highlight as well as tackle the pitfalls of a quality based retriever approach for selecting templates i.e. when generating multiple paraphrases from a single source sentence, the model might end up selecting syntactically similar templates leading to low diversity in generated paraphrases. The proposed Diverse Template Search algorithm while simple, provides an effective way of combating this issue as evident from the experimental results that show substantial improvement in diversity.\n\n3. The evaluation is very thorough including a wide variety of baselines (including an open source LLM), automatic evaluation metrics as well as human evaluation. This helps answer most questions the reader would have about the effectiveness of the proposed approach. Evaluation of the method when acting as a method for data-augmentation for downstream tasks is also well appreciated.", "reasons_to_reject": "I didn't find any clear weakness in the work, except that the proposed method seems to be less effective when used with SISCP SPG model compared to the performance gains observed for AESOP.", "questions_to_authors": "None", "paper_topic_and_main_contributions": "The paper tackles the problem of selecting (syntactic) templates for syntactically controlled paraphrase generation (SPG) and proposes a quality-based retriever that ranks the quality of to-be-generated paraphrase from a source sentence given a template. The retriever model is a two-tower siamese network that scores a syntactic template for a provided source sentence. Further, the authors propose a Diverse Templates Search algorithm to ensure diversity in generated paraphrases when generating multiple paraphrases from a single source sentence. Through experiments on QQP-Pos and ParaNMT-small datasets, it is shown that augmenting existing SPG methods with the proposed retriever results in a substantial improvements in the performance on both automated and human evaluation. Further, the proposed method when used for data augmentation also helps in improving downstream task performance for paraphrase detection datasets (MRPC and QQP).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "JSAhJ47BxI", "reviewer": "Reviewer_7mbF", "summary": "", "questions": "Question A: Does the retrieval step affect the training/inference time of the model?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your time and effort to review our paper, but we are afraid that you may have some misunderstandings about our paper. To clarify your misunderstandings, we will provide point-by-point explanations on your comments in the following.\n\n---\n\n> **Q1**：Some parts of the paper are not clearly written. For example, one of the focuses of this paper is the template, but it does not emphasize the information about the template (readers can only infer from the figure that the template refers to the syntax tree), and the choice of template is not fully discussed, such as the part of speech sequence, truncated syntax tree (after all, using the whole syntax tree may lead to too long input sequence).\n\n**A1**：(1) Actually, we have introduced that we use syntax parse trees as templates in this paper in Footnote 2 of Page 1. To emphasize it, we will move them to the main content following your insightful suggestions.\n\n(2) On the choice of the template, we choose the syntax parse tree in this paper as it is one of the mainstream forms of the template in previous work on SPG. Besides, we totally agree with your concern about the too-long input, and thus we truncated all parse trees by height 4 and linearised them following previous work (Yang et al, 2022) as we have discussed and listed in Lines 316-318. In addition, we also present an example in Footnote 3 of Page 3 for a better understanding of their forms.\n\nErguang Yang et al. Learning Structural Information for Syntax-Controlled Paraphrase Generation. NAACL'22 Findings.\n\n---\n\n> **Q2**：The task is not clearly defined. As far as I know, the controlled paraphrase generation task on these two data sets generally provides only data pairs [input sentence, output sentence] at the time of training. In the testing stage, an examplar is provided and the model is required to refer to examplar examples to paraphrase results that contain specific syntactic structures. Based on this, I do not quite understand why exemplar-as-template is used as the baseline (previous articles generally Exemplar-as-Output, see article [1][2]).\n\n**A2**：In this paper, we are not aiming at proposing a new model or method for syntactically controlled paraphrase generation, but **tackling a practical issue for existing SPG models on the lack of qualified templates in practice**. To this end, we propose QSTR in this paper, which can be plugged into any existing SPG models to retrieve templates for high-quality paraphrases. \n\nIn the main experiments, to simulate practical scenarios, we assume that there is no template for sentences in the test set and we need to retrieve templates from the template library with our QSTR as well as other existing solutions for obtaining templates. Then we use the same SPG model (e.g., AESOP or SISCP) to generate paraphrases based on these templates and compare the performance of the paraphrases. Under this setting, **\"exemplar-as-template\"**, which means using the parse trees of the exemplar sentences as templates, **is just the simulation of the human-annotated templates in practice**. As shown in Table 1/8, the templates retrieved by our QSTR achieve fully comparable performance with these human-annotated templates on reference-free metrics.\n\n---\n\n\n> **Q3**：Baselines are seriously lacking and comparisons may be unfair, and the authors do not include such strong baselines [2][3][4] in the comparison. At the same time, from the conclusions and experimental results of these papers, the use of natural language sentences as templates is obviously superior to the use of syntactic trees, so I may not quite understand the reason for using syntactic trees as templates. In addition, the method in this paper has two RoBerta-based encoders, and the baselines compared are transformer based. It is hard to determine whether performance gains are due to the use of the PLMs or the method itself.\n\n**A3**：As we explained above, our goal is to retrieve qualified templates for each input sentence, so our baselines are template acquisition methods rather than other paraphrase models. However, the methods you mentioned aim to improve the controllability of the paraphrases given the annotated sentence exemplars as templates, which is inconsistent with our focus. Therefore, we are not comparing our RoBerta-based templated retrieval model with current Transformer-based SPG models, but with other template retrieval methods, and testing the performance of these retrieved templates when paraphrasing using the same SPG model (*e.g.*, AESOP and SISCP). \n\nAccording to the experimental results, the performance gains are due to our method being able to retrieve suitable templates, which can guide SPG models to generate better paraphrases.\n\nFurthermore, as reviewer TgL2 concluded, our QSTR can be plugged into any SPG models for better performance in practice, including the ones you mentioned that use sentence exemplars as templates. And we will explore this possibility in future work.\n\n---\n\n\n> **Q4**：The design of manual evaluation scores is not very reasonable, the author only evaluates quality and diversity, and lacks a separate evaluation of syntactic controllability, which is undoubtedly the focus of the task.\n\n**A4**：In terms of syntactic controllability, we think that compared to human evaluation, Tree Edit Distance (TED) between the syntax tree of the paraphrase and the template is a more precise metric to reflect how much the paraphrase is syntactically conformed with the template. Under this metric, our QSTR has shown substantial superiority compared to other template retrieval methods.\n\n---\n\n\n> **Q5**：Does the retrieval step affect the training/inference time of the model?\n\n**A5**：Like any other template retrieval method we have compared in experiments, the retrieval step will introduce the time cost of inference of the SPG models, which is inevitable in order to get better templates. However, the benefit of the two-tower architecture of our QSTR, we can encode all the candidate templates beforehand and largely improve the retrieval speed during application.\n\n---\n\nWe hope our responses will address your concerns and clarify the contributions of our work. We would greatly appreciate it if you could reassess our paper after reading our responses. Thanks again for your efforts and valuable comments on our paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It's an interesting point to consider generating diversity in the syntactically-controlled paraphrase generation task. \n2.  The tables and figures are pretty clear and easy to understand. Authors provide code and dataset for reproducibility.", "reasons_to_reject": "1. Some parts of the paper are not clearly written.  For example, one of the focuses of this paper is the template, but it does not emphasize the information about the template (readers can only infer from the figure that the template refers to the syntax tree), and the choice of template is not fully discussed, such as the part of speech sequence, truncated syntax tree (after all, using the whole syntax tree may lead to too long input sequence).\n2. The task is not clearly defined. As far as I know, the controlled paraphrase generation task on these two data sets generally provides only data pairs [input sentence, output sentence] at the time of training. In the testing stage, an examplar is provided and the model is required to refer to examplar examples to paraphrase results that contain specific syntactic structures. Based on this, I do not quite understand why exemplar-as-template is used as the baseline (previous articles generally Exemplar-as-Output, see article [1][2]).\n3. Baselines are seriously lacking and comparisons may be unfair, and the authors do not include such strong baselines [2][3][4] in the comparison. At the same time, from the conclusions and experimental results of these papers, the use of natural language sentences as templates is obviously superior to the use of syntactic trees, so I may not quite understand the reason for using syntactic trees as templates. In addition, the method in this paper has two RoBerta-based encoders, and the baselines compared are transformer based. It is hard to determine whether performance gains are due to the use of the PLMs or the method itself.\n4. The design of manual evaluation scores is not very reasonable, the author only evaluates quality and diversity, and lacks a separate evaluation of syntactic controllability, which is undoubtedly the focus of the task.\n\n[1] Syntax-Guided Controlled Generation of Paraphrases.\n\n[2] Generative Pre-training for Paraphrase Generation by Representing and Predicting Spans in Exemplars\n\n[3] GCPG: A General Framework for Controllable Paraphrase Generation\n\n[4] HCPG: a highlighted contrastive learning framework for exemplar-guided paraphrase generation", "questions_to_authors": "Question A: Does the retrieval step affect the training/inference time of the model?", "paper_topic_and_main_contributions": "This paper focuses on the syntactically-controlled paraphrase generation task and proposed a new method QSTR and DTS algorithm. In short, this method retrieves syntax-parse-tree-based templates to improve the quality of paraphrases.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See Reasons To Reject", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reasons to accept
(1) The paper addresses the important issue of syntactically controlled paraphrase generation, specifically focusing on selecting appropriate templates.
(2) The proposed method is intuitive and can be applied to any Syntactically Controlled Paraphrase Generation (SPG) model to enhance its performance.
(3) The paper highlights and addresses the potential problem of low diversity in generated paraphrases when using a quality-based retriever approach to select templates.
(4) Thorough evaluation includes a wide range of baselines, automated metrics, and human evaluations, providing comprehensive insights into the proposed approach's effectiveness, including its role in data augmentation for downstream tasks.

Reasons to Reject:
(1) The paper's proposed method appears to be less effective when used with the SISCP SPG model compared to the performance gains observed for AESOP.
(2) Some parts of the paper are not clearly written, making it challenging for readers to fully understand certain aspects, such as the choice of templates and their role.
(3) The task definition could be clearer, and the choice of exemplar-as-template as a baseline may not align with the typical setup for controlled paraphrase generation tasks.
(4) The lack of certain strong baselines from previous related work and the use of two RoBERTa-based encoders make it challenging to determine whether performance gains are due to the proposed method itself or other factors.
(5) The manual evaluation scores do not include a separate evaluation of syntactic controllability, which is a central aspect of the task.

Two of three reviewers agree that the paper is strong and exciting, however one reviewer gave ambivalent marks for the paper's strength and excitement. The authors also carefully responded to each reviewer's comments and questions by providing additional information, detailed explanations, or new experiments. I have read through the responses from the authors and consider their responses to be reasonable and valid.
Decision: Accept-Main