Paraphrase Types for Generation and Detection
Authors: Jan Philip Wahle
Bela Gipp
Terry Ruas
Venue: EMNLP 2023 Main (2023)
Keywords: paraphrase generation and detection
paraphrase types
paraphrasing tasks

Abstract: Current approaches in paraphrase generation and detection heavily rely on a single general similarity score, ignoring the intricate linguistic properties of language. This paper introduces two new tasks to address this shortcoming by considering paraphrase types - specific linguistic perturbations at particular text positions. We name these tasks Paraphrase Type Generation and Paraphrase Type Detection. Our results suggest that while current techniques perform well in a binary classification scenario, i.e., paraphrased or not, the inclusion of fine-grained paraphrase types poses a significant challenge. While most approaches are good at generating and detecting general semantic similar content, they fail to understand the intrinsic linguistic variables they manipulate. Models trained in generating and identifying paraphrase types also show improvements in tasks without them. In addition, scaling these models further improves their ability to understand paraphrase types. We believe paraphrase types can unlock a new paradigm for developing paraphrase models and solving tasks in the future.

Paraphrase Types for Generation and Detection
Jan Philip Wahle, Bela Gipp, Terry Ruas
University of Göttingen, Germany (wahle, gipp, ruas)@uni -goettineeetnnnnn de
Abstract
The President gave speech about her plan to change the Constitution
The President gave speech about plan to change Constitution
Current approaches in paraphrase generation and detection heavily rely single general similarity score, igmoring the intricate linguistic properties of language. This paper introduces two new tasks to address this shortcoming by considering paraphrase rypes- specific linguis- tic perturbations particular text positions. We name these tasks Paraphrase Type Gen- eration and Paraphrase Type Detection. Our results suggest that while current techniques perform well in a binary classification scenario, i.e. paraphrased not, the inclusion of fine- grained paraphrase types poses significant challenge. While most approaches are good at generating and detecting general semantic sim- ilar content, they fail to understand the intrin- sie linguistic variables they manipulate. Mod- els trained in generating and identifying para- phrase types also show improvements in tasks without them. In addition, scaling these mod- els further improves their ability to understand paraphrase types. We believe paraphrase types unlock new paradigm for developing para- phrase models and solving tasks in the future.
Disthssis
The President given speech about her plan change the Constitution
Similarity
Palarity Subsiik.tion
She has given speech about her plan to change the Constitution
Laxicon Change
She has given talk about plan change the Constitution
She has given a talk about her plan to change Constitution
Figure 1: Comparison of current paraphrase tasks (left) and our proposal towards paraphrase types (right).
Generative models cannot be asked to perform cer- tain types of perturbations, and detection models are unable to understand which paraphrase types they detect (Egonmwan and Chali, 2019; Meng et al., 2021; Ormazabal et al., 2022; Vizcarra and Ochoa-Luna, 2020), or they learn limited language aspects (e.g., primarily syntax (Chen al., 2019;
1 Introduction
Paraphrases are texts expressing identical meanings that use different words or structures (Vila et al., 2015, 2014; Zhou and Bhat, 2021). Paraphrases exhibit humans' complex language's 's nature and diversity, as there infinite ways transform one text into another without altering its meaning. For example, one can change text's
Goyal and Durrett, 2020; Huang and Chang, 2021)). The shallow notion of what composes paraphrases used by these systems limits their understanding of the task and makes it challenging to interpret detection decisions in practice. For example, al- though high structural and grammatical similaritics can indicate plagiarism, detection systems of- not concerned with the aspects that make texts segments alike (Ostendorff et al., 2022; Wahle et al., 2022a).
morphology: "Who they could might be?",
synlax: "He drew go was drawn by him.".
Given the limitations of current paraphrase gen- eration and detection tasks, their proposed solu- tions are also constrained (Foltýnek al.. 2020). Figure gives an example of the difference be- tween current tasks and their true linguistic di- versity. Therefore, we propose two new tasks
lexicon: "She liked enjoyed it.".
Nonetheless, current paraphrase generation and detection systems are yet unaware of the lexical variables they manipulate (Zhou and Bhat, 2021).

---

explore the role of paraphrase types, namely Paraphrase Type Generation and Paraphrase Type Detection. the generation task, a model has to generate a paraphrased text for specific segments considering multiple paraphrase types ($3.1). For the detection task, paraphrased seg- ments must be classified into one more para- phrase types (e.g., lexico-syntactic-basee changes) ($3.2). These tasks complement existing ones (without paraphrase types), enabling more gran- ular assessments of paraphrased content.
investigate the correlation between paraphrase types, generation and detection performance of existing solutions, and scaling experiments to explore our proposed tasks;
make the source code and data to reproduce our experiments publicly available;
provide interactive demo to generate para- phrases with types:
Related Work
The shift from traditional paraphrase tasks to- wards a more specific scenario, i.e., including para- phrase types, has many benefits. A direct con- sequence of incorporating paraphrase types, and maybe the most impactful, lies in plagiarism de- tection. Plagiarism detection systems based machine learning often support their decision re- sults using shallow high-level similarity metries that only indicate how much a given text is po- tentially plagiarized, thus limiting their analysis (Foltýnek et al., 2019). Incorporating paraphrase types allows for more interpretable plagiarism de- tection systems, as more informative and precise re- sults can be derived. Additionally, automated writ- ing assistants can be improved beyond simple prob- ability distributions when suggesting alterations to a text. top of that, lecond-language learners can correct their texts by considering specific para- phrase types in their daily lives (e.g., learning about contractions "does not = doesn't")) helping them to learn new languages faster.
First attempts to categorize the lexical variables ma- nipulated in paraphrases into a taxonomy have been performed by Vila et al. (2014), followed by their first typology annotated corpus (Vila et al., 2015). Gold et al. (2019) categorize paraphrases on a higher level as meaning relations and present three additional categories: textual entailment, speci- ficity, and semantic similarity. Kovatchev et al. (2020) extend Vila et al. (2015, 2014)'s typology and re-annotate the MRPC-A (Dolan and Brockett, 2005) corpus with fine-grained annotations with more than 26 lexical categories, such as negation switching and spelling changes in the ETPC dataset. Recent works model objective functions instead of taxonomies (e.g.. word position deviation, lexical deviation) to automatically categorize paraphrases (Liu and Soh, 2022a). This approach is similar Bandel et (2022); Liu al. (2020a)'s proposed metrics for paraphrase quality (e.g., semantic simi- larity, expression diversity).
Our proposed tasks indicate that language mod- els struggle generate or detect paraphrase types with acceptable performance, underlining the chal- lenging aspect of finding the linguistic aspects within paraphrases. However, learning paraphrase types is beneficial in generation and detection as the performance of trained models consistently in- creases for both tasks. In addition, scaling models also suggest improvements in their ability to un- derstand and differentiate paraphrase types when transferring unseen paraphrasing tasks. In summary, we:
Recent work requires texts to satisfy certain stylistic, semantic, or structural requirements, such as using formal language expressing thoughts using a particular template (lyyer al., 2018; Shen et al., 2017). In paraphrase generation, methods require texts meet certain quality criteria, such as semantic preservation and lexical diversity (Ban- del et al., 2022; Yang et al., 2022) or require syn- tactic criteria, such as word ordering (Chen et al.. 2019; Goyal and Durrett, 2020; Sun et al., 2021). The development of the Multi-Topic Paraphrase in Twitter (MultiPIT) corpus addresses quality is- in existing paraphrase datasets and facilitates the acquisition and generation of high-qualin para- phrases (Dou et al., 2022). (PIP) tunes large pre-trained language models for
introduce two new tasks, Paraphrase Type Generation and Paraphrase Type Detection, providing a more granular perspective over general similarity-based tasks for paraphrases;
Prefix
tht thub. com/ jowahle 1mm_p233parpphasee--
show that our proposed tasks are compatible with traditional paraphrase generation and de- tection tasks (without paraphrase types);
"tt //thggguggffaceeeeee co/ spaacs/ paraphrase-type syype-generation

---

generating paraphrases according specified syn- tactic structures in a low-data setting, significantly reducing training costs compared to traditional fine- tuning methods (Wan et al., 2023).
3.1 Paraphrase Type Generation
Given a sentence or phrase x and a set of para- phrase type(s) 4 E L, a paraphrase E should be provided, where L. is the set of all possible para- phrase types L [Hea, .... -morphh The reference paraphrase types to be incorporated in I have to take place on specific positions (i.e., segments sj). which can potentially overlap. The resulting para- phrase ő should maximize its similarity against the original text x while incorporating the segment's s reference paraphrase type(s).
Although current contributions to generate and detect different paraphrase forms, they do not use them generate detect paraphrase types directly, Instead, they rely on shallow similarity measures and binary labels for identifying paraphrases. In this work, we propose two new tasks. One for generating specific perturbations when creating new paraphrases and one for detecting the lexi- cal differences between paraphrases. We use the ETPC dataset to evaluate these tasks and show that learning paraphrase types is more challenging than considering the binary notion of paraphrase. Our results suggest that learning paraphrase types is beneficial for traditional paraphrase generation and detection. We further demonstrate these findings in our experiments ($4).
The task can be measured through multiple met- rics. This study uses BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTScore (Zhang al., 2020b) for the paraphrase segments in E in relation to x (cf. Section 4. 4.1). To measure correlations, we also include word position deviation and lexical deviation (Liu and Soh, 2022a),
3.2 Paraphrase Type Detection
Given sentence or phrase x and paraphrase ã, the task is to identify which paraphrase types 4 L the latter contains in relation to the former L(ā). where L the group of all possible paraphrase types I, - and I(x) represents the paraphrase types in स. Both x and ā include no information about which segments sj were altered or how they are correlated. Therefore, our task requires the identification of segments and their classification, which can be composed of multiple types. 8y might have different positions in x and ã, as cach phrase can have different word order. The following example shows how two phrases are related according to their paraphrase types.
3 Task Formulation
Most paraphrase-related tasks focus on generat- ing or classifying paraphrases at a general level (Foltýnek al., 2020; Wahle et al., 2022a,b, 2021). This goal is limited, as it provides little details on what composes a paraphrase or which aspects make original and paraphrase alike (Fournier and Dunbar, 2021). We believe incorporating paraphrase types in generation- and detection-related tasks can help understand paraphrasing better.
We propose specific tasks for paraphrase gen- eration and detection include paraphrase types. The goal of Paraphrase Type Generation to generate a paraphrased text that preserves the se- mantics of the source text but differs in certain linguistic aspects. These linguistic aspects are spe- cific paraphrase types (e.g., lexicon change). In the Paraphrase Type Detection task, the goal is to locate and identify the paraphrase types in which two pieces of the text differ.
A project x was funded in
New York City x4'
5: New York funded ie,, for its largest city 64"
I(#):
(s (azi (5 laia) hex))
Multiple metrics can also be considered when evaluating Paraphrase Type Detection (e.g.. Fl score, accuracy). We evaluate the detection as a weighted sum of accuracies of paraphrase types 4 in the modified segments of ã against r. Therefore, accuracies are weighted within the same phrase. Weighting prevents the dominance of specific types in phrases with multiple occurrences in the dataset. explore paraphrase types, we as-
Both tasks aim to include a fine- -reiigr under- standing of paraphrase types over current existing tasks. Each specific task can also be formulated as a simple paraphrase generation or detection task (i.c., without paraphrase types) with small error 6. Thus, our tasks complement existing ones in paraphrase generation and detection. Section 4 pro- vides more information on the composition of the proposed datasets, their splits, and their structure. Figure illustrate our proposed tasks.
As our goal sume that one of the sentences is paraphrase of the other and both are semantically related. However, this task can also be altered to identify the existence

---

Paraphrase Type Generation
Paraphrase Type Detection
The-prasident
gave
speech
about her...
The president
gave
speech
about her...
% Derwes
She
has given
talk
about her...
She
has given
talk
about her...
Figure 2: Paraphrase Type Generation (left) and Paraphrase Type Detection (right) with model f. reference segments s1, s2, s3 candidate segments 81, 82, 83-
of paraphrasing and its types separately. Another possible extension for our tasks could include un- related sentence pairs with no paraphrase types, similar to the unanswerable questions in SQuAD 2.0 (Li, 2019). In experiments, do not in- vestigate the performance considering the location of each paraphrase. Thus, correct identification is only considered if the pair (sj; 1,) is provided. We leave the investigation of such aspects to fu- ture work and invite researchers to explore other variations of our tasks.
overview of their distribution. The most common is the group "lexicon-based changes", particularly the paraphrase type "synthetic/analytic substitutions", e.g., noun replacements with the same meaning. Notably, many paraphrases additions dele- tions of words phrase. Using ETPC, we evalu- ale how well existing models perform generation and detection tasks, with and without prior training in paraphrase types. We use a 70% train and 30% eval split with an equal balance between paraphrase types. To show how our tasks are compatible with general binary paraphrase tasks and datasets, se- lected models trained with paraphrase types are tested for paraphrase types (ETPC) and general paraphrase task (QQP).
4 Experiments
In our experiments, we first investigate the dis- tinct differences in paraphrase types by evaluating their correlations. Next, we measure how much language models already know about paraphrase types and how that changes when scaling them. Fi- nally, we empirically study how existing models perform in our proposed tasks. To test whether paraphrase types are a valuable extension to tra- ditional paraphrasing tasks, we evaluate selected models after incorporating paraphrase types in their training to quantify the transfer from paraphrase types to traditional paraphrase tasks.
Metries. For the Paraphrase Type Generation task, we measure the performance of models generating paraphrase types on a segment level using BLEU and ROUGE. For Paraphrase Type Detection, we use accuracy on a segment level, i.e., each seg- ment receives an individual score which we aver- age per phrase for three categories: Binary para- phrased not; Type paraphrase type (e.g., ellip- sis); and Group groups of paraphrase types (e.g., morphology-basee changes). More details on the evaluation can be found in the Appendix A.5.
4.1 Setup
Models. We conduct experiments scaling model sizes with LLaMA (Touvron et al., 2023); gen- eration experiments with autoregressive models: BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020a) and ChatGPT; and detection exper- iments with autoencoders: BERT (Devlin et al., 2019), RoBERTa (Liu al., 2019), ERNIE 2.0 (Sun et al., 2020), DeBERTa (He et al., 2021), and ChatGPT. We use ChatGPT in the September 25th 2023 version?.
Datasets. We use the Extended Paraphrase Ty- pology Corpus (ETPC) (Kovatchev et al., 2018) and three challenging auxiliary paraphrase datasets according to (Becker et al., 2023): Quora Ques- tion Pairs (QQP) (Wang et al., 2017), Twitter News URL Corpus (TURL) (Lan et al., 2017), and Paraphrase Adversarics from Word Scrambling (PAWS) (Zhang et al., 2019). More details about the datasets can be found in Appendix A.3. ETPC is a corpus with binary labels (paraphrased or orig- inal) and 26 fine- grained paraphrase types, and six high-level paraphrase groups. Table gives an
httpps: /thtps://hel.... open.i..nn.........ooo... httpeiiccoeeeeaaeeieeeeeeeeeeeee http////n/arttees///e 4025453-chatgpt- relaseettt

---

human performance
w/o paraphrase types
w/ paraphrase types
QQP
TURL
PAWS
100
100
100
Task performance (%)
96
95
95
85
85
85
80
80
80
75
70
70
70
7B
13B
338
65B
7B
138
33B
65B
7B
138
338
65B
Model parameters
Model parameters
Model parameters
Figure 3: Task performance (accuracy) for different model sizes of LLaMA with and without learned paraphrase types against human performance as reported by the respective datasets or benchmarks.
Paraphrase Types
tions and deletions). One possible explanation is that phrases can be syntactically different and mean the same have opposing meanings, leading divergences. Larger models tend to learn the differ- ence between paraphrases and non- oon-aaaphrases for individual types better. The overall performance, though, is relatively low, reflecting that LLMs also have difficulties classifying paraphrase types in general, meaning that they are unable to identify the particular changes that led to detection, even though they show good performance for the known Binary classification case (not shown here").
Discourse based
68
emes
Loxtoo-syrtattcc based
Lestoon batton-basee
82
Syntax
Others
Pampproos
Non-Paraphrase
78
138
338
65B
Model parameters
Figure 4: Accuracy for identifying paraphrase types of six high-level groups for different model sizes in the ETPC dataset using LLaMA.
Q2. How well can language models perform type generation and detection when instructed to?
4.2 Generation & Detection
Generation. We use BART and PEGASUS to per- form Paraphrase Type Generation by adapting their last layer for token prediction. We assign each sam- ple token-level label for generating substitutions of paraphrase type.
Q1. How much are paraphrase types already rep- resented in language models?
We test the ability of models to identify paraphrase types for different model sizes of LLaMA (in terms of model parameters). Therefore, we compose in-context prompts with few-shot examples and chain-of-thought (Wang al., 2022: Wei et al.. 2023). Prompt examples are shown in Figure Appendix A.6. We measure the accuracy de- tecting the correct paraphrase type using the ETPC group categories. Figure 4 shows the results. While smaller models lead to overall low performance in identifying paraphrase types, scaling them in- creases the performance in identifying paraphrase types. As we scale the model, divergence between paraphrases and hon-parapsrases becomes more prominent for different types. We colored the three most divergent groups in red, i.c., lexicon-based changes, syntax vynta--based, and others (mainly addi-
x Amrozi accused his brother, whom called 'the witness', of deliberately distorting his evi- dence.
3 Referring him only 'the witness' Amrozi accused his brother of deliberately distorting his evidence.
L(x) (26, 26, 26, 26, 0, 5, 0, 6, 25, 25, 25, 25, 25,
25, 25, 25, 25, 25, 25)
L(#) (6, 5, 0, 25, 0, 0, 0, 26, 26, 26, 26, 0,
0, 0,
see http:://ppeerrwthhcccccc///////s

---

Paraphrase Type
Examples
lower ROUGE scores than BART. ChatGPT gener- ales text that matches the reference at the n-gram level more precisely but might be missing out on covering other parts of the reference. This means 183
Morphology-based changes
975
Derivational Changes
186
Inflectional Changes
606
Modal Verb Changes
the generated text might be very similar to some portions of the reference but does not capture the entirety or breadth of the reference content. BART and PEGASUS capture most of the content from the reference text, but how they present it (word- ings, order) might differ from the reference. This means the generated text has good recall of the critical content but may not have the exact phras- ing or structure as the reference. Table in Ap- pendix A.4 shows additional in-context predictions of ChatGPT, showing that the default model is not able to generate paraphrase types well without fine- tuning. Although ChatGPT-3.5 reached the highest BLEU scores, smaller models have an edge when fine-tuned. All tested models can Iearn and gen- erate paraphrase types to some extent. Still, there is much potential for improvement using more so- phisticated methods to generate paraphrase types.
Lexicon-based changes
666
Spelling changes
628
Change format
236
Same Polarity Substitution (contextual)
138
Same Polarity Substitution (habitual)
831
Same Polarity Substitution (namne ent.)
533
Lexico-synlactie based changes
Converse substitution
43
Opposite polarity substitution (contextual)
15
Opposite polarity substitution (habitual)
Synthetic/amalytic substitution
Syntax- bused changes
731
Coordination changes
47
Diathesis alternation Ellipsis
161
65
Negution switching
20
Subordination and nesting changes
468
Discourse-based changes
617
DirecVindirect style alternations
19
Punctuation changes
293
Syntax/discourse structure changes
305
Extremes
2 287
Entailment
81
Identity
1782
Detection. We test four autoencoder models on Paraphrase Type Detection by adapting their token- level representation with linear layer to classify one of the 26 paraphrase types. We also fine-tuned ChatGPT-3.5 the same task with prompt in- structions. To estimate the accuracy of classifying higher-level perturbations, we group the 26 types into one of six groups (Table 1). Both the type and group scores are averaged over all occurrences the sequence. For the entire sequence, we classify the binary label (i.e., paraphrase or not) using the aggregate representation of the model (e.g., [CLS]- token for BERT).
Non- Nonn-arraprase
424
Others
742
Addition/Deletion
4733
Change of order
857
Semantic-based
152
Total
16813
Table I: An overview of considered paraphrase types and their occurrences in the ETPC dataset.
Each label is mapped to a paraphrase type, and the tuple index corresponds to the tokenized word index. We balance the number of paraphrase types between training and validation sets, although this sometimes leads to low amounts of evaluation ex- amples (e.g.. 1 examples for opposite polarity sub- stitution). Thus, we consider only groups with at least 100 examples per type in their evaluation set.
Table 3 shows the accuracy for paraphrase type detection on ETPC and paraphrase detection on QQP. DeBERTa achieves the highest performance of all encoder models across all categories for de- tecting paraphrase types, with 83.0 in the tradi- tional binary case, 65.0 when detecting paraphrase types, and 67.9 when detecting the correct group. ChatGPT significantly outperforms the results of DeBERTa and when detecting paraphrase types, it achieves 11.8 percentage points higher results than DeBERTa. ERNIE 2.0 closcly follows DeBERTa in binary detection with a score of 82.7 but trailed in type and group detection. Overall, the scores for paraphrase type and group are relatively low compared to the binary case, underlining the chal- lenge for autoencoder models to grasp which lex-
We also test paraphrase type generation with ChatGPT-3.5 by formulating generation prompts as instructions (see Figure for examples).
Table shows the results. Both BART and PE- GASUS show strong performance for generating paraphrase types in ETPC. BART outperforms PE- GASUS in all metrics, particularly in ROUGE-L, suggesting that BART may be better suited for the task when contexts are longer. Although we ex- pected fine-tuned ChatGPT to be superior over smaller models, it achieves higher BLEU scores but

---

Model
BLEU
ROUGE-1
ROUGE-2
ROUGLE-L
BART
46.3
56.2
34.9
54.2
PEGASUS
45.3
54.9
33,8
50.1
ChatGPT-3.5
55.9
51.8
32.9
48.9
Table 2: Generation results of fine-tuned models on the ETPC dataset.
ETPC
QQP
Model
Binary
Type
Group
w/o Types"
w/ Types
BERT
74.1
58.7
60.0
89.3 72.1
91.6 88.6
RoBERTa
68.3
62.5
62.9
90.2 74.3
91.5 88.6
ERNIE 2.0
82.7
64.2
65.9
90.9 / 75.2
92.4 89.7
DeBERTa 83.0
65.0
67.9
90.8 / 76.2
93.0 90.6
ChatGPT-3.5
90.4
76.8
78.1
90.7 / 75.4
92.5 90.0
Table 3: Detection results for the ETPC (accuracy) and QQP (accuracy/F1) datasets. Models trained on ETPC applied in QQP (w/ Types). *Official results for autoencoders (w/o Types) from GLUE leaderboard https: / wtwp:/benncnnrkk.cco/eeeeeeeeeee for comparison. Best results in bold.
ical perturbation occurred. In the binary scenario, all models can distinguish between paraphrased and original content at general level with good performance. However, this success diminishes in the presence of paraphrase types, corroborating that these models do not understand the intrinsic variables that have been manipulated yet.
Q4. What impact has learning paraphrase types on generating paraphrases?
We test models previously trained on generating ETPC paraphrase types to generate paraphrases for the QQP dataset. Table 4 shows the results. In- tegrating paraphrase types into BART and PEGA- SUS leads to considerable performance improve- ments across all assessed metries. When using ChatGPT with in-context instructions, consider- able performance gains can be observed too but overall the results are lower than BART and PE- GASUS, again underlining that specific tasks can benefit from smaller expert models. BART experi- ences marked increase in ROUGE-L score from 41.8 to 44.2 and its ROUGE-1 score from 43.1 to 45.5, demonstrating improved results in para- phrase generation. Similar but overall less consis- tent improvements are also observed in PEGASUS, with BLEU increasing by 2.6 points and ROUGE-L score rise of 2.4 points. These results show that including paraphrase types positively affects per- formance in the QQP dataset. ChatGPT shows good performance in generating paraphrase types too, without fine-tuning, with increases of up 3.2 percentage points (ROUGE-I), Both models drop in performance from ROUGE-1 to ROUGE-2 and again increase from ROUGE-2 to ROUGE-L, finding consistent with related works (Li et al.,
Q3. How does learning paraphrase types improve task performance of traditional paraphrase tasks with model scale?
We test fine-tuned LLaMA in three binary para- phrase tasks, i.e., QQP, TURL, and PAWS, with two different settings: without paraphrase type in- structions and with instructions using the ETPC dataset. For more details the prompts used, see Figure 9. We also report the human performance of the respective dataset papers or benchmarks. The results reveal positive trend when scaling LLaMA from 7B to 65B parameters. While scaling the model improves its baseline (w/o paraphrase types), the incorporation of paraphrase types leads to increase in performances, achieving better- than-human results for all three datasets (Figure 3). Across datasets, the variation lowest for QQP. which also more than ten times larger (795k) than TURL (52k) or PAWS (65k). This analysis complements earlier findings that larger models are also more capable of paraphrase type tasks.

---

Model
BLEU
ROUGE-1
ROUGE-2 ROUGE-L
BART
+ w/o paraphrase types
44.7
43.1
25.3
41.8
+ w/ paraphrase types
46.8
45.5
27.0
44.2
PEGASUS
w/o paraphrase types
42.3
41.9
25.1
39.6
w/ paraphrase types
44.9
43.6
26.8
42.0
ChatGPT-3.5
w/o paraphrase types
34.6
31.8
14.5
29.2
w/ paraphrase types
34.7
35.0
16.9
37.4
Table 4: Generation results on the QQP dataset for trained models in the ETPC without and with prior paraphrase type generation training. Best results bold.
2019 Liu al., 2020b; Miao al., 2019; See et al., 2017). Including paraphrase types overall leads to higher ROUGE-L scores, indicating higher recall and tendencies to perform better with longer contexts. Further investigations are necessary to conclude whether specific paraphrase types ad- ditional training contributed to performance gains.
Discourse
scores,,
0.5
Lodoo syntoctic
Lodcoo-bbaadd
-05
Others
QS. What impact has learning paraphrase rypes on identifying paraphrases?
Syntioxbraee
Donmsrge bosedd
Lavioo sintaoi bexeed
Exhemor
Larioon hunsed
Celers
We also test the transfer of models trained on de- tecting ETPC paraphrase types the QQP task (right side of Table 3). For the QQP column, we fine-tuned models on QQP that have been previ- ously fine-tuned with paraphrase types on ETPC Models trained on paraphrase types consistently outperform their counterparts in the binary setup (w/o Types) for the QQP dataset. For example, the accuracy of DeBERTa improved from 90.8/76.2 to 93.0/90.6 with the integration of paraphrase types. Similarly, when paraphrase types are incorporated, BERT's performance improves from 89.3/72.1 to 91.6/88.60. ChatGPT achieves comparable perfor- mance to autoencoders.
Synias
Figure 5: Rescaled Spearman correlations between para- phrase types six higher-level families using word po- sition deviation and lexical deviation. Correlations normalized around the origin using mean and standard deviation such that 0, a
4.3 Type Correlations and Similarity
To show how different paraphrase types correlate, we compute word position deviation and lexical deviation (Liu and Soh, 2022b) of examples in the ETPC dataset. Next, we compute the Spearman correlations between these scores for all examples per paraphrase type to another and average over all examples per paraphrase type. Correlations high overall, with average of 0.89. We rescale the correlations Figure with μ 0 and to visualize the differences better. We include the correlation
These results underscore the value of integrating paraphrase types in enhancing models' detection capabilities across different datasets and detection metrics. DeBERTa achieves the best performance with significant improvements when the model trained to recognize paraphrase types. Although ETPC has relatively small amount of examples, the performance benefits are clear and can poten- tially accelerate the development of new paraphrase detection methods using LLMs in the future.
26 paraphrase types in Figure 8 in Appendix A.2.
Within groups, lexicon-basse changes have the highest correlation, followed by the "Others" cate- gory that contains "Addition/Delltion". "Change
We do not fine-tune ChatGPT on the full QQP dataset to reduce training cost. We sample 20% of training examples.

---

order", and "Semantic-based" changes. Between groups, we observe a thaher-than-average corre- lation between syntax-based based and lexico-syntactic- based changes, both containing syntactic compo- nents. The most prominent cases regarding group types are between "synthetic/amalytie substitution" and "diathesis alternation"; and "opposite polarity substitution (contextual)" and "ellipsis" (see Ap- pendix A.2 for more details). In summary, differ- ent paraphrase types show overall high correlations even between groups while lexicon-basee bassed and other changes correlate less.
rent datasets could take place either by automated systems (for large datasets) or by human annotators. In addition to generating new paraphrase-typed datasets, a prospective direction is to use large language models to paraphrase original content and qualitatively identify which paraphrase types these models learn during their training through humans. Although metrics such as BLUE and ROUGE have known deficiencies (e.g., high scores for low-quality generations), they are currently the standard practice in generation tasks. Thus, a met- ric incorporating paraphrase types with their lo- cation and segment length could provide a more accurate assessment of our proposed tasks.
4.4 Demo for Paraphrase Types
We provide an interactive chat-like demo Hug- gingface" 6 generate paraphrase types interactively or automatically using the Python Gradio Client API. Figure 6 in the Appendix provides a screen- shot of that tool.
Limitations
As the of paraphrase types in generation and detection tasks is still incipient, much work is quired establish this as the new paradigm in the field. To the best of our knowledge, our paper one of the first contributions to define tasks for para- phrase types for automatic investigations, probe tfate-of-the-art models under these conditions, and verify the compatibility of specifically trained mod- els with proposed and existing tasks. However, several points remain open to be explored. In this section, we go over some of them.
5 Final Considerations
Conclusion. In this paper, we proposed Paraphrase Type Generation and Paraphrase Type Detection, two specific paraphrase type-based tasks. These tasks extend traditional binary paraphrase genera- tion and detection tasks with more realistic and granular objective. In the generation task, para- phrases have to be produced according to spe- cific lexical variations, while the detection task, these lexical perturbations need to be identified and classified. Our results suggest that the pro- posed paradigm poses more challenging scenario current models, but learning paraphrase types is beneficial in the generation and detection tasks. Additionally, both of our proposed tasks are com- patible with existing ones. All models trained on paraphrase types consistently improve their per- formance for generation and detection tasks, with and without paraphrases. The shift from general paraphrasing to including specific types encourages the development of solutions that understand the linguistic aspects they manipulate. Systems that incorporate specific paraphrase types, can there- fore support more interpretability, accuracy, and transparency of current model results.
Two factors limit the experimental setup of our tasks: the datasets used and the considered metrics. Our analysis is based on ETPC to probe paraphrase types, we are bounded to the limited number of examples of that dataset. In addition, we only test the transfer from paraphrase types more general paraphrase tasks between ETPC and QQP. Thus, more diverse datasets must be proposed and ex- plored so prospective solutions can be thoroughly evaluated. On the evaluation side, we still rely on standard metrics such as BLEU and ROUGE, which are known for their limitations (e.g., poor correlation with human preferences) and cannot account for paraphrase types, locations, or segment length in their score. A metric incorporating para- phrase types with their location and segment length would greatly support our experiments and results.
Even though we probe state-of-the-ar models in proposed tasks, no human study was con- ducted to establish human baseline for compar- ison on paraphrase types. Particularly automated generation metrics such as ROUGE and BLEU work well for particularly paraphrase types, such as, syntax-changed but obviously have problems
Future Work. As the vast majority of datasets in paraphrasing do not account for paraphrase types, the first natural step is to evaluate how to include types in their composition. The expansion of cur-
'https: / //tggggggugaccccce wttpaa/.cc...eeeeeeeeeeeeeeeeeee jpwahle/ paraphrase- typeeeettyyeeeeee generrtieatio

---

for lexicon-changes and lexico-syntactic changes. In future work, we are already exploring human annotation and alternative metrics to overcome is- sues resulting from word overlap. Part of the re- sults obtained in Section 4 are limited to automatic solutions, and only traditional tasks involve a hu- man baseline. Therefore, it is uncertain how much improvement current models still need to be com- parable to human performance in generating and detecting paraphrase types.
multiple linguistic aspects. For example, the re- search could focus on comparing multiple versions of the same emotions and then estimate whether dif- ferent linguistic concepts, such as negation, convey more or less emotion.
Text summarization. On top of Paraphrase Type Detection, researchers can build tools to identify where the summary preserves the original text's meaning, which parts of the text change, and how impacts semantic preservation and coherence. Text generation. Paraphrase Type Generation can support generating or paraphrasing stories using different paraphrase types to estimate which types lead to desirable attributes such as originality, ten- sion, and character development.
Ethics & Broader Impact
Ethical considerations. We understand that tech- niques devised for paraphrasing have many appli- cations, and some of them are potentially uneth- ical. As we push forward the necessity of para- phrase types, these can also be applied to gener- ate more complex, human-like, and hard-to tord-to-detect paraphrased texts, which can be used for plagia- rism. Plagiarism is severe act of misconduct in which one's idea, language, or work is used with- out proper reference (Foltýnek al., 2019; Kumar and Tripathi, 2013).
A Acknowledgements
This work was supported by the DAAD (German Academic Exchange Service) grant 9187215, the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) grant 437179652, as well as the Lower Saxony Ministry of Science and Cul- ture and the VW Foundation. Many thanks to An- dreas Stephan for thoughtful discussions.
Large language models capable of mimicking human text are a reality (e.g.. ChatGPT), and most of us still do not fully understand their reach. As al- ready foreshadowed (Wahle et al., 2022a,b, 2021), paraphrasing using language models can lead to more undetected plagiarism, undermining the qual- ity and veracity in several areas (e.g., academia, basic education, industry). Even though paraphrase types might encourage the development of even more sophisticated techniques that can potentially be incorporated into these models, we should not remain neutral. Therefore, artifacts probe and understand what composes paraphrases in neural language models should be welcome instead of feared.
References
Elron Bandel, Ranit Aharonov, Michal Shmueli- Scheuer, llya Shnayderman, Noam Slonim, and Liat Fin-Dor. 2022. Quality Controlled Paraphrase Gen- eration. Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 596-609 Dublin, Ireland. Association for Computational Linguistics.
Jonas Becker, Jan Philip Wahle, Terry Ruas, and Bela Gipp. 2023. Paraphrase detection: Human vs. ma- chine content. ArXiv preprint, abs/2303.13989.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. Controllable Paraphrase Gener- ation with Syntactic Exemplar. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5972-5984, Florence, Italy. Association for Computational Linguistics.
Broader Impact. The presented tasks in this work and its future solutions have the potential to benefit other areas aside from paraphrase generation and detection. In the following, we list few applica- tions that can be explored.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume (Long and Short Papers), pages Minneapolis. Minnesota. Association for Computational Linguistics.
Machine translation. Paraphrase Type Detection can help identify paraphrase types in a translated text to identify areas where the translation is faith- ful to the original text. This task can also assist in identifying deficiencies in specific linguistic as- pects between models and languages.
Emotion analysis. Paraphrase Type Generation could be used to express different emotions through
William Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases.

---

In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Kuan- Muan-HHo Huang and Kai- Wei Chang. 2021. Gener- ating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairs. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguisticx: Main Volume, pages 1022-1033 1033, Online. Association for Computa- tional Linguistics.
Yao Dou, Chao Jiang, and Wei Xu. 2022. Improv- ing large-scale paraphrase acquisition and generation. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 9301-9323, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial Example Generation with Syntactically Controlled Paraphrase Networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Pupers), pages 1875-1885, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's 's guide to testing sta- tistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( VVolume
1: Long Papers), pages 1383-1392 Melbourne, Aus- tralia. Association for Computational Linguistics.
Venelin Kovatchev, Darina Gold, M. Antonia Marti, Maria Salamo, and Torsten Zesch. 2020. Decompos- ing and Comparing Meaning Relations: Paraphrasing, Textual Entailment, Contradiction, and Specificity, In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5782-5791, Marseille, France. European Language Resources Association.
Elozino Egonmwan and Yllias Chali. 2019. Trans- former and seq2seq model for Paraphrase Generation. In Proceedings of the 3rd Workshop on Neural Gen- eration and Translation, pages 249-255, Hong Kong. Association for Computational Linguistics.
Tomášs Foltýnek, Norman Meuschke, and Bela Gipp. 2019. Academic Plagiarism Detection: A System- atic Literature Review. ACM Computing Surveys, 12:6):11::1111:443
Venclin Kovatchev, M. Antònia Martí, and Maria Salamó. 2018. ETPC A Paraphrase Identification Corpus Annotated with Extended Paraphrase Typol- ogy and Negation. In Proceedings of the Eleventh In- ternational Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Tomá's Foltýnck, Terry Ruas, Philipp Scharpf, Norman Meuschke, Moritz Schubotz, William Grosky, and Bela Gipp. 2020. Detecting Obffustt-d Pla- giarism. In Anneli Sundqvist, Gerd Berget, Jan No- lin, and Kjell Ivar Skjerdingstad, editors, Sustainable Digital Communities, volume 12051, pages 816-827. Springer International Publishing, Cham.
Ranjeet Kumar and RC Tripathi. 2013. An analysis of automated detection techniques for textual similar- ity in research documents. International Journal of Advanced Science and Technology. 56(9);99-1) -100
Louis Fournier and Ewan Dunhar, 2021. Paraphrases do not explain word analogies. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguisticx; Main Volume, pages 2129-2134, Online. Association for Computa- tional Linguisties.
Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A continuously growing dataset of sentential para- phrases. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1224-1234 Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Darina Gold, Venelin Kovatchev, and Torsten Zesch. 2019. Annotating and analyzing the interactions be- tween meaning relations. In Proceedings of the 13th Linguistic Annotation Workshop, pages 26-36, Flo- rence, Italy, Association for Computational Linguis- tics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising for natural language generation. translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880 Online. Association for Computa- tional Linguistics.
pre-training
Tanya Goyal and Greg Durrett, 2020. Neural Syntactic Preordering for Controlled Paraphrase Generation. In Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, pages 238-252, Online. Association for Computational Linguistics.
Yuanjun Li. 2019. Question answering on squad 2.0 dataset.
Pengeheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: eechdiig-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
Zichao Li, Xin Jiang, Lifeng Shang, and Qun Liu. 2019. Decomposable neural paraphrase generation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3403- 3414, Florence, Italy. Association for Computational Linguistics.

---

Chin-Yer Yew Lin. 2004, ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Aitor Ormazabal. Mikel Artetxe, Aitor Soroa, Gorka Labaka, and Eneko Agirre. 2022. Principled Para- phrase Generation with Parallel Corpora. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pu- pers), pages 1621 1-1688 Dublin, Ireland. Association for Computational Linguistics.
Timothy Liu and De Wen Soh. 2022a. Towards Bet- ter Characterization of Paraphrases. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8592 8601, Dublin, Ireland. Association for Computational Linguistics.
Malte OstendorfT, Till Blume, Terry Ruas, Bela Gipp, and Georg Rehm. 2022. Specialized Document Em- beddings for Aspect- Based Similarity of Research Papers. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries. JCDL '22, New York, NY. USA. Association for Computing Machin- ery.
Timothy Liu and De Wen Soh. 2022b. Towards bet- ter characterization of paraphrases. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 8592-8661 8601, Dublin, Ireland. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguisticx, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Xianggen Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song. 2020a. Unsupervised Para- phrasing by Simulated Annealing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 302-312, Online. Association for Computational Linguistics.
Alla Rozovskaya and Dan Roth. 2010. Annotating ESL errors: Challenges and rewards. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 28 36. Los Angeles, California Association for Computational Linguistics.
Xianggen Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song. 2020b. Unsupervised para- phrasing by simulated annealing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 302-312, Online. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1073- 1083, Vancouver. Canada. Association for Computa- tional Linguistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019, RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. ArXiv preprint, abs/1907.11699 11692.
for
Yuxian Meng, Xiang Ao, Qing He, Xiaofei Sun, Qinghong Han, Fei Wu, Chun Fan, and Jiwei Li. 2021. ConRPG; Paraphrase Generation using Con- texts as Regularizer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2551-2562, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tianxiao Shen, Tao Lei, Regina Barzilay, and T. Jaakkola. 2017. Style Transfer from Par-Plrllle Text by Cross- Aliinment,,,,, In NIPS.
Jiao Sun, Xuezhe Ma, and Nanyun Peng. 2021. AE- SOP: Paraphrase Generation with Adaptive Syntactic Control. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5176-5189, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. CGMH: constrained sentence generation by metropolis-hasting sampling. In The Thirty-Thin AAAI Conference on Artificial Intelligence, AAAI
Yu Sun, Shuohuan Wang. Yu-Kun Li, Shikun Feng. Hao Tian, Hua Wu, and Haifeng Wang. 2020. ERNIE 2.0: A continual pre-traine training framework for language un- derstanding. In The Thirty-Fourdh AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelli- gence Conference, IAAI 2020, The Tenth AAAI Sym- posium on Educational Advances in Artificial Intel- ligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8968 8975. AAAI Press.
2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii
USA, January 27- - February 1, 2019, pages 6834 6842, AAAI Press.
Saif Mohammad, 2016. A practical guide to senti- ment annotation: Challenges and solutions. In Pro- ceedings of the 7th Workshop on Computational Ap- proaches to Subjectivity, Sentiment and Social Media Analysis, pages 174-179, San Diego, California. As- sociation for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal

---

Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models.
5085-5109, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural lan- guage sentences. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelli- gence, IJCAI 2017, Melbourne, Australia, August 19-25. 2017, pages 4144 4150. ijeai.org.
Marta Vila, Manuel Bertran, M. Antòmia Marti, and Horacio Rodriguez. 2015. Corpus annotation with paraphrase types: New annotation scheme and inter- annotator agreement measures. Language Resources and Evaluation, (911:77710
Marta Vila, M. Antònia Martí, and Horacio Rodríguez. 2014. Is This a Paraphrase? What Kind? Paraphrase Boundaries and Typology. Open Journal of Modern Linguistics, 24001):20--218.
Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.
Gerson Vizcarra and Jose Ochoa-Luna. 2020. Para- phrase Generation via Adversarial Penalizations. In Proceedings of the Sixth Workshop on Noisy User- generated Text (W-NUT 2020), pages 249-259, On- line. Association for Computational Linguistics.
Frank Wilcoxon. 1992. Individual Comparisons by Ranking Methods, pages 196-202, Springer New York, New York, NY.
Kexin Yang. Dayiheng Liu, Wenqiang Lei. Baosong Yang. Haibo Zhang, Xue Zhao, Wenqing Yao, and Boxing Chen. 2022. GCPG: A General Framework for Controllable Paraphrase Generation. In Find- ings of the Association for Computational Linguis- rics: ACL 2022, pages 4035-4047, Dublin, Ireland. Association for Computational Linguistics.
Jan Philip Wahle, Terry Ruas, Tomáš Foltýnek, Nor- man Meuschke, and Bela Gipp. 2022a. Identifying Malte Smits, editor. Information for a Better World: Shaping the Global Future, volume 13192, pages 393-413. Springer International Publishing, Cham.
Machine- Paraphe-Parrppraseed Plagiarism.
Jingging Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020a. PEGASUS: pre-training with ex- tracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1328-11330 PMLR.
Jan Philip Wahle, Terry Ruas, Frederic Kirstein, and Bela Gipp. 2022b. How large language models are machine-paraphrase paraphrase plagiarism. In Pro- ceedings of the 2022 Conference Empirical Meth- ods Natural Language Processing, pages 952-963, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
transforming machine-
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinherger, and Yoav Artzi. 2020b. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenKeview.net.
Jan Philip Wahle, Terry Ruas, Norman Meuschke, and Bela Gipp. 2021. Are Neural Language Models Good Plagiarists? A Benchmark for Neural Para- phrase Detection. In 2021 ACM/IEEE Joint Confer- ence Digital Libraries (JCDL), pages 226-229, Champaign, IL, USA. IEEE.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages Minneapolis. Minnesota. Association for Computa- tional Linguistics.
Yixin Wan, Kuan-Hao Huang, and Kai-Wei Chang. 2023. PIP: Parse-instructed prefix for syntactically controlled paraphrase generation. In Findings of the Association for Computational Linguisties: ACL 2023, pages 10372-10380, Toronto, Canada. Associ- ation for Computational Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-Naturall Instrattnss Generaliza- tion via declarative instructions on 1600+ NLP tasks. Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages
Jianing Zhou and Suma Bhat. 2021. Paraphrase genera- tion: A survey of the state of the art. In Proceedings of the 2021 Conference Empirical Methods in Nut- ural Language Processing, pages 5075-5086, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

---

A Appendix
notable differences exist. For example, direct and indirect style alternations wore-than-average with the change of order. While writing style seems to have many components, a significant one seems to be the ordering of sentences. Usually, the most important keywords of sentence remain the same, but their ordering can vary one of the reasons why word-count-based metrics such BLEU and ROUGE are still used in many tasks.
A.1 Frequently Asked Questions
How do your tasks differ from syntactic and semantic learning?
The presented tasks are specifically designed to incorporate information about various linguistic aspects, not limited to syntax and semantics. They overcome shortcomings as demonstrated in our experiments in which one aspect is learned well (e.g., semantics the subjects and objects and actions the same), but others suffer (e.g., opposite polarity the meaning was reversed). Investigating only semantics and syntax can lead to underrepresented paraphrase types, which are particularly problematic when shifting to new domains.
Another high correlation exists for contextual opposite polarity substitution and change of or- der. Opposite polarity substitutions are cases in which the meaning of a term is opposed to the original (e.g. "Johnson quickly accepted the pro- posal." and "Johnson rejected the proposal without hesitation."). Changing polarity can often include changing the term's position if an exact opposite term does not exist. Therefore, order change often be explained by contextual opposite polarity. However, suppose the polarity change is habitual. In that case, there is often no need for word order changes as the same concept can be explained at the same text position (i.e., the meaning remains "Leicester failed in both enterprises" and "He did not succeed in either case"). Changing polarity also correlates with ellipses which are typically shorter versions of the same phrase.
The annotations for this task require experts. Is there an unsupervised approach that I can use?
While paraphrase types diverse and have many categories, the annotation task is similar to named entity recognition annotation, in which the text span (segment location) and entity (paraphrase type) must be annotated. High amounts of labels are also not a new challenge. Large annotation providers (e.g.. Prodigy', Scale") provide tools to simplify this process (e.g., one annotator finds three categories at time). Also, tasks with more be
Much twwrr-than-average correlations appear for converse substitution with addition and deletion and a format change. Converse substitution means the change of action from subject to object (e.g.. "Sam bought a new car from John." and "John sold his car to Sam."). As already illustrated by this simple example, converse substitution often requires format change and additions/deletionss to ensure that the subjects/objects connected to the action in both cases.
complex problems and deseriptions more beneficial for future research (Mohammad, 2016; Rozovskaya and Roth, 2010). We are currently working on a semi-supervised method for assisted paraphrase-type generation using contrastive learning and reinforcement learning from human feedback to facilitate this task. Still, for evaluation purposes, we require an annotated test set human raters.
While many more correlations exist across the spectrum (e.g., modal verb changes negation switching or subordination and nesting changes to syntax/discourse structure changes), their nature appears to be due to both often appearing in the same sentences. The correlation amalysis of this study serves as a starting point for further inves- tigations but is limited in that only 17 668 total paraphrase types oceur across 5 801 sentences.
How can I use the tasks?
Our implementation available on GitHub and a demo available through Huggingface.
A.2 Details on Correlations
Figure provides the detailled correlation of Fig- ure 5 for each paraphrase type. While the overall group correlations are also represented here, some
Figure 7 shows the rescaled BERTScore sim- ilarity scores for the example in Figure with three paraphrase-type perturbations, i.e., diathe- sis alternation, polarity substitution, and lexicon change. Since the example has no word position
"https://prodi.g
"nttps://scale.co
'nttps://github.co httph/uw...ooo//wheeeee.e
ammlp33-paraphrssetyyess

---

Paraphrase Type Generator
Selctnnee nuerlppe aeschnaviere in the spdowrrmeea
sentano bocboc.
thouggl carcr grenoon prophasse logar untree loun ventteityp
theory stocleczic brownia iudinn.nttrtuuod specil thery reletvity theory rebilly
scounnt
the behaviar of desttronegneis fidé and dememibauee that atwespe deory i soreet mao and oreag sqaivolen orch olthe
Pystems byr trrtaekioonss structare changes
speccal horyoliskeivit,
apec theory soreed, mas and emergy ureupu to
Figure An interactive demo generate paraphrase types.
deviations, we expect the column maxima to be diagonal-shaped like in the segment "about her plan to change the constitution". However, par- ticularly for the paraphrase types in question, the similarities between "She" and "The president", "has given" and "gave", and "talk" and "speech"
1.0
The
0.48
0.11
0.22
13.00
0.22
0:25
3.09
0.32
6:31
0.32
0.1
president-11
0.00
0.08
0.00
0.15
0.01
0.28 3.9
0.23
6.08
0.06
0.54
0.75
654
0:52
0.00
3:58
5.33 0:22 355
5:33
6.22
6.31
0.03
0.9
0.07
9:38
6:39 382 9:31
0.30
0.16 0.30
Speech-1
0.13
5.28
0:27 032 2.00 0.23 0.08
are lower between them and sometimes inferior to other similarities the same terms. We also tested the same example with "he/him" and "they/them" " pronouns instead of "she/her" to verify potential biases towards gender (i.c., presidents are mainly male in the training data), but the scores were com- parable.
Candidate
0.30
622 031 0.12 0.22 0.08
0.00
24
0.26
6.00
9.00 0.30 -0.37 0.06
362
charge
0.2
conste
We hypothesize that similarities between seg- ments with paraphrase types have lower similarity, on average, than their non par-parap type coun- terparts. This suggests that paraphrase types are semantically more challenging to identify.
dtange
Reference
Figure Rescaled BERTScore similarity for an exam- ple reference phrase and its paraphrased candidate.
A.3 Datasets
Quora Question Pairs (QQP) is collection of ap- proximately 400k pairs of questions extracted from Quora" 10 platform for general question and an- swers. This dataset annotated to identify whether one question is a rephrasing of another (Wang et al., 2017). QQP is one of the largest and most estab-
lished paraphrase datasets in the community and therefore received particular attention throughout experiments.
Twitter News URL Corpus (TURL) encompasses about 2.8 million pairs
sen-
tences taken from Twitter news (Lan et al., 2017).
10 hhttps://quora.com
"nttps://twitterr.om

---

Change
Charge of arder
Coortinctson changes
tertooaa Changee
Berhace atamrdion
Druckied nece sofe alaemaiora
Ertalirent
Infactional Changee
Negation owizting
-0.25
Oppoete poletty areetution (contersud)
Oppoalll abittstion (habbss)
0.50
Puncaaren charrges
0.75
Same Fo Sunsthtto (contersus)
Subcttation (hobhaa)
-1.00
Same Phianty sutechition jurned wis]
Spar ny charges
Subordination
Syntzuid socurse strucure changes
Systheicwwwyt nubet
Aritsse Dession
Charge Loomia
Enteliirnn
Game Frderiy Bui Dratdioed
Dustee altr Divettmeeed apte dhum
Geriaaae priturng aetesti Opensin gemet coaemi there Puanni
Figure 8: Rescaled Spearman correlations between paraphrase types using word position deviation, lexical deviation, BLEU, ROUGE, and BERTScore the ETPC dataset. Correlations are normalized using mean ş and standard deviation such that μ 0, 1.
Annotations in the form of binary-type markings were provided by six individual raters. In our study, we recognized paraphrase as positive based on a majority vote and neglected those pairs lacking majority consensus.
(Wilcoxon, 1992). The detection results between models are significant with p < 0.05.
A.6 Prompt examples
Figure 9 shows some prompt examples used our experiments. We rely on Wang et al. (2022) for few-show examples and prompt templates.
Paraphrase Adversaries from Word Scrambling (PAWS) comprises around 65k pairs of machine- generated texts (Zhang al., 2019). These texts are obtained from Wikipedia and were created by im- plementing word reordering and nack-translation strategies.
A.4 Supplementary Results
Table 5 shows additional in in-context prediction sults for ChatGPT (i.e., prompts with few-shot amples). Both ROUGE and BLEU scores are much lower than those of fine-tuned models, showing that ChatGPT's default capability to generate para- phrase types is limited.
A.5 Evaluation
Our detection experiments also included statisti- cal significance tests of our results, namely results of Table Following (Dror et al., 2018), we as- sess our results using a parameetit sampling- free test, namely the Wilcoxon signed-rank test

---

In-Context
TeneTunnd
Model
BLEU
ROUGE-1
ROUGE-L
BLEU
ROUGE-1
ROUGE-L
BART PEGASUS
46.3
56.2
54.2
45.3
54.9
50.1
Cha1GPT-3.5
27.1
24.0
23.3
55.9
51.8
48.9
Table 5: Generation results the ETPC dataset for BLEU, ROUGE-1 and ROUGE-L.
Paraphrase Type Generation
Paraphrase Type Detection
Given the following sentence, generate paraphrase with the following types.
Prompt:
Prompt: the following sentences, which the paraphrase types are changed between them?
Sentence: And although the precanditions for recovery remain in place, It said the prospects for British exports weaker than previously expected.
Sentence 1: was wondering who they might be.
Paraphrase Types: Addition/Deletion, Inflectional.
Sentence 2: was unclear why that could be.
Although preconditions recovery remain in place, the prospect external demand UK output weaker than previously expected
A: Modal Verb Change
Figure Example prompts used for experiments on paraphrase type generation detection.


Reviews:
[{"review_id": "WKkwdUUsMr", "reviewer": "Reviewer_aya4", "summary": "", "questions": "Please see above.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy to hear that the motivation for the proposed paraphrase-type tasks is clear and supported through “comprehensive experiments” and an “experiment setup [that] is well-executed”.  We thank the reviewer for their feedback and clarify questions in the following.\n\nRe: “The paraphrase type generation task is a sentence-level task or phrase-level task?”:\n\nYes, paraphrase type generation and detection are sentence-level tasks. By “segments”, we mean subparts of the sentence which do not necessarily convey enclosed meaning. We use segments to identify the location of where a paraphrase happened. We will adjust the use of “sentence”, “phrase”, and “segment” to avoid any interpretation issues.\n\nRe: “I am unsure if using BERTScore / BLEU / ROUGE is enough here.”:\n\nWe agree that BERTScore, BLEU, and ROUGE are limited in capturing semantic similarity but are also commonly used for evaluating generative tasks. As the community often uses these metrics as a proxy in paraphrasing tasks, we decided to keep them so our findings/results can be easily compared in future research. Further, we inspected a few examples non-systematically throughout the experiments to confirm the expected model behavior.\n\nRe: “From Figure 10 in the appendix, it seems the input to this task is two sentences. Will the two \"re-phrased word pairs\" be provided?”:\n\nFor detection, the system receives two sentences with a specific indication about the location of the paraphrase. Therefore, multiple paraphrases can occur within the same sentence but at different locations. The system returns the single type or group of the paraphrase (e.g., Inflection Change, Lexicon Change). We will clarify this in the figure.\n\nThank you for the suggestions on additional related work. We will include these references in our related work for the camera-ready version of the paper. Once again, thank you for your time and for engaging with the material in this paper and our response. We see the time and thought you have put into this and very much appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation for incorporating fine-grained types into paraphrase generation and identification makes sense, though I am unclear on some details about the task setup.\n\n2. I think the experiment setup is reasonable and well-executed. The author conducts pretty comprehensive experiments to support their claim.", "reasons_to_reject": "1. I have a hard time following the paper. Here are some of my confusions: \n\n    - In line 193 and line 211, when you say give a phrase x, do you mean a sentence x or a multi-word phrase x? \n\nConfusions about paraphrase type generation: \n\n- The paraphrase type generation task is a sentence-level task or phrase-level task? From the example in Figure 10 in the appendix, it seems to be a sentence-level task. Then why in line 206, paraphrase *segments* are mentioned?\n- Assuming the paraphrase type generation is a sentence-level task, given an input x and a set of paraphrase types, assume the system returns x$^*$. How do you make sure  x$^*$ has the designated paraphrase types? I am unsure if using BERTScore / BLEU / ROUGE is enough here. Maybe you can do a small portion of manual inspection to see if the output has the paraphrase types, if not using some automatic method? \n\nConfusions about paraphrase type detection:\n\n- From Figure 10 in the appendix, it seems the input to this task are two sentences.  Will the two \"re-phrased word pairs\" be provided? Or they are the system outputs, in addition to the system output? What if there are multiple re-phrased word pairs?", "questions_to_authors": "Please see above.", "paper_topic_and_main_contributions": "This paper studies the task of paraphrase identification and generation. Unlike traditional binary settings, the authors consider 26 fine-grained paraphrase types.\n\nThe authors experiment with several baseline methods for both tasks and demonstrate that incorporating fine-grained paraphrase types can help boost paraphrase generation and detection tasks under binary labels.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Please consider citing the following papers, which are closely related to your work.\n\n1. PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation; Yixin Wan, Kuan-Hao Huang and Kai-Wei Chang; Findings of ACL 2023\n\n2. Improving Large-scale Paraphrase Acquisition and Generation; Yao Dou, Chao Jiang, Wei Xu; EMNLP 2022"}}, {"review_id": "HfJVq1IXIu", "reviewer": "Reviewer_ow9w", "summary": "", "questions": "Did you consider existing augmentation literature like SSMBA, ROTOM and UDA which also introduce fine-grained linguistic variations as augmentation operators? I believe comparing with some of those methods might give more insights into the general category of operators needed for controlled paraphrasing. \n\nAlso please consider adding statistical significance tests for the various performance measures reported across tasks and datasets.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewer for their feedback and confident observations about our work. We want to underline the assessment that this work is “of interest to the research community as paraphrase types help provide granular analysis of paraphrased text and also to control generated text”. We are happy to hear that “the experiments carried out are extensive and demonstrate the performance gain on ETPC corpus and general applicability to duplicate detection datasets”.\n\nRe \"Did you consider existing augmentation literature like SSMBA, ROTOM and UDA\":\n\nThank you for the interesting suggestions for data augmentation methods. We will investigate them thoroughly and add relevant comparisons to our related work and discussion.\n\nRe \"Also please consider adding statistical significance tests\":\n\nWe have conducted statistical tests for the detection task (see Appendix D.1). We will add them more prominently in the main body of the paper for the camera-ready version.\n\nOnce again, thank you for your time and for engaging with the material in this paper and our response. We see the time and thought you have put into this and very much appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The work proposes new paradigm for paraphrase detection by introducing paraphrase types which cover diverse linguistic variations that could be introduced by paraphrasing a given piece of text. It leverages the fine-grained types annotated in ETPC corpus for the paraphrase type identification task. The proposed task is interesting and is of interest to the research community as paraphrase types help provide granular analysis of paraphrased text and also to control generated text.\n\n2. The experiments carried out are extensive and not only demonstrate the performance gain on ETPC corpus but also demonstrate general applicability to duplicate detection datasets like Quora question pairs. The authors also evaluate the impact of incorporating paraphrase types for tasks of paraphrase identification and generation.\n\n3. The authors experiment with diverse models of different scales, also providing insight into the emergent capabilities of large language models and also smaller models in transfer learning setting.\n\nIn summary an interesting preliminary work into new task of paraphrase type generation and detection which of interest to the community", "reasons_to_reject": "1. I think statistical significance tests  along with reports of effect size are necessary to establish claims of superior performance. \n2. Most paraphrasing datasets have short single sentence samples. More analysis on realistic cases of paraphrasing paragraphs where semantic context has to be maintained for longer sequences would help determine generality. \n3. While it is appreciated that authors discuss limitations, it is not clear if the proposed paraphrase types are universally applicable to different datasets.  Though it is shown that including types helps in paraphrase detection on QQP the model trained on ETPC is applied to QQP without any type annotations for QQP. This may miss out some linguistic variations particular to QQP.", "questions_to_authors": "Did you consider existing augmentation literature like SSMBA, ROTOM and UDA which also introduce fine-grained linguistic variations as augmentation operators? I believe comparing with some of those methods might give more insights into the general category of operators needed for controlled paraphrasing. \n\nAlso please consider adding statistical significance tests for the various performance measures reported across tasks and datasets.", "paper_topic_and_main_contributions": "The work introduces two new paraphrase based tasks namely the paraphrase type detection and paraphrase type generation. The authors argue that existing semantic similarity based measures are limiting in paraphrase detection tasks as they do not consider the fine-grained linguistic variations in the paraphrases. The authors propose 7 different paraphrase types and study the effect of paraphrase types on paraphrase generation and paraphrase experiments. It is an interesting paradigm and the experimental results demonstrate positive impact of the proposed tasks and paraphrase types.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xI5vG9U5J5", "reviewer": "Reviewer_qoRy", "summary": "", "questions": "1)\tWere any significance test done on the results and improvements? \n2)\tWhy BART and PEGASUS (in table 2 and table 4) and not more recent decoder models like OPT? ", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for highlighting that this work is “well motivated and [...] shows the need of paraphrase type generation and detection” and that its findings “could be beneficial for future works”. We are also keen to hear that “the experiments are well motivated” and executed with multiple pre-trained language models and paraphrase identification datasets.\n\t\nQuestion 1: Were any significance test done on the results and improvements?\n\t\nAnswer 1: Yes. We have conducted statistical tests for the detection task (see Appendix D.1). We will add them more prominently in the main body of the paper for the camera-ready version and consider performing statistical tests for the generation task too.\n\nQuestion 2: Why BART and PEGASUS (in table 2 and table 4) and not more recent decoder models like OPT?\n\nAnswer 2: We chose BART and PEGASUS as they are usually a strong backbone of recent text generation and summarization systems (including in our previous experiments in correlated tasks). Additionally, as we propose new tasks, we wanted to establish a solid baseline for future contributions. This study’s goal is not to show/compare state-of-the-art performance but to confirm the hypothesis that paraphrase types improve detection and generation tasks.\n\nThank you for mentioning the recent work “Paraphrase Identification with Deep Learning: A Review of Datasets and Methods” by Zhou et al. We will discuss and compare that work for the camera-ready version.\n\nOnce again, thank you for your time and for engaging with the material in this paper and our response. We see the time", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)\tThe paper is well motivated and it shows the need of paraphrase type generation and detection. The work could be beneficial for future works to consider paraphrase types while solving paraphrase identification task. \n2)\tA good set of PLM were evaluated on multiple paraphrase identification datasets\n3)\tThe experiments are well motivated with corresponding research questions. ", "reasons_to_reject": "1)\tThe work intuitively makes that paraphrase type generation and detection would improve paraphrase identification task performance. \nIt could strengthen the impact of work if paraphrase generation (from different types) are shown to improve query rewriting/simplification and more downstream tasks. \n2)\tThe proposed work seems more of an analyses of various PLM and LM on paraphrase type generation and detection. The correlation between models on different paraphrase types and groups is interesting but would strengthen the work if these can be shown with more concrete applications or their usefulness. \n3)\tLLaMa with paraphrase types show some improvement (a little unclear from the plots in figure 3, and if the difference is significant or not) in few shot prompt setting. It would be good to know if the difference becomes negligible or more if LLMs like LLaMa are instruction finetuned with lets say 10 examples of each paraphrase type.\n4)\tSome of the previous works have explored similar directions of Paraphrase Identification with types (see Paraphrase Identification with Deep Learning: A Review of Datasets and Methods). It would be good to discuss some of the previous works in this direction. The correlation analyses is given noticeable focus in the paper but could be very helpful to understand their applications as well. ", "questions_to_authors": "1)\tWere any significance test done on the results and improvements? \n2)\tWhy BART and PEGASUS (in table 2 and table 4) and not more recent decoder models like OPT? ", "paper_topic_and_main_contributions": "The work is focused on tasks of paraphrase type generation and paraphrase type detection where authors evaluate several PLM, and LLaMa utilizing Extended Paraphrase Typology Corpus (ETPC).  Authors show correlations between different paraphrase type groups and show how paraphrase detection(and generation) could help in the end task of paraphrase detection (binary). Work shows several examples and discusses paraphrase types from ETPC dataset highlighting the potential benefits on paraphrase identification (and other relevant tasks).", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1)\tIt is difficult to read figure 3 without numbers to understand the improvements\n2)\tSmall point - It could improve readability if eval metric full forms are mentioned in table 2 caption. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the task of paraphrase identification and generation. Unlike traditional binary settings, the authors consider 26 fine-grained paraphrase types (most of which proposed by Kovatchev, Marti, Salamo 2018) -- this is a promising direction of research on paraphrase, and addresses the issue that the definition of paraphrase is often not precise enough. Authors conducted some interesting analyses on four paraphrase datasets, with Llama and several other LLMs. 

Author's responses also raised some confidence -- two of the reviewers are happy with the author responses. We hope the authors will make improvements based on the review comments for their camera-ready, if the paper gets accepted. We also want to ask the authors to discuss more thoroughly the related work and limitations.
Decision: Accept-Main