An Intent-based and Annotation-free Method for Duplicate Question Detection in CQA Forums
Authors: Yubo Shu
Hansu Gu
Peng Zhang
Tun Lu
Ning Gu
Venue: EMNLP 2023 Findings (2023)
Keywords: Sentence-level Semantics
Textual Inference
Data deduplicating
Instruct-tuning

Abstract: With the advent of large language models (LLMs), Community Question Answering (CQA) forums offer well-curated questions and answers that can be utilized for instruction-tuning, effectively training LLMs to be aligned with human intents. However, the issue of duplicate questions arises as the volume of content within CQA continues to grow, posing a threat to content quality. Recent research highlights the benefits of detecting and eliminating duplicate content. It not only enhances the LLMs' ability to generalize across diverse intents but also improves the efficiency of training data utilization while addressing concerns related to information leakage. However, existing methods for detecting duplicate questions in CQA typically rely on generic text-pair matching models, overlooking the intent behind the questions. In this paper, we propose a novel intent-based duplication detector named Intent-DQD that comprehensively leverages intent information to address the problem of duplicate question detection in CQA. Intent-DQD first leverages the characteristics in CQA forums and extracts training labels to recognize and match intents without human annotation. Intent-DQD then effectively aggregates intent-level relations and establishes question-level relations to enable intent-aware duplication detection. Experimental results on fifteen distinct domains from both CQADupStack and Stack Overflow datasets demonstrate the effectiveness of Intent-DQD. Reproducible codes and datasets will be released upon publication of the paper.

Intent-DQD:
With the advent of large language models (LLMs), Community Question Answering (CQA) forums offer well-curated questions and answers that be utilized for instruction- luning, effectively training LLMs to be aligned with human intents. However, the issue of du- plicate questions arises as the volume of con- tent within CQA continues to grow, posing threat to content quality. Recent research high- lights the benefits of detecting and eliminat- ing duplicate content. It not only enhances the LLMs' ability to generalize across diverse intents but also improves the efficiency of train- ing data utilization while addressing concerns related to information leakage. However, isting methods for detecting duplicate ques- tions in CQA typically rely on generic text- pair matching models, overlooking the intent behind the questions. In this paper, we pro- pose novel intent-based duplication detector named Intent-DQU that comprehensively lever- ages intent information to address the problem of duplicate question detection in CQA. Intent- An Intent-based and Annotation-free Method for Duplicate Question Detection in CQA Forums
Yubo Shu', Hansu Gu², Peng Zhang'; Tun Lu'; Ning Gu'
Fuudnn University, Shanghai, China
Seattle, United States
ybshu200fudan.edd edu. cn, hansug@acm. org
{zhangpeng_, lutun, ninggu)efudan. mnnguu)efudaneedu.cc edu. cn
Abstract
well-curated questions and answers that can be ef- fectively utilized for instruction-tuning for large language models (LLMs). Instruction tuning plays a vital role in the Reinforcement Learning from Hu- man Feedback (RLHF) process employed in build- ing systems like ChatGPT and Claude (Ouyang et al., 2022; Bai et al., 2022). The purpose of in- struction tuning is train LLMs better follow human intents. For instance, when human posts question to an LLM, instruction tuning helps the LLM learn to provide an answer instead of merely imitating the act of asking a question. As the vol- ume of content in CQA forums continues to in- crease, the issue of duplicate questions becomes more prevalent, thereby posing a significant chal- lenge to content quality.
Recent research emphasizes the significance of detecting and removing duplicate content in the instruction-tuning of LLMs. It promotes gener- alization across diverse intents (Hemannezz et al., 2022), reduces overlap between training and test sets (Lee et al., 2022), mitigates information leak- age risks (Kandpal et al., 2022), enhances learning stability, and improves training data efficiency. In- spiringly, Zhou et al. (2023) discovers that even carefully selected set of 1,000 unique questions can effectively train LLM to follow intents and achieve competitive performance when compared to OpenAl's
DQD first leverages the characteristics in CQA forums extracts training labels to recognize and match intents without human annotation. Intent-DQD then effectively aggregates intent- level relations and establishes question-lewe level relations enable intent-aware duplication de- tection. Experimental results on fifteen distinct domains from both CQADupStack and Stack Overflow datasets demonstrate the effective- ness of Intent-DQD. Reproducible codes and datasets will be released upon publication of the paper.
model.
Despite the impressive results of previous stud- ies in detecting duplication Liang et al. (2019); Xia et al. (2021); Peng et al. (2022), these methods do not consider intents in question, which may lead to sub-optimal results when deduplicating content. Figure shows two question pairs that illustrate the importance of considering intents in duplication detection for instruction-tuning. From a generic text-pair matching perspective, both the exemplar duplicate and neutral pairs are partially semanti- cally matched. Despite being mismatched the non-intent part, the first pair is labeled as duplica-
1 Introduction
With the rapid development of web technology, Community Question Answering (CQA) forums have emerged as crucial platforms for individuals to seek solutions to their problems. These forums serve high-quality knowledge bases, offering
"Corresponding author

---

Ql:
Label: Duplicate
We propose method to aggregate intent-level relations and obtain question-level relations detect duplicate questions with multiple intents.
Goughel auppeling vorkil
Q2
match
mismmate atch
Jin aos--dee
My Panction wtw.eetttttttttttapap..p...o
buidie.
Label: Neutral
Q3
heer gpan chootr gane lam milenng sootto by sund may tire par batton frm.
The experimental results on two datasets covering fifteen domains demonstrate that our method can leverage question intent and achieve better performance over previous methods.
Q4
malch
mismatch
minnn spar gare, fan
dep
Figure 1: Example question pairs sampled from CQA forums. The highlights are the intents in questions. In the first pair, QI and Q2 are annotated as "Duplicate" by forum users. QI and Q2 have matched intents to solve a particular java error while having mismatched backgrounds. In the second pair, Q3 and Q4 are treated as neutral questions in the forum. Q3 and Q4 have matched backgrounds mismatched intents.
Annotation-free Label Collection
The primary challenge is the absence of labels for (1) intent recognition, which involves determining whether sentence expresses an intent or not; (2) in- tent matching, which involves classifying whether the relationship between intents is matched or mis- matched. In order to overcome this challenge, we introduce strategies for mining labels without hu- man annotation by leveraging the characteristics of CQA forums.
tion due the matched question intent. Meanwhile, the second pair only matches in non-intent content and thus should be treated as neutral.
Therefore, our goal is to highlight intents in du- plicate question detection. However, this task re- mains challenging as follows: (1) Accurately rec- ognizing intent from users' posts is difficult due to the high diversity of intent in different QA forums, especially without sufficient human annotation. (2) Determining whether two lexically differrnnt intents have the same semantics can be difficult because most annotations are at question-level instead of intent-level. (3) One question posted online fo- rums often carries more than one intent, and it is unclear how to handle multiple intents for duplicate question detection.
2.1 Labels for Intent Recognition
The intent of a question refers to the user's goal purpose asking it. We have found that in most cases, the question title serves as a concise repre- sentation of what users are asking, which can be considered as intent. However, only using titles may lead to noise when titles are too short or do not provide enough information, such "Order list" and "What should do?". Inspired by previ- ous research (Beyer al., 2020), we additionally use regular expression patterns to match more ac- curate intents and eliminate overly generic intents only with pronouns, such as "how to do it/this/that". More information about used regex patterns is in Appendix A.I.
To address the above challenges, first lever- age the characteristics in CQA forums and ex- tract abundant training labels to both recognize and match intents without human annotation. Based on matched intent pairs, we leverage supervised con- trastive learning (Khosla et al., 2020) and properly learn intent representations in a low-dimensional space. Finally, we design similarity matrix on the question pair level, based inter and intra similar- ities between multiple intents for each question to detect duplicate questions.
2.2 Labels for Intent Matching
After recognizing intents, there exist three types of links which can used to derive valid labels for intent matching. Consider common situation where a pair of questions is labeled as duplication in CQA forums, as shown in Figure 2, we can extract semantically matched intent pairs from inter, intra and mixed-links:
The key contributions of this paper are summa- rized as follows:
We leverage the features of intent in CQA forums and implement an effective intent rec- ognizer without human annotation.
Intra-link Inside one question, the intent in the title and the intent in the body can be linked. As the title and the body are expected be consistent, their intents are likely seman- tically the same.
We automatically match semantically iden- tical intent pairs and separate non-identical ones through supervised contrastive learning.

---

Find number of lines in csv without reading it
How to line count of large file cheaply in Python?
arredy
How on olancel lage Ceuply Nydioe anowed
1255
opend (1lemmo) ummurate(()):
Choudd
pans reture
there way finding the number lines
withhout actually loading the whole file memory (in Python]?
count file the
expect can special accimbedd function
time efficient manner?
imegin now read by line and the lnee, but the possible sense since only need the number of lines, not actual content.
Intent
Inter-link
Intra-link
Mixed-link
Figure 2: An example of mining matched intents through inter, intra, mixed-links pair of duplicate questions.
Inter-link Between a pair of duplicate ques- tions, intents in their titles can be duplicate as well.
3.1 Intent Recognition
To ensure the accuracy intent recognition, we incorporate three types of features in addition to transformer- language models. We first fo- cus keywords that match tags in the QA forum. Matches such "Tensorflow" or "PyTorch" indi- cate the mention of important concepts that are recognized by the QA forum, and are thus helpful for the model to learn if they are related to intent detection. We build a keyword vocabulary from all tags in the forum and use a binary flag to in- dicate whether the current word matches the tag, Additionally, considering the sentence "What is the difference between Concept A and Concept B?", can classify the sentence intent without knowing the specific concepts. So, we add part-of- speech (POS) information help the model utilize the linguistic features of sentence. Lastly, we use sentence position in the question as a feature because some users may prefer to put intents at the start or the end of post. We implement all three features extra embeddings added on top of existing text token embedding illustrated in Figure 3(b).
Mixed-link We can combine intra-links and inter-links to create mixed-links and further generate more training samples for intent matching.
Besides question pairs, it is worth noting that the intra-link can also work even within single ques- tion. Similarly, we extract negative intent pairs from questions labeled as "related but not dupli- cate" in CQA forums by utilizing inter and mixed links. These negative intent pairs often exhibit lexi- cal similarity but possess distinct semantic mean- ings. Consequently, pairs formed by these intents serve hard negative labels for training.
3 Method
After label collection, we proceed to present our method for detecting duplicate questions. Intent- DQD consists of three key components: intent recognition module, intent matching module, and duplicate question detection module. Figure shows the overall structure of our method. In the intent recognition module, we exploit the inner fea- tures of intent sentences to augment recognition accuracy. In the intent matching module, we lever- age supervised contrastive learning (Khosla et al., 2020) and properly learn intent representations in low-dimensional space. the duplicate question detection module, we design similarity matrix on the question pair level to integrate inter and intra-similarities between multiple intents.
3.2 Intent Matching
To distinguish matched or mismatched intents, we develop a dedicated intent matching layer based on contrastive learning. After the label collec- tion process, we have collected both matched in- tents pairs (positive samples) and mismatched in- tent pairs (negative samples). We then feed these intent pairs to calculate the loss function of pervised contrastive learning. More specifically, Int.Sct represents dataset of intents, and inenty is the k-th intent in the dataset. Inside a training batch, we define: P(k) contains positive intents

---

Softmax Classifier
Dup Question Detection
Inter Interaation
Pooling
intent
non, Intent
RQa
hop
Softmax Classifier
Bi-Encoder
RAA
R
R
Eoertencee Posttion
Reyworrd
Flag
Intent Matching
15
15
Contrastive Encoder
PCS
Intent
+ internte intenat intene?
Embedding layers
Recognition
Intents
Gifteroncc
Pysorch
Intent Recognizer
Sentence,
Sentence,
Sentencey
(Qa Qu)
@a) Intent-DQD
(b) Intent Recognizer
The overall structure of our method. Part (a) the left shows proposed intent-DQU processing question pairs through intent recognition, intent matching and duplicate question detection module. Part (b) on the right shows the details of the intent recognizer. Filtering by Intent prediction. After intent recognition, method will select sentences predicted as the Intent label, passing them to the intent matching module.
Figure 3:
of inenly; and N(k) contains negative intents of inenty; A(k) N (k) P(k) \ iinenty is union set excluding inentg. For inentg, the supervised contrastive loss is formulated as:
RBB = -IP 1I,,I,,... IR
1
1
I
RAB = [71, 15 IA] [7 I
The first matrix RAA MxM is used match intents within QA. Similarly, the second RBB RNN is used to match intents within QB- The third RAB E M is used to match intents be- tween Q^ and QB- The idea behind matching in- tent in this way is to provide a comprehensive view of question-level relation identification. In the next module, we will introduce how to aggregate these intent-level relations into question-level relations for duplication detection.
exp(sim(Tk: Ip)/T) exp(sim( Je, Ia)/T)
Σ log Σ REA(K)
IP(R)
PEP())
where denotes the encoded vector of a intent, and sim donates cosine similarity function and I E + is a scalar temperature parameter. During the training process, optimizing the loss function for each intent in Int.St will decrease the distance between positive intents in the embedding space and increase the distance between negative intents.
3.3 Duplicate Question Detection
We use the intent embeddings from contrastive learning to match intents within a pair of ques- tions. As shown in figure 3, given two questions QA and QB. we encode the intents of QA as a list of intent embeddings [1 A .... M and intents of QB || 1 .... [. The M and N represent the number of intents in Q and QB, respectively (e.g., M and N = for the case in the fig- ure). We match intents within each question and across questions, generating the following intent similarity matrices:
worth noting that question can have one multiple intents. Therefore, in the duplicate ques- tion detection module, we combine the above ma- trices and calculate a holistic similarity matrix S between QA and QB. The shape of S is M * N. The similarity score at the i-th row and j-th column, S.y, represents not only an intent-level relation be- tween intent, and intent but also the degree which this intent-level relation contributes the question-level relation. For example, if intents and intent B are semantically matched, its contri- bution degree to S,y depends on two situations: (1)
R4A 1 1
[1 1

---

intent, is also matched with other intents in QA. indicating that QA is a focused question, which makes the match between intenty and inent more significant for duplication.
2017). The dataset includes data from 12 CQA sites Stack Exchange, such as Android, English, Game, Mathematics, Physics and Unix. We use duplicate questions and neutral questions in the dataset and randomly split them into training, vali- dation, and test sets with ratio of 8:1:1. However, the dataset does not include the largest forum Stack Overflow (SO) as the authors stated that SO data was too large to process at that time. To overcome the limitation, we have constructed an additional dataset, the Stack Overflow duplicate questions dataset (SODup).
(2) intentf is mismatched with other intents in QA meaning that the match between intent; and intenty has mi- nor contribution for duplication. For this purpose, when calculating Sij. we take into account the in- ner intent similarity matrices AAA and RBB with RAB, The explicit formula of Sij is below:
M
RAA
RAB
REB
Siy
(1)
SODup dataset. Stack Overflow is a widely used CQA forum that focuses on code-related ques- tions. All questions and question links are avail- able the official data dump.' We downloaded the data dump and then filtered questions accord- ing to the three most popular programming lan- guages: JavaScript, Python and Java. We extract the duplicate/neuutal question relations based on LinkTypeld field of PostLinks table: (1) Duplicate relation: If question pair has LinkTypeld=3, it means that the pair is marked as "Duplicate" by forum users. Therefore, recognize the ques- tion pairs having LinkTyyeld=3 as duplicate rela- tions. (2) Neutral relation: Instead of just using random question pairs which are too simple for classífication, we also gather indirectly referenced questions as neutral relations which serve as hard neutrals. Specifically, a question pair with Link- Typeld=1 means that one question references the other. However, some of the question pairs with a LinkTypeld=1 in the data dump indeed dupli- cates in reality due to the delay of user annotation. To increase the probability of being true neutrals, we use questions with indirect references as neutral relations. The ratio of random and indirectly ref- erenced questions is 1:1 ,add the ratio of duplicate and neutral relations is also 1:1. We adopt the same setting 8:1:1 for training, validation and test split in the SODup dataset.
RAA
REB)
After calculating S E RMXN, as shown in Fig- ure 3, we obtain the final feature f by concatenating the max pooling of S with the interaction of ques- tion embedding h A and hB from the bi-encoder:
f = [hh hB: I - hal: Mar Poolingg)
We use a linear layer followed by a Softmax to make the final prediction:
= softmantReLUWW b)).
where ÿ is the prediction of probability in (duplicate, neutral). For duplicate question de- tection, the ultimate goal is to optimize the standard binary cross-entropy loss function.
4 Experiment Setting
4.1 Datasets
We introduce two datasets covering fifteen different domains evaluate our method. The first dataset is CQADupStack, and the second dataset comes from the Stack Overflow data dump. Table shows the statistics of two datasets.
Table I: The statics of datasets
CQADupStack
SODup
4.2 Training Details
of question pairs
48090
100000
In order to avoid label leakages, there no overlap on question ids between the test dataset for dupli- cation question detection and the dataset for intent recognition and matching. We train intent recogni- tion and duplicate question detection with a batch size of 32. The batch size for contrastive learn- ing in intent matching is 16. We set maximum
of intents of non-intents
259588
467816
404373
595488
intent rate
39.1%
44.0%
average intents per question
3.12
3.56
CQADupStack dataset was released by
A Ahasanuzzaman et al. (2016) and has become a popular benchmark for CQA research (Nakov et al.,

---

Table 2: Overall performance of duplicate question detection.
SODup
CQADupStack
J8
Py
Java
And.
Ein..
Game
Gis
Math
Phys.
Prog.
Stats
Tex
Unix
Weh.
Word.
BiMPM
0.847
0.870 0.878 0.892 0.882 0.908 0.766 0.820 0.913 0.850 0.819 0.886 0.860 0.921 0.818
MFAE
0.853
0.869 0.890 0.901 0.870 0.857
0.758 0.832 0.899 0.876 0.818 0.897 0.894 0.917 0.784
SBERT
0.868
0.873 0.887 0.901
0.911 0.888
0.797
0.826
0.924 0.890
0.802
0.894
0.892
0,932
0.804
SRoBERTa
0.896
0.891 0.896
0.933
0.893
0.923
0.820
0.803
0.929
0.902
0.852 0.901
0.904
0.942
0.838
Intent-DQD 0.887 0.882 0.893
0.936
0.925
0.910
0.837
0.880
0.944
0.913
0.885
0.898
0.906
0.946 0.838
Bert
Intent-DQD 0.908 0.902 0.914 0.956 0.925 0.956 0.838
0.874 0.946 0.931 0.852 0.921 0.921 0.953 0.878
Roberta
sequence length of 512 for single question and 128 for a single intent. We train 20 epochs for con- trastive learning and 10 epochs for classification tasks. We set the learning rate to 5e-5 and used a training scheduler with a weight decay rate of 0.01 and one epoch warm-up. up. In intent recognition, we use Spacy package to extract POS information. For duplicate question detection, the top-3 performing checkpoints on the validation set will be evaluated on the test set to report average results.
does not differentiate intent, and all emphasized keywords can also appear in non-intent sentences, which may introduce extra noise when detecting duplicates.
The result of ablation study, It shows the overall duplicate question detection performance on the two datasets. A, B and C donate three combinations of intent recognition (ir), intent matching(im) and relation aggregation(ra) illustrated Formula
Table 3:
ir
im
ra
SODup
CQADup.
5 Result and Analysis
X
X
0.894
0.896
B
V
X
X
0.903
0.912
5.1 Duplicate Question Detection Result
/
x
0.900
0.914
We compare Intent-DQD with a few base- lines, including BiMPM((Wng et al., 2017), and MFAE(Zhang et al., 2020) that are representative duplicate question detection methods, as well as general-purposed SBERT and SRoBERTa models with fine-tuning. BiMPM encodes each question individually and then employs a multi-persppectvy matching mechanism for question pairs. MF'A uti- lizes pre-trained language models to encode ques- tions and enhance keywords in question match- ing. Additionally, compare with SBERT and SRoBERTa, we implement a BERT-based and RoBeRTa-based IQD, respectively.
Intent-DQD
V
V
V
0.908
0.921
5.2 Ablation Study
Since Intent-DQD has different components, it is essential to dissect the impact of each component contributing to performance improvement. Consid- ering that Roberta-based Intent-DQD achieves the highest performance in most forums, choose it for the following ablation experiments.
As shown in Table 3, the combination is a SRoBERTa-only model which performs the worst, and the complete Intent-DQD achieves the high- est performance on both datasets. With only in- tent recognition, combination B can only feed the recognized intents into Bi-Encoder and combine encoded intents with encoded questions. Even so, see an increase in performance, which suggests the significance of utilizing intents for detecting du- plicate questions. However, combination C, only using intent matching can slightly increase perfor- mance on CQADupStack dataset and even decrease
Table demonstrates the F1-score of classifica- tion results on fifteen domains. From the results, our Intent-DQD outperforms the baselines in all fifteen domains. The largest improvement is on the Math domain at 4.8%, and the least increase is 1. 1% on the Python domain. We observe that even incor- porating keyword emphasis on top of BERT, MFAE is not able to surpass most domains. One possible reason is that MFAE
SBERT in

---

im
Inteir-DQQ
SRoBERTa
Diffxrence betwenn
expressions are used almost idemically in different contexts, Is there difference
16
got gotien?
Case
15
14
Lntice that Americans the word ppotien' when Hritain just use bott pottent necented American Englishs, that is used and accepted English ctamination papers, ot bipe slang, speech, but not written! The previous answer you directed me to doesa't quite answer er my question, ulthough was interesting, can deduce from However, That 'eatten. if used correctly, indeed acceptable and setual word, is right?
Duplicate
Neutral
Duplicale
12
12
.56
1
.58
.22
12
.56
1
.62
.21
QA: Puthon masttion ion How can Inon back my code? Currenily have
19
12
code cxiting when an crror detected. It will otit if your income is
inputted negative and will exit if posteode entered string.
1
4
But need be able to create a loop so will give you another chance to values without
exiting, New python and haven't figured it our. Any help would he very approcisted
15
15
Case 2:
Noutral
Duplicate
Neutral
QR: How to loop back to specific point in code. So I'm programming
1
12
game
erying something
don't understund
do. I've defined function, and when none of the conditions with
12
.57
.49
.45
code works, want
other
code.
be as clear possible whenn snsw because I've just started learn programming and I know almost nothing about it.
19
.57
1
.39
.35
Figure 4: Two cases that Intent-Dog DQD correctly makes predictions while SRoBERTa makes wrong predictions
the performance on SODup dataset. This is likely due the fact that questions forums tend to have multiple intents, and without effectively us- ing intent-level relations, it is not comprehensive enough to determine question-level relations. As shown in Table 1, SODup has higher average in- tent per question than CQADupStack, thus more challenging without proper question-level aggre- gation. Lastly, we observe that aggregating intent- level relation can significantly improve classifica- tion performance, which further verifies that the relation aggregation module crucial and it can only achieve its effectiveness when paired with the intent matching modulc.
Table 4: As mentioned in Sec 3.1. R uses Regex patterns find intents for train data. treats Titles as intents for train data. S integrates sentence Structure features(POS. Sentence Position, and Keyword) in training, Our result in table 4 shows that the combination of R, T and S can produce an effective intent recogmizer without human annotation.
Precision
Recall
F1
0.816
0.621
0.706
T
0.914
0.633
0.748
R+T R+T+S
0.862
0.809
0,834
0.864
0.831
0.847
5.4 Case Study
5.3 Intent Recognition Verification
To better understand how Intent-DQD works, we have studied two test from the CQADupStack and SODup datasets. To analyze the contributions of each component, we further provide interme- diate outputs to illustrate how intent recognition, intent matching, and duplicate question detection contribute to duplicate detection in the two cases.
Since the intent recognition is the basis of our method, it is essential to verify the performance of intent recognizer. We randomly sample 5,000 sentences from QA forums and manually annotate whether sentence is intent or not. After reaching an agreement (kappa > 0.9), authors annotated a test dataset including 1,916 sentences labeled as intent and 3,084 labeled as non-intent. We then verify our intent recognition module on the test dataset. Table 4 shows that either regex title strategy is valuable for recognizing intents. The extra improvement made when combining regex and title suggests that they are complementary to each other. Lastly, the Fi-score reaching 0.847 with structure features verifies the effectiveness of all sub-modules within our intent recognizer.
The first case consists of two questions from the English domain. The label of the question pair is "Duplicate" as both questions are asking for the dif- ference between "got" and "gotten". The prediction of intent-DQD is "Duplicate" while SRoBERTa predicts "Neutral". There two main reasons for SRoBERTa's mistake: (1) The non-intent content has semantic discrepancy. (2) The intent content between the two questions has a large lexical gap. With the help of contrastive learning, our intent

---

matching component in Intent-DQD is able out- put a high matching score of the intent pairs, thus closing the lexical gap. As seen in Figure 4, QA has quite consistent intents, and the main intent of QB matches the intent of QA.
continue the question generate similar questions. whereas an aligned LLM trained through instruc-
tion tuning is more likely to generate an answer based on the question.
The second case consists of two questions from the python domain. The label of the question pair is "Neutral". Although both questions are about looping in python, the nuances of the intents make a difference i.e. the second intent of QA is to catch errors and loop, while the second intent of QB is to loop after condition control such as if-else. Ad- ditionally, the non-intents of the two questions are similar, and they both indicate that the question authors have little experience in the python domain. Therefore, it is not surprising that SRoBERTa pre- diets it as "Duplicate". As for Intent-DQD. the intent matching component can discover the differ- ence between I and I because of training from hard negative samples. In the end, the relation aggregation component can comprehensively con- sider intent-level relations and produce relatively low question-level match scores for the duplication classification.
Collecting high-quality training data is funda- mental for effective instruction tuning, CQA fo- rums have millions of well-aligned Q&A A data, mak- ing them ideal sources for instruction tuning (Zhou et al., 2023). However, the issue of duplicate ques- tions arises as the volume of content within CQA continues grow, posing threat to content qual- ity. Recent research highlights the importance of detecting and removing duplication to optimize for LLM instruction tuning. Hernandez et al. (2022) find deduplication increases generalization ability and learning stability. It can also save computation and decrease train-test set overlap Lee et al. (2022). Deduplication also reduces privacy risks because LLM is vulnerable to privacy attacks when sensi- tive information duplicated in the instruct-tuning dataset (Kandpal et al., 2022).
Existing duplication detection studics mainly build text-pair classifiers based on bi-encoders or cross-encoders, Bi-encoders (Mueller and Thya- garajan, 2016; Conneau et al., 2017; Reimers and Gurevych, 2019; Zhang et al., 2020), encode each question individually while cross cross-encodees (De- vlin al., 2019; Liang et al., 2019; Liu al., 2019; He et al., 2020), use cross-attention between questions or self-attention by concatenating mul- tiple questions. Despite the popularity of cross- encoders, they typically have high computational cost (Reimers and Gurevych, 2019) and inconsis- tent performance (Chen et al.. 2020). In this work. focus bi-encoders, which are more computa- tionally efficient and have shown consistent perfor- mance in duplicate question detection. However, the thhove-mentiones methods are generally generic text-pair matching and do not consider intents in question, which can lead to sub-optimal results in deduplicating datasets for instruction tuning.
5.5 Scalability in Real-world
our method, given a new question, the interme- diate results of intent recognition and intent em- beddings can be stored and reused. Additionally, a scalable paradigm for detecting duplicates typically involves two stages: (1) coarse- -rannee stage that retrieves duplication candidate question pairs and (2) fine-grained stage that classifies candidates to detect duplication. Intent-DQD contributes directly the fine-gre -reeggraiedd stage and thus our evaluation focuses on the classification results. As for the coarse-grained stage, our intent embeddings can be utilized to construct question-level embeddings and extend to duplication candidate retrieval.
6 Related Work
Instruction tuning is commonly regarded as the first step in the RLHF training process for train- ing human-aligned LLMs including GPT-3.5, GPT- 4, Claude (Ouyang et al., 2022; Bai et al., 2022). The goal of instruction tuning is to train LLM in following human intents expressed in instructions (OpenAI, 2023; Taori et al., 2023; Chiang al., 2023). For instance, when human asks a question to a LLM, their intent is to receive a proper answer. Before instruction tuning, a pre-trained LLM may
Therefore, we incorporate intent features in Intent-DQD to detect duplicate questions. Unlike existing research on intents (Kim et al., 2016; Coucke al., 2018; Weld et al., 2021, 2022) which typically rely on human annotation, our proposed method is annotation- which leverages charac- teristics of CQA forums and automatically mine labels for intents.

---

7 Conclusion
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:221..08073.
In this work, we propose an intent-based duplicate question detection method Intent-DQD. We design intent recognition to identify question intent with- out human annotation. Intent matching employs contrastive learning to enhance the embeddings. Duplicate question detection effectively aggregates intent-level relations and obtains question-level re- lations. Evaluation on fifteen domains from two real-world datasets shows that our method outper- forms existing tate-of-the-aa methods.
Stefanie Beyer, Christian Macho, Massimiliano Di Penta, and Martin Pinzger. 2020. What kind of questions do developers ask on stack overflow? comparison of automated approaches classify posts into question categories. Empirical Software Engineering, 25(3):2258-25) 23011.
Hannah Chen, Yangfeng Ji, and David K Evans. 2020. Pointwise paraphrase appraisal potentially prob- lematic. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Linguistics: Student Research Workshop. pages 150-155.
8 Limitations
The main assumption of this work is that the user- generated content in CQA forums contains explicit intent. The effectiveness of our carefully designed intent recognition, intent matching and duplicate detection modules may have limited success if the questions do not have clear intent. This can be true in forums that focus open discussions, where the objective of the discussion is unclear.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chathot impressing gpt-4 with 90%* chatgpt quality.
Alexis Conneau, Douwe Kiela, Holger Sehwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 670-680.
9 Ethies Statement
We acknowledge and conform ACL and ACM Code of Ethics. The datasets we used are from the official public data source, and we use the data under the license. In data processing, we do not use user private information. Our method does not create toxic content or bias in different groups.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, al. 2018. Snips voice plat- form: an embedded spoken language understanding system for private-by-oe by-design voice interfaces. arXiv preprint arXiv: 10790.
10 Acknowledgments
We would like to thank the anonymous review- ers and appreciate their insightful comments that helped improve the quality of the paper. This work supported by the National Natural Sci- ence Foundation of China (NSFC) under Grants 61932007 and 62172106. Peng Zhang is a faculty of School of Computer Science, Fudan University. Tun Lu is a faculty of School of Computer Science and Shanghai Key Laboratory of Data Science, Fu- dan University and Shanghai Institute of Intelligent Electronics & Systems. Peng Zhang and Tun Lu are the corresponding authors.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pages 4171- 4186.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El- El-Showk, Nelson Elhage, Zac Hltfield-Dodds, Tom Henighan. Tristan Hume, et al. 2022. Scaling laws and interpretabil- ity of learning from repeated data. arXiv preprint arXiv:2205. 10487.
References
Muhammad Ahasanuzzaman, Muhammad Asaduzza- man. Chanchal K Roy, and Kevin A Schneider. 2016.
Mining duplicate questions of stack overflow. In 2016 IEEE/ACM 13th Working Conference on Min- ing Software Repositories (MSR), pages 402 412.
Nikhil Kandpal, Eric Wallace and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks
IEE.

---

in language models. In International Conference on Machine Learning, pages 10697-10707. PMLR.
Nils Reimers and Iryna Gurevych. 2019, Sentence-berts Sentence embeddings using siamese bert-networke In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joinl Conference on Natural Language Processing (EMNLP-IJCN(P)) pages 3982-3992.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce I. Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. Advances in Neural Information Processing Systems, 11:16661-
Rohan Taori, Ishaan Gulrajami, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsumori B. Hashimoto. 2023. Stanford alpaca: An instruction-followiin llama model. https:// gi thub. com/ hwtsssullabsstafordddro
Seokhwan Kim, Lais Fernando D'Haro, Rafael E. Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino. 2016. The fifth dialog state track- ing challenge. In 2016 IEEE Spoken Language Tech- nology Workshop (SLT), pages 511-517. IEEE.
Zhiguo Wang, Wael Hamza, and Radu Florian, 2017. Bilateral multi-perspective matching for natural lan- guage sentences. In Proceedings of the 26th Inter- national Joint Conference on Artificial Intelligence. pages 4 4154-4150.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang. Douglas Eck, Chris Kallison-Burchh and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8424 8445.
Henry Weld, Guanghao Huang. Jean Lee, Tongshu Zhang, Kunze Wang. Xinghong Guo, Siqu Long, Josiah Poon, and Caren Han. 2021. Conda: con- textual dual--nnotttedd dataset for in ga-a toxicity understanding and detection. In Findings of the Asso- ciation for Computational Linguistics; ACI-JJCNLP 2021, pages 1406-241.
Di Liang, Fubao Zhang, Weidong Zhang, Qi Zhang, Jin- lan Fu, Minlong Peng. Tao Gui, and Xuanjing Huang. 2019. Adaptive multi-attention network incorporat- ing answer information for duplicate question detec- tion. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 95-104.
Henry Weld, Xiaoqi Huang. Siqu Long, Josiah Poon, and Soyeon Caren Han. 2022. A survey joint intent detection and slot filling models in natural language understanding. ACM Computing Surveys, 55 38.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 1eXXiv:997..1. 11692.
Tingyu Xia, Yue Wang, Yuan Tian, and Yi Chang. 2021. Using prior knowledge to guide bert's attention in semantic textual matching tasks. Proceedings of the Web Conference 2021, pages 1466-2475.
Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence simi- larity. In Proceedings of the AAAI conference on artificial intelligence, volume 30.
Rong Zhang, Qifei Zhou, Bo Wu, Weiping Li, and Tong Mo. 2020. What do questions exactly ask? mfae: Duplicate question identification with multi-fusion asking emphasis. In Proceedings of the 2020 SIAM International Conference on Data Mining, pages 226-
Preslav Nakov, Doris Hoogeveen, Lluís Mårquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017. SemEval-2017 task 3: Community question answering. In Proceed- ings of the 11th International Workshop on Semantic Evaluation somnEval-2017. pages 27-48, Vancou- ver, Canada. Association for Computational Linguis- tics.
234. SIAM.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, 2023. Lima: Less more for alignment. arXiv preprint arXiv:2305.1 1206.
A Appendix
OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
A.1 Regex Expression Examples
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Regex expressions for mining intents:
doox ["]]+)]033 work can ([jwe) (osduppyytt)))))))))) dellesplaint ("4]+1(033 why
(dosfknow dox [10]5 (exis)
GeX ["s] 5) Gabmy someX (",3))00,3) (way)
(canloould (canknoldshhouddd should (lyoulx 100)0 solvesoleefffof (105)) (eroo]/w(00 (0,35) lexception)
(emonn fexcepioonn) (11,5)) (logcat sustmmelommppgeggggg
Qiwei Peng, David Weir, Julie Weeds, and Yekun Chai. 2022. Predicate-are aardicctte-ruummnt argument based bi bicenr for para- phrase identification. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 5579 5589.
(log's Morssttstadesttree logigndleX (105) (emoofocccprion)
Patterns
for
removing
noise:
howt (1055) salve (ilhhsshhat++ understandi ("(5))11155 how (0.5) (ithistaat)+ (try miedrrynng (*1)))(115) how (1 (1*))+) (ratisthaat+

---

A.2 Detail Staties of Datasets
A.4 ChatGPT for Duplicate Question Detection
Table shows the statics of each domain in SODup dataset, and Table 6 shows the CQADupStack dataset.
We have randomly selected 100 question pairs from SODup test data and 100 pairs from CQADupStack test data. Then we ask ChatGPT (GPT-3.5-turbb to classify these pairs, and the final F1 Fl-scores are 0.855 and 0.861 for SODup and CQADup whereas the F1-scores for Intent-DQD 0.92 respectively. The results imply that dupli- cate question detection can be more complex than initially anticipated. The complexity arises from the inherent difficulty in precisely defining what constitutes duplication. In CQA forums, duplicate questions are not strictly limited to having exact lexical or semantic equality. Instead, duplication encompass scenarios where question be inferred from infer another question to some degree. However, it is difficult to have an explicit definition, making it equally challenging for Chat- GPT to accurately detect duplication.
Table The statics SODup dataset.
Domain
# of Question Pairs
# of Intents
0.9 and
Javascript Python
33334 33334 33334
171061 153704
Jäva
143051
Table 6: The statics of CQADupStack dataset.
Domain
# of Question Pairs
# of Intents
Android
3426
18529
English
7782
32412
Game
4558
23813
Gis
2232
14270
mathematics
2744
15793
Physics
3936
22075
Programmers
3476
22315
Stats
1832
11714
Tex
10390
54887
Unix
3428
20164
A.5
Relationship between the intent recognition and DQD
Webmasters
2790
14474
Wordpress
1496
9142
Table 7 presents an extended experiment examin- ing the relationship between the accuracy of in- tent recognition and the performance of duplicate question Detection(DQDD), The table indicates that higher Fl score in intent recognition correlates with a higher F1 score in DQD.
A.3 The Effect of Contrastive Learning
We randomly sample a batch matched intents and mismatched intents. We visualize the similarity between intents in two situations: (1) without con- trastive learning(CL),, we encode intents through pre-trained BERT (2) with contrastive learning, we further train the BERT through the contrastive loss function. As shown in Figure 5, after contrastive learning, matched intents (diagonal elements) have a more prominent distinction with mismatched in- tents (off-diagonal elements).
Table employs Regex patterns. interprets titles intents. incorppratess sentence structure features. including Part-of-Speech (POS), sentence position, and keyword identification. The first two columns display the performance of detecting duplication in SODup and CQADup, respectively, while the final column show- cases the Fi-score for intent recognition.
SODup F1
CQADup F1
Intent FI
0.861
0.879
0.706
0.875
0.892
0.748
R+T
0.904
0,916
0.834
R+T+S
0.908
0.921
0.847
The rank of DQD performance across various strategies consistent with the rank of the score for intent recognition. This correlation can be at- tributed to the phenomenon error propagation: inaccuracies in intent recognition can lead to sub- sequent mistakes in intent matching, which in turn influence the final DQD results.
(a) without CL
(b) with
Figure 5: Comparison of intent cosine similarity before or after contrastive learning. The diagonal elements are matched intent pairs, and the non- diagonal elements mismatched intent pairs. The color of the cell indicates the similarity score.


Reviews:
[{"review_id": "eJeBkuolSv", "reviewer": "Reviewer_Q6eK", "summary": "", "questions": "\n1. As intent recognization plays a key role in the whole process, how the different strategies (R, T, R+T) for extracting training data to learn an intent recognizer affect the overall performance of DQD?\n\n2. How can we apply the Intent-DQD method in a new CQA forum?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response 1: \n> As intent recognization plays a key role in the whole process, how the different strategies (R, T, R+T) for extracting training data to learn an intent recognizer affect the overall performance of DQD?\n\nWe have observed that a higher F1 of intent recognition results in a higher F1 of duplicate question detection (DQD) as shown in the table below:\n\n|      |SODup F1|CQADup F1| Intent F1|\n| ---- | ---- | ---- | -------- | \n|   R     |0.861 |0.879 | 0.706    | \n|   T     |0.875 |0.892 | 0.748    | \n|   R+T   |0.904 |0.916 | 0.834    | \n|   R+T+S |0.908 |0.921 | 0.847    | \n\n&emsp; The rank of the overall performance of DQD with different strategies follows the rank of F1 for intent recognition. One possible reason is the error transition. Errors in intent recognition will cause subsequent errors in intent matching, and then they can be reflected in the final DQD.\n\n&nbsp;\n\n### Response 2: \n> How can we apply the Intent-DQD method in a new CQA forum?\n\nWhen applying the Intent-DQD method in a new CQA forum using the training data in the new forum, the main challenge is how to adopt the domain features of the new forum. Although the regex pattern has difficulties in handling unknown/new domain features (including specific text style and specific terms), the following key points in our method can help:\n\n* The question title is a part of the forum content. In intent-DQD, treating titles as intents can address partial domain features. (line 140)\n* The tag of questions usually collects specific items in a forum. To handle specific terms, we build a keyword library based on the tag, and mark each token whether it belongs to a keyword during intent recognition. (line 192)\n* We use POS to emphasize the structure feature. (line 204)\n\n&emsp; For a cold start situation where a new forum is just formed and has almost no data available for training, we train an intent recognizer on data from SODup and test it on data from CQADupStack. The intent recognition F1 is 0.814, which is comparable to the overall 0.847 F1. The overall deuplicate detection F1 is 0.806, compared to SRoBERTa in the same cold-start situation with an F1 of 0.757. It suggests that Intent-DQD is able to generalize to new CQA forums in a cold start situation using training data from old forums. As we build more data and knowledge in the new forum, e.g., regular expression patterns and human-labeled duplication links, we will be able to retrain Intent-DQD using training data from the new forum and achieve better performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. An intent-based and annotation-free method for duplicate question detection in CQA forums is proposed. It detects duplicate questions based on intents they have, and does not require human annotation for preparing training data.\n\n2. Extensive experiments on 15 domains from two real-world datasets demonstrate the effectiveness of Intent-DQD.\n\n3. The paper is clearly written.", "reasons_to_reject": "\n1. It's not clear whether the proposed method can be easily applied in a new CQA forum, as regex patterns may not be always valid across different forums.", "questions_to_authors": "\n1. As intent recognization plays a key role in the whole process, how the different strategies (R, T, R+T) for extracting training data to learn an intent recognizer affect the overall performance of DQD?\n\n2. How can we apply the Intent-DQD method in a new CQA forum?", "paper_topic_and_main_contributions": "The authors propose an intent-based and annotation-free method (Intent-DQD) for duplicate question detection in CQA forums. The Intent-DQD method works in three steps: intent recognition, intent matching, and duplication detection. They leverage the characteristics of CQA forums to automatically extract training data for recognizing and matching intents. Experiments on datasets from two CQA forums demonstrate the effectiveness of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\nTypos:\n\n1. lines 072 and 073, missing parentheses around the citations\n2. in the caption of figure 5, \"before of after\""}, "extra_text": {}}, {"review_id": "51BaA64P4A", "reviewer": "Reviewer_118e", "summary": "", "questions": "Pls see \"Reasons To Reject\"", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "###  Response 1: \n> It is unclear how the intents are extracted from section 3.1. 2.Figure 3 has issues: the output of the intent recognizer is the binary label (b), however, in (a), it should be the intents. It is not clear how the two parts are connected.\n\nSorry for the confusion in Figure 3. The process of how we extract intents and how part (a) and part (b) are connected are as follows :\n * We first use an intent recognizer to predict the binary label (Intent or Not-intent) for each sentence.\n * After collecting all predictions, we filter sentences with Intent labels as intents.\n * These intents will passed to the encoder of the intent matcher. Here, we encode each intent into a real number vector.\n * The intent matcher calculates the similarity of vectors. A higher similarity between intents indicates a higher possibility of being matched.\n\n&emsp; Thanks for your advice on Figure 3. We will add the \"filtering by Intent prediction\" for clarity.\n\n&nbsp;\n\n###  Response 2: \n> The intent recognizer has a transformer-based encoder, which will dominate the features-- i.e. making the three features (tag, position, POS) show less importance. Also, experiments on the effectiveness of the three features are needed.\n\nAdding extra features to a transformer-based encoder can refer to existing research: (1) The work [1]  proves that the feature indicating whether a token belongs to a code or an entity can improve the performance of the NER task in code-mixed text. (2) The embedding of sentence position can be considered as the extension of segment embedding in transformer-based models like BERT. The segment embedding is effective for segment-sensitive tasks such as next-sentence prediction and sentence pair classification. In our study, since we observed that some users prefer to put intents at the start or the end of a post and other usage of tags and POS features, we try to incorporate the extra tag, position, and POS features for best prediction results.\n\n&emsp; **Experiments of the three features.** The individual POS, Keyword and SentencePOS boost F1 from 0.834 (no structure features) to 0.844, 0.843 and 0.839, respectively. Table 4 demonstrates that we increase the intent recognition F1 from 0.834 to 0.847 with all structure features. The overall duplicate question detection F1 increases from 0.916 to 0.921 on CQADup with the three features.\n\n*[1] Tabassum J, Maddela M, Xu W, et al. Code and Named Entity Recognition in StackOverflow[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020: 4913-4926.*\n\n&nbsp;\n\n###  Response 3: \n> The writing also has issues. For example, before line 180, the method does not mention contrastive learning. So it is better to first have an overall method framework introduced, then talk about the details.\n\nThanks for your advice, and we will improve the method section. We will add a concise description of the technique of each submodule, especially describing the role of contrastive learning in intent matching.\n\n&nbsp;\n\n###  Response 4: \n> Inter-link depends on the existing duplicate links provided, so it is not a generic method.\n\nWe would like to clarify that inter-links (existing duplicate links) are used in training only. They are not used in inference. During the training process, existing duplicate links are used as both the labels for the final duplicate question detection and also part of the contrastive loss in the intent matching module. Intent-DQD, as well as all the baseline methods for CQA duplicate detection we compared in the paper, relies on existing duplicate links as training labels.\n\n&emsp; There are sufficient existing duplicate links in CQA forums for training. Currently, multiple CQA forums provide human-voted duplicate links. For instance, the number of duplicate links in the Stack Overflow forum has reached 1,453,083. Votes are of high quality because only experienced users with a reputation >= 3000 can vote for duplicate links, and at least three votes to duplicate can finally create a duplicate link. \n\n&emsp; To further test the situation where a new forum has just formed and has no existing duplicate link for training, we conduct a cold-start experiment: training Intent-DQD on SODup data and testing it on CQADup. The overall F1 score of intent-DQD reaches 0.806, compared to SRoBERTa in the same cold-start situation with an F1 of 0.757. This indicates the generalization of intent-DQD in the situation where it cannot use any existing duplicate link for training in a new forum.\n\n&nbsp;\n\n###  Response 5: \n> Figure 1: Q1 and Q2 should have matched intent, not mismatch\n\nThe involving parts of our paper are:\n * In Figure 1, the highlight means the intent and Q1 and Q2 indeed have matched the intent.\n * In the caption of Figure 1, our description is \"Q1 and Q2 have matched intent to solve a particular java error while having mismatched backgrounds.\"\n\n&emsp; Therefore, we indeed express a consistent meaning that Q1 and Q2 have matched the intent but mismatched background information. We will make the description more clear in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It targets an interesting and useful topic as the CQA forum is not just for human users to find solutions, but also is leveraged to pret-train language models.\n2. The solution was tested effective and outperforms many pre-train models", "reasons_to_reject": "1. It is unclear how the intents are extracted from section 3.1. \n2.Figure 3 has issues: the out put of the intent recognizer is the binary label (b), however, in (a), it should be the intents.  It is not clear how the two parts are connected.\n3. The intent recognizer has a transformer-based encoder, which will dominate the features-- i.e. making the three features (tag, position, POS) show less importance. Also, experiments on the effectiveness of the three features are needed.\n4. The writing also has issues. For example, before line 180, the method does not mention contrastive learning.  So it is better to first have a overall method framework introduced, then talk about the details.\n 5. Inter-link depends on the existing duplicate links provided, so it is not a generic method.", "questions_to_authors": "Pls see \"Reasons To Reject\"", "paper_topic_and_main_contributions": "The paper targets the duplicate question problem in CQA forums.  It particularly focuses on leveraging question intents to check the duplication.  The work aims to have more accurate duplicate detection.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1:  Q1 and Q2 should have matched intent, not mismatch"}, "extra_text": {}}, {"review_id": "zIbq7W3H3d", "reviewer": "Reviewer_xaBD", "summary": "", "questions": "What’s the ration of positive and negative samples in intent matching module? And does it affect the training of the module?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response 1:\n> Requires intent to be explicitly mentioned in the title/body. Doesn't work well where intents are implicit.\n\nOur method handles implicit intents. Compared with an explicit intent in the title or body, such as \"How to solve GPU OOM error in Pytorch?\", an implicit one can be \"I have GPU OOM error in Pytorch\", which implicitly indicates that the user needs a solution to solve the OOM error. As mentioned in section 5.3, we have conducted verification of intent recognition through manually annotated data. Among the 1,916 labeled intents, we further identify 189 intents belonging to implicit intents similar to the above example. We then evaluate the intent recognizer, and the F1 is 0.833. It is comparable to the F1 of 0.847 for all the explicit and implicit intents, showing the intent recognition module is able to detect implicit intent.\n\n&emsp; However, our method does not work well on open conversations with undefined intents. Examples such as \"I love Python and it is an amazing language\" can confuse answerers on what the question is asking for and are commonly considered low-quality questions, which are marked with negative scores and not considered useful in CQA forums. We will revise the limitations section to reflect the above data points and clarifications.\n\n&nbsp;\n\n### Response 2:\n> The paper refers to the arxiv references when the conference references exist such as line 663, 639, etc.\nThank you for pointing this out and we will update the references.\n\n&nbsp;\n\n### Response 3:\n> The paper has typo and grammar mistakes which hinders the readability of the paper such as missing commas (line 185), misformed sentence (line 166), extra hyphens (line 158), combining (line 440), and missing equation number (line 290, 293) etc.\nThank you and we will fix the grammar mistakes and improve readability.\n\n&nbsp;\n\n### Response 4:\n> What's the ratio of positive and negative samples in intent matching module? And does it affect the training of the module?\n\nThe ratio of positive and negative samples is 1:1. The reason behind this is that there are two equally important goals as follows:\n * Goal_A: decreasing the distance among matched intents\n * Goal_B:  increasing the distance among mismatched intents.\n\n&emsp; The positive/negative ratio indeed affects the training of the intent matching module. We conducted training on the ratio of 2:1 and 1:2. We observed that a lower proportion of positive samples or negative samples can cause sub-optimal performance on either Goal_A or Goal_B. Therefore, we set the ratio of matched intents and mismatched intents to 1:1 during contrastive learning. We will incorporate this into the paper's final version to explain the design choices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper contributes towards one of the critical challenges in the field of NLP i.e. detecting and eliminating duplicate content from the CQA forums to improve the content quality.\n\nTackles the challenge of missing labels by utilizing regular expression patterns. The proposed method is annotation-free which helps in saving the cost of labeling.\n\nHandles multi-intent questions by exploiting intent-level relations.\n\nThe proposed model is scalable in real-world scenarios.", "reasons_to_reject": "Requires intent to be explicitly mentioned in the title/body. Doesn’t work well where intents are implicit.\n\nThe paper refers to the arxiv references when the conference references exist such as line 663, 639, etc.\n\nThe paper has typo and grammar mistakes which hinders the readability of the paper such as missing commas (line 185), misformed sentence (line 166), extra hyphens (line 158), combining (line 440), and missing equation number (line 290, 293) etc.", "questions_to_authors": "What’s the ration of positive and negative samples in intent matching module? And does it affect the training of the module?", "paper_topic_and_main_contributions": "The paper focuses on leveraging intent information to detect duplicate question on Community Question Answering (CQA) forums. These CQA forums can serve as a high-quality knowledge bases for instruction tuning for LLMs, which in turn, can be employed in designing systems like ChatGPT etc.\n\nThe proposed model, Intent-DQD, extracts abundant training labels to recognize and match intents without human annotations. The authors then utilize supervised contrastive learning to match semantically similar intent pairs and separate the non-identical ones. They further propose a method to aggregate intent-level relations and obtain question-level relations to detect duplicate questions with multiple intents. They experimented on two datasets covering 15 domains to show the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Paper doesn’t discuss the following significant works in the same direction:\n\n1. Divide and Conquer: Text Semantic Matching with Disentangled Keywords and Intents.\n\n2. Match2: Matching over Matching Model for Similar Question Identification"}}]

Author rebuttal:

Meta-review: All the reviewers agree on the fact that the paper provides a valuable contribution to the research on Community Question Answering (CQA) forums. In this paper, an intent-based and annotation-free method for duplicate question detection in CQA forums is proposed, that detects duplicate questions based on the intents they have, and does not require human annotation for preparing training data. Extensive experiments on several domains from two real-world datasets have been carried out to demonstrate the effectiveness of the proposed method. However, some concerns have been raised by the reviewers concerning the complexity of applying such method to a new CQA forum, and the effectiveness of the intent extraction method in particular for implicit and undefined intents. The writing of the paper should also be improved.
Decision: Accept-Findings