Stance Detection on Social Media with Background Knowledge
Authors: Ang Li
Bin Liang
Jingqian Zhao
Bowen Zhang
Min Yang
Ruifeng Xu
Venue: EMNLP 2023 Main (2023)
Keywords: Stance Detection
Knowledge Augmentation
Background Knowledge
TL;DR: Knowledge-augmented stance detection on social media.

Abstract: Identifying users' stances regarding specific targets/topics is a significant route to learning public opinion from social media platforms. Most existing studies of stance detection strive to learn stance information about specific targets from the context, in order to determine the user's stance on the target. However, in real-world scenarios, we usually have a certain understanding of a target when we express our stance on it. In this paper, we investigate stance detection from a novel perspective, where the background knowledge of the targets is taken into account for better stance detection. To be specific, we categorize background knowledge into two categories: episodic knowledge and discourse knowledge, and propose a novel Knowledge-Augmented Stance Detection (KASD) framework. For episodic knowledge, we devise a heuristic retrieval algorithm based on the topic to retrieve the Wikipedia documents relevant to the sample. Further, we construct a prompt for ChatGPT to filter the Wikipedia documents to derive episodic knowledge. For discourse knowledge, we construct a prompt for ChatGPT to paraphrase the hashtags, references, etc., in the sample, thereby injecting discourse knowledge into the sample. Experimental results on four benchmark datasets demonstrate that our KASD achieves state-of-the-art performance in in-target and zero-shot stance detection.

Stance Detection on Social Media with Background Knowledge
Ang Li Bin Liang" 12,4 Jingqian Zhao 1.2 Bowen Zhang",
Min Yang,, and Ruifeng Xu3.2.3
Harbin Insitute of Technology, Shenzhen, China
2 Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies
Peng Cheng Laboratory, Shenzhen, China
The Chinese University of Hong Kong, Hong Kong, China
Shenzhen Technology University, Shenzhen, China
6 SIAT, Chinese Academy of Sciences, Shenzhen, China (angli, 2350510222 hit.edu. cn, bin. liwng@cuhk,,...m edw.. hk, Shang_bo_wenn@fxxxee oxmail com, min. yang@sineesi ac. cn, xurui suruifeng@hit t. edu. cn
Abstract
Tagett Brazil World Cup
Stance: Against
Identifying users' stances regarding specific targets/topics a significant route to learning public opinion from social media platforms. Most existing studies of stance detection strive learn stance information about specific tar- gets from the context, in order to determine the user's stance on the target. However, in real-world scenarios, we usually have a cer- tain understanding of a target when we express our stance on it. this paper, we investi- gate stance detection from novel perspective, where the background knowledge of the targets taken into account better stance detec- tion. To be specific, categorize background knowledge into two categories: episodic knowl- edge and discourse knowledge, and propose a novel Showledge-Augmentee Stance Detec- tion (KASD) framework. For episodic knowl- edge, devise a heuristic retrieval algorithm based on the topic to retrieve the Wikipedia documents relevant to the sample. Further, we construct prompt for ChatGPT filter the Wikipedia documents to derive episodic knowl- edge. For discourse knowledge, we construct a prompt for ChaiGPT to paraphrase the hash- tags, references, etc., in the sample, thereby injecting discourse knowledge into the sam- ple. Experimental results on four benchmark datasets demonstrate that our KASD achieves state-of-the-ae thrt performance in in-target and zero-shot stance detection.
Text: In the future, the FIFA World Cup should only be held in countries within the top 15 GDP per capita.
Target:
Joe Biden
Stance: Favor
Text:
The whole idea of POTUS ... #VotcBlue #Blue- Wave
Table 1: Two examples to show the episodic knowledge and the discourse knowledge. The first from the VAST dataset and the second is from the P-stance dataset.
or Neutral, etc.) expressed the content towards specific target, topic, or proposition (Somasun- daran and Wiebe, 2010; Augenstein et al., 2016 6; Stefanov et al., 2020). Existing work has achieved promising results different types of stance detec- tion tasks on text, based on conventional machine learning methods (Hasan and Ng. 2013; Moham- mad et al., 2016; Ebrahimi et al., 2016) and deep learning methods (Sun et al., 2018; Zhang et al., 2020; Chen et al., 2021; Liang et al.. 2022a).
However, identifying stance on social media is still challenging because the background knowl- edge of targets is not included in the posts, and the content often comprises implicit information in concise format. For this reason, it is necessary to integrate background knowledge into the stance learning of the target to enhance the ability of the model's 's stance detection by fully understanding the target. To better exploit the background knowledge of the target, we divide it into two types: episodic knowledge and discourse knowledge.
1 Introduction
Stance detection has been essential learning the public options from social media platforms, which aims to automatically identify the author's opin- ionated standpoint or attitude (e.g., Favor, Against,
Episodic knowledge (Ma et al., 2019) refers our understanding of a target, which is the basis for us to express our stance on a target. That is, when we express our stance on a topic, we usu-
Corresponding Author

---

ally have a certain understanding of it. Here, the episodic knowledge generally not explicitly men- tioned in the text. As the red part of the first ex- ample in Table I shows, The author's opposition to hosting the World Cup in Brazil can only be un- derstood by knowing the background knowledge that Brazil's GDP per capita ranks lower than 15th. Previous research (Hanawa et al., 2019; He et al., 2022) has shown that Wikipedia is good source of background knowledge. However, the limitation of input length within the language model makes it impossible to directly input lengthy Wikipedia articles.
rephrase the original text. We apply KASD both line-tuned model and large language model, and conduct experiments on four benchmark stance detection datasets. The results show that, on the fine-tuned model, KASD outperforms the baseline models which have additional designs or back- ground knowledge incorporation. On the large lan- guage model, the results demonstrate that knowl- edge retrieval-augmenntr ChatGPT based on KASD can effectively improve the performance on stance detection. Additionally, we find that with KASD dis- tilling the understanding and relevant background knowledge of the large language model, the fine- tuned model can achieve better results with signifi- cantly fewer parameters.
In addition, in real-world social media platforms, users are accustomed to using nicknames to ex- press certain targets. Therefore, we present dis- course knowledge (Fang et al., 2021) to understand the expressions of acronyms, hashtags, slang, and references in social media texts. The blue part of the second example in Table illustrates that "POTUS" in the text refers to the "President of the United States," and "#VoteBlue, #BlueWave" rep- resents the Democratic Party with implied support for Joe Biden.
The main contributions of our work are summa- rized follows:
1) We investigate stance detection from a novel perspective by exploring background knowledge for an adequate understanding of the targets. The background knowledge is divided into episodic knowledge and discourse knowledge for better learning of stance features.
2) We design the KASD framework, which lever- ages ChatGPT heuristically retrieve episodic knowledge and incorporate discourse knowledge. 3) A series of experiments have demonstrated that our knowledge-augee augmentation framework can effectively improve the accuracy and generalization ability of the fine-tuned model and large language model stance detection
Incorporating background knowledge into stance detection social media poses two major chal- lenges. First, the required knowledge lacks ground truth labels. Second, the knowledge retrieval module necessitates an in-depth comprehension of expressions to retrieve relevant background knowledge rooted semantics and incorporate the knowledge into the original text for making stance judgments. Typically, unsupervised algo- rithms lack these abilities. However, large language models (LLMs), such ChatGPT!, Bard?, and LLaMA (Touvron al., 2023), exhibit exceptional abilities reading and generating text. They have been pre-trained on extensive Wikipedia data and hence possess an immense amount of knowledge. In this paper, we propose Knowledge-Augmented Stance Detection Framework (KASD), leveraging ChatGPT to extract and inject the aforementioned two types of knowledge. We crawl Wikipedia pages related to each target and develop heuristic retrieval algorithm based on topic modeling and an instruct-based filter to obtain episodic knowl- edge. To incorporate discourse knowledge, we employ instruct prompting to decipher acronyms, hashtags, slang, and references in the text and
2 Related Work
Incorporating Episodic Knowledge Current retricval methods typically employ keyword-based filtering (Zhu et al., 2022b) or di- rect use of knowledge graphs (Liu et al., 2021) for knowledge retrieval. However, these retrieval methods necessitate that the required background knowledge overlaps with the text, which is not al- ways the case and could result in poor retrieval effects. Conforti al. (2022) introduced finan- cial signals as background knowledge to improve the stance detection of the WTWT dataset. Zhu et al. (2022a) leveraged unannotated data with a variational nuto-encoding architecture for detecting vaccine attitudes on social media. The knowledge incorporated in these works lacks generality. Incorporating Discourse Knowledge
'https://openaii http:/opena..cc/bl.loooo com/blog/chattpt/ /https://bbrd.ppe com/
'The source code of this paper is available at https:// gi tit. cow/ Ht///HISSZHHLTIAE HLT/KA-Stance Sww---ttaneeeetttiii Detection

---

Tanger: Trump
propapantinic
foom Tramp
the Wikipedia API Each Wikipedia page typically contains between 2,000 20,000 words, which ex- ceeds the capacity of most encoding models. To differentiate among the various episodic knowl- edge, we segment each section of the Wikipedia page into separate documents. This segmentation allows to group relevant information and assign an average length of approximately 400 words per document. Furthermore, this approach is readily extensible, with new targets easily added the knowledge base using the same method.
How is that hoas wocking out for you' Finump KiVilii
W
alternase acwsworthy
Topic Modeling
Pusdeniic Covidit, He..
America,, Greer. Partior...
Large Language Model Prompting
Teeget: Trump
Trung propapenlatic and satement dusing the outhresk.
Mr. Thremy your abour cazsing results
(CLS) Target, Text (SEP) Knowledge
3.2 Retrieval and Filtering
Figure 1: The architecture of our KASD framework.
Existing method (Zhu et al., 2022b) typically begins with word-based retrieval, treating the episodic knowledge as the posterior P(E)(r,, of the In, and uses keywords in the text for retrieval. However, we argue that authors form their opinions on target with a certain stance based underly- ing topics and express these opinions accordingly. Therefore, the episodic knowledge behind these topics should be treated as the prior of the text zn
Zhang et al. (2020) proposed a framework that ex- tracts word-level semantic and emotional under- standing to facilitate knowledge transfer across various targets. Ghosh et al. (2019) splited hash- tags into individual words and employed substi- tute vocabulary to clarify these expressions. Zheng et al. (2022) proposed a prompt-basen contrastive learning approach to stimulate knowledge for low- resource stance detection tasks. Xu et al. (2022)
M
P(Inldn) Σ P(EnEE)) P(Eilda) (1)
and Li and Yuan (2022) utilized data augmentation to enhance the model's comprehension of both text and target. Huang et al. (2023) introduced back- ground knowledge injection modeling hashtags and targets. As experiment results reported this paper, their approaches are suboptimal.
where document dn denotes the combination of words.
3.2.1 Topic Modeling
For each episodic knowledge E,, we assume cor- responds topic Tr related to the sample 2n- To model this prior relationship, we establish topic model for each target in and use the fitted topics to retrieve the relevant episodic knowledge.
3 Methodology
Given X {In, tu IN the labeled dataset, where x denotes the input text and denotes the corresponding target, the goal of stance detection is to identify the stance label y for the correspond- = In represents each word in the given text. As shown in Fig- ure 1, we aim to retrieve multiple episodic knowl- edge { needed for the sample In. and in- ject the required discourse knowledge into Ina re- sulting in D.,- To achieve knowledge augmen- tation, we detect the stance of the sample using {tn, Dn, E] Ek...K)) EK.
We set Ta as the number of topics, which is a hyper- aeper-parameter that affects the effectiveness of the topic model. For the modeling process, employ the Latent Dirichlet Allocation (LDA) algorithm, assuming the word distribution of topic Te denoted by P( TT,), where each word in the vocabulary V assigned a probability B E [o, 1jV of belonging to the topic T., and assuming the topic distribution of a document in denoted by p( (T) represents the probability of each word in the document be- longing to each topic. To estimate these two dis- tributions, employ the online variational Bayes algorithm (Hoffman et al., 2010) implemented in sklearn
ing target f. {w) ...... wi
3.1 Episodic Knowledge Acquisition
For episodic knowledge, followed Zhu al. (2022b); He al. (2022), we conduct our knowl- edge base from Wikipedia. We retrieve the top 10 most relevant Wikipedia pages for each target using
http:///ckkkkt-ller..org/ssstlll
modules/generated/sklenn http:/raaaaeeeeeeeeeeeeeeoesspps tion. Latentirrichheeklllcattio..

---

3.2.2 Heuristic Retrieval
3.2.3 Large Language Model Filtering
After obtaining the word distribution probability I(8'))) for each topic and the distribution prob- ability P( for each document belonging to a topic, we propose a heuristic TF-IDF retrieval algorithm that combines the prior topics Ti with the original sample Zn to match Wikipedia docu- ments in our knowledge base. Initially, establish TF-IDF model for all Wikipedia documents in our knowledge base. Then, we design retrieval method by combining the topic probability distri- bution, arriving at a topic-based retrieval score for each document:
After the heuristic retrieval process. the relevant Wikipedia documents W,, may contain redundant information which brings negative effect on both the injection of background knowledge and the de- termination of the stance. However, dividing the Wikipedia documents and subjecting them to re- fined retrieval process may cause significant loss of contextual information. Therefore, we leverage ChatGPT a filter for episodic knowledge. We build the prompt as:
USER: Sentence: In- Target: 4. Wikipedia Document: W; If [Wikipedia Document] is not related to the given [Sentence] and the given [Target], output None. Otherwise, summarize the knowledge from the document which related the given [Sentence] and the given [Tarret].
Suopie Σ (P(E. Σ (1f-id(()))xx(((T)))
(2)
where L represents the number of words con- tained in the prior topic T. In our experiments. we find that by selecting the top 50 words based on their probability distribution, all topics satisfy Σ' P() > 0.99, indicating that most words relevant to each topic are covered. Besides retriev- ing based on prior topic, we also consider the unique expression patterns of each sample. This retrieval score is based sample-specific caleu- lation:
One sample may have multiple relevant Wikipedia documents, resulting in several prompts. We in- put each prompt into ChatGPT, and if the response is None, we consider the document irrelevant the sample and discard it. If the response is a fil- tered knowledge, we concatenate them to obtain the filtered episodic knowledge &n. Here, Chat- GPT is only allowed to extract knowledge from Wikipedia documents, thus preventing leakage of its stance labels. The ablation experiments con- ducted in Section 5.3 show that filtering can sig- nificantly enhance stance detection compared unfiltered knowledge.
M
Stext Σ (Fiid((w")) x )))
(3)
172
where P(w represents the normalization coeffi- cient that balances with ((3))T) and both have a value 1/M. The final retrieval score is the sum
3.3 Discourse Knowledge Injection
of these two parts without the need for any coef- ficient control, as normalization has already been performed:
To take advantage of the advanced contextual un- derstanding capability and internal knowledge of ChatGPT to inject discourse knowledge, we design prompt that allows ChatGPT to paraphrase the sample Wn, supplementing its acronyms, hashtags, slang, and references, and yielding the knowledge integrated sample D,;
Similarity Smpie
Stext
(4)
The selection of relevant Wikipedia documents Wn, is ultimately controlled by threshold:
In. Please expand the abbre- viations, slang, and hashtags in the [Sentence] into complete phrases and sentences to restate the text.
USER: Sentence:
Wn (Similatryy (tresholld) (5)
From our observations of each dataset, we ulti- mately choose threshold of 0.02, which filters out the majority of irrelevant Wikipedia documents. By utilizing our proposed retrieval method, both the synthesis the sample's prior themes and the differentiated expressions are considered.
Our experiment has demonstrated that injecting discourse knowledge in this way is more effective and capable of boosting the generalization ability of fine-tuned models than merely pre-training or utilizing a substitution dictionary.

---

3,4 Knowledge-Auggmented Stance Detection
Dataset
Target
Favor
Against
Neutral
163
We utilize KASD on both the fine-tuned model and the large language model.
Sem16
170
167
222
Biden
3217
4079
3.4.1 Fine-tuned Model Stance Detection
P-Stance
Sanders
3551
2774
Trump
3663
4290
To demonstrate the effectiveness of KASD in knowl- edge augmentation, we use simple structure for the Fine-tuned Model. We input the sample which injecting discourse knowledge and concating fil- tered episodic knowledge into the BERT model for encoding.
Fauci
492
610
762
Home
615
250
325
Covid19
Mask
190
400
782
School
693
668
346
Table Statistics of SemEval-2016 Task6, P-stance
and COVID-19-Stance datasets.
(6) Then, the representation hn is fed into a softmax classifier, and predicts the distribution of stance.
h = BERT ([CLS)t,, D, (SSPEE,
Train
Valid
Test
Examples
13477
2062
3006
Unique Comments
1845
786
Zero-shöt Topics
2003
383
600
Few-shot
638
114
159
Pn BantmaxxWhh,, + +b)
where Pa. E Rd is the predicted stance proba- bility of the input instance En. dp is the dimen- sionality of stance labels. W E xdm and b d are trainable parameters. The represen- tation is fed into single fully connected layer and softmax layer to predict the stance label ý E { [favor, against, neutral], which is optimized by a cross-entropy loss:
Table Statistics of the VAST dataset.
SemEval-2016 Task6 (Sem16) (Mohammad et al., 2016) consists of tweets containing six pre- defined targets. Following Huang al. (2023) and Zhang et al. (2023), we conduct experiment the three targets: Hillary Clinton (HC), Feminist Movement (FM), Legalization of Abortion (LA), as these targets have larger number of samples. P-stance (Li ct al., 2021) consists of tweets re- lated to three politicians: Joe Biden (Biden), Bernie Sanders (Sanders) and Donald Trump (Trump). As noted in their paper, samples labeled "None" have a low level of annotation consistency. Similar to prior research, we eliminate samples labeled as "None".
N
dp
min L =
Σ
y log of
(8)
==
where yn is the ground-trun truth stance label distribu- tion of instance Zn. In is the estimated distribution, e denotes all trainable parameters of the model, represents the coefficient of I-zrreuularization.
3.4.2 Large Language Model Stance Detection
COVID-I9-Stance (Covid19) (Glandt et al.. 2021) consists of tweets containing four pre- defined targets: Anthony Fauci (Fauci), stay-at- home orders (Home), wear face mask (Mask) and keeping school closed (School).
Although large language models may internally contain the background knowledge to detect the stance of samples, believe that explicit knowl- edge retrieval augmentation would substantially en- hance the large language models' efficacy. There- fore, we apply our KASD framework large lan- guage models as well. To better compare with the baseline, we use the same prompt as Zhang et al. (2023) and applied KASD for knowledge augmenta- tion.
Varied Stance Topics (VAST) (Allaway and McKeown, 2020) is for zero/few-shot stance de- tection and comprises comments from The New York Times "Room for Debate" section range of topics covering broad themes. It has about 6000 targets, far more than the other three datasets.
large
4 Experimental Setup
4.2 Implementation Details
4.1 Datasets
For the fine-tuned model, employ the RoBERTa (Liu et al., 2019) as the encoding module and a fully connected layer with batch normalization and LeakyReL.U as the classifier, namely KASD-BERT. The models are trained using an AdamW W optimizer with batch size 16 for a
We conduct experiments on four benchmark datasets in stance detection including SemEival- 2016 Task6, P-stance, COVID-19-SMN 19-Snnce,,, and Var- ied Stance Topics. The statistics is shown is Table and Table 3.

---

Seml6(%)
Ps)tance(%))
COVID19(%)
HC
FM
LA
Avg
Biden
Sanders
Trump
Avg
Fauci
Home
Mask
School
Avg
Fine-tuned Model
RoBERTa
55.97
68.19
67.60
63.92
84.29
79.56
82,70
82.18
77.44
79.46
80.89
72.84
77.66
BERTweet
62.31
64.20
64.14
63.55
82.90
79.00
84,41
82.10
82.91
80.90
77.98
78.77
80.14
KPT
71.30
63.30*
63.50*
66.03
80.40
77.10
80.20
79.23
84.37
81.34
84.27
76.71
81.67
RoBERTa-Ghost
55.19
62.02
70.41
62.54
83.54
79.35
84.04
82.31
77.29
78.07
82.58
75.75
78.42
BHRTweet-Ghhsh
56.72
64.46
64.80
61.99
82.72
77.65
84,60
81,66
82.05
83.97
80.81
75.89
80.68
KEprompi
77.10
68.30'
76.30
"1190F
84,40
83720"
WS-BERT-Dual
75.26
66.02
70.42
70.57
83,50"
79.00"
85,80"
82.77"
83.60'
85,00'
86,60'
82.20'
84.35'
KASD-BERT BERT
77.60
70.38"
72.29"
73.42"
85.66"
80.39
85.35
83.80
87.49"
87.97"
86.20
83.03
86.17"
Large Language Model
ChatGPT
78.90'
68.70'
61.80'
69.801
82.80
80.801
85.70|
83.101
77.48
72.02
69.58
57.95
69.26
KASD-ChatGPT
80.92
70.37
63.26
71.52
84.59
79.96
85.06
83.20
77.64
72.47
77.24
59.20
71.64
Table 4: In-target stance detection experiment results Sem16, P- Stancee and COVID19 dataset. The results with are retrieved from (Huang et al., 2023), > from (He et al., 2022), from (Zhang et al., 2023). Best scores are in bold. Results with denote the significance tests of our KASD over the baseline models at p-value < 0.05. Since the results based on ChatGPT are the same each time, significance test cannot be conducted.
maximum of 30 epochs with warm-up ratio of 0.2. A learning rate of le-5 and a weight decay of le-3 are utilized. We report averaged scores of 5 runs to obtain statistically stable results. For the Large Language Model, we utilize the gpt-3.5-turbo-030 version of ChatGPT and set the temperature zero, ensuring replicable. We use the same prompt as the baselines and applied our framework for knowl- edge augmentation, namely KASD-ChatGPT.
use their original zero-shot dataset settings. We use standard vrain/valiation/test splits for in-target and zero-shot stance detection across the four datasets.
4.4 Comparison Models
The fine-tuned model baselines include vanilla RoBERTa (Liu et al., 2019), domain pre-trained model: BERTweet (Nguyen et al., 2020), prompt based model: KPT (Shin et al.. 2020), joint contrastive learning framework: JoinCL (Liang et al., 2022b), incorporating discourse knowl- edge method (Ghosh et al., 2019): RoBERTa- Ghosh and BERTweer-Ghosh, incorporating Con- ceptGraph knowledge model: KFprromp (Huang et al., 2023), and incorporating Wikipedia knowl- edge model: TarBK-BERT (Zhu et al., 2022b) and WS-BERT (He et al., 2022). It should be noted that WS-BERT uses the COVID-Twiteer- BERT model, which is large-sized model, for the Covid19 dataset. Thus, the Covid19 dataset, all BERT-based baselines, including KASD-BERT, are compared using the large model. Apart from that, all other baselines and our KASD-BERT are utilized on the base model. For large language models, we compare KASD-ChatGPT with Chat- GPT (Zhang et al., 2023), which utilizes few-shot chain-of- tfft-ught prompt for in-target stance detec- tion and zero-shot prompt for zero-shot stance de- tection. Therefore, we guarantee that comparisons with all baselines are fair.
4.3 Evaluation Metric
Following previous works (He et al., 2022), adopt the macro-average of the F1-score the evaluation metric. P-stance is binary classifi- cation task, where each sample labeled either as 'favor' or 'against'. Thus, we report Faeg = (Fravor + Fagninst)/2. For Sem16, we follow the setup in Mohammad et al. (2016) and report F Fang (Fjavor + Faguinst)/22 For COVID19 and VAST, we follow the setup in Glandt et al. (2021); Allaway and McKeown (2020) and report Faog (Fjavor + Fageinst FNone)/3. /3. In in- target stance detection, we select the one target to divide training, validation and test sets, consistent with other baselines. In zero-shot stance detection, for the SemEval16 dataset, following Huang et al. (2023), we select two targets as training and val- idation sets and the remaining one as test set. For the P-Stance dataset, following Huang et al. (2023); Liang al. (2022b), we select two targets training and validation sets and the remaining one as a test set. (Which the "DT, JB->BS", "DT, BS->JB," and "JB, BS->DT" described in dataset paper (Li et al., 2021)). For the VAST dataset, we
Experimental Results
We conduct experiments on two different meth- ods stance detection: in-target stance detection,

---

Sem 16(%)
P-stance($))
VAST(%)
HC
FM
LA
Avg
Biden
Sanders
Trump
Avg.
Avg
Fine-tuned Model
RoBERTa
43.45
40.38
38.79
40.87
76.29
72.07
67.56
71.97
73.18
BERTweet
44.82
21.97
31.91
32.90
73.13
68.22
67.66
69.67
71.10
JoinCLL
54.80
53.80
49,50
52.70
72.3
RoBERTa-Ghosh
44.78
41.33
29.21
38.44
76.28
70.57
66.81
71.22
73.07
BERTweet-
44.51
36.21
34,43
38.38
73.18
68.39
66.36
69.31
71.90
TarBK--BER
33.10
$3.80"
48.770"
52.53"
75.49"
770.45
65,80
76.38
73.601
WS-BERT-Dual
53.65
47.06
42.61
47.77
77.91
71.63
69.24
72.93
75.30'
KASD-BERT
64.78"
57.13"
51.63"
57.85"
79.04"
75.09*
70.84"
74.99
76.82"
Large Language Model
ChatGPT 79.50
68.40!
58.20!
68.
68.
82.30'
79.401
82.80 !
81.50
62.30'
KASD-ChatGPT
80.32
70.41
62.71
71.15
83.60
79.66
84.31
82.52
67.03
Table 5: Zero-shot stance detection experiment results Seml6, P-Stance and VAST dataset. The results with are retrieved from (Liang et al., 2022b), & from (Zhu et al., 2022h), from (He et al., 2022), i from (Zhang et al., 2023). Best scores are in bold. Results with denote the significance tests of our KASD over the baseline models at p-valuc 0.05. Since the results based on ChatGPT are the same each time, significance test cannot be conducted.
which is trained and tested on the same target, and zero-shot stance detection which performs stance detection on unseen targets based on the known
Shot Stance Detection task.
5.3 Ablation Study
targets.
To examine the impact of episodic knowledge and discourse knowledge, we provide two types of our proposed KASD in the ablation study: (I) "w/o E" denotes without the filtered episodic knowledge.
5.1 In- Target Stance Detection
We perform experiments on Seml6, P-Stance, and COVID-19 for in-target stance detection. The ex- perimental results are presented Table It shows that our proposed KASD framework outperforms all baseline methods in terms of both average results across all datasets and for most of the targets. It indicates that our KASD framework can improve the ability of the fine-tuned model to determine the stance through knowledge augmentation, without modifying the model itself. The results also indi- cate that knowledge strievall-uggmene ChatGPT using KASD can improve its performance in the In- Target Stance Detection task.
(2) "w/o D" denotes without the injected discourse knowledge.
We conduct experiments the P-Stance dataset for In-Target Stance Detection and the VAST dataset for Zero-Shot Stance Detection. The sults are presented Table 6. indicates that for in-target stance detection on the P-Stance dataset, removing either episodic knowledge or discourse knowledge resulted in a significant decrease in per- formance across all targets. This suggests that for the P-Stance dataset, which contains a signif- icant amount of hashtags and slang used in tweet posts, discourse knowledge, which enhances un- derstanding of the samples, and episodic knowl- edge, which helps them judge difficult samples through background information, are both effective. While the VAST dataset consists mostly of stan- dardized online debate texts, which usually exhibit standard conventions, discourse knowledge did not significantly improve the performance. However, since the VAST dataset contains a large number of targets, related episodic knowledge can help the model more effectively understand the targets, thereby enhancing the model's stance detection ability. Furthermore, Case Study provided fur- ther evidence to support the above observations.
5.2 Zero-Shot Stance Detection
We conduct experiments on Seml6, P-Stance, and VAST for zero-shot stance detection. The experi- mental results, as shown Table indicate that our proposed KASD framework can significantly im- prove zero-shot performance in fine-tuned mod- els(the results of p-value on most of the evaluation metrics are less than 0.05). It suggests that our mhowledgo-augmente method is more effective in zero-shot tasks for fine-tuned models which can enhance the model's understanding and generaliza- tion capabilities. The results based on ChatGPT show that our KASD framework can also largely improve the performance of ChatGPT on the Zero-

---

P-stance(S9) %+)
VAST(%)
100
Biden
Sanders Trump
Avg
Avg
126
KASD-BERT
85.66
80.39
85.35
83.80
76.82
w/o
83.41
78.54
85.03
82.33
74.53
w/o D
83.69
79.01
84.19
82.29
76.44
MASD-ChatGPP
84.59
79.96
85.06
83.20
67.03
42-
50
w/o &
82.59
78.10
81.69
80.69
65.22
w/o D
82.87
77.79
83.09
81.25
66.79
Table 6: Experimental results of ablation study in de- tecting in-target stance the P-Stance dataset, and zero-shot stance on the VAST dataset.
[c]
Figure The first-order differences of PPI. across the number of topics T, ranging from to 19 in Seml6, P-Stance, and Covid19 datasets.
Surram Krowkids
Sueeury Keowhidgs
Bantored Keowledgs
Rettocnd Keowlides
F-bardd Kannlagg
Fhand Kansladge Restoved Thual
Restived Fitrrl
The experiment results, shown in Figure 2, indicate that redundancy may negatively impact the stance classification Wikipedia documents are not effectively retrieved and filtered. Our retrieval and filtering structure this paper can effectively obtain the necessary background knowledge for the samples, resulting in significant improvements.
75
70
45
70
(a) Fine- funed Modil:: Rebarta-bsse
P-Stasee
Semilo
VIST
(b)) Lange Moded. CheGGPT
P-Staxse
ViST
5.4 Analysis of Topic Model
During the process of topic modeling, the num- ber of topics Tn acts as hyper- papperrprrametr that affects the effectiveness of episodic knowledge. Perplexity (PPL) is commonly used metric measure the quality of language models. As the number of topies T,, increases, the PPL of the topic model naturally increases. To address this, we use the first-order difference of PPL as an evaluation metric to evaluate the effectiveness of the Topic Model. Based on preliminary understanding of the dataset, we limit T, between 5 and 19. The results are shown in Figure 3. For each target, we select the result with the minimum metric and set it as the value of T₁ Note that the VAST dataset fea- tures small sample size for each target, rendering modeling an effective Topic Model difficult. Thus, the texts are utilized for the retrieval of episodic knowledge in the VAST dataset. From the results, we observe that for Seml6 and P-Stance, each tar- get demonstrates strong topics. In contrast, for the C dataset, there are fewer topics, which is consistent with all targets in the Covid19 dataset that are nearly based on the same topic.
Figure 2: Experimental results of three methods of re- trieving background knowledge detecting in-target stance the P-Stance and Semle 6 datasets, and zero- shot stance on the VAST dataset.
To validate the effectiveness of the heuristic re- trieval algorithm and the filtering method designed in this paper, we conduct comparative experiments three datasets, namely P-Stance, Seml6, and VAST. The experiments conducted based on the fine-tuned models and the large language model, with four groups of comparisons:
Summary Knowledge:
Following the ap- proach proposed by He et al. (2022), only utilize the summary section from the Wikipedia page as knowledge.
Retrieved Knowledge: We use the heuristic retrieval algorithm proposed in this paper to obtain the most similar Wikipedia document as knowledge.
Filtered Knowledge: We use ChatGPT to directly extract episodic knowledge from the knowledge base without retrieval.
5.5 Human Evaluation
We randomly select 500 samples from the Semla 16, P-Stance, Covid19, and VAST datasets and use hu- man evaluation (with three evaluators who are not involved in this work) to measure the quality of the data generated by ChatGPT. Here, To assess the
Retrieved + Filtered: We use the framework proposed this paper to retrieve and filter knowledge.

---

Generating
Filtering Generating redundant discourse knowledge 96.87%
(sstanco(%e)
episodic knowledge
Biden
Sanders
Trump
Avg
content 95.13%
RoRERTa-
76.68
74.67
68.71
73.35
Human Eval
96.00%
BERTweet-large
78.76
78.04
63.01
73.27
CharGPT
82.30
79.40
82.80
81.50
Table 7:
Results of human evaluation Seml6, P-
KASD-RoBERTa-Larg
84.36
79.69
85.25
83.10
Stance, Covid19, and VAST datasets.
Table 8: Experimental results of RoBERTa-large. BFRTweet-large, ChatGPT and KASD-RoBERT alaage detecting zero-shot stance the P-Stance dataset.
quality of the generated episodic knowledge, we evaluate whether the filtered episodic knowledge is relevant the respective sample and whether the filtered redundant content does not contain the required episodic knowledge. For the quality of the generated discourse knowledge, we evaluate the consistency of the generated discourse knowledge with the original content. The evaluators are asked to answer either "yes" or "no" to each of the three questions. Finally, we compute the mean propor- tion of "yes" responses from three evaluators for each question. A higher proportion indicates better data quality. The results are shown in Table 7.
smaller model through knowledge augmentation, the fine-tuned model can outperform the large lan- guage model with much fewer parametcrs(about 500 1000 times fewer).
CharGPT's suboptimal performance can be at- tributed to its limited understanding of the stance detection task. The advantage of the fine-tuned model lies in its ability to better understand the task through task-specific supervised learning. How- ever, the limited amount of training data hinders the development of general comprehension abilities. By utilizing KASD, we distill the understanding ca- pabilities of large models and external Inowledge in the forms of discourse knowledge and episodic knowledge, the performance of fine-tuned models can be effectively improved, thus surpassing large language models.
The results show that ChatGPT is capable of generating bigh-quality background knowledge in the majority of cases (over 95%). This can be attributed to the fact that filtering redundant knowl- edge and generating discourse knowledge can be considered retrieval and generation tasks. Given that ChatGPT has been extensively trained on substantial amount of similar data, consequently led to enhanced generation quality.
Conclusion
In this paper, we propose Bhowledge-Augmenter Stance Detection (KASD) framework, providing heuristic retrieval and filtering of episodic knowl- edge and utilizing contextual information to inject discourse knowledge. We conduct experiments on in-target stance detection and zero-shot stance detection using four benchmark datasets. The ex- perimental results demonstrate significant improve- ment in performance for both fine-tuned models and large language models utilizing KASD.
5.6 Fine-tuned Model vs Large Language Model
this section, we compare the performance be- tween the fine-tumed model and the large language model based on our KASD framework. Concerning in-target stance detection, Table 4 demonstrates that the effect of the RoBERTa-base model af- ter knowledge augmentation is superior to Chat- GPT model, which uses the few-shot chain of thought prompt. In zero-shot setup, Table sug- gests that the RoBERTa-base model, after knowl- edge augmentation, performs better than the Chat- GPT model on the VAST dataset. Additionally, Table 8 presents results for zero-shot stance detec- tion on the P-Stance dataset, which is considered more challenging. It shows that the knowledge- augmented RoBERTa-large outperforms ChatGPT, with significantly fewer parameters the former em- ploys, compared the latter. These results imply that by distilling the large language model's 's under- standing ability and background knowledge into a
Acknowledgments
We thank the anonymous reviewers for their valu- able suggestions to improve the quality of this work. This work was partially supported by the National Natural Science Foundation of China (62006062, 62 66176076), Natural Science Foundation of Guang- Dong 2023A1515012991 Shenzhen Foundational Research Funding JCYJ202103241 15614039 and JCYJ2022081 10200088111111111 10325 Guangdong Provincial
Key Laboratory of Novel Security Intelligence Technologies 2022B 12 20100055

---

Limitations
the 60th Annual Meeting of the Association for Com- putational Linguistics ( (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4074- 4091. Association for Computational Linguistics.
Our framework requires crawling relevant Wikipedia pages in advance to construct knowledge base. We did not consider alternative sources, such as more timely news websites, for our knowledge base. Future research on exploiting this knowledge to explore timely stance detection methods is promising.
Javid Ebrahimi, Dejing Dou, and Daniel Lowd. 2016. Weakly supervised tweet stance classification by re- lational bootstrapping. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1012-1017.
Tianqing Fang, Hongming Zhang, Weiqi Wang, Yangqiu Song. and Bin He. 2021. DISCOS: bridging the gap between discourse knowledge and common- sense knowledge. In wWW 21: The Web Conference 2021. Virtual Event Liubljana, Slovenia, April 19- 23, 2021, pages 2648 2659. ACM/ IW3C2.
Ethies Statement
The datasets used in this paper are sourced from open-access datasets. The VAST dataset provides complete text data in open access. In compli- ance with the privacy agreement of Twitter for aca- demic usage, the Seml6, P-Stance, and COVID19 datasets were accessed using the official Twitter API through the Tweet IDs to fetch complete text data. In these datasets, we touch on stance detec- tion for some sensitive targets (c.g., belief, politics, etc.). Due to the noise of the training data and the imperfection of the models, unreasonable results may appear. We used the textual data obtained from Wikipedia and the ChatGPT service from OpenAl during the research process. We followed their term and policies.
Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019. Stance detection in web and social media: A comparative study. In Experimental IR Meets Multilingualltyy, Multimodality, and Interaction 10th International Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, September 9-12, 2019, Pro- ceedings, volume 11696 of Lecture Notes in Com- puter Science pages 75-87. Springer.
Kyle Glandt, Sarthak Khanal, Yingjie Li, Doina Caragea, and Cornelia Caragea. 2021, Stance detec- tion COVID-19 19 tweets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, ACL/JCNLP 2021, (Volume 1: Long Pupers). Virtual Event, Au- gust 1-6, 2021, pages 1,596-1611, 1. Association for Computational Linguistics.
References
Emily Allaway and Kathleen R. McKeown. 2020. Zero- shot stance detection: A dataset and model using gen- cralized topic representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020. pages 8913-8933 Association for Computational Linguistics.
Kazuaki Hanawa, Akira Sasaki, Naoaki Okazaki, and Kentaro Inui. 2019. Stance detection attending ex- ternal knowledge from wikipedia. J. Inf. Process., 27:499-506.
Kazi Saidul Hasan and Vincent Ng. 2013. Stance clas- sification of idcological debates: Data, models. fea- tures, and constraints. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1348-1356.
Isabelle Augenstein, Tim Rocktäschel, Andreas Vla- chos, and Kalina Bontcheva. 2016. Stance detection with bidircctional conditional encoding. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876-885, Austin, Texas. Association for Computational Lin- guistics.
Zihao He, Negar Mokhberian, and Kristina Lerman. 2022. Infusing knowledge from Wikipedia to en- hance stance detection. In Proceedings of the 12th Workshop Computational Approuches to Subjectiv- ity, Sentiment & Social Media Analysis, pages 71 71-77, Dublin, Ireland. Association for Computational Lin- guistics.
Pengyuan Chen, Kai Ye, and Xiaohui Cui. 2021. In- tegrating n-gram features into pre-trained model: a novel ensemble model for multi-taree tarlet stance detec- tion. In International Conference on Artificial Neural Networks, pages 269-279, Springer.
Matthew D. Hoffman, David M. Blei, and Francis R. Bach. 2010, Online learning for latent dirichlet allo- cation. In Advances in Neural Information Process- ing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pages 856-866- 864, Curran Associates, Inc.
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2022. Incorporating stock market sig- nals for twitter stance detection. In Proceedings of

---

Hu Huang, Bowen Zhang, Yangyang Li. Baoquan Zhang, Yuxi Sun, Chuyao Luo, and Cheng Peng. 2023. Knowledge-enhanced prompt-tuning for stance detection. ACM Transactions on Asian and Low-Resource Language Information Processing.
System Demonstrations, EMNLP 2020- Demos, Online, November 16-20, 2020, pages 9-14. Association for Computational Linguisties.
Processing:
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with au- tomatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 4222-4233 Association for Computational Linguistics.
Yang Li and Jiawei Yuan. 2022. Generative data aug- mentation with contrastive learning for zero-shot stance detection. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 6985-6995. Association for Computational Linguistics.
Swapna Somasundaran and Janyce Wiebe. 2010. Rec- ognizing stances ideological on-line debates. Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Genera- tion of Emotion in lext, pages 116-124, Los Angeles, CA. Association for Computational Linguistics.
Yingjie Li, Tiberiu Sosea, Aditya Sawant, Ajith Jayara-
man Nair, Diana Inkpen, and Cornelia Caragea. 2021.
P-stance: A large dataset for stance detection in po- litical domain. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, On- line Event, August 1-6, 2021, volume ACL/ICNLP 2021 of Findings of ACL, pages 2355-2365, Associ- ation for Computational Linguistics.
Peter Stefamov, Kareem Darwish, Atanas Atanasov, and Preslav Nakov. 2020. Predicting the topical stance and political leaning of media using tweets. In Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 527-537, Online. Association for Computational Linguistics.
Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang. and Ruifeng Xu. 2022a. Zero-shot stance detection via contrastive learning. In Proceedings of the ACM Web Conference 2022, pages 2738-2747.
Qingying Sun, Zhongging Wang. Qiaoming Zhu, and Guodong Zhou. 2018. Stance detection with hierar- chical attention network. In Proceedings of the 27th International Conference Computational Linguis- tics, pages 2399-2409, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Bin Liang, Qinglin Zhu, Xiang Li, Min Yang, Lin Gui, Yulan He, and Ruifeng Xu. 2022b. Jointel: A joint contrastive learning framework for zero-shot stance detection. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ire- land, May 22-27, 2022, pages 81-91. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.
Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021. Enhancing zero-shot and few-shot stance de- tection with commonsense knowledge graph. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 3152-3157, On- line. Association for Computational Linguistics.
Hanzi Xu, Slobodan Vucetic, and Wenpeng Yin. 2022. Openstance: Real-world zero-shot stance detection. CoRR, abs/2210.14299
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1662..
Bowen Zhang, Xianghua Fu, Daijun Ding, Hu Huang, Yangyang Li, and Liwen Jing. 2023. Investigating -thhugt with chatgpt for stance detection on social media. CoRR, Abs/2304.03088
Yunpu Ma. Volker Tresp, and Erik A. Daxberger. 2019, Embedding models for episodic knowledge graphs. J. Web Semant., 59.
Bowen Zhang, Min Yang, Xutao Li, Yumming Ye, Xi- aofei Xu, and Kuai Dai. 2020. Enhancing cross- target stance detection with transferable semantic- emotion knowledge. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 3188-3197, Online, Association for Computational Linguistics.
Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiao-Dan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval NAACL-NAA 2016.
San Diego, CA, USA, June 16-17, 2016, pages 31-41. The Association for Computer Linguistics.
Kai Zheng, Qingfeng Sun, Yaming Yang, and Fei Xu. 2022. Knowledge stimulated contrastive prompting for low low-resource stance detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1168- 1178, Association for Com- putational Linguistics.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. Bertweet: A pre-trained language model for english tweets. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language

---

Lixing Zhu, Zheng Fang, Gabriele Pergola, Robert Proc- ter, and Yulan He. 2022a. Disentangled learning of stance and aspect topics for vaccine attitude detec- tion social media. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL. 2022, Seattle, WA, United States, July 10-15, 2022, pages 1566-1580. Association for Computational Linguistics.
Qinglin Zhu, Bin Liang, Jingyi Sun, Jiachen Du, Lanjun Zhou, and Ruifeng Xu. 2022b. Enhancing zero-shot stance detection via targeted background knowledge In SIGIR '22 The 45th International ACM SIGIR Conference Research and Development in Infor- mation Retrieval, Madrid, Spain, July 15, 2022, pages 2070-2077 ACM.
A Case Study
We conduct a case study on Seml6, P-Stance, COVID19, and VAST datasets, and compared the results and knowledge with WS-BERT-Dual. The results are shown in Table 9, 10 and 11. We can observe that the background knowledge obtained based ChatGPT has no label leakage and can help stance detection more effectively than other methods. We found that episodic knowledge can help the model solve difficult samples, while dis- course knowledge can improve the generalization ability the model and smooth out the differences between languages and expressions. This corre- sponds to the results of the ablation study.

---

Text (Target on: Climate Change a Real Concern)
Stance
Sea Level Rise above meters what does that mean? It favor means 20 ft above current heights.
Original Sentence
Contemporary climate change includes both global warm- ing and its impacts Earth's weather patterns. There have been previous periods of climate change, but the current changes are distinctly more rapid and not due natural causes. Instead, they are caused by the emission of green- house gases, mostly carbon dioxide (CO2) and methane... Sea level rise above 6 meters means that it will be 20 feet above current heights.
WS-BERT-Dual knowledge
KASD discourse knowledge
humans face risks due to sea level rise, sea surface warming, and increased frequency and intensity of extreme weather events because of climate change.
favor
KASD episodic knowledge
Text (Target on: Hillary Clinton)
Stance
@thehill Women deserve a better candidate for the HIGH HONOR if first woman President: We ALL do!
Original Sentence
against
WWhyl''mNoVootmnnee MNoooootingFooHHlllla
Hillary Diane Clinton (née Rodham; born October 26, 1947) is an American politician, diplomat, lawyer, writer, and pub- lic speaker who served as the 67th United States secretary of state from 2009 to 2013...
WS-BERT-Dual knowledge
favor
The Hill tweeted that women deserve a better candidate for the high honor of being the first woman President, and that everyone deserves better candidate. This is the reason why the person voting for Hillary Clinton. None
KASD discourse knowledge
against
KASD episodic knowledge
Text (Target on: Legalization of Abortion)
Stance
Antis don't It care because the only people the laws are hurting are females and they don't care about things that don't effect men.
Original Sentence
favor
Abortion-rights movements, also referred to as pro-choice movements, advocate for legal access to induced abortion services including elective abortion. It is the argument against the anti-abortion movement. The Abortion rights movement seeks wish terminate their pregnancy at any point...
WS-BERT-Dual knowledge
favor
represent and support women who
People who oppose the legalization of abortion do not care because they only believe that females are negatively af- fected by these laws and they ignore issues that do not impact men.
KASD discourse knowledge
Margarei Sanger wrote: "No woman can call herself free" until she choose consciously whether she will will not be mother." From this perspective the right to abortion can be construed to be necessary in order for women achieve equality with men whose freedom not nearly so restricted by having children.
against
KASD episodic knowledge
Table Three examples from the Sem16 dataset.

---

Text (Target Donald Trump)
Stance
#Trump planning to divert additional $7.2 billion Pen- tagon funds for border wall @ TeamPelosi @RepJerry- Nadler @ RepSwalwelll @ @SenSchumer
Original Sentence
against
Donald John Trump (born June 14, 1946) is an American politician, media personality, and businessman who served the 45th president of the United States from 2017 to 2021...
WS-BERT-Dual knowledge
favor
President Trump reportedly planning to use an additional $7.2 billion in funds originally allocated for the Pentagon's budget and put it towards constructing a wall at the southern border.
WS-BERT-Dual knowledge
In 2018, Trump refused to extend government funding un- against
less Congress allocated $5.6 billion in funds for the border wall, resulting in the federal government partially shutting down for 35 days from December 2018 January 2019, the longest U.S. government shutdown history.
KASD episodic knowledge
Text (Target on: Joe Biden)
Stance
Now OAN Rudy is "interviewing" corrupt #Ukraine politicians who were *sworn in (in Ukraine) about Joc and Hunter #Biden
Original Sentence
against
"Joseph Robinette Biden Jr. (born November 20, 1942) is an American politician who is the 46th and current president of the United States. A member of the Democratic Party, he served as the 47th vice president from 2009 to 2017 under Barack Obama and represented Delaware in the United States Senate from 1973 to 2009...
WS-BERT-Dual knowledge
favor
Currently on One America News Network (OAN), Rudy Giuliani is conducting an interview with allegedly corrupt Ukrainian politicians who were inaugurated Ukraine and are being questioned about the involvement of Joe and Hunter Biden.
KASD discourse knowledge
Since the early months of 2019, Biden and his father have been the subjects of unevidenced claims of corrupt activities in Biden-Ukraine conspiracy theory pushed by then-U.S. President Donald Trump and his allies, concerning Hunter Biden's 's business dealings in Ukraine and Joe Biden's anti- corruption efforts there on behalf of the United States during the time he was vice president.
against
KASD episodic knowledge
Table Two examples from the P-Stance dataset.

---

Text (Target on: Gun Allowed College)
Stance
A friend of mines father lost his leg fighting in Germany [Battle of the Bulge]. He never owned any firearms ever after the war.....andd scarcely spoke it. Having to kill others with guns changes your life forever.... That's what war does. Do we really want to make universities feel like a war zone..... If you think having classroom of armed students is going to make learning better... Good Luck.
Original Sentence
against
A gun is ranged weapon designed to shooting tube (gun barrel) to launch typically solid projectiles, but can also project pressurized liquid...
WS-BERT-Dual knowledge
none
friend's father lost his leg in the Battle of the Bulge during the war in Germany. He was deeply affected by having to kill others and never owned any firearms after the war. War changes lives forever, and we should not want universities to resemble war zones. The idea of arming students in classroom is unlikely improve education, it is good solution.
KASD discourse knowledge
against
Many do believe that permitting firearms in classroom would lead to disruption in the lcarning processes of stu- dents but also diminish the overall safety of students.
KASD episodic knowledge
Text (Target on: Olympics)
Stance
Holding the games in Brazil pure insanity. Zika and dengue fever are out control. The Olympic committee expects swimmers to compete in an open sewer. Partici- pants and fans will be risking their health, and the health of their families and even their future children, to attend these games, and also risk the health of their home countries upon return. All that NBC & the olympic committee make big bucks. It's not worth it.
Original Sentence
against
The modern Olympic Games or Olympies (French: Jeux olympiques) are the leading international sporting events featuring summer and winter sports competitions in which thousands of athletes from around the world participate in a variety of competitions...
WS-BERT-Dual knowledge
none
The Hill tweeted that women deserve a better candidate for the high honor of being the first woman President, and that everyone deserves a better candidate. This the reason why the person is not voting for Hillary Clinton.
KASD discourse knowledge
against
Some controversies during the Rio Olympics included the Zika virus epidemic and significant pollution in Guanabara Bay
KASD episodic knowledge
Table 11: Two examples from the VAST dataset.


Reviews:
[{"review_id": "QFMBTJpKMI", "reviewer": "Reviewer_ZhbM", "summary": "", "questions": "QA: I imagine there might be error propagation when using ChatGPT to filter redundant knowledge and generate discourse knowledge. I am just curious whether the authors have done any analysis to measure the quality of data generated by ChatGPT?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the time and effort you've taken to review our work. \n\n>**Reject #1:**\\\n>How does KASD compare when directly using ChatGPT to extract episodic knowledge?\n\n**Response of Reject #1:**\\\nThank you for your valuable and positive comments. In the preliminary experiment, we conducted a comparison of directly using ChatGPT to extract episodic knowledge. We found that our KASD performs better than directly using ChatGPT to extract episodic knowledge for both in-target stance detection and zero-shot stance detection. The results are shown in Table 1 and Table 2. We will add the experiments and analysis to the revision.\n\n|           | P-Stance | Sem16 | VAST  |\n|:--------- |:--------:|:-----:|:-----:|\n| KASD-BERT | 83.80    | 73.42 | 76.92 |\n| DUC-BERT  | 77.41    | 66.26 | 72.94 |\n\n*Table 1: The results of extracting episodic knowledge through our KASD and extracting episodic knowledge directly using ChatGPT (short for DUC) in detecting in-target stance on the P-Stance and Sem16 datasets and detecting zero-shot stance on the VAST dataset based on the BERT model*\n\n|              | P-Stance | Sem16 | VAST  |\n|:------------ |:--------:|:-----:|:-----:|\n| KASD-ChatGPT | 83.20    | 71.52 | 67.03 |\n| DUC-ChatGPT  | 79.58    | 65.93 | 59.37 |\n\n*Table 2: The results of extracting episodic knowledge through our KASD and extracting episodic knowledge directly using ChatGPT (short for DUC) in detecting in-target stance on the P-Stance and Sem16 datasets and detecting zero-shot stance on the VAST dataset based on ChatGPT.*\n\n>**Reject #2:**\\\n>For SemEval16, what is the performance for the remaining three targets, and how was the dataset split?\n\n**Response of Reject #2:**\n1) We apologize for not including this clarification in the paper due to page limitations. Following the baselines of KEprompt [1] and ChatGPT [2], we selected HC, FM, and LA as these targets have a larger number of samples. We also conducted experiments to evaluate KASD's performance on the remaining three targets on in-target stance detection and zero-shot stance detection. Because the Sem16 dataset [3] only provided zero-shot stance detection for the target \"Donald Trump\", it was only experimented with for zero-shot stance detection. The results shown in Table 3 and Table 4 demonstrate that our KASD also achieves state-of-the-art performance.\n\n|              |  A    |  CC   |\n|:------------ |:-----:|:-----:|\n| RoBERTa      | 65.40 | 43.08 |\n| BERTweet     | 68.12 | 41.30 |\n| WS-BERT-Dual | 71.57 | 57.31 |\n| KASD-BERT    | **72.32** | **61.47** |\n| ChatGPT      | 61.18 | 60.37  |\n| KASD-ChatGPT | **61.92** | **62.72** |\n\n*Table 3: The results of in-target stance detection on target \"Atheism\" (A) and \"Climate Change is a Real Concern\" (CC). Best scores are in bold.*\n\n|              |  A      |  CC     | DT      |\n|:------------ |:-------:|:-------:|:-------:|\n| RoBERTa      | 26.80   | 18.70   | 32.12   |\n| BERTweet     | 30.49   | 12.48   | 26.88   |\n| TarBK-BERT   | **56.20**   | 39.50   | 50.80   |\n| KASD-BERT    | 55.97   | **40.11**   | **54.74**   |\n| ChatGPT      | 60.94   | 53.03   | 62.30   |\n| KASD-ChatGPT | **63.95**  | **55.83**   | **64.23**   |\n\n*Table 4: The results of zero-shot stance detection on target \"Atheism\" (A), \"Climate Change is a Real Concern\" (CC) and Donald Trump (DT). Best scores are in bold.*\n\n\n2) The SemEval16 dataset was split by its publishers [3] into training and test sets. For in-target stance detection, we divided the original training set into training and validation sets with an 8:2 ratio. For zero-shot stance detection, following [4], we selected 5 targets and divided them into training and validation sets with an 8:2 ratio, while the remaining targets were used as the test set. In the camera-ready version, we will include an introduction to dataset partitioning.\n\n[1]: [Knowledge-enhanced Prompt-tuning for Stance Detection](https://dl.acm.org/doi/abs/10.1145/3588767) (Huang et al., TALLIP 2023)\\\n[2]: [Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media](https://arxiv.org/abs/2304.03087) (Zhang et al.)\\\n[3]: [SemEval-2016 Task 6: Detecting Stance in Tweets](https://aclanthology.org/S16-1003) (Mohammad et al., SemEval 2016)\\\n[4]: [JointCL: A Joint Contrastive Learning Framework for Zero-Shot Stance Detection](https://aclanthology.org/2022.acl-long.7) (Liang et al., ACL 2022)\n\n>**Reject #3:**\\\n>The authors should provide more detail on how SemEval and PSTANCE are applied in zero-shot stance detection.\n\n**Response of Reject#3:**\\\nWe apologize for the lack of clarity on zero-shot stance detection due to page limitations. As described in lines 442-444, zero-shot stance detection performs stance detection on unseen targets based on known targets. For the SemEval16 dataset, following [1], we select five targets as training and validation sets and the remaining one as a test set. For the P-Stance dataset, following [1], we select two targets as training and validation sets and the remaining one as a test set. (Which is the \"DT, JB->BS\", \"DT, BS->JB,\" and \"JB, BS->DT\" described in their paper.) This experimental setup is commonly employed in other zero-shot stance detection research.\nIn the final version, we will include a detailed introduction to zero-shot stance detection.\n\n[1]: [JointCL: A Joint Contrastive Learning Framework for Zero-Shot Stance Detection](https://aclanthology.org/2022.acl-long.7) (Liang et al., ACL 2022)\n\n>**Question #A:**\\\n>I imagine there might be error propagation when using ChatGPT to filter redundant knowledge and generate discourse knowledge. I am just curious whether the authors have done any analysis to measure the quality of data generated by ChatGPT?\n\n**Response of Questions#A:**\\\nThank you for your valuable comments about conducting an analysis to measure the quality of data generated by ChatGPT and for your deep insight into our work. We do agree that there might be error propagation when using ChatGPT to filter redundant knowledge and generate discourse knowledge. Therefore, in our preliminary experiment, we randomly selected a total of 500 samples from the Sem16, P-Stance, Covid19, and VAST datasets and incorporated human evaluation (with three evaluators who are not involved in this paper) to measure the quality of the data generated by ChatGPT. Here, To assess the quality of the generated episodic knowledge, we evaluated whether the filtered episodic knowledge is relevant to the respective sample and whether the filtered redundant content does not contain the required episodic knowledge. For the quality of the generated discourse knowledge, we evaluated the consistency of the generated discourse knowledge with the original content. The evaluators were asked to answer either \"yes\" or \"no\" to each of the three questions. Finally, we computed the mean proportion of \"yes\" responses from three evaluators for each question. A higher proportion indicates better data quality. The results are shown in Table 5.\n\n|             | Generating episodic knowledge| Filtering redundant content| Generating discourse knowledge |\n| :---        |    :----:                    |   :----:                   |   :----:                       |\n| Human Eval  | 96.00%                       | 95.13%                     |     96.87%                     |\n\n*Table 5: Results of human evaluation.*\n\nThe results demonstrate that ChatGPT is capable of generating high-quality background knowledge in the majority of cases (with an average of over 95%). This can be attributed to the fact that filtering redundant knowledge and generating discourse knowledge can be considered retrieval and generation tasks. Given that ChatGPT has been extensively trained on a substantial amount of similar data, it is highly adept at performing these tasks, consequently leading to enhanced generation quality.\nThanks again for your valuable comments; we will add the knowledge quality analysis to the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses stance detection from a novel angle by introducing background knowledge to aid in stance detection. The proposed approach has shown improvement over several baselines for both in-target stance detection and zero-shot stance detection. The work appears technically sound, and the contributions are clear.", "reasons_to_reject": "There are low risks associated with accepting this work, possible reasons for rejection are minor, mostly related to the clarification of some questions that might make the work more convincing. \nFor instance, how does KASD compare when directly using ChatGPT to extract episodic knowledge? \nFor SemEval16, what is the performance for the remaining three targets, and how was the dataset split? \nMoreover, the authors should provide more detail on how SemEval and PSTANCE are applied in zero-shot stance detection.", "questions_to_authors": "QA: I imagine there might be error propagation when using ChatGPT to filter redundant knowledge and generate discourse knowledge. I am just curious whether the authors have done any analysis to measure the quality of data generated by ChatGPT?", "paper_topic_and_main_contributions": "This paper explores background knowledge to assist in the stance detection task, which refers to information that is not explicitly expressed but is often informative for identifying the correct stance label. In particular, background knowledge includes episodic knowledge (prior knowledge that is not expressed in the text) and discourse knowledge (certain special expressions).\n\nThe main contribution of this work is that it addresses stance detection from an effective and new angle: introducing background knowledge.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The caption of Table 7 should be: “zero-shot stance” instead of \"in-target stance\""}, "extra_text": {}}, {"review_id": "n8gSD65cVz", "reviewer": "Reviewer_sQss", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the time and effort you've taken to review our work. \n\n>**Reject#1:**\\\n>It seems like in Tables 4 and 5, some of the results are retrieved from other papers. Do the other papers have the same experiment setup as yours? Is this comparison fair and reliable? Please elaborate more on this in the paper.\n\n**Response of Reject#1:**\\\nThank you for your insightful comments. Yes, we would like to clarify that our comparisons are fair and reliable. For the splitting of datasets and evaluation metrics, as mentioned in Section 4.3, we used the settings from the original dataset paper [1], [2], [3], and [4], which are also adopted by other baselines. The introduction to the model and training settings is in Section 4.4. For the configuration of fine-tuned models, we adopted the settings from [5] and [6], using the same model and training settings as them. As for the configuration of large language models, we followed [7], using the same model and prompts as them. We will provide detailed explanations of this aspect in the final version.\n\n[1]: [SemEval-2016 Task 6: Detecting Stance in Tweets](https://aclanthology.org/S16-1003) (Mohammad et al., SemEval 2016)\\\n[2]: [P-Stance: A Large Dataset for Stance Detection in Political Domain](https://aclanthology.org/2021.findings-acl.208) (Li et al., ACL Findings 2021)\\\n[3]: [Stance Detection in COVID-19 Tweets](https://aclanthology.org/2021.acl-long.127) (Glandt et al., ACL-IJCNLP 2021)\\\n[4]: [Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations](https://aclanthology.org/2020.emnlp-main.717) (Allaway & McKeown, EMNLP 2020)\\\n[5]: [Knowledge-enhanced Prompt-tuning for Stance Detection](https://dl.acm.org/doi/abs/10.1145/3588767) (Huang et al., TALLIP 2023)\\\n[6]: [Infusing Knowledge from Wikipedia to Enhance Stance Detection](https://aclanthology.org/2022.wassa-1.7) (He et al., WASSA 2022)\\\n[7]: [Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media](https://arxiv.org/abs/2304.03087) (Zhang et al.)\n\n>**Reject#2:**\\\n>zero-shot learning is mentioned quite a few times in the text but is formally introduced in section 5 for the first time. I suggest introducing this definition earlier in the paper for better readability.\n\n**Response of Reject#2:**\\\nThank you for your careful reading and valuable comment. We will provide a comprehensive introduction to zero-shot stance detection within the Introduction section.\n\n>**Reject#3:**\\\n>typo line 353: argumentation.\n\n**Response of Reject#3:**\\\nThank you for your comments. We have corrected 'argumentation' to 'augmentation' in the revision. We have thoroughly checked for spelling errors and will avoid them in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The topic of the paper is very interesting, hence worthy of investigation. It also aligns perfectly with the conference's theme.\n- The paper is very well motivated and well written. The authors nicely motivate the work and position the paper in the literature.\n- The experiment section is well-designed, the datasets and baseline methods are all appropriately chosen and well explained. The evaluation metric is nicely chosen and the experiment setup is detailed which make it reproducible.\n- The numerical results shows significant improvement over the baseline methods which shows the effectiveness of the proposed framework.\n- The authors also zoom in on various components of their framework to investigate their effectiveness which make their analysis more reliable and comprehensive.", "reasons_to_reject": "Major: \n- It seems like in Tables 4 and 5, some of the results are retrieved from other papers. Do the other papers have the same experiment setup as yours? Is this comparison fair and reliable? Please elaborate more on this in the paper.\n\nminor:\n- zero-shot learning is mentioned quite a few times in the text but is formally introduced in section 5 for the first time. I suggest introducing this definition earlier in the paper for better readability.\n- typo line 353: argumentation", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new approach for identifying users' stances on specific topics from social media. While traditional stance detection relies on context alone, this method acknowledges that users often possess background knowledge about the subjects they discuss. The proposed technique, called Knowledge-Augmented Stance Detection (KASD) framework, leverages two types of background knowledge: episodic knowledge and discourse knowledge.\nEpisodic knowledge is obtained from relevant Wikipedia documents using a heuristic retrieval algorithm. ChatGPT is then prompted to filter and extract relevant information from these documents. For discourse knowledge, prompts are utilized to paraphrase hashtags, references, and other elements in the text, thereby integrating additional contextual knowledge.\n\nThe experimental results, based on four benchmark datasets, demonstrate that KASD outperforms existing methods in stance detection for both in-target and zero-shot scenarios.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yiRd6qNYFO", "reviewer": "Reviewer_d5bz", "summary": "", "questions": "（1）Why you use the topic model to retrieval the target page, not use the BERT embedding to look up the pages?\n(2) Why the fine-tuned model perform better than the ChatGPT? How many parameters fine-tuned,10% ,30%,...,can exceed the ChatGPT?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the time and effort you've taken to review our work. \n\n>**Reject#1:**\\\n>The work only use a LLM, ChatGPT, you can also test it in LLaMA.\n\n**Response of Reject#1:**\\\nThanks a lot for your valuable comments. Because the previous LLM-based work only used ChatGPT in this task, we primarily conducted experiments on ChatGPT. Based on your valuable suggestions, we also experimented with LLaMA-2-70b-chat for in-target stance detection and zero-shot stance detection. The comparison results are shown in Tables 1 and 2.\n\n|              | HC    | FM    | LA    | Avg   | Biden | Sanders | Trump | Avg   | Fauci | Home  | Mask  | School | Avg    |\n|:------------ |:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:-----:|:-----:|:-----:|:-----:|:-----:|:------:|:------:|\n| ChatGPT      |   78.90   |   68.70   |   61.80   |   69.80   |   82.40   | **80.80**   | **85.40** |   82.87   |   77.48   |   72.02   |   69.58   |   57.95    |   69.26    |\n| KASD-ChatGPT | **80.92** | **70.37** | **63.26** | **71.52** | **84.59** |   79.96     |   85.06   | **83.20** | **77.64** | **72.47** | **77.24** | **59.20**  | **71.64**  |\n| LLaMA2       |   72.91   |   65.04   |   49.71   |   62.55   |   78.57   | **71.64**   |   65.57   |   71.87   |   67.69   |   59.07   |   64.05   |   45.29    |   59.03    |\n| KASD-LLaMA2  | **77.89** | **67.29** | **52.00** | **65.73** | **79.59** |   71.32     | **67.89** | **72.93** | **71.38** | **67.33** | **67.10** | **52.89**  | **64.68**  |\n\n*Table 1: In-target stance detection experiment results on Sem16, P-Stance, and COVID-19 datasets based on ChatGPT and LLaMA2. Best scores are in bold.*\n\n|              | HC    | FM    | LA    | Avg   | Biden | Sanders | Trump | Avg   | VAST   |\n|:------------ |:-----:|:-----:|:-----:|:-----:|:-----:|:-------:|:-----:|:-----:|:------:|\n| ChatGPT      |   79.50   |   68.40   |   58.20   |   68.70   |   82.30   |   79.40     |   82.80   |   81.50   |   62.30    |\n| KASD-ChatGPT | **80.32** | **70.41** | **62.71** | **71.15** | **83.60** | **79.66**   | **84.31** | **82.52** | **67.03**  |\n| LLaMA2       |   74.24   | **67.19** |   55.92   |   65.78   | **76.24** |   73.67     |   64.47   |   71.46   |   39.85    |\n| KASD-LLaMA2  | **77.70** |   65.57   | **57.07** | **66.78** |   75.28   | **74.09**   | **69.27** | **72.88** | **43.42**  |\n\n*Table 2: Zero-shot stance detection experiment results on Sem16, P-Stance, and VAST datasets based on ChatGPT and LLaMA2. Best scores are in bold.*\n\nThe results indicate that by using our KASD to enhance background knowledge, LLaMA2 could also achieve significant performance improvements in both in-target stance detection and zero-shot stance detection tasks. We will add the experiments and analysis to the final version.\n\n>**Reject#2:**\\\n>In line 96, large language models(LLMs), such as ChatGPT, Bard, and LLaMA. It lacks the references about the three models\n\n**Response of Reject#2:**\\\nThank you for your valuable suggestions, we have added the corresponding references in the revision.\n\n>**Question#1:**\\\n>Why you use the topic model to retrieval the target page, not use the BERT embedding to look up the pages?\n\n**Response of Questions#1:**\\\nThank you for your question. First, for the stance detection task, target-related information is crucial, and the text is distributed around several topics related to multiple targets. In this context, it is natural and effective to use a topic model to retrieve the knowledge pages. Secondly, we need knowledge pages to provide the model with additional information for stance judgment. However, the prerequisite for using BERT embedding to retrieve knowledge pages is that the knowledge and the original samples have the same semantics, which is not always the case. For example, as shown in Table 8 of Appendix A Case Study, the first sample is \"Sea Level Rise above 6 meters—what does that mean? It means 20 feet above current heights, while the background knowledge needed is that human activities are causing rising sea levels and triggering extreme weather events and climate disasters. They are not strongly semantically related, but they describe the same topic. Therefore, our heuristic retrieval method based on the topic model can successfully retrieve the relevant information.\n\n>**Question#2:**\\\n>Why the fine-tuned model perform better than the ChatGPT? How many parameters fine-tuned,10% ,30%,...,can exceed the ChatGPT?\n\n**Response of Questions#2:**\\\nThank you for your valuable comments about fine-tuning and your deep insight into this task. ChatGPT's suboptimal results can be attributed to its limited understanding of stance detection task. The advantage of the fine-tuned model lies in its ability to better understand the task through task-specific supervised learning. However, the limitation of the amount of training data prevents the development of general comprehension abilities. One promising method is distilling the understanding capabilities of large models and external knowledge into the forms of discourse knowledge and episodic knowledge so that the performance of fine-tuned models can be effectively improved, thus surpassing ChatGPT.\\\nThe question of how many parameters to fine-tune can exceed ChatGPT is influenced by various training strategies, such as which layer of network parameters to fine-tune or the use of parameter-efficient learning methods like prefix tuning, and exceeds the scope of this paper since we primarily focus on knowledge enhancement in stance detection tasks rather than complex model design and training strategies. Based on your suggestions, we conducted experiments on fine-tuning parameters on the 0th layer of RoBERTa (KASD-RoBERTa-large-layer-0) and the 11th layer of RoBERTa (KASD-RoBERTa-large-layer-11), two layers commonly used for study. The results are shown in Table 3.\n\n|                              |     Biden                   |      Sanders              |      Trump                    |       Avg                     |\n| :---                         |    :----:                   |    :----:                 |    :----:                     |    :----:                     |\n| ChatGPT                      | 82.30                       | 79.40                     |     82.80                     |     81.50                     |\n| KASD-RoBERTa-large           | 84.36                       | 79.69                     |     85.25                     |     83.10                     |\n| KASD-RoBERTa-large-layer-0   | 79.30                       | 74.26                     |     78.12                     |     77.23                     |\n| KASD-RoBERTa-large-layer-11  | 83.34                       | 79.12                     |     85.17                     |     82.54                    |\n\n*Table 3: Results of ChatGPT, KASD-RoBERTa-large, finetuning parameters on the 0th layer of RoBERTa (KASD-RoBERTa-large-layer-0) and the 11th layer of RoBERTa (KASD-RoBERTa-large-layer-11) detecting zero-shot stance on the P-Stance dataset.*\n\nFrom the results, we can observe that with knowledge-enhanced fine-tuned models, KASD-RoBERTa-large-layer-11 can outperform ChatGPT while only fine-tuning approximately 9% of its parameters. Thanks again for your comments; we will add the corresponding experiments and analysis to the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) It is an interesting work to incorporate the two background knowledge to stance detection, KASD framework can conduct the task with a simple and effective way.\n(2) The experimental result can well show the performance of KASD-based model, it can indeed improve the quality of the stance detection.", "reasons_to_reject": "(1) The work only use a LLM, ChatGPT, you can also test it in LLaMA.\n(2) In line 96, large language models(LLMs), such as ChatGPT, Bard, and LLaMA. It lacks the references about the three models", "questions_to_authors": "（1）Why you use the topic model to retrieval the target page, not use the BERT embedding to look up the pages?\n(2) Why the fine-tuned model perform better than the ChatGPT? How many parameters fine-tuned,10% ,30%,...,can exceed the ChatGPT?\n", "paper_topic_and_main_contributions": "This paper investigate stance detection from a perspective by exploring background knowledge for an understanding of the targets. The background knowledge is divided into episodic knowledge and discourse knowledge for learning of stance features. It design the KASD framework, which leverages ChatGPT to heuristically retrieve episodic knowledge and incorporate discourse knowledge.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to incorporate background knowledge into stance detection models, which is retrieved from Wikipedia and by prompting ChatGPT. To increase reproducability, the authors have additionally added ablation results with the open-source LLM LLaMA-2 in their rebuttal. They perform experiments on several stance benchmarks and for different levels of supervision, and show empirically that their proposed setting works well. The reviewers agree that the paper is well executed, though the approach is relatively pedestrian, as it is in line with a lot of contemporary work on prompting ChatGPT.
Decision: Accept-Main