TREA: Tree-structure Reasoning Schema for Conversational Recommendation
Year: 2023

Abstract:
Conversational recommender systems (CRS) aim to timely trace the dynamic interests of users through dialogues and generate relevant responses for item recommendations. Recently, various external knowledge bases (especially knowledge graphs) are incorporated into CRS to enhance the understanding of conversation contexts. However, recent reasoning-based models heavily rely on simplified structures such as linear structures or fixed-hierarchical structures for causality reasoning, hence they cannot fully figure out sophisticated relationships among utterances with external knowledge. To address this, we propose a novel Treestructure Reasoning schEmA named TREA. TREA constructs a multi-hierarchical scalable tree as the reasoning structure to clarify the causal relationships between mentioned entities, and fully utilizes historical conversations to generate more reasonable and suitable responses for recommended results. Extensive experiments on two public CRS datasets have demonstrated the effectiveness of our approach. Our code is available at https: //github.com/WindyLee0822/TREA

heading: 1 Introduction
text: Conversation Recommender System (CRS) has become increasingly popular as its superiority in timely discovering user dynamic preferences in practice. As opposed to traditional passive-mode recommendation systems, it highlights the importance of proactively clarifying and tracing user interests through live conversation interactions, which notably enhance the success rate of item recommendations.
Since sole contextual utterances are insufficient for comprehensively understanding user preferences, there are many efforts devoted to incorporat-
Corresponding Author
ing various external knowledge (Chen et al., 2019; Zhou et al., 2020a, 2022; Wang et al., 2022; Yang et al., 2022), which typically enrich the contextual information with mentioned entities recognized over utterances. However, these methods fail to model the complex causal relations among mentioned entities, owing to the diversity of user interest expression and the frequent shift of conversation topic as shown in Figure 1.
Actually, it is non-trivial to explicitly model the complex causal relationships of conversations. Although there are several reasoning-based methods proposed for CRS, their simplified structures make the objective unattainable. Some researches (Zhou et al., 2021) track the mentioned entities as linear sequential fragments analogous to (1) in Figure 1. However, the linear structure is only suitable for adjacent relation modeling, which may not always work well since the actual causality between mentioned entities exists multi-hop jumps ("comedy""La La Land" in Figure 1). Other studies (Ma et al., 2021) propose other forms of specially-designed structures for reasoning akin to (2) in Figure 1, but they generally have fixed hierarchies, which often degenerate into a simple 2-layer hierarchy "history"-"prediction", neglecting the causal relations of historical entities. Therefore, neither of them is applicable for full modeling of the complex reasoning causality within conversations.
To improve the reasoning capability of CRS, the challenges are twofold. The first challenge lies in empowering the model to illuminate the causal inference between all mentioned entities. To tackle this, we performs abductive reasoning for each mentioned entity to construct the multi-hierarchical reasoning tree. The reasoning tree explicitly preserves logical relations between all entities and can be continuously expanded as the conversation continues, which provides the model with a clear
2970
reference to historical information for prediction. The second challenge is how to utilize reasoning information in response generation. We enable the model to extract relevant textual information from the historical conversation with the corresponding reasoning branch, thus promoting the correlation between generated responses and recommended items. We name this Tree-structure Reasoning schEmA TREA.
To validate the effectiveness of our approach, we conduct experiments on two public CRS datasets. Experimental results show that our TREA outperforms competitive baselines on both the recommendation and conversation tasks. Our main contributions are summarized as follows:
• To the best of our knowledge, it is the first trial of CRS to reason every mentioned entity for its causation.
• We propose a novel tree-structured reasoning schema to clarify the causality relationships between entities and mutual the reasoning information with the generation module.
• Extensive experiments demonstrate the effectiveness of our approach in both the recommendation and conversation tasks.

heading: 2 Related Work
text: Conversational Recommender System (CRS) explores user preference through natural language
dialogues. Previous works can be roughly categorized into two types. The first category of CRS is recommendation-biased CRS (Sun and Zhang, 2018; Lei et al., 2020b,a; Deng et al., 2021; Zhang et al., 2022). This category focuses solely on interactive recommendations but the function of natural language is ignored. Several fixed response templates are preset on the agents and users cannot use free text but only have limited options, which can be detrimental to the user experience.
The other category of CRSs is dialog-biased CRS (Li et al., 2018; Moon et al., 2019; Chen et al., 2020; Liu et al., 2021; Sarkar et al., 2020). This category emphasizes the critical effect of natural language, aiming to understand user utterances for accurate recommendations and generate human-like responses. Noticing that entities (Gu et al., 2022; Qu et al., 2022, 2023) mentioned in conversations are important cues for modeling user preferences, Chen et al. (2019) firstly integrates KG to enhance the user representation. Zhou et al. (2020a); Liang et al. (2021) use two KGs on entity-granularity and word-granularity respectively to represent the user preference more comprehensively. Subsequent researches introduce other types of external knowledge e.g. item description (Lu et al., 2021; Zhou et al., 2022) or pretrained language models (PLMs) (Yang et al., 2022; Wang et al., 2022) to further assist the user representations. However, they commonly treat each mentioned knowledge piece equally and integrate them into an aggregated representation.
Recently, some researches manage to model the reasoning process during conversations. Zhou et al. (2021) linearize the mentioned entity sequence and reasoning the inferential causality between the adjacent entity pairs. Ma et al. (2021) create non-linear reasoning structures, but they do not preserve the hierarchy of historical turns. Therefore these reasoning methods have limited performance improvement.
To sort out the causal relations among utterances, our model performs tree-structured reasoning on the entire dialogue history for each mentioned entity. We also inject the reasoning information into the generation process to make responses more relevant, achieving that the reasoning process facilitates both recommendation and generation tasks simultaneously.

heading: 3 Methods
text: In this section, we present the Tree-structure reasoning schema TREA as demonstrated in Figure 2. Specifically, we first introduce the encoding of entities and word tokens. Then we illustrate the construction procedure of the reasoning tree. Later, we describe how the reasoning information supports the generation module. Finally, we explain the process of parameter optimization.

heading: 3.1 Entity and Dialog Encoding
text: Following previous works (Chen et al., 2019; Zhou et al., 2020a; Ma et al., 2021; Zhou et al., 2022),
we first perform entity linking based on an external KG DBpedia (Bizer et al., 2009), and then encode the relational semantics via a relational graph neural network (RGCN) (Schlichtkrull et al., 2018) to obtain the corresponding entity embeddings. Formally, the embedding nl+1e of entity e at the l+1-th graph layer is calculated as:
nl+1e = σ( ∑
r∈R
∑
e′∈N re
1
Ze,r Wlrn l e′ + W lnle) (1)
where R is a relation set, N re denotes the set of neighboring nodes for e under the relation r, Wlr,Wl are learnable matrices for relation-specific aggregation with neighboring nodes and representation transformation respectively, Ze,r is a normalization factor, σ denotes the sigmoid function. The semantic information of word tokens is encoded by an external lexical knowledge graph ConceptNet (Speer et al., 2017). We further adopt a graph convolutional neural network (GCN) (Kipf and Welling, 2016) to propagate and aggregate information over the entire graph.

heading: 3.2 Reasoning Tree Construction.
text: The construction of reasoning trees is introduced in a manner similar to mathematical induction. We first explain the structure initialization at the first conversation round, then illustrate the structure transition from the (n-1)-th round to the n-th round. The structure of the whole tree can be deduced accordingly.
To initialize the reasoning tree, we first set a pseudo node as the root node. The root node does not represent any entity in the conversations but is just a placeholder. When the first utterance is coming, the first mentioned entity is directly connected to the root node. The subsequent entities in the first utterance are connected following the Algorithm 1.
When the conversation progresses to (n-1)-th round, the known conditions are as follows: the current reasoning tree Tn−1, utterance tokens sequences st. They are utilized for the extension of the reasoning tree Tn−1, which is described in two parts, tree-structure reasoning and the selection & connection of candidate entities.
Tree-Structure Reasoning. We embed all the reasoning branches and pad them to a certain length lr. A path from the root node to any leaf node of the tree is referred to as a reasoning branch since it expresses a chain of coherent inferences. To represent the sequential information for each reasoning branch, we inject a learnable position embedding into the embedding of each entity element. The position-enhanced branch embedding matrix is denoted as P ∈ Rnr×lr×d where nr is the branch number of Tn−1 and d is the dimension of embeddings. We incorporate a linear attention mechanism to integrate the representation of each path. The attention scores are calculated as follows:
P̃ = Attn(P) = Pαr αr = Softmax(br tanh(WrP))
(2)
where Wr,br are learnable parameters. Embeddings of entities in a certain reasoning branch are aggregated according to the attention score. Then we can obtain the comprehensive representations of reasoning branches denoted as P̃ ∈ Rnr×d.
Selection & Connection. Since the reasoning branches have varying-degrees contributions to the next-hop entity, the model analyzes the semantics of word tokens st to measure the impact of each branch. The formulas are as follows:
p = Attn(γP̃ + (1− γ)s) γ = σ(WsConcat(P̃ , s))
(3)
where Ws is a learnable parameter, s is the comprehensive semantic representation of the word tokens in ConceptNet which are aggregated with the linear attention mechanism in Eq.2. Then we can obtain the user representation pu that combines semantic and reasoning information. Since the latest turn has a prominent significance to the response (Li et al.,
2022), we collect the entities and word tokens from the current conversation turn, embedded to ec, sc. Then we aggregate the current turn information and mutual it with acquired representation p as follows:
pu = g(p, g ′(Attn(ec),Attn(sc)) (4)
where g(· , ·), g′(· , ·) are two gate layers like Eq.3. Then we derive the next-hop possibility distribution from the overall user representation:
Pur = Softmax([pueT0 , · · · , pueTn]) (5) where e0, · · · , en are representations of all entities. The entity with the largest probability is selected and connected to the reasoning tree. The connection strategy is as Algorithm 1.
Algorithm 1: Connection Strategy input :Selected entity e∗; Entity sequence
ES in reverse order of mention; Reasoning Tree T with root node r
1 foreach e in ES do 2 if IsAdj(e,e∗) then 3 // Two entities are adjacent in KG; 4 AddEdge(e,e∗); 5 // Add an edge (e, e∗) in T ; 6 return 7 end 8 end 9 AddEdge(r,e∗);
10 return

heading: 3.3 Reasoning-guided Response Generation
text: After adding the predicted entity to the reasoning tree, the objective of the conversation module is to generate utterances with high relevance to the predicted entity. Reasoning branches that involve the new entity and the historical utterances that mention the relevant entities in branches are extracted, which are encoded by RGCN and standard Transformer (Vaswani et al., 2017) respectively. The corresponding embedding matrices are denoted as E,U. Following (Zhou et al., 2020a), we incorporate multiple cross-attention layers in a Transformer-variant decoder to fuse the two groups of information. The probability distribution over the vocabulary is calculated as follows:
Rl = Decoder(Rl−1,E,U) (6)
Rb = FFN(Concat(Attn(E),Rl)) (7)
Pg = Softmax(RlVT + RbWv) (8)
where V is the embedding matrix of all words in the vocabulary, Wv is a learnable parameter that converts the Rb dimension to |V|. The copy mechanism is adopted in Eq.7 to enhance the generation of knowledge-related words. The transformation chain (Zhou et al., 2020a) in the decoder of Eq.6 is generated words → relevant entities → historical utterances.

heading: 3.4 Optimization
text: The parameters can be categorized into two parts, the reasoning parameters and the generation parameters, denoted by θr, θg. The reasoning objective is to maximize the predicted probability of the upcoming entity. The cross-entropy loss is adopted to train the reasoning module. During the training, we propose two auxiliary loss functions, isolation loss to maintain the independence of each reasoning branch, and alignment loss to bridge the representation gap.
Isolation Loss. Since reasoning branches that have no shared parts are generally irrelevant, representations from different reasoning branches are expected to be dissimilar. To maintain the isolation of each reasoning branch, we propose isolation loss. Given representations of different reasoning branches, the isolation loss is calculated as
LI = ∑
i =j sim(p̃i, p̃j) =
∑
i =j
p̃ip̃j |p̃i| · ∣∣p̃j ∣∣ (9)
where p̃i, p̃j are representations of two different reasoning branches extracted from P̃.
Alignment Loss. The representation gap exists between the semantics and the entities since their encoding processes are based on two separate networks. Hence the entity representation and semantic representation of the same user should be dragged closer; those of different users should be pushed further to reduce the gap. The formula is as follows:
La = λcsim(pc, sc) + (1− λc)sim(p, s) (10) where pc, sc are aggregated representation Attn(ec),Attn(wc) in Eq.4, λc is a hyperparameter.
Then We can optimize parameters θr through the following formula:
Lr = − ∑
u
∑
ei
logPur [ei] + λILI + λaLa (11)
where ei is the order of the target entity at the i-th conversation round of user u, λI , λal are hyperparameters.
When the reasoning loss Lr converges, we optimize the parameters in θg. After obtaining the relevant entities and utterances via the reasoning tree, we calculate the probability distribution of the next token. To learn the generation module, we set the cross-entropy loss as:
Lg = − 1
N
N∑
t=1
logPtg(st|s1, s2, . . . , st−1) (12)
where N is the number of turns in a certain conversation C. We compute this loss for each utterance st from C.

heading: 4 Experiment

heading: 4.1 Dataset.
text: We conduct our experiments on two widely-applied benchmark datasets on CRS, which are multilingual including English (ReDial) and Chinese (TGReDial). ReDial(Li et al., 2018) collects highquality dialogues for recommendations on movies through crowd-sourcing workers on Amazon Mechanical Turk(AMT). The workers create conversations for the task of movie recommendation in a user-recommender pair setting following a set of detailed instructions. It contains 10,006 conversations consisting of 182,150 utterances. TGReDial(Zhou et al., 2020b) is annotated in a semiautomatic way. It emphasizes natural topic transitions from non-recommendation scenarios to the desired recommendation scenario. Each conversation includes a topic path to enforce natural semantic transitions. It contains 10,000 conversations consisting of 129,392 utterances.

heading: 4.2 Baselines
text: We evaluate the effectiveness of our model with following competitive baselines:
ReDial (Li et al., 2018) comprises a conversation module based on hierarchical encoder-decoder architecture(Serban et al., 2017) and a recommendation module based on auto-encoder.
KBRD (Chen et al., 2019) firstly utilizes KG to enhance the user representation. The Transformer(Vaswani et al., 2017) architecture is applied in the conversation module.
KGSF (Zhou et al., 2020a) incorporate two external knowledge graphs on different aspects to further enhance the user representations. The KG information is employed in the decoding process.
CRFR (Zhou et al., 2021) can generate several linear reasoning fragments through reinforcement learning to track the user preference shift.
CR-Walker (Ma et al., 2021) create a twohierarchy reasoning tree between history and forecast and preset several dialog intents to guide the reasoning.
C2-CRS (Zhou et al., 2022) proposed a contrastive learning based pretraining approach to bridge the semantic gap between three external knowledge bases.
UCCR (Li et al., 2022) considers multi-aspect information from the current session, historical sessions, and look-alike users for comprehensive user modeling.

heading: 4.3 Metrics
text: For recommendation evaluation, we used Recall@n (R@n,n=10,50), which shows whether the top-n recommended items include the ground truth suggested by human recommenders. For the response generation task, we evaluate models by Bleu-n(n=2,3) (Papineni et al., 2002), Dist-n(n=3,4) (Li et al., 2016) for word-level matches and diversity. To evaluate the generation performance more equitably, three annotators are invited to score the generated candidates from two datasets for human evaluation on the following three aspects: Fluency, Relevance, and Informativeness. The interannotator coherence is measured by Fleiss’ Kappa.

heading: 4.4 Implementation Details
text: We keep the same data preprocessing steps and hyperparameter settings as previous researches (Zhou et al., 2022; Ma et al., 2021). We adopt the same mask mechanism as NTRD(Liang et al., 2021).
The embedding dimensions of reasoning and generation are set to 300 and 128 respectively. In the encoding module, the word embeddings are initialized via Word2Vec1 and the layer number is set to 1 for both GNN networks. The normalization constant of RGCN is 1. We use Adam optimizer (Kingma and Ba, 2015) with the default parameter setting. For training, the batch size is set to 64, the learning rate is 0.001, gradient clipping restricts the gradients within [0,0.02]. For hyperparameters, Ze, r of RGCN in Eq.1 is 1, λc of representation alignment in Eq.10 is 0.9, λI , λa in Eq.11 is 0.008, 0.002 respectively.

heading: 4.5 Overall Performance Analysis
text: Recommendation. The columns R@10,R@50 of Table 1 present the evaluation results on the recommendation task. It shows that our TREA significantly outperforms all the baselines by a large margin on both two datasets, which verifies that TREA can clarify the sophisticated causality between the historical entities and accurately model the user preferences. Moreover, even though RevCore and C2-CRS utilize the additional knowledge, they are still not as effective as TREA, which further proves the significance of correct reasoning. CR-walker and CRFR are two previous methods that manage to reason over the background knowledge. CRWalker does not preserve the hierarchy between the historical information and CRFR linearizes the reasoning structure. Therefore even though CR-walker conducts the additional annotations of dialog intents and CRFR applies the reasoning on another
1https://radimrehurek.com/gensim/models/ word2vec.html
KG to assist, the performance raising is limited, which certifies that our non-linear tree-structured reasoning over all mentioned entities does facilitate the user modeling.
Generation. The columns Dist-n, Bleu-n of Table 1 present the automatic evaluation results on the conversation task. Since CR-walker adopts GPT-2 in the original model, we initialize the generation module with Word2Vec instead for a fair comparison. It shows that TREA surpasses all baselines on generation diversity and matchness. Table 2 presents the human evaluation results. All Fleiss’s kappa values exceed 0.6, indicating crowd-sourcing annotators have reached an agreement. The results show that our TREA leads to a higher relevance of generated utterances. It can be derived that the extraction of relevant information with the reasoning tree does improve the relevance of the generation.

heading: 4.6 Ablation Study
text: Recommendation. The parameter optimization for the reasoning module involves two additional loss, isolation loss (Iso.) LI and alignment loss (Aln.) La. We would like to verify the effectiveness of each part. We incorporate three variants of our model for ablation analysis on the recommendation task, namely TREA w/o Iso., TREA w/o Aln. and TREA w/o IA., which remove the isolation loss, the alignment loss and both of them respectively. As shown in Table 3, both components contribute to the final performance. Furthermore, we can see that removing the isolation loss leads to a large performance decrease, which suggests that maintaining the representation dependence of each reasoning branch is crucial to the correctness of the reasoning.
To further confirm that the performance improvement is consistent and stable instead of acciden-
tal. We test the models under different iteration steps and display the corresponding results in Figure 3. It can be seen that when the training loss converges, each ablation component contributes to the model performance regardless of the iteration number, which proves that the two additional loss functions are stably effective.
The Effect of Isolation Loss. The above subsection has verified the great impact of the isolation loss. We take a deeper dive to determine how it benefits model performance. If removing the isolation loss, since each reasoning branch participates in the calculation of the predicted possibility distribution, the representations of entities in different reasoning branches would approach each other for sharper descending of the loss value, which means that the representation of unrelevant entities would get similar irrationally and finally lead to the representation convergence of the entire knowledge graph. To confirm the assumption, we display the entity embeddings trained by TREA and TREA
w/o Iso. in Figure 4. It shows that representations of KG entities in model without the isolation loss are more congested and less distinguishable. It demonstrates the isolation loss can prohibit the clustering of the nodes in KG, which is consistent with the above conjecture.
Generation. To examine whether the extraction of the relevant information through the reasoning tree benefits the generation, we conduct the ablation study based on three variants of our complete model, which utilize the whole historical entities, the whole historical utterances and both of the above without extraction, namely TREA w/o Ent., TREA w/o Utt., TREA w/o EU. respectively. The results in Table 4 show that deleting either extraction brings a performance decrease on all generation metrics. PPL (Perplexity) is an automatic evaluation metric for the fluency of generations and confidence in the responses. The results of PPL show that the extraction of the relevant information reduced the model confusion. A substantial decrease on Rel. shows that reasoning-guided extraction especially influences the relevance of the generation.

heading: 4.7 Evaluation on Long Conversations
text: We further evaluate TREA in long conversation scenarios. To the best of our knowledge, it is the
first time to discuss this aspect of CRS. When the dialogue becomes longer and more knowledge information appears, if the relationships between knowledge pieces are not clarified, the model is not able to utilize the historical information effectively. We evaluate our TREA and a competitive baseline UCCR on data of different conversation rounds, measured by the metric Recall@50. The results in Figure 5 shows that the performance of UCCR decreases sharply when the conversation rounds exceed 12 in ReDial and 14 in TG-ReDial. On the contrary, the performance of TREA fluctuates less as the number of conversation rounds increases. It indicates that the reasoning process of TREA can illuminate sophisticated relationships between historical entities for a better reference to the current situation, which further proves that nonlinear reasoning with historical hierarchy is vital to modeling user preference, especially when the conversation is long and the informativeness is great.

heading: 5 Conclusion
text: In this paper, we propose a novel tree-structure reasoning schema for CRS to clarify the sophisticated relationships between mentioned entities for accurate user modeling. In the constructed reasoning tree, each entity is connected to its cause which motivates the mention of the entity to pro-
vide a clear reference for the current recommendation. The generation module also interacts with the reasoning tree to extract relevant textual information. Extensive experimental results have shown that our approach outperforms several competitive baselines, especially in long conversation scenarios.

heading: 6 Limitations
text: The construction of the reasoning tree may be affected by the KG quality since the connection operations are variant with the KG structure. Hence the unsolved problem in Knowledge Graph such as incompleteness or noise could disturb the reasoning process. In the future, we will explore a solution to alleviate the influence of the side information.

heading: Acknowledgements
text: This work was supported in part by the National Natural Science Foundation of China under Grant No.62276110, No.62172039 and in part by the fund of Joint Laboratory of HUST and Pingan Property Casualty Research (HPL). The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
3 A2. Did you discuss any potential risks of your work? 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? 4

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 4

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? 4
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 4

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: 4

heading: Future Work
text: In the future, we will explore a solution to alleviate the influence of the side information