SlowGuess/NAACL_2021_OCR huggingface dataset:
Lu Ji $^{1}$ , Zhongyu Wei $^{2*}$ , Jing Li $^{3}$ , Qi Zhang $^{1}$ , Xuanjing Huang $^{1}$
$^{2}$ School of Data Science, Fudan University, China
$^{3}$ Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China {17210240034,zywei,qz,xjhuang} @fudan.edu.cn
In this paper, we focus on identifying interactive argument pairs from two posts with opposite stances to a certain topic. Considering opinions are exchanged from different perspectives of the discussing topic, we study the discrete representations for arguments to capture varying aspects in argumentation languages (e.g., the debate focus and the participant behavior). Moreover, we utilize hierarchical structure to model post-wise information incorporating contextual knowledge. Experimental results on the large-scale dataset collected from CMV show that our proposed framework can significantly outperform the competitive baselines. Further analyses reveal why our model yields superior performance and prove the usefulness of our learned representations.
Arguments play a central role in decision making on social issues. Striving to automatically understand human arguments, computational argumentation becomes a growing field in natural language processing. It can be analyzed at two levels — monological argumentation and dialogical argumentation. Existing research on monological argumentation covers argument structure prediction (Stab and Gurevych, 2014), claims generation (Bilu and Slonim, 2016), essay scoring (Taghipour and Ng, 2016), etc. Recently, dialogical argumentation becomes an active topic.
In the process of dialogical arguments, participants exchange arguments on a given topic (Asterhan and Schwarz, 2007; Hunter, 2013). With the popularity of online debating forums, large volume of dialogical arguments are daily formed, concerning wide range of topics. A social media dialogical argumentation example from ChangeMyView subreddit is shown in Figure 1. There we show two
CMV: The position of vice president of the USA should be eliminated from our government.
Post A: a1: [If the president is either killed or resigns, the vice president is a horrible choice to take over office.] a2: The speaker of the House would be more qualified for the position. a3: [I'm willing to bet that John Boehner would have an easier time dealing with congress as president than Joe Biden would due to his constant interaction with it.] a4: If Boehner took office, as a republican, would he do something to veto bills Obama supported?
Post B: b1:[Seriously, stop this hyperbole.] b2:[Do you think that have anything to do with the fact that Boehner is a republican, and republicans control congress?] b3:[That argument has much less to do with the individuals than it does with the current party in control.
Figure 1: An example of dialogical argumentation consists of two posts from change my view, a sub-forum of Reddit.com. Different types of underlines are used to highlight the interactive argument pairs.
posts holding opposite stances over the same topic. One is the original post and the other is reply. As can be seen, opinions from both sides are voiced with multiple arguments and the reply post  $B$  is organized in-line with post  $A$ 's arguments. Here we define an interactive argument pair formed with two arguments from both sides (with the same underline), which focuses on the same perspective of the discussion topic. The automatic identification of these pairs will be a fundamental step towards the understanding of dialogical argumentative structure. Moreover, it can benefit downstream tasks, such as debate summarization (Sanchan et al., 2017) and logical chain extraction in debates (Botschen et al., 2018).
However, it is non-trivial to extract the interactive argument pairs holding opposite stances. Back to the example. Given argument b1 with only four words contained, it is difficult, without richer contextual information, to understand why it has interactive relationship with a1. In addition, without modeling the debating focuses of arguments, it is likely for models to wrongly predict that b2 has interactive relationship with a4 for sharing more words. Motivated by these observations, we pro
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5467-5478 June 6-11, 2021. ©2021 Association for Computational Linguistics
pose to explore discrete argument representations to capture varying aspects (e.g., the debate focus) in argumentation language and learn context-sensitive argumentative representations for the automatic identification of interactive argument pairs.
For argument representation learning, different from previous methods focusing on the modeling of continuous argument representations, we obtain discrete latent representations via discrete variational autoencoders and investigate their effects on the understanding of dialogical argumentative structure. For context representation modeling, we employ a hierarchical neural network to explore what content an argument conveys and how they interact with each other in the argumentative structure. To the best of our knowledge, we are the first to explore discrete representations on argumentative structure understanding. In model evaluation, we construct a dataset collected from  $CMV^{1}$ , which is built as part of our work and has been publicly released<sup>2</sup>. Experimental results show that our proposed model can significantly outperform the competitive baselines. Further analysis on discrete latent variables reveals why our model yields superior performance. At last, we show that the representations learned by our model can successfully boost the performance of argument persuasiveness evaluation.
2 Task Definition and Dataset Collection
In this section, we first define our task of interactive argument pair identification, followed by a description of how we collect the data for this task.
Given a argument  $q$  from the original post, a candidate set of replies consisting of one positive reply  $r^{+}$ , several negative replies  $r_{1}^{-} \sim r_{u}^{-}$ , and their corresponding argumentative contexts, our goal is to automatically identify which reply has interactive relationship with the quotation  $q$ .
We formulate the task of identifying interactive argument pairs as a pairwise ranking problem. In practice, we calculate the matching score  $S(q,r)$  for each reply in the candidate set with the quotation  $q$  and treat the one with the highest matching score as the winner.
2.2 Dataset Collection
Our data collection is built on the CMV dataset released by Tan et al. (2016). In CMV, users submit posts to elaborate their perspectives on a specific topic and other users are invited to argue for the other side to change the posters' stances. The original dataset is crawled using Reddit API. Discussion threads from the period between January 2013 and May 2015 are collected as training set, besides, threads between May 2015 and September 2015 are considered as test set. In total, there are 18,363 and 2,263 discussion threads in training set and test set, respectively.
An observation on CMV shows that when users reply to a certain argument in the original post, they quote the argument first and write responsive argument directly, forming a quotation-reply pair. Figure 2 shows how quotation-reply pairs could be identified. Inspired by this finding, we decide to
Original Post: ... Strong family values in society lead to great results. I want society to take positive aspects of the early Americans and implement that into society. This would be a huge improvement than what we have now. ... User Post: &gt; I want society to take positive aspects of the early Americans and implement that into society. What do you believe those aspects to be? ...
Figure 2: An example illustrating the formation process of a quotation-reply pair in CMV.
extract interactive argument pairs with the relation of quotation-reply. In general, the content of posts in CMV is informal, making it difficult to parse an argument in a finer-grain with premise, conclusion and other components. Therefore, following previous setting in Ji et al. (2018), we treat each sentence as an argument. Specifically, we only consider the quotation containing one argument and view the first sentence after the quotation as the reply. We treat the quotation-reply pairs extracted as positive samples and randomly select four replies from other posts that are also related to the original post to pair with the quotation as negative samples. In detail, each instance in our dataset includes the quotation, one positive reply, four negative replies, and the posts where they exist. The posts where they exist refer to argumentative contexts mentioned below. What's more, we remove quotations from argumentative contexts of replies.
We keep words with the frequency higher than 15 and this makes the word vocabulary with 20,692 distinct entries. In order to assure the quality of quotation-reply pairs, we only keep the instance where the number of words in the quotation and
1https://reddit.com/r/changemyview 2http://fudan-disc.com/data/ arg-pairs-fudanU.zip
training set	test set
# of arg. per post	11.8±6.6	11.4±6.2
# of token per post	209.7±117.2	205.9±114.6
# of token per q	20.0±8.6	20.0±8.6
# of token per pr	16.9±8.1	17.3±8.4
# of token per nr	19.0±8.0	19.1±8.1
max # of q-pr pairs	12	9
avg. # of q-pr pairs	1.5±0.9	1.4±0.9
Table 1: Overview statistics of the constructed dataset (mean and standard deviation). arg.,  $q, p_r, n_r$  represent argument, quotation, positive reply and negative reply respectively.  $q - p_r$  represents the quotation-reply pair between posts.
repplies range from 7 to 45. We regard the instances extracted from training set and test set in Tan et al. (2016) for training and test. The number of instances in training and test set is 11,565 and 1,481, respectively. We randomly select  $10\%$  of the training instances to form the development set. The statistic information of our dataset is shown in Table 1.
To further demonstrate that quotation-reply pairs have interactive relationships, we randomly select 100 instances from the test set and hire two trained annotators who are fluent English speakers to identify interactive argument pairs. The accuracy of the two annotators is 0.83 and 0.93, respectively. The inter-annotator agreement measured by Co-hens Kappa (Carletta, 1996) is 0.82. This confirms the quality of the constructed dataset.
The overall architecture of our model is shown in Figure 3(a). It takes a quotation, a reply and their corresponding argumentative contexts as inputs, and outputs a real value as its matching score. It mainly consists of three components, namely, Discrete Variational AutoEncoders (DVAE, Figure 3(c)), Argumentative Context Modeling (Figure 3(b)) and Argument Matching and Scoring. We learn discrete argument representations via DVAE and employ a hierarchical architecture to obtain the argumentative context representations. The Argument Matching and Scoring integrates some semantic features between the quotation and the reply to calculate the matching score.
3.1 Discrete Variational AutoEncoders
We employ discrete variational autoencoders (Rolfe, 2017) to reconstruct arguments from auto-encoding and obtain argument representations based on discrete latent variables to capture different aspects of argumentation languages.
Encoder. Given an argument  $x$  with words  $w_{1}, w_{2}, \ldots, w_{T}$ , we first embed each word to a dense vector obtaining  $w_{1}', w_{2}', \ldots, w_{T}'$  correspondingly. Then we use a bi-directional GRU (Wang et al., 2018) to encode the argument.
$$
h _ {t} = B i G R U \left(w _ {t} ^ {\prime}, h _ {t - 1}\right) \tag {1}
$$
We obtain the hidden state for a given word  $w_{t}^{\prime}$  by concatenating the forward hidden state and backward hidden state. Finally, we consider the last hidden state  $h^{T}$  as the continuous representation of the argument.
Discrete Latent Variables. We introduce  $z$  as a set of K-way categorical variables  $z = \{z_{1},z_{2},\dots,z_{M}\}$ , where  $M$  is the number of variables. Here, each  $z_{i}$  is independent and we can easily extend the calculation process below to every latent variables. Firstly, we calculate the logits  $l_{i}$  as follows.
$$
l _ {i} = W _ {l} h _ {i} ^ {T} + b _ {l} \tag {2}
$$
where  $W_{l} \in R^{K \times E}$  stands for the weight matrix,  $E$  is the dimension of hidden units in encoder, while  $b_{l}$  is a weight vector.
After obtaining the logits  $l_{i}$ , we can calculate the posterior distribution and discrete code of  $z_{i}$ .
$$
q \left(z _ {i} | x\right) = \operatorname {S o f t m a x} \left(l _ {i}\right) \tag {3}
$$
$$
Z _ {\text {c o d e}} (i) = \underset {k \in [ 1, 2, \dots , K ]} {\arg \max } \left(l _ {i k}\right) \tag {4}
$$
However, using discrete latent variables is challenging when training models end-to-end. To alleviate this problem, we use the recently proposed Gumbel-Softmax trick (Lu et al., 2017) to create a differentiable estimator for categorical variables. During training we draw samples  $g_{1}, g_{2}, \ldots, g_{K}$  from the Gumbel distribution:  $g_{k} \sim -\log(-\log(u))$ , where  $u \sim U(0,1)$  are uniform samples. Then, we compute the log-softmax of  $l_{i}$  to get  $\omega_{i} \in R^{K}$ :
$$
\omega_ {i k} = \frac {\exp \left(\left(l _ {i k} + g _ {k}\right) / \tau\right)}{\sum_ {k} \exp \left(\left(l _ {i k} + g _ {k}\right) / \tau\right)} \tag {5}
$$
$\tau$  is a hyper-parameter. With low temperature  $\tau$ , this vector  $\omega_{i}$  is close to the one-hot vector representing the maximum index of  $l_{i}$ . But with higher temperature, this vector  $\omega_{i}$  is smoother.
Then we map the latent samples to the initial state of the decoder as follows:
$$
h _ {d e c} ^ {0} = \sum_ {i = 1} ^ {M} W _ {e i} \omega_ {i} \tag {6}
$$
where  $W_{ei}\in R^{K\times D}$  is the embedding matrix,  $D$  is the dimension of hidden units in decoder. Finally, we use a GRU as the decoder to reconstruct the
Figure 3: (a) Overall architecture of the proposed model. (b) Hierarchical architecture for argumentative context modeling. (c) Detailed structure of the discrete variational autoencoders  $(DVAE)$ . (d) Structure of the quotation-guided attention in argument matching.
argument given  $h_{dec}^{0}$ .
Discrete Argument Representations. Through the process of auto-encoding mentioned above, we can reconstruct the argument. The representation that we want to find can capture varying aspects in argumentation languages and contain salient features of the argument.  $q(z_{i}|x)$  shows the probability distribution of  $z_{i}$  over  $K$  categories, which contains salient features of the argument on varying aspects. Therefore, we obtain the discrete argument representation by the posterior distribution of discrete latent variables  $z$ .
$$
R = \sum_ {i = 1} ^ {M} W _ {e i} q \left(z _ {i} \mid x\right) \tag {7}
$$
3.2 Argumentative Context Modeling
Here, we introduce contextual information of the quotation and the reply to help identify the interactive argument pairs. The argumentative context contains a list of arguments. Following previous setting in Ji et al. (2018), we consider each sentence as an argument in the context. Inspired by Dong et al. (2017), we employ a hierarchical architecture to obtain argumentative context representations.
Argument-level CNN. Given an argument and their embedding forms  $\{e_1, e_2, \dots, e_n\}$ , we employ a convolution layer to incorporate the context information on word level.
$$
s _ {i} = f \left(W _ {s} \cdot \left[ e _ {i}: e _ {i + w s - 1} \right] + b _ {s}\right) \tag {8}
$$
where  $W_{s}$  and  $b_{s}$  are weight matrix and bias vector.  $ws$  is the window size in the convolution layer and  $s_i$  is the feature representation. Then, we conduct an attention pooling operation over all the words to get argument embedding vectors.
$$
m _ {i} = \tanh  \left(W _ {m} \cdot s _ {i} + b _ {m}\right) \tag {9}
$$
$$
u _ {i} = \frac {e ^ {W _ {u} \cdot m _ {i}}}{\sum_ {j} e ^ {W _ {u} \cdot m _ {j}}} \tag {10}
$$
$$
a = \sum_ {i} u _ {i} \cdot s _ {i} \tag {11}
$$
where  $W_{m}$  and  $W_{u}$  are weight matrix and vector,  $b_{m}$  is the bias vector,  $m_{i}$  and  $u_{i}$  are attention vector and attention weight of the  $i$ -th word.  $a$  is the argument representation.
Document-level BiGRU. Given the argument embedding  $\{a_1, a_2, \ldots, a_N\}$ , we employ a bidirectional GRU to incorporate the contextual information on argument level.
$$
h _ {i} ^ {c} = B i G R U \left(a _ {i}, h _ {i - 1} ^ {c}\right) \tag {12}
$$
Finally, we employ an average pooling over arguments to obtain the context representation  $C$ .
3.3 Argument Matching and Scoring
Once representations of the quotation and the reply are generated, three matching methods are applied to analyze relevance between the two arguments. We conduct element-wise product and
element-wise difference to get the semantic features  $f_{p} = R_{q} * R_{r}$  and  $f_{d} = R_{q} - R_{r}$ . Furthermore, to evaluate the relevance between each word in the reply and the discrete representation of the quotation, we propose the quotation-guided attention and obtain a new representation of the reply.
Quotation-Guided Attention. We conduct dot product between  $R_{q}$  and each hidden state representation  $h_j^r$  in the reply. Then, a softmax layer is used to obtain an attention distribution.
$$
v _ {j} = \operatorname {s o f t m a x} \left(R _ {q} \cdot h _ {j} ^ {r}\right) \tag {13}
$$
Based on the attention probability  $v_{j}$  of the  $j$ -th word in the reply, the new representation of the reply can then be constructed as follows:
$$
f _ {r} = \sum_ {j} v _ {j} \cdot h _ {j} ^ {r} \tag {14}
$$
After obtaining the discrete representations, argumentative context representations and some semantic matching features  $f_{p}, f_{d}, f_{r}$  of the quotation and the reply, we use two fully connected layers to obtain a higher-level representation  $H$ . Finally, the matching score  $S$  is obtained by a linear transformation.
$$
f _ {m} = \left[ f _ {p}; f _ {d}; f _ {r} \right] \tag {15}
$$
$$
H = f \left(W _ {H} \left[ R _ {q}; R _ {r}; C _ {q}; C _ {r}; f _ {m} \right] + b _ {H}\right) \tag {16}
$$
$$
S = W _ {s} H + b _ {s} \tag {17}
$$
where  $W_{H}$  and  $W_{S}$  stand for the weight matrices, while  $b_{H}$  and  $b_{S}$  are weight vectors.
The proposed model contains three modules, i.e., the  $DVAE$ , argumentative context modeling and argument matching, which are trained jointly. We define the loss function of the overall framework to combine the two effects.
$$
L = L _ {D V A E} + \lambda L _ {m} \tag {18}
$$
where  $\lambda$  is a hyper-parameter to balance the two loss terms. The first loss term is defined on the DVAE and cross entropy loss is defined as the reconstruction loss. We apply the regularization on  $KL$  cost term to solve posterior collapse issue. Due to the space limitation, we leave out the derivation details and refer the readers to Zhao et al. (2018).
$$
L _ {D V A E} = E _ {q (z \mid x)} [ \log p (x \mid z) ] - K L (q (z \mid x) | | p (z)) \tag {19}
$$
The second loss term is defined on the argument matching. We formalize this issue as a ranking task
and utilize hinge loss for training.
$$
L _ {m} = \sum_ {i = 1} ^ {u} \max  \left(0, \gamma - S \left(q, r ^ {+}\right) + S \left(q, r _ {i} ^ {-}\right)\right) \tag {20}
$$
where  $u$  is the number of negative replies in each instance.  $\gamma$  is a margin parameter,  $S(q,r^{+})$  is the matching score of the positive pair and  $S(q,r_{i}^{-})$  is the matching score of the  $i$ -th negative pair.
We use Glove (Pennington et al., 2014) word embeddings with dimension of 50. The number of discrete latent variables  $M$  is 5 and the number of categories for each latent variable is also 5. What's more, the hidden units of GRU cell in encoder are 200 while that for the decoder is 400. We set batch size to 32, filter sizes to 5, filter numbers to 100, dropout with probability of 0.5, temperature  $\tau$  to 1. The hyper-parameters in loss function are set as  $\gamma = 10$  for max margin and  $\lambda = 1$  for controlling the effects of discrete argument representation learning and argument matching.
The proposed model is optimized by SGD and applied the strategy of learning rate decay with initial learning rate of 0.1. We evaluate our model on development set at every epoch to select the best model. During training, we run our model for 200 epochs with early-stop (Caruana et al., 2000).
4.2 Comparison Models
For baselines, we consider simple models that rank argument pairs with cosine similarity measured with two types of word vectors: TF-IDF scores (henceforth TF-IDF) and the pre-trained word embeddings from word2vec corpus (henceforth WORD2VEC). Also, we compare with the neural models from related areas: MALSTM (Mueller and Thyagarajan, 2016), the popular method for sentence-level semantic matching, and CBCA-WOF (Ji et al., 2018), the state-of-the-art model to evaluate the persuasiveness of argumentative comments, which is tailored to fit our task. In addition, we compare with some ablations to study the contribution from our components. Here we first consider  $\mathrm{MATCH}_{rnn}$ , which uses BiGRU to learn argument representations and explore the match of arguments without modeling the context therein. Then we compare with other ablations that adopt varying argument context modeling methods. Here we consider BiGRU (henceforth  $\mathrm{MATCH}_{rnn} + \mathrm{C}_b$ ), which
Models	P@1	MRR
Cosine Similarity based
TF-IDF	28.36*	51.66*
WORD2VEC	28.70*	52.03*
Neural-Network based
MALSTM (Mueller and Thya-garajan, 2016)	31.26*	52.97*
CBCAWOF (Ji et al., 2018)	56.04*	73.03*
Ablation Study
MATCHrnn	51.52*	70.57*
MATCHrnn+Cb	55.98*	73.20*
MATCHrnn+Ch	57.46*	73.72*
MATCHae+Ch	58.27‡	74.16*
MATCHvae+Ch	58.61‡	74.66‡
Our model	61.17	76.16
Table 2: The performances of different models on our dataset in terms of Mean Reciprocal Rank (MRR) and Precision at 1 (denoted as  $P@1$ ). The proposed model significantly outperforms all the comparison methods marked with * or  $\ddagger$  (*:  $p<0.01$ ;  $\ddagger$ :  $p<0.05$ , Wilcoxon signed rank test). Best results are in bold.
focuses on words in argument context and ignores the argument interaction structure. We also consider a hierarchical neural network ablation (henceforth  $\mathrm{MATCH}_{rnn} + \mathrm{C}_h$ ), which models argument interactions with BiGRU and the words therein with CNN. In addition, we compare with  $\mathrm{MATCH}_{ae} + \mathrm{C}_h$  and  $\mathrm{MATCH}_{vae} + \mathrm{C}_h$ , employing auto-encoder (AE) and variational AE (VAE), respectively, to take the duty of the DVAE module of our full model.
To evaluate the performance of different models, we first show the overall performance of different models for argument pair identification. Then, we conduct three analyses including hyper-parameters sensitivity analysis, discrete latent variables analysis and error analysis to study the impact of hyperparameters, explain why DVAE performs well on interactive argument pair identification and analyze the major causes of errors. Finally, we apply our model to a downstream task to investigate the usefulness of discrete argument representations.
5.1 Overall Performance Comparison
The overall results of different models are shown in Table 2. Mean Reciprocal Rank (MRR) and Precision at 1 (denoted as  $P@1$ ) are used for evaluation metrics. We have following findings.
- Our model significantly outperforms all comparison models in terms of both evaluation metrics. This proves the effectiveness of our model.
- Neural network models perform better than TFIDF and WORD2VEC. This observation shows
Figure 4: The impact of varying the number of discrete latent variables  $M$  and categories for each latent variable  $K$  on  $P@1$ . We find that our model still outperforms VAE which is the most competitive baseline.
the effectiveness of argument representation learning in neural networks.