SlowGuess/NAACL_2021_OCR huggingface dataset:
Yingxue Zhang $^{1}$ , Fandong Meng $^{1}$ , Peng Li $^{1}$ , Ping Jian $^{2}$ , Jie Zhou $^{1}$
$^{1}$ Pattern Recognition Center, WeChat AI, Tencent Inc, China
$^{2}$ Beijing Institute of Technology, China
{yxuezhang, fandongmeng, patrickpli, withtomzhou} @tencent.com
pjian@bit.edu.cn
Implicit discourse relation recognition (IDRR) aims to identify logical relations between two adjacent sentences in the discourse. Existing models fail to fully utilize the contextual information which plays an important role in interpreting each local sentence. In this paper, we thus propose a novel graph-based Context Tracking Network (CT-Net) to model the discourse context for IDRR. The CT-Net firstly converts the discourse into the paragraph association graph (PAG), where each sentence tracks their closely related context from the intricate discourse through different types of edges. Then, the CT-Net extracts contextual representation from the PAG through a specially designed cross-grained updating mechanism, which can effectively integrate both sentence-level and token-level contextual semantics. Experiments on PDTB 2.0 show that the CT-Net gains better performance than models that roughly model the context.
Implicit discourse relation recognition (IDRR) aims to identify logical relations between two adjacent sentences in discourse without the guidance of connectives (e.g., because, but), which is one of the major challenges in discourse parsing. With the rise of deep learning, lots of sentence-modeling based methods (Liu and Li, 2016; Ronqvist et al., 2017; Bai and Zhao, 2018; Xu et al., 2019; Shi and Demberg, 2019) have emerged in the field of IDRR. These methods typically focus on modeling the local semantics of these two sentences, without considering wider discourse context.
Contextual information plays an important role in understanding sentences. Take the paragraph  $P = \{S_1, S_2, S_3, S_4\}$  in Figure 1 as an example, the ground-truth relation between  $S_3$  and  $S_4$  is "Comparison". Combining the contextual information carried by  $S_1$  and  $S_2$ , we can more easily identify the "Comparison" relation reflected by
S1:The manufacturer went public at $15.75 a share in August 1987.
S2: Mr. Sim's goal then was a $29 per-share price.
$S_{3}$ : Strong earnings growth helped achieve that price far ahead of schedule.
S4:The stock has since softened, trading around $25 a share
Figure 1: The paragraph association graph (PAG) (right) built for a case (left) of PDTB 2.0.
"achieve that price" (rising: "\(15.75 a share" to "\(29 per-share") and "softened" (falling: "\(29 per-share" to "\(25 a share"). Dai and Huang (2018) move one step on utilizing wider discourse context, where they use a hierarchical BiLSTM (H-LSTM) to model the whole paragraph rather than only the two sentences, to obtain context-aware sentence representation. However, there are still two limitations in their model. First, they roughly merge all the information in the paragraph, which dilutes the role of key context that closely related to the current sentence. Second, the H-LSTM suffers from the long-distance forgetting problem, which may fail to model the long-distance and non-continuous dependency across multiple sentences (like green lines in Figure 1).
To overcome these limitations, we propose a novel Context Tracking Network (CT-Net), which can track essential context for each sentence from the intricate discourse, without being affected by the spatial distance. The CT-Net computes contextual representation through two main steps. Firstly, it converts the paragraph into the paragraph association graph (PAG) (Figure 1), which contains three types of edges between sentences, namely (1) adjacency edge (black lines): connecting adjacent sentences, (2) co-reference edge (purple lines): connecting sentences with co-reference associations, and (3) lexical chain edge (green lines): connecting sentences containing related words. Each sentence can track closely related context along these
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1592-1599 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: The overall architecture of the CT-Net. Given a paragraph  $P = (S_{1}, S_{2}, S_{3}, S_{4})$ , it converts  $P$  into the PAG  $\mathcal{G}$ , then employs the cross-grained updating mechanism on  $\mathcal{G}$  to get contextual representation for classification.
edges, including long-distance sentences involving the same object or topic. Secondly, the CT-Net extracts contextual representation over the PAG. To effectively incorporate fine-grained information carried by tokens, we propose the cross-grained updating mechanism, which will be executed multiple recurrent rounds. At each round, it performs semantic exchange via three processes:
- Token-to-Sentence Updating: updating the sentence representation with its tokens to grasp fine-grained semantics.
- Sentence-to-Sentence Updating: performing interaction between sentences on the PAG to get context-aware sentence representation.
- Sentence-to-Token Updating: using the context-aware sentence representation to update tokens, so that each token can also incorporate contextual information. The obtained context-aware token representation will be used for the computation of the next round.
After multiple rounds, the CT-Net obtains the contextual representation that fully combines sentence-level and token-level contextual semantics.
Our main contributions are two folds. First, we propose a novel CT-Net for IDRR, which builds the PAG to track closely related context for each sentence in the intricate discourse, and incorporates multi-grained contextual semantics via the crossgrained updating mechanism. Second, experiments on PDTB 2.0 demonstrate that the CT-Net gains better performance than a variety of approaches that roughly model the discourse context.
The input of the CT-Net is a paragraph  $P = (S_{1}, S_{2}, \dots, S_{n - 1}, S_{n})$ . Here,  $S_{n - 1}$  and  $S_{n}$  are the adjacent sentences to be classified, while  $S_{1}, \dots, S_{n - 2}$  are context with background information. Our goal is to identify the relation between  $S_{n - 1}$  and  $S_{n}$ . We firstly build a paragraph association graph (PAG) for  $P$  (Section 2.1), then employ the cross-grained updating mechanism on the PAG to extract the contextual representation of  $S_{n - 1}$  and  $S_{n}$  (Section 2.2). The contextual representation is then used for the final classification (Section 2.3).
2.1 Paragraph Association Graph
The CT-Net firstly converts the  $P$  into a PAG  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V}$  and  $\mathcal{E}$  are the sets of nodes and edges respectively. As shown in Figure 2, the PAG contains sentence nodes (blue) and token nodes (orange). Each token node is connected with its corresponding sentence node. We carefully design the edges between sentence nodes so that each sentence only connects the ones that are closely related to it. Specifically, there are three types of edges between sentence nodes in the PAG:
- Adjacency Edge (black edges). Adjacent sentences tend to carry important contextual information. Therefore, we add adjacency edges between the neighbors in the discourse.
- Co-reference Edge (purple edges). Sentences with co-reference associations tend to involve the same object and be highly related, so we add a co-reference edge between them.
- Lexical Chain Edge (green edges). Lexical chain tracks related words that run through the whole paragraph. Sentences containing the
$^{1}$ Code is available at: https://github.com/ yxuezhang/CTNet
same words or synonyms (except stop words) tend to involve the same topic, therefore, we add a lexical chain edge between them.
We give more details of the PAG in Section 3.2.
2.2 Cross-Grained Updating Mechanism
The CT-Net then extracts contextual representation of  $S_{n - 1}$  and  $S_{n}$  from the PAG  $\mathcal{G}$  through crossgrained updating mechanism, which is executed  $T$  rounds. At the  $t$ -th round, we denote the state of the  $i$ -th sentence node as  $g_{i}^{t}$ , and the state of the  $j$ -th token node of the  $i$ -th sentence as  $h_{i,j}^{t}$ . The states transition from the (t-1)-th to the  $t$ -th round consists of three computation processes: token-to-sentence updating, sentence-to-sentence updating and sentence-to-token updating. The first two processes are responsible for updating sentence nodes, while the last one is for updating token nodes.
Node Initialization. When  $t = 0$ , we initialize token nodes with the concatenation of char, GloVe (Pennington et al., 2014) and ELMo (Peters et al., 2018) embeddings. And the dimension is reduced:
$$
h _ {i, j} ^ {0} = x _ {i, j} = W \left[ x _ {i, j} ^ {\text {c h a r}}; x _ {i, j} ^ {\text {g l o v e}}; x _ {i, j} ^ {\text {e l m o}} \right] + b \tag {1}
$$
where  $W, b$  are parameters. The sentence node  $g_{i}^{0}$  is initialized as the average of its token nodes.
Token-to-Sentence Updating. This process updates the sentence state  $g_{i}^{t}$  with the token states of last round  $h_{i,j}^{t-1}$ . We employ Sentence-state LSTM (SLSTM) (Zhang et al., 2018) to achieve this. SLSTM is a novel graph RNN that converts a sentence into a graph with one global sentence node and several local word nodes, just like the sub-graph in the PAG (inside the dotted ellipse in Figure 2). At the  $t$ -th round, the hidden state of  $i$ -th sentence  $g_{i}^{t}$  is computed as follows:
$$
g _ {i} ^ {t} = \operatorname {S L S T M} _ {h \rightarrow g} \left(h _ {i, 0} ^ {t - 1}, h _ {i, 1} ^ {t - 1} \dots , h _ {i, | S _ {i} |} ^ {t - 1}, g _ {i} ^ {t - 1}\right) \tag {2}
$$
where  $\mathrm{SLSTM}_{h\to g}$  represents the process of updating the sentence state with token states by SLSTM, and its detailed equations are shown in Appendix A.  $|S_{i}|$  is the number of tokens in  $S_{i}$ .
Sentence-to-Sentence Updating. After merging token semantics, sentences further grasp sentence-level contextual semantics through the interaction between sentence nodes on the PAG. Since there are three types of edges, we employ Multi-Relational GCN (Schlichtkrull et al., 2018) to get contextual sentence representation  $c_{i}^{t}$  of  $S_{i}$ :
$$
c _ {i} ^ {t} = \sigma \left(W _ {g} g _ {i} ^ {t} + \sum_ {r \in R} \sum_ {k \in N _ {i} ^ {r}} \frac {1}{\left| N _ {i} ^ {r} \right|} W _ {r} g _ {k} ^ {t}\right) \tag {3}
$$
where  $W_{g}, W_{r}$  are model parameters.  $R$  is the set of edge types between sentence nodes.  $N_{i}^{r}$  denotes neighbours of the  $i$ -th sentence node of relation  $r$ , where  $r \in R$ .  $\sigma$  is the ReLU function.
Sentence-to-Token Updating. This process is for updating token states. It conveys the sentence-level contextual information  $c_{i}^{t - 1}$  to the token, which is also achieved by the SLSTM. At the  $t$ -th round, the hidden state of each token  $h_{i,j}^{t}$  is computed as follows:
$$
h _ {i, j} ^ {t} = \operatorname {S L S T M} _ {g \rightarrow h} \left(x _ {i, j}, c _ {i} ^ {t - 1}, h _ {i, j - 1} ^ {t - 1}, h _ {i, j} ^ {t - 1}, h _ {i, j + 1} ^ {t - 1}\right) \tag {4}
$$
where  $x_{i,j}$  is the initial token embedding. We show the detailed equations of  $\mathrm{SLSTM}_{g\to h}$  in Appendix A. Then, the obtained  $h_{i,j}^{t}$  is used for the token-to-sentence updating of the next round.
After  $T$  rounds, we get  $c_{n - 1}^T$  and  $c_{n}^{T}$  as the final contextual representations of  $S_{n - 1}$  and  $S_{n}$ , respectively, which fully combine token-level and sentence-level contextual semantics.
2.3 Classification Layer
After obtaining global contextual representations  $c_{n-1}^T$  and  $c_n^T$ , we use a one-layer BiLSTM (Hochreiter and Schmidhuber, 1997) to encode  $S_{n-1}$  into  $l_{n-1}$  by concatenating the last hidden states in two directions, and encode  $S_n$  into  $l_n$  in the same way.  $l_{n-1}$  and  $l_n$  are local representations without considering wider context. We then concatenate global and local features as follows:
$$
X _ {c l s} = \operatorname {c o n c a t} \left(l _ {n - 1}, l _ {n}, c _ {n - 1} ^ {T}, c _ {n} ^ {T}\right) \tag {5}
$$
$X_{cls}$  is then fed into a two-layer MLP (a fully-connected layer with ReLU activation followed by a softmax output layer) for classification.
Multi-Task Training. Following previous works (Dai and Huang, 2018; Nguyen et al., 2019), we apply multi-task learning to improve the performance. The main task is implicit discourse relation recognition (IDRR), while the auxiliary tasks are explicit discourse relation recognition (EDRR) and connective prediction (CP). These three tasks share the same encoder but use three different MLPs. The objective function is as follows:
$$
\begin{array}{l} L = - \alpha \sum_ {j = 1} ^ {C _ {i d r r}} y _ {i d r r} ^ {j} \log \widehat {y} _ {i d r r} ^ {j} - \beta \sum_ {j = 1} ^ {C _ {e d r r}} y _ {e d r r} ^ {j} \log \widehat {y} _ {e d r r} ^ {j} \\ - \gamma \sum_ {j = 1} ^ {C _ {c p}} y _ {c p} ^ {j} \log \widehat {y} _ {c p} ^ {j} \tag {6} \\ \end{array}
$$
where  $\alpha, \beta, \gamma$  are adjustable hyper-parameters.  $y_{idrr}$ ,  $y_{edrr}$  and  $y_{cp}$  are ground-truth labels of IDRR, EDRR and CP respectively, while  $\widehat{y}_{idrr}$ ,  $\widehat{y}_{edrr}$  and  $\widehat{y}_{cp}$  are corresponding predictions.  $C_{idrr}$ ,  $C_{edrr}$  and  $C_{cp}$  represent the number of classes of IDRR, EDRR, and CP respectively.
We conduct experiments on PDTB 2.0 (Prasad et al., 2008), which contains 16, 224 implicit instances and 18, 459 explicit instances. We perform one-vs-others binary classification and 4-way classification on 4 top-level discourse relations: comparison (Comp.), contingency (Cont.), expansion (Exp.), and temporal (Temp.). Following Pittler et al. (2009), we use sections 2-20 for training, sections 21-22 for test and sections 0-1 for validation. The metric is F1 score, and for 4-way classification, we calculate the macro-average F1 score.
Details of the PAG. We set the number of sentences to build PAGs as 6, and use zero padding when the text is less than 6 sentences. When building the PAG, we employ spaCy (https://spacy.io/) to identify co-reference chains, use simple matching to recognize the same words and use WordNet (Miller, 1995) to recognize synonyms. The WordNet covers  $59.38\%$  (7558/12632) training samples,  $59.05\%$  (699/1183) developing samples, and  $56.98\%$  (596/1046) testing samples. The average number of edges in the PAG is 11.
Details of Parameters and Training. For the node embedding initialization, we use 150-dimensional char embedding obtained by a CNN (Kim, 2014) with kernel window size of [1, 2, 3], 300-dimensional-GloVe embedding, and ELMo with 1024 dimension (the output of the second layer of BiLSTM). We reduce the dimension of node states as 512, so that the dimensions of SLSTM and MR-GCN are also 512. The iteration rounds of the cross-grained updating mechanism is set as 6. The size of the BiLSTM which is used to compute local features (Section 2.3) is 128. For multi-task learning, we set the  $\alpha$ ,  $\beta$ ,  $\gamma$  as 1.0, 0.5, 0.5. The learning rate is 0.001 with batch size of 64. The number of parameters of the CT-Net is about 16M. We use the F1 score as the criterion when manually tuning the hyper-parameter values. The
Model	Comp.	Cont.	Exp.	Temp.	4-way
NoContext	44.90	53.44	72.20	44.96	51.64
BiLSTM	45.25	53.75	72.66	45.38	51.02
H-LSTM	45.56	53.84	73.23	45.11	51.92
FCG-Net	46.42	54.75	72.43	45.57	52.45
CT-Net	46.86	55.63	73.71	45.90	53.11
Table 1: Comparison (F1, %) with models using different paragraph encoders (introduced in Section 3.3).
Row	Edge Type	Number of Sentences	F1 (%)
Adj.	Coref.	Lex.	n=4	n=6	n=8
1	X	✓	✓	✓	52.14
2	✓	X	✓	✓	52.58
3	✓	✓	X	✓	52.61
4	✓	✓	✓	✓	52.33
5	✓	✓	✓	✓	52.78
6	✓	✓	✓	✓	53.11
Table 2: Results of CT-Net with different PAG settings on 4-way classification.
whole model is trained end to end with the ADAM optimizer (Kingma and Ba, 2014) on two Tesla P40s with 24GB GPU memory, and the average runtime is about 6 hours.
3.3 Results and Discussion
Main Results (Table 1). We carefully design four baselines with different paragraph encoders for a full comparison: (1) "NoContext", the model only using BiLSTM to get local features without considering wider context. (2) "BiLSTM", the model using BiLSTM to encode the paragraph. (3) "H-LSTM", the model using hierarchical BiLSTM as paragraph encoder. (4) "FCG-Net", the model replacing the PAG in the CT-Net with a fully-connected graph (FCG). Except for the way of encoding paragraph, the other settings of these models are the same as the CT-Net. We can draw the following three conclusions. First, "NoContext" obtains the worst performance in most cases, demonstrating the necessity of using contextual representations. Second, the CT-Net gains better performance than models with sequential paragraph encoders "BiLSTM" and "H-LSTM", which proves the superiority of our graph-based CT-Net. The reason is that the CT-Net can track and model closely related context for sentences including long-distance ones. Third, replacing the PAG in the CT-Net with the FCG (FCG-Net) brings a quality drop, which proves the PAG effectively pick out appropriate context that benefits on sentence understanding. We also performed paired t-test between CT-Net and these 4 baselines. The CT-Net is significantly
Model	Comp.	Cont.	Exp.	Temp.	4-way
Chen et al. (2016)	40.17	54.76	-	31.32	-
Qin et al. (2017)	40.87	54.56	72.38	36.20	-
Lan et al. (2017)	40.73	58.96	72.47	38.50	47.80
Dai and Huang (2018)	46.79	57.09	70.41	45.61	51.84*
Lei et al. (2018)	43.24	57.82	72.88	29.10	47.15
Bai and Zhao (2018)	47.85	54.47	70.60	36.87	51.06
Nguyen et al. (2019)	48.44	56.84	73.66	38.60	53.00
Dai and Huang (2019)	-	-	-	-	52.89
Guo et al. (2020)	42.92	57.67	73.45	36.33	47.90
Ours	46.86	55.63	73.71	45.90	53.11
Table 3: Comparison (F1, %) with existing models on binary and 4-way settings. * means ensemble result.
Row	Multi-Task	F1 (%)
EDRR	CP
1	✓	✓	53.11
2	X	✓	51.32
3	✓	X	51.95
Table 4: Ablation study of multi-task learning on the 4-way classification.
better than all these baselines with  $\mathrm{p} < 0.05$ .
Analysis of the PAG (Table 2). The PAG contains three types of edges: adjacency edge (Adj.), co-reference edge (Coref.) and lexical chain edge (Lex.). To understand the impact of these edges, we conduct ablation experiments on 4-way classification. Rows 1-3 report the results of removing "Adj.", "Coref.", and "Lex." respectively. Removing "Adj." brings the biggest drop  $(0.97\%)$ , which reflects that the adjacency edge plays the most important role in the PAG. We also explore the impact of the number of sentences in the PAG. Rows 4-6 report the results. The CT-Net gains the best performance when the PAG contains 6 sentences, and modeling a longer paragraph of 8 sentences causes a decline. We hypothesize that modeling a paragraph this is too long may introduce some irrelevant context, resulting in a reduction in performance.
Comparison with Existing Systems (Table 3). Table 3 shows the comparison with existing systems. Our method outperforms other models on 4-way classification, and also gains the best performance on the binary classifications of temporal (Temp.) and expansion (Exp.).
Ablation Study of Multi-task Learning (Table 4). Following Dai and Huang (2018) and Nguyen et al. (2019), we utilize the explicit discourse relation recognition (EDRR) and connective prediction (CP) as auxiliary tasks to help implicit
discourse relation recognition (IDRR). We conduct ablation experiments of the two auxiliary tasks on 4-way classification (Table 4) to show their impact. Row 1 is the performance of the CT-Net. Rows 2-3 report the performance of removing the auxiliary task. As expected, the EDRR contributes more to the IDRR than the CP does, which is because that the EDRR is a more similar task with the IDRR.
We propose a novel graph-based Context Tracking Network (CT-Net) to model the context for implicit discourse relation classification. The CT-Net first converts the paragraph into the paragraph association graph (PAG), where each sentence tracks their appropriate context through different edges, then employs the cross-grained updating mechanism to combine sentence-level and token-level contextual information. Experiments on PDTB 2.0 demonstrate that the CT-Net captures more effective contextual information than carefully designed baselines with different context encoders.
Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 571-583, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1726-1735, Berlin, Germany. Association for Computational Linguistics.
Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 141-151, New Orleans, Louisiana. Association for Computational Linguistics.
Zeyu Dai and Ruihong Huang. 2019. A regularization approach for incorporating event knowledge and coreference relations into neural discourse parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2976-2987, Hong Kong, China. Association for Computational Linguistics.
Fengyu Guo, Ruifang He, Jianwu Dang, and Jr-Yan Wang. 2020. Working memory-driven neural networks with a novel knowledge enhancement paradigm for implicit discourse relation recognition. In AAAI 2020.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural Computation*, 9:1735-1780.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference on Learning Representations.
Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1299-1308, Copenhagen, Denmark. Association for Computational Linguistics.
Wenqiang Lei, Yuanxin Xiang, Yuwei Wang, Qian Zhong, Meichun Liu, and Min-Yen Kan. 2018. Linguistic properties matter for implicit discourse relation recognition: Combining semantic interaction, topic continuity and attribution. In Thirty-Second AAAI Conference on Artificial Intelligence.
Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1224-1233, Austin, Texas. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.