SlowGuess/ACL_2020_OCR huggingface dataset:
Ruifang He $^{1,2,3}$ , Jian Wang $^{1,2}$ , Fengyu Guo $^{1,2*}$ , and Yugui Han $^{1,2}$
$^{1}$ College of Intelligence and Computing, Tianjin University, Tianjin, China  
 $^{2}$ Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin, China  
 $^{3}$ State Key Laboratory of Cognitive Intelligence, iFLYTEK, China {rfhe, jian_wang, fengyuguo, yghan}@tju.edu.cn
Implicit discourse relation recognition is a challenging task due to the lack of connectives as strong linguistic clues. Previous methods primarily encode two arguments separately or extract the specific interaction patterns for the task, which have not fully exploited the annotated relation signal. Therefore, we propose a novel TransS-driven joint learning architecture to address the issues. Specifically, based on the multi-level encoder, we 1) translate discourse relations in low-dimensional embedding space (called TransS), which could mine the latent geometric structure information of argumentrelation instances; 2) further exploit the semantic features of arguments to assist discourse understanding; 3) jointly learn 1) and 2) to mutually reinforce each other to obtain the better argument representations, so as to improve the performance of the task. Extensive experimental results on the Penn Discourse TreeBank (PDTB) show that our model achieves competitive results against several state-of-the-art systems.
Discourse relation describes how two adjacent text units (e.g., clauses, sentences, and larger sentence groups) are connected logically to one another. A discourse relation instance is usually defined as a connective taking two arguments (as  $\text{Arg1}$  and  $\text{Arg2}$ , respectively). Implicit discourse relation recognition without explicit connectives (Pitler et al., 2009) is still a challenging problem of discourse analysis, which needs to infer the discourse relation from a specific context. It is beneficial to many downstream natural language processing (NLP) applications, such as machine translation (Meyer and Popescu-Belis, 2012) and text summarization (Gerani et al., 2014).
The existing neural network-based models have shown great success in recognizing implicit discourse relations. It mainly includes 1) Basic neural networks (Braud and Denis, 2015; Zhang et al., 2015; Liu et al., 2016) can learn the dense vector representations of discourse arguments, which can capture the semantic information to some extent. Further studies exploit different attention or memory mechanisms (Liu and Li, 2016; Zhang et al., 2016) to capture the critical information of argument pairs. 2) Complex neural models (Chen et al., 2016; Lei et al., 2017; Guo et al., 2018) utilize gated relevance networks or neural tensor networks to capture the deeper interactions between two discourse arguments. 3) Joint learning architectures (Qin et al., 2017; Bai and Zhao, 2018; Xu et al., 2019) exploit implicit connective cues, different granularity of text, or topic-level relevant information to improve the discourse relation prediction. However, these approaches still have the following drawbacks: 1) do not make full use of the annotated discourse relation signal to explore the argument-relation features; 2) neglect the extra information in the low-dimensional continuous embedding space, i.e., the direction or structure information of the vectors.
Notice that Translating Embeddings (TransE) is a method for the prediction of entities' missing relations in knowledge graphs. Bordes et al. (2013) model relations by interpreting them as translating operation not on the graph structure directly but in a learned low-dimensional embedding of the knowledge graph entities: if  $(h_e, l_e, t_e)$  holds, then the embedding of the tail entity  $t_e$  should be close to the embedding of the head entity  $h_e$  plus some vector that depends on the relation  $l_e$ . Similar to the entity relation extraction, our task aims to identify the semantic relations between two arguments (i.e., sentences).
Inspired by TransE, we design a new method
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 139-148 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(TransS), which translates discourse relations in sentence embedding spaces to mine the argument-relation features. Intuitively, these features reflect the latent geometric structure among the arguments and their discourse relation by performing the algebraic operation, and the argument-relation instances with the same discourse relation may have similar direction and position information in the embedding space. Therefore, we propose a novel TransS-driven joint learning neural network framework that leverages the latent geometric structure information of argument-relation instances, in addition to using the semantic features to improve the comprehension of discourse argument. Among them, we adopt a multi-level encoder to further enrich the argument representations, which could obtain the deeper semantics of discourse.
- Propose a novel TransS-driven joint learning architecture, including the latent geometric structure information learning (GSL) and semantic feature learning (SFL);
- Design TransS approach to translate discourse relations in low-dimensional embedding space from the sentence-level perspective, which could induce the geometric structure of argument-relation instances to some extent;
- Employ the mutual reinforcing between the GSL and SFL to optimize the argument representations: 1) the GSL adopts its geometric structure clues to facilitate the SFL; 2) the SFL utilizes its semantic cues to improve the learning capability of GSL;
- The experimental results on the PDTB demonstrate the effectiveness of our model.
2 The Proposed Model
The implicit discourse relation recognition task is usually formalized as a classification problem. In this section, we give an overview of the TransS-driven joint learning framework, which consists of four parts: embedding layer, multi-level encoder, latent geometric structure learning, and semantic feature learning, as shown in Figure 1.
2.1 Embedding Layer
In order to model two discourse arguments with neural networks, we transform the one-hot repre
sentations of arguments and their discourse relation into the distributed representations. Formally, the embedding layer could be seen as a simple projection layer where the word embedding is achieved by lookup table operation according to the indexes. All words of two arguments  $Arg1$ ,  $Arg2$ , and their relation will be mapped into low dimensional vector representations, which are taken as the input of our model.
2.2 Multi-level Encoder
To enrich the discourse argument representations, we exploit multi-level encoder shown in Figure 2 to learn the argument representations at the different levels. Particularly, the higher-level states of multi-level encoder could capture context-dependent aspects of words while the lower-level states could model aspects of syntax (Peters et al., 2018). The multi-level encoder is composed of stacked encoder layers.
2.2.1 Encoder Layer
Referring to the previous work, we implement the bidirectional LSTM (BiLSTM) neural network to model the argument sequences, which could preserve both the historical and future information in forward and reverse directions. Therefore, we can obtain two representations  $\vec{h}_t$  and  $\vec{h}_t$  at each time step  $t$  of the sequence. Then we concatenate them to get the intermediate state  $h_t = [\vec{h}_t, \vec{h}_t]$ .
Attention Controller. Due to the limitations of treating each word equally in the general representations, we use attention mechanism to point out the words particularly useful for our task. Let  $H$  be the matrix consisting of output vectors  $[h_1, h_2, \dots, h_n]$  of the last layer produced, where  $n$  is the length of the argument. The new representation  $\tilde{h}$  of the argument is formed by a weighted sum of the output vectors:
$$
M = \tanh  (H), \tag {1}
$$
$$
\alpha = \operatorname {s o f t m a x} \left(w ^ {T} M\right), \tag {2}
$$
$$
\tilde {h} = H \alpha^ {T}. \tag {3}
$$
where  $H\in \mathbb{R}^{n\times d}$ $d$  is the dimension of word embedding,  $w$  is a parameter vector. Then we could obtain the argument representation with important information from Eq. (4) for the next step.
$$
h ^ {*} = \tanh  (\tilde {h}) \tag {4}
$$
Figure 1: TransS-driven joint learning architecture of our proposed model.
Figure 2: The illustration of multi-level encoder.
2.2.2 Pooling Layer
Finally, we can receive the overall argument representations by averaging pooling operation for the word embedding sequence, defined as:
$$
h _ {A r g} ^ {*} = \frac {1}{n} \sum_ {i = 1} ^ {n} h _ {i} ^ {* (m)} \tag {5}
$$
where  $h_{Arg}^{*}$  is the argument representation,  $h_{i}^{*(m)}$  is the representation of the  $i$ -th word in the word embedding sequence of the  $m$ -th encoder layer,  $n$  is the number of words in an argument.
2.3 Latent Geometric Structure Learning
TransE, as a model for learning low-dimensional embeddings of entities, is to enforce the structure of embedding space in which different relations between entities of different types may be represented by translation (Bordes et al., 2013). Discourse relation recognition and entity relation extraction are
similar to some extent. Intuitively, the argument-relation instances with the same discourse relation may also have similar direction and position information in embedding space. However, discourse argument embedding is a sentence-level representation, which is different from the reuse of entities in other sentences, and more diverse and complex than entity representation. Therefore, we design TransS, a method which models discourse relations by interpreting them as translations operating in the low-dimensional embedding space from the sentence perspective. Moreover, it could mine the latent geometric structure of argument-relation instances. Specifically, to define two arguments as head vector  $h_s$  and tail vector  $t_s$  respectively, their annotated relation signal as relation vector  $r_s$ , the latent geometric structure is reflected by  $h_s + r_s \approx t_s$ , their score function is defined as follows:
$$
d _ {s} \left(h _ {s}, t _ {s}\right) = \left\| h _ {s} + r _ {s} - t _ {s} \right\| _ {2} ^ {2}. \tag {6}
$$
where  $h_s, t_s$  denote the representations of  $Arg1$  and  $Arg2$  respectively;  $r_s \in \mathbb{R}^d$  is the embedding of discourse relation and  $d$  is the dimension of word embedding.
GSL Loss. Under the framework of TransS, given a training set  $T$  of triplets  $(h_s, r_s, t_s)$  composed of two arguments  $h_s, t_s \in V$  (the set of sentence vectors) and a relation  $r_s \in R$  (the set of relation), our model would learn the embeddings of the words in arguments and the discourse relation. The GSL
loss function is defined as:
$$
\begin{array}{l} \mathcal {L} _ {G S L} = \sum_ {(h _ {s}, r _ {s}, t _ {s}) \in T} \sum_ {(h _ {s} ^ {\prime}, r _ {s}, t _ {s} ^ {\prime}) \in T _ {(h _ {s}, r _ {s}, t _ {s})} ^ {\prime}} [ \gamma + d _ {s} (h _ {s} \\ \left. + r _ {s}, t _ {s}\right) - d _ {s} \left(h _ {s} ^ {\prime} + r _ {s}, t _ {s} ^ {\prime}\right) ] _ {+} + \lambda_ {G S L} \| \theta \| _ {2} ^ {2}. \tag {7} \\ \end{array}
$$
where  $[\cdot ]_{+}$  denotes the positive instances,  $\gamma >0$  is a margin hyper-parameter, and the set of negative triplets, constructed according to Eq.(8), in which the head or tail is replaced by a random argument vector (but not simultaneously).  $\theta$  denotes the other parameters of the network.  $L_{2}$  regularization is used to penalize the size of all parameters for preventing overfitting, weighted by  $\lambda_{GSL}$
$$
\begin{array}{l} T _ {\left(h _ {s}, r _ {s}, t _ {s}\right)} ^ {\prime} = \left\{\left(h _ {s} ^ {\prime}, r _ {s}, t _ {s}\right) \mid h _ {s} ^ {\prime} \in V \right\} \cup \tag {8} \\ \{(h _ {s}, r _ {s}, t _ {s} ^ {\prime}) | t _ {s} ^ {\prime} \in V) \}. \\ \end{array}
$$
By optimizing the GSL loss, we could obtain the latent geometric structure information about argument-relation instances. Different from TransE, we could not directly utilize TransS to recognize discourse relations, for that each argument could not be reused in discourse. Therefore, we exploit TransS to mine the latent geometric structure information and further guide the semantic feature learning.
2.4 Semantic Feature Learning
The new argument representations  $(h_{Arg1}^{*}, h_{Arg2}^{*})$  with latent geometric structure information learned by the GSL are as inputs of the semantic feature learning (SFL). The  $h_{Arg1}^{*}$  (i.e.,  $h_s$ ) and  $h_{Arg2}^{*}$  (i.e.,  $t_s$ ) are obtained from the multi-level encoder. We further stack a softmax layer upon the representations:
$$
y = f \left(W _ {f} \left[ \begin{array}{l} h _ {A r g 1} ^ {*} \\ h _ {A r g 2} ^ {*} \end{array} \right] + b _ {f}\right). \tag {9}
$$
where  $f$  is the softmax function,  $W_{f}\in \mathbb{R}^{C\times 2d}$ ,  $b_{f}\in \mathbb{R}^{C}$  are the weights and bias term respectively,  $d$  denotes the dimension of word embedding and  $C$  denotes the number of relation classes.
SFL Loss. Under the framework of basic neural networks for our task, given training set  $T$ , two argument vectors  $h_s, t_s$  in the triplet  $(h_s, r_s, t_s)$  are concatenated to a new sentence vector during the training process, and then the generated vector is used for relation recognition. The SFL loss is a cross-entropy style shown as:
$$
\mathcal {L} _ {S F L} = - \sum_ {j = 1} ^ {C} y _ {j} \log \left(\hat {y} _ {j}\right) \tag {10}
$$
where  $y$  is the one-hot representation of the groundtruth relation;  $\hat{y}$  is the predicted probabilities of relations;  $C$  is the number of relation class.
2.5 Joint Learning
After obtaining the new representations  $Arg1$  as head vector  $h_s$ ,  $Arg2$  as tail vector  $t_s$ , and the relation vector  $r_s$ , our model is trained using joint learning mechanism. The goal of our model is to minimize the loss function (Eq.(11))
$$
\mathcal {L} = \mathcal {L} _ {G S L} + \lambda \mathcal {L} _ {S F L}. \tag {11}
$$
where,  $\mathcal{L}_{GSL}$  and  $\mathcal{L}_{SFL}$  are from Eq.(7) and (10), respectively;  $\lambda$  is the trade-off parameter controlling the balance between GSL and SFL.
Our model jointly learns the GSL and SFL to optimize the argument representations. On the one hand, the GSL maps the discourse relation between two arguments to the low-dimensional embedding space and obtains the vectors  $h_s, r_s, t_s$  with geometric structure information to constrain the SFL. On the other hand, the SFL alternately optimizes the discourse representations and provides the necessary semantic clues for geometric structure information mining. Generally, the GSL and SFL reinforce with each other, and finally get the better argument representations containing the semantics and the latent geometric structure information of argument-relation.
The PDTB 2.0, a large scale corpus annotated on 2,312 Wall Street Journal articles, is utilized for all experiments. It contains three hierarchies: Level-1 Class, Level-2 Type, and Level-3 Subtype. We focus on the first level, which contains four classes: Comparison (Comp.), Contingency (Cont.), Expansion (Exp.), and Temporal (Temp.). As (Rutherford and Xue, 2014), we use Sections 2-21 as the training set, Section 22 as the development set, Section 23 as the test set.
Relation	Train	Dev	Test
Comp.	1945	196	152
Cont.	3242	284	272
Exp.	6794	646	546
Temp.	709	61	79
Total	12690	1187	1049
Table 1: The statistical distribution of PDTB.
All the arguments are padded at the same length of 100. Word embedding is randomly initialized by uniformly distributed samples [-0.1, 0.1] with 300-dimension. The learning rate is set to 0.001, the batch size is 128, and the number of iteration is 100. For the GSL, the margin of loss is set to 0.5, the trade-off parameter  $\lambda$  in Eq.(11) is set to 1.0, and we use  $L_{2}$  distance as dissimilarity; For the SFL, the sizes of the input and the hidden layer of the BiLSTMs are both 300; we choose three encoder layers, and set the dimension of pre-trained embeddings from ELMo (Peters et al., 2018) to 300.
3.3 The Comparison Models
3.3.1 The State-of-the-art Systems
To validate the effectiveness of our model, we select some state-of-the-art systems from the following three aspects to compare with our model:
- Discourse Argument Representation
1) Ji2015: Ji and Eisenstein (2015) computed distributed representations for each discourse argument by composition up the syntactic parse tree.
2) Zhang2015: Zhang et al. (2015) proposed pure neural networks with three different pooling operations to learn shallow representations in tasks.
3) Liu2016a: Liu and Li (2016) combined attention mechanism and external memory to focus on specific words that helps determine discourse relations.
4) Lan2017: Lan et al. (2017) designed an attention-based neural network for learning discourse argument representations and a multi-task framework for learning knowledge from annotated and unannotated corpora.
- Complex Neural Models
5) Chen2016: Chen et al. (2016) adopted a gated relevance network to capture interaction information between two arguments to enhance relation recognition.
6) Qin2016: Qin et al. (2016a) adopted context-aware character-enhanced embeddings to address implicit discourse relation recognition task.
7) Lei2017: Lei et al. (2017) devised the Simple Word Interaction Model (SWIM) to learn the interactions between word pairs.
8) Dai2018: Dai and Huang (2018) modeled interdependencies between discourse units as well as discourse relation continuity and patterns, and pre
dict a sequence of discourse relations in a paragraph.
- Joint Learning
9) Liu2016b: Liu et al. (2016) designed related discourse classification tasks specific to a corpus, and proposed a novel Convolutional Neural Network embedded multi-task learning system to synthesize these tasks by learning both unique and shared representations for each task.
10) Bai2018: Bai and Zhao (2018) employed different grained text representations, including character, subword, word, sentence, and sentence pair levels, and transferred the knowledge from the implicit connectives to support discourse relation prediction.
3.3.2 The Ablation Methods
In order to validate the effectiveness of each component of our model, we present the following ablation methods:
- Baseline (Including SFL) We use three encoder layers to encode the argument pairs separately, then concatenate them together, and feed them to the SFL module for relation recognition.
- +GSL We encode two arguments based on the Baseline, and then feed them into GSL and SFL modules, respectively. Finally, we use the two modules to help recognize the discourse relation.
- +ELMo We utilize the Baseline to receive the argument representations, and then we use the pre-trained ELMo vector to enhance the argument representations. Finally, we feed them to the SFL module for relation recognition.
- +GSL & ELMo (Ours) We feed the two argument representations, encoded by the Baseline and enhanced by the pre-trained ELMo vector, into GSL and SFL modules, respectively. And then, we utilize the integrated representation to recognize the discourse relation.
3.4 Results and Discussion
Consistent with previous studies, we choose  $F_{1}$  score and accuracy as evaluation metrics. For binary classification, the result is computed by  $F_{1}$  score, and for 4-way classification, the result is computed by macro average  $F_{1}$  score.
Model	Comp.	Cont.	Exp.	Temp.	4-way	Acc.
Ji2015	35.93	52.78	-	27.63	-	-
Zhang2015	33.22	52.04	69.59	30.54	-	-
Liu2016a	32.13	46.09	69.88	31.82	44.98	57.27
Lan2017	40.73	58.96	72.47	38.50	47.80	57.39
Chen2016	40.17	54.76	-	31.32	-	-
Qin2016	38.67	54.91	80.66	32.76	-	-
Lei2017	40.47	55.36	69.50	35.34	46.46	-
Dai2018	37.72	49.39	67.45	40.70	48.82	59.75
Liu2016b	39.86	54.48	70.43	38.84	46.29	57.57
Bai2018	47.85	54.47	70.60	36.87	51.06	-
Ours	47.98	55.62	69.37	38.94	51.24	59.94
Table 2:  ${F}_{1}$  score  $\left( \% \right)$  and Accuracy(Acc., %) of different comparison models on binary and 4-way classification.
Model	Comp.	Cont.	Exp.	Temp.	4-way	Acc.
Baseline	32.32	49.53	65.91	34.86	46.46	54.02
+ GSL	44.88	53.17	67.91	37.38	48.91	57.65
+ ELMo	46.85	54.57	68.44	38.71	50.07	58.89
+ GSL & ELMo (Ours)	47.98	55.62	69.37	38.94	51.24	59.94
Table 3:  $F_{1}$  score  $(\%)$  and Accuracy(Acc.,  $\%$  ) of ablation models on binary and 4-way classification.
3.4.1 Comparison with the state-of-the-art Systems
Table 2 shows the results of the compared state-of-the-art systems on binary and 4-way classification. We could make the following observations:
- Overall, i) our model achieves state-of-the-art performance, i.e., the  $F_{1}$  score and accuracy are 51.24% and 59.94% on the 4-way classification, respectively; ii) the results of binary classification are keeping a similar tendency with the 4-way classification. In particular, our model gains the best  $F_{1}$  score on Comparison relation. The main reasons may be that the instances with different discourse relations have different directions and position (geometric structure) features in the low-dimensional continuous embedding space, and the Comparison instances have more obvious indicative structure features.
- Comparing our model with Chen2016 and Lei2017, the  $F_{1}$  scores of our model are higher than those of the latter two. It proves that our model is better than the two methods only considering the content interactions, since we jointly leverage the geometric structure information and the semantic information
of the argument-relation instances to obtain deeper interactions.
- In the comparison models, Bai2018 with joint learning framework achieves the best performance, which illustrates that jointly utilizing the discourse relation and the implicit connectives are helpful to the task. Moreover, the performance of our model is better than that of Bai2018. It not only indicates that the effectiveness of joint learning, but also proves considering the geometric structure is beneficial to our task.
3.4.2 Ablation Models
For the ablation models, we can make the observations from Table 3:
Overall:1) Our model gains state-of-the-art performance than that of the other ablation models. This demonstrates that the geometric structure information could enrich the argument representation and promote implicit discourse relation recognition. 2) All models have a higher  $F_{1}$  values on the Expansion relation than those of the other relations. The unbalanced data may cause that.
GSL: The  $F_{1}$  score of our model using the GSL module is  $48.91\%$ , higher than the performance of Baseline. In addition, compared with ELMo,
(a) without geometric structure features.
(b) with geometric structure features.
Figure 3: Visualization of the interaction information of argument representation.
although the performance of GSL does not exceed ELMo's, GSL obtain comparable results. This manifests that the two modules (GSL and SFL) could reinforce with each other, which utilizes the geometric structure information by the algebraic operation. Moreover, we exploit the geometric structure clues to augment the semantic understanding of discourse from a new aspect, which is different from the ELMo only focusing on the semantic information of the text itself.
ELMo: The third row of Table 3 is the result of our model, which only uses the pre-trained ELMo vector to enhance argument representations. The  $F_{1}$  score and accuracy are  $50.07\%$  and  $58.89\%$ , respectively, which achieve  $3.61\%$  and  $4.87\%$  improvements than those of the Baseline. It verifies that ELMo, as pre-trained contextualized word embeddings, could contain more contextual information.
GSL & ELMo: Compared with ELMo, GSL & ELMo gains better performance, which demonstrates that inducing spatial geometry structure information based on argument enhancement could understand the semantics of discourse better.
3.4.3 Impact of TransS