SlowGuess/NAACL_2022_OCR huggingface dataset:
Le Qi $^{1}$ , Yu Zhang $^{1*}$ , Qingyu Yin $^{1}$ , Guidong Zheng $^{2}$ , Junjie Wen $^{2}$ , Jinlong Li $^{2}$ , and Ting Liu $^{1}$
<sup>1</sup>Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China
$^{2}$ AI Lab of China Merchants Bank
{lqi, zhangyu, qyyin, tliu}  $@$  ir.hit.edu.cn
{zhengguidong, wenjunjiee, lucida} @cmbchina.com
Question matching is the task of identifying whether two questions have the same intent. For better reasoning the relationship between questions, existing studies adopt multiple interaction modules and perform multi-round reasoning via deep neural networks. In this process, there are two kinds of critical information that are commonly employed: the representation information of original questions and the interactive information between pairs of questions. However, previous studies tend to transmit only one kind of information, while failing to utilize both kinds of information simultaneously. To address this problem, in this paper, we propose a Full Information Transmission Network (FITN) that can transmit both representation and interactive information together in a simultaneous fashion. More specifically, we employ a novel memory-based attention for keeping and transmitting the interactive information through a global interaction matrix. Besides, we apply an original-average mixed connection method to effectively transmit the representation information between different reasoning rounds, which helps to preserve the original representation features of questions along with the historical hidden features. Experiments on two standard benchmarks demonstrate that our approach outperforms strong baseline models.
Question Matching (QM) aims to identify whether two questions have the same intent, which is widely applied in Question Answering (QA) applications such as community QA and intelligent customer services. Typically, QM is regarded as a semantic matching task (Hu et al., 2021). To correctly infer the relationship of a given question pair, there are two kinds of information that should be considered: the representation information of questions that captures the semantics of the texts, and the inter
active information between questions that contains critical hints for relationship reasoning.
For better detecting the relationship between question pairs, it's far from being enough to conduct only one single round of reasoning. Existing methods commonly resort to multiple interaction modules to do deep reasoning, where each module is generally composed of an encoding layer (can be omitted (Gong et al., 2018)) to update the representation information of questions and an interaction layer for capturing the interactive information between questions (Kim et al., 2019; Hu et al., 2021). In such a multi-round reasoning procedure, both the representation and interactive information in history rounds play a vital role in guiding the future inference. However, previous studies either only transmit the representation information (Kim et al., 2019) or only the interactive information (Gong et al., 2018), while failing to utilize both kinds of information simultaneously.
As shown in Figure 1 (i), when performing multi-round reasoning, if a model only transmits the representation information, the interactive information between questions will then be simply utilized to generate the representation of questions for future rounds. Consequently, the critical hints for relationship reasoning conveyed by interactive information are abandoned and cannot be directly used for future inferences. On the other side, if a model only transmits the interactive information, it is equivalent to conduct multi-round reasoning with only one single pass on question pairs, as shown in Figure 1 (ii). Admittedly, missing the representation information of original questions may lead to understanding deviation and thus bring cascading errors. Therefore, as shown in Figure 1 (iii), to better perform reasoning between question pairs, a desirable solution should be able to transmit both the representation and interactive information from historical rounds to the current round simultaneously.
To address the aforementioned problems, in this
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1431 - 1440 July 10-15, 2022 ©2022 Association for Computational Linguistics
(i) Transmit only the representation information
(ii) Transmit only the interactive information
(iii) Transmit both the representation and the interactive information
Figure 1: Comparison between different reasoning process of the relationship between questions S and T when transmitting different kinds of information.
paper, we propose a Full Information Transmission Network (FITN) that learns to transmit both the representation and interactive information between each round of reasoning. In particular, we propose a novel Memory-based Attention (Mem-Att) to transmit the interactive information between question pairs. In the Mem-Att, we maintain a global interaction matrix as a memory for keeping the interactive information and do inference on top of it. Compared with traditional attentions that calculate the alignment score directly, the proposed interaction matrix keeps rich interactive information and is more stable in the update process due to its redundancy. Thanks to the global interaction matrix, each round of inference could benefit from the historical interactive information and the whole reasoning procedure is progressive.
Meanwhile, to effectively transmit the representation information of questions, we introduce an interesting connection method, namely the Original-Average Mixed Connection (OA-mixed Connection). Instead of feeding only the hidden features from the last reasoning round, when performing reasoning at the current round, we regard both the hidden features and the original representation embeddings of questions as the input. Such a connection method enables our model an ability to explicitly utilize the entire rich information of original texts when inference. In addition, the OA-mixed Connection employs the average operation over hidden features from the last two rounds to build the input hidden feature for the current reasoning round. Compared with the residual connection (He et al., 2016) that treats the information in each round equally, the average connection pays more attention to the information in the nearer rounds, and thus brings better discrimination ability.
We evaluate our proposed method on the Quora and LCQMC benchmarks. Experimental results
show that FITN outperforms the non-pretrained baselines with considerable margins. Furthermore, compared with pre-trained models (small ones with comparable parameter sizes as FITN), our FITN also achieves better performance, which reveals the advantage of proposed method under resource-constrained conditions. All these illustrates the effectiveness of our method.
In sum, our major contributions are three-fold:
- We propose the Full Information Transmission Network (FITN) that can better utilize the historical information, capturing both the representation and interactive information of questions for question matching.
- We propose the memory-based attention for keeping and transmitting the interactive information and the original-average mixed connection to fully utilize the original embedding features of texts and historical hidden features.
- We evaluate the proposed FITN on two benchmark datasets, where considerable improvements are gained over strong baseline models.
In this section, we introduce our proposed full information transmission network (FITN) in detail. As shown in Figure 2, FITN comprises three modules: the embedding module, the interaction module and the prediction module. In FITN, we first embed each question in the embedding module, then do inference in the interaction module and finally predict their relationship in the prediction module.
We denote two input questions as  $S = \{s_1, s_2, \dots, s_m\}$  and  $T = \{t_1, t_2, \dots, t_n\}$  where  $s_i / t_j$  is the  $i^{th} / j^{th}$  token of question  $S / T$  and  $m / n$  is the token length of  $S / T$ .
Figure 2: Architecture of the FITN model.
2.1 Embedding Module
In the embedding module, we apply the word embedding along with the character embedding to embed tokens in each question. The character embedding is randomly initialized and then processed by a convolutional neural network (CNN) with a max-pooling operation. Formally, the final representation  $e_{s_i}$  of token  $s_i$  is calculated as follows:
$$
e _ {s _ {i}} = \left[ \operatorname {E m b} \left(s _ {i}\right); \operatorname {C h C o n v} \left(s _ {i}\right) \right] \tag {1}
$$
where  $[;]$  denotes the concatenation operation, Emb is the word embedding and ChConv is the character-level CNN. Each word in  $S$  and  $T$  is treated in the same procedure and then  $S$  and  $T$  can be represented as  $E_{S}\in \mathbb{R}^{m\times d_{e}}$  and  $E_{T}\in \mathbb{R}^{n\times d_{e}}$ .
2.2 Interaction Module
The interaction module is the core of our FITN, composed of N same-structured blocks for doing N rounds of inference. Each block contains 3 components: the encoding layer, the memory-based attention layer and the original-average mixed connection layer. We denote  $I_S^l$  and  $I_T^l$  as the inputs of the  $l^{th}$  block, where  $I_S^0 = E_S$  and  $I_T^0 = E_T$ .
2.2.1 Encoding Layer
We encode two questions through a Bi-LSTM encoder to extract the contextual representation of
each token in questions, shown as:
$$
H _ {S} ^ {l} = \operatorname {B i L S T M} ^ {1} \left(I _ {S} ^ {l}\right) \tag {2}
$$
$$
H _ {T} ^ {l} = \operatorname {B i L S T M} ^ {1} \left(I _ {T} ^ {l}\right) \tag {3}
$$
where  $H_{S}^{l}\in \mathbb{R}^{m\times d_{h}}$  and  $H_T^l\in \mathbb{R}^{m\times d_h}$  are the hidden representations of  $I_S^l$  and  $I_T^l$  in the  $l^{th}$  round, respectively.
2.2.2 Memory-based Attention Layer
As shown in Figure 3, we maintain a global interaction matrix for keeping and transmitting the interactive information in the memory-based attention (Mem-Att) layer. The global interaction matrix is treated as a memory, which keeps all the historical interactive information and will be updated when getting the new one. For each pair of tokens, we keep an interactive vector instead of an attention score in the global interaction matrix. The interactive vector keeps richer information and is more stable in the update process due to its redundancy.
Figure 3: Architecture of the Mem-Att.
In each round, we firstly update the global interaction matrix and then do attention based on this matrix. In this way, the interactive information in history can be transmitted into the current round and provides assistance on the soft-alignment and inference between the two questions.
Global Interaction Matrix Update The global interaction matrix is updated through two steps: current interaction matrix calculation and global interaction matrix combination.
Current Interaction Matrix Calculation The current interaction matrix in the  $l^{th}$  round  $M^l \in \mathbb{R}^{n \times m \times d_h}$  is calculated as follows:
$$
M ^ {l} = H _ {S} ^ {l} \odot H _ {T} ^ {l} \tag {4}
$$
For each pair of tokens  $s_i$  and  $t_j$  in the question  $S$  and  $T$ , the interaction vector  $M_{i,j}^{l} \in \mathbb{R}^{d_h}$  in  $M^l$  is calculated through the element-wise multiplication operation, shown as:
$$
M _ {i, j} ^ {l} = H _ {s _ {i}} ^ {l} \circ H _ {t _ {j}} ^ {l} \tag {5}
$$
where  $\circ$  is the element-wise multiplication.
Global Interaction Matrix Combination After that, we combine the current interaction matrix  $M^l$  and the global interaction matrix  $\bar{M}^{l - 1}$  in the previous round and feed the concatenation result of them into a fully-connected layer with a non-linear activation function as the global interaction matrix  $\bar{M}^l\in \mathbb{R}^{n\times m\times d_m}$  in the  $l^{th}$  round:
$$
\bar {M} ^ {l} = \left\{ \begin{array}{l l} F (M ^ {l}) & l = 0 \\ F ([ M ^ {l}; \bar {M} ^ {l - 1} ]) & l > 0 \end{array} \right. \tag {6}
$$
where  $\bar{M}_{i,j}^{l}\in \mathbb{R}^{d_{m}}$  in  $\bar{M}$  is calculated as:
$$
\bar {M} _ {i, j} ^ {l} = f \left(w _ {m} ^ {l} \left[ \bar {M} _ {i, j} ^ {l - 1}; M _ {i, j} ^ {l} \right] + b _ {m} ^ {l}\right) \tag {7}
$$
where  $[\cdot ]$  is vector concatenation across row,  $w_{m}^{l}\in$ $\mathbb{R}^{(d_h + d_m)\times d_m}$  and  $b_{m}^{l}\in \mathbb{R}^{d_{m}}$  correspond to the weight and bias respectively.
Attention over Interaction Matrix Next, we do inference and alignment through the global interaction matrix. We firstly adopt a dense-pooling method to extract an attention map from the global interaction matrix. More specifically, we utilize a fully-connected layer with a nonlinear function to convert each vector into the attention value. Each element  $Att_{i,j}^{l}$  in the attention map  $Att^{l} \in \mathbb{R}^{m \times n}$  is calculated as:
$$
A t t _ {i, j} ^ {l} = f \left(w _ {p} ^ {l} \bar {M} _ {i, j} ^ {l} + b _ {p} ^ {l}\right) \tag {8}
$$
where  $w_{p}^{l}\in \mathbb{R}^{d_{m}\times 1}$  and  $b_{p}^{l}\in \mathbb{R}$  correspond to the weight and bias, respectively. Then, the attentive representation  $A_{s_i}^l$  of  $s_i$  in the  $l^{th}$  round is weighted summed by  $H_{t_j}^l$ , where the weights are calculated by the softmax operation over  $Att_{i,j}^l$ :
$$
A _ {s _ {i}} ^ {l} = \sum_ {j = 1} ^ {n} \operatorname {s o f t m a x} \left(A t t _ {i, j} ^ {l}\right) H _ {t _ {j}} ^ {l} \tag {9}
$$
Finally, we calculate the average and the difference between the attentive representation  $A_{S / T}$  and the contextual representation  $H_{S / T}$ , concatenate the results with themselves together, and then feed the concatenation result into a fully-connected layer to get the outputs of the block.
$$
U _ {S} ^ {l} = \left[ H _ {S} ^ {l}; A _ {S} ^ {l}; \left(H _ {S} ^ {l} + A _ {S} ^ {l}\right) / 2; H _ {S} ^ {l} - A _ {S} ^ {l} \right] \tag {10}
$$
$$
U _ {T} ^ {l} = \left[ H _ {T} ^ {l}; A _ {T} ^ {l}; \left(H _ {T} ^ {l} + A _ {T} ^ {l}\right) / 2; H _ {T} ^ {l} - A _ {T} ^ {l} \right] \tag {11}
$$
$$
O _ {S} ^ {l} = f \left(w _ {f} ^ {l} U _ {S} ^ {l} + b _ {f} ^ {l}\right) \tag {12}
$$
$$
O _ {T} ^ {l} = f \left(w _ {f} ^ {l} U _ {T} ^ {l} + b _ {f} ^ {l}\right) \tag {13}
$$
where  $O_S^l \in \mathbb{R}^{m \times d_h}$ ,  $O_T^l \in \mathbb{R}^{n \times d_h}$ ,  $w_f^l \in \mathbb{R}^{4d_h \times d_h}$ , and  $b_f^l \in \mathbb{R}^{d_h}$  are the weight and bias respectively.
2.2.3 Original-Average Mixed Connection Layer
Finally, we transmit the representation information through the original-average mixed connectivity pattern (OA-mixed connection) in this layer. The question representation input to each round of inference can be divided into two parts: the original features from the initial embedding of questions and the hidden features extracted from previous inference rounds. Both of them play a vital role in each round of inference, where the original features can lead the model to make inference in the right direction, and the hidden features contain deeper contextual and interactive information. Besides, the hidden features can be seen as the information enhancement of the original features. Formally, the whole process can be shown as:
$$
I ^ {l} = \left\{ \begin{array}{l l} I _ {E} & l = 0 \\ \left[ I _ {E}; I _ {H} ^ {l} \right] & l > 0 \end{array} \right. \tag {14}
$$
where  $I^l \in \mathbb{R}^{m \times (d_e + d_h)}$  ( $l > 0$ ) is the  $l$ -th round input,  $I_E$  is the initial embedding, and  $I_H^l$  is the  $l$ -th round hidden input, calculated as:
$$
I _ {H} ^ {l} = \left\{ \begin{array}{l l} O ^ {0} & l = 1 \\ \left(O ^ {l - 1} + I _ {H} ^ {l - 1}\right) / 2 & l > 1 \\ = \frac {O ^ {l - 1}}{2} + \dots + \frac {O ^ {1}}{2 ^ {l - 1}} + \frac {O ^ {0}}{2 ^ {l - 1}} \end{array} \right. \tag {15}
$$
where  $O^l$  are the hidden outputs of the interaction module before the average connection.
Here, instead of the residual connection, we apply the average connection to capture the hidden features. Compared with the residual connection that treats the information in each round equally, the average connection pay more attention to the information in the nearer rounds. Besides, the residual connection's summation operation may cause the variance of the vectors in the hidden part to go larger as the layers deepen. In comparison, the average connection can balance the variance between the two parts of the question representation.
2.3 Prediction Module
The final representations of the two questions in the interaction module are the last block's next inputs  $I_S^{N + 1}$  and  $I_T^{N + 1}$ . To extract a proper representation for each question, we apply the max-pooling operation over them, i.e.:
$$
V _ {S} = \max  \left(I _ {S} ^ {N + 1}\right) \tag {16}
$$
$$
V _ {T} = \max  \left(I _ {T} ^ {N + 1}\right) \tag {17}
$$
Table 1: Experimental results on the Quora and LCQMC datasets. Para. denotes the number of parameters. The evaluation metric of Quora is accuracy  $(\%)$ , and that of LCQMC is accuracy  $(\%)$  and F1. The results are average scores using 5 different seeds along with the standard deviation.
Type	Model	Para.	Quora	LCQMC
Non-pretrained	BiMPM(Wang et al., 2017)	1.6m	88.2	83.3/84.9
DIIN (Gong et al., 2018)	4.4m	89.1	-/-
CSRAN (Tay et al., 2018)	-	89.2	-/-
RE2 (Yang et al., 2019)	2.8m	89.4	-/-
Enhanced-RCNN (Peng et al., 2020)	7.3m	89.5	-/-
TIM-W (Zhou et al., 2020)	-	89.6	-/-
DRCN (Kim et al., 2019)	6.7m	90.2	-/-
GMN (Chen et al., 2020)	-	-	84.6/86.0
LET (Lyu et al., 2021)	-	-	84.8/86.1
COIN (Hu et al., 2021)	6.5m	89.4	85.6/86.5
Pre-trained	AlBERT-tiny (Lan et al., 2020)	4.1m	-	85.3/86.3
BERT-tiny (Turc et al., 2019)	4.4m	87.2	-/-
BERT-mini (Turc et al., 2019)	11.3m	88.8	-/-
AlBERT-base (Lan et al., 2020)	11.7m	90.0	86.3/87.0
Ours	FITN	2.5m	90.6±0.1	86.0±0.5/87.1±0.4
where  $V_{S}, V_{T} \in \mathbb{R}^{d_{h} + d_{e}}$  and max extracts the maximum value in each column of the inputs.
Finally, we concatenate  $V_{S}$  and  $V_{T}$  to get the feature vector  $V$  and feed the feature vector  $V$  into a two-layer feed-forward network with one hidden layer and one softmax layer to make the prediction.
We evaluate our FITN on two QM benchmarks: the Quora (English dataset) (Iyer et al., 2017) and LCQMC (Chinese dataset) (Liu et al., 2018). The Quora dataset contains over  $400\mathrm{k}$  question pairs collecting from Quora, an English community question answering (cQA) website, and the data splits  $(380\mathrm{K} / 10\mathrm{K} / 10\mathrm{K})$  are provided in BIMPM (Wang et al., 2017). The LCQMC collects over  $260\mathrm{k}$  question pairs from a Chinese cQA website called BaiduKnows  $(240\mathrm{K} / 8\mathrm{K} / 12\mathrm{K})$ .
In the original FITN, we initialize the word embedding with 300d Fasttext vectors (Bojanowski et al., 2017) for the English task and 300d Word2Vec vectors trained in Baidu Encyclopedia (Qiu et al., 2018) for the Chinese task, respectively. We randomly initialize the character embedding with a 25d vector and extract a 50d character representation by CNN. Then, we conduct three rounds of inference and set the hidden size of each layer to 100d in the interaction module. Finally, we set
500 hidden units for the 2-layer FFN in the prediction module. We apply an Adam (Kingma and Ba, 2015) optimizer with a learning rate of 1e-3. We train 100 epochs on the Quora dataset and 20 epochs on the LCQMC dataset. We run 5 times with 5 different randomly selected seeds and report the mean value with the standard deviation selected according to the best performance in the development set.
The main experimental results are shown in Table 1. We compare our FITN with non-pretrained models at first. In particular, we employ the baselines including: 1) DIIN(Gong et al., 2018): a CNN-based model that employs a DenseNet on the interaction matrix; 2) RE2 (Yang et al., 2019): a CNN-based model with the augmented residual connection; 3) Enhanced RCNN (Peng et al., 2020): a model that encodes sentences by multi-layer CNNs and adopts the attention-based RNNs for relationship inference; 4) TIM-W (Zhou et al., 2020): a model based on deep mutual information estimation; 5) DRCN (Kim et al., 2019): a co-attention BiLSTM model with dense-connection; 6) GMN (Zhou et al., 2020): a neural graph matching model; 7) LET (Lyu et al., 2021): a transformer-based model that employs the external linguistic knowledge derived from Graph Attention Networks; and 8) COIN (Hu et al., 2021): A CNN-based model
with a deep context-aware cross-attention based interaction module.
We can see that our model outperforms all these baselines on the two benchmarks. More specifically, our model beats DIIN because we can keep updating the representation information based on the historical representation information during iterations. Compared with DRCN, our FITN utilizes the historical interactive information for inference and in return acquires performance improvements with fewer inference rounds. Besides, the historical interactive information can also benefit our model on deeper inference. Therefore, the performance of our model is unsurprisingly better than RE2.
In addition, to further verify the effectiveness of our FITN under restricted computing resources, we compare our FITN with 4 publicly available tiny pre-trained models, which are distilled from large pre-trained models (BERT-tiny and BERT-mini (Turc et al., 2019) that are distilled from BERT-base (Devlin et al., 2019)) or directly pretrained by large-scale datasets (AlBERT-tiny and AlBERT-base (Lan et al., 2020)). As shown in Table 1, our model can achieve competitive or even better performance than pre-trained models with similar model size. It demonstrates that our FITN can be a desirable choice compared with pretrained models in resource-constrained scenarios.
In this subsection, we firstly verify the effectiveness of our proposed Mem-Att and OA-mixed connection, then show the impact of inference rounds on model performance, and finally further analyze the Mem-Att by a statistical analysis and a case study.
3.3.1 Effectiveness of the Mem-Att
We compare Mem-Att with three attention mechanisms to verify the ability of Mem-Att to maintain richer interactive information and leverage historical interactive information to aid future inference, containing 1) the scaled dot product attention (DotAtt); 2) the scaled weighted dot product attention (wDot-Att); and 3) the interactive attention (InterAtt), a variance of Mem-Att, which is only based on the current interaction matrix. The functions of these attentions are shown as following:
$$
A t t = \left\{ \begin{array}{l l} P o o l _ {a t t} \left(F \left(M ^ {c}\right)\right) & \text {I n t e r - A t t} \\ \frac {S T ^ {T}}{\sqrt {d}} & \text {D o t - A t t} \\ \frac {S W T ^ {T}}{\sqrt {d}} & \text {w D o t - A t t} \end{array} \right. \tag {18}
$$
Table 2: Comparison experiments about different attention mechanisms on the Quora dataset.
Attentions	Dev Acc.	Test Acc.
Dot-Att	88.4	88.0
wDot-Att	88.8	88.3
Inter-Att	90.4	90.1
Mem-Att	90.8	90.6
Table 3: Comparison experiments about different connectivity patterns on the Quora dataset.
Patterns	Dev Acc.	Test Acc.
Direct	90.3	90.0
Dense	90.4	90.3
Residual	90.5	90.2
OA-mixed	90.8	90.6
where  $d$  is the dimension of the question representation,  $S\in \mathbb{R}^{d\times m}$ $T\in \mathbb{R}^{d\times n}$  and  $W\in \mathbb{R}^{d\times d}$
The comparison results are shown in Table 2. With an intuition that the 3D interaction matrix can keep richer interactive information than the 2D attention map, the performance of the Int-Att is unsurprisingly better than those of the wDot-Att and the Dot-Att, which demonstrates that richer interactive information can bring benefit to the model on conducting more proper inference. Furthermore, the performance of the Mem-Att is better than that of the Int-Att, which reflects that the historical interactive information can provide assistance on the current and the future inference.
3.3.2 Effectiveness of the OA-mixed Connection