SlowGuess/NAACL_2022_OCR huggingface dataset:
To illustrate the advantage of the OA-mixed connection, we compare our method with the following connective patterns: 1) residual connection (He et al., 2016); 2) dense connection (Huang et al., 2017), and 3) direct connection that directly treats the output of the previous round as the input.
As shown in Table 3, the direct connection unsurprisingly performs worst. These results show that the historical representation information provides benefits for the current round of inference and it is critical to design advanced connectivity patterns to effectively transmit important information between different reasoning rounds. Moreover, our OA-mixed connection beats both the residual
connection and the dense connection. We attribute it to the fact that our method can preserve the entire information of original texts. Meanwhile, the average connection we proposed can help the model to focus more on the information conveyed by the surrounding reasoning rounds. All these bring rich information and helpful hints to determine the relationships between the question pair.
Figure 4: The accuracy curve for different rounds of inference on the Quora dataset.
3.3.3 Impact of the Inference Rounds
In this part, we design a comparison experiment to demonstrate the impact of the inference rounds. We set the inference round in our FITN from 1 to 5 and compare their performance on Quora's development and test set. The comparison result is shown in Figure 4. Obviously, as the number of the inference round increases, our model's accuracy increases, verifying the utility of the multi-round inference. However, the increasing trend of the accuracy gradually slows down as the number of inference rounds grows. Continue stacking layers may not bring further significant improvements. We attribute this to the model capturing enough information from a limited multi-round inference under the assistance of our proposed Mem-Att and OA-mixed connection. There is no need to stack too many inference modules.
3.3.4 Analysis of the Mem-Att
In order to further analyze how the Mem-Att works, we compare our Mem-Att with the Dot-Att and conduct a statistical analysis along with a case study to verify that the Mem-Att can pay higher attention to the critical word pairs and the inference round
Table 4: Statistical analysis on the Quora dataset. "Mean±std" denotes the mean value and the standard deviation of the attention distribution. R1., R2., and R3. denote the attention in the round 1, 2, and 3 respectively.
Metric	Mem-Att	Dot-Att
R1. Mean±std	0.0967±0.0026	0.0967±0.0017
R2. Mean±std	0.0967±0.0028	0.0967±0.0003
R3. Mean±std	0.0967±0.0029	0.0967±0.0013
Pearson(R2,R1)	0.7327	0.5456
Pearson(R3,R2)	0.8521	0.5390
in the Mem-Att is progressive.
Statistical Analysis We conduct the statistical analysis on the development set of Quora and compare our Mem-Att with the Dot-Att. We calculate the mean value and the standard deviation of the attention distributions in each inference round to observe the distribution characteristics. Then, we calculate the Pearson correlation coefficient (Benesty et al., 2009) to quantify the relevance between two attention distributions in adjacent rounds. We take the average of the above metrics among all samples as the final metrics.
As shown in Table 4, the standard deviation of the attention distributions in the Mem-Att is larger than that in the Dot-Att and the distribution of the Dot-Att tends to be uniform. It demonstrates that our Mem-Att is more discrete and pays more attention to the specific token pairs. Besides, the Pearson correlation coefficient between the attention distributions of the Mem-Att in adjacent rounds is higher than that in the Dot-Att, which denotes that the inference between adjacent rounds has more relevance in the Mem-Att. The inference procedure in the Mem-Att is progressive.
Case Study Then, we take a pair of similar questions "What is the cost of a Snapdragon 2100 SoC ?" and "What is the Snapdragon 2100 SoC pricing ?" as an example and visualize the attention distributions of the Mem-Att and the Dot-Att in each round of inference. Here, the Mem-Att predicts right and the Dot-Att predicts wrong.
As shown in Figure 5, both the Dot-Att and the Mem-Att can align all pairs of the same word in the first inference round, where the Mem-Att focuses more on these word pairs than the Dot-Att. The distribution of the Mem-Att is more concentrated than that of the Dot-Att, which denotes that
(a) The heat maps of Mem-Att.
(b) The heat maps of Dot-Att.
Figure 5: The heat maps of Mem-Att and Dot-Att. Take "What is the cost of a Snapdragon 2100 SoC ?" and "What is the Snapdragon 2100 SoC pricing ?" as an example.
the Mem-Att has obvious tendency to pay attention. With the increase in the number of inference rounds, the Mem-Att's distribution does not tend to be uniform. Furthermore, the change of the Mem-Att's distribution is continuous, where the Mem-Att gradually deepens its focus on "cost" and "pricing". It demonstrates that the inference in the Mem-Att is progressive. The Mem-Att can gradually align word pairs with similar semantics.
Question Matching can be regarded as a semantic matching task, which core lies in how to model the vector representation of texts (Shen et al., 2018; Reimers and Gurevych, 2019; Gao et al., 2021) and reason about the semantic relationship between text pairs. ESIM (Chen et al., 2017) encodes texts through BiLSTM or TreeLSTM (Socher et al., 2013) and applies the coattention to extract fine-grained alignment information for inference. BiMPM (Wang et al., 2017) matches texts from multiple perspectives by multiple kinds of attentions. For better inference, many studies tend to employ deeper models. DIIN (Gong et al., 2018) applies a dense-net on the interaction matrix extracted from two texts for deep inference. DRCN (Kim et al., 2019) iterates one same block multiple times for multi-turn inference. TIM-W (Zhou et al., 2020) is based on deep mutual in
formation estimation. ADIN (Liang et al., 2019) performs multiple rounds of asynchronous reasoning for the NLI task. In comparison, our FITN performs better due to the better utilization of historical information.
Thanks to the knowledge obtained from massive data, pre-trained models can greatly improve the performance of semantic matching, such as BERT (Devlin et al., 2019) AlBERT (Lan et al., 2020). However, the complexity of the model and the time consumption of reasoning are greatly increased, making them not suitable to resource-constrained scenarios. Enhanced-RCNN (Peng et al., 2020) compares itself with BERT in inference speed and accuracy. Although the performance is relative low, its inference speed is 10 times faster than BERT-base. Under the resource-constrained condition, directly using publicly available tiny pre-trained models is another solution. These models are commonly pre-trained with large-scale corpus (like AlBERT-tiny and AlBERT-base (Lan et al., 2020)) or distilled from large pretrained models (like BERT-tiny (Turc et al., 2019)). Compared with these tiny pre-trained models, our FITN achieves better performance.
In this paper, we study the task of question matching and propose a Full Information Transmission
Network (FITN) that can utilize both the historical representation and the historical interactive information together in a simultaneous fashion. Specifically, the FITN employs a memory-based attention to keep and transmit the historical interactive information and an original-average mixed connectivity pattern to transmit the representation information. Experimental results on two benchmarks show that our FITN takes advantage of both kinds of information and outperforms strong baselines with considerable margin.
This work is supported by the Key Development Program of the Ministry of Science and Technology (No.2019YFF0303003), the National Natural Science Foundation of China (No.61976068) and "Hundreds, Millions" Engineering Science and Technology Major Special Project of Heilongjiang Province (No.2020ZX14A02).
This paper proposes a general model for question matching. This paper neither introduces any social/ethical bias to the model nor amplifies any bias in the data. In all the experiments, we use public datasets and consist their intended use. We build our algorithms using public code bases (PyTorch). We do not foresee any direct social consequences or ethical issues.
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1-4. Springer.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Lu Chen, Yanbin Zhao, Boer Lyu, Lesheng Jin, Zhi Chen, Su Zhu, and Kai Yu. 2020. Neural graph matching networks for Chinese short text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6152-6158, Online. Association for Computational Linguistics.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 1657-1668, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910.
Yichen Gong, Heng Luo, and Jian Zhang. 2018. Natural language inference over interaction space. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Zhe Hu, Zuohui Fu, Yu Yin, and Gerard de Melo. 2021. Context-aware interaction network for question matching. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3846-3853.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2261-2269. IEEE Computer Society.
Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. 2017. First quora dataset release: Question pairs. data. quora. com.
Seonhoon Kim, Inho Kang, and Nojun Kwak. 2019. Semantic sentence matching with densely-connected recurrent and co-attentive information. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6586-6593. AAAI Press.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In
3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Di Liang, Fubao Zhang, Qi Zhang, and Xuanjing Huang. 2019. Asynchronous deep interaction network for natural language inference. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2692-2700, Hong Kong, China. Association for Computational Linguistics.
Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. 2018. LCQMC:a large-scale Chinese question matching corpus. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1952-1962, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Boer Lyu, Lu Chen, Su Zhu, and Kai Yu. 2021. Let: Linguistic knowledge enhanced graph transformer for chinese short text matching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13498-13506.
Shuang Peng, Hengbin Cui, Niantao Xie, Sujian Li, Ji-axing Zhang, and Xiaolong Li. 2020. Enhanced-rcnn: An efficient method for learning sentence similarity. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 2500-2506. ACM / IW3C2.
Yuanyuan Qiu, Hongzheng Li, Shen Li, Yingdi Jiang, Renfen Hu, and Lijiao Yang. 2018. Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pages 209-221. Springer.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. In Proceedings of the Thirty-Second AAAI Conference on Artificial
Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5446-5455. AAAI Press.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Yi Tay, Anh Tuan Luu, and Siu Cheung Hui. 2018. Co-stack residual affinity networks with multi-level attention refinement for matching text sequences. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4492-4502, Brussels, Belgium. Association for Computational Linguistics.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/1908.08962.
Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 4144-4150. ijcai.org.
Runqi Yang, Jianhai Zhang, Xing Gao, Feng Ji, and Haiqing Chen. 2019. Simple and effective text matching with richer alignment features. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4699-4709, Florence, Italy. Association for Computational Linguistics.
Xixi Zhou, Chengxi Li, Jiajun Bu, Chengwei Yao, Keyue Shi, Zhi Yu, and Zhou Yu. 2020. Matching text with deep mutual information estimation. arXiv preprint arXiv:2003.11521.