SlowGuess/NAACL_2021_OCR huggingface dataset:
Comparing with the mono-granularity PLMs in base-size, LBERT takes benefits from word-level information and outperforms its character-level counterpart, BERT-our, by  $1.5\%$  averagely. Meanwhile, LBERT performs better than the word-level model, BERT-word, remarkably on CLUE tasks. We think the lattice inputs incorporate
coarse-grained semantics while avoiding segmentation errors by combining multiple segmentation results. Therefore, with the multi-granularity treatments in word lattices, PLMs obtain better performances in downstream tasks than the monogranularity settings.
Furthermore, LBERT outperforms the previous state-of-the-art (sota) multi-granularity PLM, AMBERT (Zhang and Li, 2020), by  $0.9\%$  in text classification and  $1.3\%$  in MRC, averagely. Different from modeling the characters and words separately, the graph representations of word lattices could enhance the interaction between multi-granularity tokens and utilize all possible segmentation results simultaneously. As a result, LBERT achieves a new sota among the base-size models on the CLUE leaderboard as well as the sub-leaderboards for text classification and MRC tasks. $^7$
With lite-size settings, LBERT brings  $2.0\%$  improvement over BERT-our on average, which is larger than the case in base-size. In CWS, TNEWS, and CSL, the lite-size LBERT even outperforms the base-size BERT-our. With more coarse-grained inputs, the shallower architectures do not require complicated interactions to identify character combinations but utilizing word representations explicitly, thus, narrowing the gap with the deeper ones.
Ablation Study. We conduct ablation experiments to investigate the effectiveness of our proposed lattice position attention (LPA) and masked segment prediction (MSP) in downstream tasks. To reduce the computational costs, we base our
$^{6}$ https://huggingface.co/hfl/chinese-roberta-wwm-ext
<sup>7</sup>https://www.cluebenchmarks.com/rank.html, the sota by the time of submission, Oct. 31st, 2020.
WSC.	NER. (EF1)	CMRC	avg.
BERT-our	66.3 (1.9)	92.8 (0.2)	57.2 (0.7)	72.1
LBERT	75.3 (1.3)	94.1 (0.1)	64.5 (0.5)	78.0
-Rel.	75.7 (1.2)	93.7 (0.2)	63.8 (0.8)	77.7
-Dis.	73.8 (0.7)	93.8 (0.2)	63.1 (0.4)	76.9
-Dis. -Rel.	72.8 (0.5)	93.6 (0.1)	61.7 (0.4)	76.0
-MSP	72.2 (1.0)	93.9 (0.1)	63.0 (0.7)	76.4
Table 2: Ablation results in lite-size settings with standard deviation in subscripts. EF1 is the entity-level F1 score. -Dis. and -Rel. represent the ablation of the relative distances and positional relations in LPA, respectively. The small numbers in brackets are standard deviation scores over five runs.
pre-training settings on lite-size with the sequence length of 128 characters. We select one task from each of the task clusters. We use the entity-level F1 score for NER to highlight the influence on boundary prediction. We report the average scores over 5 runs and use the development sets for CLUE tasks.
We can see in Table 2 that the ablation of either module (-Dis.-Rel. & -MSP) leads to a substantial drop in the average scores. In particular, replacing MSP with vanilla MLM, the average score of - MSP drops by  $1.6\%$ . For the WSC task, where long-range dependency is required to resolve the coreference, the gap is high up to  $3.1\%$ . We trace this drop into the pre-training procedure and observe the MLM accuracy for the -MSP setting on the development set is  $88.3\%$ . However, if we mask the tokens within the segment and avoid potential leakages, the accuracy drastically drops to  $48.8\%$ , much lower than the performance of LBERT training with MSP  $(56.6\%)$ . This gap provides evidence that the MSP task prevents the PLMs from tricking the target by peeking the overlapping text units in one segment, thus encourages the PLMs to characterize the long-range dependency.
For the LPA method, without the positional relation  $(- \text{Rel.})$ , the entity-level F1 score on NER decreases by  $0.4\%$ , and the performance on CMRC decreases by  $0.7\%$ . The performance drops are similar to the case without distance information  $(- \text{Dis.})$ . Without either of them  $(- \text{Dis.} - \text{Rel.})$ , the gaps widen to  $0.5\%$  and  $2.8\%$ , respectively. The boundary predictions in NER and CMRC are more sensitive to the local linguistic structures like nested words or overlapping ambiguity. With the positional relation and distance characterized in attention, LBERT could accurately model the interaction between the nested and overlapping tokens in different segmentation results. Meanwhile, the
accuracy of WSC. remarkably drops without distance information. The performance drops by  $7.5\%$  and  $5.8\%$  when the number of characters between the pronouns and candidate phrases is larger than 30, or between 20 to 30, respectively. For the rest cases, the drop is only  $0.4\%$ . With explicitly modeling of distance, LBERT predicts the long-distance coreference relations more accurately. Averagely, without the positional relation and distance modeling in LPA, the performance drops by  $2.0\%$  on the three tasks, showing the importance of LPA in assisting the PLMs to exploit the multi-granularity structures in word lattices.
How LBERT Improves Fine-grained PLMs? We compare the prediction results of LBERT and the character-level BERT-our in base-size on development sets to investigate how the LBERT outperforms the vanilla fine-grained PLMs. Intuitively, the word-level tokens in lattices provide coarse-grained semantics, which argument the character-level inputs.
We observe in TNEWS, the short text classification task, LBERT brings more improvement in the shorter instances, where the statements may be too short to provide enough context for predictions. By dividing the development set into five bins with equal size according to the sentence length, LBERT outperforms BERT-our by  $2.3\%$  and  $1.3\%$  in the shortest and second shortest bins, respectively, larger than the average gain on the rest instances  $(0.6\%)$ . We think the redundant tokens in word lattices provide rich context for the semantics of these short statements. For example, for the short title我们村的电影院/the cinema in our village, with the redundant words, 电影/movie, 影院/cinema, and 电影院/cinema, introduced in the lattice, LBERT classifies the instance as entertainment news instead of news stories.
Another case is the CSL task, where the target is to predict whether the candidate words are keywords for a given paragraph. For those instances, where LBERT identifies more than two word-level tokens from each candidate word averagely, which accounts for  $47\%$  of the dataset, the performance gain is  $3.0\%$ , significantly larger than the average improvement of the rest,  $1.0\%$ . We think LBERT understands the key words from various aspects by exploiting the redundant expressions in lattices. For example, from the keyword candidate '太阳能电池/solar battery, the太阳/solar,太阳能/solar energy,电池/battery, and太阳能电池/solar battery
Figure 4: Visualization of the attention scores of.   
research life is very fulfilling. The three lines from top to bottom are the attentions before fine-tuning and after fine-tuning with MSRA-NER and TNEWS tasks, respectively.
are lattice tokens. With these word-level tokens, LBERT could match this candidate with the expressions in the paragraph like 阳极/positive electrode, 光/light, 电子/electron, 离子/ion, etc.
On the other side, for MSRA-NER, LBERT reduces the errors in identifying entities with nested structures. Averagely, the number of error cases where the predicted entities are nested with the golden ones are reduced by  $25\%$  in LBERT. For example, the organization entity解放巴勒斯坦运动/Palestine National Liberation Movement is nested with the location entity 巴勒斯坦/Palestine and ends with an indicator to organizations, 运动/movement. The character-level baseline model mistakenly recognizes the 巴勒斯坦/Palestine and 动/move as a location and an organization, separately. While LBERT identifies this entity correctly after integrating the words, 解放/liberate, 巴勒斯坦/Palestine, and 运动/movement. With the pretrained multi-granularity representations, LBERT fuse the contextual information from words and characters simultaneously, and detects the correct entity in success.
How does LBERT harness multi-Granularity representations? LBERT consumes all the words and characters from input sequences simultaneously, but how does the model utilize such multi-granularity representations during pre-training and downstream tasks? To investigate this, we use the average attention scores that each lattice token receives among all layers and all heads to represent its importance.
As the example shown in Figure 4, before finetuning, LBERT focuses on tokens including 'live',充实/fulfilling,研究/research,研究生/graduate student, 究/investigate, etc. Before fine-tuning on specific tasks, the model captures various aspects of the sentence. After fine-tuning with MSRA-NER, the most focused words become '充实/fulfilling',
Question: What is the game with the theme song which is sung by Chen Yiting and is composed by Zeng Zhihao?
Document: ... He was recommended by Lu Shengfei, the No.7 music director of the Cape, as the composer of the theme song The South of China[err]. ... In cooperation with singer Chen Yiting, Zeng Zhihao was responsible for the composition of the theme song Wish and the promotion song Youth Love for The Legend of Sword and Fairy V[ans].
Table 3: An example in CMRC. Given the Question, LBERT predicts the corrected answer[ans] in the Document, while BERT-our predicts the wrong answer[err].
很/very, 生活/life, and 研究/research, i.e., the tokens from the golden segmentation result, “研究|生活|很|充实”, which is intuitively beneficial for the NER tasks. The attention score of the wrong segmented word, 研究生/graduate student, drops remarkably.
On the other hand, after fine-tuning with the news title classification task, TNEWS, LBERT tends to focus on充实/fulfilling,研究生/graduate student, 生活/life, etc. Although these tokens can not co-exist in one Chinese word segmentation result, LBERT can still utilize the redundant information from various plausible segmentations to identify the topics of inputs. These results indicate that Lattice-BERT can well manage the lattice inputs by shifting the attention to different aspects among the multi-granularity representations according to specific downstream tasks.
Case Study. Table 3 shows an example in CMRC, a span selection MRC task, where models choose a text span from the given document to answer the question. In this case, the question asks for a game, restricted by its theme song. BERT our incorrectly outputs a theme song, The Song of China, since there is no expression in the document explicitly related to game. However, LBERT find the correct answer, The Legend of Sword and Fairy V. One possible reason is that The Legend of Sword and Fairy is an entry in the vocabulary for lattice construction. LBERT may have learned this word as an entity for a famous video game from the context in pre-training by explicitly exploiting its representation as a whole. With the coarse-grain text units in pre-training, LBERT directly encodes knowledge about these units to benefit the downstream tasks.
Computational Costs. For fair comparisons, we ensure LBERT and the character-level baselines
(i.e., BERT-our) have the same training epochs when the training steps are equal following previous works (Diao et al., 2020; Zhang and Li, 2020). Thus, comparing with BERT-our,  $35\%$  more text units are introduced in the pre-training instances of LBERT, which introduces  $48\%$  more computational resources comparing with BERT-our to process the additional word-level tokens (See Appendix C). To illustrate the gains attribute to the incorporation of lattices instead of additional computations, we investigate the lite-size BERT-our with longer input sequences in pre-training, which has the same computational costs as LBERT. We find LBERT still outperforms BERT-our by  $2.2\%$  averagely on CLUE classification tasks. More details are elaborated in Appendix D.
Recently, several works utilize the lattice structures to explore multi-granularity information in Chinese NLU tasks. Buckman and Neubig (2018) incorporate lattice into recurrent neural networks based language modeling to capture marginal possibilities across all possible paths. In NER, Lattice-LSTM (Zhang and Yang, 2018), graph neural networks (Gui et al., 2019), and flat-lattice transformers (Li et al., 2020a) are adopted to incorporate words from lattice inputs. Lai et al. (2019) adapt convolutional neural networks to lattice for matching based question answering. Chen et al. (2020) adopt graph matching networks to perform multi-granularity interaction between lattices for text similarity. These works are designed to explore word lattices in specific tasks. We explore the multi-granularity representations with word lattices in PLMs, investigating the previously attempted downstream tasks as well as other tasks, e.g., MRC. We design LPA to meet various interaction needs of the downstream tasks and propose MSP to avoid the leakages.
In the field of Chinese PLMs, some efforts incorporate coarse-grained information with the character-level inputs. ERNIE 1.0 (Sun et al., 2019) and BERT-wwm (Cui et al., 2019) propose to mask the words, entities, and phrases as a whole in the MLM task to encourage the modeling of coarse-grained features. ZEN (Diao et al., 2020) adopt auxiliary networks to integrate n-gram representations. BERT-MWA (Li et al., 2020b) propose word-aligned attention to use multiple segmentation boundaries. Different from their methods, we propose Lattice-BERT to consume multi
granularity tokens simultaneously into one PLM via lattice graphs. Thus, Lattice-BERT explicitly exploits the representations of the coarse-grained units, as well as the interactions among word- and character-level tokens. The proposed MSP task can be treated as an extension of the whole word masking (Cui et al., 2019), while considering the span information like Joshi et al. (2020, SpanBERT) according the lattice structures. The concurrent work, Zhang and Li (2020, AMBERT) investigate multi-granularity inputs similarly, but they use two transformer encoders to separately deal the word and character sequences. We treat the words and characters as lattice graphs, which enables thorough interactions among multi-granularity tokens and utilizes all potential segmentation results.
In this paper, we propose Lattice-BERT to leverage multi-granularity representations of input sentences for Chinese PLMs. Specifically, Lattice-BERT takes a word-lattice as input, modeling the representations of words and characters simultaneously. We design the lattice position attention to embed the multi-granularity structure into transformers and propose the masked segment prediction task to avoid potential leakage in original MLM caused by the redundancy information in lattices. We conduct extensive experiments on 11 Chinese NLU tasks and observe consistent gains over character-level baselines, achieving new sota on CLUE benchmarks. We show that Lattice-BERT can well manage the lattice inputs and utilize multi-granularity representations to augment the character-level inputs. We believe the lattice structure can be adapted to integrate the phrase and word representations into the word-piece based PLMs in other languages, which we leave for future exploration.
This work is supported in part by the National Hi-Tech RD Program of China (No. 2020AAA0106600), the NSFC under grant agreements (61672057, 61672058). For any correspondence, please contact Yansong Feng.
Alfred V Aho and Margaret J Corasick. 1975. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333-340.
Jacob Buckman and Graham Neubig. 2018. Neural lattice language models. Transactions of the Association for Computational Linguistics, 6:529-541.
Lu Chen, Yanbin Zhao, Boer Lyu, Lesheng Jin, Zhi Chen, Su Zhu, and Kai Yu. 2020. Neural graph matching networks for Chinese short text matching. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6152-6158, Online. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pretrained models for Chinese natural language processing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 657-668, Online. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2020. ZEN: Pre-training Chinese text encoder enhanced by n-gram representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4729-4740, Online. Association for Computational Linguistics.
Thomas Emerson. 2005. The second international Chinese word segmentation bakeoff. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing.
Tao Gui, Yicheng Zou, Qi Zhang, Minlong Peng, Jinlan Fu, Zhongyu Wei, and Xuanjing Huang. 2019. A lexicon-based graph neural network for Chinese NER. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1040-1050, Hong Kong, China. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Guolin Ke, Di He, and Tie-Yan Liu. 2020. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Yuxuan Lai, Yansong Feng, Xiaohan Yu, Zheng Wang, Kun Xu, and Dongyan Zhao. 2019. Lattice cnns for matching based chinese question answering. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 6634-6641. AAAI Press.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108-117, Sydney, Australia. Association for Computational Linguistics.
Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020a. FLAT: Chinese NER using flat-lattice transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6836-6842, Online. Association for Computational Linguistics.
Yanzeng Li, Bowen Yu, Xue Mengge, and Tingwen Liu. 2020b. Enhancing pre-trained Chinese character representation with word-aligned attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3442-3448, Online. Association for Computational Linguistics.
Ziran Li, Ning Ding, Zhiyuan Liu, Haitao Zheng, and Ying Shen. 2019. Chinese relation extraction with multi-grained information and external linguistic knowledge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4377-4386, Florence, Italy. Association for Computational Linguistics.
Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. 2018. Nugget proposal networks for Chinese event detection. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1565-1574, Melbourne, Australia. Association for Computational Linguistics.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-BERT: enabling language representation with knowledge graph. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 2901-2908. AAAI Press.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Boer Lyu, Lu Chen, Su Zhu, and Kai Yu. 2021. Let: Linguistic knowledge enhanced graph transformer for chinese short text matching. In AAAI.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. In AAAI, pages 8968-8975.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. 2019. Nezha: Neural contextualized representation for chinese language understanding. arXiv preprint arXiv:1909.00204.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaowehua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4762-4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural
Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Xinsong Zhang and Hang Li. 2020. Ambert: A pretrained language model with multi-grained tokenization. arXiv preprint arXiv:2008.11869.
Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1554-1564, Melbourne, Australia. Association for Computational Linguistics.