Self-Distillation and Pinyin Character Prediction for Chinese Spelling Correction Based on Multimodality  
[PERSON]  
1 School of Information Science and Technology, North China University of Technology, Beijing 100144, China; [EMAIL_ADDRESS]. In recent years, leveraging pre-trained language models (PLM) has emerged as a dominant approach for Chinese spelling correction tasks. Prominent examples include FASpell [4], Softmasked-BERT [5], SpellGCN [6], and PLOME [7]. Concurrently, certain researchers have directed their focus towards the phonological and glyph features of Chinese characters, aiming to enhance the model's error correction capabilities by amalgamating phonetic and visual information [7; 8]. Previous studies underscore the significance of pinyin and glyph similarities, revealing that 83% of spelling errors are attributed to pinyin similarity and 48% to glyph similarity [9]. Effectively modeling Chinese pronunciation in the context of the CSC task remains a pivotal challenge, and nearly all contemporary state-of-the-art CSC methods explicitly or implicitly incorporate Chinese pronunciation. Explicit methods delve into the Chinese pronunciation of entire characters, such as encoding the pinyin of a Chinese character and integrating it into the character representation using specific strategies [10]. Alternatively, Chinese pronunciation prediction models the relationship between similar characters in pinyin [7]. Implicit methods primarily focus on pinyin glyph similarity between Chinese character pairs, either by increasing the decoding probability of characters with akin pronunciations [6] or by incorporating pinyin glyph similarity into the encoding process via graph convolutional networks (GCN) [6]. Despite the notable performance gains achieved, these approaches confront two potential challenges: during training, pinyin information may either be overlooked or overshadowed by textual information. For instance, a specific BERT model that exclusively considers pinyin sequences, excluding Chinese characters, can still detect and rectify erroneous characters, while models like REALISE [8] encode and merge textual and pinyin information through a gating mechanism that overlooks an error. Second, the introduction of pinyin features may compromise the representation of normal text. Using Figure 2 as an illustration, a standard BERT model can correct the misspelled character \"#!\" in the input, whereas REALISE fails to do so. This issue is attributed to REALISE's tendency to over-rely or over-fit on pinyin information. Currently, only a fraction of the available multimodal information is harnessed in the existing Chinese spelling correction (CSC) methods, highlighting the underutilization of the inherent value in Chinese character multimodality. This underscores the unexplored potential for enhancement within the field of multimodality for the CSC task. Building upon this realization, we present a novel approach named SPMSpell (Self-Distillation and Character Pinyin Prediction Based on Multimodality). SPMSpell introduces a finely tuned pinyin prediction task, an adaptive weighting mechanism, and a self-distillation module, aiming to refine the performance of the CSC task. Concretely, when provided with a sentence containing spelling errors, Chinese-BERT [11] serves as the encoder backbone to fuse three pivotal feature dimensions: semantic, phonological, and graphemic. Three distinct decoders are then constructed based on this fusion: one for accurate character prediction, another for predicting the consonant-rhyme of each target character (i.e., pinyin prediction), and the last one devoted to the self-distillation module, ensuring the model's prediction consistency when presented with the original text input. Throughout the training phase, the pinyin prediction task serves as an auxiliary task, with its predictions discarded during inference. Figure 1: Examples of pinyin error and glyph error, where red color indicates the wrong character.