Exploring Conditional Variational Mechanism to Pinyin Input Method for Addressing One-to-Many Mappings in Low-Resource Scenarios (2024)

Abstract: Pinyin input method engine (IME) refers to the transformation tool from pinyin sequence to Chinese characters, which is widely used on mobile phone applications. Due to the homophones, Pinyin IME suffers from the oneto-many mapping problem in the process of pinyin sequences to Chinese characters. To solve the above issue, this paper makes the first exploration to leverage an effective conditional variational mechanism (CVM) for pinyin IME. However, to ensure the stable and smooth operation of Pinyin IME under low-resource conditions (e.g., on offline mobile devices), we should balance diversity, accuracy, and efficiency with CVM, which is still challenging. To this end, we employ a novel strategy that simplifies the complexity of semantic encoding by facilitating the interaction between pinyin and the Chinese character information during the construction of continuous latent variables. Concurrently, the accuracy of the outcomes is enhanced by capitalizing on the discrete latent variables. Experimental results demonstrate the superior performance of our method.

heading: 1 Introduction
text: Input method engines (IMEs)1 are important tools to connect users with mobile applications, drawing dramatic attentions (Chen and Lee, 2000; Li et al., 2004; Zheng et al., 2011; Han and Chang, 2013; Chen et al., 2013; Jia and Zhao, 2014; Huang et al., 2015, 2018; Zhang et al., 2019; Liu et al., 2021; Tan et al., 2022; Ding et al., 2023). In China, there are two common Pinyin IMEs2 for cellphones: the 9-key IMEs and the 26-key IMEs, which are used by more than 97% of Chinese people (Hu et al., 2022). As shown in Figure 1, the 26-key keyboard
*Work done at WeChat AI, Tencent Inc. †Corresponding Author
1https://en.wikipedia.org/wiki/Input_method 2https://en.wikipedia.org/wiki/Pinyin_input_method
uses the 26 English letters as Chinese pinyin syllables, while the 9-key keyboard maps the 26 pinyin syllables onto 8 keys.
Due to the Chinese homophones, the process of converting pinyin sequences to Chinese character sequences inevitably presents a one-to-many mapping challenge for Pinyin IME. In the perfect pinyin mode of a 26-key IME, 500 pinyin combinations need to correspond to nearly 10,000 Chinese characters (Jia and Zhao, 2014; Zhang et al., 2019). For instance, inputting the pinyin sequence "bei zi" can map to various Chinese characters with completely different meanings, such as "被子" (blanket), "杯 子" (cup), and "辈子" (lifetime). In the case of the abbreviated pinyin mode, entering the initial letters "b z" for "bei zi" can result in not only the aforementioned characters but also others like "不 止" (more than), "不在" (not present), and "步骤" (steps). As for 9-key IMEs, each key can represent 3 to 4 pinyin syllables, which means that inputting "23494" offers 323 possible pinyin combinations except "beizi". While some pinyin combinations that do not adhere to standard rules can be pruned, this undoubtedly expands the solution space.
One effective method to alleviate the one-tomany problem is to generate more candidates for users to autonomously choose the one they need. Existing methods typically employ beam search to generate additional candidates. However, Holtzman et al. (2020) found that unlike beam search, which selects the token with the highest probability, humans tend to choose more surprising and
616
diverse tokens. Furthermore, beam search requires sorting multiple candidates during the generation process, and in some low-resource scenarios (such as on offline mobile devices), it is challenging to ensure stable and rapid generation due to the lack of sufficient memory and computational resources.
To alleviate the aforementioned problems, we take inspirations from conditional variational mechanism, which models the one-to-many cases through the latent variable space and generate various results by sampling different latent variables (Shen et al., 2017; Zhao et al., 2017; Bao et al., 2020; Lin et al., 2020; Fang et al., 2021; Sun et al., 2021). Therefore, instead of prioritizing the arrangement of the highest-scoring candidate result, our primary study of interest is to recall more eligible candidates within the same inference time. To this end, we propose a conditional variational IME model (CV-IME) with a novel hybrid latent variables strategy. Please refer to § 2.2 for details.
Our contributions are as follow: To the best of our knowledge, this is the first exploration and investigation of the impact of CVM on the performance of Pinyin IME in low-resource scenarios, specifically on offline mobile platforms. Furthermore, we propose a novel hybrid latent variable that designed to balance the performance and efficiency of our CV-IME model.

heading: 2 Methodology

heading: 2.1 Base Model
text: With the advancement of technology, the latest Pinyin IMEs, e.g., PinyinGPT (Tan et al., 2022) and GeneInput (Ding et al., 2023), primarily adopt models based on the transformer architecture (Vaswani et al., 2017). Therefore, we have adopted the transformer structure and conducted a series of experiments to identify the most suitable configuration.

heading: 2.2 Conditional Variational IME
text: Following the previous work of CVM, CV-IME primarily consists of four components: a encoderdecoder model, a prior network pθ(z|c), a recognition network qϕ(z|r, c) and a discrete latent variable matrix M . c, r and z represent the user input (i.e., context and pinyin sequence), the character result and the continues latent variable.
Hybrid Latent Variable. Previous researches indicate that continuous latent variables can enhance diversity but may reduce relevance, whereas discrete latent variables strengthen relevance but lack
diversity (Gao et al., 2019; Bao et al., 2020; Sun et al., 2021, 2023). Therefore, a promising direction is to hybrid the continuous and discrete latent variables, leveraging their respective strengths to complement and offset their weaknesses. To build the hybrid latent variables H , we follow Sun et al. (2023), adding sentence-level continuous latent variable z′s to the discrete latent variables M : H = (z′s +M [1], · · · , z′s +M [k]), where K represents the number of discrete latent variables.
Continuous Latent Variables. We initially employ the model encoder to transform c and c+r into prior memory h and posterior memory h′. Given that there is a degree of alignment between the pinyin and character sequences in the task of pinyinto-character conversion, relying solely on the encoder’s self-attention mechanism for interaction may not yield effective information. Therefore, we have introduced an interaction between the prior memory and the posterior memory:
h′ = SoftMax(h · h′T ) · h′ (1)
To enhance the recognition process, we use h and h′ together to estimate the isotropic Gaussian distribution qϕ(z|c, r) ∼ N (µ′, σ′2I) :
( µ′1, ..., µ ′ n
log(σ′21 ), ..., log(σ ′2 n )
) =   [h1;h ′ 1]
· · · [hn;h ′ n]
 W ′u ,
where W ′u is trainable parameters of qϕ(z|r, c). After that, we follow additive Gaussian mixing (Wang et al., 2017) to obtain the sentence-level continuous latent variables. (see more details in Appendix B)

heading: 3 Experimental Settings
text: Benchmarks. We used two public benchmarks, namely the People’s Daily (PD) corpus (Yang et al., 2012) and WD dataset (Tan et al., 2022), in the experiments. PD is extracted from the People’s Daily from 1992 to 1998, while WD is extracted from the WuDaoCorpora (Yuan et al., 2021). Different from PD, WD contains test cases from 16 different domains of test cases.
Evaluation Metrics. We use the precision of topN, indicating whether the desired result is included in the generated top-N results. We also use the inference time for one instance as a metric.
Training Dataset. To train our CV-IME and base model, we built a training dataset for the pinyin-tocharacter task based on the news2016 corpus3. We randomly extract sentence from news2016 corpus and incorporate “pypinyin” tool to convert Chinese Characters into Pinyin syllables. Table 1 shows the statistics of this dataset. (Please refer to appendix C for more details)
Baseline Models. We introduced some IMEs, i.e., Google IME4, On-OMWA (Zhang et al., 2017) and On-P2C (Zhang et al., 2019) as baselines.
• GoogleIME is a commercial Chinese IME that offers an API with debugging capabilities.
• On-OMWA system, introduced by Zhang et al. (2017), is an adaptive online model designed for the acquisition of new words, specifically tailored for Chinese IMEs.
• On-P2C model, as described in Zhang et al. (2019) on open vocabulary learning, is a neural network-based Pinyin-to-Chinese conversion system that improves its performance by dynamically updating its word database to facilitate learning of an open vocabulary.
Training Detail. The hidden size of all models is set to 512. Our CV-IME employs a Transformer model with 2 encoder layers and 1 decoder layer, and additionally incorporates two fully-connected layers as a prior network. We set the batch sizes to 1024 and 256 for base model and CV-IME, respectively. Adam is used for optimization. The initial learning rate is set to 0.0001. We also introduce KL annealing trick to leverage the KL divergence during the training. The KL weight increases linearly from 0 to 1 in the first 3000000 batches. We train all models in 100 epochs on four A100 GPU cards with Pytorch, and save the model parameters when the validation loss reaching minimum.
3https://github.com/brightmart/nlp_chinese_corpus 4https://www.google.com/inputtools/services/features/input-
method.html

heading: 4 Result and Analysis

heading: 4.1 Model Structure Selection.
text: Figure 2 and Table 2 show the generation latency, accuracy and parameters of base models with different configurations, which illustrates that: (1) maintaining a fixed number of encoder while solely increasing decoder layers significantly raises latency (≈10ms) without notably improving accuracy. (2) while the increase in the number of encoder layers leads to a gradual rise in latency (≈2ms), accompanied by an upward trend in accuracy. (3) the number of parameters in an encoder is to some extent positively correlated with accuracy.
In selecting the final encoder-decoder configuration, we primarily considered constraints on memory storage and latency. In this work, we posit that under low-resource constraints, with a storage ceiling of no more than 32MB and a generation latency not exceeding 30ms for a single candidate, the system can operate reliably.
Regarding storage, we aimed to emulate a realistic mobile environment, mindful of the fact that an IME system houses multiple models, such as PinyinIME, speech recognition, handwriting recognition, etc. Given the overall storage consumption of the system must remain low, we endeavored to limit the size of the PinyinIME model to within
32MB. To ensure stable performance, the model was restricted to using at most INT8 quantization, which means the parameters had to be kept under 32M. As for latency, we set a strict benchmark: the CPU latency for generating a single case under lengthy text conditions must not exceed 30ms. Since the model needs to regenerate results immediately with each pinyin character input by the user, to prevent perceptible delays, we aimed to set even more stringent latency requirements.
Therefore, a configuration comprising a 4-layer encoder and a 1-layer decoder represents the most cost-effective choice.

heading: 4.2 Generation Latency.
text: Figure 3 shows a comparative analysis of the latency incurred by CV-IME (ours), Base+Beam search (beam) and Base+TopK sample (topk) (Fan et al., 2018) in generating a varying number of candidates. As can be observed from the figure, the latency of CV-IME when generating four candidates is nearly identical to that of the base model when producing two candidates, which demonstrates the superiority of the CV-IME approach in recalling more candidate results under low-resource conditions. We also observed that the latency of the topk significantly increases when generating on CPU devices, which may be attributed to the higher computational complexity of the multinomial function in PyTorch on CPU.

heading: 4.3 PD Benchmark.
text: Table 3, 4 show the results of PD. CV-IME-imeans the results of CV-IME using i-th hybrid latent variable. Beam represents the beam search. From these results, we can observe that: (1) Our models outperform the baselines on the PD benchmark; (2) our models show more significant results in the task of converting from a abbreviated pinyin to characters; (3) Under the condition of equivalent
time expenditure, our model is capable of generating more candidates and achieve better accuracy compared to the baselines; (4) The four hybrid latent variables exhibited a clustering effect in the 9-key IME, where CV-IME-1 excelled in abbreviated pinyin and CV-IME-3 in perfect pinyin. However, this phenomenon was not replicated in the 26- key IMEs. These results suggest that the current training methodology for hybrid latent variables has certain limitations, as it struggles to encourage different latent variables to focus on distinct data categories during training. This will be a direction for our future research.

heading: 4.4 WD Benchmark.
text: Table 5 reports the results of different domains over WD. We have selected the results from four domains where the differences between CV-IME and the Base model are the smallest and the largest under various pinyin input patterns. This result demonstrates that the CV-IME achieves a superior performance than base model in terms of all domains in WD. We also conduct experiments with different configurations on WD, which are detailed
in the appendix D.3.

heading: 4.5 Ablation Study.
text: Table 6 presents the results of ablation experiments, where “CLS” and “CHVT” are two alternative strategies for constructing hybrid latent variables that differ from our approach:
• “CLS” means using the [CLS] token to determine the prior distribution of continuous latent variables.
• “CHVT” stands for Conditional Hybrid Variational Transformer (Sun et al., 2023), which also utilizes hybrid latent variables, but it is primarily used in dialogue tasks.
Compared to CLS and CHVT, CV-IME achieves better performance on PD and WD benchmarks, indicating the effectiveness of the proposed strategy in the pinyin-to-character task. Moreover, “w/o. CLV” and “w/o. DLV” denote the CV-IME model variants with the continuous latent variables (CLV) and discrete latent variables (DLV) removed, respectively. The findings indicate that DLV may excel in accuracy but fall short in generalizing across diverse scenarios. Therefore, the “w/o. CLV” performs well on news data (PD) similar to the training set but not as well on OOD data (WD). Similarly,
CLV may excel in diversity but compromise on precision, which exhibit a marked decline in performance when the DLV is removed. Furthermore, from perfect to abbreviated pinyin, the degradation in performance becomes more pronounced.

heading: 4.6 Discussion on Top-1 Results.
text: For the Pinyin-to-character task, there is a clear correlation between the top-1 accuracy and the distribution differences between training and testing data. This is due to the presence of one-to-many samples in the data, where identical Pinyin corresponds to completely different outcomes. If the most proportionate samples in the test data also happen to be the highest probability samples in the training data, the model’s top-1 results are likely to be high. The CV-IME is proposed to internalize one-to-many data through latent variables, mitigating the excessive influence of training data distribution on the test data distribution. Experimental results reveal differentiated outcomes presented by various hybrid latent variables, indicating that latent variables can indeed diversify data distributions. However, the current training is unsupervised, and the overall differentiation effect is not pronounced, necessitating further research.

heading: 5 Conclusion
text: This paper introduces the conditional variational mechanism into the IME model, presenting the CVIME model. By incorporating hybrid latent variables, CV-IME enhances diversity while maintaining the quality of generated results. In comparison to existing IME models, the experimental results demonstrate that CV-IME can recall more diverse and accurate results within similar time constraints, exhibiting significant advantages in low-resource scenarios, such as offline mobile devices.

heading: Limitations
text: Application Scenarios. The CV-IME model is proposed to effectively mitigate the severe one-tomany problem in the task of pinyin-to-character conversion under low-resource conditions (e.g., on the mobile phone devices). Therefore, under conditions of abundant computational and storage resources, the introduction of larger pre-trained language models with more parameters may yield better results. After all, the practical application of latent variable-based pre-training techniques remains to be tested, which also constitutes one of our future research directions.
Data Distribution. The training data and evaluation benchmarks are extracted from different Chinese corpora, which are not not consistent with the data generated by real users of the Pinyin IME, and there are certain differences in their distributions. Consequently, in constructing our training dataset, we selected news data closely aligned with the PD benchmark to approximate independent and identically distributed scenarios (comparison with PD), and out of domain scenarios (comparison with WD). Through the aforementioned configurations, we have rudimentarily simulated real-world scenarios of general distribution and user-specific personalization, which to some extent, demonstrates the efficacy of our approach in practical applications.
Flexibility and Differentiation. The hyperparameters (e.g., the number of discrete latent variables, the annealing steps of KL and so on) need to be determined through multiple experiments, which cannot be set adaptively. Additionally, the experimental results indicate that the different mixed latent variables are not sufficiently independent, as the corresponding generated texts are not entirely distinct. This may be attributed to the current training methodology being guided by unsupervised gradient backpropagation. It might be necessary to introduce regularization terms or to devise a novel training approach to enhance the discriminability between the mixed latent variables. These initial promising results for distinguishing different hybrid latent variables for recalling diverse candidate results will hopefully lead to future work in this interesting direction.

heading: Ethics Statement
text: We acknowledge and ensure that our study is compatible with the provided Code of Ethics. Pinyin
input method engine (IME) is crucial for building connection between Chinese people and mobile applications, which is an import topic in Chinese natural language process field. All our experiments are conducted on public available datasets to avoid ethical concerns. All terms for using these datasets are strictly followed in our study. There are no direct ethical concerns in our research.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their constructive comments. This work is supported by the Beijing Natural Science Foundation, China (Nos. 4222037, L181010).

heading: A Related Work
text: A.1 Input Method Engine Input method engines (IMEs) are important tools to connect users with mobile applications, By providing an efficient and user-friendly interface, it enables users to input text with ease, thereby enhancing their overall experience with the apps. Different from alphabetic languages, the input of some Asian language (i.e. Chinese) characters must rely on the IMEs. In China, there are two common Pinyin IMEs for cellphones: the 9-key IMEs and the 26-key IMEs (Hu et al., 2022). Previous research on Chinese IMEs primarily focused on three tasks associated with the 26-key keyboard:
(1) The perfect (abbreviated) pinyin to Chinese characters (PTC) task (Chen and Lee, 2000; Li et al., 2004; Zhang et al., 2017; Huang et al., 2018; Zhang et al., 2019; Tan et al., 2022). This task represents the most fundamental aspect of the Pinyin IME, revealing the core performance capabilities of the IME model.
(2) The input noise correction tasks, such as input typo correction (Zheng et al., 2011; Jia and Zhao, 2014; Liu et al., 2021) and Chinese spelling check (Chiu et al., 2013; Han and Chang, 2013; Chen et al., 2013). Due to the limited screen size of mobile phones, users may accidentally press the wrong keys on a 26-key keyboard, leading to incorrect pinyin syllables being entered. For instance, while typing ‘songgei’ (送给, give), the ‘i’ might be mistakenly hit as ‘u’, resulting in ‘songgeu’. Identifying the noise caused by these accidental touches and correcting them to output what the user intended is a significant challenge.
(3) The intelligent association task (Huang et al., 2015; Huang and Zhao, 2018; Ding et al., 2023). Usually Pinyin IMEs simply predict a list of character sequences for user choice only according to the pinyin input. However, Chinese inputting is a multi-turn procedure, which can be supposed to be exploited for further user experience promoting. This task is a commonly used input assistance function, which predicts possible next sentences
based on the content already entered by the user for selection, to improve input efficiency.

heading: A.2 Conditional Variational Mechanism
text: Conditional variational mechanisms (Kingma and Welling, 2014; Sohn et al., 2015; Yan et al., 2016; Bowman et al., 2016) are powerful tools in text generation task, and they are usually used in dialogue generation models. By using continuous latent variables, previous conditional variational mechanisms are introduced into dialogue generation models to tackle short, dull and general responses problem (Shen et al., 2017; Zhao et al., 2017; Chen et al., 2018; Lin et al., 2020; Fang et al., 2021; Sun et al., 2021; Chen et al., 2022; Sun et al., 2023).
The conditional variational mechanism estimates the posterior probability distributions p(z|c, r) and the prior probability distribution p(z|c) of latent variable z based on the dialogue corpora, where c denotes the context, r denotes the response, and a context and a response together constitute a singleturn dialogue pair. During training, these models sample the continuous latent variable z from p(z|c, r) and maximize the conditional probability p(r|c, z) to encode context and response into latent space. Meanwhile, they also minimize the KL-divergence DKL(p(z|c, r)||p(z|c)) to bring the two distributions closer together, thus constraining the continuous latent variables z sampled from the prior distribution p(z|c) for inference.
In practically, the continuous latent variables effectively help dialogue models to generate diverse responses. Nevertheless, owing to the one-to-many and many-to-one phenomena, the continuous latent variables frequently struggle to encapsulate the precise contextual semantics, leading to responses that are irrelevant and lack coherence (Sun et al., 2021). Different from the continuous latent variables, discrete latent variables are better at producing relevant and coherent responses. For example, Bao et al. (2020) uses Latent Act Recognition to model the relationship between discrete latent variables and multiple responses, and proposes Response Selection to chose the generated responses of most coherent with the context. However, owing to their limited scale, discrete latent variables might encapsulate a narrower range of features compared to their continuous counterparts.
Therefore, combining continuous and discrete latent variables presents a promising direction. By doing so, the strengths of each can be harnessed and their weaknesses mitigated, allowing for a more
balanced approach that capitalizes on the diversity provided by continuous variables and the specificity afforded by discrete variables. This hybrid approach could potentially lead to more robust and nuanced models that better capture the complexities of the data they are designed to represent. Based on this, (Sun et al., 2023) propose a hybrid latent variable strategy and a Conditional Hybrid Variational Transformer (CHVT) for dialogue generation task. Different from the CHVT, our CV-IME focus on the pinyin-to-characters task. Owing to the pronounced alignment between the input pinyin sequences and the target character sequences within IME data, the conventional approach to information interchange employed during the construction of continuous latent variables in the CHVT framework can inadvertently overlook salient character sequence details. This oversight has the potential to compromise model performance. In response to this challenge, we introduce an innovative strategy for the formulation of continuous latent variables. This strategy is designed to intensify the interaction of information between pinyin and character sequences, consequently bolstering the efficacy of the training phase.

heading: A.3 Generation Methods
text: Beam Search (BS), a popular breadth-first decoding method, is widely used in text generation task. Unfortunately, they inherently exhibit a deficiency in diversity, which frequently results in performance degradation within human-like contexts (Holtzman et al., 2020). Additionally, BS necessitates the computation of cumulative scores for each candidate during the decoding process, and concurrently requires the sorting and recombination of samples, thereby augmenting the computational burden. Under conditions of constrained computational resources, this may impede the realization of its advantages.
To enrich the diversity of BS, stochastic decoding strategies are introduced in the generation phrase. Ancestral sampling (AS) (Bishop and Nasrabadi, 2006) is the most straightforward but less effective sampling method. Temperature sampling (Ackley et al., 1985) is an improvement of AS, which introduces temperature to shape the probability distribution. However, due to the randomness, both of them will damage the quality of generated results. To mitigate this problem, top-k (Fan et al., 2018), nucleus sampling (Holtzman et al., 2020), and locally typical (Meister
et al., 2022) sampling are proposed to truncate the distributions, which aim at improving quality while preserving diversity. However, the truncation and re-scaling of probabilities also demand additional computational effort, similarly presenting challenges with respect to latency.
Diverging from the aforementioned approaches, we introduce a conditional variational mechanism into the IME model, optimizing the sampling process through adjustments to the model structure, while exclusively employing greedy search to circumvent additional computational overhead. Leveraging the latent variables in sampling, CV-IME is capable of enhancing the diversity of generated results under conditions of limited latency.

heading: B Method
text: Construction of Continuous Latent Variables. The prior and recognition network are responsible for estimating the prior and the posterior distribution of continues latent variables. We first use the Transformer encoder to encode the input sequence (x = x1, x2, · · · , xn) to obtain its final hidden state (h = h1, h2, · · · , hn) as prior memory, where n denotes the length of c. Then, we use the same encoder to encode the input and target sequence (x′ = x1, · · · , xn, · · · , xn+m) to obtain the posterior memory (h′ = h′1, · · · , h′n, · · · , h′n+m), where m means the length of r. Next, we use prior memory to recompute the posterior memory:
h′ = SoftMax(h · h′T ) · h′ (2)
Finally, similar with previous works (Bowman et al., 2016; Zhao et al., 2017; Shen et al., 2017) that assume z follows isotropic Gaussian distribution, we use fully-connected networks as pθ(z|c) ∼ N (µ, σ2I) and qϕ(z|c, r) ∼ N (µ′, σ′2I) :
( µ1, ..., µn
log(σ21), ..., log(σ 2 n)
) = tanh(   h1 · · · hn  Wd)Wu
( µ′1, ..., µ ′ n
log(σ′21 ), ..., log(σ ′2 n )
) =   [h1;h ′ 1]
· · · [hn;h ′ n]
 W ′u ,
whereW{d,u},W ′u are trainable parameters of prior network and recognition network. At this point we have n token-level probability distributions for n tokens in c. To take full use of these distributions, we follow the additive Gaussian mixing (Wang et al.,
2017) to compute the sentence-level distribution:
pθ(zs|c) ∼ N ( n∑
i=1
wiµi, n∏
i=1
σ2wii )
qϕ(z ′ s|c, r) ∼ N (
n∑
i=1
wiµ ′ i,
n∏
i=1
σ′2wii ) ,
where zs represents the sentence-level latent variable, wi denotes the weight of the i-th distribution.
Finally, we use the reparameterization trick (Kingma and Welling, 2014; Zhao et al., 2017) to obtain samples of zs either from p(zs|c, r) (training) or p(zs|c) (inference). The sentence-level latent variable zs will be used for constructing the hybrid latent variable afterwards.

heading: Construction of Hybrid Latent Variables. To
text: build the hybrid latent variables H , during training, we first sample the z′s from the p(z ′ s|c, r) and then expandedK times that make it added to the discrete latent variables M :
H =
  z′s +M [1]
· · · z′s +M [K]
  ,
where K represents the number of discrete latent variables.
Loss Function. During training, CV-IME introduce the self-separation training and aims to maximizing the variational lower bound of the conditional log likelihood (Kingma and Welling, 2014; Sohn et al., 2015; Yan et al., 2016):
L(θ, ϕ,Ω,M ; r, c)
=
K∑
i=1
αiEqϕ(z′s|r,c) [log p(r|[Hi; c])]
− λDKL(qϕ(z′s|r, c)||pθ(zs|c))
αi = { 1 if Ei = max(E1, · · · ,EK) 0 otherwise
Ei = Eqϕ(z′s|r,c)[log p(r|[Hi; c])],
where θ, ϕ, ψ,Ω,M are parameters of CV-IME, and λ is the scale factor of KL divergence.
Inference Phase. During inference, CV-IME use the prior distribution p(zs|c) to sample the sentence-level continuous latent variable zs and mix zs with discrete latent variables to construct hybrid latent variables. Based on the K discrete latent variables, CV-IME can directly generate K results for the same input.

heading: C Experimental Settings
text: Benchmarks. We used two benchmarks: (1) PD benchmark, a commonly used benchmark dataset for the Chinese IME task, is extracted from the People’s Daily from 1992 to 1998 that has word segmentation annotations by Peking University. It contains 2,000 segments of consecutive Chinese characters for testing. For each test case, the input pinyin are all perfect pinyin and the context is null.
(2) WD benchmarks is extracted from the WuDaoCorpora (Yuan et al., 2021) that contains 3TB Chinese corpus collected from 822 million Web pages. Tan et al. (2022) randomly select 16 domains from WuDaoCorpora, and segment those documents into sentences. For each sentence, they randomly selected a context ranging from 0-3, 4-9, and 10+ words, while continuously selecting a target of 1-3, 4-9, or 10+ words. Each context-target length tuple like (0-3, 1-3) serves as an evaluation configuration and contains 2,000 test instances.
Training Data. To train our CV-IME and the base model, we built a new pinyin-to-character dataset based on the news2016 corpus5. The news2016 corpus comprises 2.5 million news articles, each containing keywords and descriptions. Initially, we extract paragraphs from the data and segment them into sentences using periods, exclamation points, and question marks as delimiters. Subsequently, we divide each sentence from the end into two parts: context and target, omitting segmentation points where the target part includes numbers or special characters. Following this, we retain the case with a probability of 50%. Subsequent to the initial processing, we incorporate the ’pypinyin’ package to construct Pinyin sequences for the target portion of the retained cases. For each case, we determine with a 50% probability whether the data will represent a ’perfect’ Pinyin or an ’abbreviated’ Pinyin. Ultimately, we retained a total of 19,330,170 training samples. Table 1 shows the statistics of our training set.
Training Detail. The hidden size of all models is set to 512. The base model consists of 4 layers of encoder and 1 layer of decoder. The CV-IME employs a Transformer model with 2 encoder layers and 1 decoder layer, and additionally incorporates two fully-connected layers as a prior network. The maximum length of input sequence (pinyin + context) and result are set to 42 and 20, respectively.
5https://github.com/brightmart/nlp_chinese_corpus
We set the batch sizes to 1024 and 256 for base model and CV-IME, respectively. Adam is used for optimization. The initial learning rate is set to 0.0001. We also introduce KL annealing trick to leverage the KL divergence during the training. The KL weight increases linearly from 0 to 1 in the first 3000000 batches. We train all models in 100 epochs on four A100 GPU cards with Pytorch, and save the model parameters when the validation loss reaching minimum.

heading: D Experimental Results

heading: D.1 Model Structure Selection
text: Figure 4 elucidates the following points: (1) Merely augmenting the number of layers in the decoder significantly increases the generation latency without
improving the accuracy of the generated results; (2) A model with a single-layer encoder and a sixlayer decoder exhibits a base latency of 69.62 on CPU devices, yet its accuracy is inferior to that of a model with a four-layer encoder and a single-layer decoder; (3) Increasing the number of encoder layers effectively enhances the accuracy of the generated results. Consequently, retaining a single-layer decoder offers the best cost-effectiveness in lowresource scenarios, and, where possible, augmenting the number of encoder layers under constrained conditions contributes to improved accuracy.

heading: D.2 PD Benchmark
text: Table 7 and Table 8 show the results of different context-target length configurations over PD benchmakrs. Sample-i means that the target in the set
contains i tokens, and Sample means that the number of tokens in the context in this data set is 0, and all tokens are in the target. CV-IME-i means the results of CV-IME using i-th hybrid latent variable. Base+Beam represents the results of base model with beam search. In the tables presented, we observe that our model outperforms across nearly all configurations of context-target lengths.

heading: D.3 WD Benchmark
text: We conducted experiments on the WD dataset across different domains and with various contexttarget length configurations.
Table 9, Table 10, Table 11 and Table 12 report the results of different domains over WD. From these tables, it can be observed that: (1) Our CV-
IME model consistently outperforms the baseline model across various domains. (2) The improvement ratio varies across different domains, ranging from a minimum of 0.75 points to a maximum of 4.97 points. We hypothesize that this variability may be attributed to the fact that the training data is extracted from news data, which differs in domain information from the various domains in WD.
Table 13 presents the results of different contexttarget length configurations over WD. The data from the tables indicate the following observations: (1) CV-IME model achieves superior performance over the baseline in most configurations; (2) As the length of the target increases, the difficulty of achieving an exact match between the generated results and the ground truth progressively rises; (3) Extending the length of the context portion effectively enhances the accuracy of the generated outcomes; (4) CV-IME model exhibits improved performance in the target length phases of 0-3 and 4-9, yet its performance diminishes in scenarios where the target length exceeds 10. This may be attributed to the diversity introduced by latent variables, which leads to a discrepancy between the generated content and the ground-truth.

heading: D.4 Case Study
text: Table 14 presents examples of perfect Pinyin mode in 9-key IME. From the table, it can be observed that our CV-IME can recall more diverse and accurate results within similar generation time constraints. However, since CV-IME utilizes unsupervised training of latent variables, the 4 generated results from CV-IME only represent the corresponding mixed latent variables and do not imply the priority of the results. Thus, identifying a timeefficient sorting method is our future research.

heading: Future Work
text: Thus, identifying a timeefficient sorting method is our future research