Real-time Neural-based Input Method

The input method is an essential service on every mobile and desktop devices that provides text suggestions . It converts sequential keyboard inputs to the characters in its target language , which is indispensable for Japanese and Chinese users . Due to critical resource constraints and limited network bandwidth of the target devices , applying neural models to input method is not well explored . In this work , we apply a LSTM-based language model to input method and evaluate its performance for both prediction and conversion tasks with Japanese BCCWJ corpus . We articulate the bottleneck to be the slow softmax computation during conversion . To solve the issue , we propose incremental softmax approximation approach , which computes softmax with a selected subset vocabulary and fix the stale probabilities when the vocabulary is updated in future steps . We refer to this method as incremental selective softmax . The results show a two order speedup for the softmax computation when converting Japanese input sequences with a large vocabulary , reaching real-time speed on commodity CPU . We also exploit the model compressing potential to achieve a 92 % model size reduction without losing accuracy . 1 INTRODUCTION . Typing is one of the core activities users interact with devices . The input method is an intelligent service in operating systems that provides text suggestions to boost typing efficiency . With a large user base , commercial input products have helped users save trillions of key presses . Being able to provide an input method service in high quality makes a huge business impact . Among various features an input method provides , we consider two essential text suggestion tasks in this work : next word prediction task and conversion task . Next word prediction task relies on a language model , which gives the probability of the next word p ( wt|w1 : t−1 ) . Usually , top 3 or 5 candidates are shown to users . Conversion task , however , is not a common scenario for Latin users . Latin users can type directly with a keyboard , but there are thousands of characters in the Chinese and Japanese language . Conversion task takes a key sequence and converts it to target word sequence that best matches users ’ original intentions . It is a sequence decoding task similar to machine translation . Conventionally , an n-gram language model is used to solve these two tasks ( Stolcke , 2002 ) . However , due to the exponential increase in model size , the n-gram model can only support up to tri-gram . The decoding results often fail to have global consistency as the tri-gram language model is not capable of considering distant context . Also , n-gram models are often pruned to meet model size requirement with a loss in accuracy . In contrast , neural-based language models are proven to solve such issues more efficiently ( Bengio et al. , 2003 ; Mikolov et al. , 2010 ; Jozefowicz et al. , 2016 ) . Although neural-based language model has achieved enormous success , conventionally , it is not considered feasible for input method task . One major reason is that the input method has to run interactively at user devices with critical constraints of resources and run-time speed . This is different from speech recognition or machine translation services that are normally served from online servers . Input applications have to sacrifice accuracy for execution speed and model size . In this work , we enhance a neural language model tailored for input method task to meet both runtime speed and model size requirement . Our baseline model is composed of a LSTM-based language model ( Hochreiter & Schmidhuber , 1997 ) with a Viterbi decoder ( Forney , 1973 ) as Figure 1 shows . It improves the performance for both conversion and next word prediction task and is capable of capturing distant context . For languages like Japanese and Chinese , a large vocabulary is required in the conversion task . we articulate the bottleneck of run-time speed as the vocabulary size matrix operation before softmax . We propose an incremental selective softmax approach that builds a subset vocabulary Vt at each decoding step t. By only computing softmax over Vt , the cost of softmax significantly drops since |Vt| is usually a small number that is less than 1 % of the original vocabulary size . We evaluate the speedup comparing to other softmax optimization . To further handle the model disk size and memory size , we apply k-means based quantization algorithm and analyze its performance with various compression ratio . We summarize the main contributions of this paper as follows : 1 ) We empirically evaluate using a LSTM-based language model to address input method challenges and demonstrate its performance for a complex language with large vocabulary size . 2 ) We propose an effective approach to accelerate the inference time with incremental selective softmax . By avoiding computing the full softmax , our approach achieves two orders inference speed boost without negatively impact the accuracy of conversion . 3 ) To let the neural language model with a large vocabulary fit into the execution environment on consumer devices , we compress the model with quantization techniques , achieving a 92 % reduction on model size . 2 NEURAL-BASED INPUT METHOD . The neural-based approach we propose for input method is illustrated in Figure 1 . We use LSTM language model for both the conversion and prediction tasks . A noticeable difference from the ngram approach is the use of a latent vector to capture input context . It is the hidden vector from the last recurrent unit of the previous sequence . In a real scenario , users don ’ t type the full sentence in one effort . Instead , they type and convert in fragments , where each fragment contains a few words . It is important to carry the context of the previously determined results to current conversion task . Conversion task is described in Figure 1 ( b ) . As the user starts to type in an editor control , the input method first converts the alphabet user typed into syllables . Take Japanese as an example , Japanese Kana , also called “ fifty sounds ” , can be converted from the alphabet with fixed rules . In lexicon , Japanese words are indexed by Kana characters with an efficient structure such as Trie . A word lattice is then built with all possible words from a Kana sequence by lexicon lookup . To find the best combination of words aligned with the Kana sequence , we use a Viterbi decoder to decode with a beam size S. LSTM language model is used here to evaluate path probabilities . The best path is then shown to the user as the conversion candidate . Once the user selects the conversion candidate , last hidden vector of the best path becomes the new context vector . We compute the softmax based on the context vector and provides a probability distribution of the next words as shown in Figure 1 ( c ) . Although several commercial input products have already tried to apply neural models1 , they mainly aim to solve the prediction task for Latin languages . Conversion task has a different scale of computational cost and is not yet tackled . 3 MODEL ACCELERATION . 3.1 BACKGROUND . Comparing to a prediction task , the conversion task is even more costly to compute . We further articulate three issues that make the conversion task difficult to solve : 1 ) For Japanese or Chinese input method , a much larger vocabulary size is required , which is usually several times larger than English . 2 ) During decoding , we have to evaluate probabilities for all paths with a beam size S. 3 ) Most importantly , there are no natural word boundaries in Chinese or Japanese , such as a space character . The path evaluation has to run each time a new key input arrives , instead of when space is inserted . With all above constraints , the conversion task has to compute S × |V | × |X| steps of LSTM computation for one input sequence , where |V | is the vocabulary size , |X| is the sequence length . Heavy computation cost of LSTM model comes from two operations : the matrix operations inside LSTM cell and the matrix projection at softmax . In the case of a LSTM model with a vocabulary size of 100K , a hidden size of 512 , and an embedding size of 256 , estimated by simple matrix multiplication , the number of multiplication operations for LSTM cell is about 1.5M , while the softmax has 50M operations . In practice , the softmax occupies 97 % of the total computation cost , which is the bottleneck for the conversion task with a large vocabulary size . 3.2 INCREMENTAL SELECTIVE SOFTMAX . To solve the challenges , we propose an incremental selective softmax approach to obtain correct probabilities for paths in the lattice . Given a partial observed character input sequence ( x1 , ... , xt ) , we search for a set of words in the dictionary that match any suffix of the observation : Dt = t⋃ i=1 match ( xi , ... , xt ) , ( 1 ) where the function match ( · ) returns all lexicon items matching the partial sequence . For example , given the observation “ ha ru ka ” , the lexicon set Dt contains all words matching “ ha ru ka ” or “ ru ka ” or “ ka ” . With the lexicon set in each step , we can construct a subset vocabulary Vt that covers all possible output words until step t as : Vt = t⋃ i=1 Di . ( 2 ) In this work , we refer to Vt as the lattice vocabulary . In contrast to the beam search in machine translation , we have to rank paths that contain different number of words . If we directly compute path probabilities on Vt with a language model , as the denominator of the softmax becomes smaller , all words will have higher probabilities . Therefore , the paths containing more words will have advantage due to the incorrect probabilities . As a result , the paths can be ranked wrongly . To circumvent the probability issue , we also sample a subset vocabulary from the full vocabulary , denoted by Ṽ . We use Vt jointly with Ṽ to make the word probabilities closer to their original probabilities , thus minimizing the impact of exposure bias . 1SwiftKey , Fleksy , etc . As shown in Figure 2 ( a ) , we experiment two sampling methods . The first one samples mostly frequent words from the vocabulary , whereas the second one uniformly samples from the full vocabulary . We found the first sampling method ( top sampling ) delivers the best performance . In practice , both Vt and Ṽ have less than 1 % of the words in the original vocabulary . Our proposed softmax approximation technique has two passes . When a new input character xt+1 arrives , we merge the current vocabulary Vt with new words from the lexicon set Dt+1 , forming a new vocabulary Vt+1 . Then for each path in the lattice , we compute the language model probability at step t+ 1 with the joint vocabulary Vt+1 ∪ Ṽ as : P ( yt = i|ht ) = exp ( h > t Wi ) ∑ j ∈ Vt+1 ∪ Ṽ exp ( h > t Wj ) , ( 3 ) where ht is the LSTM hidden state in step t. In this pass , we only compute the word probabilities for top paths in batch with a beam size S. Now we computed the latest word probabilities using the lattice vocabulary Vt+1 . Unfortunately , as previous word probabilities are computed on old vocabularies , they are not comparable with the new ones . Therefore , in the second pass , we correct the stale word probabilities to let them have the same denominator as the ones computed on the latest lattice vocabulary . In principle , we can correct an old probability in step k by adding the logits of missing vocabulary to the denominator as : P ( yk = i|hk ) = exp ( h > k Wi ) ∑ j ∈ Vk+1 ∪ Ṽ exp ( h > k Wj ) + ∑ j ∈ Vt+1 , j /∈ Vk+1 ∪ Ṽ exp ( h > k Wj ) . ( 4 ) However , in practice , as each path has different missing vocabularies , we compute a union of all missing vocabularies , and then recompute the logits of them in batch . Experiments show that our proposed incremental selective softmax strategy achieves a 76x speedup comparing to the baseline .

Summary: The paper demonstrates the main challenge of using LSTM-based language models for input method in real time is the huge amount of computation in the softmax. The authors present a system to speed up the inference by avoiding computing the full softmax in the Japanese conversion task, where the number of output words can be limited from the mapping of the input sequence through a lexicon. The experiment result is encouraging in that the proposed incremental selective softmax approach significantly reduces latency over the standard inference with the full softmax computation while not hurting accuracy much. The paper also evaluates the effect of quantization for LSTM LM model compression in terms of size and accuracy.