Introduction	Recently, pre-trained language representation has shown to be useful for improving many NLP tasks , , , . Embeddings from Language Models (ELMo)  is one of the most outstanding works, which uses a character-aware language model to augment word representation. An essential challenge in training word-based language models is how to control vocabulary size for better rare word representation. No matter how large the vocabulary is, rare words are always insufficiently trained. Besides, an extensive vocabulary takes too much time and computational resource for the model to converge. Whereas, if the vocabulary is too small, the out-of-vocabulary (OOV) issue will harm the model performance heavily . To obtain effective word representation,  introduce character-driven word embedding using convolutional neural network (CNN) , following the language model in  for deep contextual representation. However, there is potential insufficiency when modeling word from characters which hold little linguistic sense, especially, the morphological source . Only 86 characters(also included some common punctuations) are adopted in English writing, making the input too coarse for embedding learning.

Subword from Unsupervised Segmentation	To segment subwords from a word, we adopt the generalized unsupervised segmentation framework proposed by . The generalized framework can be divided into two collocative parts, goodness measure (score), which evaluates how likely a subword is to be a ‘proper’ one, and a segmentation or decoding algorithm. For the sake of simplicity, we choose frequency as the goodness score and two representative decoding algorithms, byte pair encoding (BPE)  which uses a greedy decoding algorithm and unigram language model (ULM)  which adopts a Viterbi-style decoding algorithm. For a group of character sequences, the working procedure of BPE is as follows: $\bullet $ All the input sequences are tokenized into a sequence of single-character subwords. $\bullet $ Repeatedly, we calculate the frequencies of all bigrams and merge the bigram with the highest one until we get the desired subword vocabulary. ULM is proposed based on the assumption that each subword occurs independently. The working procedure of ULM segmentation is as follows. $\bullet $ Heuristically make a reasonably large seed vocabulary from the training corpus.

General Language Model	The overall architecture of our subword-aware language model shows in Figure FIGREF1. It consists of four parts, word segmentation, word-level CNN, highway network and sentence-level RNN. Given a sentence $S = \lbrace W_1, W_2, ... , W_n\rbrace $, we first use a segmentation algorithm to divide each word into a sequence of subwords , . where $M_i$ is the output of the segmentation algorithm, $x_{i, j}$ is the subword unit and $f$ represents the segmentation algorithm. Then a look-up table is applied to transform the subword sequence into subword embeddings . To further augment the word representation from the subwords, we apply a narrow convolution between subword embeddings and several kernels. where $Concat$ is the concatenation operation for all the input vectors, $\mathbf {K}_i$ is convolution kernel and $g$ is CNN-MaxPooling operation. A highway network  is then applied to the output of CNN. A bidirectional long short-term memory network (Bi-LSTM)  generates the hidden states for the given sentence representation in forward and backward.

Discussion	Subword Vocabulary Size Tables TABREF5 and TABREF10 show the performance of ESuLMo drops with the vocabulary size increases . We explain the trend that neural network pipeline especially CNN would fail to capture necessary details of building word embeddings as more subwords are introduced. Subword Segmentation Algorithms Tables TABREF5 and TABREF10 show that ESuLMo based on both ULM and BPE segmentation with 500 subwords outperform the original ELMo, and BPE is consistently better than ULM on all evaluations under the same settings. We notice that BPE can give static subword segmentation for the same word in different sentences, while ULM cannot. It suggests that ESuLMo is sensitive to segmentation consistency. We also analyze the subword vocabularies from two algorithms and find that the overlapping rates for 500, 1K and 2K sizes are 60.2%, 55.1% and 51.9% respectively. This indicates subword mechanism can stably work in different vocabularies. Task Independent vs. Task Specific To discover the necessary training progress, we show the accuracy in SNLI and PPL for language model in Figure FIGREF15.

$\bullet $ Iteratively, the probability of each subword is estimated by the expectation maximization (EM) algorithm and the top $\eta \%$ subwords with the highest probabilities are kept. Note that we always keep the single character in subword vocabulary to avoid out-of-vocabulary. For a specific dataset, the BPE algorithm keeps the same segmentation for the same word in different sequences, whereas ULM cannot promise such segmentation. Both segmentation algorithms have their strengths,  show that BPE can fix the OOV issue well, and  proves that ULM is a subword regularization which is helpful in neural machine translation.

Experiments ::: Language Model	In this section, we examine the pre-trained language models of ESuLMo in terms of PPL. All the models' training and evaluation are done on One Billion Word dataset  . During training, we strictly follow the same hyper-parameter published by ELMo, including the hidden size, embedding size, and the number of LSTM layers. Meanwhile, we train each model on 4 Nvidia P40 GPUs, which takes about three days for each epoch. Table TABREF5 shows that our pre-trained language models can improve the performance of RNN-based language models by a large margin and our subword-aware language models outperform all previous RNN-based language models, including ELMo, in terms of PPL. During the experiment, we find that 500 is the best vocabulary size for both segmentation algorithms, and BPE is better than ULM in our setting.

As we argue that for better representation from a refined granularity, word is too large and character is too small, it is natural for us to consider subword unit between character and word levels. Splitting a word into subwords and using them to augment the word representation may recover the latent syntactic or semantic information . For example, uselessness could be split into the following subwords: $<$use, less, ness$>$. Previous work usually considers linguistic knowledge-based methods to tokenize each word into subwords (namely, morphemes) , , . However, such treatment may encounter three main inconveniences. First, the subwords from linguistic knowledge, typically including the morphological suffix, prefix, and stem, may not be suitable for a targeted NLP task  or mislead the representation of some words, like the meaning of understand cannot be formed by under and stand. Second, linguistic knowledge, including related annotated lexicons or corpora, may not even be available for a specific low-resource language. Due to these limitations, we focus on computationally motivated subword tokenization approaches in this work.

Subword ELMo	Embedding from Language Models (ELMo) has shown to be effective for improving many natural language processing (NLP) tasks, and ELMo takes character information to compose word representation to train language models.However, the character is an insufficient and unnatural linguistic unit for word representation.Thus we introduce Embedding from Subword-aware Language Models (ESuLMo) which learns word representation from subwords using unsupervised segmentation over words.We show that ESuLMo can enhance four benchmark NLP tasks more effectively than ELMo, including syntactic dependency parsing, semantic role labeling, implicit discourse relation recognition and textual entailment, which brings a meaningful improvement over ELMo.

Conclusion	In this paper, we present Embedding from Subword-aware Language Model (ESuLMo). The experiments show that the language models of ESuLMo outperform all RNN-based language models, including ELMo, in terms of PPL. The empirical evaluations in benchmark NLP tasks show that subwords can represent word better than characters to let ESuLMo more effectively promote downstream tasks than the original ELMo.

In this paper, we propose Embedding from Subword-aware Language Models (ESuLMo), which takes subword as input to augment word representation and release a sizeable pre-trained language model research communities. Evaluations show that the pre-trained language models of ESuLMo outperform all RNN-based language models, including ELMo, in terms of PPL and ESuLMo outperforms state-of-the-art results in three of four downstream NLP tasks.

The training curves show that our ESuLMo helps ESIM reach stable accuracy for SNLI while the corresponding PPL of the language model is far away from convergence. Word Sense Disambiguation To explore the word sense disambiguation capability of our ESuLMo, we isolate the representation encoded by our ESuLMo and use them to directly make predictions for a fine-grained word sense disambiguation (WSD) task. We choose the dataset and perform this experiment using the same setting as ELMo with only the last layer's representation. Table TABREF16 shows that our model can outperform the original ELMo.

Textual Entailment (TE) is a task to determine the relationship between a hypothesis and a premise. The Stanford Natural Language Inference (SNLI) corpus  provides approximately 550K hypothesis/premise pairs. Our baseline adopts ESIM  which uses a Bi-LSTM encoder layer and a Bi-LSTM inference composition layer which are connected by an attention layer to model the relation between hypothesis and premise. Our ESuLMo outperforms ELMo by 0.8% in terms of accuracy. Though our performance does not reach the state-of-the-art, it is second-best in all single models according to the SNLI leaderboard .

Our ESuLMo gets 96.65% UAS in PTB-SD 3.5.0, which is better than the state-of-the-art result . Semantic Role Labeling (SRL) is to model the predicate-argument structure of a sentence.  model SRL as a words pair classification problem and directly use a bi-affine scorer to predict the relation given two words in a sentence. By adding our ESuLMo to the baseline model , we can not only outperform the original ELMo by 0.5% F1-score but also outperform the state-of-the-art model  which has three times more parameters than our model in CoNLL 2009 benchmark dataset. Implicit Discourse Relation Recognition (IDRR) is a task to model the relation between two sentences without explicit connective.  use a hierarchical structure to capture four levels of information, including character, word, sentence and pair. We choose it as our baseline model for 11-way classification on PDTB 2.0 following 's setting. Our model outperforms ELMo significantly and reaches a new state-of-the-art result.

Experiments ::: Downstream Tasks	While applying our pre-trained ESuLMo to other NLP tasks, we have two different strategies: (1) Fine-tuning our ESuLMo while training other NLP tasks; (2) Fixing our ESuLMo while training other NLP tasks. During the experiment, we find there is no significant difference between these two strategies. However, the first strategy consumes much more resource than the second one. Therefore, we choose the second strategy to conduct all the remaining experiments. We apply ESuLMo to four benchmark NLP tasks. And we choose the fine-tuned model by validation set and report the results in the test set. The comparisons in Table TABREF10 show that ESuLMo outperforms ELMo significantly in all tasks and achieves the new state-of-the-art result in three of four tasks . Syntactic Dependency Parsing (SDP) is to disclose the dependency structure over a given sentence.  use a Bi-LSTM encoder and a bi-affine scorer to determine the relationship between two words in a sentence.

Experiments	The ESuLMo is evaluated in two ways, task independent and task dependent. For the former, we examine the perplexity of the pre-trained language models. For the latter, we examine on four benchmark NLP tasks, syntactic dependency parsing, semantic role labeling, implicity discourse relation recognition, and textual entailment.

Finally, the probability of each token is calculated by applying an affine transformation to all the hidden states followed by a $SoftMax$ function. During the training, our objective is to minimize the negative log-likelihood of all training samples. To apply our pre-trained language models to other NLP tasks, we combine the input vector and the last layer's hidden state of the Bi-LSTM to represent each word.

Question: How long is the vocabulary of subwords?
Answer: 500