Introduction
Recently, pre-trained language representation has shown to be useful for improving many NLP tasks BIBREF0, BIBREF1, BIBREF2, BIBREF3. Embeddings from Language Models (ELMo) BIBREF0 is one of the most outstanding works, which uses a character-aware language model to augment word representation.
An essential challenge in training word-based language models is how to control vocabulary size for better rare word representation. No matter how large the vocabulary is, rare words are always insufficiently trained. Besides, an extensive vocabulary takes too much time and computational resource for the model to converge. Whereas, if the vocabulary is too small, the out-of-vocabulary (OOV) issue will harm the model performance heavily BIBREF4. To obtain effective word representation, BIBREF4 introduce character-driven word embedding using convolutional neural network (CNN) BIBREF5, following the language model in BIBREF6 for deep contextual representation.
However, there is potential insufficiency when modeling word from characters which hold little linguistic sense, especially, the morphological source BIBREF7. Only 86 characters(also included some common punctuations) are adopted in English writing, making the input too coarse for embedding learning. As we argue that for better representation from a refined granularity, word is too large and character is too small, it is natural for us to consider subword unit between character and word levels.
Splitting a word into subwords and using them to augment the word representation may recover the latent syntactic or semantic information BIBREF8. For example, uselessness could be split into the following subwords: $<$use, less, ness$>$. Previous work usually considers linguistic knowledge-based methods to tokenize each word into subwords (namely, morphemes) BIBREF9, BIBREF10, BIBREF11. However, such treatment may encounter three main inconveniences. First, the subwords from linguistic knowledge, typically including the morphological suffix, prefix, and stem, may not be suitable for a targeted NLP task BIBREF12 or mislead the representation of some words, like the meaning of understand cannot be formed by under and stand. Second, linguistic knowledge, including related annotated lexicons or corpora, may not even be available for a specific low-resource language. Due to these limitations, we focus on computationally motivated subword tokenization approaches in this work.
In this paper, we propose Embedding from Subword-aware Language Models (ESuLMo), which takes subword as input to augment word representation and release a sizeable pre-trained language model research communities. Evaluations show that the pre-trained language models of ESuLMo outperform all RNN-based language models, including ELMo, in terms of PPL and ESuLMo outperforms state-of-the-art results in three of four downstream NLP tasks.
General Language Model
The overall architecture of our subword-aware language model shows in Figure FIGREF1. It consists of four parts, word segmentation, word-level CNN, highway network and sentence-level RNN.
Given a sentence $S = \lbrace W_1, W_2, ... , W_n\rbrace $, we first use a segmentation algorithm to divide each word into a sequence of subwords BIBREF13, BIBREF14.
where $M_i$ is the output of the segmentation algorithm, $x_{i, j}$ is the subword unit and $f$ represents the segmentation algorithm. Then a look-up table is applied to transform the subword sequence into subword embeddings BIBREF15.
To further augment the word representation from the subwords, we apply a narrow convolution between subword embeddings and several kernels.
where $Concat$ is the concatenation operation for all the input vectors, $\mathbf {K}_i$ is convolution kernel and $g$ is CNN-MaxPooling operation.
A highway network BIBREF16 is then applied to the output of CNN. A bidirectional long short-term memory network (Bi-LSTM) BIBREF17 generates the hidden states for the given sentence representation in forward and backward. Finally, the probability of each token is calculated by applying an affine transformation to all the hidden states followed by a $SoftMax$ function. During the training, our objective is to minimize the negative log-likelihood of all training samples.
To apply our pre-trained language models to other NLP tasks, we combine the input vector and the last layer's hidden state of the Bi-LSTM to represent each word.
Subword from Unsupervised Segmentation
To segment subwords from a word, we adopt the generalized unsupervised segmentation framework proposed by BIBREF18. The generalized framework can be divided into two collocative parts, goodness measure (score), which evaluates how likely a subword is to be a ‘proper’ one, and a segmentation or decoding algorithm. For the sake of simplicity, we choose frequency as the goodness score and two representative decoding algorithms, byte pair encoding (BPE) BIBREF13 which uses a greedy decoding algorithm and unigram language model (ULM) BIBREF14 which adopts a Viterbi-style decoding algorithm.
For a group of character sequences, the working procedure of BPE is as follows:
$\bullet $ All the input sequences are tokenized into a sequence of single-character subwords.
$\bullet $ Repeatedly, we calculate the frequencies of all bigrams and merge the bigram with the highest one until we get the desired subword vocabulary.
ULM is proposed based on the assumption that each subword occurs independently. The working procedure of ULM segmentation is as follows.
$\bullet $ Heuristically make a reasonably large seed vocabulary from the training corpus.
$\bullet $ Iteratively, the probability of each subword is estimated by the expectation maximization (EM) algorithm and the top $\eta \%$ subwords with the highest probabilities are kept. Note that we always keep the single character in subword vocabulary to avoid out-of-vocabulary.
For a specific dataset, the BPE algorithm keeps the same segmentation for the same word in different sequences, whereas ULM cannot promise such segmentation. Both segmentation algorithms have their strengths, BIBREF13 show that BPE can fix the OOV issue well, and BIBREF14 proves that ULM is a subword regularization which is helpful in neural machine translation.
Experiments
The ESuLMo is evaluated in two ways, task independent and task dependent. For the former, we examine the perplexity of the pre-trained language models. For the latter, we examine on four benchmark NLP tasks, syntactic dependency parsing, semantic role labeling, implicity discourse relation recognition, and textual entailment.
Experiments ::: Language Model
In this section, we examine the pre-trained language models of ESuLMo in terms of PPL. All the models' training and evaluation are done on One Billion Word dataset BIBREF19 . During training, we strictly follow the same hyper-parameter published by ELMo, including the hidden size, embedding size, and the number of LSTM layers. Meanwhile, we train each model on 4 Nvidia P40 GPUs, which takes about three days for each epoch. Table TABREF5 shows that our pre-trained language models can improve the performance of RNN-based language models by a large margin and our subword-aware language models outperform all previous RNN-based language models, including ELMo, in terms of PPL. During the experiment, we find that 500 is the best vocabulary size for both segmentation algorithms, and BPE is better than ULM in our setting.
Experiments ::: Downstream Tasks
While applying our pre-trained ESuLMo to other NLP tasks, we have two different strategies: (1) Fine-tuning our ESuLMo while training other NLP tasks; (2) Fixing our ESuLMo while training other NLP tasks. During the experiment, we find there is no significant difference between these two strategies. However, the first strategy consumes much more resource than the second one. Therefore, we choose the second strategy to conduct all the remaining experiments.
We apply ESuLMo to four benchmark NLP tasks. And we choose the fine-tuned model by validation set and report the results in the test set. The comparisons in Table TABREF10 show that ESuLMo outperforms ELMo significantly in all tasks and achieves the new state-of-the-art result in three of four tasks .
Syntactic Dependency Parsing (SDP) is to disclose the dependency structure over a given sentence. BIBREF20 use a Bi-LSTM encoder and a bi-affine scorer to determine the relationship between two words in a sentence. Our ESuLMo gets 96.65% UAS in PTB-SD 3.5.0, which is better than the state-of-the-art result BIBREF21.
Semantic Role Labeling (SRL) is to model the predicate-argument structure of a sentence. BIBREF22 model SRL as a words pair classification problem and directly use a bi-affine scorer to predict the relation given two words in a sentence. By adding our ESuLMo to the baseline model BIBREF22, we can not only outperform the original ELMo by 0.5% F1-score but also outperform the state-of-the-art model BIBREF23 which has three times more parameters than our model in CoNLL 2009 benchmark dataset.
Implicit Discourse Relation Recognition (IDRR) is a task to model the relation between two sentences without explicit connective. BIBREF24 use a hierarchical structure to capture four levels of information, including character, word, sentence and pair. We choose it as our baseline model for 11-way classification on PDTB 2.0 following BIBREF25's setting. Our model outperforms ELMo significantly and reaches a new state-of-the-art result.
Textual Entailment (TE) is a task to determine the relationship between a hypothesis and a premise. The Stanford Natural Language Inference (SNLI) corpus BIBREF26 provides approximately 550K hypothesis/premise pairs. Our baseline adopts ESIM BIBREF27 which uses a Bi-LSTM encoder layer and a Bi-LSTM inference composition layer which are connected by an attention layer to model the relation between hypothesis and premise. Our ESuLMo outperforms ELMo by 0.8% in terms of accuracy. Though our performance does not reach the state-of-the-art, it is second-best in all single models according to the SNLI leaderboard .
Discussion
Subword Vocabulary Size Tables TABREF5 and TABREF10 show the performance of ESuLMo drops with the vocabulary size increases . We explain the trend that neural network pipeline especially CNN would fail to capture necessary details of building word embeddings as more subwords are introduced.
Subword Segmentation Algorithms Tables TABREF5 and TABREF10 show that ESuLMo based on both ULM and BPE segmentation with 500 subwords outperform the original ELMo, and BPE is consistently better than ULM on all evaluations under the same settings. We notice that BPE can give static subword segmentation for the same word in different sentences, while ULM cannot. It suggests that ESuLMo is sensitive to segmentation consistency.
We also analyze the subword vocabularies from two algorithms and find that the overlapping rates for 500, 1K and 2K sizes are 60.2%, 55.1% and 51.9% respectively. This indicates subword mechanism can stably work in different vocabularies.
Task Independent vs. Task Specific To discover the necessary training progress, we show the accuracy in SNLI and PPL for language model in Figure FIGREF15. The training curves show that our ESuLMo helps ESIM reach stable accuracy for SNLI while the corresponding PPL of the language model is far away from convergence.
Word Sense Disambiguation To explore the word sense disambiguation capability of our ESuLMo, we isolate the representation encoded by our ESuLMo and use them to directly make predictions for a fine-grained word sense disambiguation (WSD) task. We choose the dataset and perform this experiment using the same setting as ELMo with only the last layer's representation. Table TABREF16 shows that our model can outperform the original ELMo.
Conclusion
In this paper, we present Embedding from Subword-aware Language Model (ESuLMo). The experiments show that the language models of ESuLMo outperform all RNN-based language models, including ELMo, in terms of PPL. The empirical evaluations in benchmark NLP tasks show that subwords can represent word better than characters to let ESuLMo more effectively promote downstream tasks than the original ELMo.

Question: What languages are evaluated?
Answer: Unanswerable

Unanswerable

Unanswerable

Question: Does the training of ESuLMo take longer compared to ELMo?
Answer: Unanswerable

Unanswerable

Question: How long is the vocabulary of subwords?
Answer: 500

500