Introduction
Incorporating sub-word structures like substrings, morphemes and characters to the creation of word representations significantly increases their quality as reflected both by intrinsic metrics and performance in a wide range of downstream tasks BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 .
The reason for this improvement is related to sub-word structures containing information that is usually ignored by standard word-level models. Indeed, when representing words as vectors extracted from a lookup table, semantically related words resulting from inflectional processes such as surf, surfing, and surfed, are treated as being independent from one another. Further, word-level embeddings do not account for derivational processes resulting in syntactically-similar words with different meanings such as break, breakable, and unbreakable. This causes derived words, which are usually less frequent, to have lower-quality (or no) vector representations.
Previous works have successfully combined character-level and word-level word representations, obtaining overall better results than using only word-level representations. For example BIBREF1 achieved state-of-the-art results in a machine translation task by representing unknown words as a composition of their characters. BIBREF4 created word representations by adding the vector representations of the words' surface forms and their morphemes ( INLINEFORM0 ), obtaining significant improvements on intrinsic evaluation tasks, word similarity and machine translation. BIBREF5 concatenated character-level and word-level representations for creating word representations, and then used them as input to their models for obtaining state-of-the-art results in Named Entity Recognition on several languages.
What these works have in common is that the models they describe first learn how to represent subword information, at character BIBREF1 , morpheme BIBREF4 , or substring BIBREF0 levels, and then combine these learned representations at the word level. The incorporation of information at a finer-grained hierarchy results in higher-quality modeling of rare words, morphological processes, and semantics BIBREF6 .
There is no consensus, however, on which combination method works better in which case, or how the choice of a combination method affects downstream performance, either measured intrinsically at the word level, or extrinsically at the sentence level.
In this paper we aim to provide some intuitions about how the choice of mechanism for combining character-level with word-level representations influences the quality of the final word representations, and the subsequent effect these have in the performance of downstream tasks. Our contributions are as follows:
Background
We are interested in studying different ways of combining word representations, obtained from different hierarchies, into a single word representation. Specifically, we want to study how combining word representations (1) taken directly from a word embedding lookup table, and (2) obtained from a function over the characters composing them, affects the quality of the final word representations.
Let INLINEFORM0 be a set, or vocabulary, of words with INLINEFORM1 elements, and INLINEFORM2 a vocabulary of characters with INLINEFORM3 elements. Further, let INLINEFORM4 be a sequence of words, and INLINEFORM5 be the sequence of characters composing INLINEFORM6 . Each token INLINEFORM7 can be represented as a vector INLINEFORM8 extracted directly from an embedding lookup table INLINEFORM9 , pre-trained or otherwise, and as a vector INLINEFORM10 built from the characters that compose it; in other words, INLINEFORM11 , where INLINEFORM12 is a function that maps a sequence of characters to a vector.
The methods for combining word and character-level representations we study, are of the form INLINEFORM0 where INLINEFORM1 is the final word representation.
Mapping Characters to Character-level Word Representations
The function INLINEFORM0 is composed of an embedding layer, an optional context function, and an aggregation function.
The embedding layer transforms each character INLINEFORM0 into a vector INLINEFORM1 of dimension INLINEFORM2 , by directly taking it from a trainable embedding lookup table INLINEFORM3 . We define the matrix representation of word INLINEFORM4 as INLINEFORM5 .
The context function takes INLINEFORM0 as input and returns a context-enriched matrix representation INLINEFORM1 , in which each INLINEFORM2 contains a measure of information about its context, and interactions with its neighbors. In particular, we chose to do this by feeding INLINEFORM3 to a BiLSTM BIBREF7 , BIBREF8 .
Informally, we can think of LSTM BIBREF10 as a function INLINEFORM0 that takes a matrix INLINEFORM1 as input and returns a context-enriched matrix representation INLINEFORM2 , where each INLINEFORM3 encodes information about the previous elements INLINEFORM4 .
A BiLSTM is simply composed of 2 LSTM, one that reads the input from left to right (forward), and another that does so from right to left (backward). The output of the forward and backward LSTM are INLINEFORM0 and INLINEFORM1 respectively. In the backward case the LSTM reads INLINEFORM2 first and INLINEFORM3 last, therefore INLINEFORM4 will encode the context from INLINEFORM5 .
The aggregation function takes the context-enriched matrix representation of word INLINEFORM0 for both directions, INLINEFORM1 and INLINEFORM2 , and returns a single vector INLINEFORM3 . To do so we followed BIBREF11 , and defined the character-level representation INLINEFORM4 of word INLINEFORM5 as the linear combination of the forward and backward last hidden states returned by the context function: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are trainable parameters, and INLINEFORM2 represents the concatenation operation between two vectors.
Combining Character and Word-level Representations
We tested three different methods for combining INLINEFORM0 with INLINEFORM1 : simple concatenation, a learned scalar gate BIBREF11 , and a learned vector gate (also referred to as feature-wise sigmoidal gate). Additionally, we compared these methods to two baselines: using pre-trained word vectors only, and using character-only features for representing words. See fig:methods for a visual description of the proposed methods.
word-only (w) considers only INLINEFORM0 and ignores INLINEFORM1 : DISPLAYFORM0
char-only (c) considers only INLINEFORM0 and ignores INLINEFORM1 : DISPLAYFORM0
concat (cat) concatenates both word and character-level representations: DISPLAYFORM0
scalar gate (sg) implements the scalar gating mechanism described by BIBREF11 : DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are trainable parameters, INLINEFORM2 , and INLINEFORM3 is the sigmoid function.
vector gate (vg): DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 are trainable parameters, INLINEFORM2 , INLINEFORM3 is the element-wise sigmoid function, INLINEFORM4 is the element-wise product for vectors, and INLINEFORM5 is a vector of ones.
The vector gate is inspired by BIBREF11 and BIBREF12 , but is different to the former in that the gating mechanism acts upon each dimension of the word and character-level vectors, and different to the latter in that it does not rely on external sources of information for calculating the gating mechanism.
Finally, note that word only and char only are special cases of both gating mechanisms: INLINEFORM0 (scalar gate) and INLINEFORM1 (vector gate) correspond to word only; INLINEFORM2 and INLINEFORM3 correspond to char only.
Obtaining Sentence Representations
To enable sentence-level classification we need to obtain a sentence representation from the word vectors INLINEFORM0 . We achieved this by using a BiLSTM with max pooling, which was shown to be a good universal sentence encoding mechanism BIBREF13 .
Let INLINEFORM0 , be an input sentence and INLINEFORM1 its matrix representation, where each INLINEFORM2 was obtained by one of the methods described in subsec:methods. INLINEFORM3 is the context-enriched matrix representation of INLINEFORM4 obtained by feeding INLINEFORM5 to a BiLSTM of output dimension INLINEFORM6 . Lastly, INLINEFORM11 is the final sentence representation of INLINEFORM12 obtained by max-pooling INLINEFORM13 along the sequence dimension.
Finally, we initialized the word representations INLINEFORM0 using GloVe embeddings BIBREF14 , and fine-tuned them during training. Refer to app:hyperparams for details on the other hyperparameters we used.
Experimental Setup
We trained our models for solving the Natural Language Inference (NLI) task in two datasets, SNLI BIBREF15 and MultiNLI BIBREF16 , and validated them in each corresponding development set (including the matched and mismatched development sets of MultiNLI).
For each dataset-method combination we trained 7 models initialized with different random seeds, and saved each when it reached its best validation accuracy. We then evaluated the quality of each trained model's word representations INLINEFORM0 in 10 word similarity tasks, using the system created by BIBREF17 .
Finally, we fed these obtained word vectors to a BiLSTM with max-pooling and evaluated the final sentence representations in 11 downstream transfer tasks BIBREF13 , BIBREF18 .
Datasets
Word-level Semantic Similarity A desirable property of vector representations of words is that semantically similar words should have similar vector representations. Assessing whether a set of word representations possesses this quality is referred to as the semantic similarity task. This is the most widely-used evaluation method for evaluating word representations, despite its shortcomings BIBREF20 .
This task consists of comparing the similarity between word vectors measured by a distance metric (usually cosine distance), with a similarity score obtained from human judgements. High correlation between these similarities is an indicator of good performance.
A problem with this formulation though, is that the definition of “similarity” often confounds the meaning of both similarity and relatedness. For example, cup and tea are related but dissimilar words, and this type of distinction is not always clear BIBREF21 , BIBREF22 .
To face the previous problem, we tested our methods in a wide variety of datasets, including some that explicitly model relatedness (WS353R), some that explicitly consider similarity (WS353S, SimLex999, SimVerb3500), and some where the distinction is not clear (MEN, MTurk287, MTurk771, RG, WS353). We also included the RareWords (RW) dataset for evaluating the quality of rare word representations. See appendix:datasets for a more complete description of the datasets we used.
Sentence-level Evaluation Tasks Unlike word-level representations, there is no consensus on the desirable properties sentence representations should have. In response to this, BIBREF13 created SentEval, a sentence representation evaluation benchmark designed for assessing how well sentence representations perform in various downstream tasks BIBREF23 .
Some of the datasets included in SentEval correspond to sentiment classification (CR, MPQA, MR, SST2, and SST5), subjectivity classification (SUBJ), question-type classification (TREC), recognizing textual entailment (SICK E), estimating semantic relatedness (SICK R), and measuring textual semantic similarity (STS16, STSB). The datasets are described by BIBREF13 , and we provide pointers to their original sources in the appendix table:sentence-eval-datasets.
To evaluate these sentence representations SentEval trained a linear model on top of them, and evaluated their performance in the validation sets accompanying each dataset. The only exception was the STS16 task, in which our representations were evaluated directly.
Word Similarity
table:wordlevelresults shows the quality of word representations in terms of the correlation between word similarity scores obtained by the proposed models and word similarity scores defined by humans.
First, we can see that for each task, character only models had significantly worse performance than every other model trained on the same dataset. The most likely explanation for this is that these models are the only ones that need to learn word representations from scratch, since they have no access to the global semantic knowledge encoded by the GloVe embeddings.
Further, bold results show the overall trend that vector gates outperformed the other methods regardless of training dataset. This implies that learning how to combine character and word-level representations at the dimension level produces word vector representations that capture a notion of word similarity and relatedness that is closer to that of humans.
Additionally, results from the MNLI row in general, and underlined results in particular, show that training on MultiNLI produces word representations better at capturing word similarity. This is probably due to MultiNLI data being richer than that of SNLI. Indeed, MultiNLI data was gathered from various sources (novels, reports, letters, and telephone conversations, among others), rather than the single image captions dataset from which SNLI was created.
Exceptions to the previous rule are models evaluated in MEN and RW. The former case can be explained by the MEN dataset containing only words that appear as image labels in the ESP-Game and MIRFLICKR-1M image datasets BIBREF24 , and therefore having data that is more closely distributed to SNLI than to MultiNLI.
More notably, in the RareWords dataset BIBREF25 , the word only, concat, and scalar gate methods performed equally, despite having been trained in different datasets ( INLINEFORM0 ), and the char only method performed significantly worse when trained in MultiNLI. The vector gate, however, performed significantly better than its counterpart trained in SNLI. These facts provide evidence that this method is capable of capturing linguistic phenomena that the other methods are unable to model.
table:word-similarity-dataset lists the word-similarity datasets and their corresponding reference. As mentioned in subsec:datasets, all the word-similarity datasets contain pairs of words annotated with similarity or relatedness scores, although this difference is not always explicit. Below we provide some details for each.
MEN contains 3000 annotated word pairs with integer scores ranging from 0 to 50. Words correspond to image labels appearing in the ESP-Game and MIRFLICKR-1M image datasets.
MTurk287 contains 287 annotated pairs with scores ranging from 1.0 to 5.0. It was created from words appearing in both DBpedia and in news articles from The New York Times.
MTurk771 contains 771 annotated pairs with scores ranging from 1.0 to 5.0, with words having synonymy, holonymy or meronymy relationships sampled from WordNet BIBREF56 .
RG contains 65 annotated pairs with scores ranging from 0.0 to 4.0 representing “similarity of meaning”.
RW contains 2034 pairs of words annotated with similarity scores in a scale from 0 to 10. The words included in this dataset were obtained from Wikipedia based on their frequency, and later filtered depending on their WordNet synsets, including synonymy, hyperonymy, hyponymy, holonymy and meronymy. This dataset was created with the purpose of testing how well models can represent rare and complex words.
SimLex999 contains 999 word pairs annotated with similarity scores ranging from 0 to 10. In this case the authors explicitly considered similarity and not relatedness, addressing the shortcomings of datasets that do not, such as MEN and WS353. Words include nouns, adjectives and verbs.
SimVerb3500 contains 3500 verb pairs annotated with similarity scores ranging from 0 to 10. Verbs were obtained from the USF free association database BIBREF66 , and VerbNet BIBREF63 . This dataset was created to address the lack of representativity of verbs in SimLex999, and the fact that, at the time of creation, the best performing models had already surpassed inter-annotator agreement in verb similarity evaluation resources. Like SimLex999, this dataset also explicitly considers similarity as opposed to relatedness.
WS353 contains 353 word pairs annotated with similarity scores from 0 to 10.
WS353R is a subset of WS353 containing 252 word pairs annotated with relatedness scores. This dataset was created by asking humans to classify each WS353 word pair into one of the following classes: synonyms, antonyms, identical, hyperonym-hyponym, hyponym-hyperonym, holonym-meronym, meronym-holonym, and none-of-the-above. These annotations were later used to group the pairs into: similar pairs (synonyms, antonyms, identical, hyperonym-hyponym, and hyponym-hyperonym), related pairs (holonym-meronym, meronym-holonym, and none-of-the-above with a human similarity score greater than 5), and unrelated pairs (classified as none-of-the-above with a similarity score less than or equal to 5). This dataset is composed by the union of related and unrelated pairs.
WS353S is another subset of WS353 containing 203 word pairs annotated with similarity scores. This dataset is composed by the union of similar and unrelated pairs, as described previously.
Word Frequencies and Gating Values
fig:gatingviz shows that for more common words the vector gate mechanism tends to favor only a few dimensions while keeping a low average gating value across dimensions. On the other hand, values are greater and more homogeneous across dimensions in rarer words. Further, fig:freqvsgatevalue shows this mechanism assigns, on average, a greater gating value to less frequent words, confirming the findings by BIBREF11 , and BIBREF12 .
In other words, the less frequent the word, the more this mechanism allows the character-level representation to influence the final word representation, as shown by eq:vg. A possible interpretation of this result is that exploiting character information becomes increasingly necessary as word-level representations' quality decrease.
Another observable trend in both figures is that gating values tend to be low on average. Indeed, it is possible to see in fig:freqvsgatevalue that the average gating values range from INLINEFORM0 to INLINEFORM1 . This result corroborates the findings by BIBREF11 , stating that setting INLINEFORM2 in eq:scalar-gate, was better than setting it to higher values.
In summary, the gating mechanisms learn how to compensate the lack of expressivity of underrepresented words by selectively combining their representations with those of characters.
Sentence-level Evaluation
table:sentlevelresults shows the impact that different methods for combining character and word-level word representations have in the quality of the sentence representations produced by our models.
We can observe the same trend mentioned in subsec:word-similarity-eval, and highlighted by the difference between bold values, that models trained in MultiNLI performed better than those trained in SNLI at a statistically significant level, confirming the findings of BIBREF13 . In other words, training sentence encoders on MultiNLI yields more general sentence representations than doing so on SNLI.
The two exceptions to the previous trend, SICKE and SICKR, benefited more from models trained on SNLI. We hypothesize this is again due to both SNLI and SICK BIBREF26 having similar data distributions.
Additionally, there was no method that significantly outperformed the word only baseline in classification tasks. This means that the added expressivity offered by explicitly modeling characters, be it through concatenation or gating, was not significantly better than simply fine-tuning the pre-trained GloVe embeddings for this type of task. We hypothesize this is due to the conflation of two effects. First, the fact that morphological processes might not encode important information for solving these tasks; and second, that SNLI and MultiNLI belong to domains that are too dissimilar to the domains in which the sentence representations are being tested.
On the other hand, the vector gate significantly outperformed every other method in the STSB task when trained in both datasets, and in the STS16 task when trained in SNLI. This again hints at this method being capable of modeling phenomena at the word level, resulting in improved semantic representations at the sentence level.
Relationship Between Word- and Sentence-level Evaluation Tasks
It is clear that the better performance the vector gate had in word similarity tasks did not translate into overall better performance in downstream tasks. This confirms previous findings indicating that intrinsic word evaluation metrics are not good predictors of downstream performance BIBREF29 , BIBREF30 , BIBREF20 , BIBREF31 .
subfig:mnli-correlations shows that the word representations created by the vector gate trained in MultiNLI had positively-correlated results within several word-similarity tasks. This hints at the generality of the word representations created by this method when modeling similarity and relatedness.
However, the same cannot be said about sentence-level evaluation performance; there is no clear correlation between word similarity tasks and sentence-evaluation tasks. This is clearly illustrated by performance in the STSBenchmark, the only in which the vector gate was significantly superior, not being correlated with performance in any word-similarity dataset. This can be interpreted simply as word-level representations capturing word-similarity not being a sufficient condition for good performance in sentence-level tasks.
In general, fig:correlations shows that there are no general correlation effects spanning both training datasets and combination mechanisms. For example, subfig:snli-correlations shows that, for both word-only and concat models trained in SNLI, performance in word similarity tasks correlates positively with performance in most sentence evaluation tasks, however, this does not happen as clearly for the same models trained in MultiNLI (subfig:mnli-correlations).
Gating Mechanisms for Combining Characters and Word Representations
To the best of our knowledge, there are only two recent works that specifically study how to combine word and subword-level vector representations.
BIBREF11 propose to use a trainable scalar gating mechanism capable of learning a weighting scheme for combining character-level and word-level representations. They compared their proposed method to manually weighting both levels; using characters only; words only; or their concatenation. They found that in some datasets a specific manual weighting scheme performed better, while in others the learned scalar gate did.
BIBREF12 further expand the gating concept by making the mechanism work at a finer-grained level, learning how to weight each vector's dimensions independently, conditioned on external word-level features such as part-of-speech and named-entity tags. Similarly, they compared their proposed mechanism to using words only, characters only, and a concatenation of both, with and without external features. They found that their vector gate performed better than the other methods in all the reported tasks, and beat the state of the art in two reading comprehension tasks.
Both works showed that the gating mechanisms assigned greater importance to character-level representations in rare words, and to word-level representations in common ones, reaffirming the previous findings that subword structures in general, and characters in particular, are beneficial for modeling uncommon words.
Sentence Representation Learning
The problem of representing sentences as fixed-length vectors has been widely studied.
BIBREF32 suggested a self-adaptive hierarchical model that gradually composes words into intermediate phrase representations, and adaptively selects specific hierarchical levels for specific tasks. BIBREF33 proposed an encoder-decoder model trained by attempting to reconstruct the surrounding sentences of an encoded passage, in a fashion similar to Skip-gram BIBREF34 . BIBREF35 overcame the previous model's need for ordered training sentences by using autoencoders for creating the sentence representations. BIBREF36 implemented a model simpler and faster to train than the previous two, while having competitive performance. Similar to BIBREF33 , BIBREF37 suggested predicting future sentences with a hierarchical CNN-LSTM encoder.
BIBREF13 trained several sentence encoding architectures on a combination of the SNLI and MultiNLI datasets, and showed that a BiLSTM with max-pooling was the best at producing highly transferable sentence representations. More recently, BIBREF18 empirically showed that sentence representations created in a multi-task setting BIBREF38 , performed increasingly better the more tasks they were trained in. BIBREF39 proposed using an autoencoder that relies on multi-head self-attention over the concatenation of the max and mean pooled encoder outputs for producing sentence representations. Finally, BIBREF40 show that modern sentence embedding methods are not vastly superior to random methods.
The works mentioned so far usually evaluate the quality of the produced sentence representations in sentence-level downstream tasks. Common benchmarks grouping these kind of tasks include SentEval BIBREF23 , and GLUE BIBREF41 . Another trend, however, is to probe sentence representations to understand what linguistic phenomena they encode BIBREF42 , BIBREF43 , BIBREF44 , BIBREF45 , BIBREF46 .
General Feature-wise Transformations
BIBREF47 provide a review on feature-wise transformation methods, of which the mechanisms presented in this paper form a part of. In a few words, the INLINEFORM0 parameter, in both scalar gate and vector gate mechanisms, can be understood as a scaling parameter limited to the INLINEFORM1 range and conditioned on word representations, whereas adding the scaled INLINEFORM2 and INLINEFORM3 representations can be seen as biasing word representations conditioned on character representations.
The previous review extends the work by BIBREF48 , which describes the Feature-wise Linear Modulation (FiLM) framework as a generalization of Conditional Normalization methods, and apply it in visual reasoning tasks. Some of the reported findings are that, in general, scaling has greater impact than biasing, and that in a setting similar to the scalar gate, limiting the scaling parameter to INLINEFORM0 hurt performance. Future decisions involving the design of mechanisms for combining character and word-level representations should be informed by these insights.
Conclusions
We presented an empirical study showing the effect that different ways of combining character and word representations has in word-level and sentence-level evaluation tasks.
We showed that a vector gate performed consistently better across a variety of word similarity and relatedness tasks. Additionally, despite showing inconsistent results in sentence evaluation tasks, it performed significantly better than the other methods in semantic similarity tasks.
We further showed through this mechanism, that learning character-level representations is always beneficial, and becomes increasingly so with less common words.
In the future it would be interesting to study how the choice of mechanism for combining subword and word representations affects the more recent language-model-based pretraining methods such as ELMo BIBREF49 , GPT BIBREF50 , BIBREF51 and BERT BIBREF52 .
Acknowledgements
Thanks to Edison Marrese-Taylor and Pablo Loyola for their feedback on early versions of this manuscript. We also gratefully acknowledge the support of the NVIDIA Corporation with the donation of one of the GPUs used for this research. Jorge A. Balazs is partially supported by the Japanese Government MEXT Scholarship.
Hyperparameters
We only considered words that appear at least twice, for each dataset. Those that appeared only once were considered UNK. We used the Treebank Word Tokenizer as implemented in NLTK for tokenizing the training and development datasets.
In the same fashion as conneau2017supervised, we used a batch size of 64, an SGD optmizer with an initial learning rate of INLINEFORM0 , and at each epoch divided the learning rate by 5 if the validation accuracy decreased. We also used gradient clipping when gradients where INLINEFORM1 .
We defined character vector representations as 50-dimensional vectors randomly initialized by sampling from the uniform distribution in the INLINEFORM0 range.
The output dimension of the character-level BiLSTM was 300 per direction, and remained of such size after combining forward and backward representations as depicted in eq. EQREF9 .
Word vector representations where initialized from the 300-dimensional GloVe vectors BIBREF14 , trained in 840B tokens from the Common Crawl, and finetuned during training. Words not present in the GloVe vocabulary where randomly initialized by sampling from the uniform distribution in the INLINEFORM0 range.
The input size of the word-level LSTM was 300 for every method except concat in which it was 600, and its output was always 2048 per direction, resulting in a 4096-dimensional sentence representation.
Sentence Evaluation Datasets
table:sentence-eval-datasets lists the sentence-level evaluation datasets used in this paper. The provided URLs correspond to the original sources, and not necessarily to the URLs where SentEval got the data from.
The version of the CR, MPQA, MR, and SUBJ datasets used in this paper were the ones preprocessed by BIBREF75 . Both SST2 and SST5 correspond to preprocessed versions of the SST dataset by BIBREF74 . SST2 corresponds to a subset of SST used by BIBREF54 containing flat representations of sentences annotated with binary sentiment labels, and SST5 to another subset annotated with more fine-grained sentiment labels (very negative, negative, neutral, positive, very positive).

Question: Where do they employ feature-wise sigmoid gating?
Answer: gating mechanism acts upon each dimension of the word and character-level vectors

Question: Which model architecture do they use to obtain representations?
Answer: BiLSTM with max pooling

Question: Which downstream sentence-level tasks do they evaluate on?
Answer: BIBREF13 , BIBREF18

Question: Which similarity datasets do they use?
Answer: MEN, MTurk287, MTurk771, RG, RW, SimLex999, SimVerb3500, WS353, WS353R, WS353S

WS353S, SimLex999, SimVerb3500