The Expressive Power of Word Embeddings
Authors: Yanqing Chen, Bryan P, Rami Al-Rfou, Steven Skiena
ICLR 2013

Abstract: We seek to better understand the difference in quality of the several publicly released embeddings. We propose several tasks that help to distinguish the characteristics of different embeddings. Our evaluation shows that embeddings are able to capture deep semantics even in the absence of sentence structure. Moreover, benchmarking the embeddings shows great variance in quality and characteristics of the semantics captured by the tested embeddings. Finally, we show the impact of varying the number of dimensions and the resolution of each dimension on the effective useful features captured by the embedding space. Our contributions highlight the importance of embeddings for NLP tasks and the effect of their quality on the final results.

arXiv:1301.3226v4  [cs.LG]  29 May 2013
The Expressive Power of Word Embeddings
Yanqing Chen cyanqing@cs.stonybrook.edu
Bryan Perozzi bperozzi@cs.stonybrook.edu
Rami Al-Rfou’ ralfrou@cs.stonybrook.edu
Steven Skiena skiena@cs.stonybrook.edu
Computer Science Dept. Stony Brook University Stony Brook, NY 1 1794
Abstract
We seek to better understand the informa-
tion encoded in word embeddings. We pro-
pose several tasks that help to distinguish the
characteristics of diﬀerent publicly released
embeddings. Our evaluation shows that em-
beddings are able to capture surprisingly nu-
anced semantics even in the absence of sen-
tence structure. Moreover, benchmarking the
embeddings shows great variance in quality
and characteristics of the semantics captured
by the tested embeddings. Finally, we show
the impact of varying the number of dimen-
sions and the resolution of each dimension on
the eﬀective useful features captured by the
embedding space. Our contributions high-
light the importance of embeddings for NLP
tasks and the eﬀect of their quality on the
ﬁnal results.
1. Introduction
Distributed word representations (embeddings) cap-
ture semantic and syntactic features of words out
of raw text corpus without human intervention or
language dependent processing. The features em-
bedding capture are task independent which make
them ideal for language modeling. However, em-
beddings are hard to interpret and understand. De-
spite the eﬀorts of visualizing the word embeddings
(
Van der Maaten and Hinton , 2008), points in high di-
mensional spaces carry a lot of information that is hard
to quantify. Additionally, publicly available embed-
dings generated by multiple research groups use dif-
ferent data and training procedures and there is not
yet an understanding about the best way to learn these
Proceedings of the 30th International Conference on Ma-
chine Learning, Atlanta, Georgia, USA, 2013. JMLR:
W&CP volume 28. Copyright 2013 by the author(s).
representations.
In this paper, we investigate four public released word
embeddings: (1) HLBL, (2) SENNA, (3) Turian’s and
(4) Huang’s. We use context-free classiﬁcation tasks
rather than sequence labeling tasks (such as part of
speech tagging) to isolate the eﬀects of context in mak-
ing decisions and eliminate the complexity of the learn-
ing methods. Speciﬁcally, our work makes the follow-
ing contributions:
• We show through evaluation that embeddings are
able to capture semantics in the absence of sen-
tence structure and that there is a diﬀerence in
the characteristics of the publicly released word
embeddings.
• We explore the impact of the number of dimen-
sions and the resolution of each dimension on the
quality of the information that can be encoded
in the embeddings space. That shows that mini-
mum eﬀective space needed to capture the useful
information in the embeddings.
• We demonstrate the importance of word pair ori-
entation in encoding useful linguistic information.
We run two pair classiﬁcation tasks and provide
an example with one of them where pair perfor-
mance greatly exceeds that of individual words.
The rest of the work proceeds as follows: First we de-
scribe the word embeddings we consider. Next we dis-
cuss our classiﬁcation experiments, and present their
results. Finally we discuss the eﬀects of scaling down
the size of the embeddings space.
2. Related Work
The original work for generating word embeddings was
presented by Bengio et. al. in (
Bengio et al. , 2003a).
The embeddings were a secondary output when gen-
erating language model. Since (
Bengio et al. , 2003a),
3 EXPERIMENTAL SETUP 2
there has been a signiﬁcant interest in speeding up the
generation process (
Bengio et al. , 2003b; 2009). These
original language models were evaluated using perplex-
ity. We argue that while perplexity is a good metric of
language modeling, it is not insightful about how well
the embeddings capture diverse types of information.
SENNA’s embeddings (
Collobert, 2011) are gener-
ated using a model that is discriminating and non-
probabilistic. In each training update, we read an
n-gram from the corpus, concatenating the learned
embeddings of these n words. Then a corrupted n-
gram is used by replacing the word in the middle
with a random one from the vocabulary. On top of
the two phrases, the model learns a scoring function
that scores the original phrases lower than the cor-
rupted one. The loss function used for training is
hinge loss. (
Collobert et al. , 2011) shows that embed-
dings are able to perform well on several NLP tasks
in the absence of any other features. The NLP tasks
considered by SENNA all consist of sequence labeling,
which imply that the model might learn from sequence
dependencies. Our work enriches the discussion by fo-
cusing on term classiﬁcation problems.
In (
Turian et al. , 2010), Turian et. al. duplicated the
SENNA embeddings with some diﬀerences; they cor-
rupt the last word of each n-gram instead of the word
in the middle. They also show that using embeddings
in conjunction with typical NLP features improves the
performance on the Named Entity Recognition task.
An additional result of (
Turian et al. , 2010) shows that
most of the embeddings have similar eﬀect when added
to an existing NLP task. This gives the wrong impres-
sion. Our work illustrates that not all embeddings
are created equal and there are signiﬁcant diﬀerences
in the information captured by each publicly released
model exist.
Mnih and Hinton (
Mnih and Hinton , 2007) proposed
a log-bilinear loss function to model language. Given
an n-gram, the model concatenates the embeddings of
the n-1 ﬁrst words, and learns a linear model to predict
the embedding of the last word. Mnih and Hinton later
proposed Hierarchical log-bilinear (HLBL) model em-
beddings (
Mnih and Hinton , 2009) to speed up model
evaluation during training and testing by using a
hierarchical approach (similar to (
Morin and Bengio ,
2005)) that prune the search space for the next word
by dividing the prediction into a series of predictions
that ﬁlter region of the space. The language model
eventually is evaluate using perplexity.
A fundamental challenge for neural language models
involves representing words which have multiple mean-
ings. In (
Huang et al. , 2012), Huang et. al. incorporate
global context to deal with challenges raised by words
with multiple meanings.
Recent work by Mikolov et. al. (
Mikolov et al. , 2013)
investigates linguistic regularities captured by the rel-
ative positions of points in the embedding space. Our
results regarding pair classiﬁcation are complemen-
tary.
3. Experimental setup
We will construct three term classiﬁcation problems
and two pair classiﬁcation problems to quantify the
quality of the embeddings.
3.1. Evaluation Tasks
Our evaluation tasks are as follows:
• Sentiment Polarity : We use Lydia’s sentiment
lexicon (
Godbole et al. , 2007) to create sets of
words which have positive or negative connota-
tions and construct the 2-class sentiment polarity
test. The data size is 6923 words.
• Noun Gender : We use Bergsma’s dataset
(
Bergsma and Lin , 2006) to compile a list of mas-
culine and feminine proper nouns. Names that co-
refer more frequently with she/he are respectively
considered feminine/masculine. Strings that co-
refer the most with it, appear less than 300 times
in the corpus, or consist of multiple words are ig-
nored. The total size is 2133 words.
• Plurality: We use WordNet (
Fellbaum, 2010) to
extract nouns in their singular and plural forms.
The data consists of 3012 words.
• Synonyms and Antonyms : We use Word-
Net to extract synonym and antonym pairs and
check whether we can part one kind from the
others. The relation is symmetric thus we put
each word pair together with their order-reversed-
counterparts. There are 3446 diﬀerent word pairs.
• Regional Spellings : We collect the words
that diﬀer in spelling between UK English and
the American counterpart from an online source
(
Limited, 2009). We make this task be a pair
classiﬁcation task to emphasize relative distances
between embeddings. We have 1565 pairs in this
task.
We ensure that for all tasks the class labels are bal-
anced. This allow our baseline evaluation to be either
3.3 Classiﬁcation 3
Sentiment Noun Gender Plurality
Positive Negative Feminine Masculine Plural Singular
Samples
good bad Ada Steve cats cat
talent stupid Irena Roland tables table
amazing ﬂaw Linda Leonardo systems system
Synonyms and Antonyms Regional Spellings
Synonyms Antonyms UK US
Samples
store shop rear front colour color
virgin pure polite impolite driveable drivable
permit license friend foe smash-up smashup
Table 1. Example input from each task
the random classiﬁer or the most frequent label clas-
siﬁer. Either of them will give an accuracy of 50%.
Table
1 shows examples of each of the 2-class evalua-
tion tasks. The classiﬁer is asked to identify which of
the classes a term or pair belongs to.
3.2. Embeddings’ Datasets
We choose the following publicly available embeddings
datasets for evaluation.
• SENNA’s embeddings covers 130,000 words
with 50 dimensions for each word.
• Turian’s embeddings covers 268,810 words,
each represented either with 25, 50 or 100 dimen-
sions.
• HLBL’s embeddings covers 246,122 words.
These embeddings were trained on same data used
for Turian embedding for 100 epochs (7 days), and
have been induced in 50 or 100 dimensions.
• Huang’s embeddings covers 100,232 words, in
50 dimensions. Huang’s embeddings require con-
text to disambiguate which prototype to use for
a word. Our tasks are context free so we average
the multiple prototypes to a single point in the
space. (This was the approach which worked best
in our testing.)
It should be emphasized that each of these models has
been induced under substantially diﬀerent training pa-
rameters. Each model has its own vocabulary, used a
diﬀerent context size, and was trained for a diﬀerent
number of epochs on its training set. While the control
of these variables is outside the scope of this study, we
hope to mitigate one of these challenges by running our
experiments on the vocabulary shared by all these em-
beddings. The size of this shared vocabulary is 58,411
words.
3.3. Classiﬁcation
For classiﬁcation we used Logistic Regression and a
SVM with the RBF-kernel as linear and non-linear
classiﬁers. There is a model-selection procedure by
running a grid-search on the parameter space with
the help of the development data. All experiments
were written using the Python package Scikit-Learn
(
Pedregosa et al. , 2011). For the term classiﬁcation
tasks we oﬀered the classiﬁer only the embedding of
the word as an input. For pairwise experiments, the
input consists of the embeddings of the two words con-
catenated.
The average of four folds of cross validation is used
to evaluate the performance of each classiﬁer on each
task. 50%, 25%, 25% of the data are used, as train-
ing, development and testing datasets respectively, for
evaluation and model selection.
4. Evaluation Results
Here we present the evaluation of both our term and
pair classiﬁcation results.
4.1. Term Classiﬁcation
Figure
1 shows the results over all the 2-class term
classiﬁcation tasks using logistic regression and RBF-
kernel SVM. It is surprising that all the embeddings
we considered did much better than the baseline, even
on a seemingly hard tests like sentiment detection.
What’s more, there is strong performance from both
the SENNA and Huang embeddings. SENNA embed-
dings seem to capture the plurality relationship better,
which may be from the emphasis that the SENNA em-
beddings place on shallow syntactic features.
Sentiment Gender Plurality0.4
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
SENNA
HLBL-50
HLBL-100
T urian-25
T urian-50
T urian-100
Huang
Figure 1. Results of the term-based tasks considered,
shaded areas represent improvements using kernel SVM.
5 INFORMATION REDUCTION 4
Table 2 shows examples of words from the test datasets
after classifying them using logistic regression on the
SENNA embeddings. The top and bottom rows show
the words that the classiﬁer is conﬁdent classifying,
while the rows in the middle show the words that lie
close to the decision boundary. For example, resilient
could have positive and negative connotations in text,
therefore, we ﬁnd it close to the region were the words
are more neutral than being polarized.
For SENNA, the best performing task was the Plural-
ity task. That explains the obvious contrast between
the probabilities given to the words. The top words are
given almost 100% probability and the bottom ones
are given almost 0%. The results of regional spelling
task is shown here in the term-wise setup. Despite not
performing as well as the pair-wise spelling, we can
see that classiﬁer shows meaningful results. We can
clearly notice that the British spellings of words favor
the usage of hyphens, s over z and ll over l.
Positive Prob British Prob
Sentiment
world-famous 99.85
Regional Spelling
kick-oﬀ 92.37
award-winning 99.83 hauliers 91.54
high-quality 99.83 re-exported 89.46
achievement 99.81 bullet-proof 88.69
athletic 99.81 initialled 88.42
resilient 50.14 paralysed 50.16
ragged 50.11 italicized 50.04
discriminating 50.10 exorcise 50.03
stout 49.97 fusing 49.90
lose 49.83 lacklustre 49.78
bored 49.81 subsidizing 49.77
bloodshed 0.74 signaling 32.04
burglary 0.68 hemorrhagic 21.69
robbery 0.58 tumor 21.69
panic 0.45 homologue 19.53
stone-throwing 0.28 localize 17.50
Negative 1.0-Prob American 1.0-Prob
Table 2.Examples of the results of the logistic regression
classiﬁer on diﬀerent tasks.
4.2. Pair Classiﬁcation
Sometimes however, the choice to use pair classiﬁca-
tion can make quite a diﬀerence in the results. Figure
2a shows that classifying individual words according to
their regional usage performs poorly. We can redeﬁne
the problem such that the classiﬁer is asked to decide
if the ﬁrst word, in a pair of words, is the American
spelling or not. Figure
2a shows that performance im-
proves a lot. This hints that the words under this crite-
ria are not separable by a hyper-plane in any subspace
of the original embeddings space. Instead, we draw
a similar conclusion as (
Mikolov et al. , 2013) that the
pairs’ positions relative to each other is what encodes
such information but not their absolute coordinates,
and relationship between words often indicate the rel-
ative diﬀerence vector between corresponding points.
In our previous Plurality test, the SENNA embed-
dings signiﬁcantly outperformed Huang’s. However in
our regional spelling task (which might seem similar),
Huang’s embeddings outperform SENNA in both term
and pair classiﬁcation setups. We believe that Huang’s
approach for building word prototypes from signiﬁcant
diﬀerences in context provide a signiﬁcant advantage
on this task.
We note that it is surprising that neural language mod-
els may capture the relation between a synonym and
antonym. Both the language modeling of HLBL and
the way that SENNA/Turian corrupted their exam-
ples favor words that can syntactically replace each
other; e.g. bad can replace good as easily as excellent
can. The result of this syntactic interchangeability is
that both bad and excellent are close to good in the
embedding space.
5. Information reduction
Distributed word representation exist in continuous
space, which is quite diﬀerent from common language
modeling techniques. Beside the powerful expressive-
ness that we demonstrated previously, another key ad-
vantage of distributed representations is their size -
they require far less memory and disk storage than
other techniques. In this section we seek to under-
stand exactly how much space word embeddings need
in order to serve as useful features. We also investigate
whether the powerful representation that embeddings
oﬀer is a result of having real value coordinates or the
exponential number of regions which can be described
using multiple independent dimensions.
5.1. Bitwise Truncation
To reduce the resolution of the real numbers that make
up the embeddings matrix. First we scale them to 32
bit integer values, then we divide the values by 2 b,
where b is the number of bits we wish to remove. Fi-
nally, we scale the values back to lie between ( − 1,1).
After this preprocessing we give the new values as fea-
tures to our classiﬁers. In the extreme case, when we
truncate 31 bits, the values will be all either {1,− 1}.
Figure
3 shows that when we remove 31 bits (i.e, val-
ues are {1,− 1}), the performance of an embedding
dataset drops no more than 7%. This reduced reso-
lution is equivalent to 2 50 regions which can be en-
coded in the new space. This is still a huge resolution,
but surprisingly seems to be suﬃcient at solving the
tasks we proposed. A na¨ ıve approximation of this trick
which may be of interest is to simply take the the sign
of the embedding values as the representation of the
6 CONCLUSION 5
Spellings(term) Spellings(pair)0.4
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
SENNA
HLBL-50
HLBL-100
T urian-25
T urian-50
T urian-100
Huang
(a) UK/US term vs. pair
Synonym Spellings0.4
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
SENNA
HLBL-50
HLBL-100
T urian-25
T urian-50
T urian-100
Huang
(b) 2-class pair results
Figure 2. Results of the pair-based tests. Figure 2a shows
the diﬀerence between treating the UK/US spellings as a
single word problem, or using a pair of embeddings. Figure
2b shows the results of the 2-class pair tests, shaded areas
represent improvements using kernel SVM.
embeddings themselves.
5.2. Principle Component Analysis
The bitwise truncation experiment indicates that the
number of dimensions could be a key factor into the
performance of the embeddings. To experiment on this
further, we run PCA over the embeddings datasets to
evaluate task performance on a reduced number of di-
mensions. Figure
4 shows that reducing the dimen-
sions drops the accuracy of the classiﬁers signiﬁcantly
across all embedding datasets.
Another key diﬀerence between the truncation exper-
iment and the PCA experiment is that the trunca-
tion experiment may preserve relationships captured
by non-linearities in the embedding space. Linear PCA
16128421
Bits remaining in each dimension.
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90Accuracy
SENNA HLBL-50 T urian-50 Huang
Figure 3. Results of reducing the precision of the embed-
dings, averaged by the geometric mean of classiﬁers. We
note that after removing 31 bits, each dimension of the
embeddings is a binary feature.
can not oﬀer such guarantees and this weakness may
contribute to the diﬀerence in performance.
1 2 3 5 10 15 20 25 50
Dimensions
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90Accuracy
SENNA HLBL-50 T urian-50 Huang
Figure 4. Results of reducing the dimensions of the embed-
dings through PCA, averaged by the geometric mean.
6. Conclusion
Distributed word representations show a lot of promise
to improve supervised learning and semi-supervised
learning. The practical advantages of having dense
representations make them ideal for industrial appli-
cations and software development. The previous work
mainly focused on speeding up the training process
with one metric for evaluation, perplexity. We show
that this metric is not able to provide a nuanced view
of their quality. We develop a suite of linguistic ori-
REFERENCES 6
ented tasks which might serve as a part of a com-
prehensive benchmark for word embedding evaluation.
The tasks focus on words or pairs of them in isolation
to the actual text. The goal here is not to build a useful
classiﬁer as much as it is to understand how much su-
pervised learning can beneﬁt from the features which
are encoded in the embeddings.
We succeed in showing that the publicly available
datasets diﬀer in their quality and usefulness, and our
results are consistent across tasks and classiﬁers. Our
future work will try to address the factors that lead to
such diverse quality. The eﬀect of training corpus size
and the choice of the objective functions are two main
areas where better understanding is needed.
While our tasks are simple, the diﬀerences among task
performance shed light on the features encoded by em-
beddings. We showed that in addition to the shallow
syntactic features like plural and gender agreement,
there are signiﬁcant semantic partitions regarding sen-
timent and synonym/antonym meaning. Our current
tasks focus on nouns and adjectives, and the suite of
tasks has to be extended to include tasks that address
verbs and other parts of speech.
Acknowledgments
This research was partially supported by NSF Grants
DBI-1060572 and IIS-1017181, with additional sup-
port from TexelTek, Inc.
References
Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin.
A neural probabilistic language model. Journal of
Machine Learning Research , 3:1137–1155, 2003a.
Y. Bengio, J.S. Sen´ ecal, et al. Quick training of proba-
bilistic neural nets by importance sampling. In AIS-
TATS Conference, 2003b.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston.
Curriculum learning. In Proceedings of the 26th an-
nual international conference on machine learning ,
pages 41–48. ACM, 2009.
Shane Bergsma and Dekang Lin. Bootstrapping path-
based pronoun resolution. In Proceedings of the 21st
International Conference on Computational Lin-
guistics and 44th Annual Meeting of the Association
for Computational Linguistics , pages 33–40, Sydney,
Australia, July 2006. Association for Computational
Linguistics.
R. Collobert. Deep learning for eﬃcient discriminative
parsing. In International Conference on Artiﬁcial
Intelligence and Statistics (AISTATS) , 2011.
R. Collobert, J. Weston, L. Bottou, M. Karlen,
K. Kavukcuoglu, and P. Kuksa. Natural language
processing (almost) from scratch. The Journal of
Machine Learning Research , 12:2493–2537, 2011,
JMLR. org.
C. Fellbaum. Wordnet. Theory and Applications of
Ontology: Computer Applications , pages 231–243,
2010, Springer.
N. Godbole, M. Srinivasaiah, and S. Skiena. Large-
scale sentiment analysis for news and blogs. In Pro-
ceedings of the International Conference on Weblogs
and Social Media (ICWSM) , volume 2, 2007.
Eric H. Huang, Richard Socher, Christopher D. Man-
ning, and Andrew Y. Ng. Improving word represen-
tations via global context and multiple word pro-
totypes. In Proceedings of the 50th Annual Meet-
ing of the Association for Computational Linguis-
tics: Long Papers - Volume 1 , ACL ’12, pages 873–
882, Stroudsburg, PA, USA, 2012. Association for
Computational Linguistics.
Words Worldwide Limited. Word list of
us/uk spelling variants, May 2009. URL
http://www.wordsworldwide.co.uk/docs/Words-Worldwide-Word- 
Tomas Mikolov, Wen-tau Yih, and Geoﬀrey Zweig.
Linguistic regularities in continuous space word rep-
resentations. In Proceedings of NAACL-HLT, 2013.
A. Mnih and G. Hinton. Three new graphical models
for statistical language modelling. In Proceedings of
the 24th international conference on Machine learn-
ing, pages 641–648. ACM, 2007.
A. Mnih and G.E. Hinton. A scalable hierarchical dis-
tributed language model. Advances in neural in-
formation processing systems , 21:1081–1088, 2009,
Citeseer.
F. Morin and Y. Bengio. Hierarchical probabilistic
neural network language model. In Proceedings of
the international workshop on artiﬁcial intelligence
and statistics , pages 246–252, 2005.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,
B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer,
R. Weiss, V. Dubourg, J. Vanderplas, A. Pas-
sos, D. Cournapeau, M. Brucher, M. Perrot, and
E. Duchesnay. Scikit-learn: Machine learning in
Python. Journal of Machine Learning Research , 12:
2825–2830, 2011.
REFERENCES 7
J. Turian, L. Ratinov, and Y. Bengio. Word repre-
sentations: a simple and general method for semi-
supervised learning. Urbana, 51:61801, 2010.
L. Van der Maaten and G. Hinton. Visualizing data
using t-sne. Journal of Machine Learning Research ,
9(2579-2605):85, 2008.
SUPPLEMENTAL MATERIALS 8
Supplemental Materials
3-class Tests
To strengthen these results, we performed a 3-class
version of the sentiment test, in which we evaluated
the ability to classify words as having positive, nega-
tive, or neutral sentiment value. The results are pre-
sented in Figure
5. The results are consistent with
those from our 2-label test, and all embeddings per-
form much higher than the baseline score of 33%.
Sentiment 3-Sentment0.4
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
SENNA
HLBL-50
HLBL-100
T urian-25
T urian-50
T urian-100
Huang
Figure 5. The performance on the 3-class version of the
sentiment task, shaded areas represent improvements using
kernel SVM.
In order to investigate the depth to which synonyms
and antonyms are captured, we conducted a 3-class
version of the same test. We now evaluate between
pairs of words that are synonyms, antonyms, or have
no such relation. While such a task is much harder for
the embeddings, the results in Figure
6 show that a
nonlinear classiﬁer can capture the relationship, par-
ticularly with the SENNA embeddings. An analy-
sis of the confusion matrix for the nonlinear SVM
showed that errors occurred roughly evenly between
the classes. We believe that this ﬁnding regarding the
encoding of synonym/antonym relationships is an in-
teresting contribution of our work.
Dimensional Reduction by task
Looking at Figure
8a, reducing the words embeddings
to points on a real line almost deletes all the features
that are relevant to the pair classiﬁcation and to less
a degree the sentiment features. Despite the 10%-20%
drop in accuracy in the Plurality and Gender tasks,
the classiﬁcation is still higher than random. The re-
sults show that when that shallow syntactic features
such as gender and number agreement are preserved at
Synonym 3-Synonym0.4
0.5
0.6
0.7
0.8
0.9
1.0
Accuracy
SENNA
HLBL-50
HLBL-100
T urian-25
T urian-50
T urian-100
Huang
Figure 6. The performance of the 3-class syn-
onym/antonym task, shaded areas represent improvements
using kernel SVM.
16128421
Bits remaining in each dimension.
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90Accuracy
Spellings Synonym Gender Plural Sentiment
Figure 7. Results of reducing the precision of the embed-
dings, averaged by the geometric mean of classiﬁers across-
ing tasks
the expense of more subtle semantic features such as
sentiment polarity. This gives us insight into what the
hierarchical structure of the embeddings space looks
like. Shallow semantic features are present in all as-
pects of the space, and when PCA chooses to maximize
this variance of the feature space it is at the expense
of the other semantic properties.
We also illustrate this phenomenon in Figure
8b, by
showing how the performance of the linear and non-
linear classiﬁers converge for our harder tasks (senti-
ment and synonym) as we reduce the number of di-
mensions with PCA.
SUPPLEMENTAL MATERIALS 9
1 2 3 5 10 15 20 25 50
Dimensions
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90Accuracy
Spellings Synonym Gender Plural Sentiment
(a) By task
1 2 3 5 10 15 20 25 50
Dimensions
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90Accuracy
Synonym  (RBF)
Plural (RBF)
Sentiment (RBF)
Synonym (linear)
Plural (linear)
Sentiment (linear)
(b) Linear vs. Nonlinear
Figure 8. Results of reducing the dimensions of the embeddings throug h PCA, averaged by the geometric mean across
tasks ( 8a). Figure 8b shows the diﬀerence between linear (dashed) and non-linear (solid) classiﬁers for our harder tasks
(sentiment and synonym) and an easy task (plural). The perfo rmance of the linear and nonlinear classiﬁers converges
as PCA removes more dimensions. This results in signiﬁcantl y degraded performance on nuanced tasks like sentiment
analysis.
This figure "bit_reduction_by_embedding_after_normalization.png" is available in "png"
 format from:
http://arxiv.org/ps/1301.3226v4
This figure "bit_reduction_by_task_after_normalization.png" is available in "png"
 format from:
http://arxiv.org/ps/1301.3226v4


Peer review:
[Review]:
We thank the anonymous reviewers for their thoughtful comments.  We have taken them into consideration, and have uploaded a revised manuscript to arXiv over the weekend. (it should be available in a few hours)

Specific changes include:
1. We have evaluated 3-class versions of our classifiers on the sentiment and synonym/antonym tasks.
2. We have reworked to focus more explicitly on term vs. pair tasks, and believe that this is a more clear presentation of our ideas
3. We have illustrated the convergence of linear vs. nonlinear classifiers as dimensions are reduced by PCA.
4. We have tried to modify specific language and tone that the reviewers found objectionable.

We have some specific comments to each of our reviewers:

Anonymous 406c, (1st reviewer, 2/14/2013)

- Scaling of embeddings:
We investigated scaling the embeddings to control the variance after PCA as recommended by Turian (2010).  Results did not significantly change, and so we left the original in there.  We have posted the corresponding plots with scaling:
(by embedding)  http://goo.gl/wpXmD
(by task)   http://goo.gl/hWYkX

You might also be interested in the fact that we ran all of our experiments on scaled and unscaled versions of the embeddings, but did not notice significant differences between them.  We attribute both these results to the fact that we only used embeddings as features - Turian (2010) comments that his scaling approach is for mixing embedding features with words represented by binary features.  

Anonymous af94 (2nd reviewer, 3/1/2013)

- Similar to Turian (2010)
We’re quite different actually - Turian (2010) studied enhancing existing NLP tools with a variety of embeddings.  This means that he combined the embeddings with existing features (from words,  n-grams, or characters).
Instead, we use the embeddings as the sole features to understand their quality on their own. Moreover, we propose term/pair classification tasks to isolate the effect of context that influence the results of sequence tagging tasks.

- Pairwise classification is easier because its an ensemble
You raise an interesting point here.  To explain our views: in our initial experiments we used the element-wise subtraction between two embeddings as features and they outperformed the single word version of the experiment.  This seems to indicate that the embeddings encode information in the direction of the vector between two points in the space.  We later modified the experiment to the one we report (it seemed more general at the time).

- Classifiers? Which? Where?
We use Linear SVM, Logistic Regression, RBF Kernel SVM on all our tasks and report the geometric mean of their results for each task. Each classifier result is obtained with 4-fold cross validation setup. 

We also have some general comments on the nature of our tasks, and the decision to evaluate existing embeddings instead of training new ones
On the nature of our tasks:

Most of the previous evaluation on word embeddings has been done in the context of sequence tagging problems.  While practical, we believe that this approach complicates the actual analysis of the features learned by neural language models.  

For example, in a typical part of speech tagging setup the performance of the tagger over out-of-vocabulary words (without using character features) is much higher than random and might reach 70-80% accuracy. The decision, here, is clearly induced by the context. The influence of the context on the performance of classification, makes it harder to estimate the intrinsic quality of the word embeddings.

We agree that not all of our tasks map directly to traditional NLP tasks, but this is intended - each task illustrates one type of interesting behavior that can be found in the embedding space.  Some behaviors are ones known to exist (e.g. plurality is a sub-component of Part-of-Speech tagging), one seems practical (sentiment), and one is just cool (e.g. synonym / antonym).  The list is certainly not comprehensive, and we would appreciate additional suggestions.

On not training our own embeddings:

We are interested in NLP applications of feature learning, and we believe our results are valuable to consumers of such technology. We strove to evaluate the quality of what is available other researchers would use in their work.

We do agree it is hard to compare features produced under different conditions. It is a matter of fact that some of the embeddings will be better than others. The differences could be attributed to training specific factors (e.g. training time and datasets), or to the technique itself.  In light of this difficulty we have tried to highlight that we are comparing the embeddings themselves, and not the techniques.  We have toned down language contrary to this message.

On a final note, we would like to again thank our anonymous reviewers, and the greater ICLR community.