Two SVDs produce more focal deep learning representations
Authors: Hinrich Schuetze, Christian Scheible
ICLR 2013

Abstract: A key characteristic of work on deep learning and neural networks in general is that it relies on representations of the input that support generalization, robust inference, domain adaptation and other desirable functionalities. Much recent progress in the field has focused on efficient and effective methods for computing representations. In this paper, we propose an alternative method that is more efficient than prior work and produces representations that have a property we call focality -- a property we hypothesize to be important for neural network representations. The method consists of a simple application of two consecutive SVDs and is inspired by Anandkumar (2012).

arXiv:1301.3627v2  [cs.CL]  11 May 2013
Two SVDs produce more focal deep learning
representations
Hinrich Sch¨ utze
Center for Information and Language Processing
University of Munich, Germany
hs999@ifnlp.org
Christian Scheible
Institute for NLP
University of Stuttgart, Germany
scheibcn@ims.uni-stuttgart.de
Abstract
A key characteristic of work on deep learning and neural netw orks in general is
that it relies on representations of the input that support g eneralization, robust
inference, domain adaptation and other desirable function alities. Much recent
progress in the ﬁeld has focused on efﬁcient and effective methods for computing
representations. In this paper, we propose an alternative m ethod that is more ef-
ﬁcient than prior work and produces representations that ha ve a property we call
focality – a property we hypothesize to be important for neural network represen-
tations. The method consists of a simple application of two consecutive SVDs and
is inspired by (Anandkumar et al., 2012).
In this paper, we propose to generate representations for de ep learning by two consecutive applica-
tions of singular value decomposition (SVD). In a setup inspired by (Anandkumar et al., 2012), the
ﬁrst SVD is intended for denoising. The second SVD rotates th e representation to increase what
we call focality. In this initial study, we do not evaluate the representatio ns in an application. In-
stead we employ diagnostic measures that may be useful in their own right to evaluate the quality of
representations independent of an application.
We use the following terminology. SVD 1 (resp. SVD2) refers to the method using one (resp. two)
applications of SVD; 1LAYER (resp. 2LAYER) corresponds to a single-hidden-layer (resp. two-
hidden-layer) architecture.
In Section 1, we introduce the two methods SVD1 and SVD2 and show that SVD2 generates better
(in a sense to be deﬁned below) representations than SVD 1. In Section 2, we compare 1LAYER
and 2LAYER SVD2 representations and show that 2LAYER representations are b etter. Section 3
discusses the results. We present our conclusions in Section 4.
1 SVD 1 vs. SVD2
Given a base representation of n objects in Rd, we ﬁrst compute the ﬁrst k dimensions of an SVD
on the corresponding n × d matrix C. Ck = USV T (where Ck is the rank-k approximation of C).
We then use US to represent each object as a k-dimensional vector. Each vector is normalized to
unit length because our representations are count vectors w here the absolute magnitude of a count
contains little useful information – what is important is th e relative differences between the counts
of different dimensions. This is the representation SVD1. It is motivated by standard arguments for
representations produced by dimensionality reduction: compactness and noise reduction. Denoising
is also the motivation for the ﬁrst SVD in the method proposed by Anandkumar et al. (2012).
We then perform a second SVD on the resulting matrixC′ of dimensionality n × k. C′ = U′S′V ′T
(full-rank, no dimensionality reduction). We again use U′S′ to represent each object as a k-
dimensional vector. Each vector is normalized to unit length. This is the representation SVD2.
1
Figure 1: 1LAYER architecture. Distributional word vector s form the bottom layer. A trigram –
represented as a triple of word vectors – is transformed by th e ﬁrst SVD into a 100-dimensional
vector (layer “SVD1”). These vectors are then rotated (layer “SVD2”).
Note that the operation we are applying is not equivalent to a single linear operation because of the
lenght normalization that we perform between the two SVDs.
SVD2 is intended to be a rotation of SVD 1 that is more “focal” in the following sense. Consider a
classiﬁcation problem f over a k-dimensional representation space R. Let Mθ (f, R) be the size k′
of the smallest subset of the dimensions that support an accu racy above a threshold θ for f. Then
a representation R is more focal than R′ if Mθ (f, R) < M θ (f, R′). The intuition is that good
deep learning representations have semantically interpretable hidden units that contribute input to a
decision that is to some extent independent of other hidden units. We want the second SVD to rotate
the representation into a “more focal” direction.
The role of the second SVD is somewhat analogous to that of the second SVD in the approach of
Anandkumar et al. (2012), where the goal also is to ﬁnd a representation that reveals the underlying
structure of the data set.
The architecture of the 1LAYER setup is depicted in Figure 1.
Experimental setup. We use a corpus of movie review sentences (Pang and Lee, 2004). Following
Sch¨ utze (1995), we ﬁrst compute a left vector and a right vector for each word. The dimensionality
of the vectors is 250. Entryi for the left (right) vector of wordw is the number of times that the word
with frequency rank i occurred immediately to the left (right) of w. Vectors are then tf-idf weighted
and length-normalized. We randomly select 100,000 unique trigrams from the corpus, e.g., “tangled
feelings of” or “as it pushes”. Each trigram is represented a s the concatentation of six vectors, the
left and right vectors of the three words. This deﬁnes a matrix of dimensionality n× d (n = 100000,
d = 1500). We then compute SVD1 and SVD2 on this matrix for k = 100.
Analysis of correlation coefﬁcients. Figure 2 shows histograms of the 10,000 correlation coefﬁ-
cients of SVD1 (left) and SVD2 (right). Each correlation coefﬁcient is the correlation oftwo columns
in the corresponding 100000 × 100 matrix and is transformed using the functionf(c) = log10 |c|to
produce a histogram useful for the analysis. The histogram o f SVD2 is shifted by about 0.5 to the
left. This is a factor of 100. 5 ≈ 3. Thus, SVD2 dimensions have correlations that are only a third as
large as SVD1 correlations on average.
We take this to indicate that SVD2 representations are more focal than SVD1 representations because
the distribution of correlation coefﬁcients would change t he way it changes from SVD 2 to SVD1
if we took a focal representation (in the most extreme case on e where each dimension by itself
supported a decision) and rotated it.
Discrimination task. We randomly selected 200 words in the frequency range [25, 250] from the
corpus; and randomly arranged them into 100 pairs. An exampl e of such a pair is (documentary,
special). For each pair, we ﬁrst retrieved the SVD 1 and SVD2 representations of all triples from
the set of 100,000 in which one of the two words was the central word. For the example, “typical
documentary footage”, “good documentary can”, and “really special walk” are such triples. Then
we determined for each dimension i of the 100 dimensions (for both SVD1 and SVD2) the optimal
discrimination value θ by exhaustive search; that is, we determined the threshold θ for which the
accuracy of the classifer ⃗ vi > θ (or ⃗ vi < θ ) was greatest – where the discrimination task was to
distinguish triples that had one word vs the other as their central word. So for “typical documentary
footage” and “good documentary can” the classiﬁer should predict class 1 (“documentary”), for “re-
2
Figure 2: Histograms of log10 |c|of the 10,000 correlation coefﬁcients of SVD1 (above) and SVD2
(below).
ally special walk” the classiﬁer should predict class 2 (“special”). Finally, of the 100 discrimination
accuracies we chose the largest one for this word pair.
On this discrimination task, SVD2 was better than SVD1 55 times, the two were equal 15 times and
SVD2 was worse 30 times. On average, discrimination accuracy of SVD2 was 0.7% better than that
of SVD1. This is evidence that SVD2 is better for this discrimination task than SVD1.
This indicates again that SVD 2 representations are more focal than SVD 1 representations: each
dimension is more likely to provide crucial information by itself as opposed to only being useful in
conjunction with other dimensions.
To illustrate in more detail why this discrimination task is related to focality, assume that for a
particular 100-dimensional representation r of trigrams t, the decision rule “if r(t)27 > 0.2 then
‘documentary’ else ‘special’ ” (i.e., if the value of dimension 27 is greater than 0.2, then the trigram
center is predicted to be “documentary”, else “special”) has an accuracy of 0.99; and that the decision
rule “if r(t)27 > 0.2 and r(t)91 < − 0.1 then ‘documentary’ else ‘special’ ” has an accuracy of 1.0.
Then M0. 99(f, documentary-vs-special) = 1, M1. 00(f, documentary-vs-special) = 2 and we can
view the representation r as highly focal since a single dimension sufﬁces for high accuracy and two
dimensions achieve perfect classiﬁcation results.
2 1LAYER vs. 2LAYER
We compare two representations of a word trigram: (i) the 1LAYER representation from Section 1
and (ii) a 2LAYER representation that goes through two round s of autoencoding, which is a deep
learning representation in the sense that layer 2 represents more general and higher-level properties
of the input than layer 1.
The architecture of the 2LAYER is depicted in Figure 3.
To create 2LAYER representations, we ﬁrst create a vector fo r each of the 20701 word types oc-
curring in the corpus. This vector is the concatenation of it s left vector and its right vector. The
resulting 20701 × 500 matrix is the input representation to SVD1. We again set k = 100. A trigram
3
Figure 3: 2LAYER architecture. Distributional word vectors form the bottom layer. Each word vec-
tor is transformed by SVD into a 100-dimensional vector (ﬁrst layer “SVD1”). This layer constitutes
the 1LAYER part of this architecture. A triple of vectors of three consecutive words is transformed
by SVD into a 100-dimensional vector (second layer “SVD1”). These vectors are then rotated (layer
“SVD2”). The last two layers constitute the 2LAYER part of this architecture.
is then represented as the concatenation of three of these 10 0-dimensional vectors. We apply the
SVD2 construction algorithm to the resulting 100000 × 300 matrix and truncate to k = 100.
We now have – for each trigram – two SVD 2 representations, the 1LAYER representation from
Section 1 and the 2LAYER representation we just described. We compare these two trigram repre-
sentations, again using the task from Section 1: discrimination of the 100 pairs of words.
2LAYER is better than 1LAYER 64 times on this task, the same in 18 cases and worse in 18 cases.
This is statistically signiﬁcant (p < . 01, binomial test) evidence that 2LAYER SVD2 representations
are more focal than 1LAYER SVD2 representations.
3 Discussion
3.1 Focality
One advantage of focal representations is that many classiﬁers cannot handle conjunctions of several
features unless they are explicitly deﬁned as separate feat ures. Compare two representations ⃗ xand
⃗ x′ where ⃗ x′ is a rotation of ⃗ x(as it might be obtained by an SVD). Since one vector is a rotat ion of
the other, they contain exactly the same information. Howev er, if (i) an individual “hidden unit” of
the rotated vector ⃗ x′ can directly be interpreted as “is verb” (or a similar property like “is adjective”
or “takes NP argument”) and (ii) the same feature requires a conjunction of several hidden units for
⃗ x, then the rotated representation is superior for many upstream statistical classiﬁers.
Focal representations can be argued to be closer to biological reality than broadly distributed repre-
sentations (Thorpe, 2010); and they have the nice property that they become categorical in the limit.
Thus, they include categorical representations as a special case.
A ﬁnal advantage of focal representations is that in some convolutional architectures the input to the
top-layer statistical classiﬁer consists of maxima over HU(hidden unit) activations. E.g., one way to
classify a sentence as having positive/negative sentiment is to slide a neural network whose input is
a window ofk words (e.g., k = 4) over it and to represent each window ofk words as a vector of HU
activations produced by the network. In a focal representat ion, the hidden units are more likely to
have clear semantics like “the window contains a positive sentiment word”. In this type of scenario,
taking the maximum of activations over the n − k + 1sliding windows of a sentence of length n
results in hidden units with interpretable semantics like “the activation of the positive-sentiment HU
of the window with the highest activation for this HU”. Thesemaximum values are then a good basis
for sentiment classiﬁcation of the sentence as a whole.
The notion of focality is similar to disentanglement (Gloro t et al., 2011) – in fact, the two notions
may be identical. However, Glorot et al. (2011) introduce disentanglement in the context of domain
adaptation, focusing on the idea that “disentangled” hidden units capture general cross-domain prop-
erties and for that reason are a good basis for domain adaptation. The contributions of this paper are:
proposing a way of measuring “entanglement” (i.e., measuri ng it as correlation), deﬁning focality
4
in terms of classiﬁcation accuracy (a deﬁnition that covers single hidden units as well as groups of
hidden units) and discussing the relationship to convolution and biological systems.
It is important to point out that we have not addressed how foc ality would be computed efﬁciently
in a particular context. In theory, we could use brute force methods, but these would be exponential
in the number of dimensions (systematic search over all subs ets of dimensions). However, certain
interesting questions about focality can be answered efﬁci ently; e.g., if we have M(f, R) = 1for
one representation and M(f, R′) > 1 for another, then this can be shown efﬁciently and in this case
we have established that R is more focal than R′.
3.2 mSVD method
In this section, we will use the abbreviation mSVD to refer to a stacked applications of our method
with an arbitrary number of layers even though we only experi ment with m = 2 in this paper
(2LAYER, 2-layer-stacking).
SVD and other least squares methods are probably the most wid ely used dimensionality reduction
techniques for the type of matrices in natural language proc essing that we work with in this paper
(cf. (Turney and Pantel, 2010)). Stacking a second least squ ares method on top of the ﬁrst has not
been considered widely because these types of representati ons are usually used directly in vector
classiﬁers such as Rocchio and SVM (however, see the discussion of (Chen et al., 2012) below). For
this type of classiﬁer, performing a rotation has no effect on classiﬁcation performance. In contrast,
our interest is to use SVD 2 representations as part of a multilevel deep learning archi tecture where
the hidden unit representations of any given layer are not simply interpreted as a vector, but decisions
of higher layers can be based on individual dimensions.
The potential drawback of SVD and other least squares dimens ionality reductions is that they
are linear: reduced dimensions are linear combinations of o rginal dimensions. To overcome this
limitation many nonlinear methods have been introduced: pr obabilistic latent semantic indexing
(Hofmann, 1999), kernel principal component analysis (Sch¨ olkopf et al., 1998), matrix factorization
techniques that obey additional constraints – such as non-negativity in the case of non-negative ma-
trix factorization (Lee and Seung, 1999) – , latent dirichlet allocation (Blei et al., 2003) and different
forms of autoencoding (Bengio, 2009; Chen et al., 2012). All of these can be viewed as dimension
reduction techniques that do not make the simplistic assump tions of SVD and should therefore be
able to produce better representation if these simplistic a ssumptions are not appropriate for the do-
main in question.
However, this argument does not apply to the mSVD method we pr opose in this paper since it is
also nonlinear. What should be investigated in the future is to what extent the type of nonlinearity
implemented by mSVD offers advantages over other forms of no nlinear dimensionality reduction;
e.g., if the quality of the ﬁnal representations is comparable, then mSVD would have the advantage
of being more efﬁcient.
Finally, there is one big difference between mSVD and deep learning representations such as those
proposed by Hinton et al. (2006), Collobert and Weston (2008) and Socher et al. (2012). Most deep
learning representations are induced in a setting that also includes elements of supervised learn-
ing as is the case in contrastive divergence or when labeled d ata are available for adjusting initial
representations produced by a process like autoencoding or dimensionality reduction.
This is the most important open question related to the research presented here: how can one modify
hidden layer representations initialized by multiple SVDs in a meaningful way?
The work most closely related to what we are proposing is prob ably mDA (Chen et al., 2012) – an
approach we only became aware of after the initial publication of this paper. There are a number of
differences between mDA and mSVD. Non-linearity in mDA is achieved by classical deep learning
encoding functions like tanh() whereas we renormalize vect ors and then rotate them. Second, we
do not add noise to the input vectors – mSVD is more efﬁcient fo r this reasons, but it remains to
be seen if it can achieve the same level of performance as mDA. Third, the mSVD architecture
proposed here, which changes the objects to be represented f rom small frequent units to larger less
frequent units when going one layer up, can be seen as an alter native (though less general since it’s
customized for natural language) way of extending to very high dimensions.
5
4 Conclusion
As a next step a direct comparison should be performed of SVD 2 with traditional deep learning
(Hinton et al., 2006). As we have argued, SVD 2 would be an interesting alternative to deep learn-
ing initialization methods currently used since SVD is efﬁc ient and a simple and well understood
formalism. But this argument is only valid if the resulting r epresentations are of comparable qual-
ity. Datasets and tasks for this comparative evaluation cou ld e.g. be those of Turian et al. (2010),
Maas et al. (2011), and Socher et al. (2011).
References
Anandkumar, Animashree, Dean P. Foster, Daniel Hsu, Sham M. Kakade, and Yi-Kai Liu. 2012.
Two svds sufﬁce: Spectral decompositions for probabilistic topic modeling and latent dirichlet allo-
cation. CoRR, abs/1204.6703.
Bengio, Yoshua. 2009. Learning deep architectures for AI. Foundations and Trends in Machine
Learning, 2(1):1–127.
Blei, David M., Andrew Y . Ng, and Michael I. Jordan. 2003. Lat ent Dirichlet allocation. JMLR,
3:993–1022.
Chen, Minmin, Zhixiang Eddie Xu, Kilian Q. Weinberger, and Fei Sha. 2012. Marginalized denois-
ing autoencoders for domain adaptation. In ICML.
Collobert, Ronan and Jason Weston. 2008. A uniﬁed architect ure for natural language processing:
Deep neural networks with multitask learning. In ICML.
Glorot, Xavier, Antoine Bordes, and Yoshua Bengio. 2011. Do main adaptation for large-scale sen-
timent classiﬁcation: A deep learning approach. In ICML, pages 513–520.
Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep
belief nets. Neural Computation, 18(7):1527–1554.
Hofmann, Thomas. 1999. Probabilistic latent semantic inde xing. In SIGIR, pages 50–57.
Lee, David D. and H. Sebastian Seung. 1999. Learning the part s of objects by non-negative matrix
factorization. Nature, 401:788.
Maas, Andrew L., Raymond E. Daly, Peter T. Pham, Dan Huang, An drew Y . Ng, and Christopher
Potts. 2011. Learning word vectors for sentiment analysis. In ACL, pages 142–150.
Pang, Bo and Lillian Lee. 2004. A sentimental education: Sen timent analysis using subjectivity
summarization based on minimum cuts. In Proc. of ACL.
Sch¨ olkopf, Bernhard, Alex J. Smola, and Klaus-Robert M¨ uller. 1998. Nonlinear component analy-
sis as a kernel eigenvalue problem. Neural Computation, 10(5):1299–1319.
Sch¨ utze, Hinrich. 1995. Distributional part-of-speech t agging. In Conference of the European
Chapter of the Association for Computational Linguistics, pages 141–148.
Socher, Richard, Jeffrey Pennington, Eric H. Huang, Andrew Y . Ng, and Christopher D. Manning.
2011. Semi-supervised recursive autoencoders for predict ing sentiment distributions. In EMNLP,
pages 151–161.
Socher, Richard, Brody Huval, Christopher D. Manning, and Andrew Y . Ng. 2012. Semantic com-
positionality through recursive matrix-vector spaces. In EMNLP-CoNLL, pages 1201–1211.
Thorpe, Simon. 2010. Grandmother cells and distributed representations. In Kriegeskorte, Nikolaus
and Gabriel Kreiman, editors, Understanding visual population codes. Toward a common multivari-
ate framework for cell recording and functional imaging. MIT Press.
Turian, Joseph, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and
general method for semi-supervised learning. In ACL, pages 384–394.
Turney, Peter D. and Patrick Pantel. 2010. From frequency to meaning: Vector space models of
semantics. J. Artif. Intell. Res. (JAIR), 37:141–188.
6


Peer review:
[Review]:
This paper introduces a novel method to induce word vector representations from a corpus of unlabeled text. The method relies upon 'stacking' singular value decomposition with an intermediate normalization nonlinearity. The authors propose 'focality' as a metric for quantifying the quality of a learned representation. Finally, control experiments on a small collection of sentence text demonstrates stacked SVD as producing more focal representations than a single SVD.

The method of stacked SVD is novel as far as I know, but could perhaps be generalized to use other nonlinearities between the two SVD layers than length normalization alone. As the authors acknowledge, SVD is a linear transform so the intermediate nonlinearity is important as to have the entire method not reduce to a single linear transform. There are a huge number of ways to use matrix factorization to induce word vectors, Turney & Pantel (JAIR 2010) give a nice review. I would like to better understand the proposed method in the context of the many alternatives to SVD factorization (e.g. kernel PCA etc.). 

The introduced notion of focality might serve as a good metric for analysis of learned representation quality. It seems however that measuring focality is only possible with brute force experiments which could make it an unwieldy tool. Expanding on focality as a tool for representation evaluation, both in theory and practice, could strengthen this paper significantly. 

The experiments use a small text corpus to demonstrate two SVDs as producing better representations than one. There is much room for improvement in the experiment section. In particular, there are several word representation benchmarks the authors could use to assess the quality of the proposed method relative to previous work:
- Turian et al (ACL 2010) compare several word representations and release benchmark code. 
- Socher et al (EMNLP 2011) release a multi-dimensional sentiment analysis corpus and use neural nets to train word representations
- Maas et al (ACL 2011) release a large semi-supervised sentiment analysis corpus and directly compare SVD-obtained word representations with other models

The experiments given are a reasonable sanity check for the model and demonstration of the introduced focality metric. However, the paper would be greatly improved by comparing to previous work on at least one of the tasks in papers listed above. 

The 1LAYER vs 2LAYER experiment is not clearly explained. Please expand on the difference in 1 vs 2 layers and the experimental result.

To summarize:
- Novel layer-wise SVD approach to inducing word vectors. Needs to be better explained in the context of matrix factorization alternatives
- Novel 'focality' metric which could serve as a tool for measuring learned representation quality. Metric needs more explanation / analysis.
- Experiments don't demonstrate the model relative to previous work. This is a major omission since many recent alternatives exist and comparison experiments should be straightforward with several public datasets exist
- Overall paper is fairly clear but could use some work