Learning Features with Structure-Adapting Multi-view Exponential Family
    Harmoniums
Authors: YoonSeop Kang, Seungjin Choi
ICLR 2013

Abstract: We proposea graphical model for multi-view feature extraction that automatically adapts its structure to achieve better representation of data distribution. The proposed model, structure-adapting multi-view harmonium (SA-MVH) has switch parameters that control the connection between hidden nodes and input views, and learn the switch parameter while training. Numerical experiments on synthetic and a real-world dataset demonstrate the useful behavior of the SA-MVH, compared to existing multi-view feature extraction methods.

arXiv:1301.3539v1  [cs.LG]  16 Jan 2013
Learning Features with Structure-Adapting
Multi-view Exponential Family Harmoniums
Yoonseop Kang1 Seungjin Choi1,2,3
Department of Computer Science and Engineering1,
Division of IT Convergence Engineering2,
Department of Creative Excellence Engineering3,
Pohang University of Science and Technology (POSTECH)
Pohang, South Korea, 790-784.
{e0en,seungjin}@postech.ac.kr
Abstract
We propose a graphical model for multi-view feature extraction that automatically
adapts its structure to achieve better representation of da ta distribution. The pro-
posed model, structure-adapting multi-view harmonium (SA-MVH) has switch
parameters that control the connection between hidden nodes and input v iews,
and learn the switch parameter while training. Numerical ex periments on syn-
thetic and a real-world dataset demonstrate the useful beha vior of the SA-MVH,
compared to existing multi-view feature extraction methods.
1 Introduction
Earlier multi-view feature extraction methods including canonical correlation analysis [1] and dual-
wing harmonium (DWH) [2] assume that all views can be describ ed using a single set of shared
hidden nodes. However, these methods fail when real-world data with partially correlated views are
given. More recent methods like factorized orthogonal late nt space [3] or multi-view harmonium
(MVH) [4] assume that views are generated from two sets of hid den nodes: view-speciﬁc hidden
nodes and shared ones. Still, these models rely on the pre-deﬁned connection structure, and deciding
the number of shared and view-speciﬁc hidden nodes requires a great human effort.
In this paper, we propose structure-adapting multi-view ha rmonium (SA-MVH) which avoids all
of the problems mentioned above. Instead of explicitly deﬁn ing view-speciﬁc and hidden nodes in
prior to the training, we only use one set of hidden nodes and l et each one of them to decide the
existence of connection to views using switch parameters during the training. In this manner, SA-
MVH automatically decides the number of view-speciﬁc laten t variables and also captures partial
correlation among views.
2 The Proposed Model
The deﬁnition of SA-MVH begins with choosing marginal distr ibutions of visible node sets v(k)
and a set of hidden nodes h from exponential family distributions:
p(v(k)
i ) ∝ exp(
∑
a
ξ(k)
ia f(k)
ia (v(k)
i ) − A(k)
i ({ξ(k)
ia })),
p(hj ) ∝ exp(
∑
b
λjb gjb(hj ) − Bj ({λjb })), (1)
f(·), g(·) are sufﬁcient statistics, ξ, λ are natural parameters, and A, B are log-partition functions.
1
...
... ...
(a) DWH
...
... ...
... ...
(b) MVH
...
... ...
(c) SA-MVH
Figure 1: Graphical models of (a) dual-wing harmonium, (b) m ulti-view harmonium, and (c)
structure-adapting multi-view harmonium.
Connections between visible nodes and hidden nodes of SA-MV H are deﬁned by weight matrices
{W (k)} and switch parameters σ(skj ) ∈ [0, 1], where σ(·) is a sigmoid function. A switch skj
controls the connection between k-th view and j-th hidden node by being multiplied to the j-th
column of weight matrix W (k) (Figure 1). When σ(skj ) is large (> 0. 5), we consider the view and
the hidden node to be connected. With the quadratic term including weights and switch parameters,
the joint distribution of SA-MVH is deﬁned as below:
p({v(k)}, h) ∝ exp
( ∑
k,i,j
σ(skj )W (k)
ij f(k)
i (v(k)
i )gj (hj ) −
∑
k,i
ξ(k)
i f(k)
i (v(k)
i ) −
∑
j
λj gj (hj )
)
. (2)
note that indices a and b are omitted to keep the notations uncluttered.
We learn the parameters W (k), ξ(k), λ, and switch parameters skj by maximizing the likelihood of
model via gradient ascent. The likelihood of SA-MVH is deﬁne d as the joint distribution of nodes
summed over hidden nodes h:
L = ⟨log p({v(k)})⟩data =
⟨
log
∑
h
p({v(k)}, h)
⟩
data, (3)
where ⟨·⟩data represents expectation over data distribution. Then the gradient of log-likelihood with
respect to the parameters W (k), ξ(k), λ, and skj are derived as follows:
∂L
∂W (k)
ij
∝
⟨
σ(skj )fi(v(k)
i )B′
j (ˆλj )
⟩
data −
⟨
σ(skj )fi(v(k)
i )B′
j (ˆλj )
⟩
model (4)
∂L
∂ξ(k)
i
∝
⟨
f(k)
i (v(k)
i )
⟩
data −
⟨
f(k)
i (v(k)
i )
⟩
model (5)
∂L
∂λ j
∝
⟨
B′
j (ˆλj )
⟩
data −
⟨
B′
j (ˆλj )
⟩
model, (6)
∂L
∂skj
∝
⣨
σ′(skj )W (k)
ij fi(v(k)
i )B′
j (ˆλj )
⟩
data
−
⣨
σ′(skj )W (k)
ij fi(v(k)
i )B′
j (ˆλj )
⟩
model
(7)
where ⟨·⟩model represents expectation over model distribution p({v(k)}, h) and ˆξ(k)
i = ξ(k)
i +∑
j σ(skj )W (k)
ij gj(hj ), ˆλj = λj + ∑
k,i σ(skj )W (k)
ij fi(v(k)
i ) are shifted parameters.
3 Numerical Experiments
3.1 Feature Extraction on Noisy Arabic-Roman Digit Dataset
To simulate the view-speciﬁc and shared properties of multi -view data, we designed a synthetic
dataset which contains 11,800 pairs of Arabic digits and the corresponding Roman digits written in
various fonts. For each pair, we added random vertical line n oises to Arabic digits, and horizontal
line noises to Roman digits (Figure 2-(a)). SA-MVH trained with 200 hidden nodes found 95 shared
features (with connection to both views), and 47 view-speci ﬁc features for Roman digits, and 32
for Arabic digits. Remaining 26 were not connected to any vie ws and ignored. Most of the shared
features were noise-free and encoded parts of Roman and Arab ic numbers (Figure 2-(b)). On the
other hand, the view-speciﬁc features had components with horizontal or vertical noises, as well as
the parts of the numbers (Figure 2-(c)). In this example, SA- MVH automatically separated view-
speciﬁc and shared information without any prior speciﬁcation of the graph structure.
2
(a)
 (b) Shared features
 (c) View-speciﬁc features
Figure 2: (a) 10 samples from Noisy Arabic-Roman digit dataset, (b) shared features, and (c) view-
speciﬁc features learned by SA-MVH.
Table 1: Image classiﬁcation accuracy of k-nn classiﬁer using feature extraction methods trained on
Caltech-256 dataset. For each value of k, the best result is marked as bold text.
Method # 10-NN 30-NN 50-NN 70-NN 100-NN
Sparse Filtering 0.161 0.165 0.163 0.16 0.155
DWH 0.237 0.231 0.217 0.207 0.194
MVH 0.239 0.225 0.216 0.203 0.191
SA-MVH 0.246 0.232 0.223 0.212 0.198
3.2 Image Classiﬁcation on Caltech-256 Dataset
We extracted 512 dimensions of GIST features and 1,536 dimen sions of histogram of gradients
(HoG) features from Caltech-256 dataset to simulate multi-view settings. SA-MVH and other multi-
view feature extraction methods based on harmonium – DWH andMVH were trained on the dataset
for comparison. We also compared our method to Sparse Filter ing [5], which is not a harmonium-
based method. We trained the feature extraction methods and tested the methods with k-nearest
neighbor classiﬁers (Table 1). SA-MVH resulted better than other feature extraction models in this
experiment, regardless of the value of k for nearest neighbor classiﬁer.
4 Conclusion
In this paper, we have proposed the multi-view feature extraction model that automatically decides
relations between latent variables and input views. The pro posed method, SA-MVH models multi-
view data distribution with less restrictive assumption and also reduces the number of parameters to
tune by human hand. SA-MVH introduces switch parameters that control the connections between
hidden nodes and input views, and ﬁnd the desirable conﬁguration while training. We have demon-
strated the effectiveness of our approach by comparing our model to existing models in experiments
on synthetic dataset, and image classiﬁcation with simulated multi-view setting.
References
[1] D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, “Canonical correlation analysis: An overview
with applications to learning methods,” Neural Computation, vol. 16, pp. 2639–2664, 2004.
[2] E. P. Xing, R. Yan, and A. G. Hauptmann, “Mining associate d text and images with dual-wing
harmonium,” in Proceedings of the Annual Conference on Uncertainty in Artiﬁcial Intelligence
(UAI), Edinburgh, UK, 2005.
[3] M. Salzmann, C. H. Ek, R. Urtasun, and T. Darrell, “Factor ized orthogonal latent spaces,” in
Proceedings of the International Conference on Artiﬁcial Intelligence and Statistics (AISTATS),
Sardinia, Italy, 2010.
[4] Y . Kang and S. Choi, “Restricted deep belief networks formulti-view learning,” in Proceedings
of the European Conference on Machine Learning and Principl es and Practice of Knowledge
Discovery in Databases (ECML-PKDD), Athens, Greece, 2011.
[5] J. Ngiam, P. W. Koh, Z. Chen, S. A. Bhaskar, and A. Y . Ng, “Sp arse ﬁltering,” in Advances in
Neural Information Processing Systems (NIPS), vol. 23. MIT Press, 2011.
3


Peer review:
[Review]:
The paper introduces an new algorithm for simultaneously learning a hidden layer (latent representation) for multiple data views as well as automatically segmenting that hidden layer into shared and view-specific nodes. It builds on the previous multi-view harmonium (MVH) algorithm by adding (sigmoidal) switch parameters that turn a connection on or off between a view and hidden node and uses gradient descent to learn those switch parameters. The optimization is similar to MVH, with a slight modification on the joint distribution between views and hidden nodes, resulting in a change in the gradients for all parameters and a new switch variable to descend on.

This new algorithm, therefore, is somewhat novel; the quality of the explanation and writing is high; and the experimental quality is reasonable.

Pros

1. The paper is well-written and organized.

2. The algorithm in the paper proposes a way to avoid hand designing shared and private (view-specific) nodes, which is an important contribution.

3. The experimental results indicate some interesting properties of the algorithm, in particular demonstrating that the algorithm extracts reasonable shared and view-specific hidden nodes.

Cons
1. The descent directions have W and the switch parameters, s_kj, coupled, which might make learning slow. Experimental results should indicate computation time.

2. The results do not have error bars (in Table 1), so it is unclear if they are statistically significant (the small difference suggests that they may not be).

3. The motivation in this paper is to enable learning of the private and shared representations automatically. However, DWH (only a shared representation) actually seems to perform generally better that MVH (shared and private). The experiments should better explore this question. It might also be a good idea to have a baseline comparison with CCA. 

4. In light of Con (3), the algorithm should also be compared to multi-view algorithms that learn only shared representations but do not require the size of the hidden-node set to be fixed (such as the recent relaxed-rank convex multi-view approach in 'Convex Multiview Subspace Learning', M. White, Y. Yu, X. Zhang and D. Schuurmans, NIPS 2012). In this case, the relaxed-rank regularizer does not fix the size of the hidden node set, but regularizes to set several hidden nodes to zero. This is similar to the approach proposed in this paper where a node is not used if the sigmoid value is < 0.5. 
Note that these relaxed-rank approaches do not explicitly maximize the likelihood for an exponential family distribution; instead, they allow general Bregman divergences which have been shown to have a one-to-one correspondence with exponential family distributions (see 'Clustering with Bregman divergences' A. Banerjee, S. Merugu, I. Dhillon and J. Ghosh, JMLR 2005). Therefore, by selecting a certain Bregman divergence, the approach in this paper can be compared to the relaxed-rank approaches.