Learning Features with Structure-Adapting Multi-view Exponential Family
    Harmoniums
Authors: YoonSeop Kang, Seungjin Choi
ICLR 2013

Abstract: We proposea graphical model for multi-view feature extraction that automatically adapts its structure to achieve better representation of data distribution. The proposed model, structure-adapting multi-view harmonium (SA-MVH) has switch parameters that control the connection between hidden nodes and input views, and learn the switch parameter while training. Numerical experiments on synthetic and a real-world dataset demonstrate the useful behavior of the SA-MVH, compared to existing multi-view feature extraction methods.

arXiv:1301.3539v1  [cs.LG]  16 Jan 2013
Learning Features with Structure-Adapting
Multi-view Exponential Family Harmoniums
Yoonseop Kang1 Seungjin Choi1,2,3
Department of Computer Science and Engineering1,
Division of IT Convergence Engineering2,
Department of Creative Excellence Engineering3,
Pohang University of Science and Technology (POSTECH)
Pohang, South Korea, 790-784.
{e0en,seungjin}@postech.ac.kr
Abstract
We propose a graphical model for multi-view feature extraction that automatically
adapts its structure to achieve better representation of da ta distribution. The pro-
posed model, structure-adapting multi-view harmonium (SA-MVH) has switch
parameters that control the connection between hidden nodes and input v iews,
and learn the switch parameter while training. Numerical ex periments on syn-
thetic and a real-world dataset demonstrate the useful beha vior of the SA-MVH,
compared to existing multi-view feature extraction methods.
1 Introduction
Earlier multi-view feature extraction methods including canonical correlation analysis [1] and dual-
wing harmonium (DWH) [2] assume that all views can be describ ed using a single set of shared
hidden nodes. However, these methods fail when real-world data with partially correlated views are
given. More recent methods like factorized orthogonal late nt space [3] or multi-view harmonium
(MVH) [4] assume that views are generated from two sets of hid den nodes: view-speciﬁc hidden
nodes and shared ones. Still, these models rely on the pre-deﬁned connection structure, and deciding
the number of shared and view-speciﬁc hidden nodes requires a great human effort.
In this paper, we propose structure-adapting multi-view ha rmonium (SA-MVH) which avoids all
of the problems mentioned above. Instead of explicitly deﬁn ing view-speciﬁc and hidden nodes in
prior to the training, we only use one set of hidden nodes and l et each one of them to decide the
existence of connection to views using switch parameters during the training. In this manner, SA-
MVH automatically decides the number of view-speciﬁc laten t variables and also captures partial
correlation among views.
2 The Proposed Model
The deﬁnition of SA-MVH begins with choosing marginal distr ibutions of visible node sets v(k)
and a set of hidden nodes h from exponential family distributions:
p(v(k)
i ) ∝ exp(
∑
a
ξ(k)
ia f(k)
ia (v(k)
i ) − A(k)
i ({ξ(k)
ia })),
p(hj ) ∝ exp(
∑
b
λjb gjb(hj ) − Bj ({λjb })), (1)
f(·), g(·) are sufﬁcient statistics, ξ, λ are natural parameters, and A, B are log-partition functions.
1
...
... ...
(a) DWH
...
... ...
... ...
(b) MVH
...
... ...
(c) SA-MVH
Figure 1: Graphical models of (a) dual-wing harmonium, (b) m ulti-view harmonium, and (c)
structure-adapting multi-view harmonium.
Connections between visible nodes and hidden nodes of SA-MV H are deﬁned by weight matrices
{W (k)} and switch parameters σ(skj ) ∈ [0, 1], where σ(·) is a sigmoid function. A switch skj
controls the connection between k-th view and j-th hidden node by being multiplied to the j-th
column of weight matrix W (k) (Figure 1). When σ(skj ) is large (> 0. 5), we consider the view and
the hidden node to be connected. With the quadratic term including weights and switch parameters,
the joint distribution of SA-MVH is deﬁned as below:
p({v(k)}, h) ∝ exp
( ∑
k,i,j
σ(skj )W (k)
ij f(k)
i (v(k)
i )gj (hj ) −
∑
k,i
ξ(k)
i f(k)
i (v(k)
i ) −
∑
j
λj gj (hj )
)
. (2)
note that indices a and b are omitted to keep the notations uncluttered.
We learn the parameters W (k), ξ(k), λ, and switch parameters skj by maximizing the likelihood of
model via gradient ascent. The likelihood of SA-MVH is deﬁne d as the joint distribution of nodes
summed over hidden nodes h:
L = ⟨log p({v(k)})⟩data =
⟨
log
∑
h
p({v(k)}, h)
⟩
data, (3)
where ⟨·⟩data represents expectation over data distribution. Then the gradient of log-likelihood with
respect to the parameters W (k), ξ(k), λ, and skj are derived as follows:
∂L
∂W (k)
ij
∝
⟨
σ(skj )fi(v(k)
i )B′
j (ˆλj )
⟩
data −
⟨
σ(skj )fi(v(k)
i )B′
j (ˆλj )
⟩
model (4)
∂L
∂ξ(k)
i
∝
⟨
f(k)
i (v(k)
i )
⟩
data −
⟨
f(k)
i (v(k)
i )
⟩
model (5)
∂L
∂λ j
∝
⟨
B′
j (ˆλj )
⟩
data −
⟨
B′
j (ˆλj )
⟩
model, (6)
∂L
∂skj
∝
⣨
σ′(skj )W (k)
ij fi(v(k)
i )B′
j (ˆλj )
⟩
data
−
⣨
σ′(skj )W (k)
ij fi(v(k)
i )B′
j (ˆλj )
⟩
model
(7)
where ⟨·⟩model represents expectation over model distribution p({v(k)}, h) and ˆξ(k)
i = ξ(k)
i +∑
j σ(skj )W (k)
ij gj(hj ), ˆλj = λj + ∑
k,i σ(skj )W (k)
ij fi(v(k)
i ) are shifted parameters.
3 Numerical Experiments
3.1 Feature Extraction on Noisy Arabic-Roman Digit Dataset
To simulate the view-speciﬁc and shared properties of multi -view data, we designed a synthetic
dataset which contains 11,800 pairs of Arabic digits and the corresponding Roman digits written in
various fonts. For each pair, we added random vertical line n oises to Arabic digits, and horizontal
line noises to Roman digits (Figure 2-(a)). SA-MVH trained with 200 hidden nodes found 95 shared
features (with connection to both views), and 47 view-speci ﬁc features for Roman digits, and 32
for Arabic digits. Remaining 26 were not connected to any vie ws and ignored. Most of the shared
features were noise-free and encoded parts of Roman and Arab ic numbers (Figure 2-(b)). On the
other hand, the view-speciﬁc features had components with horizontal or vertical noises, as well as
the parts of the numbers (Figure 2-(c)). In this example, SA- MVH automatically separated view-
speciﬁc and shared information without any prior speciﬁcation of the graph structure.
2
(a)
 (b) Shared features
 (c) View-speciﬁc features
Figure 2: (a) 10 samples from Noisy Arabic-Roman digit dataset, (b) shared features, and (c) view-
speciﬁc features learned by SA-MVH.
Table 1: Image classiﬁcation accuracy of k-nn classiﬁer using feature extraction methods trained on
Caltech-256 dataset. For each value of k, the best result is marked as bold text.
Method # 10-NN 30-NN 50-NN 70-NN 100-NN
Sparse Filtering 0.161 0.165 0.163 0.16 0.155
DWH 0.237 0.231 0.217 0.207 0.194
MVH 0.239 0.225 0.216 0.203 0.191
SA-MVH 0.246 0.232 0.223 0.212 0.198
3.2 Image Classiﬁcation on Caltech-256 Dataset
We extracted 512 dimensions of GIST features and 1,536 dimen sions of histogram of gradients
(HoG) features from Caltech-256 dataset to simulate multi-view settings. SA-MVH and other multi-
view feature extraction methods based on harmonium – DWH andMVH were trained on the dataset
for comparison. We also compared our method to Sparse Filter ing [5], which is not a harmonium-
based method. We trained the feature extraction methods and tested the methods with k-nearest
neighbor classiﬁers (Table 1). SA-MVH resulted better than other feature extraction models in this
experiment, regardless of the value of k for nearest neighbor classiﬁer.
4 Conclusion
In this paper, we have proposed the multi-view feature extraction model that automatically decides
relations between latent variables and input views. The pro posed method, SA-MVH models multi-
view data distribution with less restrictive assumption and also reduces the number of parameters to
tune by human hand. SA-MVH introduces switch parameters that control the connections between
hidden nodes and input views, and ﬁnd the desirable conﬁguration while training. We have demon-
strated the effectiveness of our approach by comparing our model to existing models in experiments
on synthetic dataset, and image classiﬁcation with simulated multi-view setting.
References
[1] D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, “Canonical correlation analysis: An overview
with applications to learning methods,” Neural Computation, vol. 16, pp. 2639–2664, 2004.
[2] E. P. Xing, R. Yan, and A. G. Hauptmann, “Mining associate d text and images with dual-wing
harmonium,” in Proceedings of the Annual Conference on Uncertainty in Artiﬁcial Intelligence
(UAI), Edinburgh, UK, 2005.
[3] M. Salzmann, C. H. Ek, R. Urtasun, and T. Darrell, “Factor ized orthogonal latent spaces,” in
Proceedings of the International Conference on Artiﬁcial Intelligence and Statistics (AISTATS),
Sardinia, Italy, 2010.
[4] Y . Kang and S. Choi, “Restricted deep belief networks formulti-view learning,” in Proceedings
of the European Conference on Machine Learning and Principl es and Practice of Knowledge
Discovery in Databases (ECML-PKDD), Athens, Greece, 2011.
[5] J. Ngiam, P. W. Koh, Z. Chen, S. A. Bhaskar, and A. Y . Ng, “Sp arse ﬁltering,” in Advances in
Neural Information Processing Systems (NIPS), vol. 23. MIT Press, 2011.
3


Peer review:
[Review]:
The authors propose a bipartite, undirected graphical model for multiview learning, called structure-adapting multiview harmonimum (SA-MVH). The model is based on their earlier model called multiview harmonium (MVH) (Kang&Choi, 2011) where hidden units were separated into a shared set and view-specific sets. Unlike MVH which explicitly restricts edges, the visible and hidden units in the proposed SA-MVH are fully connected to each other with switch parameters s_{kj} indicating how likely the j-th hidden unit corresponds to the k-th view.

It would have been better if the distribution of s_{kj}'s (or sigma(s_{kj})) was provided. Unless the distribution has clear modes near 0 and 1, it would be difficult to tell why this approach of learning w^{(k)}_{ij} and s_{kj} separately is better than just learning 	ilde{w}^{(k)}_{ij} = w^{(k)}_{ij} sigma s_{kj} all together (as in dual-wing harmonium, DWH). Though, the empirical results (experiment 2) show that the features extracted by SA-MVH outperform both MVH and DWH.

The visualizations of shared and view-specific features from the first experiment do not seem to clearly show the power of the proposed method. For instance, it's difficult to say that the filters of roman digits from the shared features do seem to have horizontal noise. It would be better to try some other tasks with the trained model. Would it be possible to sample clean digits (without horizontal or vertical noise) from the model if the view-speific features were forced off? Would it be possible to denoise the corrupted digits? and so on..

Typo:

- Fig. 1 (c): sigma(s_{1j}) and sigma(s_{2j})