Probabilistic Latent Variable Models for Unsupervised
Many-to-Many Object Matching
Tomoharu Iwataa,∗, Tsutomu Hiraoa , Naonori Uedaa
a NTT Communication Science Laboratories

Abstract
Object matching is an important task for finding the correspondence between
objects in different domains, such as documents in different languages and users
in different databases. In this paper, we propose probabilistic latent variable
models that offer many-to-many matching without correspondence informationor similarity measures between different domains. The proposed model assumes
that there is an infinite number of latent vectors that are shared by all domains, and that each object is generated from one of the latent vectors and adomain-specific projection. By inferring the latent vector used for generatingeach object, objects in different domains are clustered according to the vectors that they share. Thus, we can realize matching between groups of objects
in different domains in an unsupervised manner. We give learning proceduresof the proposed model based on a stochastic EM algorithm. We also derive
learning procedures in a semi-supervised setting, where correspondence information for some objects are given. The effectiveness of the proposed models isdemonstrated by experiments on synthetic and real data sets.
Keywords: latent variable model, mixture model, object matching∗ Corresponding author

Email addresses: iwata.tomoharu@lab.ntt.co.jp (Tomoharu Iwata),
hirao.tsutomu@lab.ntt.co.jp (Tsutomu Hirao), ueda.naonori@lab.ntt.co.jp (Naonori
Ueda)

Preprint submitted to Information Processing & Management

January 5, 2016

1. Introduction
Object matching is an important task for finding the correspondence between
objects in different domains. Examples of object matching include matching animage with an annotation [1], an English word with a French word [2], and5user identification in different databases for recommendation [3]. Most object
matching methods require similarity measures between objects in the different
domains, or paired data that contain correspondence information. When asimilarity measure is given, we can match objects by finding pairs of objects
that maximize the sum of the similarities. When correspondence information is10

given, we can obtain a mapping function from one domain to another by using
supervised learning methods, and then we can calculate the similarities between
objects in different domains.
However, similarity measures and correspondences might not be available.
Defining similarities and generating correspondences incur considerable cost and

15time, and they are sometimes unobtainable because of the need to preserve privacy. For example, dictionaries between some languages might not exist, anddifferent online stores cannot share user identification. For these situations, unsupervised object matching methods have been proposed; they include kernelized sorting [4], least squares object matching [5], matching canonical correlation

20

analysis [6], and its Bayesian extension [7, 8]. These methods find one-to-one
matches. However, matching is not necessarily one-to-one in some applications.
For example, when matching English and German documents, multiple English
documents with the similar topic could correspond to multiple German documents. In image annotation, related annotations ‘tree’, ‘wood’ and ‘forest’ can25

be attached to multiple images that look similar to each other. Other limitations of these methods are that the number of domains is limited to two, andthe numbers of objects in the different domains must be the same. There can be
more than two domains in some applications, for example matching multilingual
documents such as English, French and German, and the number of documents30

for each language can be different.

2

In this paper, we propose a probabilistic latent variable model for finding
correspondence between object clusters in multiple domains without correspondence information. We assume that objects in different domains share a hidden
structure, which is represented by an infinite number of latent vectors that are
35

shared by all domains. Each object is generated from one of the latent vectors
and a domain-specific linear projection. The latent vectors used for generating
objects are unknown. By assigning a latent vector to each object, we can allocate objects in different domains to common clusters, and find many-to-many
matches. The number of clusters is automatically inferred from the given data40

by using a Dirichlet process prior. The proposed model can handle more than
two domains with different numbers of objects. We infer the proposed modelusing a stochastic EM algorithm. The proposed model can ignore arbitrary linear transformations for different domains by inferring the domain-specific linear
projection, and can find cluster matching in different domains, where similarity

45

cannot be calculated directly.
The proposed model assumes a Gaussian distribution for each observed variable, and its mean is determined by a latent vector and a linear projection matrix. It is an extension of probabilistic principle component analysis (PCA) [9]and factor analysis (FA) [10], which are representative probabilistic latent vari-

50

able models. With probabilistic PCA and FA, each object is associated with alatent vector. On the other hand, with the proposed model, the latent vector
that is assigned to each object is hidden. When the number of domains is one,and every object is assigned to a cluster that is different from those of otherobjects, the proposed model corresponds to probabilistic principle component

55

analysis.
The proposed model can be also used in a semi-supervised setting, where
correspondence information for some objects is given [4, 11]. The information
assists matching by incorporating a condition stating that the cluster assignments of the corresponding objects must be the same. We derive learning pro-60

cedures for the semi-supervised setting by modifying the learning proceduresfor unsupervised setting.
3This paper is an extended version of [12]. We newly proposed the inference
procedure for a semi-supervised setting, and added derivations and experiments.
The remainder of this paper is organized as follows. In Section 2, we review
65

related work. In Section 3, we formulate the proposed model and describe its
efficient learning procedures. We also present the learning procedures for asemi-supervised setting and for missing data. In Section 4, we demonstrate theeffectiveness of the proposed models with experiments on synthetic and real
data sets. Finally, we present concluding remarks and a discussion of future

70

work in Section 5.

2. Related work
2.1. Unsupervised object matching
Unsupervised object matching is a task that involves finding the correspondence between objects in different domains without correspondence information.
75

For example, kernelized sorting [4] finds the correspondence by permuting a set
to maximize the dependence between two domains where the Hilbert Schmidt
Independence Criterion (HSIC) is used as the dependence measure. Kernelized
sorting requires the two domains have the same number of objects. Convex
kernelized sorting [13] is a convex formulation of kernelized sorting. Match-80

ing canonical correlation analysis (MCCA) [6] is another unsupervised object
matching method based on a probabilistic model, where bilingual translation
lexicons are learned from two monolingual corpora. MCCA simultaneously finds
latent variables that represent correspondences and latent vectors so that thelatent vectors of corresponding objects exhibit the maximum correlation. [14]85also proposed a method for unsupervised object matching that is related to
MCCA. These methods assume one-to-one matching of objects. On the other
hand, the proposed model can find many-to-many matching, and is applicable toobjects in more than two domains. Bayesian solution for MCCA (BMCCA) has
been proposed [7, 8]. BMCCA assumes that latent vectors are generated from a

90Gaussian distribution, and finds one-to-one matching by inferring a permutation

4

matrix. In contrast, the proposed model assumes that latent vectors are generated from an infinite Gaussian mixture model [15], and finds many-to-many
matching by inferring cluster assignments.
Manifold alignment is related to the proposed model because they both95

find latent vectors of multiple sets in a joint latent space. The unsupervised
manifold alignment method [16] finds latent vectors of different domains in ajoint latent space in an unsupervised manner. The method first identifies all
possible matches for each example by leveraging its local geometry, and thenfinds an embedding in the latent space. The method requires permutations

100of the order of the factorial of the size of neighborhoods to match the local
geometry. Note that the method does not explicitly find correspondences.
There has been some work on improving the learning performance of theclassification task by using labeled objects in different domains without correspondence information. For example, multiple outlook mapping (MOMAP) [17]105

improves the performance by matching the moments of the empirical distributions for each class of two domains. [18] proposed a transfer learning methodthat improves the learning performance by embedding both source and target
domains in a joint latent space when a limited number of target objects are labeled. These methods require labeled objects. On the other hand, the proposed110

method does not require any labeled objects.
Consensus clustering [19] tries to find a single clustering from a number ofdifferent clusterings. In the case that a clustering is obtained using data ineach domain, consensus clustering can find many-to-many matching. However,consensus clustering requires that objects are aligned across domains.

115

2.2. Latent variable models
The proposed model, as well as MCCA [6], can be considered as an unsupervised version of probabilistic canonical correlation analysis (CCA) [20]. Probabilistic CCA finds dependences between objects in two domains by projecting
objects into a latent space. It requires correspondence information between dif-120

ferent domains since it takes a set of paired objects as input. On the other hand,5the proposed model can find dependences in an unsupervised manner without
correspondence information by taking a set of objects for each domain as input.
CCA is being successfully used for a wide variety of applications, such as multilabel prediction [21, 22], information retrieval [23], and image annotation [24].
125

The proposed model can be used for these applications when supervised data
are unavailable.
Mixtures of latent variable models have been also proposed, such as mixtures of probabilistic principal component analyzers [25] and mixtures of factor
analyzer [26]. These mixture models assume that each object has its own latent130

vector, and different objects can be generated from different projection matrices,where the data are obtained from a domain. On the other hand, the proposedmodel assumes that different objects can share a latent vector, and all objects
from a domain are generated from a domain-specific projection matrix. Thus,
the proposed model can find many-to-many matching by assigning the same

135

latent vectors to different objects from multiple domain data.
The proposed model can be seen as a generalization of the infinite Gaussian
mixture model [15]. When the dimensionality of the latent space is the same asthat of the observed space and the projection matrix is the identity matrix forall domains, the proposed model corresponds to the infinite Gaussian mixture

140

model.
Recently, there have been proposed a number of deep learning methods for
obtaining latent representations of multimodal data, such as multimodal deeplearning [27], multimodal deep Boltzmann machine [28] and deep canonical correlation analysis [29]. These methods utilize deep architectures to model com-145

plex nonlinear transformations between latent representations and multimodal
observations. However, these methods require correspondence information tolearn the transformations, and is unapplicable in an unsupervised setting. We
can use deep learning to obtain latent representations for each domain in preprocessing, and apply the proposed method to the latent representations for

150

unsupervised object matching.

6

Table 1: Notation.

SymbolDescriptionD

number of domains

Ndnumber of objects in the dth domain

Mddimensionality of observed features in the dth domainKdimensionality of a latent vectorJnumber of clusters (latent vectors) to which objects are assigned

xdn

observation of the nth object in the dth domain, xdn ∈ RMd

zj

latent vector for the jth cluster, zj ∈ RK

Wd

projection matrix for the dth domain, Wd ∈ RMd ×K
P∞
mixture weight for the jth cluster, θj ≥ 0, j=1 θj = 1

θj3. Proposed method
3.1. Model
Suppose that we are given objects in D domains X = {Xd }D
d=1 , where
Mdd
Xd = {xdn }Nis then=1 is a set of objects in the dth domain, and xdn ∈ R
155feature vector of the nth object in the dth domain. Our notation is summarizedin Table 1. Note that we are unaware of any correspondence between objects
in different domains. The number of objects Nd and the dimensionality Md foreach domain can be different from those of other domains. The task is to matchgroups of objects across multiple domains in an unsupervised manner. Figure 1

160

shows an example of the input (top) and output (bottom) for the proposedmodel with two domains, where the numbers of objects in the first and second
domains are 50 and 60, and the numbers of observation features are 70 and 50,
respectively. The color shows the value of the observation feature. In this case,the proposed model found five clusters, and the object indices are permutated165

so that matched objects are aligned.
The model proposed for this task is a probabilistic latent variable model.
The proposed model assumes that 1) there is a potentially infinite number ofclusters, and 2)each cluster j has a latent vector zj ∈ RK in a K-dimensional
7

Domain 1

Domain 2
object index

object index
feature index
unsupervised
many-to-many
object matching

feature index

feature index

permutated object index

permutated object index

feature index

Figure 1: Example of the input (top) and output (bottom) for the proposed model withtwo domains. The vertical axis represents object indices, and the horizontal axis represents
observation feature indices for each matrix. The object indices of the output are permutated
so that matched objects are aligned.

latent space. Each object xdn in the dth domain is generated depending on
170

domain-specific projection matrix Wd ∈ RMd ×K and latent vector zsdn that isselected from a set of latent vectors Z = {zj }∞j=1 . Here, sdn ∈ {1, · · · , ∞} is thelatent cluster assignment of object xdn . Objects that use the same latent vector,
or that have the same cluster assignment, are considered to match. Figure 2
shows the relationship between latent vectors and objects in two domains, where

175

arrows that indicate the corresponding latent vectors for each object are hidden.
To be precise, the proposed model is an infinite mixture model, where theprobability of object xdn is given byp(xdn |Z, W , θ) =∞X

θj N (xdn |Wd zj , α−1 I),

(1)

j=1
∞
where W = {Wd }D
d=1 is a set of projection matrices, θ = {θj }j=1 is a set of8latent vectors
Z

W2

W1

X1

X2

objectsin domain 1

objectsin domain 2

Figure 2: Relationship between latent vectors and objects in two domains.

mixture weights, θj represents the probability that the jth cluster is chosen, α isa precision parameter, and N (µ, Σ) denotes a multivariate normal distribution
with mean µ and covariance matrix Σ. In the proposed model, a set of latent180

vectors Z is shared among multiple domains, but projection matrix Wd depends
on the domain. The benefit of the proposed model is as follows:• By sharing the latent vectors, we can assign objects in different domains
to common clusters, and find matchings between clusters.
• By employing domain-specific projection matrices, we can handle multiple

185

domains with different dimensionalities by adjusting the size of the matrices, and different statistical properties, such as means and covariances,by inferring the values of the matrices from the given data.
• Given latent vectors, an arbitrary number of objects can be generated
for each domain independently. Therefore, we can handle domains with190

different numbers of objects.
Specifically, the proposed model generates objects in multiple domains X
according to the following process,1. Draw mixture weights θ ∼ Stick(γ)
9

γ

a
b

α

z

θ

s

x

r

∞
N

W
D

Figure 3: Graphical model representation of the proposed model.

2. Draw a precision parameter α ∼ G(a, b)
195

3. For each cluster: j = 1, . . . , ∞(a) Draw a latent vector zj ∼ N (0, (αr)−1 I)
4. For each domain: d = 1, . . . , D(a) For each object: n = 1, . . . , Nd
i. Draw a cluster assignment sdn ∼ Categorical(θ)
ii. Draw an observation vector xdn ∼ N (Wd zsdn , α−1 I)

200Here, r is a parameter for controlling the precision of latent vectors, and Stick(γ)
is the stick-breaking process [30] that generates mixture weights for a Dirichlet
process with concentration parameter γ. By using a Dirichlet process, we can
automatically find the number of clusters from the given data. Discrete(·) rep205

resents a categorical distribution. G(a, b) represents a Gamma distribution withparameters a and b. We assume Gauss-Gamma distribution p(Z, α|a, b, r) =N (Z|0, (αr)−1 I)G(α|a, b) for the prior of latent vectors Z and α because it isa conjugate prior for a Gaussian p(xdn |zj , α, W ) = N (xdn |Wd zj , α−1 I) and it
enables us to analytically integrate out the latent vectors as shown in Appendix

210

A. Figure 3 shows a graphical model representation of the proposed model,
where shaded and unshaded nodes indicate observed and latent variables, respectively.
Dd
The joint probability of data X and cluster assignments S = {{sdn }Nn=1 }d=1

is given byp(X, S|W , a, b, r, γ) = p(S|γ)p(X|S, W , a, b, r).

10

(2)

By marginalizing out mixture weights θ, the first factor is calculated byp(S|γ) =where N =γJ

QJ

j=1 (N·j − 1)!

γ(γ + 1) · · · (γ + N − 1)

,

(3)

PD

d=1 Nd is the total number of objects, N·j represents the number ofobjects assigned to cluster j, and J is the number of clusters for which N·j > 0.
By marginalizing out latent vectors Z and precision parameter α, the second
factor of (2) is calculated byp(X|S, W , a, b, r) = (2π)−

P
d Md Nd
2

KJ

r 2

J
1
ba Γ(a0 ) Y|Cj | 2 .
00ab Γ(a) j=1

(4)

Here,
a0 = a +b0 = b +PD

d=1 Md Nd

2

,

D Nd
J
1 XX
1 X > −1x>
µ C µj ,dn xdn −2
2 j=1 j j
n=1

(5)

(6)

d=1

µj = Cj

D
X

Wd>

xdn ,

(7)

Ndj Wd> Wd + rI,(8)

n:sdn =jd=1

Cj−1 =

D
X

X

d=1

where Ndj is the number of objects assigned to cluster j in domain d. The
posterior for the precision parameter α is given byp(α|X, S, W , a, b) = G(a0 , b0 ),(9)and the posterior for the latent vector zj is given byp(zj |α, X, S, W , r) = N (µj , α−1 Cj ).
See Appendix B for the derivation.

11

(10)

3.2. Learning
We describe the learning procedures for the proposed model based on a215

stochastic EM algorithm, in which collapsed Gibbs sampling of cluster assignments S and the maximum joint likelihood estimation of projection matrices
W are alternately iterated while marginalizing out latent vectors Z, precisionparameter α, and cluster proportions θ. By collapsing, or marginalizing out,220

the variables, the time-consuming step of drawing the variables is skipped, andthe sample autocorrelations are usually reduced [31].
In the E-step, given the current state of all but one latent cluster assignments,
sdn , a new value for sdn is sampled from the following probability,p(sdn = j|X, S\dn , W , a, b, r, γ) ∝p(sdn = j, S\dn |γ) p(X|sdn = j, S\dn , W , a, b, r)·,p(S\dn |γ)p(X\dn |S\dn , W , a, b, r)(11)where \dn represents a value or set excluding the nth object in the dth domain,
and we use the fact that p(sdn = j|X, S\dn , W , a, b, r, γ) does not depend onp(xdn |W , a, b, r). See Appendix C for the derivation. The first factor is givenby
N·j\dnp(sdn = j, S\dn |γ)  N −1+γ=γ
p(S\dn |γ)N −1+γfor an existing cluster, j ∈ {1, · · · , J}
for a new cluster, j = J + 1,
(12)

using (3). By using (4), the second factor is given byp(X|sdn = j, S\dn , W , a, b, r)p(X\dn |S\dn , W , a, b, r)
0a0

M
− 2d

= (2π)r

1
2 I(j>J\dn )

\dn
1
b\dn
Γ(a0sdn =j ) |Cj,sdn =j | 2,
1
0a0s =j Γ(a0
|Cj\dn | 2
\dn )
b dn

(13)

sdn =jwhere subscript sdn = j indicates the value when object xdn is assigned tocluster j as follows,a0sdn =j = a0 ,

12

(14)

1
1 >
1 >
−1
−1
b0sdn =j = b0\dn + x>
dn xdn + µj\dn Cj\dn µj\dn − µj,sdn =j Cj,sdn =j µj,sdn =j ,
2
2
2
(15)
−1
µj,sdn =j = Cj,sdn =j (Wd> xdn + Cj\dnµj\dn ),

(16)

−1
−1
Cj,s
= Wd> Wd + Cj\dn
,
dn =j

(17)

and I(·) is used to denote the indicator function, i.e. I(A) = 1 if A is true,
I(A) = 0 otherwise.
In the M-step, the projection matrices W are estimated by maximizing thelogarithm of the joint likelihood (2). We maximize it by using a gradient-based
numerical optimization method such as the quasi-Newton method [32]. The
gradient of the joint log likelihood is calculated by∂ log p(X, S|W , a, b, r, γ)
∂Wd
J
J

X
X
a0 X
>
Ndj Wd µj µ>
−
x
µ
= −Wd
Ndj Cj − 0
dn j .
jb j=1
n:s =j
j=1

(18)

dn

We can obtain the projection matrices that maximize the joint likelihood analytically as follows,

−1
J
J
0
X
a
a0 X X>W =Ndj Cj + 0 Ndj µj µj
xdn µ>
j .
0
b
b
j=1
j=1 n:s =j

(19)

dn

In our experiments, we used gradient-based updates since it found better local
225

optimum solutions by updating parameters gradually.
Algorithm 1 shows the procedures for inferring the proposed model basedon the stochastic EM algorithm. Here, T is the number of iterations for thestochastic EM algorithm. For the input, we give the initial number of clusters
J. The cluster assignments S are initialized by randomly selecting an integer

230

from {1, · · · , J}. The projection matrices W can be initialized by Gaussianwith zero mean and small variance. By iterating the E- and M-steps, we canobtain an estimate of the cluster assignments and projection matrices. In all of
the experiments, we used the following hyperparameters: a = b = r = γ = 1.
13

Algorithm 1 Learning procedures for the proposed model.
Input: multiple domain data sets X, initial number of clusters J, hyperparameters a, b, r, γ, iteration number T
Output: cluster assignments S, projection matrices W
1: initialize S and W
2: for t = 1, · · · , T do
3://E-step4:

for d = 1, · · · , D do

5:
6:

for n = 1, · · · , Nd dosample sdn using probability p(j|X, S\dn , W , a, b, r, γ) (11) from j =1, · · · , J + 1

7:

if sdn = J + 1 thenupdate the number of clusters J ← J + 1

8:9:

end if

10:

end for11:

end for12://M-step13:

for d = 1, · · · , D do

14:
15:

update Wd using a numerical optimization method using (18)end for

16: end for

We can use cross-validation to select an appropriate dimensionality for latent
235

space K. H-fold cross-validation is as follows,
1. The elements of observation matrices X are partitioned into H subsets.
2. For each of H experiments, H − 1 subsets are used for inferring the model
while the remaining one is assumed missing. The test likelihood of themissing part is calculated by using the inferred model.

240

3. We select the K value that performed the best in terms of the average
test likelihood over H experiments.
The learning procedure for missing data is described in Section 3.4.
14

When we use a regular EM algorithm instead of the stochastic EM algorithm,
latent vectors are not integrated out, and therefore need to be estimated. We
245

can use Gibbs sampling for inferring projection matrices. However, we learned
them by maximizing the likelihood since priors, which require additional hyperparameters, are not necessary.
3.3. Semi-supervised setting
In some applications, correspondence information on a few pairs of objects
might be available. This correspondence information helps to improve matching
performance. Here, we describe learning procedures for the semi-supervised
setting, where correspondence information between some objects in different
domains is given, based on a stochastic EM algorithm. The task of finding
correspondence in a semi-supervised setting is also called informed sorting [4]or seed alignment [11]. Let i = (d, n) and i0 = (d0 , n0 ) be a pair of objects forwhich one-to-one correspondence is given. Their latent cluster assignments must
be the same because we know that they match. Thus, in the E-step, we sample
the cluster assignments for both of them, i and i0 , simultaneously so that they
have the same cluster assignments. New values for the cluster assignments, si
and si0 , are sampled from the following probability,p(si = si0 = j|X, S\ii0 , W , a, b, r, γ)=p(si = si0 = j, S\ii0 |γ) p(X|si = si0 = j, S\ii0 , W , a, b, r)·p(S\ii0 |γ)p(X\ii0 |S\ii0 , W , a, b, r)(20)

where \ii0 represents a value or set excluding objects i and i0 . The first factorof the left-hand side is calculated byN
0 (N·j\ii0 +1)p(si = si0 = j, S\ii0 |γ)  (N·j\ii−2+γ)(N −1+γ)=γ
p(S\ii0 |γ)(N −2+γ)(N −1+γ)

for an existing cluster, j ∈ {1, · · · , J}
for a new cluster, j = J + 1,
(21)

15

using (3), and the second factor is given byp(X|si = si0 = j, S\ii0 , W , a, b, r)p(X\ii0 |S\ii0 , W , a, b, r)
0a0= (2π)

M +M 0− d2 dr

1
2 I(j>J\ii0 )0b\ii\ii0Γ(a0sii0 =j ) |Cj,sii0 =j | 20a0s

Γ(a0\ii0 )=j0bsii0ii=j1

1|Cj\ii0 | 2

,

(22)

using (3) and (4). When we are given correspondence information for more than250

two objects, we can derive a sampling probability with the same manner using(4). For objects without correspondence information, the sampling probabilityof the latent cluster assignment is the same as in the unsupervised setting (11).
The M-step is the same as that in the unsupervised setting, and we can update
projection matrices using gradient-based optimization with (18).

255

3.4. Missing data
Since the proposed model is a probabilistic generative model, it can handle missing data, where some observation features are missing. Let hdn =d(hdnm )M
m=1 be a vector indicating observed indices, where hdnm = 1 if xdnm isobserved, hdnm = 0 otherwise, and Mdn is the number of observed values forobject xdn . The posterior parameters are calculated as follows,
PD PNdMdn0,a = a + d=1 n=12b0 = b +

D Nd
J
1 XX
1 X > −1(hdn ◦ xdn )> (hdn ◦ xdn ) −µ C µj ,2
2 j=1 j j
n=1

(23)

(24)

d=1

µj = Cj

D
X

D
X
d=1

Wd>

hdn ◦ xdn ,(25)

diag(hdn ) Wd + rI,(26)

n:sdn =jd=1

Cj−1 =

X

Wd>

X
n:sdn =jwhere ◦ represents the Hadamard product, or element-wise product, and diag(hdn )returns a diagonal matrix whose diagonal elements are hdn1 , . . . , hdnMd . For
16

learning with missing data, we use (23) – (26) instead of (5) – (8) for calculating the sampling probability (11) in the E-step. In the M-step, the following
gradient is used for optimizing projection matrices,∂ log p(X, S|W , a, b, r, γ)
∂Wd
J
J

X
X
a0 X
>
= −Wd
Ndj Wd µj µ>
−
(h
◦
x
)µ
Ndj Cj − 0
dn
dn
j
j ,
b j=1
n:s =j
j=1

(27)

dn

instead of (18).

4. Experiments
4.1. Matchin rotated handwritten digitsFirst, we demonstrate the proposed model described in Section 3 using a toy
260

data set with three domains, which is created using handwritten digits from the
MNIST database [33]. The first domain contains original handwritten digits,
where each image is downsampled to 16×16 pixels. We synthesize objects for thesecond and third domains by rotating handwritten digits by 90 and 180 degrees,clockwise, respectively. Thus, we obtain three-domain objects that share a latent

265

space. The number of objects in each domain is 200 for all domains. We would
like to match the rotated digits in different domains without information aboutrotation or correspondence.
Figure 4 shows some examples of the clusters discovered by the proposedmodel with K = 5. The proposed model successfully clustered objects, the270

same digit with rotation. In each domain, similar objects are clustered. Acrossdifferent domains, dissimilar objects can be clustered. For example, objects incluster ‘6’ in the first domain are more dissimilar to those in the third domain
than objects in cluster ‘9’ in the first domain. By using domain-dependent projection matrices, the proposed model finds clusters of objects across different

275

domains that have similar latent features and that might have different observation features. When we use standard clustering methods such as k-means andGaussian mixtures for the data that are constructed by combining data in all
domains, ‘6’ is never clustered with ‘6’ with 180 degree rotation.
17

Cluster ‘1’

Cluster ‘4’

Cluster ‘6’

Cluster ‘7’

Cluster ‘9’(a) Domain 1(b) Domain 2(c) Domain 3

Figure 4: Examples of clusters discovered by the proposed model with rotated handwritten
data sets with three domains. Each row represents a cluster, and three objects are shown foreach domain.

We can map the observation features to different domains through the latent
280

space by using the inferred projection matrices. Observation feature vector x indomain d can be mapped into the latent space by ẑ = (Wd> Wd )−1 Wd> x, where
we used the pseudo-inverse of projection matrix Wd . The latent vector z canbe mapped into domain d0 by x̂d0 = Wd0 ẑ. Thus, the projection matrix thatmaps from domain d to domain d0 is Wd0 (Wd> Wd )−1 Wd> . Figure 5 shows the285

images mapped to the three domains by the projection matrices. The original
image comes from the first domain. The mapped images to second and third
domains are rotated images of the first domain by 90 and 180 degrees clockwise,
respectively, which are consistent with the procedure used to synthesize the data
set. This result indicates that we can infer the projection matrices properly,18original image

mapped images

Domain 1

Domain 2

Domain 3

Figure 5: Mapping images to different domains. The left figure shows the original image, andthe right figures show images mapped to the first, second and third domains.

290

and that the proposed model can map to different domains in an unsupervised
fashion by using the shared latent space.
4.2. Data
Next, we evaluated the proposed model quantitatively by using three synthetic and five real-world data sets. The statistics of the seven data sets are295

shown in Table 2. There are two domains for all data sets. Synth3, Synth5, andSynth10 are synthetic data sets with different true dimensionalities of the latent
space K ∗ = 3, 5 and 10, respectively. We generated the synthetic data sets using
the following procedure. First, we sampled latent vectors zj for j = 1, · · · , J ∗from a K ∗ -dimensional normal distribution with mean 0 and covariance I. Next,300

we generated projection matrices Wd for d = 1, 2, where each element is drawn
from a normal distribution with mean 0 and variance 1. Finally, we generated
N/J ∗ objects for each cluster j using a normal distribution with mean Wd zj
and covariance α−1 I, and obtained N objects in total for each domain d = 1, 2.
Iris, Glass, Wine and MNIST, the real-world data sets, were obtained from305

LIBSVM multi-class data sets [34], and generated objects in two domains byrandomly splitting the features into two parts for each data set as [4, 13] did fortheir experiments. Because there is no overlapping feature, we cannot calculate
similarities between objects in different domains. Iris, Glass, Wine and MNIST
were artificially splitted into two domains. On the other hand, Wiki data was a310

real-world two domain data set, which consisted of Wikipedia documents written in English and German. For each language, we sampled 150 documents
19

Table 2: Statistics of the data sets: the number of objects N , the dimensionality of the objects
Md , and the true number of clusters J ∗ , and the true dimensionality of latent space K ∗ .

N1 /N2

M1

M2

J∗

K∗

Synth3

200

50

50

5

3

Synth5

200

50

50

5

5

Synth10

200

50

50

5

10

Iris

150

2

2

3

N/A

Glass

214

4

5

7

N/A

Wine

178

6

7

3

N/A

MNIST

200

392

392

10

N/A

Wiki

150

1000

1000

5

N/A

that were categorized in ‘Nobel laureates in Physics’, ‘Nobel laureates in Chemistry’, ‘American basketball players’, ‘American composers’, and ‘English footballers’. We selected these categories since since they contained enough number
315

of documents, they were related to each other (‘Nobel laureates in Physics’ and‘Nobel laureates in Chemistry’, ‘American basketball players’ and ‘Americancomposers’, ‘American basketball players’ and ‘English footballers’), and their
topics seemed to be interpretable. We used 1,000 most frequently occurringwords as features for each language to reduce computational time, where the320

frequencies were computed by all documents that were categorized in the fivecategories. With the real-world data sets, we used class labels as true clusters.
4.3. Comparing methods
For the proposed method, we used the dimensionality of the latent space,
K = 5, for all data sets. To alleviate the local optimum problem, we ran the325

learning procedure five times with different initial conditions, and selected theresult that achieved the highest likelihood. The number of iterations was 100.
For comparison, we used k-means (KM), convex kernelized sorting (CKS) [13],and their combinations (KM-CKS and CKS-KM) as described below. The KM
method is widely used for clustering. Although KM is not a many-to-many
20

330

matching method, we included KM as a baseline method to evaluate the casewhen only objects in the same domain are clustered. The CKS method is anunsupervised object matching method. It directly finds correspondence between
objects, and does not cluster objects in the same domain. We used the CKS codeprovided by the authors1 , where we used their default setting and the maximum

335

number of iterations was 10,000. With KM-CKS, first we discovered clusters byapplying KM to each domain individually, and then found the correspondence
between clusters in two domains by using CKS. We used the mean vector of eachcluster as the input for cluster matching by CKS. With CKS-KM, after matching
objects using CKS, we combined matched objects in two domains into a vector,

340

and estimated clusters using KM. We employed CKS for comparison since it
achieves higher performance than kernelized sorting and matching canonicalcorrelation analysis [13, 11]. With KM, KM-CKS and CKS-KM, we used thenumber of clusters estimated by the proposed model. For comparison with
object matching based methods (CKS, CKS-KM), we used data sets that had345

the same numbers of objects in the two domains N1 = N2 for each data set.
4.4. Evaluation measurment
For the evaluation measurement, we used the adjusted Rand index [35],which quantifies the similarity between inferred clusters and true clusters. It
takes a value from −1 to 1; 0 represents random clustering. A higher value

350

indicates better clustering performance. The adjusted Rand index becomeshigh when object pairs that belong to one true cluster are assigned to one
inferred cluster simultaneously, and when object pairs that belong to different
true clusters are assigned to different inferred clusters. Here, we use object pairs
across all domains including the same domain. Even if no objects are correctly

355

matched across different domains, the adjusted Rand index can be positive ifclusters are correctly found within each domain. If objects are matched, they
are considered assigned to the same cluster. Therefore, we can calculate the1 http://astro.temple.edu/~tua95067/CKS_code.zip

21

Table 3: Average adjusted Rand index and its standard deviation. Values in bold typeface
are significantly better from those in normal typeface as indicated by a paired t-test.

Proposed

KM

KM-CKSCKSCKS-KM

Synth30.875 ± 0.101 0.525 ± 0.014

0.589 ± 0.117 0.014 ± 0.005

0.699 ± 0.135

Synth50.893 ± 0.126 0.548 ± 0.029

0.583 ± 0.198 0.006 ± 0.007

0.571 ± 0.182

Synth10 0.827 ± 0.145 0.556 ± 0.026

0.553 ± 0.165 0.009 ± 0.006

0.678 ± 0.170

Iris

0.383 ± 0.189 0.224 ± 0.091

0.254 ± 0.154 0.003 ± 0.002

0.207 ± 0.089

Glass

0.160 ± 0.020 0.050 ± 0.008

0.052 ± 0.011 0.001 ± 0.001

0.047 ± 0.010

Wine0.222 ± 0.111 0.125 ± 0.025

0.142 ± 0.046 0.001 ± 0.001

0.107 ± 0.038

MNIST

0.085 ± 0.016 0.030 ± 0.007

0.037 ± 0.008 0.008 ± 0.005

0.041 ± 0.016

Wiki

0.222 ± 0.048 0.152 ± 0.024 0.199 ± 0.062 0.013 ± 0.003 0.207 ± 0.061

adjusted Rand index of CKS although it is likely to be low because all objects
in the same domain are assigned to different clusters. The adjusted Rand index
360

for KM is calculated by assuming that any clusters across different domains do
not match. The adjusted Rand index measures how well objects with the samelabel in different domains are assigned to the same cluster as well as measuring
the clustering performance within each domain. For the real data sets, we
assume that the category label of each object is its true cluster assignment.

365

4.5. Results
Table 3 shows the adjusted Rand index for the seven data sets, which wereaveraged over ten experiments for each data set. For all data sets, the proposedmodel achieved the highest adjusted Rand index. This result indicates that the
proposed model can infer matching clusters by assuming a shared latent space.

370

KM-CKS achieved higher performance than KM by matching clusters in a postprocessing step. With KM-CKS, since clusters are inferred individually for eachdomain, the estimated clusters might be different in different domains. On theother hand, since the proposed model infers clusters in all domains simultaneously, it more successfully found shared clusters than KM-CKS as shown by its

375

higher adjusted Rand index. The adjusted Rand index obtained with the CKS
method was low, because it does not cluster objects. By clustering the result22

of CKS, CKS-KM improved the cluster matching performance. However, it
did not outperform the proposed model, because errors that accumulate in theobject matching process by CKS cannot be corrected in the clustering process
380with k-means. The better performance of the proposed model with fixed hyperparameters a = b = r = γ = 1 for all data sets indicates that the performance
is not sensitive to hyperparameter setting.
The performance with the synthetic data by the proposed model was better
than that with the real-world data since the synthetic data are generated by385

the generative process of the proposed model. Because the generative process,where observations are generated by linear projections of latent vectors, is simple
compared with the real-world data, the performance with the synthetic data by
the baselines was also better than the performance with the real-world data.
With the Wiki data, although the proposed method achieved the best adjusted

390

Rand index, it was not significantly better than KM-CKS and CKS-KM. Thismight be because the word frequency feature does not match with Gaussian
noise which is the assumption of the proposed method. Another reason would
be that linear projections is not appropriate for the Wiki data.
Figure 6 shows the adjusted Rand index for the Synth5 data set achieved by395

the proposed model with different latent dimensionalities. The value was highest
when the latent dimensionality of the model was the same as the true latentdimensionality K = K ∗ = 5. The proposed model with K 6= K ∗ also performed
better than the other methods. This result indicates that the proposed modelis robust to the latent dimensionality setting.

400

Figure 7 shows the adjusted Rand index for the Synth5 data set with different numbers of domains, D. The proposed model can handle data with morethan two domains. For kernelized sorting based methods (KM-CKS, CKS-KM
and CKS), we used one of the domains as a pivot for handling more than two
domains. In particular, with these methods, we found matching across multi-

405

ple domains by matching clusters/objects between the Dth domain and eachof the other D − 1 domains, and then combined the results. In general, theadjusted Rand index decreases as the number of domains increases, since the23

1adjusted Rand index

0.8

0.6

0.4
Proposed
KM−CKSCKS−KM
KM
CKS

0.2

0
1

2

3

4
5
6
7
latent dimensionality: K

8

9

10

Figure 6: Adjusted Rand index achieved by the proposed model with different latent dimensionalities K for the Synth5 data set whose true latent dimensionality is K ∗ = 5.

number of possible combinations of cluster matching increases. However, theproposed method consistently achieved the highest performance regardless of410

domain number.
The proposed model assumes that cluster proportions are shared by different
domains. We evaluated the performance of the proposed model with synthetic
data, in which cluster proportions were different across two domains. Eachclusters contains 50 objects except for cluster1 of domain1, where we changed415

the number of objects from 10 to 100. The number of true clusters was J ∗ = 5,
the latent dimensionality was K ∗ = 5, and the dimensionality of the objects was
M1 = M2 = 50. Figure 8 shows the adjusted Rand index. Note that CKS andCKS-KM are unapplicable since they require that different domains contain thesame number of objects. The proposed model achieved the best performance420

even with data with different cluster proportions. When the number of objects
in cluster1 of domain1 was 50, the adjusted Rand index of the proposed model
was highest, since the common cluster proportions were the same across different
domains with this data set.
The computational time of the proposed model for learning the Synth5 data425

set was 22 seconds with 100 learning iterations when using a computer with24

1

adjusted Rand index

0.8

0.6

0.4

0.2

0
2

Proposed
KM−CKSCKS−KM
KM
CKS
4

6
8
number of domains: D

10

Figure 7: Adjusted Rand index for the Synth5 data set with different numbers of domains,D.

3.20GHz CPU. The computational complexity of each E-step is O(N JK 3 ),where N is the total number of objects, J is the number of clusters, and K
is the latent dimensionality. It increases linearly with the number of objects
N because we need to perform sampling for each object. Cluster assignment
430

is sampled from J + 1 clusters, in which we calculate the probability for eachof the J + 1 clusters. Therefore, the complexity increases linearly with thenumber of clusters J. The complexity of calculating each sampling probability is dominated by the O(K 3 ) computation of inverse Cj−1 . Figure 9 showsthe experimental computation time of the proposed model including E- and M-

435

steps. The computation time linearly increases with the number of objects asshown in Figure 9(a). The dimensionality of the observation and latent spaces,
and the true number of clusters do not strongly impact the computation time
(Figure 9(b,c,d)). Because this experiment used small latent dimensionality, it
might not alter the total computation time. The computation time increases

440

superlinearly with the number of domains (Figure 9(e)). The time increases
because the number of parameters in the projection matrices increases linearlywith the number of domains, and the total number of objects is also increased.

25

1

Proposed
KM−CKS
KM

adjusted Rand index

0.9

0.8

0.7

0.6

0.5

20

40
60
#objects in cluster1

80

100

Figure 8: Adjusted Rand index for the synthetic data with different cluster proportions acrossdifferent domains.

4.6. Semi-supervised setting
Figure 10 shows the average adjusted Rand index achieved by the proposed
445

model with semi-supervised setting over ten experiments, where different objects were labeled for each experiment. Here, we used the seven data sets shownin Table 2. For each data set, we randomly selected pairs of objects to belabeled according to the labeled object rate, which is the horizontal axis of Figure 10, and correspondence information to the different domain for the selected450

objects were attached. The performance increases as the rate of labeled objects
increases. With some data sets (e.g. Synth5, Synth10 and Wine), the additionof just a few labeled objects drastically increases the adjusted Rand index. This
result indicates that these data sets can be appropriately modeled by the linear
projection of latent vectors, and the proposed model can utilize correspondence

455

information for a small number of objects for finding many-to-many matching.
Because the true latent dimensionality of the Synth5 data was the same withthe model used in this experiment, the adjusted Rand index became one withadditional labeled data. With the Synth3 data, the performance was not improved well since the proposed model overfit to the data due to its high latent

460

dimensionality.

26

80

120

100

70

100

80
60

60
40

60
time (sec)50time (sec)time (sec)

80

40
30
20

40
20

20

10
0

0
−20

0
50

100

150

200
250
#objects N

300

350

(a) #objects N

400

−10

20

40
60
80
observed dimension M

100

−20

2

4
6
latent dimension K

810(b) observed dimension M (c) latent dimension K

80

1400

70
1200

60
1000
time (sec)time (sec)

50
40
30

800
600

20
400

10
200

0
−10

2

4
6
#true clusters J*

810(d) #true clusters J ∗

0

1

2

3

4
5
#domains D

6

7

8

(e) #domains D

Figure 9: Computation time in seconds with different settings. The original data set is Synth5,
where the number of objects in each domain Nd = 200, the number of features in a domain
Md = 50, the latent dimensionality K = 5, the true number of clusters J ∗ = 5 and the numberof domains D = 2. Each figure shows the results with data sets when just one of the settings
is changed. The plot shows the average computational time over 100 experiments, and thebar shows the standard deviation.

5. Conclusion
We proposed a generative model approach for finding many-to-many matching based on probabilistic latent variable models. In experiments, we confirmedthat the proposed model can perform much better than conventional methods465

based on object matching, clustering and their combinations. Advantages of the
proposed model over the existing methods are that it can find many-to-many
matching, and can handle multiple domains with different numbers of objects
with no prior knowledge. Because the proposed approach uses probabilistic generative models, we can extend it in a probabilistically principled manner, and470

use it, for example, to handle missing data, integration with other probabilistic
models, and generalization to exponential family distributions.
Although our results have been encouraging as a first step towards unsuper27

1.1

1.1

1.1

1.05

1.05

1.05

0.9
0.85
0.8

1

adjusted Rand index

adjusted Rand index

adjusted Rand index

1
1
0.95

0.95
0.9
0.85
0.8

0.95
0.9
0.85
0.8
0.75
0.7

0.75

0.75

0.7

0.7

0

0.1

0.2
0.3
labeled objects rate

0.4

0.5

0.65
0

(a) Synth3

0.1

0.2
0.3
labeled objects rate

0.50(b) Synth5

0.8

0.1

0.2
0.3
labeled objects rate

0.4

0.5

(c) Synth10

0.21
0.2

0.7

0.19

0.6

adjusted Rand index

adjusted Rand index

0.4

0.5
0.4
0.3

0.18
0.17
0.16
0.15
0.14

0.2
0.1

0.13

0

0.1

0.2
0.3
labeled objects rate

0.4

0.12

0.5

0

(d) Iris

0.1

0.2
0.3
labeled objects rate

0.4

0.5

0.4

0.5

(e) Glass

0.65

0.13

0.6
0.12
adjusted Rand index

adjusted Rand index

0.55
0.5
0.45
0.4
0.35

0.11
0.1
0.09

0.3
0.08
0.25
0.2

0

0.1

0.2
0.3
labeled objects rate

0.4

0.5

(f) Wine

0

0.1

0.2
0.3
labeled objects rate(g) MNIST

Figure 10: Average adjusted Rand index and its standard deviation achieved by the proposedmodel in a semi-supervised setting with different rates of labeled objects.

vised many-to-many object matching, we must extend our approach in a numberof directions. First, we would like to extend the proposed model to other types
475

of data, such as time series and graph data. Second, we can relax the assumption that the observations are linear with respect to their latent vectors by using
nonlinear matrix factorization techniques [36]. Third, we need some techniques
for automatically inferring the latent dimensionality, such as automatic relevance determination [37]. Finally, we would like to use the proposed method

480

for other applications, which include image annotation [1], cross domain recommendation [3], multi-lingual corpus analysis [38, 39], machine translation [6],28and bioinformatics [40].

Appendix A. Derivation of (4)We give the derivation of the likelihood (4), in which latent vectors Z andprecision parameter α are analytically integrated out.
p(X|S, W , a, b, r)
Z Z Y
Nd
D Y
J
Y
=
N (xdn |Wd zsdn , α−1 I)G(α|a, b)
N (zj |0, (ar)−1 I)dZdα
d=1 n=1

=

j=1

Z Z Y
Nd 
D Y
d=1 n=1J Yααr K/2αrα Md /2
k zj k2 )exp(− k xdn − Wd zsdn k2 )
exp(−
2π
2
2π
2
j=1

1 a a−1b αexp(−bα)dZdα
Γ(a)



Z Z  P M N /2  KJ/2
J
X
d d
d
α
ba
αr
α
exp −  (zj − µj )> Cj (zj − µj ) dZ
=
Γ(a)
2π
2π
2 j=1



Nd
D X
J
X
X
1
1
 αa−1 dαx>µ>× exp −α 
dn xdn −j Cj µj + b
2
2
n=1
j=1

×

d=1

= (2π)−= (2π)−PP

d Md N d2

d Md N d
2

J

KJ

r 2

1
ba Y
|Cj | 2
Γ(a) j=1

Z0

exp(−b0 α)αa −1 dα

J

KJ

r 2

1
ba Γ(a0 ) Y|Cj | 2 .
00ab Γ(a) j=1(A.1)

In the third equation, factors related to Z are grouped together. In the fourth
equation, we integrated out Z using


Z
1
1
>
exp − (x − µ) C(x − µ) dx = (2π)M/2 |C| 2 ,
2

(A.2)

which is the normalization constant of M -dimensional Gaussian distribution.
Similarly, α is integrated out by using the following the normalization constantof Gamma distribution in the sixth equationZΓ(a)αa−1 exp(−bα)dα = a .
b

29

(A.3)

Appendix B. Derivation of (9) and (10)We give here the derivation of posteriors of α (9) and zj (10).
p(α|X, S, W , a, b)

JYp(zj |α, X, S, W , r)

j=1

∝ p(X|α, Z, S, W , a, b, r)p(α|a, b)

JYp(zj |α, r)

j=1

=

Nd
D Y
Y

N (xdn |Wd zsdn , α−1 I)G(α|a, b)

d=1 n=1=

×α Md /22πJ Y
αr K/2
j=1

2π

0

J
Y

exp(−

exp(−

∝ αa −1 exp(−b0 α)∝ G(a0 , b0 )N (zj |0, (ar)−1 I)

j=1

Nd 
D Y
Y
d=1 n=1JYα
1 a a−1
k xdn − Wd zsdn k2 )b αexp(−bα)2
Γ(a)αrk zj k2 )
2

J
Y

 α

1
|Cj |− 2 exp − (zj − µj )> Cj (zj − µj )2
j=1

N (µj , α−1 Cj ),

(B.1)

j=1
485

where we used Bayes’ rule.

Appendix C. Derivation of (11)We give here the derivation of the E-step (11) for learning based on a stochastic EM algorithm.
p(sdn = j|X, S\dn , W , a, b, r, γ)=p(sdn = j, S\dn , X|W , a, b, r, γ)p(S\dn , xdn , X\dn |W , a, b, r, γ)=p(sdn = j, S\dn |γ)p(X|sdn = j, S\dn , W , a, b, r, γ)p(S\dn |γ)p(X\dn |S\dn W , a, b, r, γ)p(xdn |W , a, b, r)∝p(sdn = j, S\dn |γ) p(X|sdn = j, S\dn , W , a, b, r)·,p(S\dn |γ)p(X\dn |S\dn , W , a, b, r)(C.1)

where p(xdn |W , a, b, r) does not depend on p(sdn = j|X, S\dn , W , a, b, r, γ).

30References[1] R. Socher, L. Fei-Fei, Connecting modalities: Semi-supervised segmenta490

tion and annotation of images using unaligned text corpora, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
CVPR, 2010, pp. 966–973.
[2] A. Tripathi, A. Klami, S. Virpioja, Bilingual sentence matching using kernel
CCA, in: MLSP ’10: Proceedings of the 2010 IEEE International Workshop

495

on Machine Learning for Signal Processing, 2010, pp. 130–135.
[3] B. Li, Q. Yang, X. Xue, Transfer learning for collaborative filtering via arating-matrix generative model, in: Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, 2009, pp. 617–624.
[4] N. Quadrianto, A. J. Smola, L. Song, T. Tuytelaars, Kernelized sorting,

500

IEEE Trans. on Pattern Analysis and Machine Intelligence 32 (10) (2010)
1809–1821.
[5] M. Yamada, M. Sugiyama, Cross-domain object matching with model selection, in: In Proceedings of Fourteenth International Conference on Articial
Intelligence and Statistics, AISTATS ’11, 2011, pp. 807–815.

505[6] A. Haghighi, P. Liang, T. Berg-Kirkpatrick, D. Klein, Learning bilingual
lexicons from monolingual corpora, in: Proceedings of ACL-08: HLT, 2008,
pp. 771–779.
[7] A. Klami, Variational Bayesian matching, in: Proceedings of Asian Conference on Machine Learning, 2012, pp. 205–220.

510[8] A. Klami, Bayesian object matching, Machine learning 92 (2-3) (2013) 225–
250.
[9] M. E. Tipping, C. M. Bishop, Probabilistic principal component analysis,
Journal of the Royal Statistical Society, Series B 61 (1999) 611–622.

31

[10] B. S. Everitt, An introduction to latent variable models, Chapman and
515

Hall London, 1984.
[11] J. Jagarlamudi, S. Juarez, H. Daumé III, Kernelized sorting for natural language processing, in: AAAI ’10: Proceedings of the 24th AAAI Conferenceon Artificial Intelligence, 2010.
[12] T. Iwata, T. Hirao, N. Ueda, Unsupervised cluster matching via probabilis-

520

tic latent variable models, in: Proceedings of the Twenty-Seventh AAAI
Conference on Artificial Intelligence, AAAI, 2013.
[13] N. Djuric, M. Grbovic, S. Vucetic, Convex kernelized sorting, in: TwentySixth AAAI Conference on Artificial Intelligence, 2012.
[14] A. Tripathi, A. Klami, M. Orešič, S. Kaski, Matching samples of multiple

525

views, Data Min. Knowl. Discov. 23 (2011) 300–321.
[15] C. Rasmussen, The infinite Gaussian mixture model, Advances in neural
information processing systems 12 (5.2) (2000) 2.
[16] C. Wang, S. Mahadevan, Manifold alignment without correspondence, in:Proceedings of the 21st International Joint Conference on Artifical Intelli-530

gence, IJCAI’09, 2009, pp. 1273–1278.
[17] M. Harel, S. Mannor, Learning from multiple outlooks, in: L. Getoor,
T. Scheffer (Eds.), Proceedings of the 28th International Conference on
Machine Learning, ICML ’11, 2011, pp. 401–408.
[18] X. Shi, Q. Liu, W. Fan, P. S. Yu, R. Zhu, Transfer learning on heterogenous

535

feature spaces via spectral transformation, in: Proceedings of the IEEE
International Conference on Data Mining, ICDM ’10, 2010, pp. 1049–1054.
[19] A. Strehl, J. Ghosh, Cluster ensembles—a knowledge reuse framework for
combining multiple partitions, The Journal of Machine Learning Research
3 (2003) 583–617.

32

540

[20] F. R. Bach, M. I. Jordan, A probabilistic interpretation of canonical correlation analysis, Tech. Rep. 688, Department of Statistics, University ofCalifornia, Berkeley (2005).
[21] P. Rai, H. Daumé III, Multi-label prediction via sparse infinite CCA, in:Advances in Neural Information Processing Systems, Vol. 22, 2009, pp.

545

1518–1526.
[22] L. Sun, S. Ji, J. Ye, Canonical correlation analysis for multilabel classification: A least-squares formulation, extensions, and analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 33 (2011) 194–200.
[23] D. R. Hardoon, S. Szedmak, J. Shawe-Taylor, Canonical correlation analy-

550

sis: an overview with application to learning methods, Neural Computation
16 (12) (2004) 2639–2664.
[24] A. Kimura, H. Kameoka, M. Sugiyama, T. Nakano, E. Maeda, H. Sakano,
K. Ishiguro, SemiCCA: Efficient semi-supervised learning of canonical correlations, in: Proceedings of IAPR International Conference on Pattern

555

Recognition, ICPR ’10, 2010, pp. 2933–2936.
[25] M. Tipping, C. Bishop, Mixtures of probabilistic principal component analyzers, Neural Computation 11 (2) (1999) 443–482.
[26] Z. Ghahramani, G. E. Hinton, et al., The EM algorithm for mixtures offactor analyzers, Tech. rep., Technical Report CRG-TR-96-1, University of

560

Toronto (1996).
[27] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, A. Y. Ng, Multimodal deeplearning, in: Proceedings of the 28th International Conference on Machine
Learning, 2011, pp. 689–696.
[28] N. Srivastava, R. Salakhutdinov, Multimodal learning with deep Boltzmann

565

machines, in: Advances in Neural Information Processing Systems, 2012,
pp. 2222–2230.

33

[29] G. Andrew, R. Arora, J. Bilmes, K. Livescu, Deep canonical correlation
analysis, in: Proceedings of the 30th International Conference on Machine
Learning, 2013, pp. 1247–1255.
570[30] J. Sethuraman, A constructive definition of Dirichlet priors, Statistica
Sinica 4 (1994) 639–650.
[31] J. S. Liu, The collapsed Gibbs sampler in bayesian computations with applications to a gene regulation problem, Journal of the American Statistical
Association 89 (427) (1994) 958–966.

575[32] J. Nocedal, Updating quasi-Newton matrices with limited storage, Mathematics of computation 35 (151) (1980) 773–782.
[33] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998)
2278–2324.

580[34] C. Chang, C. Lin, LIBSVM: a library for support vector machines, ACM
Transactions on Intelligent Systems and Technology (TIST) 2 (3) (2011)27.
[35] L. Hubert, P. Arabie, Comparing partitions, Journal of Classification 2 (1)(1985) 193–218.

585[36] N. D. Lawrence, R. Urtasun, Non-linear matrix factorization with Gaussian
processes, in: Proceedings of the 26th Annual International Conference on
Machine Learning, ICML ’09, 2009, pp. 601–608.
[37] D. J. MacKay, Bayesian interpolation, Neural computation 4 (3) (1992)
415–447.

590[38] J. Boyd-Graber, D. Blei, Multilingual topic models for unaligned text, in:Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial
Intelligence, AUAI Press, 2009, pp. 75–82.

34

[39] T. Iwata, D. Mochihashi, H. Sawada, Learning common grammar from multilingual corpus, in: Proceedings of 48th Annual Meeting of the Association
595

for Computational Linguistics, Association for Computational Linguistics,
2010, pp. 184–188.
[40] C. Wang, S. Mahadevan, Manifold alignment using Procrustes analysis, in:
ICML ’08: Proceedings of the 25th International Conference on Machine
Learning, 2008, pp. 1120–1127.

35