Supplemental Materials for ”Infinite Plaid Models for Infinite Bi-clustering”
Katsuhiko Ishiguro, Issei Sato*, Naonori Ueda, Masahiro Nakano, Akisato Kimura
NTT Communication Science Laboratories, Kyoto, Japan*The University of Tokyo*, Tokyo, Japan

Abstract
This supplemental material provides several details andomitted results from the main manuscript.

Inference procedure
In this section, we present our inference procedure for the
proposed Infinite Plaid models.
Here are the overview of the inference procedure development.
1. We focus on the easiness of derivations and implementations, than the computational efficiency and the mixing
speed of the posterior.
2. We do not try marginalization of observation function parameters θ and φ as opposed to the original Bayesian Plaidmodel (Caldas and Kaski 2008).
3. For the inference of hidden variables Z = {Z 1 Z 2 }, wetook the similar approach of (Meeds et al. 2007): simple Gibbs samplers for the existing sub-matrix factors,
and Metropolis-Hastings sampler for accepting the new
sub-matrix proposal. In addition, we used split-merge MH
moves for drastic searches over sub-matrix factors, basedon (Jain and Neal 2004).
4. Given Z, we can compute the exact posteriors for the parameters θ and φ.
5. For hyperparameters, we implemented the hyper-priorbased posterior sampling following (Hoff 2005).

Sampling Z
We rely on a combination of a simple Gibbs sampler andMetropolis-Hastings samplers for sampling new Z. We explain how to sample an instance z1,i,k from its posterior, since
the procedure is completely symmetric for other i, k and Z 2 .
Throughout this section, we use K as the number of currently instantiated factors (sub-matrices). The Beta variables
λ never appear in the inference implementations since we
can marginalize λ out.
Copyright c 2015, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

Sampling for existing sub-matrices For existing factorsk ∈ {1, 2, . . . , K}, we can sample z1,i,k in a standard IBP manner. First, the posterior of z1,i,k for our Gibbs sampler is formulated as follows:
p z1,i,k |Z \(i,k), Z 2 , X, θ, φ ∝ p z1,i,k |Z \(i,k)
11p X(1,i) |z1,i,k , Z \(i,k), Z 2 , θ, φ .
1
(1)
n
o
Z \(i,k)is Z 1 excluding z1,i,k , X(1,i) = xi, j , j = 1, 2, . . . , N2 .
1
Since z1,i,k is either 1 or 0, we want to compute the ratio ofp(z=1)p(z=0) .
The of the first part of the r.h.s. of Eq. (1) is the prior partwhich derives from IBP. Its form is well known (Griffiths
and Ghahramani 2011; Doshi-Velez et al. 2009). The ratio
between z = 0 and z = 1 is:
m1,k − z1,i,k,(2)
N1 − m1,k + z1,i,kP
where m1,k = i I z1,i,k = 1 .
The second part of the r.h.s. of Eq. (1) is the likelihood
part. With straightforward but cumbersome computations,
we have the following equation for the ratio:
 τ n
Y
o
0
z22, j,k θk2 − 2z2, j,k θk yi, j,k ,
(3)
exp −
2
j
where

yi, j,k = xi, j − φ −X

z1,i,l z2, j,l θl .

l\kThen the probability of z1,i,k = 1 is:
Eq. (2) × Eq. (3).
1.0 + Eq. (2) × Eq. (3)MH move for new sub-matrices A problem of our Twoway IBP is that we need to align indices and the numberof instantiated sub-matrices K between two domains Z 1 andZ 2 . Thus we need a special treatment for sampling new factors for z1,i,k as it effects on the cardinality of Z 2 .
Our MH sampling scheme follows (Meeds et al. 2007).
First, we sample a number of new sub-matrices to be added
to the model following the standard IBP definition:
α 
1knew ∼ Poisson.
KWe denote the new factors by l = 1, 2, . . . , knew . Now the proposal of z1,i has additional assignments on new sub-matrices:z∗1,i,l = 1. At the same time, we draw knew times for new θfrom its prior:θl∗ ∼ Normal µθ , (τθ )−1 .
Different from (Meeds et al. 2007), the Two-way IBP requires additional draws of z2, j,l for all j in the second domain. We extend Z 2 by N2 × knew empty entries, and put
assignments by the following prior:!
α2∗z2, j,l ∼ Bernoulli.
α2 + N2
Finally we accept the proposal by the following acceptance rate (Meeds et al. 2007): p X|Z 1 , Z 2 , z∗1,i,l , z∗2, j,l , θl∗ , knew , θ, φ, τ0 min 1,p (X|Z 1 , Z 2 , θ, φ, τ0 )Split-merge MH moves of sub-matrices To speed up
mixing of Z, we also implement the split-merge MH sampler for sub-matrices. Its implementation is straightforward:
we follow the idea of (Meeds et al. 2007) that rely on thepaper about MH for Dirichlet Process (Jain and Neal 2004).

Experiment details
Likelihoods as a evaluation measure
We first considered the test data log likelihood as a primal quantitative measure without ground-truth information.
Given the full observation matrix, we randomly keep a small
portion (10%) of the matrix entries as test data, and compute the likelihoods. However, in preliminary experiments
we found the log likelihood does not effectively present the
goodness of the extracted sub-matrices computed by NMI(based on the ground truth). One possible reason is thatthe likelihood is computed over all matrix entries including
the majority ”non-interesting” ones that are not proactively
modeled. Therefore we do not adopt the test data log likelihood for evaluation.
Fig. 1 presents the test data log likelihoods on synthetic
data sets. There was not so much differences in likelihoods
between two models. We found the same impressions on
real-world data sets.

Inference
All the latent variables of the Bayesian Plaid model are inferred by collapsed Gibbs samplers, similar to the proposed
Infinite Plaid model. For hyperparameters, we adopted two
strategies for the both models: (i) no updates or (ii) infer
them simultaneously by hyper-prior sampling.

Sampling θ and φSynthetic data experiments

Given the current Gibbs samples of the hidden assignment
Data The first data (synth1) has K = 3 non-overlapping subvariables Z 1 and Z 2 , sampling of remaining parameters θmatrices. All parameters for the second data (synth2) is theand φ are straightforward. Posteriors of θk and φ are comsame with the first one, excepting slight overlaps between
puted as follows:
sub-matrices. All θk for synth1 and synth2 are set to the same!
−1τθ µθ + Mk τ0 ȳk  θvalue. The third and fourth (synth3, synth4) datasets have, τ + Mk τ0,p θk |X, Z, θ\k , φ = Normal
K = 4 overlapping sub-matrices. Instead of larger complexiτθ + Mk τ0−1 ! ties (K), sub-matrices in the synth3 and synth4 have different
τφ µφ + N1 N2 τ0 ȳφ  φ
p (φ|X, Z, θ, φ) = Normal, τ + N1 N2 τ0.θk s from each other.
τφ + N1 N2 τ0
Initialization. Before conducting inference, we first initialize values of Z 1 , Z 2 , θ, and φ for both models. For the
In the above equations,
Bayesian Plaid models, we choose a specific K and initial 
1 X
ize the model parameters according to the generative model.
ȳk =I z1,i,k = 1 I z2, j,k = 1 yi, j,k ,For the Infinite Plaid models, we may use the original generMk i, j
ative model. However, in this paper, we want to show robustX
1 X
ness of the Infinite Plaid models under incorrect assumptionsȳφ =yi, j,φ , yi, j,φ = xi, j −z1,i,k z2, j,k θk ,N1 N2 i, jof K. Thus, we dare to initialize the values of Z of InfinitekXXPlaid models with the Bayes Plaid models with fixed K. WeMk =I z1,i,k = 1 I z2, j,k = 1 .
expect Infinite Plaid models can find a better K through ini
jference, while K-fixing Bayesian Plaid models suffer fromincorrect K. After inferences, we evaluated two models usSampling hyperparameters
ing the quantitative measures discussed earlier.
Hyperparameters. Two models have the four sets of hyWe can further infer the hyperparameter values of the Infiperparameters. First one is the hyperparameter set for λ (Z)nite Plaid models. There are a few ways to estimate hyperpaprior: aλ1,2 , bλ1,2 for Bayesian Plaid models, and α1,2 for Inrameters. One is to directly optimize hyperparameters (possibly by numerical gradients) to maximize the marginalized
finite Plaid Models. Second is the hyperparameter set forlog likelihoods. Another is to perform posterior samplings
θ prior: µθ , τθ . Third is the hyperparameter set for φ prior:by assuming ”flat” hyper priors on those hyperparameters.
µφ , τφ . And finally we have τ0 for the observation function.
To focus on the impact of the initial sub-matrix numbers,
In our implementations, we took the second approach. We
we initialized all the hyperparameters of the models to therefer (Hoff 2005) for an efficient and easy implementations
true values i.e. hyperparameters used in synthesizing theof hyper prior-based estimations.

Bayes Plaid

Ave. Test data Log Likelihood

Ave. Test data Log Likelihood
0.0Synth 1 (true K = 3)

0.0

-0.5

-2.0
-2.5
K=3 K=10 K=20 K=3 K=10 K=20
H.I. H.I. H.I.

-2.5

-1.5

-2.0

-2.0

-2.0

-2.5
K=3 K=10 K=20 K=3 K=10 K=20
H.I. H.I. H.I.

-3.0

Synth 4 (true K = 4)

-1.0

-1.5

-1.5

0.0
-0.5

-1.0

-1.0

-1.5

0.0

Infinite Plaid

Synth 3 (true K = 4)

-0.5

-0.5

-1.0

-3.0

Synth 2 (true K = 3)

K=4 K=10 K=20 K=4 K=10 K=20
H.I. H.I. H.I.

-2.5

K=4 K=10 K=20 K=4 K=10 K=20
H.I. H.I. H.I.

Figure 1: Averaged test data log likelihoods on synth4 data set. All log likelihoods are averaged over Gibbs iterations to obtain
posterior averages. H.I. indicates that the hyperparameter inference is employed. Averages and standard deviations of 20 runs
are presented.

Table 1: The averages and the standard deviations of the inferred final K by the Infinite Plaid models.
Synth 1 (K = 3)
Synth 2 (K true = 3)
Synth 3 (K true = 4)
Synth 4 (K true = 4)
true

K init = K true2.86 ± 0.25
2.87 ± 0.44
4.32 ± 0.39
4.00 ± 0.25

K init = 55.23 ± 2.65
4.16 ± 2.08
4.22 ± 0.42
4.77 ± 1.03

K init = 10
7.45 ± 4.63
7.56 ± 4.11
4.53 ± 0.85
4.96 ± 1.36

data (excepting α for Infinite Plaid models). These hyperparameters are either (i) fixed during inferences or (ii) inferred
simultaneously. In the manuscript, we only presented the results of (ii): all hyperparameters are inferred simultaneously.
Number of bi-clusters found. Table 1 presents the averages and the standard deviations of the final K inferred by
the Infinite Plaid models on synthetic data sets. The numbers
are in good accordance with the NMI measures. In general,
the inference of K naturally becomes harder with the larger
K init . K init = 10 for Synth1, 2 seems very difficult, judging
from the averages. Larger standard deviations imply that theinference are trapped with local solutions with a variety of
Ks.

Real-world data experiments
Pre-processing. For Enron datasets, we scaled the E-mailcount values by log(1+ x) where x is the actual E-mail count,
to fit the Normal distribution. For Lastfm dataset, we remove
all artists who have been tagged at most one word, and remove all tags that have been attached at most to one artist
because we are interested in sub-matrices that group multiple objects.
Hyperparameters. It is fundamentally difficult to determine the best hyperparameters without the ground-truth information. In this experiment, we heuristically determined
the initial values of hyperparameters. Collected real-world
datasets are very sparse, which means that most of matrix
entries are zero-valued. Thus we set the hyperparameters for
φ (µφ , τφ ) so that the distribution of φ concentrates near zero.
We compute the averages and variance of non-zero entries,
and use them as the hyperparameters for θ (µθ , τθ ). For τ0 ,we use the inverse of variances of the whole matrix data.
These hyperparameters are either (i) fixed during inferencesor (ii) inferred simultaneously. In the manuscript, we only

presented the results of (ii): all hyperparameters are inferred
simultaneously.
Result on Enron Aug. Data. Fig. 2 presents an example
of sub-matrices from Enron Aug. data. The k = 6th submatrix (purple colored) is a cluster of VIP members. Notethat the receivers consist of several Presidents, legal persons and the risk management head. We may imagine thatsome fatal problems are reported to these important persons to settle them. Another interesting sub-matrix is thek = 3rd (green colored) sub-matrix. This sub-matrix contains only one sender (1st domain object), who is the founderof Enron. He sends e-mails to many employees including many VIPs of group companies. (Ishiguro et al. 2010;
Ishiguro, Ueda, and Sawada 2012) also found a similar partition, concluding that this specific relations suit well to thefact that ”the founder actually made an announcement tocalm down the public” ((Ishiguro et al. 2010)) concerning
the resign of the Enron CEO at that time.
Result on Enron Nov. Data. Fig. 3 presents an example
of sub-matrix extraction from Enron Nov. data. The k = 1st
sub-matrix (red colored) sub-matrix is a VIP + legal expert
community similar to the 8th sub-matrix at Enron Oct. data.
Interestingly, there are a more bi-directional connections between executives than other months, implying more frequentcontacts among VIPs at one month before the bankruptcy.
The k = 2nd sub-matrix (orange colored) represents a small
and tightly connected community. Unfortunately, we cannot
examine details of the membership because included objects
lacks demographic information, but we point out that thesame community (with same members) appeared in all four
datasets. One particular sub-matrix that was never found inother dataset is the k = 8th sub-matrix (sky-blue colored).
This is a small community of traders and cash analysts.

Demonstration code sources
A Matlab implementation of the Baysian Plaid models andthe Infinite Plaid models with simpler observation models
is published in the GitHub: https://github.com/k-ishiguro/
InfinitePlaidModels

References
Caldas, J., and Kaski, S. 2008. Bayesian Biclusteringwith the Plaid Model. In Proceedings of the IEEE Interna-

All assignments Z

k=6: “VIP cluster”
Domain 1 objects
CEO
Employee, Government Relation Executive
Vice President & Chief of Staff
Vice President, Government Affairs......
Domain 2 objects
President, Enron online
Founder of Enron, CEO
President, Enron Gas Pipeline
Employee, Government Relation Executive
Vice President, Regulatory Affairs
Chief Risk Officer
In House Lawyer
Managing Director, Legal Department......

Magnified

k=3: “founder”
Domain 1 objects
Founder of Enron, CEO(only one object)
Domain 2 objects
Exhaustive bi-clustering
President, Enron online
President, Enron Gas Pipeline
also found similar sub-matrices.
Employee, COO
Vice President, Regulatory Affairs
Vice President, Chief Financial Officer and Treasurer(Many other VPs and employees)

Figure 2: Results on Enron Aug. data. Visualizations and descriptions of some sub-matrices.

Found in all months.

All assignments Z

k=2: “tight community”(no demographic info.)
k=1: “VIP w/ legal expert”
Domain 1 objects
President, Enron online
Employee, Government Relation Executive
Vice President, Regulatory Affairs
Vice President & Chief of Staff
Vice President, Government Affairs(one object w/o demographic info.)
Domain 2 objects
President, Enron online
Employee, Government Relation Executive
Vice President, Regulatory Affairs
Vice President & Chief of Staff
Vice President, Government Affairs
President
Founder of Enron, CEOIn House Lawyer
President, Enron Global Markets
CEO, Enron North America and Enron Energy Services......

One month before bankruptcy,
more frequent & bidirectional
contacts between VIPs.

Magnified

k=8: “Monetary cluster”
Domain-common objects
Employee, Senior Analyst Cash
Trader
Domain 2 specific objects
Director
Employee
Employee, Cash Analysis(one object w/o demographic info.)

Only found at Nov.

Figure 3: Results on Enron Nov. data. Visualizations and descriptions of some sub-matrices.

tional Workshop on Machine Leaning for Signal Proceesing(MLSP).
Doshi-Velez, F.; Miller, K. T.; Van Gael, J.; and Teh, Y. W.
2009. Variational Inference for the Indian Buffet Process. In
Proceeding of the 12th International Conference on Artificial Intelligence and Statistics (AISTATS).
Griffiths, T. L., and Ghahramani, Z. 2011. The Indian Buffet
Process : An Introduction and Review. Journal of Machine
Learning Research 12:1185–1224.
Hoff, P. D. 2005. Subset clustering of binary sequences,with an application to genomic abnormality data. Biometrics
61(4):1027–1036.
Ishiguro, K.; Iwata, T.; Ueda, N.; and Tenenbaum, J. 2010.
Dynamic Infinite Relational Model for Time-varying Relational Data Analysis. In Lafferty, J.; Williams, C. K. I.;
Shawe-Taylor, J.; Zemel, R. S.; and Culotta, A., eds., Advances in Neural Information Processing Systems 23 (Proceedings of NIPS).
Ishiguro, K.; Ueda, N.; and Sawada, H. 2012. Subset Infinite Relational Models. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics(AISTATS), volume XX, 547–555.
Jain, S., and Neal, R. M. 2004. A Split-Merge Markov Chain
Monte Carlo Procedure for the Dirichlet Process Mixture
Model. Journal of Computational and Graphical Statistics
13:158–182.
Meeds, E. W.; Ghahramani, Z.; Neal, R. M.; and Roweis, S.
2007. Modeling Dyadic Data with Binary Latent Factors.
In Advances in Neural Information Processing Systems 19(NIPS), 977–984.