Learning Discrete
Representations via
Information Maximizing
Self-Augmented Training
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, Masashi Sugiyama
University of Tokyo

Preferred Networks

Preferred Networks

Preferred Networks

RIKEN

RIKEN

ATR

University of Tokyo

University of Tokyo

University of Tokyo*Based on the work
performed at
Preferred Networks

Longer version is toappear in ICML 20171

Unsupervised Discrete Representation
Learning
Unlabeled data2

Unsupervised Discrete Representation
Learning
Discrete representations
Unlabeled data

Learn tomap

3Unsupervised Discrete Representation
Learning
Discrete representationsUnlabeled data

Learn tomap

Clustering
Map to cluster assignments
0, 1, 5, 8, 9, 1, 3, 2, 4, 3,9, 3, 2, 0, 2, 1, 4, 3, 1, 3

4

Unsupervised Discrete Representation
Learning
Discrete representations
Unlabeled data

Learn tomap

Clustering
Map to cluster assignments
0, 1, 5, 8, 9, 1, 3, 2, 4, 3,9, 3, 2, 0, 2, 1, 4, 3, 1, 3

Hash Learning
Map to binary codes
0001, 0101, 1110, 1111,0000, 0111, 0000, 1011

5

Deep Neural Networks (DNN) are
Promising
Discrete representations
Unlabeled data

DNN

Clustering
Map to cluster assignments
00
0111
11
1

Flexible
Scalable0, 1, 5, 8, 9, 1, 3, 2, 4, 3,9, 3, 2, 0, 2, 1, 4, 3, 1, 3

Hash Learning
Map to binary codes
0001, 0101, 1110, 1111,0000, 0111, 0000, 1011

6

Deep Neural Networks (DNN) are
Promising
Discrete representations
Unlabeled data

DNN

Clustering
Map to cluster assignments
00
011
1
111

Sensitive to
input perturbation

0, 1, 5, 8, 9, 1, 3, 2, 4, 3,9, 3, 2, 0, 2, 1, 4, 3, 1, 3Hash Learning
Map to binary codes
0001, 0101, 1110, 1111,0000, 0111, 0000, 1011

7

Illustration of Overfitting
Y 2 {0, . . . , K

X2X

1}

p✓ (y|x)8

Illustration of Overfitting
Mutual Information
X2X

I(X; Y )Y 2 {0, . . . , K

1}p✓ (y|x)

Information Maximization clustering[Bridle et al., 1991]

9Illustration of Over-fitting
X2X

Mutual Information

I(X; Y )p✓ (y|x)Y 2 {0, . . . , K

1}

013
2

Information Maximization clustering[Bridle et al., 1991]10

Illustration of Over-fitting
X2XMutual Information

I(X; Y )Y 2 {0, . . . , K

1}0p✓ (y|x)
Weight-decay

132

Regularized Information Maximization (RIM)[Gomes et al., 2010]11

Our Contributions
Better regularization

Our method

IMSAT

RIM [Gomes et al., 2010]
Weight-decay, Clustering

More general InfoMax
12

Our Method: IMSAT Y 2 {0, . . . , K
X2X

Mutual Information

I(X; Y )1}orY = (Y1 , . . . , YD )00p✓ (y|x)1001

SAT regularization

11
Information Maximizing
Self-Augmented Training
13

State-of-the-art Performance
State-of-the-art in• Clustering• Hash learning

Clustering accuracy (%)

98.4%
84.3%

MNIST

58.5% 59.6%

Our[Xie et al. RIM with deep and linear classifier
method 2016][Gomes et al., 2010]14

Outline
1. Introduction
2. Proposed Method: IMSAT = IM + SAT• Information Maximization (IM)• Self-Augmented Training (SAT)
3. Experiments
4. Conclusions & Future Work15

Outline
1. Introduction
2. Proposed Method: IMSAT = IM + SAT• Information Maximization (IM)• Self-Augmented Training (SAT)
3. Experiments
4. Conclusions & Future Work16

InformationMaximization
Better regularization
IMSAT

RIM [Gomes et al., 2010]
More general InfoMax
17

Information Maximization
Learn cluster assignment probability p✓ (y|x) :y 2 {0, . . . , K
max I(X; Y ) [Bridle et al., 1991,✓1}

Gomes et al.,2010]

Learn discrete representations probability p✓ (y1 , . . . , yD |x) ?

18

Information Maximization
Learn cluster assignment probability p✓ (y|x) :y 2 {0, . . . , K
max I(X; Y ) [Bridle et al., 1991,✓1}

Gomes et al.,2010]

Learn discrete representations probability p✓ (y1 , . . . , yD |x) :max I(X; Y1 , . . . , YD )✓19

Information Maximization
Learn cluster assignment probability p✓ (y|x) :y 2 {0, . . . , K
max I(X; Y ) [Bridle et al., 1991,✓1}

Gomes et al.,2010]

Learn discrete representations probability p✓ (y1 , . . . , yD |x) :max I(X; Y1 , . . . , YD )✓• Challenge: Combinatorial summation
à We need approximation!

XX
y1

y2···

XyD

20

Information Maximizationy = (y1 , y2 , · · · , yD )00
0111111p✓ (y|x)
Approximate up to second order interaction:[Brown 2009]
I(X; Y1 , . . . , YD ) ⇡DX

I(X; Yd )d=1Maximize
information

XI(Yd ; Yd0 )1d6=d0 D

Reduceredundancyx21

Outline
1. Introduction
2. Proposed Method: IMSAT = IM + SAT• Information Maximization (IM)• Self-Augmented Training (SAT)
3. Experiments
4. Conclusions & Future Work22

Self-Augmented Training(SAT)
Better regularization
IMSAT

RIM [Gomes et al., 2010]
More general InfoMax23

Similarization

Self-Augmented Training (SAT)
Augmentation function T (·) : X ! XEx.)

Data augmentation

x

T (x)
24

Self-Augmented Training (SAT)p✓ (y|T (x))

p✓ (y|x)
Similarizationy = (y1 , y2 , . . . , yD )KL[p✓ (y|x)kp✓ (y|T (x))][Bachman et al., 2014;
Miyato et al., 2016]

Data augmentation

x

T (x)
25

Self-Augmented Training (SAT)
Related work:
DNN regularization that imposes invariance.
ContinuousBachman et al., 2014;
Miyato et al., 2016;
Sajjadi et al., 2016

Supervised/Semi-supervised
Unsupervised

Discrete
Multi-dimOne-dim

Discriminativefeature learning[Dosovitskiy et al.
2014]

Self-Augmented Training

26

Self-Augmented Training (SAT)
Related work:
DNN regularization that imposes invariance.
ContinuousBachman et al., 2014;
Miyato et al., 2016;
Sajjadi et al., 2016

Supervised/Semi-supervised
Unsupervised

Discrete
Multi-dimOne-dim

Discriminative
feature learning[Dosovitskiy et al.
2014]

Borrow idea
Self-Augmented Training

27

Self-Augmented Training (SAT)
Decision boundary• Local perturbation

T (x) = x + r, krk2 = ✏• Random Perturbation Training (RPT)[Bachman et al., 2014]• Virtual Adversarial Training (VAT)[Miyato et al., 2016]p✓ (y|x + r)rp✓ (y|x)x28

Outline
1. Introduction
2. Proposed Method: IMSAT = IM + SAT• Information Maximization (IM)• Self-Augmented Training (SAT)
3. Experiments
4. Conclusions & Future Work

29

IMSAT = Information Maximizing + SAT
00
011
1

11
1

Discrete representations
Similarization

Information
Maximization

Data augmentation30

Outline
1. Introduction
2. Proposed Method: IMSAT• Information Maximization (IM)• Self-Augmented Training (SAT)
3. Experiments
4. Conclusions & Future Work

31Experiments (Clustering)• Measure clustering accuracy• Batch normalization• ReLU activation• Softmax output#(Cluster size)p✓ (y|x)1200
1200※ Implementation available online
https://github.com/weihua916/imsatx#(input dimension)
32

STL (Coates et al., 2010)
CIFAR10 (Torralba et al., 2008)
CIFAR100 (Torralba et al., 2008)
SVHN (Netzer et al., 2011)
Reuters (Lewis et al., 2004)
20news (Lang, 1995)

13000
60000
60000
99289
10000
18040

1010100104202048
2048
2048
960
2000
2000

Experiments (Clustering)

10%
10%
1%
19%
43%
5%

Table 3. Comparison of clustering accuracy on eight benchmark datasets (%). Averages and standard deviations over twelve trials werereported. Results marked with † were excerpted from Xie et al. (2016).
MethodMNIST
Omniglot
STL
CIFAR10 CIFAR100
SVHN
Reuters
20newsK-means
53.2
12.0
85.6
34.4
21.5
17.9
54.1
15.5
†
dAE+K-means
79.8
14.1
72.2
44.2
20.8
17.4
67.2
22.1
†DEC [Xie et al.,
84.3
5.7 (0.3)
78.1 (0.1) 46.9 (0.9)
14.3 (0.6)
11.9 (0.4) 67.3 (0.2) 30.8 (1.8)
Linear2014]
RIM
59.6 (2.3) 11.1 (0.2) 73.5 (6.5) 40.3 (2.1)
23.7 (0.8)
20.2 (1.4) 62.8 (7.8) 50.9 (3.1)
Linear IMSAT (VAT)
61.1 (1.9) 12.3 (0.2) 91.7 (0.5) 40.7 (0.6)
23.9 (0.4)
18.2 (1.9) 42.9 (0.8) 43.9 (3.3)
Deep RIM
58.5 (3.5)
5.8 (2.2)
92.5 (2.2) 40.3 (3.5)
13.4 (1.2)
26.8 (3.2) 62.3 (3.9) 25.1 (2.8)
IMSAT (RPT)89.6 (5.4) 16.4 (3.1) 92.8 (2.5) 45.5 (2.9)
24.7 (0.5)
35.9 (4.3) 71.9 (6.5) 24.4 (4.7)
IMSAT (VAT)
98.4 (0.4) 24.0 (0.9) 94.1 (0.4) 45.6 (0.8)
27.5 (0.4)
57.3 (3.9) 71.0 (4.9) 31.1 (1.9)

where l and c are the ground-truth label and cluster• Tested on 8 benchmark datasets.
assignment produced using the algorithm for x , respectively. The m ranges theover alldatasets.
possible one-to-one mappings• Hyper-parameters are fixed throughout

A brief summary of dataset statistics is given in Table 2. In
the experiments, our goal was to discover clusters that correspond well with the ground-truth categories. For the STL,CIFAR10 and CIFAR100 datasets, raw pixels are not suitedfor our goal because color information is dominant. We
therefore applied 50-layer pre-trained deep residual networks (He et al., 2016) to extract features and used themfor clustering. Note that since the residual network wastrained on ImageNet, each class of the STL dataset (whichis a subset of ImageNet) was expected to be well-separatedin the feature space. For Omniglot, 100 types of charac-nnn

between clusters and labels. The best mapping can be efficiently computed using the Hungarian algorithm (Kuhn,1955).
4.2.3. H YPER - PARAMETER SELECTION
In unsupervised learning, it is not straightforward
to de33
termine hyper-parameters by cross-validation. Hence, weSTL (Coates et al., 2010)
CIFAR10 (Torralba et al., 2008)
CIFAR100 (Torralba et al., 2008)
SVHN (Netzer et al., 2011)
Reuters (Lewis et al., 2004)
20news (Lang, 1995)

13000
60000
60000
99289
10000
18040

1010100104202048
2048
2048
960
2000
2000

Experiments (Clustering)

10%
10%
1%
19%43%
5%

Table 3. Comparison of clustering accuracy on eight benchmark datasets (%). Averages and standard deviations over twelve trials werereported. Results marked with † were excerpted from Xie et al. (2016).
MethodMNIST
Omniglot
STL
CIFAR10 CIFAR100
SVHN
Reuters
20newsK-means
53.2
12.0
85.6
34.4
21.5
17.9
54.1
15.5
†
dAE+K-means
79.8
14.1
72.2
44.2
20.8
17.4
67.2
22.1
†
DEC
84.3
5.7 (0.3)
78.1 (0.1) 46.9 (0.9)
14.3 (0.6)
11.9 (0.4) 67.3 (0.2) 30.8 (1.8)
Linear RIM
59.6 (2.3) 11.1 (0.2) 73.5 (6.5) 40.3 (2.1)
23.7 (0.8)
20.2 (1.4) 62.8 (7.8) 50.9 (3.1)
Linear IMSAT (VAT)
61.1 (1.9) 12.3 (0.2) 91.7 (0.5) 40.7 (0.6)
23.9 (0.4)
18.2 (1.9) 42.9 (0.8) 43.9 (3.3)
Deep RIM
58.5 (3.5)
5.8 (2.2)
92.5 (2.2) 40.3 (3.5)
13.4 (1.2)
26.8 (3.2) 62.3 (3.9) 25.1 (2.8)
IMSAT (RPT)89.6 (5.4) 16.4 (3.1) 92.8 (2.5) 45.5 (2.9)
24.7 (0.5)
35.9 (4.3) 71.9 (6.5) 24.4 (4.7)
IMSAT (VAT)
98.4 (0.4) 24.0 (0.9) 94.1 (0.4) 45.6 (0.8)
27.5 (0.4)
57.3 (3.9) 71.0 (4.9) 31.1 (1.9)• Used perturbation as augmentation
function.
where l and c are the ground-truth label and clusterr
assignment produced using the algorithm for x , respec• IMSAT (VAT) achieved state-of-the-art
tively. The m ranges over all possible one-to-one mappings
performance.
between clusters and labels. The best mapping
x can be ef-A brief summary of dataset statistics is given in Table 2. In
the experiments, our goal was to discover clusters that correspond well with the ground-truth categories. For the STL,CIFAR10 and CIFAR100 datasets, raw pixels are not suitedfor our goal because color information is dominant. We
therefore applied 50-layer pre-trained deep residual networks (He et al., 2016) to extract features and used themfor clustering. Note that since the residual network wastrained on ImageNet, each class of the STL dataset (whichis a subset of ImageNet) was expected to be well-separatedin the feature space. For Omniglot, 100 types of charac-nnn

ficiently computed using the Hungarian algorithm (Kuhn,1955).
4.2.3. H YPER - PARAMETER SELECTION

In unsupervised learning, it is not straightforward
to de34
termine hyper-parameters by cross-validation. Hence, weExperiments (Clustering)
Do domain-specific augmentation functions improve theclustering performance?
Omniglot dataset [Lake et al. 2011]35

Experiments (Clustering)• Domain-specific augmentation function
Learning Discrete Representations via Information Maximizing Self Augmented Training
Small stochastic affine transformation

202122232425262728293031323334 Figure 3. Examples of the random affine distortion used in our experiments. Images in the top left side are stochastically transformed
35 using the affine distortion.
3637
36
38

1
1
1
1
1
1
1
1
1
1
1
11
1
1
1
1
1
1

Experiments (Clustering)• Domain-specific augmentation function
Learning Discrete Representations via Information Maximizing Self Augmented Training
Small stochastic affine transformation

2021222324252627282930
31323334 Figure 3. Examples of the random affine distortion used in our experiments. Images in the top left side are stochastically transformed
35 using the affine distortion.
363737381111111111111111111

ete Representations
via Information
Maximizing Self-Augmented
Training
Learning
Discrete Representations
via Information
Maximizing Self-Augmented

VAT)

Experiments (Clustering)

Accuracy improved!
70.0%

24.0%

+
Domain-specific(a) IMSAT (VAT)(b) IMSAT (VAT & aﬃne)knowledge(b) IMSAT (VATof Omniglot
discoveredusing(a) IMSAT(VAT) discoveredand (b) IMSAT& affine).
Eachgure
2. Randomly
sampledclustersof Omniglot
using(VAT(a) IMSAT(VAT)
androw(b)contains
IMSAT (VAT
mecluster.sampled data points in same
(Affinendomlycluster. transformation)
38

Experiments (Hash Learning)p✓ (y1 , . . . , yD |x)• 3 evaluation metrics:#(hash bits)• mean average precision• precision @ sample=500• precision @ hamming dist=2• 16-bit (D = 16)x#(input dimension)39

(a) IMSAT (VAT)

Experiments (Hash Learning)(b) IMSAT (VAT & aﬃne)Figure 2. Randomly sampled clusters of Omniglot discovered using (a) IMSAT (VAT) and (b) IMSAT (VAT & affine). Each row contains
randomly sampled data points in same cluster.
Table 5. Comparison of hash performance for 16-bit hash codes (%). Averages and standard deviations over ten trials were reported.
Experimental results of Deep Hash and the previous methods were excerpted from Erin Liong et al. (2015).

Method(Dimensions of hidden layers)
Spectral hash (Weiss et al., 2009)
PCA-ITQ (Gong et al., 2013)
Deep Hash (60-30)
Linear RIM
Deep RIM (60-30)
Deep RIM (200-200)
Deep RIM (400-400)
IMSAT (VAT) (60-30)
IMSAT (VAT) (200-200)
IMSAT (VAT) (400-400)

Hamming ranking (mAP)MNIST
CIFAR10
26.6
12.6
41.2
15.7
43.1
16.2
35.9 (0.6)
24.0 (3.5)
42.7 (2.8)
15.2 (0.5)
43.7 (3.7)
15.6 (0.6)
43.9 (2.7)
15.4 (0.2)
61.2 (2.5)
19.8 (1.2)
80.7 (2.2)
21.2 (0.8)
83.9 (2.3)
21.4 (0.5)

precision @ sample = 500
MNISTCIFAR10
56.3
18.8
66.4
22.5
67.9
23.8
68.9 (1.1)
15.9 (0.5)
67.9 (2.7)
21.8 (0.9)
68.7 (4.9)
21.6 (1.2)
69.0 (3.2)
21.5 (0.4)
78.6 (2.1)
21.0 (1.8)
95.8 (1.0)
27.3 (1.3)
97.0 (0.8)
27.3 (1.1)

precision @ r = 2
MNIST
CIFAR10
57.5
18.5
65.7
22.6
66.1
23.3
71.3 (0.9) 14.2 (0.3)
65.9 (2.7) 21.2 (0.9)
67.0 (4.9) 21.1 (1.1)
66.7 (3.2) 20.9 (0.3)
76.5 (2.3) 19.3 (1.6)
94.6 (1.4) 26.1 (1.3)
96.2 (1.1) 26.4 (1.0)• Tested on 2 benchmark datasets.
constraint, we report the results for 32-bit hash codes in
5. Conclusion & Future Work
• Hyper-parameters
aretendency
fixedas throughout
the datasets.
AppendixH, but the results showed a similar
In this paper, we presented IMSAT, an informationthat of 16-bit hash codes. We see from Table 5 that IMSAT
with the largest network sizes (400-400) achieved competitive performance in both datasets. The performance of IMSAT improved significantly when slightly bigger networks(200-200) were used, while the performance of Deep RIM

theoretic method for unsupervised discrete representation
learning using deep neural networks. Through extensive
experiments, we showed that intended discrete representa40
tions can be obtained by directly imposing the invariance to(a) IMSAT (VAT)

Experiments (Hash Learning)(b) IMSAT (VAT & aﬃne)Figure 2. Randomly sampled clusters of Omniglot discovered using (a) IMSAT (VAT) and (b) IMSAT (VAT & affine). Each row contains
randomly sampled data points in same cluster.
Table 5. Comparison of hash performance for 16-bit hash codes (%). Averages and standard deviations over ten trials were reported.
Experimental results of Deep Hash and the previous methods were excerpted from Erin Liong et al. (2015).

Method(Dimensions of hidden layers)
Spectral hash (Weiss et al., 2009)
PCA-ITQ (Gong et al., 2013)
Deep Hash (60-30)
Linear RIM
Deep RIM (60-30)
Deep RIM (200-200)
Deep RIM (400-400)
IMSAT (VAT) (60-30)
IMSAT (VAT) (200-200)
IMSAT (VAT) (400-400)

Hamming ranking (mAP)MNIST
CIFAR10
26.6
12.6
41.2
15.7
43.1
16.2
35.9 (0.6)
24.0 (3.5)
42.7 (2.8)
15.2 (0.5)
43.7 (3.7)
15.6 (0.6)
43.9 (2.7)
15.4 (0.2)
61.2 (2.5)
19.8 (1.2)
80.7 (2.2)
21.2 (0.8)
83.9 (2.3)
21.4 (0.5)

precision @ sample = 500
MNISTCIFAR10
56.3
18.8
66.4
22.5
67.9
23.8
68.9 (1.1)
15.9 (0.5)
67.9 (2.7)
21.8 (0.9)
68.7 (4.9)
21.6 (1.2)
69.0 (3.2)
21.5 (0.4)
78.6 (2.1)
21.0 (1.8)
95.8 (1.0)
27.3 (1.3)
97.0 (0.8)
27.3 (1.1)

precision @ r = 2MNISTCIFAR10
57.5
18.5
65.7
22.6
66.1
23.3
71.3 (0.9) 14.2 (0.3)
65.9 (2.7) 21.2 (0.9)
67.0 (4.9) 21.1 (1.1)
66.7 (3.2) 20.9 (0.3)
76.5 (2.3) 19.3 (1.6)
94.6 (1.4) 26.1 (1.3)
96.2 (1.1) 26.4 (1.0)• IMSAT (VAT) outperformed the 5.previous
Conclusionmethods.
& Future Work

constraint, we report the results for 32-bit hash codes inAppendix H, but the results showed a similar tendency asthat of 16-bit hash codes. We see from Table 5 that IMSATwith the largest network sizes (400-400) achieved competitive performance in both datasets. The performance of IMSAT improved significantly when slightly bigger networks(200-200) were used, while the performance of Deep RIM

In this paper, we presented IMSAT, an informationtheoretic method for unsupervised discrete representation
learning using deep neural networks. Through extensiveexperiments, we showed that intended discrete representa41
tions can be obtained by directly imposing the invariance toOutline
1. Introduction
2. Background (Regularized Information Maximization[Gomes et al., 2010])
3. Proposed Method (Information Maximizing SelfAugmented Training)
4. Experiments
5. Conclusions & Future Work

42

Conclusions
Better regularization
SAT regularization

RIM [Gomes et al., 2010]

IMSAT

InfoMax for learning
discrete representations
More general InfoMax
43

Conclusions
Better regularization

IMSAT
State-of-the-art inclustering andunsupervised hash learning

RIM [Gomes et al., 2010]
More general InfoMax
44

000111111

Future Work

Discrete representations
Similarization

Augmentation function T (·) encodes invariance of representations.
à What is effective T (·) for different types of data?
Ex.) image, text, sequence, graph

Data augmentationxT (x)45