1Learning Discrete Representations via
Information Maximizing
Self-Augmented Training
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, Masashi Sugiyama
University of Tokyo

Preferred Networks

Preferred Networks

Preferred Networks

RIKEN

RIKEN

ATR

University of Tokyo

University of Tokyo

University of Tokyo*Based on the work
performed atPreferred Networks

2Big success of supervised Deep Learning
Huge number of Labeled data(Very) deep neural networks

3

Big success of supervised Deep Learning
Huge number of Labeled data(Very) deep neural networks

4

Unsupervised Discrete Representation Learning
Unlabeled data5Unsupervised Discrete Representation Learning
Discrete representations

Unlabeled data

Learn tomap

6

Unsupervised Discrete Representation Learning
Discrete representations

Unlabeled data

Learn tomap

Clustering
Map to cluster assignments
0, 1, 5, 8, 9, 1, 3, 2, 4, 3,9, 3, 2, 0, 2, 1, 4, 3, 1, 3

7

Unsupervised Discrete Representation Learning
Discrete representations

Unlabeled data

Learn tomap

Clustering
Map to cluster assignments
0, 1, 5, 8, 9, 1, 3, 2, 4, 3,9, 3, 2, 0, 2, 1, 4, 3, 1, 3Hash Learning
Map to binary codes
0001, 0101, 1110, 1111,0000, 0111, 0000, 1011

8

Deep Neural Networks (DNN) are Promising
Discrete representations

Unlabeled data

DNNClustering
Map to cluster assignments
00
0111111

Flexible
Scalable0, 1, 5, 8, 9, 1, 3, 2, 4, 3,9, 3, 2, 0, 2, 1, 4, 3, 1, 3

Hash Learning
Map to binary codes
0001, 0101, 1110, 1111,0000, 0111, 0000, 1011

9

Problem setting• Goal: learn deep probabilistic classifier.
p✓ (y|x) Discrete representations
00
01. Cluster assignment
2. Binary codes
111111X

Y

10Problem setting• Goal: learn deep probabilistic classifier.
p✓ (y|x) Discrete representations
00
01. Cluster assignment
2. Binary codes
111111YX

Regularized Information Maximization (RIM) [Gomes+ 2010]

max I(X; Y )✓R(✓) Regularization (weight-decay)

Mutual Information

11Problem setting• Goal: learn deep probabilistic classifier.
p✓ (y|x) Discrete representations
00
01. Cluster assignment
2. Binary codes
111111YX

Regularized Information Maximization (RIM) [Gomes+ 2010]

max I(X; Y )✓R(✓) Regularization (weight-decay)

MutualInformation
Onlyapplicableto clustering.

12

Problem setting• Goal: learn deep probabilistic classifier.
p✓ (y|x) Discrete representations
00
0

1. Cluster assignment
2. Binary codes
111
111

Y

X

Regularized Information Maximization (RIM) [Gomes+ 2010]

max I(X; Y )✓R(✓) Regularization
Weight-decay
is restrictive.
(weight-decay)

MutualInformation
Onlyapplicableto clustering.

13

Our Contributions in a nutshell
Better regularization

Our methodIMSATRIM [Gomes et al., 2010]
Weight-decay, Clustering

More general InfoMax

14

Outline1. Introduction
2. Proposed Method: IMSAT = IM + SAT• Information Maximization (IM)• Self-Augmented Training (SAT)
3. Experiments
4. Conclusions

15

Information Maximization
Better regularization

IMSAT

RIM [Gomes et al., 2010]
More general InfoMax

16

Information Maximization
Previous approach:
InfoMax clustering [Bridle et al., 1991, Gomes et al., 2010]:
Learn p✓ (y|x) via max I(X; Y )✓17Information Maximization
For discrete representation learning?

18

Information Maximization
For discrete representation learning:
Our proposal:
Learn p✓ (y1 , . . . , yD |x) via max I(X; Y1 , . . . , YD )✓

19

Information Maximization
For discrete representation learning:
Our proposal:
Learn p✓ (y1 , . . . , yD |x) via max I(X; Y1 , . . . , YD )✓• Challenge: Combinatorial summation
à We need approximation!

XXy1

y2···

XyD

20

Information Maximizationy = (y1 , . . . , yD )000p✓ (y|x)
Approximate up to second order interaction:[Brown 2009]
I(X; Y1 , . . . , YD ) ⇡DX
d=1

I(X; Yd )XI(Yd ; Yd0 )1d6=d0 Dx

111111

21

Self-Augmented Training (SAT)
Better regularization

IMSAT

RIM [Gomes et al., 2010]
More general InfoMax

Similarization

Self-Augmented Training (SAT)
Augmentation function T (·) : X ! X
à User-specified transformation that does not change
the meaning of data

Data augmentation

x

T (x)

22

23

Self-Augmented Training (SAT)p✓ (y|T (x))

p✓ (y|x)
Similarizationy = (y1 , y2 , . . . , yD )KL[p✓ (y|x)kp✓ (y|T (x))][Bachman et al., 2014;
Miyato et al., 2016]

Data augmentationxT (x)

24

Self-Augmented Training (SAT)• Local perturbation

T (x) = x + r, krk2 = ✏• Random Perturbation Training (RPT)[Bachman et al., 2014]• Virtual Adversarial Training (VAT)[Miyato et al., 2016]

rx

25

IMSAT = Information Maximizing + SAT
00
0111

111

Discrete representations
Similarization

Information
Maximization

Data augmentation

26

Outline1. Introduction
2. Proposed Method: IMSAT• Information Maximization (IM)• Self-Augmented Training (SAT)
3. Experiments
4. Conclusions

27Experiments (Clustering)• Measure clustering accuracy• Batch normalization• ReLU activation• Softmax output#(Cluster size)p✓ (y|x)1200
1200※ Implementation available online
https://github.com/weihua916/imsatx#(input dimension)Reuters (Lewis et al., 2004)
20news (Lang, 1995)

10000
18040

420

2000
2000

43%
5%

28Experiments(Clustering)
Table 3.

Comparison of clustering accuracy on eight benchmark datasets (%). Averages and standard deviations over twelve trials werereported. Results marked with † were excerpted from Xie et al. (2016).
MethodMNIST
Omniglot
STLCIFAR10 CIFAR100
SVHN
Reuters
20newsK-means
53.2
12.0
85.6
34.4
21.5
17.9
54.1
15.5
dAE+K-means
79.8 †
14.1
72.2
44.2
20.8
17.4
67.2
22.1
†
DEC [Xie et al., 2014]
84.3
5.7 (0.3)
78.1 (0.1) 46.9 (0.9)
14.3 (0.6)
11.9 (0.4) 67.3 (0.2) 30.8 (1.8)
Linear RIM
59.6 (2.3) 11.1 (0.2) 73.5 (6.5) 40.3 (2.1)
23.7 (0.8)
20.2 (1.4) 62.8 (7.8) 50.9 (3.1)
Linear IMSAT (VAT)
61.1 (1.9) 12.3 (0.2) 91.7 (0.5) 40.7 (0.6)
23.9 (0.4)
18.2 (1.9) 42.9 (0.8) 43.9 (3.3)
Deep RIM
58.5 (3.5)
5.8 (2.2)
92.5 (2.2) 40.3 (3.5)
13.4 (1.2)
26.8 (3.2) 62.3 (3.9) 25.1 (2.8)
IMSAT (RPT)89.6 (5.4) 16.4 (3.1) 92.8 (2.5) 45.5 (2.9)
24.7 (0.5)
35.9 (4.3) 71.9 (6.5) 24.4 (4.7)
IMSAT (VAT)
98.4 (0.4) 24.0 (0.9) 94.1 (0.4) 45.6 (0.8)
27.5 (0.4)
57.3 (3.9) 71.0 (4.9) 31.1 (1.9)

where l and c are the ground-truth label and cluster• Tested on 8 benchmark datasets.
assignment produced using the algorithm for x , respectively. The m ranges theover alldatasets.
possible one-to-one mappings• Hyper-parameters are fixed throughout

A brief summary of dataset statistics is given in Table 2. In
the experiments, our goal was to discover clusters that correspond well with the ground-truth categories. For the STL,CIFAR10 and CIFAR100 datasets, raw pixels are not suitedfor our goal because color information is dominant. We
therefore applied 50-layer pre-trained deep residual networks (He et al., 2016) to extract features and used themnnn

between clusters and labels. The best mapping can be efficiently computed using the Hungarian algorithm (Kuhn,1955).

Reuters (Lewis et al., 2004)
20news (Lang, 1995)

10000
18040

420

2000
2000

43%
5%

29Experiments(Clustering)
Table 3.

Comparison of clustering accuracy on eight benchmark datasets (%). Averages and standard deviations over twelve trials werereported. Results marked with † were excerpted from Xie et al. (2016).
MethodMNIST
Omniglot
STL
CIFAR10 CIFAR100
SVHN
Reuters
20newsK-means
53.2
12.0
85.6
34.4
21.5
17.9
54.1
15.5
dAE+K-means
79.8 †
14.1
72.2
44.2
20.8
17.4
67.2
22.1
†
DEC
84.3
5.7 (0.3)
78.1 (0.1) 46.9 (0.9)
14.3 (0.6)
11.9 (0.4) 67.3 (0.2) 30.8 (1.8)
Linear RIM
59.6 (2.3) 11.1 (0.2) 73.5 (6.5) 40.3 (2.1)
23.7 (0.8)
20.2 (1.4) 62.8 (7.8) 50.9 (3.1)
Linear IMSAT (VAT)
61.1 (1.9) 12.3 (0.2) 91.7 (0.5) 40.7 (0.6)
23.9 (0.4)
18.2 (1.9) 42.9 (0.8) 43.9 (3.3)
Deep RIM
58.5 (3.5)
5.8 (2.2)
92.5 (2.2) 40.3 (3.5)
13.4 (1.2)
26.8 (3.2) 62.3 (3.9) 25.1 (2.8)
IMSAT (RPT)89.6 (5.4) 16.4 (3.1) 92.8 (2.5) 45.5 (2.9)
24.7 (0.5)
35.9 (4.3) 71.9 (6.5) 24.4 (4.7)
IMSAT (VAT)
98.4 (0.4) 24.0 (0.9) 94.1 (0.4) 45.6 (0.8)
27.5 (0.4)
57.3 (3.9) 71.0 (4.9) 31.1 (1.9)• Used perturbation as augmentation function.
A brief summary of dataset statistics is given in Table 2. In
where l and c are the ground-truth label and clusterrthe•experiments,our (VAT)
goal was toachieveddiscover clustersstate-of-the-art
that corassignment produced using the algorithm for x , respecIMSAT
respond well with the ground-truth categories. For the STL,tively. The m ranges over all possible one-to-one mappings
CIFAR10performance.
and CIFAR100 datasets, raw pixels are not suited
between clusters and labels. The best mapping
x can be efnnnfor our goal because color information is dominant. We
therefore applied 50-layer pre-trained deep residual networks (He et al., 2016) to extract features and used themficiently computed using the Hungarian algorithm (Kuhn,1955).

30

Experiments (Hash Learning)p✓ (y1 , . . . , yD |x)• 3 evaluation metrics:#(hash bits)• mean average precision• precision @ sample=500• precision @ hamming dist=2• 16-bit (D = 16)x#(input dimension)

31

Figure 2. Randomly sampled clusters of Omniglot discovered using (a) IMSAT (VAT) and (b) IMSAT (VAT & affine). Each row contains
randomly sampled data points in same cluster.

Experiments (Hash Learning)

Table 5. Comparison of hash performance for 16-bit hash codes (%). Averages and standard deviations over ten trials were reported.
Experimental results of Deep Hash and the previous methods were excerpted from Erin Liong et al. (2015).

Method(Dimensions of hidden layers)
Spectral hash (Weiss et al., 2009)
PCA-ITQ (Gong et al., 2013)
Deep Hash (60-30)
Linear RIM
Deep RIM (60-30)
Deep RIM (200-200)
Deep RIM (400-400)
IMSAT (VAT) (60-30)
IMSAT (VAT) (200-200)
IMSAT (VAT) (400-400)

Hamming ranking (mAP)MNIST
CIFAR10
26.6
12.6
41.2
15.7
43.1
16.2
35.9 (0.6)
24.0 (3.5)
42.7 (2.8)
15.2 (0.5)
43.7 (3.7)
15.6 (0.6)
43.9 (2.7)
15.4 (0.2)
61.2 (2.5)
19.8 (1.2)
80.7 (2.2)
21.2 (0.8)
83.9 (2.3)
21.4 (0.5)

precision @ sample = 500
MNISTCIFAR10
56.3
18.8
66.4
22.5
67.9
23.8
68.9 (1.1)
15.9 (0.5)
67.9 (2.7)
21.8 (0.9)
68.7 (4.9)
21.6 (1.2)
69.0 (3.2)
21.5 (0.4)
78.6 (2.1)
21.0 (1.8)
95.8 (1.0)
27.3 (1.3)
97.0 (0.8)
27.3 (1.1)

precision @ r = 2
MNIST
CIFAR10
57.5
18.5
65.7
22.6
66.1
23.3
71.3 (0.9) 14.2 (0.3)
65.9 (2.7) 21.2 (0.9)
67.0 (4.9) 21.1 (1.1)
66.7 (3.2) 20.9 (0.3)
76.5 (2.3) 19.3 (1.6)
94.6 (1.4) 26.1 (1.3)
96.2 (1.1) 26.4 (1.0)• Tested on 2 benchmark datasets.
constraint, we report the results for 32-bit hash codes in5. Conclusion & Future Work• Hyper-parameters
aretendency
fixedas throughout
thedatasets.
AppendixH, but the results showed a similar
In this paper, we presented IMSAT, an informationthat of 16-bit hash codes. We see from Table 5 that IMSAT

32

Figure 2. Randomly sampled clusters of Omniglot discovered using (a) IMSAT (VAT) and (b) IMSAT (VAT & affine). Each row contains
randomly sampled data points in same cluster.

Experiments (Hash Learning)

Table 5. Comparison of hash performance for 16-bit hash codes (%). Averages and standard deviations over ten trials were reported.
Experimental results of Deep Hash and the previous methods were excerpted from Erin Liong et al. (2015).

Method(Dimensions of hidden layers)
Spectral hash (Weiss et al., 2009)
PCA-ITQ (Gong et al., 2013)
Deep Hash (60-30)
Linear RIM
Deep RIM (60-30)
Deep RIM (200-200)
Deep RIM (400-400)
IMSAT (VAT) (60-30)
IMSAT (VAT) (200-200)
IMSAT (VAT) (400-400)

Hamming ranking (mAP)MNIST
CIFAR10
26.6
12.6
41.2
15.7
43.1
16.2
35.9 (0.6)
24.0 (3.5)
42.7 (2.8)
15.2 (0.5)
43.7 (3.7)
15.6 (0.6)
43.9 (2.7)
15.4 (0.2)
61.2 (2.5)
19.8 (1.2)
80.7 (2.2)
21.2 (0.8)
83.9 (2.3)
21.4 (0.5)

precision @ sample = 500
MNISTCIFAR10
56.3
18.8
66.4
22.5
67.9
23.8
68.9 (1.1)
15.9 (0.5)
67.9 (2.7)
21.8 (0.9)
68.7 (4.9)
21.6 (1.2)
69.0 (3.2)
21.5 (0.4)
78.6 (2.1)
21.0 (1.8)
95.8 (1.0)
27.3 (1.3)
97.0 (0.8)
27.3 (1.1)

precision @ r = 2MNISTCIFAR10
57.5
18.5
65.7
22.6
66.1
23.3
71.3 (0.9) 14.2 (0.3)
65.9 (2.7) 21.2 (0.9)
67.0 (4.9) 21.1 (1.1)
66.7 (3.2) 20.9 (0.3)
76.5 (2.3) 19.3 (1.6)
94.6 (1.4) 26.1 (1.3)
96.2 (1.1) 26.4 (1.0)• IMSAT (VAT) outperformed the previousmethods.
5. Conclusion & Future Work

constraint, we report the results for 32-bit hash codes inAppendix H, but the results showed a similar tendency asthat of 16-bit hash codes. We see from Table 5 that IMSATIn this paper, we presented IMSAT, an information-

33

Outline1. Introduction
2. Background (Regularized Information Maximization[Gomes et al., 2010])
3. Proposed Method (Information Maximizing SelfAugmented Training)
4. Experiments
5. Conclusions

34

Conclusions
Better regularization
SAT regularization

IMSAT

DeeperRIM [Gomes et al., 2010]

InfoMax for learning
discrete representations
More general InfoMax

35

Conclusions
Better regularization

IMSAT
State-of-the-art inclustering andunsupervised hash learning

RIM [Gomes et al., 2010]
More general InfoMax