Stacked unsupervised learning with a network architecture found by supervised meta-learning
Authors: Kyle Luther
Sebastian Seung
Venue: NeurIPS 2022 Submitted (2022)
Keywords: stacked unsupervised learning
meta-learning
unsupervised clustering

Abstract: Stacked unsupervised learning (SUL) seems more biologically plausible than backpropagation, because learning is local to each layer. But SUL has fallen far short of backpropagation in practical applications, undermining the idea that SUL can explain how brains learn. Here we show an SUL algorithm that can perform completely unsupervised clustering of MNIST digits with comparable accuracy relative to unsupervised algorithms based on backpropagation. Our algorithm is exceeded only by self-supervised methods requiring training data augmentation by geometric distortions. The only prior knowledge in our unsupervised algorithm is implicit in the network architecture. Multiple convolutional ``energy layers'' contain a sum-of-squares nonlinearity, inspired by ``energy models'' of primary visual cortex. Convolutional kernels are learned with a fast minibatch implementation of the K-Subspaces algorithm. High accuracy requires preprocessing with an initial whitening layer, representations that are less sparse during inference than learning, and rescaling for gain control. The hyperparameters of the network architecture are found by supervised meta-learning, which optimizes unsupervised clustering accuracy. We regard such dependence of unsupervised learning on prior knowledge implicit in network architecture as biologically plausible, and analogous to the dependence of brain architecture on evolutionary history.

Stacked unsupervised learning with a network architecture found by supervised meta-learning
Anonymous Author(s)
Affiliation
Address
email
Abstract
Stacked unsupervised learning (SUL) seems more biologically plausible than backpropagation, because learning local to each layer. But SUL has fallen far short of backpropagation in practical applications, undermining the idea that SUL can explain how brains learn. Here show an SUL algorithm that can perform completely unsupervised clustering of MNIST digits with comparable accuracy relative to unsupervised algorithms based on backpropagation. Our algorithm is exceeded only by self-supervised methods requiring training data augmentation by geometric distortions. The only prior knowledge in our unsupervised algorithm is implicit in the network architecture. Multiple convolutional "energy layers" contain -fuuuoruqsssess nonlinearity, inspired by "energy models" of primary visual cortex. Convolutional kernels are learned with fast minibatch implementation of the K-Subspaces algorithm. High accuracy requires preprocessing with an initial whitening layer, representations that are less sparse during inference than learning, and rescaling for gain control. The hyperparameters of the network architecture are found by supervised meta-learning, which optimizes unsupervised clustering accuracy. We regard such dependence of unsupervised learning on prior knowledge implicit in network architecture biologically plausible, and analogous the dependence brain architecture on evolutionary history.
3
10
11
12
13
14
16
18
Introduction
20 Recently there has been renewed interest in the hypothesis that the brain learns through some version of the backpropagation algorithm (31]. This hypothesis runs counter to the neuroscience textbook
21
22 account that local learning mechanisms, such as Hebbian synaptic plasticity, are the basis for learning
23 by real brains. The concept of local learning has fallen out of favor because it has been far eclipsed
24 by backpropagation in practical applications. This was not always the case. Historically, a popular to repeatedly stack single-layer unsupervised learning 26 module to generate a multilayer network, as exemplified by Fukushima's 's pioneering Neocognitron Stacked unsupervised learning (SUL) avoids the need for the backward pass of backpropagation, because learning is local to each layer.
approach visual object recognition
28
29 In the 2000s, SUL was quite popular. There were attempts to stack diverse kinds of unsupervised learning modules, such as sparse coding (11,, restricted Boltzmann machines [4 29
30
denoising
autoencoders 46], K-Means 7 and independent subspace analysis 25
31
SUL managed generate impressive-looking feature hierarchies that reminiscent of the hierarchy of visual cortical areas. Stacking restricted Boltzmann machines yielded features that were sensitive oriented edges in the first layer, eyes and noses in the second, and entire faces in the third layer Stacking three sparse coding layers yielded an intuitive feature hierarchy where higher layers were more selective to whole MNIST digits and lower layers were selective small strokes [43] Although these feature hierarchies are pleasing to the eye, they have not been shown to be effective
33
34
35
29
36
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

38
58 In the special case that the sum-of-squne com-of-squares contains a single term, or equivalently the subspaces are for visual object recognition, in spite of recent efforts to revive SUL using sparse coding 43 similarity matching (39].
6
and
39
40
Here we show an SUL algorithm that can perform unsupervised clustering of MNIST digits with high accuracy (2% error). The clustering accuracy is as good the best unsupervised learning algorithms based on backpropagation. As far as we know, our accuracy is only exceeded by self-supervised methods that require training data augmentation or architectures with hand-designed geometric transformations. Such methods use explicit prior knowledge in the form of geometric distortions to aid learning.
41
42
43
44
45
46
Our network contains three convolutional energy layers inspired by energy models of primary visual cortex [T]. which contain a sum-of ouuuruuressssssssse nonlinearity. Our energy layer was previously used by
16 in their independent subspace analysis (ISA) algorithm for learning complex cells. The kernels 48
of our convolutional energy layers are trained by K-Subspaces clustering [45] rather than ISA. We
50
also provide a novel minibatch algorithm for K-Subspaces learning.
51
After training. the first energy layer contains neurons that selective for simple features but invariant to local distortions. These are analogous to the complex cells in energy models of the primary visual cortex .. The invariances are learned here rather than hand-designed, similar to previous work We go further by stacking multiple energy layers. The second and third energy layers learn more sophisticated kinds of invariant feature selectivity. As mentioned above, feature hierarchies have previously been demonstrated for SUL. The novelty here is the learning of a feature hierarchy that is shown be useful for pattern recognition.
52
54
16 15I.
56
57
59 restricted to be rank one, our convolutional energy layer reduces to a conventional convolutional layer.
60 Accuracy worsens considerably, consistent with the idea that the energy layers are important for learning invariances. The energy layers contain adaptive thresholding that allows representations 62 to be less sparse for inference than for learning. This is also shown to be important for attaining high accuracy, as has been reported for other SUL algorithms
61
[8] 24]. Representations are rescaled for gain control, and the energy layers are preceded by a convolutional whitening layer. These aspects of the network also important for high accuracy.
64
66
The detailed architecture of our unsupervised network depends on subspace number and rank, kernel size, and sparsity. We use automatic tuning software [2] to systematically search for a hyperparameter configuration that optimizes the clustering accuracy of the unsupervised network. Evaluating the clustering accuracy requires labeled examples, so the meta-learning is supervised, while the learning unsupervised. A conceptually similar meta-learning approach has previously been applied to search for biologically plausible unsupervised learning algorithms (37
67
68
70
71
72 For each iteration of the meta-learning, the weights of our network are initialized randomly before running the SUL algorithm. Therefore the only prior knowledge available to SUL resides in the architectural hyperparameters; no weights are retained from previous networks. We regard this implicit encoding of prior knowledge in network architecture as biologically plausible, because brain
73
74
75
76 architecture also contains prior knowledge gained during evolutionary history, and meta-learning is
77 analogous to biological evolution. In its own "lifetime" our network is able to learn with no labels at
78 all. This is possible because the network is "born" with an architecture inherited from networks that "lived" previously.
Related work
80
81
Independent subspace analysis Our method related to past works on independent subspace
analysis [16. 17 26), mixtures of principal components analyzers [3), and subspace clustering A core idea behind these works is that invariances can be represented via subspaces. The most similar of these works to ours is [26] who stacked 2 layers of subspace features learned with independent subspace analysis to action recognition datasets.
83
45
47
85
86
K-Means based features Mathematically the work of 9 is similar to ours. They use variant of K-Menns to learn patch features. Their learning algorithm (Algorithm 1) is a special case of our alg. where they use 1D subspaces and full batch updates. Their inference procedure is also very similar,
87
88
89 in that they use a dynamic threshold to sparsify patch vector representations. They use spatial pooling layers for invariance, whereas our pooling is learned by the energy layers. Their primary mode of

---

Input pixels
Convolutional ZCA
energy layers
AvgPool to 2x2 grid
K-Means clustering
.I...
Figure I: Our multilayer convolutional energy network. The last step of K-Means clustering can be regarded as part of the evaluation rather than the network itself.
81
evaluation linear evaluation with the full labeled dataset, whereas we will seek to produce an unsupervised learning algorithm which clusters inputs without labels. They additionally employ receptive field selection method.
92
93
94
Capsule networks Capsules are vector representations where the vector's length represents the probability of entity existence and the direction represents properties of that entity [1]] In networks, the r-dimensional subspace vectors Vx can be interpreted as learned "pose" vectors. we only propagate the norm ||Vx| thus suppressing the
96
However our goal is to learn invariance, pose details at each layer. Compared to the unsupervised capsule networks [23]. our networks do not any backpropagation of gradients, and do not rely hand-designed affine transformations to generate representations.
98
99
100
Meta-lean Meta-learning of unsupervised learning rules Our work will rely using label-based clustering objective to evaluate and tune an unsupervised learning rule. In other words there are two levels of learning, unsupervised inner loop and a supervised outer loop. This is the domain of meta-learning.
102
104
The more common scenario for meta-learning is to focus on optimizing over a distribution of tasks, but for this work will focus
task. The work of (36] is perhaps the most closely related to ours. They use a few-shot supervised learning rule to tune an unsupervised learning algorithm that is form of randomized backward propagation gradient feedback from a higher layer L to a lower layer L L- 1.
106
107
[30]. We wish to go further and remove any form of
108
109
3
Network architecture
110
The overall network architecture is shown in Figure [ and consists of a whitening layer followed by multiple convolutional energy layers and final average pooling layer. The energy layers include adaptive thresholding to control sparsity of activity, as well as normalization of activity by rescaling.
112
113
Convolutional ZCA We define ZCA whitening for image patches in terms of the eigenvalues of the pixel-pixel correlation matrix. Our ZCA filter attenuates the top k: eigenvalues, setting them equal to the kthh largest eigenvalue. The smaller eigenvalues pass through unchanged. Our definition slightly different from [10], which zeros out the smaller eigenvalues completely.
114
115
116
117
The first layer of our network is convolutional variant of ZCA, in which each pixel of the output image computed by applying ZCA whitening the input patch centered at that location, and discarding all but the center pixel of the whitened output patch (p. 118 of [10). The kernel has a single- pixel center with diffuse surround (see Appendix). Patch size and number of whitened eigenvalues are specified in Table Reflection padding is used to preserve the output size. Unsupervised learning algorithms such as sparse coding (40 and ICA are often preceded by whitening when applied to images.
118
119
120
122
123
124
Convolutional energy layer We define the following modification of convolutional layer, which computes & output feature maps given m input feature maps. We define the "feature vector" at a location consist of the values of a of feature maps at given location.
125
126
127
Convolve the m input feature maps with kernels to produce kr feature maps ("S ("S-maps")) in the standard way, except with no bias or threshold.
128
Divide the kr feature maps into l groups of r. For each group, compute the Euclidean 2.
130
norm the r-dimensional feature vector at each location. The result feature maps ("C-maps").
3

---

kr. feature maps
k feature maps
feature maps
Input feature maps
maps
C maps
Figure 2: Diagram of single energy layer. The outputs of this layer the C feature maps, while S just intermediate feature maps. We produce feature maps (S) with a standard convolution. We produce (C) feature maps by square root the sum of squares inside each group of S feature maps, followed by an adaptive thresholding and rescaling.
Individual values in the S- and C-maps of Steps and 2 will be called S-cells and C-cells, respectively. Each C-cell computes the square root of the sqm-of-squares (Euclidean norm) of r S-cells. The terms S-cells and C-cells are used in homage [T]. They are reminiscent of energy models of primary visual cortex, in which complex cells compute the sum-of-squares of simple cell outputs M.
133
135
136
The sum-of-square can be regarded location in different feature maps. Pooling typically performed on neighboring locations in the same feature map, yielding invariance to small translations by design. We will see later on that the sum-of-squares the energy layer acquires invariances due to learned kernels.
kind of pooling operation, but applied to S-cells at the same
137
138
139
140
Adaptive thresholding and normalization turns out to be important to postprocess the output of a convolutional energy layer follows.
142
3. For each k-dimensional feature vector, adaptively threshold so that there are W winners active.
143
4. Normalize the feature vector unit Euclidean length, and rescale by multiplying with the Fuclidean norm of the input patch at the same location.
145
148
Define as the k-dimensional vector which is the & feature map values at location u in the C feature maps. In Step 3, the adaptive thresholding of takes the form max fi r for i = 1 to k where the W + 1st largest element of f. Note that is set adaptively for each location. Such adaptive thresholding was used by 44 and is version of the well-known W WW-winners-tae take-all concept [33]
147
148
149
150
After thresholding, C-cells are sparsely active, with sparseness controlled by the hyperparameter W. S-cells, the other hand, will typically be densely active, since they are linear.
152
Step 4 normalizes the k-dimensional feature vector, and also multiplies by the Buclidean norm of the input patch, to prevent the normalized output from being large even if the input is vanishingly small. The kernels in the layer are size p p. so that the input patch at any location contains mp² values where m is the number of input feature maps.
153
154
155
Final average pooling layer To reduce the output dimensionality, feature map of the last energy layer to 2 x 2 grid, exactly as in [8. We have generally avoided pooling because
average pool each output
157
want learn invariances rather than design them in. However, final average pooling will turn out to advantageous later on for speeding up meta-learning. In Table we show that this pooling layer has only modest impact the final clustering accuracy.
159
160
4
Stacked unsupervised learning
161
162
The outputs of the ZCA layer are used as inputs to a convolutional energy layer. We train the kernels of this layer, and then we freeze the kernels. The outputs of the first convolutional energy layer are used as inputs to second convolutional energy layer, and the kemnlss in this layer are trained. We repeat this procedure stack a total of three convolutional energy layers, and then conclude with a final pooling layer.
163
185

---

LAYER
# SUBSPACES (k)
SUBSPACE RANK (r)
# WINNERS (w)
KERNEL SIZE
PADDING
L1
37
2
9
8
2
L2
9
5
L3
58
16
21
Table 1: Detailed architecture of three energy layer net. The first energy layer is preceded by convolutional ZCA whitening of the imput image. where the kernel size 9 and the number whitened eigenvalues These parameters are found with automated hyperparameter tuning.
REPRESENTATION
PIXELS 46.2
ZCA
LAYER 22.7
LAYER 2
LAYER 3
2x2 POOL
CLUSTERING ERROR (%)
50.0
22.2
2.3
2.1
Table 2: Clustering error after every layer of our network with three energy layers.
167
K-Subspaces clustering Consider an energy layer with & C-cells and kr S-cells at each location. The
168
S-cells at one location are linearly related to the input patch at that location by the set of lo matrices V, for j to k, each of size X mp? Here mp² the size of the flattened input patch, where m
169
170
is the number of input feature maps and is the kernel size.
We can think of these matrices as defining set of l linear subspaces of rank embedded in Rmpp We learn these matrices with convolutional extension of the K-Subspace learning algorithm [45 Let x,, be the previous layer's 's m-dimensional feature maps for pattern n. Define X.., as the mpp- dimensional feature vector created by flattening a ppx. patch centered around location i. K-Subspaces
171
172
173
174
aims to learn k r-dimensional subspaces Ve E Rrxmp" 2 such that every patch well modeled by one of these subspaces. This is formalized with the following optimization:
175
176
min min
ΣΣ n2
Crsk |>xii
V V Vuxnil
(1)
177
such that Crik E {0. 1 and Σn Crisk = 1. We provide a novel minibatch algorithm for this opti- mization in Appendix A During the learning. each matrix V, is constrained so that its rows are orthonormal. Therefore at each location the C-cells contain the Euclidean norms of the projection of the input patch onto each of the subspaces, before the adaptive thresholding and rescaling, This is why the K-Subspaces algorithm is naturally well-suited for learning the convolutional energy layer.
179
181
182
During K-Subspaces learning, an input patch is assigned to a single subspace, which means there winner-take-all competition between C-cells at given location. During inference, on the other hand, there can be many C-cells active at given location (depending on the "number of winners" hyperparameter w.)
185
185
186
This idea of making representations less sparse for inference than for learning has been exploited by number of authors 24). This may be advantageous because overly sparse representations can suffer from sensitivity to distortions 32].
187
188
Experiments with MNIST digits We train network with the architecture of Fig. The details of the ZCA layer and the three convolutional energy layers are specified by Table [ Each energy layer is trained using single pass through the 60,000 MNIST [27] training examples (without using the labels), with a minibatch size of 512. Training on single Nvidia Titan 1080-Ti GPU takes 110 seconds.
190
192
183
194
Evaluation of learned representations We adopt the intuitive notion that the output representation vectors of a "good" network should casily cluster into the underlying object classes defined by image labels. This is quantified by applying the trained network to 10,000 MNIST test examples. The resulting output representation vectors clustered with the scikit-learn implementation of K-Means. This uses full batch EM-style updates and additionally returns the lowest mean squared error clustering found by running the algorithm using 10 different initializations.
196
198
200
We set the number of clusters to be 10, to match the number of MNIST digit classes. We compute the disagreement between the cluster assignments and image labels, and minimize over permutations of the clusters. The minimal disagreement the final evaluation, which we will call the clustering error
202
49
5

---

REPRESENTATION
CLUSTERING ERROR (%)
PIXELS
46.2
NMF 128
44.0
STACKED DENOISING AUTOENCODERS!
46
18.8
UMAP B5
17.9
GENERATIVE ADVERSARIAL NETWORKS
12
17.2
VARIATIONAL AUTOENCODER 1
22
16.8
DEEP EMBEDDED CLUSTERING
48
15.7
VADE
19
5.5
CLUSTERGAN
38
5.0
UMAP + GMM
35
3.6
N2D [4
2.1
OURS (THREE LAYER NET)
2.1
INVARIANT INFORMATION CLUSTERING (AVG SUB-HEAD)
18
1.6
STACKED CAPSULE AUTOENCODERS [K
1.3
INVARIANT INFORMATION CLUSTERING (BEST SUB-HEAD)
18
0.8
Table Comparison of clustering accuracy for other unsupervised learning algorithms. For methods which do not generate clusters, k-means is used to cluster representations into 10 clusters, with the exception of UMAP + GMM in which case we use Gaussian Mixture Model to cluster. The errors for methods with dagger are all taken from [88) methods with double dagger are taken from 38
UMAP uses "out-of-the-box" parameter
settings.
ID12 1012 12L L
Subspace
dimension
16 out of 37 learned 2D subspaces
Figure 3: Layer subspaces learned with our algorithm. Each 8 8 image corresponds to kernel.
204
Table 2 quantifies the accuracy of each layer. The accuracy of the ZCA representation is actually worse than that of the raw pixels. However, the accuracy of the representations improves with each additional convolutional energy layer, until the final error is just 2.1%.
205
207
Comparisons with other algorithms are shown in Table It is helpful distinguish between algo- rithms that require training data augmentation, and those that do not. Many well-known unsupervised algorithms that do not make use of training data augmentation, such as GANs, variational autoen- coders, and stacked denoising autoencoders, yield clustering errors of 15 to 19%. Methods such as VaDE and ClusterGAN encourage clustered latent representations and these give rise to much lower clustering error. Clustering 2D UMAP representations with K-Means gives suprisingly high error,
208
209
211
212
213
and this likely the returned clusters are not spherical. Using a Gaussian Mixture Model instead gives much lower error. See the appendix for more discussion.
215
Self-supervised algorithms require training data augmentation, using prior knowledge to create same- class image pairs. For MNIST clustering one of the highest performance is Invariant Information Clustering, with clustering error 1.6 0.8% [8]. Our approach delivers roughly 2 % error and is noticeably better than the other algorithms that do not require training data augmentation.
216
217
218
Stacked capsule autoencoders 23 also achieve high accuracy. However, this algorithm incorporates a model of geometric distortions. Furthermore, despite the modifier "stacked," the algorithm does not conform the original idea of repeatedly stacking the same learning module. The architecture uses several distinct types of layers and still backpropagates gradients.
220
222
Learned kernels Figure shows that the kernels in the first energy layer look like bars or edges, more often curved than straight. Since the subspaces are rank 2 (Table [ the kernels come in pairs. The kernels in pair look quite similar to each other, and typically appear to be related by small distortions. Therefore the two S-cells in pair should prefer slightly different versions of the same feature. By computing the square root of the sum-of-squares of the S-cells, the C-cell should detect the same feature with more invariance to distortions.
224
226
228

---

Pixels
ZCA
Layer
Layer
Layer 3
Figure Output "C" feature maps for input images.
229
This behavior is reminiscent of energy models of simple and complex cells in primary visual cortex 42) A complex cell computes the sum-of-squares of simple cells, which are quadrature pairs of is invariant to phase shifts, much as the sum-of -um-of-squares of sine and cosine is constant. Our model also contains sum-of-squares but the filters are learned rather than hand-designed. Learning of complex cells was previously demonstrated by [16] for an energy model and by [00 for similar model. [15] showed how to substitute rectification for quadratic nonlinearity. Our contribution is to stack multiple energy layers, and investigate the accuracy of the
231
Gabor filters. The sum-of-square
233
235
resulting network at clustering task.
237
Feature maps Figure SaareeAshwww shows the C-maps for the first 3 MNIST digits. The first energy layer exhibits an intermediate degree of sparsity, (approximately 20 % of the maps active at central locations). The second energy layer exhibits dense representations (nearly all maps are active at central locations). The final energy layer shows quite sparse representations (approximately 5% of the maps active). The feature maps appear to be broad spots.
239
241
Meta-learning of network architecture for unsupervised learning
242
243
The detailed architecture of the network is specified in Table and one might ask where it came from. For example, the kernel size is in the third energy layers, but sizes are only single digit integers in other layers. The subspaces are rank 2 in the first energy layer, but rank 3 and 16 in the subsequent energy layers. The second energy layer is highly dense (all but one neuron is active at each location), while the third layer is highly sparse (only two neurons are active at each location). There is a total
244
246
248 of 17 numbers in Table [ and we can regard them as hyperparameters of the unsupervised learning algorithm.
250
In the initial stages of our research, we set hyperparameters by hand, guided by intuitive criteria such as incressing the subspace rank with layer (meaning more invariances learned). Later on, we resorted automated hyperparameter tuning. For this purpose, we employed the Optuna software package, which implements Bayesian method (2). We found that we were able to find hyperparameter configurations with considerably better performance than our hand-designed configurations.
251
252
254
255
In particular, the configuration of Table T was obtained by automated search through 2000 hyper- parameter configurations, which took approximately 20 hours using 8 Nvidia GTX 1080 GPUS. Each hyperparameter configuration was used to generate an unsupervised clustering of the training set, and its accuracy with respect to all 60,000 training labels was the objective function of the search.
256
257
258
259
For the ZCA layer we tune the kernel size kzea E [1,11] and number of whitened eigenvalues zco E [0, x-2 I-aa.. For each subspace layer tune number of subspaces k E [2. 64], subspace dimension r E [1, 16), number winners w E [1, k). kernel size k, [1, Linputsszee, and padding
261
262
P. [0, floor(k./22 The stride is fixed at 1.
263 In some respects, the optimized configuration of Table Toaleeeeni ended up conforming to our qualitative expectations. Sparsity and kernel size increased with layer, consistent with the idea of feature
7

---

NUMBER LABELS IN TRAINING SET
10
30
50
100
500
5000
60000
% MISCLASSIFIED (TEST)
30.0
9.2
10.7
5.3
4.6
2.9
2.
MISCLASSIFIED (TRAIN)
0.0
0.0
0.0
3.0
2.8
2.9
2.5
Table 4: Label efficiency of our stacked learning algorithm. In each case, the algorithm has access to all 60K unlabeled images from the training set. What varies is the number of labels we to evaluate each setting of learning parameters.
EXPERIMENT
% ERROR (TEST)
% ERROR (TRAIN)
4 ENERGY LAYERS
3.1
3.6
ENERGY LAYERS
2.1
2.5
ENERGY LAYERS
2.9
3.3
ENERGY LAYERS
20.2
20.7
No ZCA 3 ENERGY LAYERS
4.0
4.8
No RESCALING 3 ENERGY LAYERS
4.1
4.6
No ZCA & No RESCALING 3 ENERGY LAYERS
9.7
10.1
SUBSPACES ENERGY LAYERS
16.0
17.3
RANDOM SUBSPACES 3 ENERGY LAYERS
29.4
31.1
Table Systematic studies with our multilayer energy model. The learning hyperparameters are tuned using all 60K labels from the training set.
265
hierarchy with progressively greater selectivity and invariance. However, the number of subspaces behaved nonmonotonically with layer, which was unexpected.
288
267 We can think of the hyperparameter tuning as an outer loop surrounding the unsupervised learning algorithm. We will refer this outer loop as meta-leaming. Given architecture, the unsupervised learning algorithm requires no labels at all. However, the outer loop searches for the optimal network architecture by using training labels. Therefore, while the learning is unsupervised, the meta-learning is supervised. Alternatively, the outer loop can use only fraction of the training labels, in which case the meta-learning is seni-supervised.
269
271
272
273
interesting to vary the number of training labels used for hyperparameter search. The results shown in Table 4 The best accuracy is obtained when all 60,000 training set labels are used. Accuracy degrades slightly for 5000 labels, and more severely for fewer labels than that. The test error can be lower than the training error. This is not a mistake, and it appears to result from non-random ordering of patterns in the MNIST train and test sets.
275
277
278
To be clear about the use of data, we note that neither test images nor labels are used during learning or meta-leann meta-learinn. Training images but not labels are used during learning. Training labels are used by meta-learning. With each iteration of meta-learning, the weights of the network are randomly
279
281
initialized.
Experiments
252
283
The hyperparameter search explores the space of networks defined by Fig. 1 We can widen the space of exploration by performing ablation studies, with results given in Table In all experiments, we completely retune the hyperparameters using the full 60K training labels to evaluate clustering accuracy. With the exception of the experiment where we vary the number layers, we use the
285
286
287
energy layer network in this section.
Vary number of energy layers One vary the depth of the network by adding or removing energy layers. and energy layers yield similar accuracy, and are roughly 1% absolute error (50% relative eтror) worse than for energy layers. A energy layer net is dramatically worse (>20% train/test error), suggesting the stacking is critical for performance.
289
291
The hyperparameters for each of these optimal architectures are provided in the Appendix. The optimal energy layer net resembles the energy layer net with its 2nd energy layer removed, while simultaneously making the 1st energy layer less sparse.
293
8

---

295
Remove ZCA whitening Removing the ZCA layer increases both train and test error of the three
296
energy layer net by roughly 2x. This might seem surprising, as T that whitening by itself decreases accuracy if we directly cluster whitened pixels instead of raw pixels. Apparently, it helpful to "take a step backward, before taking steps forward" in the case of our networks with three energy layers. We have no theoretical explanation for this interesting empirical fact.
297
296
299
300
Preprocessing images by whitening has a long history. The retina has been said to perform whitening operation, which is held to be an "efficient coding" that reduces redundancy in an information theoretic sense 3] Our experiment suggests that whitening is useful because it improves the accuracy of subsequent representations produced by stacked unsupervised learning. This seems at least superficially different from efficient coding theory. because the invariant feature detectors in our
301
302
303
304
networks appear to discard some information (Figure
305
306
Remove rescaling We now remove the rescaling operation from our energy layers. We observe a roughly 2x increase in both test and train error. It may be surprising that such a seemingly innocuous change can double the resulting train/test errors. This is likely because we only have 17 hyperparameters to optimize; with a limited set of parameters to tune, small changes in architecture can lead to dramatic performance changes we only have limited flexibility to tune hyperparameters.
307
308
309
310
311
Remove rescaling and ZCA whitening Removing both ZCA and the per-layer rescaling operations causes 4x increase in train and test error. Apparently the damage to the resulting performance is multiplicative: removing ZCA alone doubles the train/test error, removing rescaling alone doubles train/test error, and removing both quadruples the train/test error.
312
313
314
315
ID subspaces We rerun the automated hyperparameter tuning experiments from the previous section, this time restricting our subspaces to be ID. The subspace norm can be thought of as conventional dot product followed by absolute value nonlinearity Vx = |V x| where v is the one row of the subspace matrix V. When the subspaces are ID, our algorithm in fact reduces to that of (9].
316
317
319
Random subspaces Finally we ask the question: does learning actually help or is simply the
320
architecture that matters? To do so, we run tuning experiments with random orthogonal subspaces.
321
We observe that performance is almost completely erased by using random subspaces. telling us that the unsupervised learning component is indeed critical for clustering performance.
322
Discussion
323
324
The elements of our SUL algorithm were already known in the 2000s: ZCA whitening, energy layers, K Subspaces, and adaptive thresholding and rescaling of activities during inference. To achieve state-of-the-ard unsupervised clustering accuracy on MNIST, we employed one more trick, automated tuning of hyperparameters. Such meta-learning is more feasible now than it was in the 2000s, because computational power has increased since then.
325
326
327
328
329
Given that meta-lcarning optimizes supervised criterion, is our SUL algorithm really unsupervised? is true that the complete system is supervised. However, even if the outer loop (meta-learning) is supervised, it is accurate to say that the inner loop (learning) is unsupervised. For any hyperparameter configuration, the network starts from randomly initialized weights, and proceeds to learn in a purely label-free unsupervised manner.
330
331
332
333
Although MNIST is an easy dataset by today's standards, think that the success of our SUL algorithm at unsupervised clustering is still surprising. We are only tuning 17 hyperparameters, and it not clear a priori that our approach would be flexible enough succeed even for MNIST.
335
336
337 In future work, it will be important to investigate more complex datasets or tasks. Following the more common scenario for meta-lcarning, future work should train unsupervised algorithm distribution of tasks, and test transfer to some held-out distribution. We have done some preliminary experiments with the CIFAR-10 dataset. The meta-learning is more time-consuming because larger network architectures must be explored. The research is still in progress, but we speculate that the winner- winner-take-all take-all behavior of K-Subspaces learning may turn out to be limitation. If so, it will be important to relax the winner-take-al condition in Equation
339
341
343
Evolution created brains through eons of trial-and-error. For us to discover how the brain learns, it 345
will be important to exploit our computational resources, and use meta-learning to empirically search the space of biologically plausible learning algorithms.
9

---

References
347
348
[1] EH Adelson and JR Bergen. Spatiotemporal energy models for the perception of motion. Journal of the Optical Society of America. A, Optics and Image Science, 2(2):284-299)) 1985.
[2] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
351
353
354
[3] JosephJ J Atick and A Norman Redlich. What does the retina know about natural scenes? Neural computation, 4(2):196-210 1992.
355
356
[4] Horace B Barlow. Unsupervised learning, Neural computation, 1(3):295-311, 1989.
357
[5] Anthony J Bell and Terrence J Sejnowski. The "independent components" of natural scenes are edge filters. Vision research, 37(23)33327-3333 1997.
358
[6] Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in neural information processing systems, 31, 2018.
360
361
[7] Adam Coates and Andrew Ng. Selecting receptive fields in deep networks. Advances in neural information processing systems, 24, 2011.
[8] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223 JMLR Workshop and Conference Proceedings, 2011.
364
99 Adam Coates and Andrew Y Ng. Learning feature representations with k-means. In Neural networks: Tricks of the trade, pages 561-580. Springer, 2012.
367
368
[10] D. Eigen. Predicting images using convolutional networks: Visual scene understanding with pixel maps, 2015.
369
370 K. Fukushima. Neocognitron: a self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern, 66(4):193-203 1980.
371
372
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
373
374
375
[13] Geoffrey E Hinton, Michael Revow, and Peter Dayan. Recognizing handwritten digits using mixtures of linear models. Advances in neural information processing systems, pages 1015- 1022, 1995.
376
[14] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313 2006.
379
380
[15] Haruo Hosoya and Aapo Hyvärinen. Learning visual spatial pooling by strong pea dimension reduction. Neural computation, 28(7):1249-1266) 2016.
382
[16] Aapo Hyvärinen and Patrik Hoyer. Emergence of phase-and shift-invariant features by decom- position of natural images into independent feature subspaces. Neural computation, 12(7):1705- 1720, 2000.
383
384
385
[17] Aapo Hyvärinen and Urs Köster. Fastisa: fast fixed-point algorithm for independent subspace analysis. In ESANN, pages 371-376. Citeseer, 2006.
386
387
[18] Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsuper- vised image classification and segmentation. Proceedings of the IEEE/CV International Conference on Computer Vision, pages 9865-9874, 2019.
388
389
[19] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint 1rX1v:1611.05148. 2016.
381
10

---

393
[20]
Yan Karklin and Michael S Lewicki. Emergence of complex cell properties by learning to generalize in natural scenes. Nature, 457(7225):83- 86, 2009.
394
395
[21] Koray Kavukeuoglu, Pierre Sermanet, Y-Lan Boureau, Karol Gregor, Michaël Mathieu, Yann Cun, et al. Learning convolutional feature hierarchies for visual recognition. Advances in neural information processing systems, 23, 2010.
396
397
398
[22] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.61 14, 2013.
389
400
[23] Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoen- coders. Advances in neural information processing systems, 32, 2019.
401
402
[24] Dmitry Krotov and John J Hopfield. Unsupervised learning by competing hidden units. Pro- ceedings of the National Academy of Sciences, 116(16):7723-7733 2019.
403
404
[25] Quoe V Le, Will Y Zou, Serena Y Yeung, and Andrew Y Ng. Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In CVPR 2011, pages 3361-3368 IEEE, 2011.
405
406
407
[26]
Quoe V Lee Will Y Zou, Serena Y Yeung, and Andrew Y Ng. Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In CVPR 2011, pages 3361-3368 IEEE, 2011.
408
409
410
[27] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmistl/ 1998.
411
412
[28] Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix
413
factorization. Nature, 401(6755):788-791 1999.
414
[29] Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th annual international conference on machine learning, pages 609-616, 2009.
415
416
417
[30] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support error backpropagation for deep learning. Nature communications, 1-1):1-10, 2016.
418
[31] Timothy P Lillicrap. Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6)3335-346, 2020.
421
422
[32] Kyle Luther and H Sebastian Seung. Sensitivity of sparse codes to image distortions. arXiv preprint arXiv:2204.07466, 2022.
424
[33]
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv: 1312,5633 2013.
425
426
[34] Ryan McConville, Raul Cantos-Rodriguue Robert J Piechocki. and Ian Craddock. N2o:(not too) deep clustering via clustering the local manifold of an autoencoded embedding. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 5145-5152. IEEE, 2021.
428
429
[35] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprini 1802,00400 2018.
[36] Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. arXiv preprint (804.00804.002) 2018.
432
433
[37] Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
434
435
[38] Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. Clustergan: Latent space clustering in generative adversarial networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4610-4617, 2019.
438

---

440
[39]
Dina Obeid, Hugo Ramambason, and Cengiz Pehlevan. Structured and deep similarity matching via structured and deep hebbian networks. Advances in neural information processing systems, 32, 2019.
441
[40] Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 (66833))6077666 1996.
444
445
[41] Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. arXiv preprint 1rXiv:1710.09822 2017.
[42] Ko Sakai and Shigeru Tanaka. Spatial pooling in the second-order spatial structure of cortical complex cells. Vision Research, (0(7))855-871 2000.
448
449
[43] Jeremias Sulam, Vardan Papyan, Yaniv Romano, and Michael Elad. Multilayer convolutional sparse modeling: Pursuit and dictionary learning. IEEE Transactions on Signal Processing, 66(15) 66(15):4090-4100 2018.
450
451
452
[44] Louis Thiry, Michael Arbel, Eugene Belilovsky, and Edouard Oyallon. The unreasonable effec- tiveness of patches in deep convolutional kernels methods. arXiv preprint arX1v:2101.00528, 2021.
453
454
455
[45]
René Vidal. Subspace clustering. IEEE Signal Processing Magazine, 28(2):52-66) 02055226666666 2011.
456
[46] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
457
[47] Dingding Wang. Chris Ding, and Tao Li. K-subspace clustering. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 506-521. Springer, 2009.
480
[48] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering 461
analysis. In International conference machine learning. pages 478 487. PMLR, 2016.
[49] Yi Yang, Dong Xu, Feiping Nie, Shuicheng Yan, and Yueting Zhuang. Image clustering using local discriminant models and global integration. IEEE Transactions on Image Processing, 19(10)2761-2773, 2010.
464
465
Checklist
466
467
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes] In the abstract we claim 2% clustering error with a uecuee--nsuperviiedd method, and we show this in Table 2, and provide the method in section 3,4,5
469
471
472
(b) Did you describe the limitations of your work? [Yes] See discussion
473
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
475
476
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In the ap- pendix, we state our assumptions for our theoretical result (that our power updates for optimizing the K-Subspace objective converges in the full batch setting)
478
480
(b) Did you include complete proofs of all theoretical results? [Yes]] In the appendix, we state prove our theoretical result that the algorithm for optimizing the K- K-Subspace objective converges in the full batch setting
481
482
483
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.)? [Yes] We will include code in supplemental material
485
12

---

487
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In section we provide these details
488
489
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] Unfortunately we did not have adequate time to complete these experiments, as each meta-learning experiment required training and evaluating 1000s of different architectures
491
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In section 5 on hyperparameter tuning, state that each 3 layer meta-learning experiment takes approximately 20 hours on 8 GPUs
494
496
497
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite mnist set creators.
489
500
(b) Did you mention the license of the assets? [No]
501
(c) Did you include any new assets either in the supplemental material or as URL? [No]
502
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [No] Dataset only includes handwritten digits.
503
504
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] Dataset only includes handwritten digits.
505
506
5. If you used crowdsourcing or conducted research with human subjects...
507
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
510
511
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
A Online minimization of Eq. 1
513
Algorithm 1: K-Subspace clustering via minibatch power iterations
Initialize subspaces {V RRxxmp"
for i = 1, 3 do
Sample patches from minibatch of images (xn. E Rmpp') } Cluster patches
X = Xu k argmin
|xx - V Voxu|/ for k 1, 3,
q
Apply one orthogonal power iteration to every subspace
USV
x X
SVD decomposition
U
for = 1,2,3,...
end
514
standard algorithm for minimizing Eq. [ is a full batch EM algorithm that alternates between cluster assignment (E-step) and using PCA to set V to the top principle components of the patches assigned each cluster & (M-step) 45). The full batch requirement makes this algorithm rather slow.
515
516
517 The algorithm we present and use in this paper is described Algorithm [
We make two core changes to the standard EM algorithm. One, we use minibatch updates instead of full batch updates. Two, we perform single step of power-iteration after each cluster assignment step, instead of performing full PCA.
519
520
521
Proper initialization can impact on the quality of learned subspaces. We initialize subspaces by setting the first dimension to randomly chosen patch, and the other dimensions to white Gaussian
13

---

523
noise with = 0, = 0.01. We then perform a few "warmup" iterations of the main loop in alg. [ except we only cluster using the first subspace dimension. We use warmupp iititteerrrr 10 in all our experiments. During these warmup iterations, the power updates are still performed on the full rank-r subspaces. Intuitively, this warmup procedure generates clusters with 1D subspace clustering, and initializes subspaces be the top components within these clusters.
524
525
526
527
528
Convergence of our algorithm is discussed in the Appendix. We prove that the clustering+power iteration step ensures the loss computed over the minibatch is non-decreasing. We show empirically that the loss computed over the whole dataset decreases with iteration for reasonable setting of parameters. Because we only use a single pass through the data to train each layer, our algorithm is much faster than applying the full batch EM-style algorithm described in [45] the whole dataset.
529
530
532
533
B
Convergence analysis of Algorithm
534
B.1
Theory: full-batch convergence
535
We will not be able to provide a full proof that Algorithm converges in the minibatch setting. However we can at least show that in the full batch setting, every iteration of Algorithm decreases the energy in Equation T Of course the cluster assignment portion of the algorithm decreases the energy. What we will show here is that the power iteration step also decreases the energy at every iteration.
536
537
539
540
We recall the notation from Algorithm 1. We define the matrix X whose rows are the input patches assigned cluster k:
541
X = (x h argmin Xx,
V Vgxu
542
The energy in Equation I can be written sum of energies for each cluster
= EE en where:
E X X V
(3)
543
We will show that a power update for cluster k now gives a non-decreasing energy eg. To avoid notational clutter, will drop the subspace index k and assume we are working with a single fixed
544
545
cluster for now. It will actually be casier to show that more general class of updates than the SVD update in Algorithm cause the energy to remain or decrease. Suppose we have any subspace update defined by:
548
547
V = Q(VCV)'CV
(4)
where Q is any orthogonal r matrix (that can also be a function of V, C. The power update 549
548
in Algorithm I is an example of such an update. We will show that the energy for every subspace non-decreasing with this update: e' S e. We do so by relating the update in Equation (4) to one sequence of steps in an alternating least squares problem. Define:
550
h(A,
X- T2
(5)
Define A = XVT and B = VT. Then e = |((A, B). Define the sequence:
552
B' argmii h(A, U) XA(ATA)
(6)
A' argmin A(U UU BB) XB (BB((B""B))
U
We multiply A and substitute the above equations to get:
A X(V')TV'
554 We therefore have that é' = M(AA, B') < ( = e, so the subspace energy is non-increasing under the updates in Algorithm 1.
B.2 Experiment: learning curves
557 We show learning curves using Algorithm 1 applied to MNIST digits in Figure 5 We run this algorithm for a single pass through the 60k training set patterns. Inputs are first whitened with
14

---

learning curves for a single layer applied to 60K MNIST digits
54
train set energy
52
test set energy
50
48
energy
46
44
42
40
50
100
150
200
iteration
Figure 5: Learning algorithms for Algorithm applied MNIST digits.
leamed ZEA kersel enerpy net
Figure 6: Learned 7CA kernel for 3 energy layer net. We have zero-ed out the central pixel, its value is 1.17 and thus would make it harder to the surround structure. This filter resembles oblique center surround filter.
559
ConvZCA using a kernel size of 11 and num component of 16. We learn 64 subspaces, each of dimension 5, and kernel size of 9. We use minibatch size of 256.
The training energy is computed over a minibatch of 256 inputs each iteration. The energy is computed over all 10K test set patterns, which is why it is less noisy. Empirically we that for this setting of parameters at least, our minibatch K-Subspace algorithm does lead decreasing energy computed over unseen patterns.
582
583
584
We apply 10 warmup iterations (only using the first subspace dimension to cluster for the first 10 iterations), which is why we observe the sharp drop-off energy after 10 iterations.
566
C Hyperparameters for 1,2,3,4 energy layer nets
567
In Table 6 we show the learned architectures for the 1,2,3,4 energy layer networks.
15

---

Table 6: Network architectures for 1,2,3,4 energy layer nets from Table 5.
HYPERPARAMETER
LAYER
LAYER
3 LAYER
4 LAYER
CONVZCA KERNEL SIZE
9
9
11
CONVZCA N COMPONENTS
18
LAYER SUBSPACE NUMBER (k)
37
LAYER SUBSPACE RANK (r)
5
2
LAYER] ACTIVE FEATURES (w)
16
9
10
LAYER KERNEL SIZE
10
10
8
LAYER PADDING
2
2
LAYER2 SUBSPACE NUMBER (k)
55
63
LAYER2 SUBSPACE RANK (r)
16
3
12
LAYER2 ACTIVE FEATURES (w)
1
8
1
LAYER2 KERNEL SIZE
19
5
17
LAYER2 PADDING
1
LAYER3 SUBSPACE NUMBER (k)
58
LAYER3 SUBSPACE RANK (r)
16
LAYER3 ACTIVE FEATURES (w)
2
6
LAYER3 KERNEL SIZE
21
8
LAYER3 PADDING
2
LAYER4 SUBSPACE
(k:
22
LAYER4 SUBSPACE RANK (r)
LAYER4 ACTIVE FEATURES (w)
LAYER4 KERNEL SIZE
LAYER4 PADDING
569 D Learned ZCA filter
570 In Figure 6 we show the Icarned ConvZCA kernel for the 3 energy layer network. It resembles an oblique center surround filter. It interesting calculate the relative weight of the negative surround 572 vs positive center term. Specifically we calculate:
[[4, 4] En. max -u 0.10
[[4,4 4]
where 7 is the 9x9 kernel and 744, 4] is the central pixel of that kernel. In other words, there is small DC component (the central pixel is not perfectly cancelled out by the negative surround).
574
E
Clustering UMAP embeddings
576
In the main text we reported a surprisingly performance for K-Means applied to UMAP em- bedding, and that using a Gaussian mixture model instead can significantly improve the accuracy. Here we show the UMAP embeddings, and corresponding clusterings generated via K-Means and Gaussian Mixture Models.
577
578
580
Note that result is inconsistent with the result given in Table of 34] who reported 82.5% clustering accuracy (compared to our result of 96.4% in Table 5) when using a GMM to cluster the
581
582
embedding vectors return by UMAP. This because we are using UMAP to embed 2D while they used UMAP to embed to 10 dimennsions. For this task it seems that the extremely low dimensional embeddings are actually more suitable for downstream elustering.
16

---

K-Means Clustering
Gaussian Mixture Modeling
Figure 7: K-Means vs. Gaussian Mixture-based clustering applied to UMAP with "out-of-the-bov box" parameters
applied to MNIST handwritten digits. Colors are assigned to cach point based the clustering (not the ground truth labels). K-Means erroneously merges and splits some of the clusters, while Gaussian Mixture models give much more intuitive clustering (and ultimately much lower clustering
shown in Table 5)
17


Reviews:
[{"review_id": "rpZ4GmTCb86", "reviewer": "Reviewer_x8sP", "summary": "This paper addresses the issue of how to design unsupervised learning(UL) architecture and learning without backpropagation that matches the performance of UL with backpropagation. It proposes an UL architecture consisting of a whitening layer, multiple convolutional energy layers and a final average pooling layer. The  convolutional kernels are learned with K-Subspace algorithm, and the hyperparameters are chosen by supervised meta learning to optimize the clustering accuracy. Experiments on MNIST dataset demonstrates that the proposed method achieves comparable accuracy as UL with backpropagation.", "questions": "The experiments are limited. It evaluates only on MNIST which is an easy dataset. Experiments on more complex datasets/tasks will help to support the claims.\n", "limitations": "It discussed the limitations of the work, but did not discuss potential negative societal impact of the work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The inner loop of the proposed algorithm has analogous to brain, and can train the model without backpropagation.\n\n+ The authors conducted extensive ablation study indicating the contribution of each proposed components.\n\n+ The related work section has a nice flow and puts the proposed method into context.\n\nWeaknesses:\n- Winner-take-all of K-subspace learning does not guarantee the learned convolutional kernels are optimal.\n\n- Searching for optimal hyperparamters requires supervised learning, which makes the proposed method not purely unsupervised.\n\n- The experiments are limited. It evaluates only on MNIST which is an easy dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bIKUsQY6_3", "reviewer": "Reviewer_CyHx", "summary": "In this manuscript the authors create an architecture and learning scheme to learn the clustering of MNIST without backpropagation. They combine energy layers, which compute responses as vector lengths in subspaces, with adaptive thresholding that introduces a form of competition between the output features. These layers are then learned consecutively to optimize K-Subspaces clustering, i.e. to minimize the distance of samples from their closest subspace. For this procedure the authors give a new mini-batched algorithm. After applying a supervised architecture search to this approach the authors find an architecture that reaches 2% error rate on MNIST.", "questions": "1) Could you please add some discussion on what the conclusions for future work on unsupervised learning are? Solving MNIST is not really an aim in itself.\n2) What is the authors opinion on biological plausibility here?  \n3) And how is the interpretation of the supervised meta-learning on top of unsupervised learning results different from unsupervised learning only?\n", "limitations": "---", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Constructing this learning system is definitely an engineering feat, but it is unclear to me what the novel insight we are supposed to gain from this manuscript. From a cautionary tale like “even fitting only the architecture we can achieve much better performance with somewhat unexpected parameters” to “this is a great architecture that may generalize to much harder unsupervised learning tasks” many interpretations would be possible and the authors do not comment on this and I don’t think their data are sufficiently clear on this on their own. Nonetheless, the authors are thorough. For example, they investigate the convergence of their batch algorithm, find a working clustering for the UMAP embedding and check the necessity of the parts of their model.\n\nConcrete points:\n1) It remains a bit unclear what the authors relationship to biological plausibility is here. They emphasize that stacked unsupervised learning is more plausible than back propagation, but other aspects like the subspace clustering for example are not questioned regarding back propagation at all.\n2) Somewhat surprisingly, the authors find that 4 layers work less well than 3 layers on MNIST. While choosing 3 layers then is consistent with the architecture search theme, this seems odd. Is there an explanation why deeper features are no longer better?\n3) While I tend to agree that supervised learning of the architecture for a task does not necessarily break biological plausibility, I don’t think all advantages of unsupervised learning still apply. Practically for example, the method still requires that supervised data is available for meta-learning. I think some discussion of this is necessary.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ULTEWHgQHBk", "reviewer": "Reviewer_WR6r", "summary": "This paper designed a stacked unsupervised learning (SUL) architecture to learn visual classification task without backpropagation. The proposed SUL has two loops. The inner loop is unsupervised learning, which trained the kernel of each layer iteratively, and thus no backpropagation was required. The outer loop is supervised learning to optimize the hyperparameters of the SUL. The proposed SUL was tested on Mnist dataset and achieved a good result compared to other unsupervised learning methods.", "questions": "1. Did other unsupervised learning methods also fine-tune their hyper parameters? Did other methods also require the labels of the images? If not, it seems unfair to compare the performance of the current method to traditional unsupervised learning method. It may be better to compare the current SUL to some few-shot learning or semi-supervised methods.\n2. Why does the automated tuning of hyperparameters affect the accuracy significantly?\n3. Will SUL also work on other more complex datasets?\n4. What's the theoretic contribution of the designed SUL?\n5. Besides biologically plausible, are there any other advantages for SUL compared to backpropagation? Will it achieve higher accuracy than backpropagation? Will it be more efficient to train? Will it explain some biological problem better? Or will it be more feasible on some special applications?\n6. As mentioned in the paper, the meta-learning is more time-consuming on CIFAR-10 dataset. Considering that the training process still requires labels. What is the advantage of SUL compared to supervised learning or few-shot learning?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper illustrated that SUL can still work on a visual classification task without backpropagation.\n2. This paper achieved a state-of-art performance on an unsupervised learning task. \n\nWeaks:\n1. All items, including ZCA, energy layers, K-subspaces, and adaptive thresholding, were already proposed in the 2000s. This paper only introduced automated tuning of hyper-parameters, which seems just a marginal experimental contribution.\n2. Because the proposed method still requires many labels (~5000) for meta-learning, it seems unfair to compare the current method with the previous unsupervised learning method. \n3. This paper lacks a theoretical explanation about why the current SUL works but SULs back in the 2000s did not work well. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject