Wider Networks Learn Better Features
Authors: Dar Gilboa
Guy Gur-Ari
Venue:  (2020)
Keywords: Interpretability
transfer learning

Abstract: Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases --- a visualization technique that captures features the entire hidden state responds to, as opposed to individual neurons alone. We find that, while individual neurons do not learn interpretable features in wide networks, groups of neurons do. In addition, the hidden state of a wide network contains more information about the inputs than that of a narrow network trained to the same test accuracy. Inspired by this observation, we show that when fine-tuning the last layer of a network on a new task, performance improves significantly as the width of the network is increased, even though test accuracy on the original task is independent of width. 

Under review as a conference paper at ICLR 2020
WIDER NETWORKS LEARN BETTER FEATURES
Anonymous authors
Paper under double-blind review
ABSTRACT
Transferability of learned features between tasks massively reduce the cost of training a neural network on novel task. We investigate the effect of network width on learned features using activation atlases that captures features the entire hidden state responds to, as opposed to individual neurons alone. We find that, while individual neurons do not learn interpretable features in wide networks, groups of neurons do. In addition, the hidden state wide network contains more information about the inputs than that of a nar- row network trained to the same test accuracy. Inspired by this observation, we show that when fine-tuming the last layer of a network on a new task, performance improves significantly as the width of the network is increased, even though test accuracy on the original task is independent of width.
visualization technique
INTRODUCTION
While the process of feature learning by deep neural networks is still poorly understood, numerous techniques have been developed for visualizing these features in trained networks in an attempt to better understand the learning process (Erhan et al. (2009); Simonyan et al. (2013); Nguyen et al. (2033) ; Mordvintsev et al. (015)) This is usually achieved by optimizing over the input space in order maximize the activation of single neuron or filter, with appropriate regularization. These techniques are commonly applied to pre-trained state-of-the-arl models, yet they can also be used to explore the effects of various architectural choices on the learned representations. In this work, specifically investigate the effeet of the number of neurons on the quality of the learned representations. It has been shown that increasing the width a neural network will generally not affect performance on a given task and it natural to study the effect of this modification on the learned representations.
improve it (Neyshabur et al. (017); Geiger et al. (2019),
We the recently developed activation atlases technique (Carter et al. (2019) in order to visualize the inputs that activate an entire hidden state, as opposed to a single neuron. the case a wide network, find that the resulting visualizations differ dramatically.
Our main findings are:
As we increase the number of neurons in network, the features that an individual neuron responds to appear less like natural images. However, there exist directions in the hidden state space that are activated by natural-looking images.
Additionally, the hidden states of wide network contain interpretable information about the inputs that appears not to be present in narrow networks trained to the same test accuracy.
Fine-tum tine-tunig& linear classifier on the learned features a wide network on a novel task leads substantial improvement performance, compared fine-tuning on the features of narrow network. This is true even when the test accuracy on the original task is the same for the wide and narrow networks. Presumably, this is result of the additional information encoded in the hidden states of wide networks.
The last observation suggests that networks designed for transfer learning tasks may benefit from increasing the width, even though this change may have little effect on performance on the original task.

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
The features that activate individual neurons have been the subject of study for a number of years, with early work focusing MNIST (Erhan et al. (2009)) well
complex image datasets (Simonyan et al. (2013)) The visualization procedure is sensitive to the choice of regularization, and numerous approaches have been explored including regularization using adversarial examples (Szegedy et al. (2013), robustness to transformations (Mordvintsev et al. (2015)) and using genera- tive models to parameterize the images (Nguyen al. [2016][017)), contrast to previous works, activation atlases (Carter et al. (2019) visualize features learned by the entire hidden state. Since these do not foeus on single neurons, they are particularly well-suited to studying wide networks
where individual neurons may not learn meaningful features.
There is also extensive literature on measuring the responses of individual biological neurons in number of organisms and brain areas in order to better understand the structure and computation performed in in the input pathways of the brain (Gross et al. (T969); Mohler et al. (1973); Theunissen
al. 2000)).
Thee transferability of learned features between tasks is a well-studied phenomenon (Long et al. 2013); Zoph et al. (2018)). Recently, it was also shown that the performance of the widely used MAML algorithm for transfer learning (Finn et al.] (2017) ) can be matched on many benchmark tasks simply by fine-tuning the last layer on the new task (Raghu et al. (2019)), which is exactly the fine-tuning setting used in the present work. This suggests that the success of MAML is in learning re-usable features opposed to converging points parameter space that are particularly well- suited to re- ee-training on novel task.
3 ACTIVATION ATLASES
We briefly summarize the visualization technique introduced in Carter et al. (2019). Given a set of inputs (f1,-...,4)) network with activations ('(x)) at layer and input and grid size g, the technique involves the following steps.
1. Collecting a set of activations
((.,
2. Reducing the dimensionality of the activations to using UMAP Mcinnes al. (0018).
3. Defining 2D grid of size 9 g. and averaging the activation vectors that fall in given grid cell after dimensionality reduction. The averaged activations are then whitened, giving a set of vectors ( P.ch.
4. For each averaged activation, solving the optimization problem
= arg min [g(' (1).
where g(x, y) distance function. Following Carter al. (2019), we use g(x; y) (x. x max 0..1, L(x, 2. The optimization problem is solved iteratively using ADAM (Kingma & Ba (2014)). At every iteration, series of transformations are applied to x that expects the network map to be invariant under (such jitters, rescalings and rotations). This provides a form of implicit regularization during training that improves the visualization results.
5. The resulting images (- (.i, ztae are then plotted in their respective locations on the grid, giving the activation atlas for layer 7.
Additional details are provided in Appendix B.1
4 FEATURE VISUALIZATION AND NETWORK WIDTH
In order to visualize the features learned by networks of different widths, we train convolutional networks with different numbers of neurons in the penultimate fully-connected layer, which
When applied convolutional layers, a spatial location is also chosen random for each input.
2

---

Under review as a conference paper at ICLR 2020
denote by n. We then generate activation atlases at this layer, reviewed in Section The generated images in the atlas are ones which activate the hidden state in a given direction (corresponding to the averaged activations in the given grid cell). As such, they provide indication of the types of inputs that the network responds to, and the type of structure in the input that it is sensitive to.
We additionally visualize the features that individual neurons respond to by applying the same op- timization procedure where the averaged activations P. are replaced by uniformly sampled one-hot vectors. As we will see, there are cases where the hidden state as whole is activated by interpretable images even though individual neurons are not. Additional experimental details are presented Ap- pendix B.2
4.1 MNIST
The results on the MNIST dataset for a narrow network with n = 20 and for a wide network with 2048 are shown in Figure Both networks reach 99.0% test accuracy, and can be in the UMAP output the classes cluster well in both cases. However, in the hidden state of the wide network we see additional fine structure that not present in the narrow network, which can be observed in the atlas images. In the case of the digit 4 for instance, there is sub-cluster of activations (corresponding to direction in the hidden state space) that are sensitive to "closed" 4s that contain a triangle. Similarly, there are sub-clusters for 2s with a closed loop and 7s with crossing line. None of this structure seems to captured by the hidden state of the narrow network.
Individual neurons in the narrow network respond to images that resemble digits more than those of the wide network. Based on works such as Frankle & Carbin (2018), one may expect that a small subset of neurons in the wide network should still respond to natural images in the same way as the neurons in the narrow network, but this does not appear to be the case. Rather, it appears that the hidden state learns representations that distributed across all neurons, and as the number of neurons increases the relative amount of information encoded by any individual neuron reduced.
One may wonder whether these results obtained from the MNIST dataset, which is close to being linearly separable in input space, are fact trivial. To check this, we generated atlases by using the original images as the activations. The results, presented in Appendix [A.2] show that the images are not as clearly interpretable as those generated from the atlases on the trained networks (and as expected, the clustering is not as good).
4.2 TRANSLATED MNIST
The difference between the information in the hidden states of narrow and wide networks is easier to discern in datasets with additional fine grained structure compared to MNIST. We construct such a dataset, which we call translated MNIST, by selecting subset of training images from MNIST and horizontally translating each image cyclically in all possible 28 ways. The size of the training set remains the same, and the test set is left unchanged. We then generated the atlases for this dataset using the same network architecture and training procedure as in the previous subsection. The final test accuracies were 96.0% and 96.8% for widths
20 and 2048, respectively.
As can be seen in Figure 2 the additional structure induced by the translations is clearly present in the hidden state of the wide network in the form of the annular structures in the UMAP output. The location of activation vectors in an annulus for a given class are seen to correspond to the degree of translation of the image. In contrast, the hidden state of the narrow network does not clearly encode the degree of translation. Since the translation information was present in the inputs, one may wonder whether this gap between the wide and narrow network is simply result of a bottleneck in the narrow network. As we show in Section however, the features learned by the wide network are more useful than either the features of the narrow network or the inputs themselves when fine-tuning a linear classifier new task.
5
4.3 CIFAR-10
Similar phenomena to those described above are also observed on the CIFAR-10 dataset. We again train convolutional networks of the same architecture, this time varying not only the width of the final layer but of all intermediate layers as well by scaling the number of filters in the convolutional
3

---

Under review as a conference paper at ICLR 2020
000
(a) Width
(b) Width 2048
Figure 1: Activation atlases for the penultimate fully-connected layer of narrow and wide convo- lutional networks trained MNIST. Top: 2D histograms of the output of UMAP applied the activation vectors, with color corresponding to class label. While both networks cluster the inputs according class as expected, there are additional sub-clusters in wide network activations. Mid- dle: Activation atlases, indicating the representations learned by the networks. The sub-clusters observed in the top plot for wide networks can be seen to correspond digits with distinct features. Bottom: The features that randomly selected individual neurons respond to. For the narrow network, the individual neuron and the activation atlas features appear to be equally interpretable, while for the wide network the individual neuron features are not interpretable.
layers linearly with n. The final test accuracies for the networks width n = 64 and n 2048 were 83% and 82% respectively.
The results are presented in Figure where there again appears to be more information in the hidden state of the wide network than the narrow one. Similar the fine structure observed in the atlases for MNIST and translated MNIST (which enabled one to distinguish the degree of translation or style of digit), here we can distinguish between automobiles of different color in the lower left section of the atlas (and not in the case of the narrow network). On the right section of the atlas, we see that left- and right-facing horses become separated. before, we also observe that individual neurons learn less interpretable features in the wide network.
4

---

Under review as a conference paper at ICLR 2020
66
0
Width 20
(b) Width 2048
Figure 2: Activation atlases for the penultimate fully-connected layer convolutional networks of different width trained translated MNIST, Top: 2D histograms of the output of UMAP applied to the activation vectors, with color corresponding to class label. Compared to MNIST (Figure [T], the additional structure due the translations is clearly visible in the wide network but not in the narrow one. Middle: Activation atlases. For the wide network, the location of the activations in an annulus for given class indicative of the degree of translation of the image. Bottom: The features that randomly selected individual neurons respond to. As in the case of MNIST, these are more interpretable for the narrow network.
5 USING LEARNED REPRESENTATIONS FOR NOVEL TASKS
The additional structure present in the hidden states of wide networks, seen in Figures I 2 is not necessary perform well the original classification task. This is attested by the fact that the performance of the two networks on the original task was nearly identical. However, the presence of the additional structure suggests that wide networks might perform better on novel tasks that rely on the additional information. We test this conjecture by training network of depth L on one task, obtaining a trained network
Soniy. (x) Wolg - (x),
and then fine-tuning a linear classifier that takes as input the features of the original network,
funed (x) W.. Wiine
sT. orig. (z)

---

Under review as a conference paper at ICLR 2020
arplane
applani
dear dog [rog
7002
Width
(b) Width 2048
Figure 3: Activation atlases for the penultimate fully-connected layer convolutional networks of different width trained CIFAR-IO. Top: 2D histograms of the output of UMAP applied to the activation vectors, with color corresponding to class label. Middle: Activation atlases. For the wide network, there is a direction large variance in the activation space corresponding color in the bottom left, and to left/right orientation of horses on the right. Bottom: The features that randomly selected individual neurons respond to. As before, these are more interpretable for the narrow network.
automobile
In training (x), all the weights in f frozen. The output dimension also change in the fine-tuned network. Additional experimental details presented in Appendix B.3
5.1 TRANSLATED MNIST DIGIT AND TRANSLATION CLASSIFICATION
We first train fully connected networks with varying width and depth to predict the degree of trans- lation of an image in the translated MNIST dataset, described Section 4.2 We refer to this as the original task. The details of the network architecture and training procedure Appendix B.2 As we reduce the width, we also increase the depth so that the total number of model parameters approximately constant. The widest network in experiment has 1 hidden layer with 2000 neurons, and the narrowest has 20 hidden layers with width 250. We observe that test accuracy after training the original task (translation classification) is essentially independent of width. These results are shown in the left panel of Figure
specified in

---

Under review as a conference paper at ICLR 2020
Translated MNIST
Depth
10
CIFAR-100
0.9
007
Test accuracy
Test accuracy
0.7
0.80
0.5
Test
0.3
0.75
0.6
0.1
250
500
1000
1500
2000
100
500
1000
1500
2000
Width
Width
Original task
Fine-tuned on new task
Linear classifler on new task
Figure 4: Wide networks learn features that are useful on novel tasks. We train networks with different numbers of features at the penultimate layer on one task, then fine-tune the last layer on a novel task. Performance on the original task is essentially independent of width, but performance of the fine-tuned classifier improves dramatically with increased width. The results are the mean and standard deviation over initializations. Left: Translated MNIST (described in Section (422). The original task is shift classification, and the new task is digit classification. The networks are fully connected and the total number of parameters held approximately constant (by making the narrow networks deeper). Right: 3 coarse class CIFAR- 100, convolutional networks. The original task is image classification by coarse classes (superclasses) of CIFAR-I00. The new task is classification by the fine classes of the same images. A linear classifier trained directly on the new task achieves test accuracy 37.5%.
Next, we fine-tune the last layer of the network to classify the digits (as in standard training on MNIST). The number of classes is reduced from 28 to 10. We refer to this as the new task. We find that performance improves dramatically with increased width, while both the test accuracy on the original task and the number of parameters are approximately constant (see the orange curve in the left panel of Figure
Some of this effect may be due to the fact that the narrower networks have a bottleneck. Indeed, we that performance on the new task increases rapidly until the width reaches the input dimension w. = 784, and beyond that point the improvements are more gradual. The performance achieved at width w. is also the performance of a linear classifier trained the task (the dashed line in the left panel of Figure 4. The fact that wider networks achieve better performance on this task suggests that they are learning useful features
5.2 10FAR-100: COARSE AND FINE LABEL CLASSIFICATION
We perform similar experiment on the CIFAR-100 dataset. Each image in the CIFAR-100 dataset has both a fine and a coarse label. There are 20 coarse labels and 100 fine sub-labels, where each coarse label is associated with 5 fine sub-labels (the aquatic mammal coarse class, for example, contains images in the fine classes beaver, dolphin, otter, seal, and whale). We use images in the first 3 coarse classes only, giving a training set of 500 images, and test set of 1, 500 images. We train convolutional networks with wide or narrow penultimate hidden layers to classify images according to the coarse label the original task in the right panel of Figure We then freeze these networks, and fine-tune the last layer to classify images according to the fine labels - this is the new task in the right panel of Figure 4 We record the best test accuracy achieved during training by the fine-tuned classifier.
As the case of fully-comnected networks on the translated MNIST task. we find that the test accuracy on the original task essentially does not depend on the number of neurons for the range of widths considered We again observe that the test accuracy of the fine-tuned classifier increases
We expect that digit classification on the translated MNIST dataset is harder than the original MNIST task. It is therefore not surprising that the lincar classifier does worse on this task than on original MNIST, where 90% test accuracy is achievable.
'It is possible that by tuning hyperparameters such as learning rate one can improve the performance of the wider networks on the original task [Goiger ct al. (2019)). We have not explored this possibility.

---

Under review as a conference paper at ICLR 2020
with width. A linear classifier trained directly on the inputs achieves test accuracy of only 37.5% on the new task.
6 DISCUSSION
Visualization of the inputs that activate the entire hidden state of a network, as opposed to individual neurons, suggests that wide networks contain more information in their hidden state than narrow networks, even if the two networks are trained to the same test accuracy. This is true even when controlling for the total number of parameters. This observation, result of generating activation at- lases (Catter et al. (2019) for networks of varying widths, suggests that wide networks may perform better than narrow ones when trained on novel tasks. We indeed observe this effect when fine-tuning the last layer of network on new labels. Considering our findings in the light of the recent results of Raghu et al. (2019), which suggest that feature re-use the dominant factor explaining the success of MAMI. (Finn et al. (2017)) it will be interesting to explore the effect of network width on the performance of this and other transfer learning algorithms.
We find that the activation vectors of wide networks are not sparse, and the visualization of features learned by individual neurons indicate that there isn't a small subset of neurons that respond to natural images. These results suggest that information is stored in a distributed fashion in the hidden state of a network. This may be surprising given recent work indicating that network performance is only slightly degraded after zeroing out most of the weights (Frankle & Carbin (218)).
Our empirical observation that there is information in the hidden state that appears to not be useful for classification (such as the degree of translation in translated MNIST, see the middle right panel in Figure 2 also appears to be at odds with the view that neural networks perform compression in later stages of training. According to this picture, training reduces the mutual information between the hidden state and the inputs while maintaining the mutual information between the hidden state and the labels (Schwartz-Ziv & Tishby (2017). Conversely, the picture that emerges from our experiments is that when the network has the capacity to do so, it can transform the data manifolds in order to enable classification, without destroying additional structure those manifolds that is not required for classification. Furthermore, it appears this additional capacity is more closely tied to larger width than to larger depth. It may still be the case that a bottleneck induced in the network (say by introducing an intermediate narrow layer) such information will be lost.
Finally, we would like to discuss our observations in the context of very wide neural networks, which have been studied for example in Neal (1996); Daniely et al.] (2016); [ee et al.] (2018); de G. Matthews et al. 2018). In the infinite width limit, when the network is properly parame- terized, activation vectors change by negligible amount compared with their initial values (Dul cct al. (2018b); Li & Liang 2018); Zou et al. (2018)) This is sometimes referred to as lazy training (Chizat et al. 2018)). In this regime, expect that activation atlases will not produce interpretable results, because the activations are almost the same as those at initialization. This raises the question of whether very wide networks learn interpretable features at all, and if so, how are these features encoded in the hidden state. Answering these questions in full is beyond the scope of this work, but we will make a few comments that we hope may shed light on this question.
Consider a network function f(x) = p=- E R²* where fx- E IR" is the activation in the penultimate layer, W E R are the weights of the linear classifier at the head of the network, and k is the number of classes. As we take the width n large, and with appropriate parameterization, the learned activations fz-)(x) will remain close to their initial values as measured (for example) by L2 distance. However, note that the classifier only makes use of a a-kkdii k-dimensional subspace of the hidden space R", spanned by the rows of WL. While the whole activation vector fl-1(a (x) does not move by much during training, the activation vector projected onto this smaller subspace can move by a significant amount, and it is only this projection that matters for the purpose of classification. Therefore, it is possible that learned, interpretable features are encoded in the hidden state projected to an appropriate subspace, even though the hidden state itself remains very close to its initial value. Correspondingly, it is possible that activation atlases can produce interpretable results when trained on these projected activations. Of course, the classifier subspace itself is updated during training, and so these learned features may in fact be encoded in larger subspace defined by the history of classifier subspaces seen during training. We leave a more careful treatment of this question to future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and Generalization in Overparameter ized Neural Networks, Going Beyond Two Layers. arXiv e-prints, art, arXiv:181..04118, Nov 8a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A Convergence Theory for Deep Learning via aee-PPrraeterizzaton.. arXiv e-prints, art. arXwv:1811.03962, Nov 2018b.
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah, Activation atlas. Distill, 4(3)=15,, 2019.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On Lazy Training in Differentiable Program- ming. arXiv e-prints, art.
Dec 2018.
Amit Daniely. Sgd learns the conjugate kernel class of the network. In Advances in Neural Infor- mation Processing Systems, pp. 2422-2430, 2017.
Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks:
The power of initialization and dual view expressivity. In Advances In Neural Information Processing Systems, pp. 2253-2261 2016.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahra- mani Gaussian process behaviour in wide deep neural networks. In International Confer- ence on Learning Representations, 2018. URL
net forum?id
W11-nGgWC-
Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai Gradient Descent Finds Global Minima of Deep Neural Networks. arXiv e-prints, art. arXiv:181 L.0804, Nov 2018a.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient Descent Provably Optimizes Over-parameetrrized Neural Networks. arXiv e-prints, art. 1 810.02054, Oct 2018b.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 13:1(3):1, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- learnie for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803vv1803.0333 03635, 2018.
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d' d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. arXiv preprint arXiv: /901.0/608, 2019.
Charles G Gross, David Bender, and Carlos Eduardo de MorhadMiranda. Visual receptive fields of neurons in inferotemporal cortex of the monkey. Science, 166(3910) 1969.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.660 2014.
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. In International Confer- ence on Learning Representations, 2018. URL http::://oppeeeviiiiie wttprrewwwwwweeeeeeeeeeeeeee
1LEA-M-O2 CZ
Yuanzhi Li and Yingyu Liang. Learning Overparameterized Neural Networks via Stochastic Gradi- ent Descent on Structured Data. arXiv e-prints, art. arXiv:1808.012: Aug 2018.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. arXiv preprint arXiv:1502.0270 2015.

---

Under review as a conference paper at ICLR 2020
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03422 2018.
Charles W Mohler, Michael E Goldberg, and Robert H Wurtz. Visual receptive fields of frontal eye field neurons. Brain Res, 61(3):85-3, 1973.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. 2015.
Radford M. Neal. Priors for Infinite Networks, pp. 29-53. Springer New York. New York, NY, 1996. ISBN 978-1-4612-0045-00 doi: 10. 1-4612-0745-0.2. URL htttps://doi. org 10. 00a0:000.0077777---162-2---33330
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. In Advances in Neural Information Processing Systems, pp. 5947-5956, 2017.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred imputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems, pp. 3387-3395, 2016.
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4467-4477, 2017.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse?
towards understanding the effectiveness of maml. arXiv preprint arXiv:1502.0277 2019.
Schwartz-Ziv and N Tishby. Opening the black box of deep neural networks via information. 2017. arXiv preprint arXiv:1703.00810, 2017.
Karen Simonyan. Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networss:: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6054 2013.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint (rXiv:1312.611 2013.
Frédéric Theunissen, Kamal Sen, and Allison Doupe. Spectral-temppral receptive fields of nonlinear auditory neurons obtained using natural sounds. Journal of Neuroscience, 20(6):2315- 2331, 2000.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6697-8710, 2018
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic Gradient Descent Optimizes Over-parameterrized Deep ReLU Networks. arXiv e-prints, art, arXiv: 1,00888,, Nov 2018.
10

---

Under review as a conference paper at ICLR 2020
A ADDITIONAL ACTIVATION ATLASES
A.1 UNTRAINED MODELS
Figure 5: Activation atlas for the penultimate fully- colnecconeetted layer of an untrained convolutional network. The atlas images suggest that the hidden states that are activated by images of a similar color adjacent each other, and this correlates with the class label to extent.
In Figure we show the activation atlas for an untrained convolutional network. The architecture is described in Appendix B.2 with the width of all layers set to 128 except the penultimate fully connected layer of width 512. Unsurprisingly, there no clustering in the UMAP output and the resulting atlas images are not interpretable. This indicated that obtaining interpretable atlas images a consequence of learning features from data.
A.2 ATLAS INPUT SPACE
Figure T shows the activation atlas generated for raw MNIST inputs. The averaged activations 7: in this case are simply whitened averages over images. While the inputs clearly cluster indicating that the data is nearly linearly separable, comparison Figure [ shows that the clusters not well separated as in the case trained networks, and the atlas images produced are less interpretable. In particular, sub-clusters corresponding to fine-grained features not visible.

---

Under review as a conference paper at ICLR 2020
Figure 6: Activation atlas for MNIST digits. The class clusters are not as well separated as those in trained network, and the atlas images are less interpretable.
B EXPERIMENTAL DETAILS
ACTIVATION ATLASES
The inputs used to generate the atlases were the entire training set for all datasets. The grid size was set to 15.
The code to generate the activation atlases was based https::/ com/ Eensorflow/Ing hwcccw/wlllllll The regularization applied at every iteration, the UMAP parameters and the optimization hyper-parameters were identical to those in the code. MNIST digits were padded with 0, while CIFAR- 10 images were padded with 1.
B.2 NETWORK ARCHITECTURES AND TRAINING DETAILS
The network architecture consists of two blocks of two convolutional layers each, with filter size 3 followed by max-poo max-pooling layer with pool size 2. These are followed by two fully-connected layers. The networks use ReLU nonlinearities and the weights drawn from Wt No 2) for all
me
12

---

Under review as a conference paper at ICLR 2020
but the last layer and Wij (0, ,,.)) while the biases initialized at O. Here, Rin is the fan-in dimension (suitably defined for the convolutional layers to include the filter size).
All networks were trained using cross-entropy loss with SGD with momentum (3 0.9), using batch size 128. The training set size was 50, 000 for MNIST and CIFAR-10, and 28 [50, 000 /8 for translated MNIST (see Section 5.T). The test set consisted of 10, 000 images for all datasets. Networks trained on MNIST and translated MNIST were trained for 10 epochs with learning rate of 0.01, while networks trained on CIFAR-10 10 were trained for 100 epochs with learning rate of 1/n, where is the width of the network. CIFAR-10 data also augmented with random shifts of up to 10% both horizontally and vertically, rotations of up to т / and horizontal flips.
B.3 FINE-TUNINA TUNING TASKS
B.3.1 TRANSLATED MNIST
The sean Figure 4 is of fully-comnected networks with to 20 hidden layers and width chosen such that the number of parameters is approximately constant. Denoting by n,(4,, no, #1,,1 T'max the width for the L hidden layer network, input dimension, output dimension and maximal width (set to 2000) respectively, we set (L according to
no - TL+1 + VIL nu + + 4(L (n + n + m)mx
n (L)
2(L -
The learning rate used for all widths was 0.01. This was obtained from a sean across 10" for linearly spaced values of between -1 and -3, since it led to the best performance across different depths.
B.3.2 CIFAR-100
The training set and test set consisted of images thee coarse classes aquatic mammals, fish and flowers. Networks were trained coarse label classification for 100 epochs. The learning rate used for all widths was 0.0129. This was obtained from a scan across 10" for 10 linearly spaced values of between -1 and -3, since led to the best performance across different depths average.
13


Reviews:
[{"review_id": "Hyl9rKAatB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates wider networks using a recent feature visualization technique named activation atlases. By analyzing what the hidden layers of wider networks respond to, the authors showed that wider networks learn more transferable features. However, I tend to reject this paper since it doesn’t show very compelling evidence through experiments.\n\n1. This paper does not present any novel methods, and so the experiments need to be very solid. But all the datasets and architectures used in this paper are quite simple from the view of deep learning. It is not clear whether the conclusions will still be valid for larger datasets or deeper networks. \n\n2. The most important observation of this paper is to find that wider networks can be easily transferred to a new task. But in Section 5.1, all layers except the last classification layer are fixed when fine-tuning the networks for the second task. It is so obvious that wider networks with fewer previous layers can perform better. In Section 5.2, the authors did not show the network details, and also it is not fair to compare the networks with a linear classifier. The authors should include more competitive baselines.\n\n3. I encourage the authors to show the training/validation curves to testify the data efficiency of the wider networks when training or fine-tuning on a new task.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkgkHmv5FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the effect of network width of the neural network and its ability to capture various intricate features of the data. In particular, the central claim of this paper is what the title claims \"Wider networks learn features that are better\". They make this claim using the visualization technique called \"activation atlasses\". They find that wider networks learn features in the hidden neurons that are more \"interpretable\" in this visualization framework. Additionally, they also notice that fine-tuning a _linear model_ using the learned features for the wider networks provide better accuracy for new (but related) tasks over the shallower counterparts. For most experiments of this paper, \"shallow network\" refers to a width of 64 and \"wide network\" refers to a width of 2048. The main datasets used for the experiments are MNIST, CIFAR 10/100 and a \"translated\" version of MNIST images.\n\n\nOverall the paper is written well and the ideas and results are communicated crisply. I have a few comments. First, regarding the related work, I think that the reader would be served better if the authors also list the recent works related to effect of network width on convergence and generalization (e.g., [1] and references that cite this). The reason I say this is so that the reader should not (wrongly) interpret that this is the first work that finds \"favorable\" properties of wider networks (the paper does not make this claim, but it is easy for a reader to interpret it). Second, I find it slightly concerning that a lot of findings have been extrapolated from just one architecture. In particular, I find the experiments in section 5 to be the most informative (and also objective), since it is a single number which is easy to think about. To be clear, I like the visualization experiments and it gives credibility to the claim about interpretability. Given that there are many levers in a neural net (batch norm, architectural choices, hyper-params etc.) one could fiddle with, to make the claim made in the introduction one needs a more extensive set of experiments. I acknowledge that the authors say they haven't explored the possibility of fine-tuning the hyper-params for instance, but I think considering some of these choices is really helpful. This will help _isolate_ the effect of width independent of the architecture choice.\n\nGiven the above observations, my current decision of this paper is that it doesn't meet the bar. I find the results promising but the paper is not yet ready. \n\n\n[1] - https://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BklafWnaFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors observed that the wider deep neural networks can learn much rich representative features than shallower deep neural networks while both networks show similar level of the test performance. They show feature visualization about their observations using two different networks n=20/n=2048.\n\nAt the first, I feel that the visualizations on figure 1 about two different width are too marginal. Almost they look similar, it's hard to say that significantly show difference.\n\nAlso there is no guarantee that the quality of the feature visualization follows linear relationship according to width. Comparison with just two different width is not enough to analyze the situation.\n\nI wonder if human-interpretable features are always better. Machine-interpretable information also do important role, as adversarial attack.\n\nEven the total number of parameter is preserved, the performance will be largely vary according to the network architecture, such as the number of the layers. Then, I have a doubt whether experiments on Sec 5.1 are meaningful not. More, the model can suffer from the gradient vanishing problem when the network has a number of layers. I wonder that the results on figure 4 are caused from this problem.\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigated the effect of network width on learned features using activation atlases. From the current view of deep learning, the novelty of the paper is limited.

As all reviews rejected the paper and the authors gave up rebuttal, I choose to reject the paper.

Decision: Reject