Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks

In contrast to traditional weight optimization in a continuous space , we demonstrate the existence of effective random networks whose weights are never updated . By selecting a weight among a fixed set of random values for each individual connection , our method uncovers combinations of random weights that match the performance of traditionally-trained networks of the same capacity . We refer to our networks as “ slot machines ” where each reel ( connection ) contains a fixed set of symbols ( random values ) . Our backpropagation algorithm “ spins ” the reels to seek “ winning ” combinations , i.e. , selections of random weight values that minimize the given loss . Quite surprisingly , we find that allocating just a few random values to each connection ( e.g. , 8 values per connection ) yields highly competitive combinations despite being dramatically more constrained compared to traditionally learned weights . Moreover , finetuning these combinations often improves performance over the trained baselines . A randomly initialized VGG19 with 8 values per connection contains a combination that achieves 90 % test accuracy on CIFAR-10 . Our method also achieves an impressive performance of 98.1 % on MNIST for neural networks containing only random weights . 1 INTRODUCTION . Innovations in how deep networks are trained have played an important role in the remarkable success deep learning has produced in a variety of application areas , including image recognition ( He et al. , 2016 ) , object detection ( Ren et al. , 2015 ; He et al. , 2017 ) , machine translation ( Vaswani et al. , 2017 ) and language modeling ( Brown et al. , 2020 ) . Learning typically involves either optimizing a network from scratch ( Krizhevsky et al. , 2012 ) , finetuning a pre-trained model ( Yosinski et al. , 2014 ) or jointly optimizing the architecture and weights ( Zoph & Le , 2017 ) . Against this predominant background , we pose the following question : can a network instantiated with only random weights achieve competitive results compared to the same model using optimized weights ? For a given task , an untrained , randomly initialized network is unlikely to produce good performance . However , we demonstrate that given sufficient random weight options for each connection , there exist selections of these random weight values that have generalization performance comparable to that of a traditionally-trained network with the same architecture . More importantly , we introduce a method that can find these high-performing randomly weighted configurations consistently and efficiently . Furthermore , we show empirically that a small number of random weight options ( e.g. , 2−8 values per connection ) are sufficient to obtain accuracy comparable to that of the traditionally-trained network . Instead of updating the weights , the algorithm simply selects for each connection a weight value from a fixed set of random weights . We use the analogy of “ slot machines ” to describe how our method operates . Each reel in a Slot Machine has a fixed set of symbols . The reels are jointly spinned in an attempt to find winning combinations . In our context , each connection has a fixed set of random weight values . Our algorithm “ spins the reels ” in order to find a winning combination of symbols , i.e. , selects a weight value for each connection so as to produce an instantiation of the network that yields strong performance . While in physical Slot Machines the spinning of the reels is governed by a fully random process , in our Slot Machines the selection of the weights is guided by a method that optimizes the given loss at each spinning iteration . More formally , we allocateK fixed random weight values to each connection . Our algorithm assigns a quality score to each of these K possible values . In the forward pass a weight value is selected for each connection based on the scores . The scores are then updated in the backward pass via stochastic gradient descent . However , the weights are never changed . By evaluating different combinations of fixed randomly generated values , this extremely simple procedure finds weight configurations that yield high accuracy . We demonstrate the efficacy of our algorithm through experiments on MNIST and CIFAR-10 . On MNIST , our randomly weighted Lenet-300-100 ( Lecun et al. , 1998 ) obtains a 97.0 % test set accuracy when using K = 2 options per connection and 98.1 % with K = 128 . On CIFAR10 ( Krizhevsky , 2009 ) , our six-layer convolutional network matches the test set performance of the traditionally-trained network when selecting from K = 64 fixed random values at each connection . Finetuning the models obtained by our procedure generally boosts performance over networks with optimized weights albeit at an additional compute cost ( see Figure 4 ) . Also , compared to traditional networks , our networks are less memory efficient due to the inclusion of scores . That said , our work casts light on some intriguing phenomena about neural networks : • First , our results suggest a performance comparability between selection from multiple random weights and traditional training by continuous weight optimization . This underscores the effectiveness of strong initializations . • Second , this paper further highlights the enormous expressive capacity of neural networks . Maennel et al . ( 2020 ) show that contemporary neural networks are so powerful that they can memorize randomly generated labels . This work builds on that revelation and demonstrates that current networks can model challenging non-linear mappings extremely well even by simple selection from random weights . • This work also connects to recent observations ( Malach et al. , 2020 ; Frankle & Carbin , 2018 ) suggesting strong performance can be obtained by utilizing gradient descent to uncover effective subnetworks . • Finally , we are hopeful that our novel model —consisting in the the introduction of multiple weight options for each edge— will inspire other initialization and optimization strategies . 2 RELATED WORK . Supermasks and the Strong Lottery Ticket Conjecture . The lottery ticket hypothesis was articulated in ( Frankle & Carbin , 2018 ) and states that a randomly initialized neural network contains sparse subnetworks which when trained in isolation from scratch can achieve accuracy similar to that of the trained dense network . Inspired by this result , Zhou et al . ( 2019 ) present a method for identifying subnetworks of randomly initialized neural networks that achieve better than chance performance without training . These subnetworks ( named “ supermasks ” ) are found by assigning a probability value to each connection . These probabilities are used to sample the connections to use and are updated via stochastic gradient descent . Without ever modifying the weights , Zhou et al . ( 2019 ) find subnetworks that perform impressively across multiple datasets . Follow up work by Ramanujan et al . ( 2019 ) finds supermasks that match the performance of a dense network . On ImageNet ( Russakovsky et al. , 2009 ) , they find subnetworks within a randomly weighted ResNet-50 ( Zagoruyko & Komodakis , 2016 ) that match the performance of a smaller , trained ResNet-34 ( He et al. , 2016 ) . Accordingly , they propose the strong lottery ticket conjecture : a sufficiently overparameterized , randomly weighted neural network contains a subnetwork that performs as well as a traditionally-trained network with the same number of parameters . Ramanujan et al . ( 2019 ) adopts a deterministic protocol in their so-called “ edge-popup ” algorithm for finding supermasks instead of the stochastic algorithm of Zhou et al . ( 2019 ) . These empirical results as well as recent theoretical ones ( Malach et al. , 2020 ; Pensia et al. , 2020 ) suggest that pruning a randomly initialized network is just as good as optimizing the weights , provided a good pruning mechanism is used . Our work corroborates this intriguing phenomenon but differs from these prior methods in a significant aspect . We eliminate pruning completely and instead introduce multiple weight values per connection . Thus , rather than selecting connections to define a subnetwork , our method selects weights for all connections in a network of fixed structure . Although our work has interesting parallels with pruning ( as illustrated in Figure 9 ) , it is different from pruning as all connections remain active in every forward pass . Pruning at Initialization . The lottery ticket hypothesis also inspired several recent work aimed towards pruning ( i.e. , predicting “ winning ” tickets ) at initialization ( Lee et al. , 2020 ; 2019 ; Tanaka et al. , 2020 ; Wang et al. , 2020 ) . Our work is different in motivation from these methods and those that train only a subset of the weights ( Hoffer et al. , 2018 ; Rosenfeld & Tsotsos , 2019 ) . Our aim is to find neural networks with random weights that match the performance of traditionally-trained networks with the same number of parameters . Weight Agnostic Neural Networks . Gaier & Ha ( 2019 ) build neural network architectures with high performance in a setting where all the weights have the same shared random value . The optimization is instead performed over the architecture ( Stanley & Miikkulainen , 2002 ) . They show empirically that the network performance is indifferent to the shared value but defaults to random chance when all the weights assume different random values . Although we do not perform weight training , the weights in this work have different random values . Further , we build our models using fixed architectures . Low-bit Networks and Quantization Methods . As in binary networks ( Courbariaux & Bengio , 2016 ; Rastegari et al. , 2016 ) and network quantization ( Hubara et al. , 2017 ; Wang et al. , 2018 ) , the parameters in slot machines are drawn from a finite set . However , whereas the primary objective in quantized networks is mostly compression and computational speedup , the motivation behind slot machines is recovering good performance from randomly initialized networks . Accordingly , slot machines use real-valued weights as opposed to the binary ( or integer ) weights used by low-bit networks . Furthermore , the weights in low-bit networks are usually optimized directly whereas only associated scores are optimized in slot machines . Random Decision Trees . Our approach is inspired by the popular use of random subsets of features in the construction of decision trees ( Breiman et al. , 1984 ) . Instead of considering all possible choices of features and all possible splitting tests at each node , random decision trees are built by restricting the selection to small random subsets of feature values and splitting hypotheses . We adapt this strategy to the training of neural network by restricting the optimization of each connection over a random subset of weight values . 3 SLOT MACHINES : NETWORKS WITH FIXED RANDOM WEIGHT OPTIONS . Our goal is to construct non-sparse neural networks that achieve high accuracy by selecting a value from a fixed set of completely random weights for each connection . We start by providing an intuition for our method in Section 3.1 , before formally defining our algorithm in Section 3.2 . 3.1 INTUITION . An untrained , randomly initialized network is unlikely to perform better than random chance . Interestingly , the impressive advances of Ramanujan et al . ( 2019 ) and Zhou et al . ( 2019 ) demonstrate that networks with random weights can in fact do well , if pruned properly . In this work , instead of pruning we explore weight selection from fixed random values as a way to obtain effective networks . To provide an intuition for our method , consider an untrained network N with one weight value for each connection , as typically done . If the weights of N are drawn randomly from an appropriate distribution D ( e.g. , ( Glorot & Bengio , 2010 ) or ( He et al. , 2015 ) ) , there is an extremely small but nonzero probability that N obtains good accuracy ( say , greater than a threshold τ ) on the given task . Let q denote this probability . Also consider another untrained network NK that has the same architectural configuration as N but with K > 1 weight choices per connection . If n is the number of connections inN , thenNK contains within itKn different network instantiations that are architecturally identical to N but that differ in weight configuration . If the weights of NK are sampled from D , then the probability that none of the Kn networks obtains good accuracy is essentially ( 1− q ) Kn . This probability decays quickly as either K or n increases . Our method finds randomly weighted networks that achieve very high accuracy even with small values of K. For instance , a six layer convolutional network with 2 random values per connection obtains 82 % test accuracy on CIFAR-10 . But how do we select a good network from these Kn different networks ? Brute-force evaluation of all possible configurations is clearly not feasible due to the massive number of different hypotheses . Instead , we present an algorithm , shown in Figure 1 , that iteratively searches the best combination of connection values for the entire network by optimizing the given loss . To do this , the method learns a real-valued quality score for each weight option . These scores are used to select the weight value of each connection during the forward pass . The scores are then updated in the backward pass based on the loss value in order to improve training performance over iterations .

Summary: The paper investigates a type of neural network in which one of K possible fixed weights is chosen in each neuronal connection. The weights themselves are fixed and random, but the scores that determine which of the weights is chosen are updated through back-propagation using a straight-through estimator. The accuracy and behavior of such networks is studied for small networks on small datasets (MNIST, CIFAR).