Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks

In contrast to traditional weight optimization in a continuous space , we demonstrate the existence of effective random networks whose weights are never updated . By selecting a weight among a fixed set of random values for each individual connection , our method uncovers combinations of random weights that match the performance of traditionally-trained networks of the same capacity . We refer to our networks as “ slot machines ” where each reel ( connection ) contains a fixed set of symbols ( random values ) . Our backpropagation algorithm “ spins ” the reels to seek “ winning ” combinations , i.e. , selections of random weight values that minimize the given loss . Quite surprisingly , we find that allocating just a few random values to each connection ( e.g. , 8 values per connection ) yields highly competitive combinations despite being dramatically more constrained compared to traditionally learned weights . Moreover , finetuning these combinations often improves performance over the trained baselines . A randomly initialized VGG19 with 8 values per connection contains a combination that achieves 90 % test accuracy on CIFAR-10 . Our method also achieves an impressive performance of 98.1 % on MNIST for neural networks containing only random weights . 1 INTRODUCTION . Innovations in how deep networks are trained have played an important role in the remarkable success deep learning has produced in a variety of application areas , including image recognition ( He et al. , 2016 ) , object detection ( Ren et al. , 2015 ; He et al. , 2017 ) , machine translation ( Vaswani et al. , 2017 ) and language modeling ( Brown et al. , 2020 ) . Learning typically involves either optimizing a network from scratch ( Krizhevsky et al. , 2012 ) , finetuning a pre-trained model ( Yosinski et al. , 2014 ) or jointly optimizing the architecture and weights ( Zoph & Le , 2017 ) . Against this predominant background , we pose the following question : can a network instantiated with only random weights achieve competitive results compared to the same model using optimized weights ? For a given task , an untrained , randomly initialized network is unlikely to produce good performance . However , we demonstrate that given sufficient random weight options for each connection , there exist selections of these random weight values that have generalization performance comparable to that of a traditionally-trained network with the same architecture . More importantly , we introduce a method that can find these high-performing randomly weighted configurations consistently and efficiently . Furthermore , we show empirically that a small number of random weight options ( e.g. , 2−8 values per connection ) are sufficient to obtain accuracy comparable to that of the traditionally-trained network . Instead of updating the weights , the algorithm simply selects for each connection a weight value from a fixed set of random weights . We use the analogy of “ slot machines ” to describe how our method operates . Each reel in a Slot Machine has a fixed set of symbols . The reels are jointly spinned in an attempt to find winning combinations . In our context , each connection has a fixed set of random weight values . Our algorithm “ spins the reels ” in order to find a winning combination of symbols , i.e. , selects a weight value for each connection so as to produce an instantiation of the network that yields strong performance . While in physical Slot Machines the spinning of the reels is governed by a fully random process , in our Slot Machines the selection of the weights is guided by a method that optimizes the given loss at each spinning iteration . More formally , we allocateK fixed random weight values to each connection . Our algorithm assigns a quality score to each of these K possible values . In the forward pass a weight value is selected for each connection based on the scores . The scores are then updated in the backward pass via stochastic gradient descent . However , the weights are never changed . By evaluating different combinations of fixed randomly generated values , this extremely simple procedure finds weight configurations that yield high accuracy . We demonstrate the efficacy of our algorithm through experiments on MNIST and CIFAR-10 . On MNIST , our randomly weighted Lenet-300-100 ( Lecun et al. , 1998 ) obtains a 97.0 % test set accuracy when using K = 2 options per connection and 98.1 % with K = 128 . On CIFAR10 ( Krizhevsky , 2009 ) , our six-layer convolutional network matches the test set performance of the traditionally-trained network when selecting from K = 64 fixed random values at each connection . Finetuning the models obtained by our procedure generally boosts performance over networks with optimized weights albeit at an additional compute cost ( see Figure 4 ) . Also , compared to traditional networks , our networks are less memory efficient due to the inclusion of scores . That said , our work casts light on some intriguing phenomena about neural networks : • First , our results suggest a performance comparability between selection from multiple random weights and traditional training by continuous weight optimization . This underscores the effectiveness of strong initializations . • Second , this paper further highlights the enormous expressive capacity of neural networks . Maennel et al . ( 2020 ) show that contemporary neural networks are so powerful that they can memorize randomly generated labels . This work builds on that revelation and demonstrates that current networks can model challenging non-linear mappings extremely well even by simple selection from random weights . • This work also connects to recent observations ( Malach et al. , 2020 ; Frankle & Carbin , 2018 ) suggesting strong performance can be obtained by utilizing gradient descent to uncover effective subnetworks . • Finally , we are hopeful that our novel model —consisting in the the introduction of multiple weight options for each edge— will inspire other initialization and optimization strategies . 2 RELATED WORK . Supermasks and the Strong Lottery Ticket Conjecture . The lottery ticket hypothesis was articulated in ( Frankle & Carbin , 2018 ) and states that a randomly initialized neural network contains sparse subnetworks which when trained in isolation from scratch can achieve accuracy similar to that of the trained dense network . Inspired by this result , Zhou et al . ( 2019 ) present a method for identifying subnetworks of randomly initialized neural networks that achieve better than chance performance without training . These subnetworks ( named “ supermasks ” ) are found by assigning a probability value to each connection . These probabilities are used to sample the connections to use and are updated via stochastic gradient descent . Without ever modifying the weights , Zhou et al . ( 2019 ) find subnetworks that perform impressively across multiple datasets . Follow up work by Ramanujan et al . ( 2019 ) finds supermasks that match the performance of a dense network . On ImageNet ( Russakovsky et al. , 2009 ) , they find subnetworks within a randomly weighted ResNet-50 ( Zagoruyko & Komodakis , 2016 ) that match the performance of a smaller , trained ResNet-34 ( He et al. , 2016 ) . Accordingly , they propose the strong lottery ticket conjecture : a sufficiently overparameterized , randomly weighted neural network contains a subnetwork that performs as well as a traditionally-trained network with the same number of parameters . Ramanujan et al . ( 2019 ) adopts a deterministic protocol in their so-called “ edge-popup ” algorithm for finding supermasks instead of the stochastic algorithm of Zhou et al . ( 2019 ) . These empirical results as well as recent theoretical ones ( Malach et al. , 2020 ; Pensia et al. , 2020 ) suggest that pruning a randomly initialized network is just as good as optimizing the weights , provided a good pruning mechanism is used . Our work corroborates this intriguing phenomenon but differs from these prior methods in a significant aspect . We eliminate pruning completely and instead introduce multiple weight values per connection . Thus , rather than selecting connections to define a subnetwork , our method selects weights for all connections in a network of fixed structure . Although our work has interesting parallels with pruning ( as illustrated in Figure 9 ) , it is different from pruning as all connections remain active in every forward pass . Pruning at Initialization . The lottery ticket hypothesis also inspired several recent work aimed towards pruning ( i.e. , predicting “ winning ” tickets ) at initialization ( Lee et al. , 2020 ; 2019 ; Tanaka et al. , 2020 ; Wang et al. , 2020 ) . Our work is different in motivation from these methods and those that train only a subset of the weights ( Hoffer et al. , 2018 ; Rosenfeld & Tsotsos , 2019 ) . Our aim is to find neural networks with random weights that match the performance of traditionally-trained networks with the same number of parameters . Weight Agnostic Neural Networks . Gaier & Ha ( 2019 ) build neural network architectures with high performance in a setting where all the weights have the same shared random value . The optimization is instead performed over the architecture ( Stanley & Miikkulainen , 2002 ) . They show empirically that the network performance is indifferent to the shared value but defaults to random chance when all the weights assume different random values . Although we do not perform weight training , the weights in this work have different random values . Further , we build our models using fixed architectures . Low-bit Networks and Quantization Methods . As in binary networks ( Courbariaux & Bengio , 2016 ; Rastegari et al. , 2016 ) and network quantization ( Hubara et al. , 2017 ; Wang et al. , 2018 ) , the parameters in slot machines are drawn from a finite set . However , whereas the primary objective in quantized networks is mostly compression and computational speedup , the motivation behind slot machines is recovering good performance from randomly initialized networks . Accordingly , slot machines use real-valued weights as opposed to the binary ( or integer ) weights used by low-bit networks . Furthermore , the weights in low-bit networks are usually optimized directly whereas only associated scores are optimized in slot machines . Random Decision Trees . Our approach is inspired by the popular use of random subsets of features in the construction of decision trees ( Breiman et al. , 1984 ) . Instead of considering all possible choices of features and all possible splitting tests at each node , random decision trees are built by restricting the selection to small random subsets of feature values and splitting hypotheses . We adapt this strategy to the training of neural network by restricting the optimization of each connection over a random subset of weight values . 3 SLOT MACHINES : NETWORKS WITH FIXED RANDOM WEIGHT OPTIONS . Our goal is to construct non-sparse neural networks that achieve high accuracy by selecting a value from a fixed set of completely random weights for each connection . We start by providing an intuition for our method in Section 3.1 , before formally defining our algorithm in Section 3.2 . 3.1 INTUITION . An untrained , randomly initialized network is unlikely to perform better than random chance . Interestingly , the impressive advances of Ramanujan et al . ( 2019 ) and Zhou et al . ( 2019 ) demonstrate that networks with random weights can in fact do well , if pruned properly . In this work , instead of pruning we explore weight selection from fixed random values as a way to obtain effective networks . To provide an intuition for our method , consider an untrained network N with one weight value for each connection , as typically done . If the weights of N are drawn randomly from an appropriate distribution D ( e.g. , ( Glorot & Bengio , 2010 ) or ( He et al. , 2015 ) ) , there is an extremely small but nonzero probability that N obtains good accuracy ( say , greater than a threshold τ ) on the given task . Let q denote this probability . Also consider another untrained network NK that has the same architectural configuration as N but with K > 1 weight choices per connection . If n is the number of connections inN , thenNK contains within itKn different network instantiations that are architecturally identical to N but that differ in weight configuration . If the weights of NK are sampled from D , then the probability that none of the Kn networks obtains good accuracy is essentially ( 1− q ) Kn . This probability decays quickly as either K or n increases . Our method finds randomly weighted networks that achieve very high accuracy even with small values of K. For instance , a six layer convolutional network with 2 random values per connection obtains 82 % test accuracy on CIFAR-10 . But how do we select a good network from these Kn different networks ? Brute-force evaluation of all possible configurations is clearly not feasible due to the massive number of different hypotheses . Instead , we present an algorithm , shown in Figure 1 , that iteratively searches the best combination of connection values for the entire network by optimizing the given loss . To do this , the method learns a real-valued quality score for each weight option . These scores are used to select the weight value of each connection during the forward pass . The scores are then updated in the backward pass based on the loss value in order to improve training performance over iterations .

Summary: This paper proposes a method to train a neural network by selecting a weight from a set of $K$ randomly generated weights for each edge in the network. Each edge has a different set of random weights. Quality score is assinged to each of $K$ random weights, which determines the weight used in the forward calculation. Instead of optimizing weights directly, the proposed method optimizes the quality scores with the straight-through gradient estimator. Experimental results show that the neural network trained by the proposed method achieves high accuracy compared to random initialization even when $K=2$.