It's Hard for Neural Networks to Learn the Game of Life
Authors: Jacob M. Springer
Garrett T. Kenyon
Venue:  (2021)
Keywords: Deep Learning
Game of Life

Abstract: Efforts to improve the learning abilities of neural networks have focused mostly on the role of optimization methods rather than on weight initializations. Recent findings, however, suggest that neural networks rely on lucky random initial weights of subnetworks called "lottery tickets" that converge quickly to a solution. To investigate how weight initializations affect performance, we examine small convolutional networks that are trained to predict $n$ steps of the two-dimensional cellular automaton Conway’s Game of Life, the update rules of which can be implemented efficiently in a small CNN. We find that networks of this architecture trained on this task rarely converge. Rather, networks require substantially more parameters to consistently converge. Furthermore, we find that the initialization parameters that gradient descent converges to a solution are sensitive to small perturbations, such as a single sign change. Finally, we observe a critical value $d_0$ such that training minimal networks with examples in which cells are alive with probability $d_0$ dramatically increases the chance of convergence to a solution. Our results are consistent with the lottery ticket hypothesis.

Under review as a conference paper at ICLR 2021
IT's HARD FOR NEURAL NETWORKS TO LEARN THE GAME OF LIFE
Anonymous authors Paper under double-bling review
ABSTRACT
Efforts to improve the learning abilities of neural networks have focused mostly on the role of optimization methods rather than on weight initializations. Recent findings, however, suggest that neural networks rely on lucky random initial weights of subnetworks called "lottery tickets" that converge quickly to solution (Frankle) & Carbin 2018). To investigate how weight initializations affect performance, we examine small convolutional networks that are trained to predict n steps of the two-dimensional cellular automaton Conway's 's Game of Life, the update rules of which can be implemented efficiently in a small CNN. We find that networks of this architecture trained on this task rarely converge. Rather, networks require substantially more parameters to consistently converge. Furthermore, we find that the initialization parameters that gradient descent converges solution are sensitive to small perturbations, such as a single sign change. Finally, we observe a critical value do such that training minimal networks with examples in which cells are alive with probability do dramatically increases the chance of convergence to a solution. Our results are consistent with the lottery ticket hypothesis (Frankle & Carbin 2018).
INTRODUCTION
Recent findings suggest that neural networks can be "pruned" by 90% or more to eliminate unneces- sary weights while maintaining performance similar the original network. Similarly, the lottery ticket hypothesis (Frankle & & Carbin) 2018) proposes that neural networks contain subnetworks, called winning tickets, that can trained isolation to reach the performance of the original. These results suggest that neural networks may rely on these lucky initializations to learn a good solution. Rather than extensively exploring weight-space, networks trained with gradient-based optimizers may converge quickly to local minima that are nearby the initialization, many of which will be poor estimators the dataset distribution. If some subset of the weights must be a winning configuration for a neural network to learn good solution random weights must be significantly larger than the minimal network configuration that would solve the problem in order to optimize the chance having winning initialization. Furthermore, small networks with winning initial configurations may be sensitive to small perturbations.
problem, then neural networks initialized with
Similarly, gradient-based optimizers sample the gradient of the loss function with respect to the weights by averaging the gradient at a few elements of the dataset. Thus, a biased training dataset may bias the gradient in a way that can be detrimental to the success of the network. Here we examine how the distribution of the training dataset affects the network's 's ability to learn.
In this paper, we explore how effectively small neural networks learn to take as input a configuration for Conway's Game of Life (Life), and then output the configuration n steps in the future. Since this task can be implemented minimally in a convolutional neural network with 2n layers and 23n 2 trainable parameters, a neural network with identical architecture should, in principle, be able to learn a similar solution. Nonetheless, we find that networks of this architecture rarely find solutions. We show that the number of weights necessary for networks to reliably converge solution increases quickly with n. Additionally, we show that the probability of convergence is highly sensitive to small perturbations of initial weights. Finally, that significantly increase the probability that a network will converge to a correct solution. While Life is toy problem, believe that these studies give insight into more general issues with training
explore properties of the training data

---

Under review as a conference paper at ICLR 2021
neural networks. In particular, we expect that other neural network architectures and problems exhibit similar issues. We expect that networks likely require a large number of parameters to learn any domain, and that small networks likely exhibit similar sensitivities small perturbations to their weights. Furthermore, optimal training datasets may be highly particular to certain parameters. Thus, with the growing interest in efficient neural networks (Han et al. 2015] Hassibi & Stork 1993 [Hinton iet al., 2015; [eCunn et al., 1990, Li et al., 2016), this results serve as an important step forward in developing ideal training conditions.
1.1 CONWAY'S S GAME OF LIFR
Prior studies have shown interest in applying neural networks to model physical phenomena in applications including weather simulation and fluid dynamics (Baboo & Shereef 2010 Maqsood
et al., |004] Mohan & Gaitonde] 2018 Shrrvvsttaaa et al.) (0022). Similarly, neural networks are trained to learn computational tasks, such as adding and multiplying (Kaiser Sutskever [0155] Graves et al. 2014 Joulin & Mikolov, 2015 Trask et al., 2018). In all of these tasks, neural networks are required to learn hidden-step processes which the network must learn some update rule that can be generalized to perform multi-step computation.
Conway's s Life is a two-dimensional cellular automaton with a simple local update rule that can produce complex global behavior. In Life configuration, cells in an m grid can be either alive or dead (represented by 1 or O respectively). To determine the state of a given cell on the next step, Life considers the 3 x 3 grid of neighbors around the cell. Every step, cells with exactly two alive neighbors will maintain their state, cells with exactly three alive neighbors will become alive, and cells with any other number of neighbors will die (Figure [.. We consider variant of Life in which cells outside of the n x m grid are always considered to be dead. Despite the simplicity of the update rule, Life can produce complex output over time, and thus can serve an idcalized problem for modeling hidden-step behavior.
RELATED WORK
Convolutional models of cellular automata including the Game of Life have been studied by Gilpin (2019). who classifies structural representations of the learned solutions to different cellular automata. Furthermore, Gilpin notes that narrow networks do not often converge, and for consistent convergence behavior, the cellular networks must be sufficiently wide. Our work this paper quantifies this result, and in addition explores the sensitivity of initial conditions to perturbations and different dataset distributions.
Prior research has shown interest in whether neural networks can learn particular tasks. Joulin & Mikolov (2015) argue that certain recurrent neural networks cannot learn addition in a way that generalizes to arbitrary number of bits. Theoretical work has shown that sufficiently overparame- terized neural networks converge to global minima (Oymak & Soltanolkotabi 2020 Du et al. 2018)). Further theoretical work has found methods to minimize local minima (Kawaguchi & Kaelbling 1019.Nguyeer & Hein 2017; Kawaguchi] 2016). Nye & Saxe (2018) show that minimal networks for the parity function and fast Fourier transform do not converge to solution unless they are initialized
close to a solution.
Increasing the depth and number of parameters of neural networks has been shown to increase the speed at which networks converge and their testing performance (Arora et al.) 20188] Park et al.) 2019). Similarly. FFankle & Carbin (2018) find that increasing parameter count can increase the chance of convergence to good solution. Similarly. Li et al. (2018) and Neyshabur et al. (2018) find that training near-minimal networks leads to poor performance. Choromanska et al. (2015) provide some thcoretical insight into why small networks are more likely to find poor local minima.
Weight initialization has been shown to matter in training deep neural networks. Glorot & Bengio (2010) find that initial weights should be normalized with respect to the size of each layer. Dauphin & Schoenholz (2019) find that tuming weight norms prior to training can increase training performance. Similarly, Mishkin & Matas (006) propose method for finding good weight initialization for learning. Zhou et al. (2020) find that the sign of initial weights can determine a particular subnetwork will converge good solution.
2

---

Under review as a conference paper at ICLR 2021
Figure 1: An example of an 8 x 8 cell board of Life over six time steps, evolving in time from left to right. White pixels are considered alive and black pixels are considered dead.
There is significant research into weight pruning and developing efficient networks (LeCun et al. 1990; Hassibi & Stork [993] Han et al.) 2015 Li et al.) 2016) Hinton et al.) 2015 Li al. (018)). including the lottery ticket hypothesis, which suggests that gradient descent allows lucky subnetworks to quickly converge to solution (Frankle & Carbin (2018).
Finally, there interest in learning hidden step computational processes including algorithmss and arithmetic (Kaiser & Sutskever [2015] Graves al. [2014] Joulin & Mikolov 2015 Trask al. 2018), fluid dynamics (Baboo & Shereef 2010; Maqsood et al.
2004 Mohan & Gaitonde 2018).
and weather simulation (Shrivastava et al. 2012
Among the above papers, there are studies that have already shown that weight initialization, over- parameterization, and training dataset statisties determine whether or a neural network can converge to a good solution to a problem. In this paper we build on these results to quantify them in the case simple problem with known minimal solution. This allows us to derive empirical insights that may not be possible to observe where perfect solution is not known, such as in the vision or language domains.
EXPERIMENTS AND RESULTS
function-learning problem. In particular, if x is matrix of 1s and Os, define Gzx) to be the next step in Life, according to the previously described update rules. Then, we define the Life problem to be the problem of predicting G(x)) given æ. Similarly, we define the n-step-Life problem as the problem of learning to predict (*(()) given æ. Since Life has a local update rule that considers 3 grid determine the state of the center cell, we can model Life with an entirely convolutional neural network, i.e., a neural network without any fully connected or pooling layers. A convolutional layer with two 3 filters that feeds into a second convolutional layer with one 1 x 1 filter, can solve the 1-step-Life problem efficiently. i.e., any fewer layers or convolutional filters would yield an architecture which cannot implement 1-step-Lifee Thus, we call it the minimal architecture for Life. We use ReLU activation functions prevent vanishing gradients for when the architecture is generalized to the ---tee-Liife problem by stacking layers. The second convolutional layer feeds into a final convolutional output layer with activation function. This forces all outputs to approximate either 0 or but does not, on its own, perform meaningful computation, and thus is included for this convenience. With appropriate weights, this constructs a three-layer convolutional neural network that can solve the 1-step-Life problem with 25 weights. We generalize this architecture solve the n-step-Life problem by stacking n copies of this network, as shown in Figure (right).
We define the Life problem
x 1 filter with sigmoid
We have hand-engineered weights for these architectures that implement the underlying rule and thus solves the n-step-Life problem with perfect accuracy. We conclude that this minimal neural network architecture can solve the n-step-Life problem with a 2n + layer convolutional neural network with 23n + 2 weights. In principle, neural network with an identical architecture should be able to learn a similar solution.
3.1 LIFE ARCHITECTURE
We construct class of architectures to measure how effectively networks of varying sizes solve a hidden-step computational problem. In particular, we employ an architecture similar to the one deseribed in the previous section: an entirely convolutional neural network with copies of convolutional layer with x 3 filters that feed into a convolutional layer with 1 x filters, and finally, convolutional layer with single 1 filter and sigmoid activation to decode the output into Life
3

---

Under review as a conference paper at ICLR 2021
Life state after n steps
Life state after 1 step
Sigmoid
Simmoi
Conv2d(dim= 1x1, nfilters=1)
Conv2d(dim==1Xl I, nfilters=1)
Retu
Conv2d(dim= x1, nfilters=1)
Conv2d[dim=1xx nflltees=m)
ReW
RetU
coples
Conv2d(dim=3x3, nfilters=2)
Conv2d(dim=333, nf
Life initial state
Life initial state
Figure 2: Neural network architecture diagrams for the 1-step minimal model and L(n, m). On the left, the 1 1-step minimal architecture consists of an input layer that feeds into convolutional layer with two 3 x filters with ReLU activation, then into a convolutional layer with one 1 x filter with ReLU activation, fed into similar convolutional layer but with sigmoid activation for decoding. On the right, the architecture for [(n, m) consists of the same as the minimal model, except where the first two hidden layers consist of 2m and filters respectively, and repeated times, where m is the factor of overcompleteness (see text).
configuration. When the architecture has n copies of the described layers, we say that it is an n-step architecture. In the minima solution, each repeated 3 x 3 convolutional layer has two filters and each repeated 1 x 1 convolutional layer has one filter. When similar architecture has 2m 3 x 3 filters and overcomplete with respect to the minimal architecture. We let C(n, m) describe the n-step m-times overcomplete architecture.
filters in each respective repeated layer, we say that the architecture m-times
To train instances of each architecture, we initialize the weights randomly from a unit normal distribution. The networks are implemented Keras (Chollet, 2013) on top of TensorFlow (Abadi et al. (2015) and trained using the Adam optimizer (a 0.001, 31 0.9, B2 0.999) (Kingma & Ba 2014)) with a binary cross-entropy loss function on the output the model. Each instance is trained with 1 million randomly generated training examples separated into 100 epochs of 10,000 training examples each, with a batch size of Each training and testing example is generated as follows: first, we uniformly draw a density d from [0. 1], and then generate 32 x 32 cell board such that each cell alive with probability d. It is extremely unlikely that the network will ever see the same training example twice. Thus, separating testing data into testing and validation set is unnecessary, since novel data can be generated on the fly. To improve computational efficiency, all networks with the identical parameters are implemented that they can be trained in parallel using the same randomly generated dataset.
3.2 THE DIFFICULTY OF LIFB
To quantify the effectiveness of a given neural network architecture, we measure the probability that random initialization of the network converges solution after being shown million training examples. Because [n,, m) can only implement a 3 x 3 update rule in each step of computation and minimal in this sense, for C(n, m) to solve the n-step-Life problem, must learn the underlying rule. Thus, we consider an instance of L(n, m) to be successful when it learns the correct underlying rule, and can therefore predict G" (x) with perfect accuracy for all initial states x. Any instance of L that does not have perfect accuracy did not learn the underlying rule and is thus considered unsuccessful. We wish to determine
P success of C(n, m) n, m
A reviewer helpfully pointed out that an even smaller network can constructed to solve Life, with a single 3 x 3 convolution that counts neighbors, outputs a 0 if there are two neighbors. 1 if there are three neighbors. and -1 otherwise, and then is added to the input and fed through Heaviside activation function. However, model minimal given the constraint that we are using a traditional feedforward CNN with ReLU activation.
4

---

Under review as a conference paper at ICLR 2021
Fraction of successful trials
0.8
0.6
014
n=2
10 15 20 25
Overcompleteness (m)
Figure 3: Measured probability that the m times overcomplete n-step-Life architecture learns successfully. Each line corresponds with particular n. Each point plots the percentage of 64 instances of [(n, m) with random initializations sampled from unit normal distribution that learned the rules of n-step-Life after I million training examples. We train instances of C(n, m) for values 1 < n 5 and S m < 24, excluding many combinations due to computational constraints. For n I, none of the instances of successfully learned with the minimal(m 1) architecture. As n increases, the degree of overcompleteness required for consistent converges increases rapidly.
To accomplish this, we train 64 instances C(n, m) for 1 S 3 we plot the percentage of instances
5 and S m S 24. We omit
certain combinations due computational limitations. In Figure of (n,, m) that successfully learn the n-step-Life problem.
We observe that of the minimal (m 1) architectures, only instances for the 1-step-Life problem (C(1, 1)) converged on solution, with a success rate of approximately 4.7%. Instances of architec- tures to solve the and two-step-Life problem had greater than 50% chance of converging solution when the architecture was at least 3 and 4-times overcomplete, respectively. Instances of architectures solve the n-step-Life problem for the highest degree of overcompleteness we tested, due to computational constraints. This explosive growth rate suggests that the degree of overcompleteness required for consistent convergence of the n-step-Life problem grows quickly with respect to
require an overcompleteness greater than 24,
Strikingly, for 3 < n < do not observe the hypothesized scaling behavior. Rather, we observe that for high overcompleteness, the architectures for n 4 outperforms = 3, and n 5 performs similarly to n While three n require many more parameters than minimal architecture to consistently converge, would expect that 3 requires fewer than = 4. which would require fewer than n 5. We have multiple hypotheses: firstly, we may observe this result due to noise dataset artifacts; secondly, our parameterization of Life may have consistent behavior for all 3 < n & 5, which may make the difficulty of learning any 3 S n Σ 5 steps similar.
We plot the loss of the instances of L(n, m) for 1 < rates at which the networks converge to a solution. of convergence for converged networks of L(n, m) architecture for 1 < n < 4 and The earliest point of converge is computed by determining the earliest epoch in which the loss of convergent network falls below 0.01 indicate the network has learned solution the n-step-Life problem. We exclude non-converged networks from this metric.
8 in Figure to illustrate typical addition, we compute the average earliest point
and
8 (Figure
3.3 WEIGHT PERTURBATIONS AND LEARNING
To observe the robustness weight initializations and of learned solutions, we perturb successful weight initializations and solutions of the minimal 1-step-Life architecture L(1, 1). In particular, we perform two perturbations: the k-sign perturbation and the uniform perturbation. The k-sign perturbation modifies weights follows: we select k weights randomly from a uniform distribution. We replace each chosen weight with a weight of the same magnitude but of opposite sign. The uniform perturbation modifies weights by adding value selected uniformly from the range [-r,r] for given perturbation magnitude r. We choose a weight initialization of a network which converges solution to the 1-step-Life problem. We initialize and train instances of the minimal architecture
5

---

Under review as a conference paper at ICLR 2021
411.8)
(12,8)
43,88
0.4-
14-
033
0.5
2.3
02
04
Time
4[4,8)
415,8)
Average number of epoths conuerge
A(n,8) for 1 s m
0.500
2
&
244
0375
220
% %
21
2200
0375
Number
Figure 4: First five graphs: Binary cross-entropy loss over the duration of training of the 64 networks trained solve the n-step-Life problem for S S 5. The horizontal axis corresponds number of epochs of training. For clarity, we omit from the graphs the loss of networks that eventually diverged to degenerate state where networks predicted all cells to be dead, regardless the input Life configuration. These examples converge point of convergence of L(n, m) for m = and S n S 4. Note that = is excluded because no instances of C(5, 8) converge. We compute the earliest point of convergence for each network by observing the first epoch where the loss falls below 0.01, indicating that the network has reached a stable 100% accuracy.
loss that is well over 1. Last graph: Average earliest
Insign perturtation, converged ghrs
J-sign perturtation, initsw
Uniftmm perturcation, Intill weights
Frodden seceoste
Prattton
Number weggtba with appeddaggn
Narbber of wewell gfra with floped skgn
Magritsck of unifamm perturbalian Inrglt
Figure 5: Left: Fraction of converged L(1, networks with weights initialized with k-sign perturbation of a converged solution to the 1-step-Life problem. Center: Same as left except weights initialized with k-sign perturbation the original initial weights that converged the solution. Right: Same as center except weights initialized with uniform perturbation of the original initial weights. k-sign perturbation of weights defined as perturbation which & weights are chosen randomly from uniform distribution and replaced with the same magnitude weight with opposite sign. In all cases, 128 networks with the specified weight initialization are trained over 50 epochs, which is well over the number of epochs required for convergence for instances of C(1, 1). We omit the graph for uniform perturbations of converged weights, as small perturbations have little effect given the magnitude of the converged weights.
with these weights perturbed by k-sign perturbations for 1 < k < 8 and uniform perturbations for (0.25, 0.5. 0.75, 1.0). Similarly, initialize and train instances of the minimal architecture with the described k-sign and uniform perturbations of the converged solution of this network. For each perturbation type, train 128 instances.
We plot the fraction of successful networks for each perturbation type in Figure 5 Notably, a 1-sign perturbation of the original initial weights of the successful network causes the network fail to

---

Under review as a conference paper at ICLR 2021
211.11
(01,1), zoomed in
40,2)
6.44
0.11
0.13
0.10
Alive cnl dunaly of bamning datios
Altve call deraity training dataset
Alox sell denaky ol delanw.
Probobiity oot 2arts alve
0.8-density dataset 0.38-density dataset
Figure 6: Top: Fraction of minimal networks that converge to solution to the I-step-Life problem (left and center) and 2-times overcomplete networks (right) when trained with datasets of given d-density dataset. The left and center graphs refer to the same training configurations, however, the left graph includes the rate of convergence for datasets with d between 0.1 and 0.9 with 0.05 intervals while the center graph has data for d between 0.2 and 0.5 with 0.0125 intervals.
Bottom: Left: the probability that an arbitrary cell is alive after 1 step of Life of an in- stance of a 32 32 cell d-density dataset, given d. This curve peaks at approximately d s 0.38. Right: two examples from the generated datasets (0.8 (0.8-density and 0.38-density). We generate d-density dataset by choosing 32 32 cell grids as training examples where cells are alive with probability d.
converge approximately 20% of the time, and only 4-6 sign perturbations are required drop the success rate below 50%. This suggests that for minimal networks, the weight initialization is sensitive perturbations. This unique sign perturbations. Even relatively small uniform perturbation of 0.25 magnitude (where weights are changed by 0.125 in expectation) causes the tested networks to fail to learn approximately 36% of the time. Finally, we observe that even 1-sign perturbation of an already converged solution causes approximately 90% of models to fail to learn, suggesting that converged solutions are very sensitive sign perturbations. Furthermore, since typical weights are small in the converged solution (weights have mean of approximately 0.270 and standard deviation of approximately 2.17), sign perturbations do not represent large-magnitude changes.
3.4 AN OPTIMAL TRAINING DATASET
Many deep learning systems are restricted by the dataset which available for training. We examine how class of training datasets affects the success rate of near-minimal networks learning the step-Life problem. In particular, we construct a class of training datasets: the d-density dataset, 32 32-cell dataset in which cells are chosen independently be alive with probability d. Note that the dataset described in Section 3.J is a generalization of this class of datasets in which d is chosen uniformly, which we call the uniform-densie dataset. We show examples of these datasets in Figure
6
We train 128 instances of C(1, 2) on the d-density datasets for 0.1
0.9 with intervals of 0.05. We train C( 1) on the same datasets, and in addition, on d-density datasets for 0.2 < d S 0.5 with intervals of 0.0125. Surprisingly, we find sharp spike probability of success for values between approximately 0.3 and 0.4. When d = 0.35, observe 14% success rate for the minimal model, a strikingly high success rate considering that it is approximately double the success rate of the 0.4-density dataset and triple the success rate of the 0.3-density dataset (Figure (6). The same result, though less exaggerated, appears for C(1, 2), where the performance increases drastically for d 0.35 and d = 0.4 (Figure 6). The tiny range in which performance increases significantly
7

---

Under review as a conference paper at ICLR 2021
suggests that there is likely critical value do such that the do-density dataset is in this sense optimal. We hypothesize that the value of du coincides with the peak of the graph shown in Figure [6 which plots the probability that cell is alive after one step of Life given that the initial configuration of Life is drawn from a d-density dataset, for a given d. This would place dg R 0.38. We predict that an optimal dataset must satisfy condition in which the probability of observing each possible 3 x 3 local configuration of Life reaches an equilibrium that allows the computed average gradient of the weights with respect to the loss function and the training examples to direct the weights quickly to solution. For example, for very small d, we expect most cells to be dead after one step of Life given an initial state sampled from the d-density dataset. Thus, in this case, the computed average gradient of the weights will tend to drive the network towards solution which predicts most cells to be dead. The density which maximizes the probability that cell will be on after one step of Life will maximize the occurrence of cells with exactly three neighbors and alive cells with exactly two neighbors, any instance of which will increase the number of cells that are alive in the next step of Life. We hypothesize that frequent observation of these configurations are critical for minimal network to solve the n-step-Life problem.
DISCUSSION
In this paper, we present four primary results:
1. The minimal network trained with gradient descent rarely converges on solution to Life.
2. As the complexity of the problem increases, the necessary network size to consistently learn a solution grows rapidly.
3. Ideal initial network parameters are highly sensitive to perturbations.
4. Gradient descent is highly sensitive to distribution parameters of the dataset.
This result is consistent with the lottery ticket hypothesis, which proposes that neural networks converge due to lucky subnetworks that occur due to random initialization which have amenable initial weights for learning the particular task. Thus, would be unlikely for a near-minimal network to be initialized with weights that are perfect for learning. We characterize this likelihood for networks that model the Game of Life, and in addition show that lucky initialization parameters are highly sensitive to perturbations or non-ideal dataset distributions.
While Conway's s Game of Life itself is toy problem and has few direct applications, the results we report here have implications for similar tasks in which a neural network is trained to predict an outcome which requires the network to follow a set of local rules with multiple hidden steps. Examples of such problems include but are not limited to machine-learning based logic or math solvers, weather and fluid dynamics simulations, and logical deduction in language or image processing. In these instances, without enormously overcomplete networks, gradient descent based optimization methods may not suffice to learn solutions to these problems. Furthermore, such a result may generalize to problems that do not explicitly involve local hidden step processes, such as classification of images and audio, and virtually every other application of machine learning. In addition, significant effort has gone into developing faster and smaller networks with similar performance to their larger counterparts. Our result suggests that these smaller networks may necessarily require alternative training methods, or methods to identify optimal weight initializations.
The highly specific ideal dataset distribution for learning the Game of Life may be specific to the particular networks we train or the problem itself. However, other neural networks, especially small networks, may suffer from similar problems. The dataset parameters may need to be tuned near perfectly in order to maximize the learning potential of neural networks.
In conclusion, we find that networks of the L architecture that are trained to predict the configuration of Life after n steps given an arbitrary initial configuration require a degree of overcompleteness that scales quickly with in order to consistently learn the rules of Life. Similarly, we show that weight initializations and converged solutions are extremely sensitive to small perturbations. Finally, we find that these networks to observe a significant increase in success probability. These observations are consistent with the predictions of the lottery ticket hypothesis, and have important consequences in the field.
dependent on very strict conditions of the dataset distribution in order
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: //ww//ww....o www.t.ensorfloo org/ from tensorflow.org.
Software available
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. arXiv:/802.004 [cs/, June 2018. URL http://
arxiv.org/abs/
06509
arXiv: 1802.06509.
S Santhosh Baboo and Kadar Shereef. An efficient weather forecasting system using artificial neural network. International journal of envirommental science and development, 1(4):221, 2010.
François Chollet. keras. htttps///sttaub..cm///ccoollel//
2015.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204, 2015.
Yann N Dauphin and Samuel Schoenholz. Metalnit: Initializing learning by learning to initialize. In Advances in Neural Information Processing Systems. pp. 30645-12651 2019.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai Gradient descent finds global minima of deep neural networks. arXiv preprint arXiv: 18I (81.:880440 03804, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint
2018
William Gilpin. Cellular automata as convolutional neural networks. Physical Review E, 100(3): 032402, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1135- 1143. Curran Associates, Inc., 2015. URL http: http:://pppprs..sssccccooco aondd--oommmtciioo-------ffeeeee
5784 hel-leernnnn--oot... pdf
network.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, pp. 164-171, 1993.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1503vv/503.000 02531, 2015.
Armand Joulin and Tomas Mikolov. Inferring algorithmic patterns with stack- augmentede recurrent nets. In Advances in neural information processing systems, pp. 190-198, 2015.
Fukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. arXiv preprint arXiv: 15/1 2015.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp. 586-594, 2016.

---

Under review as a conference paper at ICLR 2021
Kenji Kawaguchi and Leslie Pack Kaelbling. Elimination of all bad local minima in deep learning. arXiv preprint
2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXfv:1412.6980, 2014.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the Intrinsic Dimension of Objective Landscapes, arXiv: 1804vv880408888 [cs, stat), April 2018. URLhttp::/ httttt::/aaxiv.ooo/ waxxi.orrg abs 1804. 08838
arXiv: 1804.08838.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1cXiv:1608.08710 2016.
Imran Maqsood, Muhammad Riaz Khan, and Ajith Abraham. An ensemble of neural networks for weather forecasting. Neural Computing & Applications, 13(2):112-122, 2004.
Dmytro Mishkin and Jiri Matas, All you need is good init. arXiv:1511.0642 [cs], February 2016. URL httpthttp///xiieee org/ htppg//bs//111111 06422 arXiv: 151 1511.06422.
Arvind T Mohan and Datta V Gaitonde. A deep learning based approach to reduced order modeling for turbulent flow control using istm neural networks. arXiv preprint arXiv:1804. 09269,9, 2018.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards Understanding the Role of Over-Paramettrization in Generalization of Neural Networks. OrXiv:I805: 12076 fcs, stat), May 2018. URL http:://aax.....rr//bb///////a/e 12076 arXiv: 1805. 12076.
Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2603-2612. JMLR, org, 2017.
Maxwell Nye and Andrew Saxe. Are efficient deep representations learnable? 1 2018
arXiv preprint
S. Oymak and M. Soltanolkotabi. Towards moderate overparameteriizaton:: global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theury, pp. 1-1, 2020.
Daniel Park, Jascha Cohl-Dickstein, Quoc V. Le, and Samuel Smith. The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study. arXiv: 03776 [cs, stat), May 2019. URL http:: wtwp:iiwwwggaabb..b 1905. 03776 arXiv: 1905.03776.
Gyanesh Shrivastava, Sanjeev Karmakar, Manoj Kumar Kowar, and Pulak Guhathakurta. Application of artificial neural networks in weather forecasting: a comprehensive literature review. International Journal of Computer Applications, 51(18), 2012.
Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems, pp. 8035-8044 2018.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask. arXiv: (a0e::905000000 01067 [cs, stat], March 2020. URL http::/arxt orq/ axq/abs/Dae 19t.. 01067
arXiv: 1905.01067.
10

---

Under review as a conference paper at ICLR 2021
A WEIGHTS FOR MINIMAL ARCHITECTURE
We describe weights that solve Life for the minimal architecture 1).
The first layer has two 3 3 convolutional filters, each with bias, described follows:
W1,1 =
1/10
1
1
1
11,1 = -
1 1
W1,2
1
1
1
21, =-2
where W1,1 and W1,2 describes the weights of the first and second convolutional filters, respectively, and 1 and 21, similarly describes the bias. Each output is fed through a ReLU function.
The second layer has single filter which combines the output of the two filters from the previous layer.
22,1 (-10) @ (1)
where the first component corresponds with the first-layer output the first filter and the second component corresponds with the output of the second filter.Each output is fed through ReLU function.
We add one more layer, convenience for learning:
W3 = (2s). by = -s
for an arbitrary large s, and in practice, 20. The output then fed through a sigmoid function. Thus, the entire architecture, given Life input x, has that (xx) computes the next step, where N is defined as:
N1 (eLU(( @ W1,1 b1.1) (x Wi,2 + b1,2))
N₂ ReI.U(N, @ (W2,1 @ W2,))
N Sigmoid N2 @ Wa + by)


Reviews:
[{"review_id": "xxVobZ2nvo6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses Conway's Game of Life (GoL) as a testbed for evaluating the learning dynamics of neural networks as a function of their architecture or the random initialization.  In particular, they look at predicting the state of GoL $n$ steps into the future given an arbitrary random initial state, using a multilayer CNN that is shown to be capable of encoding a correct solution to this problem. This network can be $m$-times over-parametrized by adding $m$-times more filters to the convolutional layers. The authors show that in practice, this network rarely finds solutions, especially when $m$ is small or when $3 \\leq n \\leq 5$, for which it is necessary to use a much larger number of filters (for instance, when $n=1$, the system with just an additional pair of filters finds more reliably solutions than when $n=4$ but 10 times more filters are provided). Finally, the authors show that the systems' convergence probability is sensitive to the initial weights.\n\nIn this reviewer's opinion, Conway's Game of Life indeed constitutes an interesting testbed to study the capabilities of neural networks thanks to its rich dynamics, which have been shown to be Turing-complete. The authors further claim that they expect that this study might extend to other architectures and problems. Furthermore, they extrapolate their results to support the claim that neural networks require a large number of parameters to learn any domain, and small networks are sensitive to perturbations.\n\n+ (+) One the main positive aspects of this paper are the experimental results on characterizing the probability of convergence of a network for a given the $n$-step learning problem, with an $m$-times over-parametrized network.\n\n+ (+) They are also very interesting the results that characterize the probability of success of training the network as a function of the initial density, and show that the optimal training density is consistent with a well-known attracting point of $\\rho=0.37$ (see, e.g. https://arxiv.org/pdf/1407.1006.pdf).\n\n+ (+) The authors also do some experiments characterizing the basin of attraction of solutions by perturbing the weights of converged networks and of their random initializations before training, showing that the networks are sensitive to the studied perturbations. (See also http://papers.nips.cc/paper/395-back-propagation-is-sensitive-to-initial-conditions.pdf for a related study that is very relevant to this work).\n\n- (-) On the other hand, I believe there might be some issues with how the $n$-step learning problem is analyzed. In particular, it is worth noting that the authors decided to model this by replicating the convolutional blocks $n$ times. Yet, in doing so, they conflate two different issues: On the on side, there is the expected problem of learning to predict $n$ steps in the future. Yet, on the other hand, there is the fact that each of these sub-networks need to learn the transition function independently from each other! Thus, the problem becomes exponentially harder with $n$ (which might in part explain the very poor performances for $n > 2$. To differentiate between these two aspects, it would have been extremely beneficial to have a setup in which the weights for these blocks are _shared_.\n\n- (-) Given that the role of the initialization is greatly emphasized, it surprised me that just a single (and not much motivated) initialization scheme was tried, namely, the unit ball. Similarly, the strong reliance on Adam with default parameters as the only optimization algorithm would require some justification.\n\n- (-) The claim that the number of parameters of the network needed to learn a correct solution grows with the complexity cannot be fully substantiated for the reason that the the scaling breaks for $n > 2$. On one hand, the authors show that more parameters facilitate the learning for $n=1$ or $2$. On the other hand, they show that for $n>2$, if you want to have a shot at learning the problem, you better use a larger network. Yet, there is no consistent improvement, nor clear relation, and thus the claims and conclusions need to be more subtle.\n\n- (-) The title is catchy, but inaccurate. After all, the authors show that neural networks can learn the Game of Life quite easily. You just need more parameters and, say, a single evolution step, but that's exactly what both neural networks and game of life are all about (over-parametrized models and a system with markovian rules)\n\n\n**Questions for the authors** \n\n- What are you trying to show with Figure 4? It doesn't seem to drive any specific point in the main text.\n\n- \"We have multiple hypotheses: firstly, we may observe this result due to noise or dataset artifacts; secondly, our parameterization of Life may have consistent behavior for all 3 ≤ n ≤ 5, which may make the difficulty of learning any 3 ≤ n ≤ 5 steps similar.\" -> Did you do anything in order to find support for either of these two?\n\n- $\\mathcal{L}$ is for \"Life\"? You might want to mention that, because we see this symbol so many times associated with Loss that it takes some time getting used to that.\n\n**Related references** \n\nThe authors might consider referring to https://arxiv.org/pdf/2002.03896.pdf, https://arxiv.org/pdf/1903.12508.pdf and https://arxiv.org/pdf/1908.06663.pdf for other work learning on Life or Life-like environments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "vrizSK6F15j", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the problem of learning the rule for the game of life with a convolutional network. This is indeed an interesting problem that can be a great benchmark for studying the lottery ticket hypothesis and related training issues - kudos to the authors for this great idea. Unluckily, the methods and results of the paper do not stand up to the idea. While the game of life is an interesting problem, the methods employed to study it must be comparable to modern deep networks so that the results can be meaningfully interpreted in the broader context. The authors study convolutional models that are to learn multi-step processes, but they use neither residual connections in their models nor do they employ normalization or an appropriate multi-layer initialization scheme like fixup. There is also no study of the optimizer - Adam is used with the default Keras hyperparameters and no ablations are performed, even though other works find crucial benefits in tuning them - and also the epsilon parameter that's not even mentioned by the authors. Therefore, while we find the idea very promising, we need to recommend rejection of this paper until the methods are improved.", "extra_scores": {}, "extra_text": {}}, {"review_id": "POVvnyV5QEI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a numerical study of how well can deep learning frameworks learn the underlying rules of a discrete dynamical system when the training set is composed of pairs of configurations separated by some time interval. Specifically, they focus on the Game of Life and study the success rates of learning as a function of the time-interval, perturbations to the weights, and the over-parameterization. The latter is defined as the number of parameters divided by that of a certain ideal DNN that the authors hand-crafted. \n\nThey find that over-parametrization is useful when the time interval is small but, at least within their numerical reach, a time interval of 3+ Game of Life steps seems to be always hard to learn. In addition, they find that when the success rate is small taking a converged instance, flipping a weight, and continuing training has a drastic harmful effect on the success rate.\n\nI believe the above two numerical observations (hardness of 3+ steps and the drastic effect of a sign-flip on a converged instance) are interesting and may stimulate further numerical and analytical study. On the other hand, the lack of theory and the fact that these observations are shown in just one setting diminishes the potential impact of this paper. \n\nI suggest several specific points which the authors may wish to try and improve: \n\n1. It would be nice to see whether milder conditions of success, such as getting a sizable portion of the inputs exactly right, are met more often. In a related manner, I wonder if the authors can explain the second observation based on the fact that flipping a single weight, may result in a network which is perfectly accurate for some finite fractions of the inputs, whereas making this weight cross zero back to its original values must pass through a point which ruins the performance on this good finite fraction. \n\n2. The results and approach of this work have similarities with https://openreview.net/forum?id=BJGWO9k0Z, where the Game of Life is traded for a cellular automaton that percolates through a graph. I think the authors should relate both results as they may strengthen one another. \n\n3. It is unfortunate that the transition from success to failure is so sharp with the steps and makes it hard to deduce whether overparameterization improves performance for 3+ steps. Can the authors improve the quality of the numerics there? Perhaps by using more statistics or adopting a more relaxed criterion for success (such as in 1. for instance)? Perhaps by reducing the training-set size? In this aspect, the authors' definition of overparameterization is with respect to the handcrafted DNN. Some define overparameterization w.r.t. the size of the training-set,  as often happens in practical applications of DNNs. Can the authors comment what happens when they enter this latter over-parametrization regime? \n\n4. The authors argue that the success rate is highly sensitive to perturbations in the *initial state*. I see such sensitivity when sign-flipping the converged weights. However, I don't see a striking sensitivity when perturbing initial states (that is, one which is beyond that expected by the fact that the network rarely converges). Can the authors elaborate more on what they mean here? Is there some new information in the study of perturbation to the initial state which is not obvious from the success rate they find? \n\n\n\n\n\n\n\n\n\n \n\n    \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0ITyQrQHcLb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Paper Summary\n\nThis paper experimentally studies the problem of training a neural network to learn the rules of Conway's Game of Life from examples using gradient descent. The motivation is that since we can know and construct the minimal networks that can implement the Game of Life, it is interesting to study whether and under what conditions conventional gradient-based training of networks succeeds on this problem. The experiments show that training rarely works for the minimal network architectures, and substantial growth in network size is necessary to make training more reliable. Training is also very sensitive to initial network parameters and data distribution characteristics. Based on these results, the paper suggests that the Game of Life may serve as an interesting and useful test bed for studying general problems of neural network training.\n\n# Strengths\n\nThe paper is a well executed and clear experimental study on the problem of learning the Game of Life using neural networks. The experiments are based on simple genuine curiosity, and I found the results interesting (they are not predictable). There is potential for such experiments to lead to interesting ideas that we do not foresee yet.\n\n# Weaknesses\n\nWhile the experimental results are interesting, I found the contributions to the big picture issues of training neural networks to be unclear due to a couple of main reasons. \n\nA main connection made in the paper is to the lottery ticket hypothesis. The authors claim that since networks much larger than the minimal architecture are needed for reliable training, the findings support the lottery ticket hypothesis. I think there are definitely some relations because initialization plays a critical role here, the relevance of lottery ticket hypothesis is not very clear because of the differences in behavior for $n \\in \\{3, 4, 5\\}$ and $n \\in \\{1, 2\\}$ in Fig. 3. If the main issue was finding lucky subnetwork initializations, wouldn't we expect more consistent trends as Overcompleteness increases for all $n$?\n\nMoreover, I think that there is a very relevant line of prior work along similar lines that needs to be acknowledged and discussed here. [A] and [B] showed that certain synthetic problems that had been presented in prior work [C] as being very difficult for recurrent networks trained by gradient descent (due to long time lags) could actually be easily solved using random weight guessing. The key message was not that weight guessing is a good learning method, but that those benchmarks were not the right ones to focus on when looking for generalizable insights relevant to practical problems. This work is directly relevant here since it is also about the use of synthetic benchmarks to guide progress in learning algorithms.\n\n[A] Schmidhuber, J. and Hochreiter, S., 1996. Guessing can outperform many long time lag algorithms. (appears to be a TR for [B]) http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.45.1132\n\n[B] Hochreiter, S. and Schmidhuber, J., 1997. LSTM can solve hard long time lag problems. In Advances in neural information processing systems (pp. 473-479). http://papers.nips.cc/paper/1215-lstm-can-solve-hard-long-time-lag-problems.pdf\n\n[C] Bengio, Y., Simard, P. and Frasconi, P., 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2), pp.157-166.\n\n# Review Summary\n\nThe experimental study is very interesting, but I am hesitant to accept this paper due to the weaknesses mentioned above. I would be happy to re-consider my decision based on authors' responses to my concerns. In particular, how do the authors frame their work in the context of prior work that I mentioned? Is there a specific reason to believe that Life is not a \"trivial\" problem in the sense defined by [A,B], or is it simply an open question at this point? Relatedly, have they tried random weight guessing or variants for training the minimal architecture? Can the authors make any predictions relevant to the lottery ticket hypothesis based on their experimental results? In other words, how do these results indicate that if a method improves reliability of training on Life, we can also expect it to help on other practical problems?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Unfortunately, the authors did not submit a response during the rebuttal phase.
Decision: Reject