Continual Backprop: Stochastic Gradient Descent with Persistent Randomness

1 INTRODUCTION . In the last decade , deep learning methods have been successful and become the state-of-the-art in many machine learning problems and applications , including supervised classification , reinforcement learning ( Silver et al. , 2016 ) , computer vision ( Krizhevsky et al. , 2012 ) , and natural language processing ( Brown et al. , 2020 ) . These methods learn the weights of an artificial neural network using Backprop , which is primarily applied to stationary problems . However , a primary challenge to leveraging the strengths of deep learning beyond current applications is that Backprop does not work well in non-stationary problems ( McCloskey and Cohen , 1989 ; French , 1997 ; Sahoo et al. , 2018 ) , for example , a problem that consists a sequence of stationary problems . Leveraging the strengths of deep learning methods in non-stationary problems is important as many real-world applications of machine learning like robotics involve non-stationrities . Nonstationarities can arise due to changes in the environment ( Thurn , 1998 ) , high complexity ( Sutton et al. , 2007 ) , partial observability ( Khetarpal et al. , 2020 ) , or other actors ( Foerster et al. , 2018 ) . Many works have tried to make deep learning work in non-stationary problems . Some works have proposed methods that can remember previously learned information ( Kirkpatrick et al. , 2017 ; Aljundi et al. , 2018 ; Riemer et al. , 2019 ) . While , others have proposed methods that can adapt fast to nonstationarities ( Rusu et al. , 2016 ; Al-Shedivat et al. , 2018 ; Finn et al. , 2019 ) . A limitation of prior works on non-stationary problems is that they did not study cases where there are many non-stationarities . Number of non-stationarities refers to the number of times the data distribution changes . Most works only look at problems with less than ten non-stationarities ( Rusu et al. , 2016 ; Kirkpatrick et al. , 2017 ; Al-Shedivat et al. , 2018 ) . Finn et al . ( 2019 ) studied problems with up to a hundred non-stationarities . Dealing with many non-stationarities is important for systems that continually interact with the real world as non-stationarities frequently occur in the world . In this work , we study problems with a large number ( thousands ) of non-stationarities . We start with a special class of problems that we call semi-stationary problems . These are online supervised learning problems where the input distribution is non-stationary while the target function is stationary . The target function is the function being approximated , for example , the true regression function . These problems are a natural way to study problems with many non-stationarities ; slowly moving through a large input space can cause thousands of non-stationarities . These problems are also important for systems in the real world as inputs from the real world often depend on previous inputs , making the input distribution non-stationary . Finally , we study non-stationary RL problems . These are full non-stationarity problems as the input distribution changes when the agent ’ s behaviour changes and the target function , optimal policy , changes as the dynamics of the environment change . Semi-stationary problems reveal a new difficulty with Backprop ( BP ) in non-stationary problems ; they help us clearly understand one way in which Backprop fails . We show that in non-stationary problems , Backprop performs well initially , but surprisingly , its performance degrades substantially over time as Backprop loses its ability to adapt . Backprop relies on proper random initialization for its effectiveness ( Glorot et al. , 2010 ; Sutskerver et al. , 2013 ; He et al. , 2015 ) . However , randomization in Backprop only happens in the beginning . We hypothesize that Backprop ’ s ability to adapt degrades because the benefits of initial random distribution are not present at all times . To extend the benefits of initial random distribution throughout learning , we propose the Continual Backprop ( CBP ) algorithm . CBP uses a generate-and-test method to continually inject random features alongside SGD . We show that unlike BP , CBP can continually adapt in non-stationary problems . Our generate-and-test algorithm consists of two parts : the generator , which proposes new features , and the second is the tester , which finds and replaces low utility features with the features proposed by the generator . Our generate-and-test algorithm is built on the one proposed by Mahmood and Sutton , ( 2013 ) and our algorithm is compatible with modern deep learning . We overcome three significant limitations of their work . First , our algorithm is applicable to feed-forward networks with arbitrary shape and size while theirs was only applicable to networks with a single hidden layer and one output . Second , our algorithms works with modern activations and optimizers like Adam while theirs was limited to LTU activations , binary weights , and SGD . And third , we combine generate-and-test with SGD and show that the resulting algorithm , CBP , is significantly better than BP in complex non-stationary problems while their work was an isolated study of generate-and-test . The first contribution of our work is to show that in non-stationary problems with many nonstationarities BP loses its ability to adapt over time . In other words , we contribute to the understanding of why BP and its variants fails in non-stationary problems . Our second contribution is that we propose the CBP algorithm that extends the benefits of the initialization in BP to all times . 2 NON-STATIONARY PROBLEMS . We study Backprop , Continual Backprop , and other learning algorithms in semi-stationary and Reinforcement Learning ( RL ) problems . First , we consider a novel idealized semi-stationary problem . The strength of this problem is that in this problem we can study continual learning algorithms extensively and yet in a computationally inexpensive way and without the confounders that arise in more complex problems . Then we study the permuted MNIST problem , an online image classification problem , and two non-stationary RL problems . We demonstrate on these problems that the findings from the idealized problem scale to large neural networks in more realistic settings . Performances measure in semi-stationary problems In supervised learning , the task is to learn a function using examples of input-output pairs . This function is called the target function . In online supervised learning ( Orabona , 2019 ) , there is a stream of samples ( xt , yt ) , and the predictions have to be made sequentially . The performance measure is the loss on the next sample . Thus , learning and evaluation happen simultaneously . This is fundamentally different from offline supervised learning , where there are two separate phases , one for learning and another for evaluation . Another common measure in non-stationary problems is the performance on previously seen data . However , measuring performance on previously seen data is only meaningful when studying the catastrophic forgetting aspect of BP . As we do not study the forgetting problem , we do not measure the performance on old data . 2.1 BIT-FLIPPING PROBLEM . Our first problem is the Bit-Flipping problem . It differs slightly from most supervised learning in two ways . First , it is conventionally assumed that samples are independently and identically distributed , whereas we focus on the case where the sample at the current time-step depends on the previous sample . Second , it is often assumed that the learner has sufficient capacity to closely approximate the target function , whereas we assume that the target function is more complex than the learner . The best approximation continually changes in this problem as the input distribution is non-stationary , and the target function has high complexity . Therefore , there is a need for continual adaptation . The target function in the Bit-Flipping problem is represented by a multi-layered target network . The target network has two layers of weights . We limit the learning networks to networks with the same depth . This allows us to control the relative complexity of the target function and learner . If the target network has a lot more hidden units than the learner , then the target function is more complex than the learner . We set the target network to be wider . The input at time step t , xt , is a binary vector of size m. Here , xt ∈ { 0 , 1 } m where each element , xi , t ∈ { 0 , 1 } for i in 1 , ... , m. After every T time-steps , one of the first f bits is randomly selected and its value is flipped ; the values of these f bits do not change at other times . We refer to the first f bits as flipping bits . Each of the next m−f bits is randomly sampled from U { 0 , 1 } at every timestep . The value of T allows us to control the correlation among the consecutive values of the input . Note that when a flipping bit flips , the input distribution changes . We usem=20 , f=15 , and T =1e4 . In the target network , all weights are randomly sampled from U { −1 , 1 } . The activation function is a Linear Threshold Unit ( LTU ) , McCulloch ( 1943 ) . The output of an LTU , with input xt is 1 if ∑m+1 i=0 vixi , t > θi else 0 . Here , v is the input weight vector . We set θi = ( m + 1 ) ∗ β − Si , where Si is the number of input weights with the value of −1 and β ∈ [ 0 , 1 ] . This form of LTU is taken from Sutton and Whitehead , ( 1994 ) . We use β = 0.7 . The output of the network at time-step t is a scalar yt ∈ R. Figure 1 shows the input and the target network . The Bit-Flipping problem is a regression problem ; we use the squared error to measure performance . 2.2 PERMUTED MNIST We use an online variant of the Permuted MNIST problem ( Zenke et al. , 2017 ) . They used this problem with just 10 non-stationarities . This is an image classification problem with 10 classes . The images in permuted MNIST are generated from the images in the MNIST dataset by randomly permuting the pixels in the images . We present the images sequentially and measure online classification accuracy . The MNIST dataset has 60,000 images . We present these 60k images in random order , and after all the images have been presented , we use a single permutation to change all the images . This cycle of presenting all the 60,000 images and then changing the permutation of pixels can be continued indefinitely , which allows us to create a long-term continual learning problem . 2.3 NON-STATIONARY RL PROBLEMS . We use two non-stationary RL problems and both are continual variants of a corresponding Pybullet ( Coumans and Bai , 2016 ) problem . Non-stationary variants are needed as all the problems in Pybullet are stationary . In our problems , either the environment or the distribution of input changes after a pre-specified time , making it necessary for the learner to adapt . Slippery Ant : In this problem , we change the friction between the agent and the ground of the standard Pybullet Ant problem . In the standard problem , the value of the friction is 1.5 . We change the friction after every 10M time-steps by log-uniformly sampling it from [ 1e−4 , 1e4 ] . Semi-stationary Reacher : This problem is a modification of PyBullet ’ s Reacher problem . In Reacher , the goal is to get the tip of an arm to a target point in a square . At the beginning of each episode , a new target point is uniformly randomly chosen inside the square . Each episode lasts for 150 time-steps . In our problem , we change the distribution of input after every 30 episodes ( 4500 time-steps ) . We do so by dividing the square into 100 sub-squares , as shown in Figure 2 . At the beginning of each episode , the target point is randomly sampled from within a sub-square . This sub-square is changed after every 4500 time-steps when a new sub-square is randomly selected .

Summary: This paper demonstrates and proposes a solution for a new problem in continual learning which is the inverse of catastrophic forgetting. Compared to prior work, they study problems where the data distribution changes much more rapidly. They demonstrate that backpropagation based optimization loses its ability to adapt when tracking these rapidly changing continual learning problems.  They show a degradation in performance over time on permuated MNIST, non-stationary RL problems, and the bit-flipping problem. They propose a solution to this problem by reinitializing some portion of the weights of every layer. They propose a utility function to choose which layers to reinitialize based on a combination of adaptation-utility and contribution-utility. They demonstrate that utilizing this method, they can achieve better performance that does not degrade over time and that it works in more cases than l2 weight decay.