Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online
Authors: Yangchen Pan
Kirby Banman
Martha White
Venue: ICLR 2021 Poster (2021)
Keywords: Reinforcement learning
natural sparsity
sparse representation
fuzzy tiling activation function

Abstract: Recent work has shown that sparse representations---where only a small percentage of units are active---can significantly reduce interference. Those works, however, relied on relatively complex regularization or meta-learning approaches, that have only been used offline in a pre-training phase. In this work, we pursue a direction that achieves sparsity by design, rather than by learning. Specifically, we design an activation function that produces sparse representations deterministically by construction, and so is more amenable to online training. The idea relies on the simple approach of binning, but overcomes the two key limitations of binning: zero gradients for the flat regions almost everywhere,  and lost precision---reduced discrimination---due to coarse aggregation. We introduce a Fuzzy Tiling Activation (FTA) that provides non-negligible gradients and produces overlap between bins that improves discrimination. We first show that FTA is robust under covariate shift in a synthetic online supervised learning problem, where we can vary the level of correlation and drift. Then we move to the deep reinforcement learning setting and investigate both value-based and policy gradient algorithms that use neural networks with FTAs, in classic discrete control and Mujoco continuous control environments. We show that algorithms equipped with FTAs are able to learn a stable policy faster without needing target networks on most domains. 

Published as a conference paper at ICLR 2021
FUZZY TILING ACTIVATIONS: A SIMPLE APPROACH TO LEARNING SPARSE REPRESENTATIONS ONLINE
Yangchen Pan University of Alberta pan6 wet@uuabbert.ccmmm
Kirby Banman
Martha White University of Alberta whi tem@ua 1 be ee
University of Alberta
kdbanman ua berta...... ca
ABSTRACT
Recent work has shown that sparse representations--herr only a small percentage of units are active-can significantly reduce interference. Those works, however, relied on relatively complex regularization or meta-learning approaches, that have only been used offline in pre-training phase. In this work, we pursue direction that achieves sparsity by design, rather than by learning. Specifically, we design activation function that produces sparse representations deterministically by construction, and so is more amenable to online training. The idea relies on the simple approach of binning, but overcomes the two key limitations of binning: zero gradients for the flat regions almost everywhere, and lost precision-reduuced diserimination-due to coarse aggregation. We introduce Fuzzy Tiling Activation (FTA) that provides non-neglig gradients and produces overlap between bins that improves discrimination. We first show that FTA is robust under covariate shift in a synthetic online supervised learning problem, where we can vary the level of correlation and drift. Then we move to the deep reinforcement learning setting and investigate both value-based and policy gradient algorithms that use neural networks with FTAS, in classic discrete control and Mujoco continuous control environments. We show that algorithms equipped with FTAs are able to learn stable policy faster without needing target networks on most domains.
INTRODUCTION
Representation learning in online learning systems can strongly impact learning efficiency. both positively due to generalization but also negatively due interference (Liang et al., 2016: Heravi, 2019; Le al., 2017; Liu et al., 2019; Chandak al., 2019; Caselles-Dupré et al., 2018; Madjiheurem Toni, 2019). Neural networks partieularly suffer from interference-wherr updates for some inputs degrade accuracy for others others-when training on temporally correlated data (McCloskey & Cohen, 1989; French, 1999; Kemker et al., 2018).
Recent work (Liu et al., 2019; Ghiassian et al., 2020; Javed & White, 2019; Rafati & Noelle, 2019; Garrandez-Gariia Sutton, 2019), as well as older work (McCloskey & Cohen, 1989; French, 1991), have shown that sparse representation can reduce interference in training parameter updates. A sparse representation is one where only a small number features are active, for each input (Cheng et al., 2013). Each update only impacts small number of weights and so is less likely interfere with many state values. Further, when constrained to learn sparse features, the feature vectors are more likely to be orthogonal (Cover, 1965), which further mitigates interference. The learned features can still be highly expressive, and even more interpretable, as only a small number of attributes are active for a given input.
However, learning sparse representations online remains relatively open. Some previous work has relied on representations pre-trained before learning, either with regularizers that encourage sparsity (Tibshirani, 1996; Xiang et al., 2011: Liu et al., 2019) or with meta- meta-learning (Javed & White, 2019). Other work has trained the sparse-representation neural network online, by using sparsity regularizers online with replay buffers Aiernandez--arcca & Sutton, 2019) or using a winner-take-all strategy where all but the top activations are set to zero (Rafati & Noelle, 2019). Hermandez-Gariia &
"Code available htwpa///////ettttb..//aaaaaaaaaa http::/c.cccooooooooo//e/rerrere

---

Published as a conference paper at ICLR 2021
Sutton (2019) found that many of these sparsity regularizers were ineffective for obtaining sparse representations without high levels of dead neurons, though the regularizers did still often improve learning. The Shiner-Take--a on it's efficacy, some positive (Rafati & Noelle, 2019) and some negative (Liu et al., 2019). Finally, kemel representations can be used online, and when combined with WTA approach, provide sparse representations. There is some evidence that using only the closest prototypes- and setting kernel values zero for the further prototypes- not hurt approximation quality (Schlegel et al., 2017). However, kernel- based methods can be difficult to scale to large problems, due to computation and difficulties in finding a suitable distance metric. Providing a simpler approach obtain sparse representations, that are easy to train online, would make it easier for researchers from the broad online learning community to adopt sparse representations and further explore their utility.
(WTA) approach is -on-differemtiable, and there are mixed results
In this work, we pursue a strategy for what we call natural sparsity-an approach where we achieve sparsity by design rather than by encoding sparsity in the loss. We introduce an activation function that facilitates sparse representation learning in an end-to-end manner without the need of additional losses, pre-training or manual truncation. Specifically, we introduce a Fuzzy Tiling Activation (FTA) function that naturally produce sparse representation with controllable sparsity and can be conveniently used like other activation functions in a neural network. FTA relies on the idea of designing a differentiable approximate binning operation whhiee inputs aggregated into intervals. We prove that the FTA guarantees sparsity by construction. We empirically investigate the properties of FTA in an online supervised learning problem, where we can carefully control the level of correlation. We then empirically show FTA's practical utility in a more challenging online learning setting-the the deep Reinforcement Learning (RL) setting. On a variety of discrete and continuous control domains, deep RL algorithms using FTA can learn more quickly and stably compared to both those using ReLU activations and several online sparse representation learning approaches.
PROBLEM FORMULATION
FTA is a generic activation that can be applied in variety of settings. A distinct property of FTA is that it does not need to learn to ensure sparsity; instead, it provides an immediate, deterministic sparsity guarantee. We hypothesize that this property is suitable for handling highly nonstationary data in an online learning setting, where there is highly correlated data stream and a strong need for interference reduction. We therefore explicitly formalize two motivating problems: the online supervised learning problem and the reinforcement learning (RL) problem.
Online Supervised Learning problem setting. The agent observes a temporally correlated stream of data, generated by stochastic process {(x,, Yt) heN, where the observations X, depend on the past {X,-ihieN- In our supervised setting. X₁ depends only on X,-1, and the target Y, depends only on X,, according to a stationary underlying mean function f(x) = XX, = 2/|. On each time step, the agent observes X,, makes a prediction (X,),) with its parameterized function fa. receives target Y; and incurs a prediction error. The goal of the agent is to approximate function f-the ideal predictor- by learning from correlated data in an online manner, unlike standard supervised learning where data is independent and identically distributed (iid).
RL problem setting. We formalize the interaction using Markov decision processes (MDPs). An MDP consists of A, R. 7), where S is the state space, AA is the action space, IP is the transition probability kernel, R is the reward function, and y E [0. 1] is the discount factor. At each time step / = 1, 2,.... ... the agent observes state 8, E S and takes an action o, E A. Then the environment transits to the next state according to the transition probability distribution, i.e., the reward function R : S x A x S + IR A policy is a mapping from a state to an action (distribution) π. S. x A + [0, 1]. For a given state-action pair (s, a), the action-value function under policy π is defined as Qn (s, a) BSG,|S, s, A, a; A ~ T where G, Ere 7 R((s, at, $1+1) is the return of a sequence of transitions so, do, $1, a1, by following the policy π.
8141 IP(-|s,, dt), and the agent receives a scalar reward Tt-1 E IR according
The goal of an agent is to find an optimal policy that obtains maximal expected return from each state. The policy is either directly learned, as in policy gradient methods (Sutton et al., 1999; Sutton & Barto, 2018), or the action-values values are learned and the policy inferred by acting greedily with respect to the action-values, as in Q-learning (Watkins& & Dayan, 1992). In either setting, we often parameterize the policy/value function by neural network (NN). For example, Deep Q Networks (DQN) (Mnih
2

---

Published as a conference paper at ICLR 2021
al., 2015) parameterizes the action-value function Qo S A R by NN. The bootstrap target for updating state-action value is computed by using a separate target NN Qa- :S x A + R parameterized by = Yt T++1 max,'' Qs 15 a'). The target NN parameter 0 is updated by copying from every certain number time steps.
Online deep RL control problems can be highly nonstationary, for two primary reasons. First, the environment itself could be highly nonstationary, or alternatively, partially observable. Second, the data distribution is constantly shifting because of both the changing policy and shifting training targets. The latter can be mitigated by using a target NN as described above, which has become critical in successfully training many deep RL algorithms. However, it potentially slows learning as the new information is not immediately used to update action-valus valuus, instead, the slower moving and potentially out-dated target NN used. Several works reported that successful training without a target NN can improve sample efficiency of RI. algorithms (Liu et al., 2019; van Hasselt et al., 2018; Fan et al., 2020; Kim et al., 2019; Rafati & Noelle, 2019; Fan et al., 2020; Ghiassian al., 2020). We show that deep RL algorithms using activation is able to achieve superior performance without using a target NN, indicating the benefit of applying our method to nonstationary, online problems.
BINNING WITH NON-NEGLIGIBLA GRADIENTS
In this section, we develop the Fuzzy Tiling Activation (FTA), as new modular component for neural networks that provides sparse representations. We first introduce a new way to compute the binning of an input, using indicator functions. This activation provides guaranteed sparsity but has a gradient of zero almost everywhere. Then, we provide smoothed version, resulting in non-negligible gradients that make it compatible with paok-propagation algorithms. We then prove that the fuzzy version still guaranteed to provide sparse representation and the sparsity can be easily tuned.
3.1 TILING ACTIVATION
The tiling activation inputs a scalar z and outputs binned vector. This vector is one-hot, with a 1 in the bin corresponding to the value of z, and zeros elsewhere. Note that standard activation typically maps scalar to a scalar. However, the tiling activation maps a scalar to a vector, as depicted in Figure (()). This resembles tile coding, which inspires the name Tiling Activation; connection, we include a brief review of tile coding in the Appendix A. In this section, we show how to write the tiling activation compactly, using element-wise max and indicator functions.
this
0.75
0.00
1.00
(a) TA, 1: 44
(b) FTA, 4
(c) FTA, = 4, 0.25
Figure 1: a) The regular TA mapping R + R² with cach output element h; corresponds to a different bin. b) The FTA with > 0, permitting both overlap in activation, and nonzero gradient between the vertical red and gray lines. e) Larger values for 17 extends the sloped lines further from either side of each plateau, increasing the region that has neg-negli gradients.
Assume we are given range [1, u. for constants 1, u E R, where we expect the input z E [7. u]. The goal is to convert the input, a one-hot encoding, with evenly spaced bins of size d E B Without loss of generality, we assume that u / is evenly divisible by 8: if it is not, the range (4, u] could be slightly expanded, evenly on each side, to ensure divisibility. Define the k-dimensional tiling vector
def (L, + 5, + 25, ..., u 25, u 5).
where & = (u 11-1] 1)/5. The tiling activation defined as
(x)) 1 (max(c- - 2, 0) + max(2 8 - c, 0))
(2)
where (+) is an indicator function, which returns if the input is positive, and zero otherwise. The indicator function for vectors is applied elsment-wise. In Proposition 1, we prove that d returns
3

---

Published as a conference paper at ICLR 2021
binned encoding: if c; < < C,+1, then ((2) returns e; the one-hot (standard basis) vector with in the i-th entry and zero elsewhere. For values of & that fall on the boundary, 2 = C₁ the encoding returns vector with ones in both the i - 1th and ith entries. Consider the below example for intuition.
Example. Assume [1. u] = [0. 1] and set the tile width to å = 0.25. Then the tiling vector e has four tiles (A- 4): c (0, 0.25, 0.5, 0.75). If we apply the tiling activation to 0.3, because 0.25 < 0.3 < 0.5, the output should be (0, 1,0,0
0, 0). To see ((2) does in fact return this vector, we
compute each max term
max(c- 2, 0) = (0,0,0.2,0.45)
and max(2 - 6 - c, = max c, (0..0000 0,,05,,,00)
The addition of the two is (0.05, 0, 0.2, 0.45) and so 1 1+ (0.05, 0, 0.2, 0.45) 1 (1,0, (0, 1, 0, 0). The first max extracts those components in с that are strictly greater than 2, and the second max extracts those strictly less than z. The addition gives the bins that are strictly greater and strictly less than the bin for 2, leaving only the entry corresponding to that activated bin as O, with all others positive. The indicator function sets all nonzero entries to one and then using one minus this indicator function's ' output provides us the desired binary encoding. We rigorously characterize the possible output cases for the activation in the Appendix A.2.I.
3.2 FUZZY TILING ACTIVATION (FTA)
The Tiling Activation provides a way to obtain sparse, binary encodings for features learned within a NN. Unfortunately, the tiling activation has a zero derivative almost everywhere as visualized in Figure I(a). In this section, we provide a fuzzy tiling activation, that has non-zero derivatives and so amenable to use with backpropagation.
To design the FTA, we define the fuzzy indicator function ?
(,,((() (1 x)x (x (
where 17 is a small constant for controlling the sparsity. The first term (1 x) is if x < 17. and otherwise. The second term (1- is 1 if > 17, and 0 otherwise. If < 1), then T (x) r. and else (x) 1. The original indicator function 1 can be acquired by setting = 0. When 1j > 0, the derivative is non-zero for n,, and zero otherwise. Hence the derivative can be propagated
wards through those nonzero entries. Using this fuzzy indicator function, we define the following Fuzzy Tiling Activation (FTA)
(x) def 1 I (max(c 2, max(= - - c))
(4)
where again + is applied elementwise.
We depict FTA with differrnnt rys in Figure 3.1. For the smaller 77, the FTA extends the activation to the neighbouring bins. The activation in these neighbouring bins is sloped, resulting in non-zero deriva- tives. For this smaller 7,, however, there are still regions where the derivative is zero (e.g., 2 = 0.3 in Figure 1(b)). The regions where derivatives are non-zero can be expanded by increasing 17 as shown in Figure (c). Hence we can adjust 7) to control the sparsity level as we demonstrate in Section A.5. Figure 2 shows neural network with FTA applied to the second hidden layer and its output y is linear in the sparse representation. FTA itself does not introduce any new train- ing parameters, just like other activation func- tions. For input x, after computing first layer h = xW, we apply Φ, to h Wa E IRd to get the layer h2 of size kd. This layer consists of stacking the k-dimensional sparse encodings, for each element in h1 W2.
h2 FTFFFAAAAA A(hy W2)
h2
h ReLu(wWh)
W
h1
h W
y= hg W
ReLu
FTA
3.3 GUARANTEED SPARSITY FROM THE FTA
Figure 2: A visualization of an FTA layer
We now show that the FTA maintains one of the key properties of the tiling activation: sparsity. The distinction with many existing approaches is that our sparsity is guaranteed by design and hence is
TThe word fuzzy reflects that input can partially activate tile, with lower activation than 1, an analogy the concept of partial inclusion and degrees of memhership from fuzzy sets.
4

---

Published as a conference paper at ICLR 2021
not a probabilistic guarantee. We first characterize the vectors produced by the FTA, in Proposition with proof in Appendix A.2.2. Then, we provide an upper bound on the proportion of nonzero entries in the generated vector in Theorem 1, with in Appendix A.2.3.
Assumption 1. << 1, where kō u for k E N.
Theorem (Sparsity guarantee for FTA.). For any E [1 u|, (2((2) oulputs vector whose
number of nonzero entries [[),(())lo satisfies:
(z) S
7
+3
Corollary 1. Let o [0, be the desired sparsity level: the maximun proportion of nonzero entries of Φn (2).V2 u. Assume pk 2 3, i.e. some inputs have three active indices or more (even with 7 O, this minimal active number is 2). Then ni should be chosen such that
1 88_
kp - 3
equivalently n n 2
(Lkp) 1)
2
As an example, for k 100, 5 0.05 and a desired sparsity of least 10% (p = 0.1). we can use 0.05
0.225. Note that the bound in Theorem is loose in practice as the bound is for any input z. In Appendix A.2.3 and A.5, we theoretically and empirically show that the actual sparsity is usually lower than the upper bound, and quite consistent across inputs.
2 (L [100 0.1]
EXPERIMENTS IN SUPERVISED LEARNING UNDER COVARIATE SHIFT
In this section, we focus on testing the hypothesis that FTA provides representations that are more robust learning online correlated data. Specifically, we hypothesize that convergence speed and stability for ReLU networks suffer under strongly correlated training data, whereas comparable FTA networks are nearly unaffected. We create a synthetic supervised problem with a relatively simply target function, and focus the investigation on the impact of a drifting distribution inputs, which results both in covariate shift and creates temporal correlation during training. We also report results on two benchmark image classification tasks in the Appendix A.6.
The Piecewise Random Walk Problem has Gaussian X, ~ (V,S, 582) with fixed variance 3 and mean S, that drifts every T steps. More precisely, the mean S, stays fixed for T timesteps, then takes a step according to first order autoregressive random walk: S, = (1 c)S, + Ze where e E (0, 1 and Z, ~ (O0 o²) for fixed variance o². If e = 0, then this process is standard random walk; otherwise, with c < 1, it keeps S, in bounded range with high probability. For 1t ~ Xx, the training label y, is defined ye sin(2
This process is designed to modulate the level of correlation-whine we call correlation difficulty- without changing the equilibrium distribution over X,. As the correlation difficulty d varies from 0 to 1, the training data varies from low to high correlation: d = 0 recovers iid sampling. All d E (0, 1) share the same equilibrium distribution in X,, X, is ergodic and has Gaussian equilibrium distribution (0,0 )²), with variance ?² dependent upon 8 o² and c. In particular, the visitation distribution X, for any training run will converge the equilibrium distribution. This ensures that measuring loss with respect to the stationary distribution is fair comparison, because the visitation distribution of X, is identical across all settings. We depict sample trajectories with low and high d in Figure 3(a). For a rigorous construction of X, and S,, and justification for this equilibrium distribution and implementation details, see Appendix A.7.
We measure the mean squared error over the equilibrium distribution in X,, for neural networks using FTA and ReLU activations across range of correlation difficulty values. In Figure 3, we can see that FTA outperforms ReLU in two ways. First, FTA converges lower loss with less variance across all correlation difficulties. Second, FTA only marginally suffers under high difficulties d' > 0.9, whereas the performance of ReLU begins to deteriorate for relatively mild d 0.5.
Note that the FTA reaches a lower error, even on iid data (d = 0). We hypothesize this gap arises because the networks are trained online, with one sample from each X, being used for each weight update. Figure 23 in the Appendix supports this hypothesis, with the gap vanishing in an identical experiment where 50 samples drawn from each X,.
5

---

Published as a conference paper at ICLR 2021
FTA ReUU
05
Mean Final
60-
-45
Xr
-16-
S, EDX.]
AtD.
S. EXXJ
A10. E)
Training Iterations
ste
3000
Equil. Density
Training Iterations
3000
Equil. Density
Covartate Shit Dificuty
(a) Mild Correlation
(b) High Correlation
(c) ReLU and FTA, d [0.
Figure 3: (a) and (b) contain sample trajectories of X, which are (a) mildly correlated with d = 0.41 and (b) severely correlated with d = 0.98. Both share the same equilibrium distribution (in gray). (c) A plot of the prediction error (under the equilibrium distribution), averaged over the final 2.5K iterations, across range of difficulty settings. All networks are trained for 20k online updates. The lines correspond the mean of 30 runs, with the shaded region corresponding 99. 99 confidence intervals. The iid setting d = 0 is shown as a dotted line for baseline comparison.
EXPERIMENTAL RESULTS IN REINFORCEMENT LEARNING
In this section, we empirically study the effect of using FTA in RL. First, show overall performance on several benchmark discrete and continuous control environments. Second, we compare our method with other simple strategies obtain sparse representations. Third, we provide insight into different hyper-pane parameter choices of FTA and suggest potential future directions. Appendix A.4 includes details for reproducing experiments and Appendix A.5 has additional RL experiments.
5.1 ALGORITHMS AND NAMING CONVENTIONS
All the algorithms use a two-layer neural network, with the primary difference being the activation used the last hidden layer. See Appendix A.5 for results using FTA in all the hidden layers. DQN is used for the discrete action environments, and Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016) for continuous action environments. On each step, all algorithms sample mini-batch size of 64 from an experience replay (Lin, 1992; Mnih et al., 2015) buffer with maximum size set [7, u] [-20, 20),
100k. Note that we keep the same FTA setting across all experiments:
5 = 7 2.0, and hence (-2 18, -6, ..., 18). k 40/2 20.
We first compare standard DQN agents, with the same architectures except the last layer.
DQN: DQN with tanh or ReIU on the last layer (best performance reported).
DQN-FTA:
DQN with FTA on the last layer.
DQN-Large: DQN, but with the last layer of the same size as DQN-FTA. If DQN has a last (i.c., the second) hidden layer of size d, then DQN-FTA has a last layer of size dk,, since the FTA activation simply expands the number of features due to binning. Hence, we include DQN-Large with the same feature dimension as DQN-FTA in the last hidden layer. Note that DQN-Large has & times more parameters in this last hidden layer than DQN or DQN-FTA.
We also compare to several simple strategies to obtain local or sparse features. Radial basis functions (RBFs) have traditionally been used to obtain local features in RI., and recent work has used &g and l regularization directly on activations as a simple baseline (Arpit et al., 2016; Liu et al., 2019). All of these strategies have the same sized last layer the sparse feature dimension of DQN-FTA.
DQN-RBF: DQN using radial basis functions (RBFs) on the last layer, with the centers defined by the FTA: (b() [exp (-(2 (2-c1))) )...))
... exp (- (=- (=-ea)²) where is the bandwidth parameter.
DQN-L2/L1: DQN with & or & regularization on the activation functions of the final hidden layer where there is the same number of units as that in DQN-Large.
To the best of our knowledge, no suitable sparse representation approaches exist for RL. SR-NNs for RL were only developed for offline training (Liu et al., 2019). That work also showed that k-sparse NNs (Makhzani & Frey, 2013) and Kinner-Take-Ale NNs (Makhzani & Frey, 2015) performed significantly worse than l, regularization, where E actually performed quite well in most of their
Please refer to Figure 2. FTA does not augment the number of training parameters in the hidden layer where it is applied; it only augments the training parameters in the immediately next layer.

---

Published as a conference paper at ICLR 2021
-30
300
200
1000
-20
DOM-TA
D2N
-3500
200
-102
DONlane
Time Stpge
363
161
165
Acrobot (v1)
(b) MountainCar (vO)
CartPole (v1)
(d) LunarLanner (v2)
Evaluation learning curves of DON-FTA(black). DQN(red), and Lar- showing episodic return versus environment time steps. The dotted line indicates algorithms trained with target networks. The results are averaged over 20 runs and the shading indicates standard error. The learning curve smoothed over a window of size 10 before averaging across runs.
Figure 4:
experiments. One other possible option is to use Tile Coding NNs (Ghiassian et al., 2020), which first tile code inputs before feeding them into the neural network. This approach focuses on using discretization to break, overcome, incorrect generalization in the inputs; their goal is not to learn sparse representations. This approach is complementary. in that it could be added to all the agents in this work. Nonetheless, because it is of the only papers with lightweight approach to mitigate interference in online RL, we do compare to it in the Appendix A.5.2.
5.2 OVERALL PERFORMANCE
In this section, we demonstrate the overall performance of using FTAs on both discrete and continuous control environments. Our goals are to investigate if we 1) obtain improved performance with FTA, with fixed parameter choices across different domains; 2) improve stability in learning with FTA; and 3) see if that learning can in fact be faster without target networks and so that it is beneficial to use FTA without them. All experiments averaged over 20 runs (20 different random seeds), with offline evaluation performed on the policy every 1000 training/environment time steps.
remove the need to use target networks with FTA, including determining
20ta
100
2
(a) InvertedPendu
(b) Hopper
(c) Walker2d
(d) InvertedDouble
(e) Swimmer
Evaluation learning curves of DDPG-FTA(black), DDPG(red), and DDPG-Large(blue)) averaged over 10 runs with shading indicating standard error. The dotted line indicates algorithms trained with target networks. The learning curve is smoothed over window of size before averaging across runs.
Figure 5:
Discrete control. We compare performance on four discrete-action environments from Ope- nAI (Brockman et al., 2016): MountainCar, CartPole, Acrobot and LunarLanndrr. We use 64 x 64 ReLU hidden units on all these domains. Since FTA has = 20, this yields 64 x 20 = 1280 dimensional sparse features. Hence, DQN-Large, use two layers ReLU with size 64 1280.
We plot evaluation learning curves, averaged over 20 runs, in Figure 4. The results show the following. 1) With or without using target network, DQN with FTA can significantly outperform the version without using FTA. 2) FTA has significantly lower variability across runs (smaller standard errors) in most of the figures. 3) DQN-FTA trained without target network outperforms DQN-FTA trained with target network, which indicates potential gain by removing the target network. 4) Without using FTA, DQN trained without target network cannot perform well general, providing further evidence for the utility of sparse feature highlighted in previous works (Liu et al., 2019; Rafati Noelle, 2019). 5) Simply using a larger neural network does not obtain the same performance improvements, and in some cases significantly degrades performance. The exception to these conclusions LunarLander, where DQN-FTA performed similarly to DQN and actually performed slightly better with target networks. On deeper investigation, we found that this was due to using tiling bound 20, 20]. which we discuss further in Section 5.4. When using [-1, 1]. DQN-FTA performs much better and results consistent with the other domains.
7

---

Published as a conference paper at ICLR 2021
Continuous control. We compare performance continuous control tasks from Mujoco (Todorov et al., 2012). For these experiments, we use DDPG with exactly the same FTA settings as in the above discrete control domains show its generality. Corresponding to the discrete control domains, apply FTA to the critic network and do not use a target network to train it. As seen in Figure 5, on most domains, DDPG-FTA achieves comparable and sometimes significantly better performance to DDPG with target network. However, Figure 5 (c) highlights that FTA is not always sufficient on its own to achieve superior performance. Factors such as the exploration strategy, other hyper-parametent and neural network architecture may also play an important role on such challenging tasks.
5.3 COMPARISON WITH OTHER REPRESENTATION LEARNING APPROACHESS
FTA provides clear benefits, but it is natural to ask other simple strategies that provide sparse or local features could have provided similar benefits. We compare to DQN-RBE, DQN-L2 and DQN-LI, on the discrete action environments, shown in Figure 6. We find the following. 1) FTA performs consistently well across all environments using fixed parameter setting, none of the other approaches achieve consistent performance, even though we tuned their parameters per environment. 2) Both the & and &g approaches have a high variance across different random seeds. 3) The RBF variant can do better than the & and & approaches but is worse than our algorithm. It is a known issue that RBF is sensitive the bandwidth parameter choice and we observe similar phenomenon. It is also known that the exponential activation can be problematic in the propagaattt process. We empirically investigate the overlap and instance sparsities of each algorithm the Appendix A.5.
-R2-
100-
506
100
DOVA22
splices
DQWL
200
BQWARE
DOUFTA
100
-200
-30%
Time Stps
165
383
563
16
(a) Acrobot (v1)
(b) MountainCar (vO)
(c) CartPole (v1)
(d) LunarLander (v2)
Evaluation learning curves of DON-FTA(black), DON-RBF green), DQN- L2(red), and DQN-LI(blue), averaging over 20 runs with the shade indicating standard error. All algorithms are trained without using target networks. The learning curve is smoothed over a window of size 10 before averaging across runs.
Figure 6:
5.4 CHOOSING THE HYPER- PAPEERPARRMETERS FOR FTA
In this section, provide some insight into selecting the dyper-parametent of FTA. We argue that 5 can be used as a rule of thumb. Then we show that given an appropriate tiling bound (i.e., value), DQN-FTA performs well, with reasonably large number of bins, i.e., reasonably small tile width 8. However, we do observe certain level of sensitivity to the tiling bound, which may partially explain the worse performance of DQN-FTA on LunarLander and some Mujoco domains.
Sparsity control parameter. The purpose of the parameter is to provide a nonzero gradient for training an NN via backpropagation. Since FTA is an one-to-many mapping, gives a nonzero gradient as long as any single element in the resulting vector provides nonzero gradient. Setting 17 8 (i.e.. the tile width) is the minimum value to guarantee nonzero gradient as we visualize in the Figure 1(c) as long as FTA's input is within the tiling bound. In all of our experiments, we fix y d unless otherwise specified. In the Appendix A.5.4, we show that DQN-FTA does reasonably well across a broad range of 17, d parameter settings.
Number of tiles/tile width. In Figure 7(c), we show that on LunarLander, with u = 1.0, DQN- FTA performs well when the tile width 6 2u/k is reasonably small (i.e., number of tiles k is reasonably large). We show the learning curves of DQN-FTA with u = 1.0, in Figure 7(a), by using number of tiles from k E {2, 4. 6, 8, 10, 12, 14. 16), providing sparse feature dimension 64 x k E (128, 256, 384, 512, 640, 768, 896, 1024]. We also examine the performance of DQN trained by using NN size 64 x k, for k E {2, 4, 6, 8, 10, 12, 14, 16). In Figure 7(b), one can
8

---

Published as a conference paper at ICLR 2021
320
223
1.0
220
235
120
-220
-270
-100
-320
-200
-470
(a) Vary # of tiles k
(b) Vary of hidden units
(c) Bound sensitivity
(d) Sparsity ratio
Figure 7: (a) Evaluation learning curves showing episodic return versus environment time steps of DQN-FTA using different number of tiles lo as labeled. This equivalent to varying tilc width 6 5 2u/k. The results are averaged over 5 random seeds. (b) Evaluation learning curves of DQN without using a target NN as we change the number of the second hidden layer units as labeled in the figure. (c) Evaluation learning curves of DQN-FTA uses [-u, u/ as tiling bound where u E {0.01, 1. 1.0, 10.0, 20.0]. The results are averaged over 10 runs. The standard error is not shown but is sufficient small differentiate the two groups (i.e., [0.1, 1.0) and {0.01, 10.0, 20.0)) corresponding to appropriate bound and too large/small bound. To generate this figure, we fix on using FTA with 20 tiles (i.e., tile width 6 = 2u 20). (d) The overlap-insteen instancce sparsity ratio v.s. environment time steps. The standard error is very small and is ignored. The results are averaged over 10 runs.
that increasing NN size does not provide a clear benefit to DQN; however, DQN-FTA performs significantly better as & increases, in Figure 7(a).
Sensitivity to tiling bound. We find that the tiling bound [1. u] can be sensitive. Intuitively, if we set I, u extremely small, then the input of FTA may go out of the boundary and FTA provides zero gradient again. When we set the bound too large, many inputs may hit the same bins, both resulting in many dead neurons and increasing interference. In Figure 7(c), we see that very small u = 0.01 and big 10, 20 perform poorly in LunarLanderr and interim values of 0.1, perform well.
We further examine the corresponding representation interference measured by the overlap sparsity divided by instance sparsity. in Figure 7(d). Instance sparsity is the proportion of nonzero entries in the feature vector for each instance. Overlap sparsity (French, 1991) is defined as overlap(á, d')
I # 0) + ()/(k)), given two td-dimensional sparse vectors o, &'. Low overlap sparsity poientially indicates less feature interference between different input samples. Overlap sparsity divided by instance sparsity represents the proportion of overlapped entries among activated ones. We can see in Figure 7(d) that large u increases this ratio, indicating increased interference, which may explain the worse performance of DQN-FTA with u 10, in Figure 7(c). On the other extreme, 0.01, the sparsity ratio is low but performance is poor. This is likely because generalization actually becomes too low at this level, reducing sample efficiency. We refer to Appendix A.5.6 for additional experiments about gradient interference.
when the bound is too small,
6 DISCUSSION
In this work, proposed the idea of natural sparsity, aiming at achieving sparsity without learning in deep learning setting. We design an activation by drawing on the idea of binning which produces one-hot encoding of an input. We provide a Fuzzy Tiling Activation (FTA) with a sparsity control mechanism that enables backpropagation of gradients, and potentially improves generalization by increasing active feature units. We show that the FTA still has sparsity guarantees, related to the choice of 1). The FTA provides sparse representations by construction, and so it is much easier to use when learning online than conventional sparse representation learning methods. We highlight that FTA is robust to high levels of covariate shift, in a synthetic supervised learning problem. We then show across several discrete and continuous control RL environments that using the FTA significantly improve learning efficiency and stability, and in most cases even removes the need of target networks.
Our work suggests several promising future research directions. The first choices for the tiling bound. An adaptive approach for the tiling bound is particularly relevant for case-of-use, as we found more sensitivity to this choice. Second, FTA is actually complementary to approaches (Liu et al., 2018; Riemer et al., 2019; Javed & White, 2019) designed to explicitly mitigate interference. An interesting direction is to combine FTA with these methods to reduce interference. Last, it is important to study how should balance generalization and discrimination. We observe that an algorithm's 's performance degrades when either the overlap sparsity is too high or too low. Learning sparse representation with appropriate overlap sparsity may be most desirable.
further investigate the

---

Published as a conference paper at ICLR 2021
7 ACKNOWLEDGEMENTS
We would like to thank all anonymous reviewers for their helpful feedback. We thank Han Wang for carefully reading our paper and providing helpful suggestions to improve our presentation. We acknowledge the funding from the Canada CIFAR Al Chairs program and Alberta Machine Intelligence Institute.
REFERENCES
Abadi, M. and et. al. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from lensorflow.org. 2015.
Arpit, D., Zhou, Y., Ngo, H., and Govindaraju, Why regularized auto-encoders learn sparse representation? In International Conference on Machine Learning, pp. 136-144, 2016.
Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, pp. 15854, 2019.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv, 2016.
Caselles-Dupré, H., Ortiz, M. G., and Filliat, D. Continual state representation learning for reinforce- ment learning using generative replay. arXiv:1810.03880. 2018.
Chandak, Y., Theocharous, G., Kostas, J., Jordan, S., and Thomas, P. Learning action representations for reinforcement learning. In International Conference on Machine Learning, pp. 941-950, 2019.
Cheng, H., Liu, Z., Yang, L.,, and Chen, X. Sparse representation and learning in visual recognition: Theory and applications. Signal Processing, pp. 1408 1425, 2013. Special issue Machine Learning in Intelligent Image Processing.
Cover, T. M. Geometrical and Statistical Properties of Systems of Linear Inequalities with Applica- tions in Pattern Recognition. IEEE Trans. Electronic Computers, 1965.
Fan, J., Wang, Z., Xie, Y., and Yang, Z. A theoretical analysis of deep q-learning. In Proceedings of the 2nd Conference on Learning for Dynamics and Control, pp. 486-489, 2020.
French, R. M. Using semi-distributed representations to overcome catastrophic forgetting in connec- tionist networks. Annual Cognitive Science Society Conference, 1991.
French, R. M. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, pp. 128-135, 1999.
Ghiassian, S., Rafiee, B., Lo, L., and White, A. Improving performance in reinforcement learning by breaking generalization in neural networks. International Conference Autonomous Agents and Multi-agent Systems, 2020.
Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. International Conference on Artificial Intelligence and Statistics, 2010.
Grunwald, G., Hyndman, R., and Tedesco, L. A unified view of linear ar (1) models. 1995.
Gu, S., Lillicrap, T. P., Sutskever, I., and Levine, S. Continuous Deep Q-Learning with Model-based Acceleration. In International Conference on Machine Learning, pp. 2829-2838, 2016.
Heravi, J. R. Learning Representations in Reinforcement Learning. PhD thesis, University of California, Merced, 2019.
Hernandez-Garcia, J. F. and Sutton, R. S. Learning sparse representations incrementally in deep reinforcement learning. Workshop on Continual Learning at Advances in Neural Information Processing Systems, 2019.
10

---

Published as a conference paper at ICLR 2021
Javed, K. and White, M. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, pp. 1818-1828. 2019.
Kemker, R., McClure, M., Abitino, A., Hayes, L., and Kanan, C. Measuring catastrophic forgetting in neural networks. In AAAI conference on Artificial Intelligence, 2018.
Kim, S., Asadi, K., Littman, M., and Konidaris, G. Deepmellow: Removing the need for a target network deep q-learning. International Joint Conference on Artificial Intelligence, pp. 2733- 2739, 2019.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
Le, L.,, Kumaraswamy, R., and White, M. Learning sparse representations in reinforcement learning with sparse coding. International Joint Conference on Artificial Intelligence, pp. 2067-2073, 2017.
LeCun, and Cortes, C. MNIST handwritten digit database. http://yaanllccc..ccm//xx//////e 2010.
Leurent, E. environment for autonomous driving decision-making 2018. URL https://
github.
Liang, Y. Machado, M. C., Talvitie, E., and Bowling, M. State of the control of atari games using shallow reinforcement learning. International Conference on Autonomous Agents & Multiagent Systems, pp. 485 493, 2016.
Lillicrap. T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Lin, L.-J. Self-Improving Reactive Agents Based On Reinforcement Learning, Learnin Planning and Teaching. Machine Learning, 1992.
Liu, V., Kumaraswamy, R., Le, L., and White, M. The utility of sparse representations for control in reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4384 4391, 2019.
Liu, X., Masana, M., Herranz, L., Van de Weijer, J., López, A. M., and Bagdanov, A. D. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In International Conference on Pattern Recognition, pp. 2262-2268, 2018.
Madjiheurem, S. and Toni, L. Representation learning on graphs: A reinforcement learning applica- tion. International Conference on Artificial Intelligence and Statistics, 2019.
Makhzani, A. and Frey, B. k-sparse autoencoders. arXiv: 1312.5663, 2013.
Makhzani, A. and Frey, B. Winner-take-all autoencoders. In Advances in Neural Information Processing Systems, 2015.
McCloskey, M. and Cohen, J. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation, 1989.
Mnih, V., Kavukcuoglu, K., Silver. D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M.. Fidjeland, K., Ostrovski, G., and others. Human-level control through deep reinforcement learning. Nature, 2015.
Moore, A. W. and Atkeson, G. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, pp. 103-130, 1993.
Pan, Y., Zaheer, M., White, A., Patterson, A., and White, M. Organizing experience: a deeper look at replay mechanisms for bample-based planning in continuous state domains. In International Joint Conference Artificial Intelligence, pp. 4800, 2018.
11

---

Published as a conference paper at ICLR 2021
Pan, Y., Yao, H.,, Farahmand, A.- and White, M. Hill climbing on value estimates for search-contro in dyna. International Joint Conference on Artificial Intelligence, 2019.
Pan, Y., Mei, J., and massoud Farahmand, A. Frequency-based search-control in dyna. International Conference Learning Representations, 2020.
Rafati, and Noelle, orXiv:/909.0157/ 2019.
Learning sparse representations in reinforcement learning.
Riemer, M.. Cases, I., Ajemian, R., Liu, M., Rish, L., Tu, Y., and Tesauro, G. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference Learning Representations, 2019.
Schlegel, M., Pan, Y., Chen, J., and White, M. Adapting Kernel Representations Online Using
Submodular Maximization. In International Conference on Machine Learning, 2017.
Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bulletin, 2(4):160-163, 1991.
Sutton, R. S. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems 8, pp. 1038-1044 MIT Press, 1996.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Sutton, S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforce- ment learning with function approximation. In International Conference on Neural Information Processing Systems, 1999.
Sutton, R. S., Szepesvári, C., Geramifard, A., and Bowling, M. Dyna-style planning with linear function approximation and prioritized sweeping. In Conference Uncertainty in Artificial Intelligence, pp. 528-536, 2008.
Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, pp. 267-288, 1996.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: physics engine for model-based control. In International Conference Intelligent Robots and Systems, 2012.
Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, Deep reinforcement learning and the deadly triad. 11X2v:1812.02660 2018.
Watkins, C. C. H. and Dayan, P. Q-learning. Machine Learning, 1992.
Xiang, Z., Xu, H., and Ramadge, J. Learning sparse representations of high dimensional data on large scale dictionaries. Advances in Neural Information Processing Systems, 2011.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. 2017.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
This appendix includes the following contents:
1. Section A.1 briefly reviews tile coding which inspires FTA and the naming.
2. Section A.2 shows the proofs for theorems about sparsity guarantee in this paper.
3. Section A.3 discusses alternative way handle the case when the inputs of FTA go out of the boundary of the tiling vector c.
4. Section A.4 includes experimental details of Section 5 for reproducible research.
5. Section A.5 presents additional experiments in reinforcement learning setting.
6. Section A.6 reports the results of FTA two popular image classification datasets: Mnist (LeCun & Cortes, 2010) and Mnistfashion (Xiao et al., 2017).
7. Section A.7 includes the details of the synthetic supervised learning experiment from Section
A.1 TILE CODING REVIEW
We give a brief review of tile coding here, as tile coding inspires our Fuzzy Tiling Activation (and its naming).
Tile coding is a generalization of state aggregation, that uses multiple tilings (aggregations) to improve discrimination. For input [0, 1J, state-aggregation would map a one-hot vector with in the corresponding bin (bin can be also called tile), with k the number of bins discretized into intervals of length 6. In tile coding, multiple such tilings of the input are concatenated, where each tiling is offset by a small amount. This is depicted in Figure 8, where we show two tilings, one covering from [-0.05, 1] and the other from [0.0, 1.05] both with feature vector is a concatenation of these two tilings, to produce 8 features. Two nearby inputs 0.3 and =' 0.55 would be aggregated together in state-aggregation. making it impossible to produce different values for those inputs. Tile coding, on the other hand, would cause them to share one feature in the second tiling, but each have a distinct feature that enables them to have different values. The large bins/tiles allows for some information sharing, for fast generalization; and the overlapping tilings allows for improved discrimination.
and 8 1.05/4 0.2625. The resulting
Figure 8: Tile coding maps a scalar to an di- mensional binary vector. For example, = activates the second tile on both tilings which gives the fea- ture vector (0, 1,0, 1,0, Similarly, z' (0. 0, 0,0,0,
tiling2 tiling1
-0.05
1.05
Unfortunately, tile coding can scale exponentially in the dimension of the inputs. Each dimension is discretized to granularity of 5 (k regions), with the cross-product between all these regions resulting in R. bins. One natural approach to make binning more scalable and more practically usable, is to combine it with neural networks (NN). Our activation function Junction-Fuzzyy Tiling Activation-enabless differentiable binning operation.
It is known that those simple domains which are able use tile coding never require separate slowly moving weight vector (i.e., the counterpart of a target network in linear function approximation setting) (Sutton, 1996; Sutton Barto, 2018). Based on this observation, Liu al. (2019) indicates that in a deep learning setting, when using sparse representation. the target network can be removed and the sample efficiency can improved, coinciding with many of our empirical results. It is sensible that the target network possibly slower down learning. Because at each environment time step, the new information is not immediately utilized to estimate the bootstrap target and this could slower down learning. Accurately estimating bootstrap targets may be highly beneficial in Dyna- style model-based reinforcement learning Sutton (1991); Sutton et al. (2008), as the planning stage typically involves efficient improvement of value estimates (Moore & Atkeson, 1993; Pan et al., 2018; Gu et al., 2016; Pan et al., 2019). It may be an interesting future direction to carefully study the effect of removing the target network in Dyna-style planning.
13

---

Published as a conference paper at ICLR 2021
A.2 PROOFS
We now provide the proof for Proposition 1, Proposition 2, Theorem and Corollary below.
A.2.1 PROOF FOR PROPOSITION
For convenience, we define the k-dimensional one-hot vector e, whose ith entry one and otherwise zero.
Proposition! 1. Under Assumption foror any 2 E [1, u/,
1. Ifc, < for [K or cs then ((2) e;
2. If 2 = c, for ( k), then ((z) = e; + te
3. If C1, then ()=) el
Proof. In all thee cases, the first max operation in Φ is max(c- 2, 0) (0, 0, Ci+1 Ci+2 2, .,... Cj: 2).
To understand the output of the second max operation, we look the three cases separately.
Case 1: Because c; < 2 < C; we know that C,-1 < = - < C;. This implies the second max operation in Φ is:
max(= - 8 c, = - 6 C1, C2,
c;-1, 0,0,, .).
Therefore, the sum of both operations max 2, 0) max(= everywhere except the ith position, which is zero. Hence 4. (max(c z, 0) + max(= (= - 6 c, 0)) gives vector where every entry is except the ith entry which is 0. Then I (max(c - 2, 2, max(z - 8 c, 0)) = e, Case 2: C,, E (2, ...; k),, then == =
c, 0) has positive elements
-, and
max(= - 8 - 0)
(z - 8 C1 2 5 -c, .... 2 c;- 2,0,,.0...... O,...... 0).
follows that max(c - 2, 0) + max (= - c. has exactly two zero entries, indices i and
i. This gives 1 - T+ (max 2, 0) max(= - c, 0)) - e; vector with ones at indices
i i- and i.
Case 3: When Cı, max(= ô c, 0) is a zero vector and max(c 0) is positive everywhere
except the first entry, which is zero. Again this gives 1 I (max(c 2, 0) + max(2 5 - ccc e1-
A.2.2 PROOF FOR PROPOSITION
Proposition 2. Let I be the set of indices where (bz) is active (from Theorem I we know it can only contain one or two indices). Under Assumption 1, for any 2 E [L, u), the function (n(2)= = 6(z) A. where A is mostly zero, with few non-zero entries. The vector & has the following properties:
is zero at indices E I, iee, (z(2) equals (2z) at indices E II.
2. A is non-zero at indices {ili $ I.j E [kt, < = - - Cj < n. C
Proof. Part 1. Let the ith entry be active in (f2). We have one of the three cases hold as stated in Theorem 1. Assume C; z Ci+1. Note that
max(c - 2, 0) (0, 0, Ci+1 ",..., Cj. - 2) max(= - 8 0) (= 5 Ci..... - 5 C, O,... 0),
taking the sum of the above two equations gives vector as following:
Ci,... 8 C;-1, 0, Ci+1 - 2,... Ck
(6)
Then applying I,,+0 to vector equation 6 gives us a vector retaining elements < 17 and all other elements become 1. Hence the ith position is after applying T (+) to vector equation 6. Using
14

---

Published as a conference paper at ICLR 2021
minus this vector would give us vector with only the ith is one. But this is exactly Φn(2). And
since (2(2) = o(z) + A, and the ith entry of ((2) is also one, the ith entry of A must be zero. Similar reasoning applies to the cases when C, 2 {2.... k} or C₁.
Part 2.
Note that applying L, (-) to the vector max c- 2, 0) max -- d-c. c. 0) keeps all elements no
more than 17 and making all other elements one. As result, 1 + (max(c- 2, 0) max((= -- 0)) would give a vector zero everywhere except those entries in max(c - 2, 0) + max(2 - 5 c, c,0) which are S v). The set of indices which are I 1) in the vector max(c- 2, 0) max(z 8 c,
can be written as {ilá E [k],0 0 < 2 8 C₁ S 17, 0 < Cj
S n). which is also the set of
indices where (,,(2) is nonzero. Since (2) Φ(2) + A and ((2) has nonzero entries in the indices I and A has zero values at those entries by Part 1, then A must have nonzero entries at {ili E J. E K),0 < δ Cj S n, S n).
A.2.3 PROOF FOR THEOREM
Theorem 1. Sparsity guarantee for FTA. For any 2 E [1, u). n7 > 0. (x(2) outputs vector whose number of nonzero entries (=)llo satisfies:
(1),()) 2 1 +3
Proof. Similar the proof of Theorem we divide to three cases to prove the result.
Case 1. Consider the case that C; < 2 < C,+1, E [k 1]. Note that the number of nonzero entries
in Φh,, is equal the number of entries less than 1 in the vector equation 6, hence we can count the number of entries less than 1 in (z
CI - ô- Ci-1, 0, Ci+1
Ci+2 2, ..., Cs - 2).
First, we count the number of entries that are less than or equal to y on the left side of the ith position.
Since the äth position zero, which indicates = - - c, < O, hence 2 - 5 C,-1 5 < 0 and it follows that 0 < 8 C 8. Then
z δ C,-1 + 5 = 8 C < 28. Hence
G- 1)6 < z - - Ci-j j j E {1, [1..... i - -1.. Assume there are m entries S 7) on the the left
side of the ith. Then z 5 Ci-m I 17. It follows (m 1)8 < v) and hence m S 7 + 1. Hence
the total number of elements S on the left side of the ith position is at most 1 + 1.
Second, count the number of entries S 17 on the right side of the ith position. Since c, -2 < 0 < C2 + 8 on the right side of ith position is at most [ +
C++1 < 5, 0 < Ci+2
< 25, .... Hence the possible number of entries S
As result, together with the ith position which is 0 S n) the possible number of nonzero entries in this case is at most _ +3.
Case 2. When C;, E (2, .e....... A... we count the number of entries less than 1y in the vector
max(c - 0) + max(: - 8 c, 0) ((i - (16 ... 25, 0, 6, 26, ..., i)8)
Again, attempt to count the number of entries less than 1) in this vector by considering (i 1),, 26, 5 and 5, 25, (k- i)5 respectively.
We follow the exactly same argument above and now we have two zero entries at i 1, ith positions. The difference is that, the number of entries less than 7 in the vector ((i - 1)6, .... 26, 5 can be at most [ As result, the number of nonzero entries in this case is still at most 2 _ +
Case 3. When : = C1, max(2 - - c, 0) is a zero vector and max(c 2.0) + max(2 5 - c, 0) is positive everywhere except the first entry, which is zero. Then we simply count the number of entries
< 1 on the right side of the Oth position, i.e. j E {1, {1,...,kk kj). Similar to the analysis in the above Case 1, the possible number of entries < 17 on the right side of Oth position is most [ Hence in this case, the number of nonzero entries is at most ! + 1.
In summary, the number of nonzero entries does not exceed 2 +33 This completes the proof.
Remark. As we empirically demonstrated in Table 1 and below Figure 10 and Figure 9, the actual sparsity achieved by FTA is lower than the upper bound. This is because our upper bound is for any possible input z. Consider that in Case I in the above proof, count the number of entries that
15

---

Published as a conference paper at ICLR 2021
less than equal to 1 on the left side and right side of the ith position. There are L + 1 such entries on both sides only when z is exactly equal to S which is unlikely to happen in practice. 2
A.2.4 PROOF FOR COROLLARY 1
Corollary Let E (0,1) be the desired sparsity level: the maximum proportion of nonzero entries of o,, (2), E 1, Assume pk 2 3, i.e., some inputs have three active indices more (even with ?) = this minimal active number is 2). Then ii should be chosen such that
7 88
kp - 3
5 equivalently n n (Lkpl 1)
2
Proof. Because (2)lo < kk],, from Theorem it is sufficient to pick such that 2 + <
[kp] < kp. This gives 1 S ([kp) -3)/2 S (kp - 3)/2. Additionally, we know ? -15 13]
(Lkp] 3)/2, giving the second inequality.
A.3 MORE DISCUSSION ABOUT FTA
Our development of the FTA assumed that the inputs are bounded in the range [L, ul (recall that [1. | 5, 26,..., u)). This is not guaranteed for the standard inputs to activations, namely 2 x w for some weight vector w. The FTA can still be used if = & [1, uj., but then the gradient of the FTA will be zero. This means that the weights w cannot be adjusted for inputs where becomes too large or too small. This issue is usually called gradient vanish, which is common in many popular, existing activation functions such as ReLU, tanh, sigmoid, etc.
In our main paper, we proposed to use different tiling vectors (i.e., es) with a broad range of bounds to different components in the input vector of FTA reduce the chance of vanishing gradients. Here we discuss two other possible ways to avoid this issue: 1) use squashing activation before handing to the FTA and 2) regularize (penalize) z that falls outside the range [7. u). For example, tanh can be applied first to produce
Though the simplest strategy, using tanh function can be problematic in deep neural networks due to gradient vanishing problems. An alternative is to
tanh(x w) [-1,1]. 1).
use penalty, that pushes out-of-boundary back into the chosen range [-u, uj
r(=) def [(|=| > u) o |2|
(8)
This penalty is easily added to the loss, giving gradient ar(=) = [(2 u) I(2 <(=<<1)) For example, dz
z = max(x w, 0), which might produce value greater than u. If > u, then the gradient pushes the weights w to decrease 2. It should be noted that the number of nonzero entries can only decrease when 2 goes out of boundary. However, in our experiments in our main paper, we found such out of boundary loss is unnecessary on all of our tested domains. Furthermore, we further verify that our FTA performs stably across different weights for the above regularization in Section A.5.
A.4 REPRODUCING EXPERIMENTS FROM SECTION
Common settings. All discrete action domains are from OpenAl Gym (Brockman et al., 2016) with version 0.14.0. Deep learning implementation is based on tensorflow with version 1.13.0 (Abadi & et. al, 2015). We use Adam optimizer (Kingma & Ba. 2015), Xavier initializer (Glorot & Bengio, 2010), mini-batch size b = 64, buffer size 100k, and discount rate ? 0.99 across all experiments. We evaluate each algorithm every 1 training/enviromment time steps.
Algorithmic details. We use 64 x 64 ReLU units neural network for DQN and 200 x 100 ReLU units neural network for DDPG. All activation functions are ReLU except: the output layer of the Q value linear. The weights output layers were initialized from a uniform distribution [-0.003, 0.003]. Note that we keep the same FTA setting across all experiments: we set [1, u| = [-2, 20]; we set 5 = 7 2.0, c = {20 - 18, - -16, ..., 18), and hence k = 40/2 = 20. This indicates that the DQN-L and DDPG- versions have 64 x 20- 1280 and 100 x 20 2000 ReLU units in the second hidden layer. For RBF coding, we set the bandwidth as a = 2.0, and uses the same tiling (i.e. vector) as FTA.
16

---

Published as a conference paper at ICLR 2021
Meta-parameter details. For DQN, the learning rate is 0.0001 and the target network is updated every 1k steps. For DDPG, the target network moving rate is 0.001 and the actor network learning rate is 0.0001, critic network learning rate is 0.001. For [ la regularization variants, we optimize its regularization weight from (0.1,0.01, 0.001, 0.00011 on MountainCar, then we fix the chosen optimal weight 0.01 across all domains.
Environmental details on discrete control domains. We set the episode length limit as 2000 for MountainCar and keep all other episode limit default settings. We use warm-up steps 5000 for populating the experience replay buffer before training. Exploration noise is 0.1 without decaying. During policy evaluation, we keep a small noise c 0.05 when taking action.
Environmental details on continuous control domains. On Mujoco domains, we use default settings for maximum episodic length. We use 5, 000 warm-up time steps to populate the experience replay buffer. The exploration noise is as suggested in the original paper by Lillicrap et al. (2016).
A.5 ADDITIONAL EXPERIMENTS ON REINFORCEMENT LEARNING PROBLEMS
This section shows the following additional empirical results which cannot be put in the main body due to space limitations.
1 empirically investigates the representation sparsity generated by FTA.
A.5.2 shows the comparison with Tile Coding NNs (Ghiassian et al., 2020), which first tile code inputs before feeding them into a neural network.
A.5.3 shows the empirical results of DQN-FTA using linear activation function with regular- ization weights E {0.0, 0.01, 1.0} for the out of bound loss in Eq 8.
A.5.4 shows the sensitivity of FTA to the sparsity control parameter v and tile width parameter 8.
A.5.5 shows the results when applying FTA to both hidden layers rather than just the second hidden layer in RL problems.
A.5.6 shows gradient interference analysis result.
A.5.7 shows the result of using FTA on an autonomous driving application.
A.5.1 EMPIRICALLY MEASURING SPARSITY
We also report two sparsity measures of the learned representations averaged across all steps in Table 1. We compute an estimate of sparsity by sampling a mini-batch of samples from experience replay buffer and taking the average of them. Instance sparsity corresponds to proportion of nonzero entries in the feature vector for each instance. Overlap sparsity French (1991) is defined as overlap(ó, ø') E, T(o # ( + ()/(k)), given two kdd-dimensional sparse vectors d,, d'. Low overlap sparsity potentially indicates less feature interference between different input samples. Theorem 1 guarantees that the instance sparsity. with FTA, should be no more than 12.5% for our setting of k = 40, 1 = ó; and should be no more than 25% when k = 20, 17- 8. In the table, we (better) instance sparsity than the upper bound provided by the theorem. Further, FTA achieves the lowest instance and overlap sparsity among all baselines in average sense.
that FTA has lower
Figure and Figure 10 are corresponding to the learning curves as shown in Figure 6 in Section 5.3. We show learning curves of instance/overlap sparsity as a function of training steps for FTA, 4, l2 regularization and RBF The instance sparsity is computed by randomly sampling a mini-batch of states from replay buffer and count the average number of nonzero entries in that mini-batch divided by feature dimension kd. The overlap sparsity is computed by randomly sampling two mini- mini-batches of states from the replay buffer and count the average number of simultaneously activated (i.c. nonzero) entries in both mini-batches, From the sparsity learning curves, one can see that our FTA has very stable sparsity since the beginning of the learning, and the sparsity is almost constant cross domains. Particularly, the overlap sparsity is very low, which possibly indicates low representation interference.
A.5.2 COMPARISON WITH TC-NN
We include the figure of comparing with TC-NN from the work by Ghiassian et al. (2020), which use regular tile coding process the input before feeding into the neural network. It should be noted
17

---

Published as a conference paper at ICLR 2021
Table 1: Sparsity on LunarLander (average across time steps)
Sparsity FTA(k=40)
FTA(k=20)
LI
L2
RBF
Instance 7%
14%
16%
44%
99%
Overlap
4%
8%
10%
34%
99%
J2W-2
26
0.8
DON-1
66
0.5
30U-307
70V-77
02
0.2
02
20
15
20
20
10
(a) Acrobot-vl
(b) MountainCar-v0
(c) CartPole-vl
(d) LunarLander-v2
Figure 9: Instance sparsity v.s. number of time steps on MountainCar, CartPole, Acrobot, LunarLan-
der. The results are averaged over 20 random seeds and the shade indicates standard error.
D2W-2
304-1
201 201
2.6
D0V -1A
0.4
0.2
20
0.5
1-3
06
0.
1.0
15
20
25
65
20
3-4
200
23
30
60
(a) Acrobot-vl
(b) MountainCar-vo
(c) CartPole-vl
(d) LunarLander-v2
Figure 10: Overlap sparsity (number of simultaneously activated entries in the sparse feature vectors)
number of time steps by averaging over 20 random seeds and the shade indicates standard error.
18

---

Published as a conference paper at ICLR 2021
-500
DQN-L2
11000
DQN-L1
DQN-RBF
1500
DON-FTA
TC-NN
2000
0
3
4
1e5
Figure 11: DQN-FTA compares with TCNN. Evaluation learning curves are averaged over 20 random seeds and the shade indicates standard error. All variants trained without target networks.
-500
102
-1000
0.01
-3500
-100
mg0.0
-50
70
2000
165
(a) Acrohot-vl
(b) MountainCar-vo
(c) CartPole-vl
(d) Lunarl ander-v2
Figure 12: DQN-FTA trained with linear activation function with different regularization weight (-reg-1.) 1.0 means regularization is set 1.0). One that our FTA is not sensitive this choice as those learning curves are almost overlapping with each other. Evaluation learning curves are averaged over 10 random seeds and the shade indicates standard error. All variants trained without target networks.
that this method itself is not a sparse representation learning technique, and requires the original observation space be scaled within [0, 1]. Figure 11 shows that TC-NN does indeed improve performance for DQN, but not compared to FTA. DQN-FTA both learns faster and reaches a better, more stable solution. In our TC-NN implementation, as a correspondent FTA, use binning operation to turn each raw input variable to 20 binary variables through binning operation. We use the same learning rate 0.0001 other baselines. The result generated by using FTA with single tiling: [1, u] 20, 20), 5 = r = 2.0, and hence {-20, - 18, - -16, 18), k 40/2 = 20.
A.5.3 INSENSITIVITY TO OUT OF BOUNDARY LOSS
We demonstrate the effect of using out of boundary loss discussed in Section A.3. This is supplementary to our experiments in the main body, where we did not use an out of boundary loss, ie. the regularization weight is 0. To highlight the issue of out of boundary,
intentionally use a tiling vector с with a small range [-1,1] 1] with 20 tiles, i.e. = 2/200 = 0.1 and set the sparsity control parameter 7 8 0.1. In Figure 12, one can see that algorithm typically does need such a regularization even if we use a small tiling.
A.5.4 FTA'S STRONG PERFORMANCE WITH DIFFERENT TILE WIDTHS AND SPARSITY CONTROL PARAMETERS
The purpose of this section to show that given an appropriate bound the tiling vector c, our approach is insensitive to tile width and sparsity control parameter 7). Since already showed in Section that DQN-FTA works well on LumarLander with tiling vector bound [-1, 1], we fix using this setting in this section. For both 1 and 8, sweep over 0.8/2', i {0, 1, .... 8). Note that this range extreme (8 0.8 gives two bins, and = 0.8 significantly increases overlap); we do so to clear pattern.
19

---

Published as a conference paper at ICLR 2021
Figure 13 shows the early learning performance of all possible 9 9 81 combinations of 17, 8. We report the average episodic return (rounded to integers) within the first 300k time steps, for each combination. The results averaged over 5 runs. The pattern is expected. 1) The algorithms performs best with a reasonably small 5 and f/. 2) For extremely small and 6, performance is poor, expected because the gradient is very small and so learning is slow. 3) Given fixed tile width ò, the performance degrades as becomes smaller than 6, since again the activation has many regions with zero derivative. If gets too large, the sparsity level increases and again performance degrades, though performance remained quite good for a broad range between 0.025 and 0.2. 5) If gets large (big bins), performance degrades more so that with larger 7/, which matches the intuition that 1/ provides overlap rather than just increasing bin size and so losing precision.
-104
24
15
30
62
-72
-85
-99
-123
117
31
73
67
15
42
12
-99
51
5
61
108
121
124
41
16
48
148
-24
114
149
264
138
10
-30
-81
-86
31
96
153
173
175
-77
-105
-93
eta
153
115
121
29
116
74
-127
138
8
11
-65
-25
-24
25
-118
155
tile width
Figure 13: Sensitivity of 1,, ā on LunarLander--v.
A.5.5 EMPIRICAL RESULTS WHEN USING FTA TO BOTH HIDDEN LAYERS
As an activation function, FTA can be naturally applied in any hidden layer in a NN. As convention, we typically use the same activation functions fully connected NN. In this section, we present such results on the reinforcement learning domains, setting the same those in Section except that we apply FTA to both hidden layers in DQN-FTA and DDPG-FTA. It should be noted that, in this case. DQN-FTA and DDPG-FTA have the same number of training parameters as their -Large correspondents respectively. Figure 14 shows the results the discrete domains, and Figure 15 shows the results on the continuous control domains. can be seen that, the FTA versions are better than ReLu versions in the case of not using target network. This further validates the utility of our FTA in dealing with nonstationary problems.
supplement to the results in Section 5. All
A.5.6 GRADIENT INTERFERENCE ANALYSIS
We provide gradient interference analysis in this section. The results in this section are pro- duced by using FTA with single tiling: [7. u] [0, 20),
2.0, and hence
20, 18, - 166 ..... 18), k = 40/2- 20.
We consider three measures for gradient interference. Let la be the DQN loss parameterized by 0. Given two random samples (i.e. experiences) X, X' from ER buffer, we estimate
1. m1: o[Vola (X) (Xl)(")), this is to measure on average, whether the algorithm general- izes positively interfere.
( 2. m2: ([V))( Velo ONLY for those pairs of gradient vectors who have negative
inner product. This is to check for those likely interfered gradient directions, how much they interfere with each other.
3. m3: Within a minibatch of pairs of (X),', the proportion of negative gradient inner products (i.e. if 32 out of 64 pairs has negative inner products, then the proportion is 50%).
20

---

Published as a conference paper at ICLR 2021
202
leut
-350
-200
D0M
80
1.0
22
25
(a) Acrobot (v1)
(b) MountainCar (vO)
(c) CartPole (v1)
(d) LunarLander (v2)
Figure 14:
Evaluation learning curves of of DQN-FTA DQN(red), and DQN-Large(bluee showing
episodic return versus environment time steps. The dotted line indicates algorithms trained with target networks. The results are averaged over runs the shading indicates standard error. The learning curve smoothed over a window size 10 before averaging across runs.
spocie
(a) InvertedPendu
(b) Hopper
(c) Walker2d
(d) InvertedDouble
(e) Swimmer
Evaluation learning curves of DDPG-FTA(hlack), (DPG(r (DPGeLarge(blue)) on Mujaco environments, averaged over 20 runs with shading indicating standard error. All algorithms are trained
Figure 15:
without target networks. The learning curve is smoothed over window of size 30 before averaging across runs.
This is to roughly check how likely it is to get conflict gradient directions by randomly drawing two experiences from ER buffer.
It should be noted that normalize the gradient vectors unit length before computing inner product. For each point in the learning curve, we randomly draw 64 samples from the ER buffer to estimate the interference.
Figure 17 shows the algorithms' performances in terms of number of time steps taken to reach near-optimal policy: DQN-FTA DQN-RBF> DQN-LI DQN-L2 > DQN (i.e. means better (use fewer time steps) and 2 means slightly better).
There are several interesting observations. First, in Figure 16(b)(e). for the weights in the second hidden layer (m2 measurement), DQN-RBF has lower interference strength than DQN-FTA, but DQN-FTA performs better. This discrepancy indicates that is not necessarily true that the lower interference, the better. Second, DQN tends to over generalize during early learning, which may hurt and result bad performance. Third, figure (e) shows that DQN-LI is similar to DQN-FTA during late learning stage in terms of m2. But DQN-LI seems to have extremely interfered gradient directions during early learning, which may explain why it learns slower than DQN-FTA.
These observations may indicate that DQN-FTA helps generalize appropriately. but not overly generalize or highly interfered. We believe is worth separate work to thoroughly study the gradient interference issue draw some interesting conclusions.
A.5.7 TESTING STABILITY IN A SIMULATED AUTONOMOUS DRIVING DOMAIN
Our results have shown improved stability with FTA. In this section, we test FTA in an environment focused on stability, namely an autonomous driving task (Leurent, 2018). In the real world, stable policy is of vital importance to ensure safety in autonomous driving. In this simulated task, the goal not only obtain high return, but also keep the number car crashes as low as possible. FTA setting is the same as those in Section 5 on discrete control domains. Figure 18(a) shows the domain, where the agent-the green car har-has to learn to switch lanes, avoid car crashes, and go as fast as possible. The observations 25-dimensional, with vehicle dynamics that follow the Kinematic Bicycle Model. The action space is discrete.
FTA learns faster, with significantly fewer crashes incurred during the evaluation time steps as shown in Figure 18(c). Target networks are harmful in this environment, potentially because they slow early learning; the agent accumulate significant number of crashes before improving.
21

---

Published as a conference paper at ICLR 2021
DON-12
DQN-22
0.5
0.6
DON RDF
-0.2
DQN RDF
DON
0.4
DON-FTA
-0.4
DON-FTA
DON-12
-0.6
DQN-RBF
-0.8
0.1
DON
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
165
165
1a5
(a) ml
(b) m2
(c) m3
1.0
DON-L2
0.0
0.5
DON-L1
DON-LI
RBF
RBF
DON
-0.4
DON
DON-FTA
DON-FTA
0.4
00.6
0.2
DON-L1
DQN RBF
0.1
0.0
-1.0
DQN
0.5
2.0
1.0
0.0
0.5
1.5
2.0
105
165
165
(d) ml in 2nd
m2 in 2nd
(1) m3 in 2nd
Figure 16: Gradient inference measurements on mountain car domains, averaging over 10 runs with the shade indicating standard error. (f)(e)(f) are measured for those parameters in the second hidden layer only (which are supposed to directly affect representation). All algorithms are trained without using target networks except DQN. The curve is smoothed over a window size 20 before averaging across runs.
-100
DQN-L2
DQN-L1
DQN-RBF
DQN-FTA
-200
0.5
1.0
1.5
2.0
1e5
Figure 17:
Evaluation learning curves on mountain car. The results are averaged over 20 runs with the shade indicating standard error. All algorithms are trained without using target networks.
A.6 RESULTS ON IMAGE CLASSIFICATION TASKS
We now report the empirical results on two popular image classification tasks: MNIST (LeCun Cortes, 2010) and Mnistfashion (Xiao et al., 2017). We found that FTA does not present any clear advantage or disadvantage such a conventional supervised learning setting.
On MNIST, FTA achieves testing error 1.22%. and ReLu achieves 1.38%. On Mnistfashion, FTA achieves testing error 10.67%, and ReLu achieves 10.87%.
Details. The NN architecture we use is two convolution layer followed by two fully connected 32 x 32 layers. The first convolutional layer has filters with size 5 x 5 and followed by max pooling operation. The second convolutional layer has 16 filters with the same size followed by a max pooling. FTA is applied to the second hidden layer and the setting is exactly the same used in the RI. experiment A.4, To optimize learning rate, optimize over the range
22

---

Published as a conference paper at ICLR 2021
Camadth
pcr
DON-TA
D0N-TTA
DON
20N
Time Suge
Time Stezo
(a) Highway
(b) Evaluation LC
(c) Car Crashes LC
Figure 18: (a) The Highway environment, (b) The evaluation learning curve. (c) The cumulative number of car crashes as function of driving time steps. Results are averaged over 30 runs with shading indicating standard error. The learning curve is smoothed over window of size 10 before averaging across runs.
{0.00003, 0.00001, 0.0003, 0.0001, 0.003, 0.001}. We use 10-fold cross validation to choose the best learning rate and the above error rate is reported on testing set by using the optimal learning rate at the end of learning. The standard deviation of the testing error (randomness comes from NN initialization and random shufiling) is sufficiently small to get ignored.
A.7 CONSTRUCTION PIECEWISE RANDOM WALK PROBLEM
In Section 4, recall that we train with data generating process {(X,, Yt) so that interference difficulty is controllable, while permitting fair comparison via fixed equilibrium distribution.
The temporal correlation high difficulty {Xi)ee is designed to mimic state space trajectories through MDPs with high degree of local state space connectedness. For example, agent can only move to adjacent cells in GridWorld, the paddle and ball Atari Breakout can only move so far in single frame, and most successor states in Go only differ by few pieces. Figure 19 depicts sample trajectories across a range of difficulties, alongside the fixed equilibrium distribution.
0.1
66
X; Sy FLXi
-10
Sy EEA
MB. 5
-1.0
e
-10
ARO,
Rerations
Equil Dersity
Training
200
Equil. Dersity
Training Iterationn ans
130
Equil. Density
(a) 0=0
(b) 0.21
(c) d 0.41
-20
S, (RR1
A00.29)
-20
.79.E9)
Sy EXJ
A10, E*)
300
180
Equil
sia
190 Equi.
sic Training Iterations
1850 Equil. Density
Training
Densty
Training Iberations
Denelty
(d) d = 0.62
(e) d = 0.83
(f) d = 0.98
Figure 19: Sample trajectories of {X, JEN (blue) and HEN (black) across range of difficulties. Note in particular the fixed equilibrium distribution (gray) and i.i.d sampling for = 0 (top left).
Here rigorously construct (Xe and (St) and show they have the claimed properties.
Let random variable E, ~ (N(e; 02) be a source of Gaussian noise for the recursive random variable S,
23

---

Published as a conference paper at ICLR 2021
S411 = (1 c)S; - E.
If e E (0, 1], then {St)ee a linear Gaussian first order autoregressive process, denoted AR(1). Then the equilibrium distribution of heN also Gaussian. If So is sampled from the equilibrium distribution, then the process HeN is stationary with [ and variance
2²
2² = E|S
(10)
2c-c c²
It also follows that (Se) is ergodic. See Grunwald et al. (1995) for reference AR(1) processes, especially the discussion of equation (3.3) for linear Gaussian AR(1) processes, and section 5.3 for their stationarity and ergodicity.
Let 8t be realization of St. We define X, Sc S2 as a Gaussian r.v. with mean and variance g²:
S. 8t N 8t,
(11)
The process HeN is also a linear Gaussian AR(1) process, so we can again rely established AR(1) process theory to conclude that the equilibrium distribution (X,)eN Gaussian. Moreover, if Xo is sampled from the equilibrium distribution, then the process {Xi)IeN is stationary with
E X = and variance g2
a²
e = E[X? = 3 1 +B+
(12)
2c c
Refer to Theorem 4 in Section A.7.3 for proof that (ei is linear Gaussian AR(1) process, and that the resulting equilibrium distribution has variance specified by equation 12. Note there are several variance terms in the above construction.
0² is the noise variance in the random walk, i.e. the source of jumps in {Sa heN
2 is the equilibrium distributon variance of {St)ee
32 the variance in any particular X, given 8; at time step
< is the equilibrium distribution variance of {Xe)ee
The processes {S², heN and (X₂ Jee are fully determined by the quantities c, 0. 8 For our experiments we wish have single parameter such that the equilibrium distribution of (X, fixed for all d E [0, 1), but temporal correlation variable. Also, d= 0 should correspond to zero temporal correlation. The following two theorems specify this behaviour more rigorously.
Let difficulty parameter [0, 1) and g2 (2)2 for some B E TR+ Then (Xt)en will have fixed equilibrium distribution N (0, {?), invariant to d, parameters e, o, , are set as follows
Theorem 2.
-
0² d B 2
8 = (1 d) 2
<2 is defined in terms of B E B is high probability bound with P(X, E [- B)) < 0.95 for all 1, since (X.hen ergodic.
+ simply because B is intuitive design parameter. In partieular,
Theorem 3. = 0 and S. = induces i..d X, from Nx 0, ) the equilibrium distribution of (X.)ee
c O. then S. is a simple Gaussian random walk. We do not use such a process, because sample paths unbounded, hitting + infinitely often as 0o
24

---

Published as a conference paper at ICLR 2021
We defer proof of both theorems to Section A.7.3.
Correlation difficulty is intuitively characterized as follows.
As increases, 8 decreases, so smaller portion of the overall state space is supported at any given time. i.e. P(X+ = x| = 84) becomes increasingly narrow, inducing higher temporal correlation.
As increases, the noise in random walk {St increases in amplitue, so that larger jumps in the mean of any particular X, are more likely.
At d = 0, all correlation difficulty from {Xi)ieN has been removed, in that recover iid sampling.
As + 1, hEN converges towards pathological correlation, with X, 4 0(s)), where 5(-) is the Dirac delta distribution. i.c. X, becomes constant everywhere except the jumps in realization 8, of S..
Despite the above relationships between d and the other process parameters, the equilibrium distribu- tion (x;; 0, g²) is identical for all difficulty settings d [0, 1), because the increase (or decrease) in parameters o², is tuned specifically to counteract the decrease (or increase) in , Having identical cquilibrium distributions means that a hypothetical ideal algorithm, capable of perfectly mitigating interference, would train the identical approximator fo for any correlation difficulty d E [0, 1). Hence, we use approximation loss on the equilibrium distribution as a measure of robustness to interference.
To summarize, correlation difficulty d E [o, 1) controls the likelihood of interference-inducin samples throughout training, but also permits fair comparison between values of d by the (fixed) equilibrium distribution of {Xe]
A.7.1 PIECEWISE RANDOM WALK ADDITIONAL PLOTS
Figure 20 depicts learning curves of loss training time at different levels of correlation difficulty. Figure 21 shows learning rate sensitivity curves for the FTA and ReLU networks for three different difficulty settings. The ADAM optimizer Kingma & Ba (2015) was used in all experiments.
For the experiment in Figure 3, where one sample from each X, is used for each weight update, FTA outperforms ReLU even on i.i.d data (d = 0). In order to find the conditions under which ReLU and FTA both perform equally well, we repeat the same experiment, but with 50 samples drawn from each X, Figure 23 depicts the results, where the performance is indeed equalized for iid sampling (d = 0). but ReLU's performance still diverges similarly to other experiments as correlation difficulty increased.
The size of the FTA and ReLU neural networks used on the Piecewise Random Walk Problem were chosen based on best performance on iid data, and the best performers do not have the same number of parameters. (67K vs 5.2K learnable parameters.) In order to verify that the difference in parameter count was not a significant factor in handling high correlation difficulty, we additionally include results for a ReI.U network with wider hidden layers totalling 81 K learnable parameters. See Figure 22. Both ReLU networks perform very similarly- in most cases agrecing within p = with-withh loss steeply running away as correlation difficulty increases. So we conclude FTA's robustness to high correlation difficulties is not explained simply by parameter count.
A.7.2 PIECEWISE RANDOM WALK PROBLEM HYPERPARAMETER SELECTION
Experimental Parameters
Configuration for Piecewise Random Walk problem across all experiments:
25

---

Published as a conference paper at ICLR 2021
d-0 88
d-0
FTA ReLU
Mean Loss
Over
200 2004 U00 Training
2000
Training
1004
1968
2000
SA (c) ReLU and FTA, d E [0, 1)
Difcutty
10
(a) FTA
(b) ReLU
Figure 20: Left, Middle: Learning curve of loss over stationary distribution during training on low difficulty (dotted) and high difficulty (solid) settings for two layer neural nets. The curves smoothed over a window of 50. Right: The final loss over stationary distribution after 20K training iterations across range of difficulty settings, shown the mean of 30 runs with the shaded region corresponding to = 0.001. The final loss per run is computed as the mean over the final 2.5K iterations, with the iid setting d (dotted) shown for baseline comparison.
Mean
Mean Finil
Hean Fiial
d' 0.00
dm 0.41
d' 0.98
FTA
22th
(a) Lowest Difficulty
(b) Moderate Difficulty
High Difficulty
Figure 21: Learning rate sensitivity of FTA and ReLU for iid, mildly correlated, and severely correlated X, (left, middle, right, respectively.) Final loss performance is shown as the mean of 30 runs with the shaded region corresponding p = 0.001. These curves corroborate our findings that, in general, FTA prefers lower learning rates (but converges more quickly nonetheless.)
Y1
d-0
ReLü
Mean
RelU (aange)
Loss
Finsl Loss
list.
Training Rerations
Training Berations
Coveriate Shilt Difiesity
(a) FTA
(b) ReL.Ularre
(c) ReI.U-large and FTA, d E [0, 1)
Figure 22: Left, Middle: Learning curve for single using loss stationary distribution during training on low difficulty (dotted) and high difficulty (solid) settings for two layer neural nets, both having similar numbers of learnable parameters. The curves smoothed The final loss over stationary distribution after 20K training iterations across a range of difficulty settings, shown the mean of 10 runs with the shaded region corresponding 0.05. The final loss per run is computed as the mean over the final 2.5K iterations, with the iid setting d = 0 (dotted) shown for baseline comparison.
window of 50. Right:
26

---

Published as a conference paper at ICLR 2021
FTA
ReLU
8.5-
Mean Final Loss
0.5
000
Covariate Shift Difficulty
Figure 23:
The final loss over stationary distribution after 20K training iterations across a range of difficulty settings, shown the mean of 10 runs with the shaded region corresponding 0.001. For each weight update, a batch of 50 samples are drawn from each Xt, rather than a single sample done in other similar figures.
Iterations per training rum Test batch size per iteration Train batch size per iteration Min, max difficulty settings d Number of difficulty settings swept Training runs per difficulty setting
20K
100
(0.0, 0.98)
20 (linear range over (min, max), inclusive)
10 for ReLU Large (Figure 22)
10 for batched run (Figure 23)
otherwise
Number of piecewise stationary segments
50
Target function High probability bound B
Yú = sin 2n X?xX;) (-1,1)
Configuration for the neural networks used on the Piecewise Random Walk problem:
FTA
Hidden layers
Layer width
40
Layer binning interval bounds (1, u)
(-1, 1)
Layer bins &
40
Layer sparsity parameter 7
ReLU
Hidden layers
Layer width w
50
ReLU Large
Hidden layers
Layer width w
200
Layer widths w, bin count k and sparsity parameter 1) were chosen by sweeping range, and choosing the best performing configuration on the iid setting (i.e. difficulty == 0). In the sweep, the same w, k, 77 are used for each hidden layer of FTA, and the same w is used for each hidden layer of ReLU. See the Parameter Selection Details below, For all three networks, the ADAM optimizer Kingma & Ba (2015) was used with parameters (B,, (2) (0.9, 0.999), and the following learning rates were swept for each difficulty setting.
Learning rates X 0.01, 0.005, 0.001, 0.0005, 0.0001, 0.00005, 0.00001, 0.000005
The results from the best performing learning rate are reported in Figures 3(c), 20(c), 22(c), and 23. That is. the performing learning rate is determined separately for each difficulty setting, using the mean final loss across 30 runs for the FTA and ReLU networks. 10 runs were used for ReLU Large and the batching run. Results from all learning rates reported in Figure 21 for three different difficulty settings.
Parameter Selection Details
27

---

Published as a conference paper at ICLR 2021
Before sweeping any difficulty settings, first the network architectures established for the FTA and ReLU networks so that loss was minimized on the iid setting (d = 0). We avoid the overparametrized regime, that standard bias varsivaie tradeoff applies here Belkin et al. (2019). (However, the ReLU Large network is overparametrized. Learning rate was also swept during the architecture sweep, to ensure that the chosen architecture performed best within the learning rate range chosen for the main experiment depicted in Figure 3(c).
For the layer FTA network, each hidden layer uses the same width w, and sparsity parameter n). Those parameters, along with learning rate À were optimized in the following ranges with grid search:
Layer width E {10, 15, 20, 40)
Sparsity parameter 1) =
Learning rate À E {0.0005, 0.0001, 0.00005, 0.00001, 0.000005}
For the 2 layer ReLU network, fully connected hidden layers were used. Width and learning rate À were found from the following ranges, also with grid search:
Layer width E {5, 10, 20, 30, 40, 50, 60, 70, 80, 100, 120}
Learning rate À E (0.005, 0.001, 0.0005, 0.0001, 0.00005)
The learning rates are of different magnitude between FTA ReLU because they perform best in different ranges, as explained in A. For the main experiments, where range of difficulties are swept, the range of tested learning rates is shared between FTA and ReLU, and is broad enough to cover the optimal range for both. Also note that set &. rather than separately sweeping it. This is because FTA performs consistently well with in this order of magnitude.
A.7.3 PROOFS FOR COVARIATE SHIFT PROPERTIES
Theorem 4. Let {Eehien (Sthee, {XiHee be stochastic processes such that
E, ~ O,)
S4+1 (1-c)S++
(13)
X NNx;; St, 82)
(14)
where E (0, 1] and 8; is realization Sr. Then (XtheN is linear Gaussian AR(I) process with equilibrium distribution [(x; 0, 82 + )
Proof. Begin by rewriting equation 14 to give X, a sum of S, and mean zero Gaussian B,
X, = Sz + B,
B, ~ (~(,00 iid
(15)
Now substitute according equation 13
X, = -1 (1- Eç-1 + Br
(16)
Let Gaussian r.v. O NN a²) with some variance a².
Er-1, B, are independent Gaussian, so E B, can be written as a linear combination of B O since all are independent. Specifically, fix a² that the following holds
E + B, = ( B1 +
(17)
and substitute into equation 16
28

---

Published as a conference paper at ICLR 2021
X₁ = (1 c)S:-1 + c) B +
(1 (), + BB-1) + O2
= (1 -
By inspection, (EN a linear Gaussian AR(I) process with coefficient (1 (1-c) and noise variance
a Elementary AR process theory gives the equilibrium distribution as
Nr (0.
2
(18)
(1
For equation 17 to hold, we need the first and second moments of LHS and RHS to be equal. All terms are mean zero Gaussian, so suffices to show when the LHS and RHS have equal variance:
Var F + B₂) = Var((1 - B + 0
2² + B² = (1 8) +
a = 0² + 8 - -(1 c)
a² 0² + - (1 )3
Substituting into equation 18
a²
N
0,
(1 - c)2
2 (1 (1 2)2)
0,
1- (1 c)²
d²
0,
- (1 2)2 + $2
2² =
0,
+ 82
2c c²
So (Xihee is linear Gaussian AR(1) with the desired equilibrium distribution.
Theorem 2
Let difficulty parameter E (0, and 5² (g) for some B R+ Then (XthEN will have fixed cquilibrium distribution (0, {²), invariant if parameters 0², f2 are set follows
c c= v1-
B 0² d B)2
8e (1 d) B
Proof. By the above theorem 4, " completes the proof.
B² +
(1 d)
+ (g) This
20-2
Theorem 3
d 0 and So 5(00) induces iid X, from (2;; 0, (2)), the equilibrium distribution of (X1)
29

---

Published as a conference paper at ICLR 2021
Proof.
S1+1 = (1 c) + E
(by 9)
S + E
(d= = c=
St
(d (d=00 0 = o² =0=0 = E 6(0))
Hence, d = 0 implies S is constant over all time i, and So = 5(0) gives S. 8(0) for all time t.
L.v B, (0, (2) iid, then X, S; B, B, when S, is Dirac delta concentrating zero. = - () ( by the setting from Theorem 2 and setting d = 0. Hence, d = 0 induces iid sampling from the equilibrium distribution of X, This completes the proof.
30


Reviews:
[{"review_id": "z0eECQpSjV_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper presents a new activation function \"Leaky Tiling Activation\", designed with the goal of learning sparse representations (where only a few units activate for a given input). This new activation function, rather than mapping a scalar to another scalar, maps a scalar to a smoothed one-hot representation based on bins., which encourages sparse representations.\n\n\nReasons for score:\n\nAlthough I have some issues with the sparsity claims of this paper (see my comments below), I found the paper intriguing and I am curious to try similar ideas in other problems. So, with some work on the sparsity argument sections, I think this could be an interesting paper with ideas people would want to play with.\n\n\nAdditional feedback:\n\n- page 3: \"In the following theorem\" -> which theorem? [edit after reading further: Do you mean Theorem 1 in Section 3.3? (probably not, as that's for LTA, not TA)]\n- I am not sure if \"leaky\" is the best name for LTA, as more than \"leaky\", it is almost like the operators in fuzzy logic, with the term \"leaky\", I was imagining it more like a Leaky ReLU, where derivative is non zero everywhere.\n- page 4: about the \"guaranteed sparsity\". I do not think it's very fair to compare the sparsity achieved by this operation with that achieved via regularization for the following reason: LTA increases the number of outputs of the unit. So, if in a regular layer with, say, a ReLU activation, you would have n outputs, with an LTA activation you would have k*n outputs. Assuming TA, instead of LTA, where will be exactly \"n\" outputs that are non zero, the same as in the original network with ReLU. So, LTA is not increasing sparsity, but just encoding the output in a different way that makes it look sparse as it increases the number of outputs (Theorem 1 and its Corollary feel totally unnecessary, as the network is sparse by construction, no need to prove it). It's like claiming that a OHE representation of a discrete label is \"sparser\" than the label itself, which is an iffy argument, as they encode the same information. I am not implying that LTA is not useful (I still find the paper very intriguing!), but just that the sparsity argument is questionable, and in my view it is not fair.\n- page 5: it seems all results are in the appendix, which is unfortunate, as papers should stand on their own and appendix only used for additional explanation, not for main results. So, I am going to judge the paper just based on what is on the first 10 pages, to be fair to other papers. I'd expect sentences like \"as figure X shows, blah, blah (more details in appendix)\", but not \"as figure X in the appendix shows, blah blah main result\". This is trying to game the page limitations.", "extra_scores": {}, "extra_text": {}}, {"review_id": "U0DfeVsJ1e", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Update during review period\n\n- The reproducibility of the paper is now much better. It's great that the authors promised to release the LTA code. I hope that this includes the code for the experiments. \n- Based on the above, I changed my review score to 7. \n\n### Summary \n\nThe paper presents a novel activation function (Leaky Tiling Activation - LTA) to produce sparse activations, which have been found to stabilize learning in continual learning and RL settings. The new nonlinearity and its theoretical properties are described well, and the authors present convincing experiments demonstrating that the method yields practical benefits on synthetic datasets and RL games (e.g. Atari). \n\n### Reasons for score: \n \nThe paper should certainly be published somewhere, but maybe in a workshop that focuses on continual learning or RL. \n\nWhile the proposed new activation function may be useful in some settings, there is not enough evidence in the paper that it would become a go-to solution, or significantly change the way we think about interference in continual learning and RL. In particular, the authors compare LTA variants of DQN, but it might have been useful to compare with e.g. Rainbow too. While I agree that interference is still an issue in modern RL with function approximation, it would be useful for potential users of LTA to know whether LTA provides benefits when used in conjunction with an existing state-of-the-art RL algorithm. Adding an experiment to this effect to the appendix would make this paper stronger. \n \n### Pros\n \n1. The paper addresses a key problem in continuous learning and RL: interference and catastrophic forgetting. It presents a novel method to combat this problem, and demonstrates its usefulness in a number of experiments. \n2. The paper is generally well-written. \n3. The experiments on synthetic data were compelling, and made for a very nice controlled experiment.  \n \n\n### Cons\n \n1. The authors could have benchmarked against stronger baselines. The authors might be able to do this during the review process. \n2. The precise set-ups that the authors used for their experiments should be described more clearly. As it is, the paper's work is not reproducible. See my the section \"Questions during rebuttal period\" below for details. \n\n### Questions during rebuttal period: \n \nHave the authors considered to benchmark using stronger (state-of-the-art) baselines? Someone who considers using LTAs would likely use a state-of-the-art method already, not DQN, which the authors benchmarked against. The question is then whether LTA yields benefits when used in conjunction with state-of-the-art methods, not when used in conjunction with DQN. \n\nI did not fully understand what architecture the authors used in their experiments. The architecture appears to be described mostly in this single sentence: “All the algorithms use a two-layer neural network, with the primary difference being the activation used on the last layer. ” \n- Between what and what is this difference? Between baselines and the LTA experiments? \n- Later on the page from which I quoted above, the authors mention that they experimented with DQN-like networks. The original DQN paper used 3 layers (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). Did the authors mean to write ““All the algorithms use a two-layer neural network”?\n- Do the authors insert the LTA before the third layer?\n\nMore generally, it would be helpful if the authors could describe the architectures used in more detail and maybe ask a colleague who is not yet familiar with the paper to review for clarity and reproducibility. \n \n### Some suggestions\n\nI would rephrase the first sentence of the abstract in order to introduce “interference” in a gentler way. While interference is an active research topic in the RL community, many members of the ICLR community might wonder “what kind of interference”? A half-sentence like “where updates for some inputs degrade accuracy for others“ (copied from the paper’s introduction)  could suffice here. \n\nIn section 5.1, the authors have lines beginning with bolded “DQN”, “DQN-LTA”, “DQN-Large” et cetera. For readability’s sake, it might be useful to format these as bulleted lists. \n\nA small grammatical issue: “This issue is usually called gradient vanish”. Maybe rephrase this as “This issue is known as the vanishing gradient problem”. \n\nThe authors list ReLUs as an example of an activation function with vanishing gradients. As vanishing gradient problems go, ReLU is a bit different from tanh and other nonlinearities that saturate, so I would avoid listing it here to avoid unnecessary debates. \n\nThe authors write “Mnist” in a number of places. The correct spelling is “MNIST”: this is an acronym for “Modified National Institute of Standards and Technology”. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EeTU4igBX3k", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(I will keep my review short, as I don't have much constructive criticisms to share)\n\nThe authors propose a new activation function that guarantees sparsity, which can reduce interference/forgetting when learning on non-stationary data. The Leaky Tiling Activations (LTA) are relaxations of the binning operation to make it differentiable. The authors proved that LTA guarantees sparse features. \n\nNext, LTA is experimentally evaluated in online supervised learning on non-stationary data, as well as in reinforcement learning (RL).  In the supervised learning experiment, LTA is shown to outperform ReLU on a toy task. In the RL experiments, the story is repeated in discrete action domains (with a DQN backbone) and in continuous action domains (with a DDPG backbone). \n\nThe paper is well written and an enjoyable read. It is well motivated given that sparsity is an important characteristic for learning on non-stationary data. The empirical section is extensive. I am not really knowledgeable of those experiments, however. This is why I chose a confidence score of 4.\n\nMy main criticism would be that no code was shared. I am willing to keep my score and fight for acceptance if the authors commit to releasing the code upon acceptance. I would have appreciated, however, that the codebase would be available during the review period. This is why I chose a score of 7 instead of 8.\nFurthermore, I encourage the authors to build simple PyTorch and/or TF packages such that one can easily use the LTA without much overhead.\n\n_________\n\n**Post rebuttal**\n\nI am happy with the response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CClSwGY-Xb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel activation function based on tiling, with a careful consideration of the tradeoff between differentiable regions of the tiling (which would otherwise be an undifferentiable one-hot) and sparsity. Using such activations in domains where we expect sparsity to help, online learning and deep RL, shows some encouraging improvements.\n\nStrengths:\n- The paper is fairly well written and easy to understand\n- The method is simple and addresses important problems in deep RL\n- Results are encouraging\n\nWeaknesses [the two following weaknesses we addressed in the rebuttal]:\n- The central motivation for this method is the reduction in gradient interference, yet, it is never measured, nor other similar values. Sparsity is measured, but it is expected to be lower, by _construction_. Proposing new methods without understanding why they work is detrimental to progress. \"My number is bigger than your number\" is not a great way to do science.\n- The comparisons and baselines may not be entirely fair (although the methodology is otherwise good, with repeated runs and confidence intervals)\n\nI gave the paper a score of 6 because while the proposed method is interesting and possibly useful, more needs to be done to understand it, and advance our understanding of deep RL.\n[Rebuttal update: the extra results clear up some uncertainty about this method. I think the success of this method is interesting and teaches us a thing or two about deep RL.]\n\n\nComments:\n- \"past $X_{t−1}$, $X_{t−2}$\", that would be a 2-Markov problem, and is needlessly specific? I'd suggest just writing \"the past $X_{t-i}$\" since general online problems can have arbitrary time-dependencies.\n- \"such encodings can enable faster learning and reduce interference\", that may be true, but the authors need to test this.\n- \"LTA itself does not introduce any new training parameters\", that's not true. If a layer has a 100 units and 4 LTA bins, then the output of LTA is 400 units, and the following layers has to be 4 times as big, and thus introduces 300 new training parameters.\n  - \"One potentially surprising point is that the LTA reaches a lower error, even on iid data\", this should be a warning sign. When a new method outperforms an old method in settings where it shouldn't, it's often because the setting is unfair to the old method. I see in Figure 17 that the authors compared LTA to a wider ReLU network, that's good, although it's still not clear to me that the comparison is fair, as values of k and eta are not given, nor how many units and layers there are. It would be good to have all hyperparameters clearly laid out, especially in the appendix (which can be arbitrarily long).\n  - (The RL section is much clearer in that regard!)\n- Is the (presumed) reduction in gradient interference really due to the sparse encoding? or due to LTA still being mostly 0-gradient everywhere (e.g. in Figure 1c, with k=3 and eta=0.1, 60% of the x-axis has gradient 0)? This should be tested.\n  - If it's not clear what I mean, let's say we have 100 LTA units with k=4 and so 400 outputs. By construction we're going to have at most a 25% sparsity, let's say for argument's sake that all LTA inputs are positive and so there exactly 25% or 100 units that are on. Some proportion of these 100 active units will be in the 0-gradient regime, even though they are active. If that proportion is 5%, then activation sparsity and gradient sparsity will be roughly the same, but if that proportion is 80%, then the gradients will be _even more_ sparse than the activations. That seems like something that's important to know to understand how/why the proposed method works.\n- The RL experiments compare DQN-LTA with DQN-Large, this is good, but perhaps misleading\n  - it's not clear how k/delta/eta were chosen (Figure 13 hints at good values, but the legend is missing and it's not clear if this was done post-hoc or if it informed the choice in the main experiments, I assume the latter)\n  - it's not clear that the k that's optimal for DQN-LTA is the same that's optimal for DQN-Large. Considering Figure 13, it seems that the authors spent more time doing hyperparameter search for DQN-LTA than for DQN-Large, in that sense it's likely an unfair comparison. (in any case it would be interesting to see a plot similar to Figure 13 for DQN)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new sparsity-inducing activation function, and demonstrates its benefits on continual learning and reinforcement learning tasks.

After the discussion period, all reviewers agree that this is a solid paper, and so do I. I am thus recommending it for acceptance as a poster. Hopefully, such visibility (combined with the open source release of the code) will encourage other researchers to try this new technique, and we will see more evidence confirming its usefulness in more varied settings and versus stronger baselines (that remain somewhat limited in the current work: this is the main weakness of the paper).
Decision: Accept (Poster)