Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper presents a new activation function "Leaky Tiling Activation", designed with the goal of learning sparse representations (where only a few units activate for a given input). This new activation function, rather than mapping a scalar to another scalar, maps a scalar to a smoothed one-hot representation based on bins., which encourages sparse representations.


Reasons for score:

Although I have some issues with the sparsity claims of this paper (see my comments below), I found the paper intriguing and I am curious to try similar ideas in other problems. So, with some work on the sparsity argument sections, I think this could be an interesting paper with ideas people would want to play with.


Additional feedback:

- page 3: "In the following theorem" -> which theorem? [edit after reading further: Do you mean Theorem 1 in Section 3.3? (probably not, as that's for LTA, not TA)]
- I am not sure if "leaky" is the best name for LTA, as more than "leaky", it is almost like the operators in fuzzy logic, with the term "leaky", I was imagining it more like a Leaky ReLU, where derivative is non zero everywhere.
- page 4: about the "guaranteed sparsity". I do not think it's very fair to compare the sparsity achieved by this operation with that achieved via regularization for the following reason: LTA increases the number of outputs of the unit. So, if in a regular layer with, say, a ReLU activation, you would have n outputs, with an LTA activation you would have k*n outputs. Assuming TA, instead of LTA, where will be exactly "n" outputs that are non zero, the same as in the original network with ReLU. So, LTA is not increasing sparsity, but just encoding the output in a different way that makes it look sparse as it increases the number of outputs (Theorem 1 and its Corollary feel totally unnecessary, as the network is sparse by construction, no need to prove it). It's like claiming that a OHE representation of a discrete label is "sparser" than the label itself, which is an iffy argument, as they encode the same information. I am not implying that LTA is not useful (I still find the paper very intriguing!), but just that the sparsity argument is questionable, and in my view it is not fair.
- page 5: it seems all results are in the appendix, which is unfortunate, as papers should stand on their own and appendix only used for additional explanation, not for main results. So, I am going to judge the paper just based on what is on the first 10 pages, to be fair to other papers. I'd expect sentences like "as figure X shows, blah, blah (more details in appendix)", but not "as figure X in the appendix shows, blah blah main result". This is trying to game the page limitations.
review_title: Interesting idea with convincing results, some issues in the sparsity claims though.

final_score: rating: 7: Good paper, accept
review_content: ### Update during review period

- The reproducibility of the paper is now much better. It's great that the authors promised to release the LTA code. I hope that this includes the code for the experiments. 
- Based on the above, I changed my review score to 7. 

### Summary 

The paper presents a novel activation function (Leaky Tiling Activation - LTA) to produce sparse activations, which have been found to stabilize learning in continual learning and RL settings. The new nonlinearity and its theoretical properties are described well, and the authors present convincing experiments demonstrating that the method yields practical benefits on synthetic datasets and RL games (e.g. Atari). 

### Reasons for score: 
 
The paper should certainly be published somewhere, but maybe in a workshop that focuses on continual learning or RL. 

While the proposed new activation function may be useful in some settings, there is not enough evidence in the paper that it would become a go-to solution, or significantly change the way we think about interference in continual learning and RL. In particular, the authors compare LTA variants of DQN, but it might have been useful to compare with e.g. Rainbow too. While I agree that interference is still an issue in modern RL with function approximation, it would be useful for potential users of LTA to know whether LTA provides benefits when used in conjunction with an existing state-of-the-art RL algorithm. Adding an experiment to this effect to the appendix would make this paper stronger. 
 
### Pros
 
1. The paper addresses a key problem in continuous learning and RL: interference and catastrophic forgetting. It presents a novel method to combat this problem, and demonstrates its usefulness in a number of experiments. 
2. The paper is generally well-written. 
3. The experiments on synthetic data were compelling, and made for a very nice controlled experiment.  
 

### Cons
 
1. The authors could have benchmarked against stronger baselines. The authors might be able to do this during the review process. 
2. The precise set-ups that the authors used for their experiments should be described more clearly. As it is, the paper's work is not reproducible. See my the section "Questions during rebuttal period" below for details. 

### Questions during rebuttal period: 
 
Have the authors considered to benchmark using stronger (state-of-the-art) baselines? Someone who considers using LTAs would likely use a state-of-the-art method already, not DQN, which the authors benchmarked against. The question is then whether LTA yields benefits when used in conjunction with state-of-the-art methods, not when used in conjunction with DQN. 

I did not fully understand what architecture the authors used in their experiments. The architecture appears to be described mostly in this single sentence: “All the algorithms use a two-layer neural network, with the primary difference being the activation used on the last layer. ” 
- Between what and what is this difference? Between baselines and the LTA experiments? 
- Later on the page from which I quoted above, the authors mention that they experimented with DQN-like networks. The original DQN paper used 3 layers (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). Did the authors mean to write ““All the algorithms use a two-layer neural network”?
- Do the authors insert the LTA before the third layer?

More generally, it would be helpful if the authors could describe the architectures used in more detail and maybe ask a colleague who is not yet familiar with the paper to review for clarity and reproducibility. 
 
### Some suggestions

I would rephrase the first sentence of the abstract in order to introduce “interference” in a gentler way. While interference is an active research topic in the RL community, many members of the ICLR community might wonder “what kind of interference”? A half-sentence like “where updates for some inputs degrade accuracy for others“ (copied from the paper’s introduction)  could suffice here. 

In section 5.1, the authors have lines beginning with bolded “DQN”, “DQN-LTA”, “DQN-Large” et cetera. For readability’s sake, it might be useful to format these as bulleted lists. 

A small grammatical issue: “This issue is usually called gradient vanish”. Maybe rephrase this as “This issue is known as the vanishing gradient problem”. 

The authors list ReLUs as an example of an activation function with vanishing gradients. As vanishing gradient problems go, ReLU is a bit different from tanh and other nonlinearities that saturate, so I would avoid listing it here to avoid unnecessary debates. 

The authors write “Mnist” in a number of places. The correct spelling is “MNIST”: this is an acronym for “Modified National Institute of Standards and Technology”. 

review_title: Weak accept: interesting idea, could have benchmarked against stronger baselines, needs revision for better reproducibility

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a novel activation function based on tiling, with a careful consideration of the tradeoff between differentiable regions of the tiling (which would otherwise be an undifferentiable one-hot) and sparsity. Using such activations in domains where we expect sparsity to help, online learning and deep RL, shows some encouraging improvements.

Strengths:
- The paper is fairly well written and easy to understand
- The method is simple and addresses important problems in deep RL
- Results are encouraging

Weaknesses [the two following weaknesses we addressed in the rebuttal]:
- The central motivation for this method is the reduction in gradient interference, yet, it is never measured, nor other similar values. Sparsity is measured, but it is expected to be lower, by _construction_. Proposing new methods without understanding why they work is detrimental to progress. "My number is bigger than your number" is not a great way to do science.
- The comparisons and baselines may not be entirely fair (although the methodology is otherwise good, with repeated runs and confidence intervals)

I gave the paper a score of 6 because while the proposed method is interesting and possibly useful, more needs to be done to understand it, and advance our understanding of deep RL.
[Rebuttal update: the extra results clear up some uncertainty about this method. I think the success of this method is interesting and teaches us a thing or two about deep RL.]


Comments:
- "past $X_{t−1}$, $X_{t−2}$", that would be a 2-Markov problem, and is needlessly specific? I'd suggest just writing "the past $X_{t-i}$" since general online problems can have arbitrary time-dependencies.
- "such encodings can enable faster learning and reduce interference", that may be true, but the authors need to test this.
- "LTA itself does not introduce any new training parameters", that's not true. If a layer has a 100 units and 4 LTA bins, then the output of LTA is 400 units, and the following layers has to be 4 times as big, and thus introduces 300 new training parameters.
  - "One potentially surprising point is that the LTA reaches a lower error, even on iid data", this should be a warning sign. When a new method outperforms an old method in settings where it shouldn't, it's often because the setting is unfair to the old method. I see in Figure 17 that the authors compared LTA to a wider ReLU network, that's good, although it's still not clear to me that the comparison is fair, as values of k and eta are not given, nor how many units and layers there are. It would be good to have all hyperparameters clearly laid out, especially in the appendix (which can be arbitrarily long).
  - (The RL section is much clearer in that regard!)
- Is the (presumed) reduction in gradient interference really due to the sparse encoding? or due to LTA still being mostly 0-gradient everywhere (e.g. in Figure 1c, with k=3 and eta=0.1, 60% of the x-axis has gradient 0)? This should be tested.
  - If it's not clear what I mean, let's say we have 100 LTA units with k=4 and so 400 outputs. By construction we're going to have at most a 25% sparsity, let's say for argument's sake that all LTA inputs are positive and so there exactly 25% or 100 units that are on. Some proportion of these 100 active units will be in the 0-gradient regime, even though they are active. If that proportion is 5%, then activation sparsity and gradient sparsity will be roughly the same, but if that proportion is 80%, then the gradients will be _even more_ sparse than the activations. That seems like something that's important to know to understand how/why the proposed method works.
- The RL experiments compare DQN-LTA with DQN-Large, this is good, but perhaps misleading
  - it's not clear how k/delta/eta were chosen (Figure 13 hints at good values, but the legend is missing and it's not clear if this was done post-hoc or if it informed the choice in the main experiments, I assume the latter)
  - it's not clear that the k that's optimal for DQN-LTA is the same that's optimal for DQN-Large. Considering Figure 13, it seems that the authors spent more time doing hyperparameter search for DQN-LTA than for DQN-Large, in that sense it's likely an unfair comparison. (in any case it would be interesting to see a plot similar to Figure 13 for DQN)

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: (I will keep my review short, as I don't have much constructive criticisms to share)

The authors propose a new activation function that guarantees sparsity, which can reduce interference/forgetting when learning on non-stationary data. The Leaky Tiling Activations (LTA) are relaxations of the binning operation to make it differentiable. The authors proved that LTA guarantees sparse features. 

Next, LTA is experimentally evaluated in online supervised learning on non-stationary data, as well as in reinforcement learning (RL).  In the supervised learning experiment, LTA is shown to outperform ReLU on a toy task. In the RL experiments, the story is repeated in discrete action domains (with a DQN backbone) and in continuous action domains (with a DDPG backbone). 

The paper is well written and an enjoyable read. It is well motivated given that sparsity is an important characteristic for learning on non-stationary data. The empirical section is extensive. I am not really knowledgeable of those experiments, however. This is why I chose a confidence score of 4.

My main criticism would be that no code was shared. I am willing to keep my score and fight for acceptance if the authors commit to releasing the code upon acceptance. I would have appreciated, however, that the codebase would be available during the review period. This is why I chose a score of 7 instead of 8.
Furthermore, I encourage the authors to build simple PyTorch and/or TF packages such that one can easily use the LTA without much overhead.

_________

**Post rebuttal**

I am happy with the response.
review_title: promising and simple idea, great paper

Meta-review: This paper proposes a new sparsity-inducing activation function, and demonstrates its benefits on continual learning and reinforcement learning tasks.

After the discussion period, all reviewers agree that this is a solid paper, and so do I. I am thus recommending it for acceptance as a poster. Hopefully, such visibility (combined with the open source release of the code) will encourage other researchers to try this new technique, and we will see more evidence confirming its usefulness in more varied settings and versus stronger baselines (that remain somewhat limited in the current work: this is the main weakness of the paper).
Decision: Accept (Poster)