Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Update after response: 

The authors have quite thoroughly addressed most of my concerns with updates and new experiments. So I will increase my score to an accept at this point.

Summary:

The authors consider the question of the effect of temporally correlated data on learning. They show that while standard networks are adversely affected by this smoothness, using mechanisms such as leaky memory in activation units and memory gating allows networks to take advantage of this data. The authors also further study the representations that emerge from said mechanisms.

Overall, the work is quite interesting and insightful. But some of the concerns listed below put this work below the threshold of acceptance for me.

Strengths:

+ The question is very well motivated and relevant, since, as the authors point out, a lot of data in the real world is highly temporally correlated.
+ This work provides a nice compact explanation for the role of leaky processes in biology.

Weaknesses:

- With temporally correlated data, once expects that the ability of the network to temporally process the data would also have a significant effect on performance. In this context, it seems to me that a comparison with RNNs is also quite relevant as a baseline.
- The Auto-encoder with no memory seems to perform better than other models. This needs an explanation at the least. This could either indicate that the experiment they consider is too simple, or it could point to some more fundamental underlying issues with the combination of auto-encoders and sequentially correlated data. Depending on which, the rest of the analysis may not generalise. This is a major weakness in the paper.
- A more detailed explanation for why BPTT is not used (or comparison of all results with BPTT baselines) would make the results a lot more useful. BPTT implies propagating the gradient through the leaky memory. 
- A discussion of the universal assumption of i.i.d data in machine learning deserves more space in the introduction.

Other questions:

* Could the authors comment on what sort of processes could control event-related resetting in biology?

Minor:

* Defining what incremental learning is early on would improve the clarity of the paper.
* Visualisation of the dynamics of units of the AE could prove to be quite interesting.
* The first line of section 4.1.2 is a bit confusing: "We tested MNIST, Fashion-MNIST, and further synthetic datasets containing low category overlap" seems to imply MNIST and Fashion-MNIST have low category overlap, which is not the case and I don't think that's what the authors meant to imply.
review_title: Authors address a very relevant and interesting question

final_score: rating: 6: Marginally above acceptance threshold
review_content: Temporal smoothness is a recurring feature of real-world data that has been unaccounted for when training neural networks. Much of the random sampling in training neural networks is done to remove the temporal correlations originally present when the data is collected. This work aims to propose a method to train on this 'less processed' form of data.

There are two aspects of their method which makes training on smooth data possible:
1) Hidden units with 'multi-scale leaky memory'
2) Memory gating- between category transitions in time, memory is reset by setting $\alpha = 0$

For supervised learning, the authors create an artificially smooth dataset by presenting a model with examples from the same class repeatedly. They show as compared to a baseline model, their proposed method is able to learn effectively on highly repetitive data.

For unsupervised learning, the authors show that the model learns to match internal hidden unit representations with different \alpha = \{0.0, 0.3, 0.6\} with the corresponding timescale of \{fast, medium, slow\} features on a toy dataset.

Strengths:
- Specifies the right problem. Temporally smooth learning mechanisms are noticeably absent in the field.

- Interesting property where the proposed method does not use backpropagation through time despite having a recurrent hidden unit function

- Leaky memory is a simple idea to resolve this learning issue.

Places for improvement:

-Despite making the case that data in the real world is temporally smooth, the datasets which were used were artificially generated from a dataset that is not smooth (MNIST). Is there any issue in applying this method to a video segmentation as described in their example in Fig 1A?

-For the unsupervised learning scenario, the toy data only has variations that exactly match the timescales setup in the architecture of the network. What happens if these are mismatched with the input data?

-The accuracies also seem off for 1 repetition training of the baseline. The accuracy should be well above 90% on MNIST.

-How does mini-batching work in this temporally smooth data? Are the examples within a mini-batch temporally aligned? Temporally aligned mini-batches does not seem like a realistic assumption to make.




review_title: On the right track, but would like to see less artificial experiments and more realistic ones

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Update after rebuttal:** I appreciate the detailed responses by the authors. I'm willing to increase my score based on the responses, but unfortunately I'm still not ready to recommend acceptance. In my opinion, the paper is simply not mature enough yet for publication (the significant amount of revisions required during the rebuttal period attests to this, I think; a mature conference paper should not have to require this much revision during review). In particular, the following fundamental issues still remain for me even after the revisions: 

1. The misleading language about "temporal smoothness" in real-word data remains throughout the paper despite the fact that the paper doesn't address temporal smoothness as it exists in real-world data.

2. The authors promise some new experiments on more realistic stimuli, but as it stands the paper still only includes experiments on static images with mostly toy data and I have no way of knowing whether any of their results would generalize to more realistic data. The experiments with multi-scale stimuli suggest that that generalization may be non-trivial (e.g. in that experiment, the baseline model with no memory or gating mechanisms actually performs the best).

3. Which brings me to my final point: I still don't think the authors have adequately explained why and how the proposed mechanisms work. For example, the authors say: *"Our working hypothesis is that averaging across multiple members of the same category increases (in some datasets) the proportion of variance in the hidden units that is associated with category-diagnostic features."* Why the hedging *in some datasets*? The experiments with multi-scale stimuli clearly demonstrate that the proposed scheme doesn't work in all cases, but what exactly are the conditions under which it would work better than the baseline model? The authors need to make these a lot clearer.

------------------------------------------
This paper mainly investigates the effect of iteration-to-iteration correlations in online learning. It recapitulates a fairly obvious and pretty well-known result that such iteration-to-iteration correlations will slow learning. I find the motivating question (the effect of temporal correlations on learning) somewhat interesting, but unfortunately, I think the research reported in this paper is really not very well-executed:

(1) Unlike what the title and sections 1 and 2 claim, the experiments in this paper do not test the effect of temporal smoothness. There is no actual time dimension in the data used in this paper. It rather tests something else: more accurately described as “iteration-to-iteration correlations in online learning”. This makes the set-up considerably less general and less interesting in my mind compared to the actual temporal ordering question, which has some practical relevance. Relatedly, the illustration in Figure 1A is misleading. This is not the setup tested in the experiments. 

(2) The models and datasets used in this paper are extremely toy, there is no reason why more realistic datasets with actual temporal structure could not be used for this research. 

(3) The paper only studies the online learning scenario (Appendix A5 reports the results of an experiment with minibatch training, but this is very limited, and not nearly rigorous enough). This limits the relevance of this work both for machine learning and for neuroscience/psychology. Most machine learning research does not do online learning. Even animals do not have to do purely online learning, because they have offline replay mechanisms that don’t have to respect temporal order strictly. 

(4) The authors propose two mechanisms to alleviate the learning slow-down caused by iteration-to-iteration correlations in online learning. However, it isn’t at all clear why the proposed mechanisms help with correlated data. No explanation is given for how these mechanisms are supposed to help with learning from correlated data in the online setting. Please note claiming that these mechanisms are brain-inspired is not an explanation. Moreover, the set-up in these experiments is also not described clearly. Section 5.1.1 says “The learning algorithm, optimization and initialization methods, and the hyperparameters were identical to those used in training and testing feedforward neural networks”, but you can’t do online learning with leaky neurons anymore. Later on (right at the very end of the paper in the Conclusion section!), we learn that the learning setup is actually not identical: backprop is truncated in these models to prevent gradients from flowing into previous time steps. This important detail is somehow never mentioned in section 5. 

(5) Is it possible that the effect of leaky memory is just due to reduced gradient variance via some sort of mini-batching mechanism? (note that Appendix A5 doesn’t address this question). Since the hidden state contains information about previous examples in this model, the memory may be acting as some sort of implicit mini-batching mechanism that reduces the gradient variance.

(6) The results in Figure 4A: the baseline no-memory model is outperforming the other models. This seems to contradict the results earlier in the paper (e.g. Figure 2) showing the benefits of memory+gating. What is the explanation for this discrepancy? 

(7) The experiments are also in general not done very rigorously. For example, no hyperparameter tuning was done for the “smooth” case, but maybe the problem is just that the learning rate in this case should be slightly different (i.e. no need for special mechanisms like memory or gating). We can never know this unless the experiments are done more rigorously.

review_title: official review

final_score: rating: 6: Marginally above acceptance threshold
review_content: In regular DNN training mini-batches are selected at random and temporal smoothness of data is not used. In fact it is expected that temporal smoothness can lead to catastrophic forgetting which may lead to poorer performance. The authors of this paper first verify this hypothesis and prove this to be true.

Then they propose that two memory inspired mechanisms can take advantage of the temporal smoothness of data: leaky memory and memory-gating. They show that leaky memory helps when data is presented smoothly and additional gating helps even further (however, the legends in Figure 2 are not clear). They also show that similar results hold for unsupervised learning.

While the results are promising (and I also liked the link the authors draw from human brain to the two proposed mechanisms) there are certain drawbacks as well. One of the main advantages of random training is that generating the order is not costly. However, in the temporarily smooth approach this can become a bottleneck due to the large size of datasets used to train DNNs. Another potential issue pops up due to data imbalance found in practice. While undersampling/oversampling can be used, data imbalance can still be a big problem. In that respect I would have preferred to have seen experiments with more datasets and more complex neural networks.

Comments:
- Show the legends for all curves in Figure 2 (only Minimum smoothness is marked as green)
- Why is the random sampling in 2B different from 2A and 2C? They should all be the same or just different. Any particular reason for treating 2B differently?
- I would suggest showing cross-entropy results in the main paper as that is the popular loss function used.
- I am confused by 4A. It seems that No memory has the lowest test error. Doesn't that mean no memory is the best?
- Typo: Related Work: line 2: speeded -> sped 
review_title: This is an empirical paper which shows that leaky memory and memory-gating can take advantage of temporal smoothness in data. They run multiple experiments to show the effectiveness. However, one of my main concern is that such a mechanism may be hard to apply in practical scenarios in particular with class imbalance.

Meta-review: It appears that this paper can benefit from additional detail and work before it becomes a stronger publication that is more convincing. The authors have done an impressive job responding to the reviewers and updating their paper, and multiple reviewers raised their score consequently. However, while multiple reviewers now recommend acceptance, there is no agreement on it. Even among the reviewers who recommended acceptance, there is a feeling on being on the fence specifically about the ability of the paper to make a convincing argument without considering a real life scenario and while only using toy settings. Indeed, this is a problematic aspect of the paper because the value of the paper lies in making that argument. Further, the paper would gain further from clarifying the writing further and connecting the paper more directly with the neuroscientific literature it aims to be connected to.
Decision: Reject