Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes a sampler for LSTM video models. The sampler works by repeating frames in a training batch. The method is evaluated for the task of action recognition on the UCF 101 dataset. 
main_review: This paper presents many substantial flaws, which I detail below.

# Anachronistic temporal information representation

The authors start providing a 3 page long formulation of “video temporal information”. The authors base their formulation on the correlation of objects between frames. While this might make sense for some actions, such correlation is solely modelled as the ratio between an object bounding box and the video frame size. The authors then go on with many equations (10 of them) based on the Bayes theorem and Mutual information theory to “model video temporal information”, again based only on bounding boxes ratios. 

This approach is completely oblivious of what has been done in the past 20 years to model actions. No optical flow, no visual features, no CNNs are mentioned at all. In fact, the authors do not include a single paper related to action recognition in their related work. Moreover, it is not clear how this temporal information formulation is even used, if it is used at all in this work. Indeed, after presenting their formulation, the authors continue saying they use a simple CNN with an LSTM to recognise actions. My guess is that the object-box-based formulation is not used, and that CNN features are instead employed, which leaves me to wonder why such formulation is presented in the first place. 

To summarise this point, Section 3.2 (Video Temporal Information) is entirely disconnected to the rest of the paper. The temporal information formulation is overly simplistic and completely unaware of any previous work in the field. It is not clear whether this formulation is even employed.

# Insufficient contribution

The core idea of this work is to repeat frames in a batch when training an LSTM to recognise actions. This very simple idea does not constitute a sufficient contribution, because it essentially corresponds to augmenting the training data with more frames, which naturally speeds convergence. In short, no novel contribution is presented in any way since the method is just a sampler that repeats frame in a trivial way.

# Flawed and incomplete evaluation

Quoting from page 6: 

> Our test uses the shuffle operation, to make the test results more objective. Since the test data is shuffled, there should be less temporal between the data, the test results may depend on the sampler and model only, which could reflect the detection effect well.

Firstly, it is unclear how shuffling would make test results more “subjective”. Secondly, it is arguably incorrect to test a model shuffling the data as this introduces noise in the evaluation. Finally and most importantly, it is not clear why the method would perform better or worse by shuffling data. This confusion stems from the fact that the authors do not specify whether they shuffle the batch (in which case shuffling makes no sense during inference) or whether they shuffle frames within a sequence (which would be a major flaw since the videos would be altered).

Besides, the method is evaluated only on a single dataset without comparison to previous work. Results show that the proposed method under-performs in many cases as well.
summary_of_the_review: This paper presents major problems: it is severely unaware of previous work in the field, does not present a sufficient contribution and it is not well evaluated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a stepped sampling to improve the learning capabilities of neural network models such as LSTM. Specifically, the stepped sampling procedure repeats the same input data in multiple batches, in other words, the batches "overlap" with one another in terms of the contained input data. This follows from the authors' argument that repeatedly providing the same data leads to faster and stable convergence in training, as well as higher accuracy in testing. The authors experimentally show these benefits of their stepped sampling procedure over traditional sampling techniques (e.g., random sampling) in the context of action detection from videos using LSTMs.
main_review: ### Strengths
1. The proposed methodology is technically sound and clearly presented.
2. The experiments show promising results, lending good empirical credibility to the authors' claims.

### Weaknesses
1. My main concern is that while the authors argue that their proposed sampling technique is more "human-like" or closer to "how humans learn from data", I did not find any references in the paper backing up this claim. Are there any studies in psychology, neuroscience, or any other relevant field that grounds the key intuition for stepped sampling, which is repeating the parts of the same input in different batches? Without such grounding, the claim can come across as being made post-hoc, following the experimental success of stepped sampling.

2. Since $d$ in Eq. 11 is called the iteration number, I presumed it is a (non-negative) integer. How, then, is $d$ determined for $L=25$, $m=20$, and strides $n=2, 3, 4$ in the results shown in Figure 3?

3. The authors make claims of stepped sampling leading to stabler convergence compared to traditional sampling techniques in a few places in the paper (e.g., last paragraph in Section 4.3). Is there a way to quantify this claim, e.g., by plotting the rate of change of the loss function across the training epochs?

### Minor comments
1. Is there any specific reason the authors worked with PyTorch version 1.0.1 (Section 4.1), which came out in early 2019, when much newer versions such as 1.8.0 have been available since early 2021?

2. For the sake of completeness, please specify that $L>m \geq n$ in Algorithm 1.
summary_of_the_review: Overall, the authors propose a technically sound sampling technique and experimentally show its benefits over traditional sampling techniques, such as faster and stable convergence and higher test accuracy, in the context of action detection from videos using LSTMs. However, the authors also attribute the superiority of their technique to the claim that it is closer to how humans learn from data, but this is not grounded with any relevant prior studies or observations. Without such grounding, the claim comes off as post-hoc and the technique comes off as a successful engineering effort rather than a novel research contribution, thus making me lean towards rejection.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper mainly provides a stepped sampling method only for video detection and only adopted in LSTM structure. This idea is inspired from human's repeatedly memory working mechanism. This method can achieve fast convergence during training, smooth the training loss curves and get a good test accuracy when batch size equal to 15.
main_review: Strengths:
1, This paper gives very detailly derivation processes, equations, algorithms for the idea. The illustrated graph clearly states its idea.

Weaknesses:
1, Performances: It is useless to discuss the convergence speed without getting the optimal convergence value.  For methods comparison in RESULTS part, authors can only achieve better results when batch size equal to 15. And authors don't provide details to illustrate why choose 25 or 15 batch size. The parameters selection seems randomly. The results cannot be predicted and repeated in other situations.
2, As this paper limits its usage and performances under very narrow scope (for certain task Video detection and certain network structure LSTM) and the results are not good and seem to be unpredictable in further reproduction, there is no need to discuss other weakness. The method is useless.
summary_of_the_review: This paper proposes a very simple idea: repeatedly inputting previous frames into the same units in LSTM structure attempting to imitate humans' repeatedly memory system. But the results seem ugly, the training parameters (batch size) seem to be randomly selected. This method seems impossible to generalize to other training parameters, tasks and network structures.
review_title: null

Meta-review: This paper proposes a stepped sampler for LSTM-based video detection. However, reviewers raised a series of issues of this paper, including the weakness in novelty, experiment evaluations, and generalizability of the method. Considering the limited contribution of this paper, and limited experiment evaluations, the AC agrees with the reviewers and recommends reject for this paper.
Decision: Reject