Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new data augmentation for video classification that consists of shifting one of the RGB channels forwards or backwards by 1 frame. They call this augmentation "Ghost Motion" (GM). They also show that scaling the logits by a temperature helps generalization. The authors claim that GM improves results for a variety of different models and datasets.
strength_and_weaknesses: Strengths:
1) The proposed data augmentation technique is well explored and has an extensive empirical evaluation. I appreciate that the authors investigated it under a number of different models and datasets. While they briefly also investigate modern approaches based on Transformers, most of their results focus on out-dated models based on 2D CNNs. I feel that the paper could be improved by focusing more on modern models. E.g. Table 1 contains results for TSN, which is by now a 6 year old method. I don't think these results have any relevance for the research community. Adding a modern, state-of-the-art method like Video Swin Transformer[4], MViT[5] or MTV[6] instead would considerably strengthen the paper. 


Weaknesses
1)  The "logit scaling" part of the current manuscript is weak: it's a completely unrelated regularization technique to Ghost Motion, and as the authors do mention, the same technique was furthermore already proposed and investigated for classification tasks in [3]. So mentioning this as a novel contribution is a stretch.  Furthermore it seems to be just another variation of label smoothing [1] or logit squeezing [2], neither of which the authors compare to or even mention. I'd suggest de-empathising this part of the paper and focusing completely on Ghost Motion instead. Otherwise, the authors need to properly compare logit scaling to competing methods.

2)  The results for the baselines stated by the authors do not match those from the original publications. For example, the authors claim in Table 1 that the basic TSM module achieves 45.34% accuracy on SomethingSomethingV1, and that TSM+GM together achieve 46.88%.
However, the original TSM publication states that TSM alone achieves 47.3% accuracy. Similarly, the original TDN publication claims 52.3% top1 accuracy on SomethingSomethingV1, while this manuscript claims that TDN alone achieves 49.69% and TDN+GN  50.51%. I did not check the rest of the numbers, but this small spot check seems weird.  The authors need to address this discrepancy.

3) While the authors claim several times that GM "brings almost zero costs", that claim was never backed  up. I'd appreciate if the authors could provide insights on the computational cos of their method: does adding GM really not increase the wall-clock time of training? 

4) The authors give all accuracies with 4 significant digits, which would suggest a large number of repeat experiments, yet they do neither mention this nor give error bars. I would strongly suggest they add unambiguous error bars for their results.


[1] Mueller et al., "When does label smoothing help?", NeurIPS 2019 (and references therein)

[2] Shafahi et al, "Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training? ", ICLR 2019

[3] Agarwala et al, Temperature check: theory and practice for training models with softmax-cross-entropy losses, arxiv 2019

[4] Liu et al, "Video Swin Transformer", CVPR 2022

[5] Fan et al, "Multiscale Vision Transformers", ICCV 2021

[6] Yan et al, "Multiview Transformers for Video Recognition", CVPR 2022
clarity,_quality,_novelty_and_reproducibility: 1) The paper is riddled with grammatical errors and un-idiomatic language. This needs to be addressed before this manuscript can be accepted for publication. The mistakes were too abundant to all write down and note, so I stopped taking notes on this after the abstract:

“overfitting is an even severe problem in 2D video recognition models” => “overfitting is a severe problem”

“current pipeline treats background and semantic frames” => pipelines. (Also, “non-nformative” and “informative” might be better than 

“background and semantic”. The latter made me assume this work was about semantic segmentation, but YMMV.

“named as Ghost Motion” => named Ghost Motion

“GM shifts channels along temporal dimension” => along the temporal dimension

“leading to improvement” => improvements


I suggest the authors revise the manuscript intensively and extensively for such errors.



2) Regarding Originality, GM is a simplification of the Temporal shift module (TSM), but the authors show in their experiments that GM improves upon TSM models. As mentioned above, I have strong doubts about the originality of logit scaling.
summary_of_the_review: The authors propose a  input augmentation that appears to be effective on a wide range of problems and models. The empirical evaluation covers a wide range of models and datasets, yet the numbers raise some questions (see above).

UPDATE: after the author's revision of the paper, I have updated my score to reflect that I think the method now meets the standards for publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper makes two contributions to reduce overfitting of video recognition models: Ghost Motion (GM) data augmentation which randomly shifts one RGB channel forward or backward in time, and logit smoothing (temperature scaling) to reduce overconfidence on background frames. The authors hypothesize that GM works by exchanging information between neighboring frames, creating motion artifacts that guide the model to focus on temporal cues. Experiments on multiple video action datasets show the proposed training scheme reduces overfitting and improves test accuracy of classifiers.
strength_and_weaknesses: ## Pros

- This paper studies an important problem of video data augmentation, where there has been limited prior work in the literature.  
- The proposed method is simple and appears effective across datasets and models.  
- Ablation experiments on the augmentation parameters are provided.  

## Cons

- Methods  
	- I don't feel that the motivation for GM is strong enough. Specifically, how does the information transfer between frames lead to stronger temporal modeling at test time (when the channels are not shifted)? It would be nice to have experiments supporting that models trained with GM is more sensitive to temporal cues than spatial ones.  
	- I am also not sure about the design choice, where only red or blue channels are shifted, but not the green channel. It would be nice if more explanations or justifications are provided. It would also be interesting to see if shifting all channels with interpolation can achieve similar effects.  
	- The technical contribution from this work is somewhat limited: GM mirrors the temporal shift operation of TSM in the input image space, with similar objectives (improve temporal modeling); temperature scaling has been widely used in model calibration literatures to reduce overfitting, and should not be considered an original contribution from this work. 

- Experiments
	- The effect of logit smoothing and ghost motion is not well disentangled in the experimental results. From figure 3 it appears that temperature tuning alone improves accuracy by ~1%. Is this the main source of improvements in the main tables, or do the gains come from the GM augmentation instead?  
	- The improvements over existing image-based augmentations is not particularly significant (e.g. <1% over MixUp).
	- There does not seem to be comparison to prior video data augmentation methods such as VideoMix (https://arxiv.org/abs/2012.03457).  

- Minor comments
	- I assume that the effectiveness of method depends on the video frame rate. If frame sampling is dense enough, adjacent frames will be too similar that shifting does not make a significant difference. Maybe it would be nice to include additional discussion/experiments on this.
clarity,_quality,_novelty_and_reproducibility: The method is presented with clarity, which adds to the reproducibility of the work, though there are some confusions in the experimental results. I am less confident in originality as the proposed techniques are either well studied in prior work (legit smoothing) or simply adapted from related problems (channel-wise temporal shifts).
summary_of_the_review: Although this work has its strengths, I lean toward rejection of the paper at its current state, due to lack of technical contributions and empirical justifications of some claims made in the paper, as detailed in the weaknesses above. I look forward to the authors' response to resolve any confusions/misunderstandings.

UPDATE: Increased rating to borderline reject post-rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an augmentation technique for video action recognition named GM (ghost motion). In specific, GM generates the fused clips via the shifted channel and temporal misalignment, encouraging the model to alleviate over-fitting. Besides, the authors apply logits smoothing on the temporal-sensitive benchmark (SSv1/v2) for emphasizing temporal dependency. The experiments demonstrate that GM improves frame-wise accuracy and boosts multiple baseline architectures significantly.
strength_and_weaknesses: Strength:
+ The way of combining channel disorder and temporal misalignment is novel and interesting. This approach enlarges the input space and transfers the semantic frames to adjacent frames. 
+ The method is computationally friendly and introduces minimal overhead.
+ The experimental results are comprehensive on various benchmarks including SSv1, SSv2, UCF, HMDB, ActivityNet and Mini-Kinetics.
+ The paper is generally easy to follow.

Weakness:
- Miss the ablation study on the disorder channel, which is one of the key components in the paper. Namely, what the performance will be if the mixing video is TSM-like, $(C^R_{i}, C^G_{i}, C^B_{i+1})\times H \times W$ or $(C^R_{i-1}, C^G_{i}, C^B_{i})\times H \times W$. 
- All the baseline performance is far lower than SOTA. I understand for a fair comparison, the paper unifies the training and test recipes and reproduces the 'bad' results of many previous methods. But the community would be more interested in how much the augmentation technique pushes prior arts and the absolute improvement on top of the baseline methods. 
- The logits smoothing is orthogonal to the ghost motion augmentation and more looks like add-on to improve the performance on Something-Something dataset. What is the result only equipped with ghost motion augmentation? Is it still effective?
clarity,_quality,_novelty_and_reproducibility: Clarity: The citing format is not precise. Many citations in the paper are supposed to be ~\citep{} instead of ~\cite{}. For example, in the first paragraph of the introduction: TSM Lin et al. (2019) ---> TSM (Lin et al. 2019). Abundant wrong citation formats distract the clarity of the paper. 

Reproducibility: The implementation details are specific and easy to follow for the practitioners. 

Originality: The method is novel and original, though the shifting trick is introduced in the video action recognition field before [1].

[1] TSM: Temporal Shift Module for Efficient Video Understanding. ICCV 2019
summary_of_the_review: Overall, it is a good paper. The paper introduces a simple yet effective augmentation method to resist over-fitting for video action recognition. The experiments verify the augmentation boosts the performance considerably across the board. Thus, my initial rating is borderline accept. I would like to see the responses from the authors during the discussion period.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a new method for data augmentation, Ghost Motion, applicable to video understanding problems, such as action recognition. The proposed method synthesizes the augmented video by taking a video and applying a random shift in time by 1 frame to one of the RGB channels. The authors also investigate temperature scaling as a technique to reduce overfitting in this setting. The empirical evaluation shows that the proposed method improves performance across multiple architectures and downstream tasks.

This was a borderline submission. The reviewers appreciated the simplicity of the proposed method, somewhat extensive evaluation, and the fact that this problem is often overlooked. They also pointed out several weaknesses: (1) Most of the architectures considered were not modern so it's unclear whether the benefits of this method transfer to the practically relevant setting. (2) Logit scaling is essentially orthogonal and already well-understood aspect of this work, and puts many results outlined in the paper under question as it's unclear how to disentangle the performance gain. (3) Motivation behind Ghost Motion in terms of why would this specific design choice bring significant practical benefits.

During the rebuttal some of these issues were partially addressed and some reviewers improved their score. Nevertheless, at the end of the discussion the general consensus remained -- the submission is still borderline and the reviewers feel that the work would mostly benefit a more focused audience. I will hence recommend rejection and suggest the authors to improve on (1-3), ideally removing the confounding factor (2), and provide a more detailed analysis in terms of running time impact across all the benchmarks.
justification_for_why_not_higher_score: (1) Most of the architectures considered were not modern so it's unclear whether the benefits of this method transfer to the practically relevant setting.
(2) Logit scaling is essentially orthogonal and already well-understood aspect of this work, and puts many results outlined in the paper under question as it's unclear how to disentangle the performance gain. 
(3) Motivation behind Ghost Motion in terms of why would this specific design choice bring significant practical benefits.
Decision: Reject