Probing into Overfitting for Video Recognition
Authors: Yitian Zhang
Yue Bai
Huan Wang
Yizhou Wang
Yun Fu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Action Recognition
Data Augmentation
Overfitting

Abstract: Video recognition methods based on 2D networks have thrived in recent years, leveraging advanced image classification techniques. However, overfitting is an even severe problem in 2D video recognition models as 1) the scale of video datasets is relatively small compared to image recognition datasets like ImageNet; 2) current pipeline treats background and semantic frames equally during optimization which aggravates overfitting. Based on these challenges, we design a video-specific data augmentation approach, named as Ghost Motion (GM), to alleviate overfitting. Specifically, GM shifts channels along temporal dimension to enable semantic motion information diffused into other frames which may be irrelevant originally, leading to improvement in frame-wise accuracy. In addition, for challenging video samples with significant temporal dependency (e.g., Something-Something), we further scale the logits during training to prevent overconfident predictions on background frames. Comprehensive empirical validation on various popular datasets shows that the proposed method can improve the generalization of existing methods and is compatible to other competing data augmentation approaches.

Under review as a conference paper at ICLR 2023
PROBING INTO OVERFITTING FOR VIDEO RECOGNI- TION
Anonymous authors
Paper under double-blind review
ABSTRACT
Video recognition methods based on 2D networks have thrived in recent years, leveraging the advanced image classification techniques. However, overfitting a severe problem in 2D video recognition models as 1) the scale of video datasets is relatively small compared to image recognition datasets like ImageNet: 2) current pipeline treats informative and -on-informative (e.g.. background) frames equally during optimization which aggravates overfitting. Based on these challenges, we design a video-specific data augmentation approach, named Ghost Motion (GM), to alleviate overfitting. Specifically, GM shifts channels along the temporal di- mension to enable semantic motion information diffused into other frames which may be irrelevant originally, leading to motion artifacts which describe the ap- pearance change and emphasize the motion salient frames. In this manner, GM can prevent the model from overfitting non non-informative frames and results in bet- ter generalization ability. Comprehensive empirical validation on various archi- tectures and datasets shows that GM can improve the generalization of existing methods and compatible to existing image-level data augmentation approaches to further boost the performance.
INTRODUCTION
Video recognition methods has evolved rapidly due the increasing number of online videos and success of advanced deep neural networks. Even if 3D networks (Feichtenhofer et al. 2019: Car- reira Zisserman| 2017 Tran al. 2015) provide straightforward solutions for video recognition,
2D based methods (Wang et al. 2016, 6 Lin et al. 2019 Li et al.) 2020: Wang et al.] 2021 still arouse researchers' interests because of their efficiency in both computing and storage. However, 2D net- works still suffer from overfitting issue. For instance, Something-Somethingg V1 (Goyal et al.) 2017) dataset which contains strong temporal dependency, the training and validation accuracy of TSM (Lin et al. 2219) is 31.22% and 45. 45.34%. respectively. Besides, its Expected Calibration Error (ECE) (Guo et al. 2017) is 25.83% which means the model gives overconfident predictions and brings negative impact when deploying the model in real scenarios.
There are two main reasons for overfitting in video: 1) video recognition benchmarks usually have fewer training samples compared with image classification datasets (c.g.. ImageNet (Deng et al. 2009) with 1.2 million training samples compared to Kinetics (Kay et al. 2017) with 240K videos). Furthermore, spatial-temporal modeling for video harder than recogmizing static images, which should require even more samples. 2) 2D based methods average the logits of all frames to vote for final decision which increases the tendency to overfit frames which contain less semantic informa- tion (e.g., background scene). In this view, these frames can be regarded as noise in optimization because they do not provide motion information for temporal modeling.
To alleviate overfitting, many attempts have been proposed. Dropout (Srivastava et al., 2014) and Label Smoothing (Szegedy et al.) 2016) are widely used in deep networks because of the regular- ization effects they bring. Designing data augmentation methods (Zhang et al.) 2017 Yun et al.) 2019 Vries & Taylor 2017: Cubuk et al.) 2020) to relieve overfitting is another line of research. Although some methods have shown effectiveness in image-level tasks, directly employing them video tasks may result in detrimental temporal representations as these methods are not specifically designed for video data and some transformations will break the motion pattern.

---

Under review as a conference paper at ICLR 2023
80
51
79
50
+0.82
78
+0.94
Acc (%)
77
48
TSM
TSM GhostMotion
76
+0.69
47
TAM
+1.10
TAM, GhestMotion
+1.54
46
+1.54
TEA, GhestMotion
45
TDN
73
TDN GhostMetion
TSM_ TSM_ GhostMotion
44
72
TSM
TAM
TEA
TDN
12
16
32
2D ConvNets
Number of Sampled Frames
(a) Ghost Motion leads to consistent improvement on Something Sovethinn-Sometingg VI with different models.
(b) Ghost Motion continuously improves the perfor- mance with different sampled frames on ActivityNet.
Figure The performance of Ghost Motion on different methods and datasets. Ghost Motion introduces negligible cost current competing methods and improves their generalization abilities.
In our work, we propose a data augmentation approach Ghost Motion (GM) which shifts channels along the temporal dimension propagate motion information adjacent frames. Specifically, all channels will be shifted for one step along the temporal axis leading to misaligned channel orders, and interpolate between the original video and the misaligned named ghost video. It diffuses motion patterns from salient frames into other frames to enhance the overall representative capacity. In this way, we can enforce the model to focus more on the informative frames and prevent it from overfitting non-informative frames to relieve overfitting. Although the shifting operation results in mismatched RGB orders, we surprisingly find the disorder is beneficial to improve generalization as well where it is elaborated in the analysis of Sec. 4.5 Ghost Motion effectively enlarges the input space without crashing the motion pattern and offers continuous variance in the input space which beneficial to improve the generalization abilitics of video recognition methods. Moreover, we find utilizing a hyperparameter Temperature to scale the logits before Softmax further mitigate overfitting and reduce the calibration error, especially challenging datasets such as Something-
form the new video,
V1&V2.
The proposed GM can be easily plugged into existing methods improve their generalization abili- ties with a few line codes and brings negligible computational costs. Shown in Fig. (a). GM results in consistent improvements performance of bascline model with different number of sampled frames. In addition, our method is compatible with existing image-level augmentations. We can jointly apply them to relieve overfitting and further boost the performance. The main contributions are summarized as follow:
various 2D models. In Fig (b), GM continuously improves the
We propose video recognition data augmentation method Ghost Motion (GM) which can effectively improve the generalization of current video benchmark models and is compatible with existing image-level data augmentation approaches.
We find smoothing the logits can prevent overconfident predictions to further alleviate over- fitting for temporal cemporal-dependent datasets such Something-Sommetinng Something.
We conduct comprehensive experiments validate the strength of Ghost Motion on vari- ous datasets and methods. Extensive ablation with detailed analysis illustrate the motiva- tion and intuition about our GM strategy.
2 RELATED WORK
2.1 VIDEO RECOGNITION
Video recognition has benefited lot from the development image classification, especially the 2D based methods which share the same backbone with the models in image classification. The main focus of these 2D methods lies in temporal modeling. For instance, TSN (Wang et al. (016) pro- poses average the prediction of all the frames to present the final prediction and lays a foundation for later 2D models. While TSM (Lin et al. 2019) is designed to shift parts the channels among adjacent frames for temporal modeling. Recently, people have paid more attention multi-scale
2

---

Under review as a conference paper at ICLR 2023
temporal modeling and many methods are proposed to not only model the motion variance locally, but capture the global temporal information as well (Li et al. 2020 Wang et al. 2021). Another line of research focuses on using 3D-CNNs to Iearn the spatial and temporal information jointly, such as C3D (Tran et al. 2.13), 13D (Carreira & Zisserman (017) and SlowFast (Feichtenhofer et al.) 2019). Though being effective, these methods usually cost great computations and are hard to be deployed on edge devices. Based on the structure of Vision Transformers (Dosovitskiy et al. 2020 Liu et al. 202 many Transformer-basse networks (Fan al.) 2021: Liu al. [2022] Li et al.
2022 have been proposed recently for spatial-temporal learning in video recognition and shown powerful performance.
Ghost Motion is similar with temporal shift module (TSM) (Lin et al., 2019) as both methods in- volves the shifting operation and enforces information exchange between neighboring frames. Nev- ertheless, the differences are: (1) Motivation: TSM is designed as a temporal modeling module which densely inserted in the network and hard to generalize to other architectures for further improvement; Ghost Motion is video data augmentation method which manipulates the input data, thus can be generalized to different networks and other image-level data augmentation methods. (2) Specific Designs: TSM shifts limited number of channels at every convolution block and the rest channels are kept in the original place; Ghost Motion shifts all channels initially which causes mis- match in the order of channels and the ghost video will be determined by the sampling coefficient, leading to a enlarged input space.
2.2 MITIGATING OVERFITTING
Lots of efforts have been made towards the study of regularization in the image domain to pre- vent overfitting. Dropout (Srivastava et al. 2014), Weight Decay (Krogh & Hertz 199T) and Label Smoothing (Szegedy et al. (0166) are widely utilized in image recogmition to reduce overfitting. Recently, RMS (Kim et al. 2020) is proposed to vary the magnitude of oow-freuenccy components randomly to regularize the 3D models. Besides these regularization methods, data augmentation methods have also proven to be effective to mitigate the effect of overfitting, especially for datasets which only have limited samples. Traditional augmentation methods, including random flipping, resizing and cropping, are frequently used to enforce invariance in deep networks (He et al.] 2016 Huang et al. 2017). Some studies have investigated to enlarge the input space by randomly occlud- ing certain regions in images (DeVsies & Taylor 2017). blending two images (Zhang et al. (2177) or replacing an image patch with the one in another image (Yun et al. 2019). In recent years, automatic augmentation strategies (Cubuk et al. 2018: 2020) are shown to be effective as it considers exten- sive augmentation types, such as color jittering, shear, translation, etc. However, there has been lack of studies for data augmentation methods in video recognition where overfitting is even severe problem compared to image classification.
Mixup iZhang et al. 2017) is widely used data augmentation method which encourages the model to behave linearly in-between training samples. Ghost Motion is related to mixup as both methods adopt interpolation to generate new training samples. Yet, we emphasize that the two methods enlarge the input space from two perspective and are complementary to each other. Mixup trains the network convex combinations of pairs of samples and their labels which can be understood as introducing more inter-class data. The success of mixup can be mostly attributed to the linear relationship between training samples and corresponding labels because of the regularization effects it brings (Carratino et al., 2020). Ghost Motion does not touch the label of training sapmples and can be considered as doing interpolation between the original input space and the generated input space which introduces more smoothed intra-class data.
METHODOLOGY
3.1 PRELIMINARIES
Given a video X containing T frames x {f1, f2, .... fr) with its one-hot class label y e Y. we denote X as the input video space, Y = [0, 1]" and K is the number of classes. A 2D network F maps x the vector p = F (u) where p- {PI, Po,,P,..... Pr} correspond to the predicted logits of T frames. The final prediction P of video is formed as:
P = 9 (pr,
3

---

Under review as a conference paper at ICLR 2023
where G is the aggregation function and the most common adapted form is the average function. Incorporated with standard categorical entrspy loss, the loss function is defined as:
Ph
C
=
yk log
Σ== e²
k=1
(2)
( PPgg
yk log
Σj= ( Po2
As = and ys. = O for all h o k, minimizing this loss function equals maximizing the log- likelihood of the right label. The maximum can hardly be achieved with finite Pk. but it can be approached if Ps >> Pa which means that the logit corresponding to the correct label is much greater than other logits. With this objective, it may result in overfitting as the model encouraged to produce overconfident predictions and brings difficulties to generalization. This effect may be even disastrous in video recognition as:
P Ph. (Pk2 Phan Pkrt) >>>>> (pha Phay Phrr)
which be achieved more easily if there is an imbalance distribution of (p1, P2..... Prr) in each video. Unfortunately, ing overconfident predictions may result in large logits of background frames which will overwhelm other frames' predictions and lead to wrong final decision.
frames exists in many video samples. The tendency of giv-
There challenging datasets in video recognition such Gonnthing-SSomehhig V1&V2 which involve significant temporal dependency and consists of abundant motion-irrevelan frames. To pre- vent the final predictions being overwhelmed by background frames. we propose smooth the logits of each sample on these datasets and we denote it as logits smoothing. This procedure is simi- lar with (Agarwala al. 2020) where the temperature tuned delicately improve the accuracy in image recognition. However, our aim is to prevent overconfident predictions of background frames and relieve the overfitting problem. We introduce a hyperparameter temperature т and assign it to each logits, so the standard cross-entropy loss can be redefined as:
C
yk log
P1
f=1
K
0
(4)
Uk log
K
Σj=1
As G is the average function, this operation equals smoothing the logits of every frame. When r > 1, the gap between I'm and Pa can be reduced which alleviates the overfitting problem. While temper- ature scaling is commonly used in model calibration (Guo et al. 2017), it only applies temperature during testing for calibration which does not help to improve the generalization of models.
3.2 GHOST MOTION
In this section, we introduce our proposed data augmentation method Ghost Motion (GM) which is specially designed for video recognition. Considering a RGB frame ft E (C cf (. with 3 channels in total, we can denote the video sequence as Tx((" c "C")xxXW
Given random probability μ, we generate misaligned video x' E X' by shifting channels for one step along the temporal dimension with random direction, so that we can represent the generated misaligned frame as:
f' E'ER(e (ER(C (h.):.) r S 0.5, C'ER(CE,CP.C)).R
(5)
/ > 0.5.
With this operation, enable information exchange in x'. The misaligned frame f is com- posed of three channels, with two channels from the original frame and one from adjacent frame which explicitly facilitate information interaction between neighboring frames. However, directly
4

---

Under review as a conference paper at ICLR 2023
fi
Right Shifting
fi+1
# 0.5
Misaligned x'
Bi
R
G
B
Ri-1
GH+I
p- (0.1)
Original x
R;
G
B
R
G
B
A-Beta(a, a)
Misaligned x
G
B
R
G
B
R
1-2
H1 > 0.5
Left Shifting
Illustration of Ghost Motion. Given video (/1, J2. fr), its channels will be shifted along the temporal dimension to generate a misaligned video x' and the shifting direction is decided by random probability μ sampling from uniform distribution. Interpolation beween origi- nal video and misaligned video will be implemented to generate the Ghost Video. The coefficient is sampled from beta distribution to control the degree of interpolation.
Figure 2:
using the misaligned video as the input source may not be optimal because the generation fol- lows a fixed pattern and the discrete variance may not favorable. Therefore, we interpolate be- tween the original video and misaligned video, and sample coefficient À from the Beta distribution Beta (a, a) to control the strength. our experiments, we set a to so that À is sampled from the uniform distribution (0, 1) and we can denote the generated training sample it and the corresponding label as:
i = Ax + (1 - A) 1 y
(6)
We name E ghost video as there exists artifacts in certain frames. For adjacent frames with great motion variance, the shifting operation will transfer semantic information to other frames, resulting in motion artifacts which looks like the RGB difference data. The reason is motion patterns locates in different spatial regions for neighboring frames and the overlay of channels from different frames will preserve the motion information of both. With this design, the generated data will enforce the model to focus more on the temporal variance guided by the motion artifacts. Besides, motion in- formation may be diffused into non-informative frames and increase their chance be correctly classified to alleviate the imbalance in aggregation function G. Though the shifting operation leads to mismatch RGB orders, we find the color alteration is beneficial to further improve the gener- alization. Note that x' is produced by x, so ghost video v shares the same label with original video sample We denote the generated channels of ghost frame Je as:
{
ČR XCR + (1 X) CG
CR XCR + (1 (1-X) CB
c X + (11+++1110 (1- CB
1 U.S.O. 0.5,
G A + (1- C) C
/ p>0.5, 0.5,
(7)
CB XC + (1-11 A) C) C
CR
C AC + C1-
where Č, ÖG ö. stand for the RGB channels of the ghost frame respectively. Due to the introduced X. the degree of temporal variance can be measured with a linear relation, leading to continuous variance in the input space.
EMPIRICAL VALIDATION
In this section, empirically evaluate the performance of the proposed method various video recognition datasets. We first validate the effectiveness of Ghost Motion (GM) on different mod- els and various datasets. Second, we present the comparisons of our method with competing data augmentation methods in image recognition and prove that GM is compatible with them. Further, extend GM to other structures and depths of network. Finally, analyze the effect of logits smoothing, discuss the training cost of different data augmentation methods, and present compre- hensive ablation results of design choices and visualizations results.
4.1 EXPERIMENTAL SETUP
Datasets. Our method is evaluated on six widely-used video datasets. For datasets which contain strong temporal dependency: (1) -SaethiingSSmmthingg V1&V2 (Goyal et al. 2017) are hu-
5

---

Under review as a conference paper at ICLR 2023
Table I: Evaluation of Ghost Motion (GM) Something-Something datasets with different models. The best results are faced Note that we does not apply random flipping on these two datasets, so the reproduced results of TANet and TDN are lower than the number reported in their papers.
Sth-Sth V1
Sth-Sth V2
Method
Accl.(%)
Acc5.(%)
Accl.(()
Accl.(%)
Acc5.(%)
Acc1.(%)
TSN (Wang el al.
2016
17.21
43.17
30.56
61.21
+1.02
+1.23
TSN+GM
18.23
44.88
31.79
62.52
TSM (Lin el al. TSM-GM
2019
45.
74.45
59.14 19.14
85.60
+1.54
+0.69
46.88
75.76
59.83
TANet (Täu et al. TANe++GIM
2021b
45.94
75.56
59.17
85.18
+1.10
+0.87
47.04
76.05
60.04
86.25
TEA eL 2020
48.67
77.94
61.11
87.20
+0.94
+0.64
TEA+GM
49.61
78.03
61.75
87.04
TDN (Wang et al. 2021
78.74
63.10
88.12
+0.82
+0.71
TDN-GM
50.51
79.06
63.81
88.59
Table 2: Evaluation of Ghost Motion (GM) on UCF101, HMDB51, ActivityNet and Mini-Kinetics with TSM (Lin et al. 2019) and TEA (Li et al. 2020 The best results are bold-faced.
UCFI01
HMDB51
ActivityNet
Mini- Kinetiess
Method
Acc1.(%)
A.c1.(%)
A A%)
mAP(%)
A mAP(%)
Aoc1.(%)
& Ac)1.
TSM
79.57
48.63
73.10
74.11
TSM+GM
80,57
+1.00
51.11
+2.48
74.72
+1.62
74.82
+0.71
TEA
80.10
+ +1.05
49.61
+3.00
73.09
+1.40
75.96
+00.5
TEA+GM
81.15
52.61
74.49
76.51
man action datasets which includes 98k and 194k videos respectively. For datasets which require less temporal modeling: (2) ActivityNet-vI.3 (Caba Heilbron et al. 2015) contains 10,024 training videos and 4,926 validation videos with 200 categories and the average duration is 117 seconds; (3)
UCF101 (Soomro al.) 2012) dataset originally consists of 13,320 videos with 101 classes and we adopt the first training/ testing split for training and evaluation; (4) HMDB51 (Kuchne et al. 2011) dataset made up of 6, 6,766 videos with 51 classes and we use the original first training/testing split for training and evaluation as well; (5) Kinetics400 (Kay et al. 2017 is large scale video recog- nition dataset which is categorized into 400 action classes. Kinetics a subset of Kinetics400 with 200 categories in total and we select the classes following (Meng al. 2020).
Implementation details. We uniformly sample 8 frames for each video unless specified. During training, the training data is random cropped to 224 x 224 following (Zolfaghari et al., 2018)) and we did not perform random flipping on Nomething-Sometthne frames are center-cropped to 224 x 224 and adopt one-crop one-clip per video during evaluation for efficiency. Similar with CutMix (Yun et al.] 2019). we add a hyperparamter @ which denotes the probability whether to implement GM for every batch samples. We @ as 0.5 Something- and 0.2 in other datasets. Note that we only
V1&V2. At inference stage, all
Something V1&V2 which are temporal-depeneee apply logits smoothing on SonnethingSSomething V1& V2 to calibrate the overconfident predictions and we use the symbol θ to denote that this dataset is trained with logits smoothing. Due to limited space, more implementation details can be found in supplementary material.
4.2 MAIN RESULTS
Results on different methods. We first empirically evaluate the performance of Ghost Motion (GM) on Something-Some Something VI and V2 datasets using different models in Tab. [ It can be observed that GM consistently increases the generalization ability of these methods on both datasets, including the state-of-the-e model TDN (Wang et al. 202T) T) in 2D methods. The improvements on Sth-Sth V1 are generally larger than those on Sth-Sth V2 as V2 contains more training samples and models will suffer less from overfitting this dataset. Specifically, the accuracy of TEA (Li al, 2020) can be improved to 49.61% % on Sth-Sth VI which is similar with our reproduced results of TDN (Wang al. 2021 adapting more complex structures and taking much longer training time.
Results on different datasets. We first validate Ghost Motion on UCF101, HMDB51 and Activi- tyNet datasets in Tab. 2 The improvements brought by GM are, quite obvious, greater than 1% as the scales of these datasets are relatively small and the overfitting phenomenon is more severe on them. We further implement GM Mini-Kined which is one of the large scale datasets in video

---

Under review as a conference paper at ICLR 2023
Table 3: Comparisons with competing data augmentation methods on different datasets using TSM (Lin et al. 2019). The best results are bold-faced.
UCFI01
HMDB51
Sth-Sth V1
Method
Acc1.(%)
Accl.%)
Accl((%)
Accl.(%)
Accl((%)
Accl.(%E)
79.57
48.63
45.34
TSM+Cutout DeVeiss & Taylor 2017
79.04
-0.53
48.69
+0.09
44.68
00,66
TSM+CutMix Yum et m. 2019
79.16
-0.41
48.17
-0.46
44,99
0.35
TSM+Mixup Zhang cl al:
79.83
+0.26
50.39
+1.76
+0.69
TiM+Auggix (Hendrycks
2019
79.94
+0.40
50.59
+1.96
46.23
+0,89
TSM+Rand Augient (Cubuk et
2020
82.13
+2.16
$2.55
+3.92
47.20
+1.86
TSM+GM
80.57
+1.00
51.11
+2488
46.88
+1.34
TSM+CiM-MMxxpp
N0.99
+1.42
3T.44
7-281
47.19
71.85
TSM+
81.10
+1.53
52.68
44.05
46.93
+1.59
TSM+GM+RandAugmeen
83.24
+3.67
53.79
+5.16
48.06
+2.72
Table Evaluation of Ghost Mo-
Table 5: Evaluation of Ghost Motion (GM) on Something- Something VI and Mini-Kinetics with SlowOnly (Feichten-
tion (GM) on Something-Sometthne VI with different depths.
hofer al. 2019) and Uniformer (Li et 2022).
Sth-Sth V1
Sth-Sth V1@
Mini-Kinetics
Method
Method
Aec1.(%)
Accl.(%)
Acc1.(%)
Accl.(%)
Acc1.(%)
Acc1.(%)
TSM(R18)
TSM(R18)+GH
39.80
39.04
+0.76
SlowOnly
SlowOnly+GM
43.00
+1.19
65.90
+0.61
(SM(R101))
45.85
Uniformer
54.45
71.47
+2.20
+0.84
+1.00
TSM(RI01)+CM
48.05
Uniformer+GM
55.29
72.47
recognition but GM still increases the generalization ability of the baseline models which validates has the same number of samples per class
the effectiveness of our method. Note that with Kinetics400 which means the overfitting phenomenon will not be more severe on this dataset and conduct experiments it for efficiency.
4.3 COMPARISON WITH OTHER APPROACHES
We compare our method with competing data augmentation methods on UCF101, HMDB51 and Something-Sone Something V1 based on TSM in Tab. 3 It can be observed that popular data augmen- tation methods like Cutout Vrees Taylor 20.7). CutMix (Yun et al.) 2019) results negative effects in video recognition. One possible explanation is that these methods break the motion pattern in video sequences and bring challenges to temporal modeling which is extremely essential in video recognition. Mixup (Zhang et al. (017)) shows consistent improvement on these three datasets as it enforces a regularization effects which is helpful to deal with overfitting. AugMix (Hendrycks et al.) 2019) and RandAugment (Cubuk et al. 2020) exhibits great generalization ability as both methods involve multiple augmentation operations and their magnitudes will be adaptively sampled.
Our method outperforms all approaches except for RandAugment, which is composed of various augmentation operations. However, note that GM is compatible with these methods as they only consider image-level transformations and GM mainly focuses on temporal-wise interactions to large the input space. We further combine GM with Mixup, AugMix and RandAugment on these three datasets and the performance can be improved by 3.67%, 5.16% and 2.72% at most.
4.4 FURTHER VERIFICATION
Results on different depths of network. In previous experiments, methods are built on ResNet- 50 (He et al. 2016) and we further conduct experiments on ResNet-18 18 and ResNet- to verify the effectiveness our method when evaluated on models with different representation abilities. We can observe from Tab. 4 that GM consistently improves the performance of baseline method and the improvements are increasing (0.76%. 1.54%, 2.20%) with deeper network. The explanation is that the overfitting phenomenon is more obvious deeper network as it has stronger representation ability to fit the training data and GM can effectively alleviate this phenomenon which leads to greater improvements in accuracy compared with shallow networks.
Results on different structures of network. Although Logits Smoothing and Ghost Motion are in- spired by the analysis in based methods, method can be implemented other video recog-

---

Under review as a conference paper at ICLR 2023
Val Frame Accuracy (96)
Accuracy (%6
40-
ECE (%)
20
16
14-
30
15
25
10
10
20
30
40
20
30
40
50
10
20
30
50
Epoch
Epech
Epoch
(a) Validation Accuracy
(b) Validation Frame Accuracy
(c) Expected Calibration Error
Figure 3: Validation Accuracy, Validation Frame Accuracy and Expected Calibration Error (ECE)
curves of TSM trained with Temperature T = 1,2, 3 on Something-Something VI dataset.
Table 6: Comparison of Label Smooth-
Table 7:
Training time of different data augmenta- tion methods on Something-Something VI, includ- ing Cutout (CU), CutMix (CM), Mixup (MU), Aug- Mix (AM), RandAugment (RA) and GM.
ing (LAS) and Logits Smoothing (LOS) on Something Something-Something Something VI. We set 0.2 in LAS and 7 in LOS.
Accl.(%)
Training cost (GPU hours)
Method
Method
Baseline
LAS
LOS
LAS+LOS
Baseline
CU
CM
MU
AM
RA
GM
TSM
45.34
45.88
45.85
46.21
TSM
11
12
12
15
13
12
14
nition architecture as it does not touch the structures inside the networks. Therefore, we implement our method on 3D-based method SlowFast (Feichtenhofer et al. [201)] with the slow branch only and Transformer-basee
method Uniformer (Li et al. (002 to test the generalization capability of our method. The results in Tab, show that GM leads to improvements in accuracy on both structures and datasets, indicating our method can be applied to other structures to improve their generalization capability well. The possible explanation is that the data generated by GM has motion artifacts and the explicit pattern in data will enforce the model to focus more on the temporal variance, pre- venting it from overfitting non-informative frames.
4.5 ABLATION AND ANALYSIS
Effects of logits smoothing. In this part, we train TSM with T = 1, respectively and try to explain what effects logits smoothing brings. We denote Frame Accuracy as the average accuracy of all sampled frames in the training set. Shown in Fig. (a)(b). the model trained with higher tem- perature has clearly higher frame accuracy which leads to advantages in accuracy of video samples as well. From Fig. (c). one can see that higher temperature results in smaller Expected Calibration Error (ECE) (Guo et al. 2017) which is beneficial to prevent overconfident predictions. Further, we compare logits smoothing with label smoothing (Szegedy et al. 2016) in Tab 6 Results show that both methods lead to increase in accuracy because of the regularization effects and combining two methods can lead to further improvement they are complementary to each other.
Training cost. We measure the training time of different data augmentation methods with the same training setting in Tab 7 Theoretically, these should bring negligible extra computational costs, but we observe time increases in all methods. Mixup costs the most time which means the interpolation of raw data and the manipulation of labels will increase wall-clock time despite their theoretical efficiency. The training time of Cutout and CutMix are less as they only process a small crop of the raw data compared Mixup and GM, which need to do interpolation on the whole data. Though AugMix and RandAugment involve multiple augmentation operations, these transformations are traditional data augmentation techniques which can be efficiently implemented with small time creases. We hope these analysis can provide insights for later works.
Design choices. this part, we first empirically compare different design choices of Ghost Motion (GM) in Tab. [X At first, we explore other selections of hyperparameter a which control the beta distribution and the results show that a = 1 results in the best performance. Besides, we compare GM with its variants which shift channels for two step and shift for single direction. The two- step shift obtains little worse results than one-step shifts and the possible reason is that two-step shift will lead to offset in frame location. GM outperforms one-way shift obviously as it offers more variance in data which is helpful to improve the generalization of baseline method. Moreover, implement GM in the feature space following (Verma et al. 2019) which applies Mixup in the
8

---

Under review as a conference paper at ICLR 2023
intermediate features. However, results show that this measure leads to negative effects compared with GM and we consider the misaligned channel order caused by GM in the feature space may be one of the possible reasons.
As implementing GM leads to the stress of the motion information and alteration of the video color, we compare GM with two methods: (I) Motion Exchange(MIE): shift- ing channels along temporal dimension to enforce information exchange but keep the original RGB order; (2) Color Jittering(CJ):
Table 8: Ablation of design choices of Ghost Motion ActivityNet. The best results fold-faced and mAPs are averaged over multiple runs.
Method
Specification
mAP(%)
TSM (Lin einetar al. 2019
73.10=0.14
TSM+GM
Beta (0.5, 0.5)
0.095
randomly changing the brightness, contrast, saturation and hue of the video. We can ob- serve both ME and CJ lead increase in accuracy and the improvement that GM re- sults is bigger than the increase brought by ME+CJ which means the combination of the two effects can boost the performance of each other. Another important design of GM is the interpolation between the origi- nal video and the misaligned video. Therefore, we compare GM with its variant which directly using the misaligned video as the input. The performance of GM without interpolation is lower than our method which demonstrates the superiority of this design it offers a continuous distribution of the input data which enlarges the input space and is beneficial for classification.
TSM+GM
Beta (2.0. 2.0)
0.09
TSM+GM
Two-Step Shift
7.23
ISM+GM
One- One-Way Shift
73.91=0.03
TSM+GM
Manifold
73.22=0015
TSM+ME
Motion Exchange
73.79=0.10
TSM+CJ
Color Jittering
73.89=0.25
TSM+GM
w/o interpolation 74.03=0.05
TSM+GM
74.72-0.04 0.04
(a) Frame-wise predictions of TSM trained with stan- dard protocol. Original frames shown.
(b) Frame- predictions of ISM trained with Ghost Motion. Frames generated GM shown.
Figure 4: Visualization results on Something- Something-Something VI dataset. Frames annotated with red rectangles wrongly classified and the green represent correctly categorized frames.
Visualizations. We visualize the frame-wise predictions of TSM in Fig. The left two samples are wrongly classified by the model trained with standard protocol as the final predictions are over- whelmed by motion-irreleve irrelevant frames. In contrast, the right samples are correctly classified by the model trained with GM and there exists motion artifacts in some frames as each frame exchanges parts of the channels with adjacent frames. The motion artifacts looks like the RGB difference be- tween two consecutive frames which describe the appearance change. However, compared to RGB difference, the data generated by GM does not loss information the static regions parts of the channels rather than subtracting the similar parts. Therefore, the data generated by GM could be seen result, the model trained with GM can focus more on motion frames guided by the motion artifacts, which prevents the model from overfitting background frames towards the ground truth label and results in better generalization ability.
only shift
modality which restore the good attributes of RGB and RGB difference. As
5 CONCLUSION
This paper probes into the overfitting problem in video recognition and proposes special designed data augmentation method Ghost Motion (GM). Specifically, GM shifts channels along the tempo- ral dimension to enforce motion-relevant information exchange among neighboring frames. This can enhance the learning process as motion information can be diffused into other frames, result- ing in motion artifacts which emphasize the motion-relevant frames. By doing GM can prevent the model from overfitting non -on-informativ frames and increase its generalization ability for better performance. Comprehensive experiments popular datasets and methods demonstrate the ef- fectiveness and generalization of GM. Moreover, GM is compatible with existing image-level data augmentation approaches further improve the performance of video recognition methods.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
In our paper, we strictly follow the ICLR ethical research standards and laws. To the best of our knowledge, our work abides by the General Ethical Principles.
REPRODUCIBILITY STATEMENT
We adhere to ICLR reproducibility standards and ensure the reproducibility of our work. All datasets we employed are publicly available. As we have included comprehensive empirical validations various methods and datasets in work, provide corresponding training setting for method in the appendix. Moreover, we will provide the code to reviewers and area chairs using an anonymous link when the discussion forums open.
REFERENCES
Atish Agarwala, Jeffrey Pennington, Yann Dauphin, and Sam Schoenholz. Temperature check: theory and practice for training models with softmax-cross-entropy losses. arXiv preprint arXiv:2010.07344 2020.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regular- ization. arXiv preprint arXiv:2006.060449 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinctics dataset. In CVPR, 2017.
Ekin Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment:
Learning augmentation policies from data. arXiv preprint 1805v:/805.09500 2018.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated
data augmentation with reduced search space. In CVPRW, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Terrance De Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /708. 04552, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19929 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz StuellrrFreittgg et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
10

---

Under review as a conference paper at ICLR 2023
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint wwXh::112.002781, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint (rXiv:/705..00
2017.
Jinhyung Kim, Seunghwan Cha, Dongyoon Wee, Soonmin Bae, and Junmo Kim. Regularization on spatio-temporally smoothed feature for action recognition. CVPR, 2020.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. NeurlPS, 1991.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 2011 International conference on computer vision, pp.
IEEE, 2011.
Kunchang Li, Yali Wang, Junhao Zhang. Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint arXiv:2201. 00940,, 2022.
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In CVPR, 2020.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021a.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022.
Zhaoyang Liu. Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recognition. In ICCV, 2021b.
Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. Ar-net: Adaptive frame resolution for efficient action recogni- tion. In ECCV, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2t:. 2012.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, 2015.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
11

---

Under review as a conference paper at ICLR 2023
Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. In CVPR, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1710.0921. 2017.
Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Efficient convolutional net- work for online video understanding. In ECCV, 2018.
A APPENDIX
A.1 DATA PRE-PROCESSING
We uniformly sample 8 frames for each video all datasets, except the experiment over Uni- VI dataset where we sample 16 frames for ev-
former (Li et al. 2022) on
ery video. During training, the training data is randomly cropped to 224 224 following (Lin) (Goyal et al. 2017)
et al. 2019). and we perform random flipping except for Something-Somett
datasets. At inference stage, all frames are center-cropped to 224 224, obtaining a one-crop one- Something-
clip per video for evaluation, except the experiment over Uniformer (Li al. 2022 Something VI dataset where we adopt three-crop for evaluation.
A.2 IMPLEMENTATION DETAILS
All results reported in our paper the models are pretrained on ImageNet except for the experiment of Uniformer (Li al. (022) on Something-Somen Something VI which utilize the pre-trained model on Kinetics400 following the origi- nal setting. We reimplement all 2D methods in our code and use the official implementations for Slowonly Feichtenhofer al. 2019 and Uniformet
NVIDIA Tesla V100 GPUs with mixed precision. All
For Ghost Motion, a is set to in all our experiments and we change the hyperparamter which de- notes the probability whether to implement GM for every batch samples on different datasets. Note that we only apply Logits Smoothing to temporal-dependent datasets such as Something-Something V1&V2.
A.2.1 TEMPORAL SENSTTIVE DATASETS
We set @ to 0.5 to enable more variance in training samples on Something-SSmeethnng V1&V2 as the overfitting phenomenon is the most severe on these two datasets. For Logits Smoothing, the temperature set to 3,2 for
V1&V2, respectively.
A.2.2 TEMPORAL INSENSITIVE DATASETS
We e to 0.2 for Ghost Motion on Mini-Kinetics (Kay al. 2017). ActivityNet (Caba Heilbron et al. 2015), HMDB51 (Kuehne et al. 2011) and UCF101 Soommo et al. 2012) and we did not apply Logits Smoothing on these datasets.
http://withhic..ooo.eeeeeeeeteet fication
12


Reviews:
[{"review_id": "u4_v7_F4NWE", "reviewer": "Reviewer_Xnqd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new data augmentation for video classification that consists of shifting one of the RGB channels forwards or backwards by 1 frame. They call this augmentation \"Ghost Motion\" (GM). They also show that scaling the logits by a temperature helps generalization. The authors claim that GM improves results for a variety of different models and datasets.", "summary_of_the_review": "The authors propose a  input augmentation that appears to be effective on a wide range of problems and models. The empirical evaluation covers a wide range of models and datasets, yet the numbers raise some questions (see above).\n\nUPDATE: after the author's revision of the paper, I have updated my score to reflect that I think the method now meets the standards for publication.", "strength_and_weaknesses": "Strengths:\n1) The proposed data augmentation technique is well explored and has an extensive empirical evaluation. I appreciate that the authors investigated it under a number of different models and datasets. While they briefly also investigate modern approaches based on Transformers, most of their results focus on out-dated models based on 2D CNNs. I feel that the paper could be improved by focusing more on modern models. E.g. Table 1 contains results for TSN, which is by now a 6 year old method. I don't think these results have any relevance for the research community. Adding a modern, state-of-the-art method like Video Swin Transformer[4], MViT[5] or MTV[6] instead would considerably strengthen the paper. \n\n\nWeaknesses\n1)  The \"logit scaling\" part of the current manuscript is weak: it's a completely unrelated regularization technique to Ghost Motion, and as the authors do mention, the same technique was furthermore already proposed and investigated for classification tasks in [3]. So mentioning this as a novel contribution is a stretch.  Furthermore it seems to be just another variation of label smoothing [1] or logit squeezing [2], neither of which the authors compare to or even mention. I'd suggest de-empathising this part of the paper and focusing completely on Ghost Motion instead. Otherwise, the authors need to properly compare logit scaling to competing methods.\n\n2)  The results for the baselines stated by the authors do not match those from the original publications. For example, the authors claim in Table 1 that the basic TSM module achieves 45.34% accuracy on SomethingSomethingV1, and that TSM+GM together achieve 46.88%.\nHowever, the original TSM publication states that TSM alone achieves 47.3% accuracy. Similarly, the original TDN publication claims 52.3% top1 accuracy on SomethingSomethingV1, while this manuscript claims that TDN alone achieves 49.69% and TDN+GN  50.51%. I did not check the rest of the numbers, but this small spot check seems weird.  The authors need to address this discrepancy.\n\n3) While the authors claim several times that GM \"brings almost zero costs\", that claim was never backed  up. I'd appreciate if the authors could provide insights on the computational cos of their method: does adding GM really not increase the wall-clock time of training? \n\n4) The authors give all accuracies with 4 significant digits, which would suggest a large number of repeat experiments, yet they do neither mention this nor give error bars. I would strongly suggest they add unambiguous error bars for their results.\n\n\n[1] Mueller et al., \"When does label smoothing help?\", NeurIPS 2019 (and references therein)\n\n[2] Shafahi et al, \"Label Smoothing and Logit Squeezing: A Replacement for Adversarial Training? \", ICLR 2019\n\n[3] Agarwala et al, Temperature check: theory and practice for training models with softmax-cross-entropy losses, arxiv 2019\n\n[4] Liu et al, \"Video Swin Transformer\", CVPR 2022\n\n[5] Fan et al, \"Multiscale Vision Transformers\", ICCV 2021\n\n[6] Yan et al, \"Multiview Transformers for Video Recognition\", CVPR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eveHfO7L3IV", "reviewer": "Reviewer_6srd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes two contributions to reduce overfitting of video recognition models: Ghost Motion (GM) data augmentation which randomly shifts one RGB channel forward or backward in time, and logit smoothing (temperature scaling) to reduce overconfidence on background frames. The authors hypothesize that GM works by exchanging information between neighboring frames, creating motion artifacts that guide the model to focus on temporal cues. Experiments on multiple video action datasets show the proposed training scheme reduces overfitting and improves test accuracy of classifiers.", "summary_of_the_review": "Although this work has its strengths, I lean toward rejection of the paper at its current state, due to lack of technical contributions and empirical justifications of some claims made in the paper, as detailed in the weaknesses above. I look forward to the authors' response to resolve any confusions/misunderstandings.\n\nUPDATE: Increased rating to borderline reject post-rebuttal. ", "strength_and_weaknesses": "## Pros\n\n- This paper studies an important problem of video data augmentation, where there has been limited prior work in the literature.  \n- The proposed method is simple and appears effective across datasets and models.  \n- Ablation experiments on the augmentation parameters are provided.  \n\n## Cons\n\n- Methods  \n\t- I don't feel that the motivation for GM is strong enough. Specifically, how does the information transfer between frames lead to stronger temporal modeling at test time (when the channels are not shifted)? It would be nice to have experiments supporting that models trained with GM is more sensitive to temporal cues than spatial ones.  \n\t- I am also not sure about the design choice, where only red or blue channels are shifted, but not the green channel. It would be nice if more explanations or justifications are provided. It would also be interesting to see if shifting all channels with interpolation can achieve similar effects.  \n\t- The technical contribution from this work is somewhat limited: GM mirrors the temporal shift operation of TSM in the input image space, with similar objectives (improve temporal modeling); temperature scaling has been widely used in model calibration literatures to reduce overfitting, and should not be considered an original contribution from this work. \n\n- Experiments\n\t- The effect of logit smoothing and ghost motion is not well disentangled in the experimental results. From figure 3 it appears that temperature tuning alone improves accuracy by ~1%. Is this the main source of improvements in the main tables, or do the gains come from the GM augmentation instead?  \n\t- The improvements over existing image-based augmentations is not particularly significant (e.g. <1% over MixUp).\n\t- There does not seem to be comparison to prior video data augmentation methods such as VideoMix (https://arxiv.org/abs/2012.03457).  \n\n- Minor comments\n\t- I assume that the effectiveness of method depends on the video frame rate. If frame sampling is dense enough, adjacent frames will be too similar that shifting does not make a significant difference. Maybe it would be nice to include additional discussion/experiments on this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UgGANSuJ2pm", "reviewer": "Reviewer_Xyj5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an augmentation technique for video action recognition named GM (ghost motion). In specific, GM generates the fused clips via the shifted channel and temporal misalignment, encouraging the model to alleviate over-fitting. Besides, the authors apply logits smoothing on the temporal-sensitive benchmark (SSv1/v2) for emphasizing temporal dependency. The experiments demonstrate that GM improves frame-wise accuracy and boosts multiple baseline architectures significantly.", "summary_of_the_review": "Overall, it is a good paper. The paper introduces a simple yet effective augmentation method to resist over-fitting for video action recognition. The experiments verify the augmentation boosts the performance considerably across the board. Thus, my initial rating is borderline accept. I would like to see the responses from the authors during the discussion period.", "strength_and_weaknesses": "Strength:\n+ The way of combining channel disorder and temporal misalignment is novel and interesting. This approach enlarges the input space and transfers the semantic frames to adjacent frames. \n+ The method is computationally friendly and introduces minimal overhead.\n+ The experimental results are comprehensive on various benchmarks including SSv1, SSv2, UCF, HMDB, ActivityNet and Mini-Kinetics.\n+ The paper is generally easy to follow.\n\nWeakness:\n- Miss the ablation study on the disorder channel, which is one of the key components in the paper. Namely, what the performance will be if the mixing video is TSM-like, $(C^R_{i}, C^G_{i}, C^B_{i+1})\\times H \\times W$ or $(C^R_{i-1}, C^G_{i}, C^B_{i})\\times H \\times W$. \n- All the baseline performance is far lower than SOTA. I understand for a fair comparison, the paper unifies the training and test recipes and reproduces the 'bad' results of many previous methods. But the community would be more interested in how much the augmentation technique pushes prior arts and the absolute improvement on top of the baseline methods. \n- The logits smoothing is orthogonal to the ghost motion augmentation and more looks like add-on to improve the performance on Something-Something dataset. What is the result only equipped with ghost motion augmentation? Is it still effective?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject