Don't Judge by the Look: Towards Motion Coherent Video Representation
Authors: Yitian Zhang
Yue Bai
Huan Wang
Yizhou Wang
Yun Fu
Venue: ICLR 2024 poster (2024)
Keywords: Video Understanding
Data Augmentation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present a motion coherent augmentation for video understanding which could enforce the model to learn appearance invariant representations.

Abstract: Current training pipelines in object recognition neglect Hue Jittering when doing data augmentation as it not only brings appearance changes that are detrimental to classification, but also the implementation is inefficient in practice. In this study, we investigate the effect of hue variance in the context of video understanding and find this variance to be beneficial since static appearances are less important in videos that contain motion information. Based on this observation, we propose a data augmentation method for video understanding, named Motion Coherent Augmentation (MCA), that introduces appearance variation in videos and implicitly encourages the model to prioritize motion patterns, rather than static appearances. Concretely, we propose an operation SwapMix to efficiently modify the appearance of video samples, and introduce Variation Alignment (VA) to resolve the distribution shift caused by SwapMix, enforcing the model to learn appearance invariant representations. Comprehensive empirical evaluation across various architectures and different datasets solidly validates the effectiveness and generalization ability of MCA, and the application of VA in other augmentation methods. Code is available at https://github.com/BeSpontaneous/MCA-pytorch.

Published as a conference paper at ICLR 2024
DON'T JUDGE BY THE Look: TOWARDS MOTION Co- HERENT VIDEO REPRESENTATION
Yitian Zhang Yue Bai' Huan Wang' Yizhou Wang' Yun Fu
'Department of Electrical and Computer Engineering, Northeastern University 2 Khouury College of Computer Science, Northeastern University
ABSTRACT
Current training pipelines in object recognition neglect Hue Jittering when doing data augmentation not only brings appearance changes that are detrimental classification, but also the implementation is inefficient in practice. In this study, we investigate the effect of hue variance in the context of video understanding and find this variance to be beneficial since static appearances are less impor- tant in videos that contain motion information. Based on this observation, we propose data augmentation method for video understanding, named Motion Co- herent Augmentation (MCA), that introduces appearance variation in videos and implicitly encourages the model to prioritize motion patterns, rather than static appearances. Concretely, we propose an operation SwapMix to efficiently mod- ify the appearance of video samples, and introduce Variation Alignment (VA) resolve the distribution shift caused by SwapMix, enforcing the model to learn appearance invariant representations. Comprehensive empirical evaluation across various architectures and different datasets solidly validates the effectiveness and generalization ability of MCA, and the application of VA in other augmentation methods. Code is available at
INTRODUCTION
Video understanding has evolved rapidly due the increasing number of online videos. Even though, current methods still suffer from the overfitting issue. For instance, TSM et al. (2019)
demonstrates a substantial disparity between its training (78.46%) and validation accuracy (45.63%
on Something-Something VI Goyal et al. (017) dataset with a notably high Expected Calibration Error (ECE) Guo et al. (2017) of 23.25%. indicating its tendency for overconfident predictions. A plausible explanation is that video understanding benchmarks usually have a smaller pool of train- ing samples compared with object recognition datasets, e.g., ImageNet Deng et al. (2009) with 1.2 million training samples compared to Kinetics400 Kay et al. (2017) with 240K videos.
To alleviate overfitting, there are many data augmentation methods Zhang et al. (2017); DeVries & Taylor (2017); Yun et al. (2019); Cubuk et al. (2020) being designed to conduct various transfor- mations to training samples, and current state-of-the-ar methods Li et al. 2202aa b) in video under- standing often incorporate multiple data augmentation operations for better generalization ability. Among them, Color Jittering is widely used to transform color attributes like saturation, brightness, and contrast. However, notice that Hue Jittering, which changes the attribute of hue in color, is often overlooked in current object recognition Liu et al. [2022b] 2021) methods and the reasons can be concluded in two folds: first, Hue littering is thought to be detrimental to object recognition as the effect of hue variance looks like changing the appearance of objects which will increase the difficulty in classification; second, the current implementation of Hue Jittering Paszke al. (2019) is highly time-consuming as it will transform the samples from RGB to HSV space back and forth.
The effect of Hue Jittering in video understanding has not been systematically studied in prior works. To examine its efficacy, we conduct experiments on object recognition dataset CIFAR 10 Krizhevsky al. (2009) with ResNet-56 He et al. (206)) and video recognition dataset
VI Goyal et al. (017)) with TSM Lin et al. (201)). Fig. [ shows that Hue Jittering results in
"Corresponding Author: markcheung9248&gmallel

---

Published as a conference paper at ICLR 2024
Polar Bear
94.0
93.8
93.6
Ursus arctos
Hue
Hittring
93.4
Catch /
Hlur
Ratering
45.7
93.2
45.5
93.0
Res56 Res56*
TSM
TSM*
Object Recognition
Video Recognition
Figure 1: Illustration of the effect caused by Hue Jittering (denoted by *). Hue Jittering will lead to unrealistic and confusing appearances which are detrimental to object recognition. However, it can improve the performance in video recognition where static appearances are less important.
an accuracy drop CIFARIO, as hue variance will lead to unrealistic or confusing appearances of objects. For example, given an image of 'Polar Bear', it might be misclassified to 'Ursus arctos' (brown bear) with huc variation. In contrast, the performance of video recognition method improved, as its goal is to categorize the action in the given video, where static appearances are less important even misleading. From Fig. [ one video remains unaffected despite the hands and football turning into green color with Hue Jittering.
that the action 'Catch' in the generated
Despite its effectiveness in video understanding, the current implementation of Hue Jittering Paszke et al. (2019) still suffers from inefficiency because of the transformation between RGB and HSV space. To address the issue, of video samples in the RGB space. Specifically, we generate vidcos with new appearances by randomly permuting the RGB order, while ensuring that other color attributes such as saturation and lightness will remain unchanged. However, swapping the channel order can only lead to discrete variations and the generated video may follow fixed pattern. Thus, we further mix the original video and the generated video with linear interpolation to enlarge the input space. In this manner, can efficiently alter the hue value of video samples and simulate the effect of appearance variation.
propose an efficient operation SwapMix modify the appearance
Nevertheless, it is worth noting that videos generated by SwapMix may exhibit unrealistic appear- ances, c.g., green-corn colooee hands and football, which are unlikely to occur in the real world. Although SwapMix effectively enlarges the training set, the augmented samples will result in distribution shift compared to the original training set which may limit its performance. To address this issue, we pro- pose Variation Alignment (VA) construct training pairs with different appearances and encourage their predictions to align with each other. By providing the network with the prior knowledge of "what is the relation between inputs with different variations', appearance invariant representation, which is shown to be beneficial for video understanding.
can enforce the model learn
Considering SwapMix and Variation Alignment (VA) in uniform manner, build Motion Coher- ent Augmentation (MCA), which efficiently generates video with new appearances and resolves the distribution shift caused by appearance variation. In this way, we can implicitly guide the model to prioritize the motion pattern in videos, instead of solely relying on static appearance for predictions. Notably, most training pipelines do not consider the effect of hue variance, which means that MCA compatible with existing data augmentation approaches and can further improve the generalization ability of competing methods like Uniformer Li al. (0022)). Moreover, we conduct experiments to validate that VA, as plug-in module, can resolve the distribution shift of other data augmentation methods for even better performance. We summarize the contributions as follows:
Despite its negative impact on object recognition, reveal that Hue Jittering is beneficial in video understanding as appearance variance does not affect the action conveyed by video.
We propose a data augmentation method Motion Coherent Augmentation (MCA) to learn the appearance invariant representation for video understanding. Concretely, we present SwapMix to efficiently simulate the effect of hue variance and introduce Variation Align- ment (VA) to resolve the distribution shift caused by SwapMix.
Comprehensive empirical evaluation across different architectures and benchmarks sub- stantiates that MCA, which can be seamlessly integrated into established video under- standing approaches with minimal code modifications, yields consistent improvement and demonstrates excellent compatibility with existing data augmentation techniques.
Analysis of Variation Alignment (VA) validates that it can utilized to resolve the distri- bution shift of other data augmentation methods for even better performance.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Video Recognition has benefited a lot from the development in object recognition, especially the 2D>based methods which share the same backbone with the models in object recognition. The search focus of these 2D methods lies in temporal modeling Wang et al. (0016); Lin et al. (2019);
Li et al. 2220)) Another line of research focuses using 3D-CNNs to capture the spatial and tem- poral relation jointly, such as 13D Carreira & Zisserman (2017), C3D Tran et al. (2015) and Slow- Fast Feichtenhofer et al.] (2019). Though effective, these methods usually cost great computations. Based on the structure of Vision Transformers Dosovitskiy et al. (2020), many Transformer based networks Fan et al. (202T: y; Liu et al. (2022a); ; Li al. (2022a) have been introduced for spatial-
temporal learning in video recognition and exhibited impressive performance.
Data Augmentation has proven to be effective in mitigating the effect of overfitting. Traditional augmentation methods, including random flipping, resizing, and cropping, are frequently used to enforce invariance in deep networks He et al.] (2016); Huang et al. (2017). Some studies have investigated to enlarge the input space by randomly occluding certain regions in images De Vries & Taylor (2077). blending two images Zhang et al. (2017), or replacing an image patch with the one in another image Yun al. (2019). In recent years, automatic augmentation strategies Cubuk jet al. (2018) 2020) are shown to be effective as they consider extensive augmentation types, such color jittering, shear, translation, etc. Apart from those amage-based approaches, recent works] Li ); Zou et al. (2023); /ooda et al. (2022); Kimata et al. (2022); Yun et al. (2020a); Wang al. (202 have made attempts to address the background bias issue in video recognition, allowing the model to concentrate more on the motion patterns in videos. While we do not focus on this particular problem, our method can partially address this issue as it will also cause hue variance in the background area and help the model to rely less on the foreground bias information as well.
iet al. (2023);
Knowledge Distillation is proposed to train student network to mimic the behavior of a larger teacher model Hinton et al. (2015). To avoid the extra costs of teacher network in previous meth- ods Park et al. (2019); AAhn et al. (2019); Tian et al. (2019), researchers have developed self- distillation approaches that allow models to transfer their own knowledge into themselve et al. (2018); Xu & Liu (2019); Yun et al. (2020b); Zhang et al. (2019). Among them, CS-KD Yun et al. (020b) and data distortion Xu & Liu (2019) relevant to work as both of them construct training pairs and encourage similar predictions. However, CS-KD uses different training samples within the class to construct the training pair, and data distortion applies the same augmentation to both training samples. In contrast, our method mainly focuses on the appearance variation in videos and utilizes the same sample with different appearances to learn the invariant representations.
MOTION COHERENT AUGMENTATION
We first introduce the preliminaries of Affinity and Diversity Cubuk et al. (2021). 1). two metries that measure the distribution shift and uniqueness of the augmentation operations. Then, we present our method Motion Coherent Augmentation (MCA) which is shown in Fig. Specifically, we introduce the operation of SwapMix which can efficiently result in hue variance and alter the appearance of the given videos. Further, we propose Variation Alignment (VA) resolve the distribution shift caused by SwapMix. By encouraging the model to generate consistent predictions for videos with varying appearances, we can enforce the model learn appearance invariant representations and prioritize the extraction of motion-related information.
3.1 PRELIMINARIES
Prior work Cubuk et al. (021) has proposed two quantitative measures to analyze existing data augmentation methods. The first one is Affinity which describes how the augmentation operation shifts data with respect to the decision boundary, while the other one is Diversity which quantifies the uniqueness of the augmented training set.
Formally, given a video x E X containing T frames x = f2., fri with its one-hot class label Y, we denote Y = [0, 1]kk and K is the number of classes. Assume a data augmentation operation a (-), the augmented video sample ã can be obtained:
f (3:)
3

---

Published as a conference paper at ICLR 2024
Mis
X
CE Loss
O
Alignment
AV Loss
Training Stream
CE Backward
Inference Stream
AV Backward
Figure 2: Illustraiion of Motion Coherent Augmentation (MCA). During training. given input video I, we will modify its appearance by SwapMix to get æ. We feed the input pair into the deep network M (+) to obtain corresponding predictions p and p. p will be used to calculate CE loss and ý will be encouraged to align with p to enforce the model to learn appearance invariant representation and resolve the distribution shift caused by SwapMix. During inference, only x will be used.
Let Xtrain and X vat be the training and validation sets drawn IID from the clean data distribution X, we denote Xeal which is derived from vat by applying a (.) to each video in X
Xuat = {(a (z) 3) V (ax y) E Xeat).
(2)
Suppose M (-) to be model trained on Xtrain and A (M, xwait represents the accuracy of the model when evaluated on dataset X Xaal, the Affinity of augmentation operation a (-) can be defined as:
M;; x -A (M, /A X.at).
T (a; M; maa]] implies that there is distribution shift and smaller number indicates the larger distribution shift caused by the augmentation operation. Similarly, we define Xtrain by applying augmentation a (.) to each video in X Xtrain and we further denote the train loss on Xtrain over the model M (-) as Livain so that Diversity can be written as:
D [a; M; Xtrain] Ltrain/ Ltrain:
where higher Diversity suggests that there is a large variance in the training samples. Typically, an ideal data augmentation operation should exhibit high Diversity which brings more variance to the training samples, and high Affinity to avoid huge distribution shift.
3.2 SWAPMIX
Considering a pixel in an RGB format image, we represent the values of RGB channels as r, g, b. and max, min stand for the maximum and minimum values of the three. At first, Hue Jittering will transform the pixel to HSV space based on:
0%
max = min,
60% x g-h
max min + +
max & g > b,
0, max min max
max = O,
H
60% x 9- + 360%,
max = & g < b, S =
V = max,
max b- min
max + O,
60 x max
min + 120%,
max = g,
60% x r-g min + 240",
max = b,
(5)
where H represents hue, S stands for saturation, and V denotes value. Then, Hue Jittering will adjust hue the HSV space and map the altered value to RGB space which is inefficient practice.
We can observe from Eq. 5 that S, V are only determined by the maximum and minimum values of the RGB channels, and the value H can be easily changed by shuffling the orders of the RGB channels without affecting S and V. Considering an RGB frame ft E R (o,,0g (e0l..o)) )f)xxxX) with channels in total, the video sequence can be represented as x E (T* (C,,c))))xxxWW We can simulate the effect of hue variance by the operation acs (-), which randomly permutes the RGB channels. The generated video x' can be written as:
s' = acs (4), Ji' (((C)) (C where o {(BBG) (BRG), (BGR), (GRB), (GBR)} and the sampling procedure is random. In this manner, we can efficiently create video x' with different appearance and we denote the operation acs (+) as Channel Swap.
6
4

---

Published as a conference paper at ICLR 2024
Nevertheless, Channel Swap can only lead to dis- crete variance and this behavior may not be op- timal because the generation follows fixed pat- tern. As shown in Fig. 3 we interpolate between a and x to generate the augmented sample with continuous range of appearance variation and the coefficient À is drawn from the Beta distribu- tion Beta (a, a:) control the strength. In our implementation, we set a to 1 so À will be sam- pled from the uniform distribution (0. 1). We de- fine the operation of SwapMix which generates training sample I and the label as:
fres
R-
Bwo
A-Buta(o.a)
Chunnel
Swep
By
Ru..
Figure 3: Illustration of SwapMixx Given video
(2........ its channel order will shuffled to create the video with new appearance x'. Then, interpolation between and will be implemented to generate E with an enlarged input space. The coefficient À is sampled from the Beta distribution to control the degree of interpolation.
I Ar + (1- (1-1)x', y y
Due to the introduced X, the degree of appearance variance can be measured with a linear relation, leading to continuous variance in the input space. We empirically compare the running time of SwapMix and Hue Jittering on different platforms in Fig. 4 and the results suggest that our method is way more efficient in practice.
CPU
Original
jittering SwapMix
SwapMiix
GPU
Jittering
3
6 9
12
15
18
21
38
40
42
44
46
Running Time (ms)
Top-1 (%)
Figure 4: Rumning time comparisons Hue Jit- tering and SwapMix on CPU (Intel(R) Cor)(TM) i7-6850K) and GPU (NVIDIA GeForce GTX TI- TAN X). Results are averaged over 500 runs.
Figure 5: Evaluation TSM Lin et al. (2019) on different Something-Somethhe tion sets, where the data is original videos, and videos generated by SwapMix, Hue Jittering.
VI valida-
3.3
VARIATION ALIGNMENT
Though SwapMix can change the appearance of video samples which enlarges the input space and mitigates overfitting, the appearance of the generated video 2 may differ significantly from the orig- inal data which results in distribution shift and low Affinity. Following the procedure in Sec.B.1] we empirically measure the accuracy of TSM Lin et al. (2019) when evaluated validation sets that are generated by different augmentation operations Fig. One can observe that SwapMix leads to less performance decline compared to Hue Jittering and the reasons be summarized (1) Channel Swap does not change the distribution inside the channels as we only shuffle the channel or- ders, but Hue Jittering will alter the values of pixels because of the non-linear transformation across RGB and HSV spaces. (2) The interpolation results continuous input space which preserves more information about the original data.
Nonetheless, both augmentation operations suffer from tremendous performance drops which im- plies that the distribution shift caused by appearance variance non-negligible. To address this issue, we propose Variation Alignment (VA) which constructs pairs of videos with different appear- ances, and explicitly enforces the network learn the appearance invariant representations. Shown in Fig. 2] given video sample x, we can obtain its predictions by:
p M (ær; 0)
where M (.) is the deep network to extract features and the parameters. Meanwhile, we will transform to ē by SwapMix and get its corresponding prediction by passing it through M (+) well. By calculating Cross-Entropy loss on p, we obtain:
LoF Σ Uv log (px).
where ys represents the one-hot label of class k. Traditional data augmentation methods calculate the loss of the whole training by Eq. which inevitably introduces distribution shift and leads to
5

---

Published as a conference paper at ICLR 2024
non-optimal performance. Instead of optimizing P towards specified one-hot label, enforce the predictions of p and p to align with each other by minimizing the appearance variation loss:
K Σ
Pk
LAV = =
Pk log
(10)
PE
In this manner, we encourage the network to learn the appearance invariant representations by pro- viding with the prior knowledge of "what is the relation between inputs with different variations', rather than 'what is the right prediction for each input'.
Combining the two losses together, we update the parameters 0 in M (-) by:
[ = LCE + AAv LAv,
(11)
where AAv is introduced to balance the two terms. In this way, can resolve the distribution shift caused by SwapMix and implicitly encourage the model to focus more on the motion information, rather than solely relying on static appearance.
4 EMPIRICAL VALIDATION
We empirically evaluate the performance of Motion Coherent Augmentation (MCA) various chitectures and benchmarks in this segment. We first validate the effectiveness of MCA on different methods, datasets, and frames. Second, we present the comparisons of method with other com- peting data augmentation methods and demonstrate that MCA is compatible with them. Further, we analyze the effect of Variation Alignment (VA), the training process, and robustness against probability change. We provide comprehensive ablation and qualitative analysis in the end.
4.1 EXPERIMENTAL. SETUP
Datasets. We validate our method on five video benchmarks: (1) Something-Sometthng Something VI & V2 et al. (2017) made up of 98k videos and 194k videos samples, which exhibit significant temporal dependencies and are employed for the majority of evaluations.
(2) UCF101 [Soomro et al. (2022) comprises 13,320 videos across 101 categories and the first training/testing split is adopted for training and evaluation. (3) HMDB51 Kuehne et al. [201T) consists of 6,766 videos categorized into 51 classes. Similarly, employ the first training/testing split for both training and testing. (4)
Kinetics400 Kay et al. (017) is a large-scale dataset which is categorized into 400 action classes.
Implementation details. We sample 8 frames uniformly for all methods except for SlowFast Feich/ enhofer et al. (2019) which samples 32 frames for fast pathway. During training, we crop the training data randomly to 224 224, and we abstain from applying random flipping to the Something- Something datasets. In the inference phase, frames will be center-cropped to 224 x 224 except Slow Fast which is cropped to 256 x 256. We adopt one-crop one-clip per video during evaluation for efficiency unless specified. More implementation details can be found in the appendix.
4.2 MAIN RESULTS
Evaluation across different datasets. In this part, we validate Motion Coherent Augmentation (MCA) across different datasets in Tab. [] As for UCF101 and HMDB51, the improvements brought by MCA are, quite obviously, greater than 2% as the scales of these datasets are relatively small, leading to a more pronounced occurrence of overfitting. video recognition dataset with strong temporal dependency and MCA can effectively increase the generalization ability of the baseline method by implicitly encouraging the model to prioritize the motion patterns during training. We further implement MCA Kinetics400 which is a large-scale dataset and MCA still improves the performance of the bascline model. Note that MCA is still shown to be effective on datasets like UCF101 which contain less motion information, indicating the effectiveness of appearance invariant representations in video understanding.
V2 is a large-scale
Evaluation across different architectures. In Tab. empirically validate the performance of MCA across different architectures on Something Soontg--Smtnnng V1 dataset which exhibits obvious temporal dependency. We first implement MCA 2D-nework TSM Lin al. (2019) and one can

---

Published as a conference paper at ICLR 2024
Table I: Evaluation of Motion Coherent Augmentation (MCA) on UCF101, HMDB51, Something- Something V2 and Kinetics400. The best results are bold-faced.
UCF101
HMDB51
Something-Something V2
Kinetics400
Method
Acc1.(%)
Acc1.(%)
Acc1.(%)
Acc1.(%)
Accl.(%)
Acc1.(%)
Acc1.(%)
A Acc1.(%5)
TSM
79.57
48.63
59.29
70.28
+2.30
+3.00
+1.42
+0.80
TSM-MCA
81.87
$1.63
60.71
71.08
Table 2: Evaluation of MCA on Something- Something V1 dataset with different architectures, in- cluding 2D, 3D and Transformer network. The best results are bold-faced.
Table Comparisons with competing data augmentation methods on Something- Something V1 dataset. The best results are bold-faced.
Method
Acc1.(%)
Acc5.(%)
A Acc1.(%))
Method
Accl.(%)
A Acc1.(%)
TSM
45.63
TSM
45.63
75.00
+1.94
TSM+Cutout
44.68
-0,95
TSM+MCA
47.57
76.27
TSM+CalMis
44,99
-0.64
TSM+ VidcoMix
45.62
-0.01
SlowFast
44.12
72.58
+1.76
MSxup
46.03
40.40
SlowFast+MCA
45.88
74.06
TSM+AugMix
46.03
+0,40
TSM+BE
40.82
Uniformer
48.48
76.69
+2.03
75M-RandAugmann
47.21
+1558
Uniforer+MCA
50.51
78.20
TSM+MCA
47.57
+1.94
observe that MCA significantly improves its performance by 1.94%. Further, we extend MCA to 3D-network: SlowFast [Fiichtenhofer et al. (2019) and Transformer-network. Uniformer-S et al. (0022a). The results show that MCA consistently increases the accuracy of these methods which validates that the idea of learning appearance invariant representation is effective and has great generalization ability across different architectures.
Evaluation across different frames on competing method. We further validate the performance of MCA across various frames the state-of-the-art method Uniformer et al. (2022a). From Fig. 6 one can notice that MCA leads to consistent improvement at all frames with the mean value of 1.95% and the standard deviation is only 0.25 which demonstrates the stability of MCA. Moreover, as current state-of tate--of-the-ar works normally adopt multi-crop evaluation to pursue better performance, evaluate our method with the same procedure in Uniformer Li al, (2022a) and present them in dotted lines. Similarly, one can observe similar enhancements across all these frames which implies that MCA can also result improvement even on state-of-the-art works.
4.3 COMPARISON WITH OTHER APPROACHES
Quantitative comparison. We compare our method with competing data augmentation methods on Something-Somethin V1 based on TSM in Tab. 3 It is shown that popular data augmentation methods like Cutout DeVries & Taylor (017). CutMix Yun et al. (2019) result in negative effects in video understanding. One possible explanation is that these methods break the motion pattern in video sequences and bring challenges to temporal modeling which is extremely essential in video understanding. Mixup ZZang al. (2017) shows improvement it enforces regularization effect which is helpful to deal with overfitting. VideoMix Yun et a (2020a) leads to similar performance with baseline method and Background Erasing (BE) Wang et al. (2021) is helpful due the stress of motion information. AugMix Hendrycks et al. (2019) and Rand Augment Cubuk et al. (2020) exhibit great performance they involve multiple augmentation operations and their magniudes will be adaptively sampled. MCA outperforms all these approaches even though we only consider the effect of hue variance in videos, which proves the efficacy of our method.
Compatibility with other approaches. Current usually incorporated with multiple strong data augmentation methods. For example, the official implementation of Uniformer-S
video understanding methods are
et al. (2022a) intrinsically involves CutMix Yun et al. (2019), Mixup Zhang et al. (2017) and Rugmentt Cubuk et al. (2020) and the improvement brought by MCA in Tab. 2 is made upon these methods. This indicates that our method is compatible with existing works as MCA only considers hue variance in videos which is overlooked in previous approaches. To better examine the effect of MCA, remove other augmentations in Uniformer and build our method on top of it. From Fig. 7 we can see that MCA can lead to even more improvement in accuracy when remove other augmentations, which demonstrates the strength of our method.

---

Published as a conference paper at ICLR 2024
60
Untormor
56-
Uniformer MCA
50
Uniormer MCA
Top- L.ccc (%)
-2-
Topa Acc (h)
48
2.94
48-
Moan Stl
44
Aeclasio) Cno-Cnp
Accl MalisChop
133
195
0.25
0.40
40
16
42
Frame Number
with Aug
w/o Aug
Figure 6: Evaluation of MCA at different frames over competing method Uniformer-S on Something- Something V1. Solid, dotted lines denote one-crop and multi-crop evaluation, respectively.
Figure 7: Compatibility of MCA with
other competing data augmentation meth- ods (include RandAugment, Mixup, Cut- Mix) Something-Something VI.
Table 4: Extending Variation Alignment (VA) to competing data augmentation methods on Something- Something V1. The best results are bold-faced.
0.20
val Acc (9)
0.15
40
0.10
Method
VA
Train Accl.(%)
Val Acc1.(%)
Vall AccITSM
AccIMCA)
0.05
TSM
78.46
45.63
TSg+AugMix
X
73.20
46.03
20
0
20
30
50
0.00
ISM-RandAugment
63,94
47.21
Epoch
Figure 8:
The curves of Validation Ac- curacy and AV Loss during training on Something-Sometiiee V1 dataset.
TSM+AugMix
81,53
46.55(0.521)
TSM-RamlAAugmen
81.55
48.53(1.321)
4.4 ABLATION AND ANALYSIS
Analysis of Variation Alignment. The motivation of VA is to alleviate the distribution shift re- sulting from SwapMix and encourage the model to learn representations invariant to appearance. Similarly, this idea can easily be extended to address the distribution shift of other augmentation methods. Shown in Tab. when we extend VA to AugMix Hendrycks AugMMxx|endryckks et 2019)) and RandAug- ment Cubuk et al. 2020). we can further improve their performance by 0.52% and 1.32%. The results prove the generalization ability of our design and imply that VA can easily be applied to cur- rent augmentation methods to mitigate the distribution shift and result in even better performance. Besides, we notice a significant increase in the training accuracy when the methods are incorporated with VA, which suggests that VA enables the model to learn better representations.
Analysis of Training Process. We plot the curves of validation accuracy and AV loss in Fig. based on the experiments of TSM Lin et al. (2019). One can notice that the accuracy will increase sharply at Epoch 20 and 40 because of the learning rate decay schedule. After Epoch 20, our method outperforms TSM the AV loss starts to decrease after that, indicating that the model starts to learn the appearance invariant representations which is beneficial for the performance.
Robustness against probability change. Compared to the baseline method, our method will con- struet a training pair and enforce appearance invariant learning at each iteration. We introduce a hyperparameter P and the model will apply the operation of MCA at this iteration if the value, ran- domly drawn from a uniform distribution, is less than the predetermined probability p. To study the robustness of MCA against probability change, we empirically conduct experiments over different and the results are in Tab. 3 The first observation is that MCA clearly outperforms the baseline method with smaller ECE Guo et al. (2017), i.e., the difference between predicted probabilities and their true accuracy, under different p. which implies that models trained with MCA are more well-calibrated. Besides, there only a small accuracy drop when we decrease the probability to 0.25 which means that our method is still effective even when we change the appearance of lim- ited number of data. Typically, applying data augmentation operations during training will result in longer training hours different extents, and so does our method. This phenomenon can help to alleviate this problem, as we can choose a small probability of applying MCA we want to attain a better trade-off between training efficiency and validation accuracy.
Ablation of design choices. In this part, we provide ablations of design choices to verify our method in Tab. 6 We first implement Hue Jittering which changes the hue of videos with the official
8

---

Published as a conference paper at ICLR 2024
Table 5:
Table 6: Ablation of design choices on Something- change on Something-Somethine VI. The Something V1. The best results are bold-faced. best results are fold-faced.
Robustoness against probability
Method
Specification
Acc1.(%)
Method
(CE.(%)
Accl.(%)
TSM+HU
Hue Jittering
46.32
TSM
23.25
45.63
TSM+CS
Channel Swap
45.98
TSM+MCA
0.25
22.04
47.26
TSM+SwapMix
Remove VA from MCA
46.53
TSM+MCA
0.50
21.85
47.38
TSM+SwapMix
Expand Training Set
45.69
TSM+MCA
0.75
21.09
47.60
TSM+MCA
Add CR Loss on P
46.89
TSM+MCA
1.00
21.11
47.57
TSM+MCA
47.57
(a) Dropping something onto something.
(b) Pushing something off of something.
Figure 9: Visualizations produced by CAM Zhou al. (2016) on Soattiinnnn VI with TSM Lin et al. (2119) (second row) and MSM+MCA (third row). TSM misclassifies the first sample as PPutting something onto something" and the second sample 'Hitting something with something'.
implementation provided by Pytorch Paszke et al. (019). It exhibits worse performance compared to SwapMix with more inefficient operation. Further, implement Channel Swap and SwapMix, and results show that both methods can lead to improvement over the baseline. However, their performance is restrained by the distribution shift caused by appearance variation. The performance of SwapMix is better than Channel Swap as the interpolation offers continuous distribution of the training data which enlarges the input space. Further, we explicitly expand the training set by creating training pairs like MCA and calculating CE loss on all the video samples to update the parameters. The result even worse than SwapMix which indicates that the improvement brought by MCA does not come from the larger training set. Finally, we add CE loss prediction ğ and the inferior performance suggests that computing CE loss on augmented training samples will introduce the distribution shift again and it indeed limits the performance of SwapMix.
Qualitative analysis. To demonstrate that MCA can encourage the model to concentrate more on the motion patterns, we utilize CAM ChM et al. (2016) to visualize the attention maps produced by TSM Lin et al. (2019) and TSM+MCA on the first sample as *Putting something onto something because it entirely concentrates box at all the frames. While MCA helps the model to focus on the movement in the first three frames and make the right prediction. In the second case, the baseline model detects the action of the hand, but it places more emphasis on the wrist and overlooks the pen movement. In contrast, MCA directs the model concentrate more precisely on the action of push.
V1 in Fig. 9 TSM misclassifies
the yellow
CONCLUSION AND LIMITATIONS
In this work, we reveal that hue variance, which thought be detrimental object recognition as it changes the appearance of images, is beneficial in video understanding. Based on the obser- vation, we propose Motion Coherent Augmentation (MCA) to encourage the model to prioritize motion information, rather than static appearances. Specifically, we propose SwapMix to efficiently transform the appearance of video samples and leverage Variation Alignment (VA) to resolve the distribution shift caused by SwapMix, enforcing the model to learn appearance invariant representa- tions. Comprehensive evaluation validates the efffcttivneess and generalization ability of MCA, and the application of VA in other data augmentation methods for even better performance.
One limitation of MCA is its increased demand for GPU memory throughout the training phase as will import another batch of data with different appearances. Second, MCA will cause longer training time because we need to construct training pairs to calculate AV loss. In subsequent re- search, we aim to enhance the training efficiency.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
In our paper, strictly follow the ICLR ethical research standards and laws. To the best of our knowledge, our work abides by the General Ethical Principles.
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint
2018.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPRW, 2020.
Ekin Dogus Cubuk, Ethan S Dyer, Rapha Gontijo Lopes, and Sylvia Smullin. Tradeoffs in data augmentation: An empirical study. 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Terrance De Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1w..iv:708..444 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In /CCV, 2019.
Shreyank N Gowda, Marcus Rohrbach, Frank Keller, and Laura Sevilla-Lara. Lear2augmentt learning to composite videos for data augmentation in action recognition. ECCV, 2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freetag et al. The" something something" video database for learning and evaluating visual common In ICCV, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In ICMI, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 19/2.0078 2019.
10

---

Published as a conference paper at ICLR 2024
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint orXiv:1503.025:: 2(7), 2015.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
Jun Kimata, Tomoya Nitta, and Toru Tamaki. Objectmix: Data augmentation by copy-pasting ob- jects videos for action recognition. In MM Asia, 2022.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In ICCV, 2011.
Haoxin Li, Yuan Liu, Hanwang Zhang, and Boyang Li. Mitigating and evaluating static bias of action representations in the background and the foreground. ICCV, 2023.
Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint wrXiv:2201.094550 2022a.
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. In CVPR, 2020.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022b.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022a.
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022b.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurIPS, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 12 2012.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: /910. 10699, 2019.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, 2015.
Jinpeng Wang, Yuting Gao, Ke Li. Yiqi Lin, Andy J Ma, Hao Cheng, Pai Peng, Feiyue Huang, Rongrong Ji, and Xing Sun. Removing the background by adding the background: Towards background robust self-supervised video representation learning. In CVPR, 2021.
11

---

Published as a conference paper at ICLR 2024
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
Ting-Bing Xu and Cheng-Lin Liu, Data-distortion guided self-distillation for deep neural networks. In AAAI, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019.
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, and Jinhyung Kim. Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457, 2020a.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In CVPR, 2020b.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1710.0942, 2017.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. NeurlPS, 2018.
Yuliang Zou, Jinwoo Choi, Qitong Wang, and Jia- Bin-Bin Huang. Learning representational invariances for data-efficient action recognition. Computer Vision and Image Understanding, 227: 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 IMPLEMENTATION DETAILS
All models are trained on NVIDIA Tesla V100 GPUs with the same training hyperparameters as the official implementations. We set the hyperparameter XAv = to be 0.65, 0.4 on Something- Something VI, V2 Goyal et al. (2017), Kinetics400 Kay al. (2017) datasets, respectively. The default probability of applying Motion Coherent Augmentation (MCA) at each iteration is = 1 in order to strive for better performance.
A.2 VALIDATION IN SELF-SUPERVISED LEARNING
Table 7: Evaluation of MCA in self-supervised learning UCF101 dataset. The best results are bold-faced.
UCF101
Method
Acc1.(%)
A A%)1.(%)
MoCo He et 2020
63.68
+4.31
MoCo+MCA
67.99
To demonstrate the effectiveness of MCA in self-supervised learning, implement MoCo He
et al. (2020) on UCF101 SSommr et al.) 2012) and further combine it with MCA. Specifically, we conduct pre-training and fine-tuning both UCF101 following the setting of Background Erasing (BE) Wang et al. (2021) and utilize the 3D network I3D Carreira & Zisserman (2011) as the base en- coder. Further, we reduce the resolution during pre-training 112 x 112 and increase the batch size for efficiency. One can observe from Tab. that MCA leads to significant improvement over MoCo on the UCF101 dataset which proves the efficacy and generalization ability of MCA in different video understanding tasks.
A.3 TRANSFERABILITY
Table 8: Evaluation of MCA with linear probing on HMDB51 dataset. The models are pre-trained on Kinetics400 with different data augmentation approaches. The best results are bold-faced.
Method
Acc1.(%)
TSM Lin et al. 2019)
63.59
Mug
64.18
ISMMRand/Augment
65.10
TSM+MCA
65.95
In order to prove the representation learned by MCA can be generalized to other domains, we first pre-train TSM Lin al. (2019) Kinetics400 Kay al. (2017) with different data augmentation methods: AugMix Hendrycks et al. (2019) and RandAugment Cubuk et al. (2020), and then conduct linear probing HMDB51 Kuehne al. (2011) dataset to compare the downstream performance. As shown in Tab. S one can notice that MCA results in the highest accuracy compared to other methods, which proves that the representation learned by MCA is general and can be transferred to other domains.
A.4 HYPERPARAMETER STUDY
We further conduct ablation with different λA over TSM et al. (2019) on Something-Somettine VI Goyal et al. (0117) dataset in this section. One can observe from Tab. 9 that MCA leads to consistent improvements with different choices of λAv and MCA is robust to this hyperparameter variation. However, if dav is too large small, we observe that the performance will start to decrease which meets our expectation as MCA will degenerate to the baseline if X = 0 and LAv will dominate the training if Aay is too large.
13

---

Published as a conference paper at ICLR 2024
Table Evaluation of MCA with different A. The best results are bold-faced.
Method
Acc1.(%) A
TSM Lin et al. (0199)
45.63
TSM+MCA
0.25
46.79
TSM+MCA
0.50
47.40
TSM+MCA
1.00
47.57
TSM+MCA
2.00
47.03
TSM+MCA
3.00
46.57
A.5 HUE VARIANCE VERSUS BACKGROUND SCENE VARIANCE
In this part, we discuss the relation between our method MCA, and methods that try to address the background bias issue. First, 'video appearance" is a general concept that contains many attributes, such color and background scene information. The motivation of MCA is to cause hue variance in training samples which leads to appearance changes videos so that MCA can force the model to learn appearance invariant representations. For methods that target the background issue, they aim to cause background scene variation in videos which will also result in appearance changes and encourage the model learn representations invariant appearance. Although starting from different perspectives, the goal of these two research directions is similar: learn appearance invariant representations and prioritize the motion information. Moreover, we highlight that these two lines of research can be combined with each other. Shown in Tab. I0 we combine our method with prior work Background Frasing (BE) Wang et al. (202T) that tries address the background bias issue, and leads to further improvement when combined with it.
Table 10: Compatibility of MCA with BE. The best results are bold-faced.
Method
Acc1.(%)
TSM Lin et al. (0019)
45.63
TSM+BE
46.45
TSM+MCA
47.57
TSM+BE+MCA
48.05
A.6 RESULTS OF DIFFERENT DEPTHS
In previous experiments, most of the results TSM Lin et al. (2019) are built upon ResNet-50 He et al. (2016) and we further conduct experiments on ResNet-18 and ResNet-101 to verify the effec- liveness of MCA when evaluated models with different representation abilities. One observe from Fig. TO that MCA leads to consistent improvement at different depths, which demonstrates that our method is effective regardless of the representation ability of the model. Further, we notice that TSM+MCA on ResNet-50 outperforms TSM on ResNet- 101 with much fewer parameters and computational costs, suggesting the strength MCA.
TSM
TSM_MC
Top-1 Acc (%))
46
43
Mean Std
40
(ccc..5))
1.94
0.10
Res18
Res50 Depth
Res101
Figure 10: Experiments with varied levels of depth on Something Something-Somethingg and standard deviation of Top-1 Accuracy improvement are shown in the table.
Something VI dataset. The mean
14

---

Published as a conference paper at ICLR 2024
A.7
RESULTS ON ACTIVITYNET DATASET
Table 11: Evaluation of MCA on ActivityNet dataset. The best results are bold-faced.
ActivityNet
Method
mAP(%)
mAP(%)
TSM et al. (0019)
73.10
+1.65
TSM+MCA
74.75
The ActivityNet-v1. dataset Caba Heilbron et al. (0155) is an extensive collection of untrimmed videos, featuring 200 action categories and average length of 117 seconds per video. It includes 10,024 video samples designated for training purposes and another 4,926 videos set aside for vali- dation. From Tab video dataset which validates the effectiveness of our method.
be observed that MCA still leads to performance increase the long
A.8 ANALYSIS OF TRAINING PROCESS
TSMICE Loss)
TSM_MCA
TSM, MCCIAV_LLoS)
MCCACE Loss)
Train Acc (%6
3.75
TSM, MwCAAAAAAaaaLsSS
0.175
AV Loss
50
0.150
1.75
0.125
30
10
20
40
50
Epoch
Epoch
Figure 11: The curves of Training Accuracy dur- ing training on Something-Somettie V1.
Figure 12: The curves of Training Losses during
training on Something-Sometie V1.
We have analyzed the training process Sec. 4.3 and we further plot more training curves for anal- ysis. We compare the training accuracy in Fig. [T] where is noticeable that the two curves closely align with each other. While traditional data augmentation methods usually result in decreases in training accuracy to alleviate overfitting, our method introduces Variation Alignment (VA) which resolves the distribution shift caused by augmentation operations and can obtain similar training accuracy compared to the baseline method TSM Lin et al. (2019).
We further plot the curves of training losses of two models. The total loss of TSM+MCA is larger than the training loss of TSM because of the AV Loss introduced by VA. and we find that the curves of CE Loss of TSM+MCA and TSM are well-aligned with each other. With the decrease of AV Loss, the model is enforced to learn appearance invariant representation which is beneficial for video recognition.
A.9 QUALITATIVE ANALYSIS
Here we show more visualization results on Something-Something V1 validation set in Fig. 13 We find that MCA effectively lead the model to concentrate the motion patterns in videos and result in more accurate predictions.
A. TRAINING TIME ANALYSIS
As we mentioned, training efficiency the limitation of MCA and we will analyze the training time in this section. Specifically. we measure the training time of MCA across different probabilities NVIDIA Tesla V100 GPUs and compare them with Hue Jittering (HJ) and Background Erasing (BE) Wang al. 2021) which shares similar vision with stress the motion patterns in videos.
15

---

Published as a conference paper at ICLR 2024
(a) Moving something and something so they collide with each other.
(b) Throwing something against something.
(c) Plugging something into something.
Figure 13: Visualizations produced by CAM Zhou al. (0166) -Soeetiin VI with Lin et al.) (2019) (second row) and TSM+MCA (third row). TSM misclassifies the first sample 'Altaching something something' and the second, the third sample 'Something falling like feather or paper'.
One can see from Tab. 12 that MCA will inevitably increase the training time of the baseline method other data augmentation methods. However, our method requires fewer training hours compared to and BE when p is smaller than 0.75 with a clear advantage accuracy.
Meanwhile, it can be observed that the training time of MCA varies lot with different values of p, while there is only a small change in accuracy. This finding can somehow mitigate this issue, as choose small probability of applying MCA if we want to attain better trade-off between training efficiency and validation accuracy.
Table 12: Training time MCA across different probabilities on 4 NVIDIA Tesla V100 GPUs.
Method P
Training Hours
Acc1.(%)
TSM
10
45.63
TSM+HJ
15
46.32
TSM+BE
18
46.45
TSM+MCA
0.25
12
47.26
TSM+MCA
0.50
13
47.38
TSM+MCA
0.75
16
47.60
TSM+MCA
1.00
47.57
16


Reviews:
[{"review_id": "zMzaf23kwM", "reviewer": "Reviewer_sFSx", "summary": "This paper presents an interesting trick to improve action recognition performance in benchmark datasets. Specifically, hue jittering or hue variance has been shown to help improve performance most likely due to adding robustness to the appearance of the content and focusing more on the motion to perform better action recognition. The proposed MCA approach focuses on better data augmentation for video recognition tasks. Performance on multiple benchmarks shows evidence for this idea to be effective.", "questions": "1. How does this approach compare to the one that uses only optical flow or other motion representation and not appearance?\n2. Does the MCA approach work better with other more modern and robust core networks than TCM?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Proposes a simple yet effective trick to improve the quantified results by a small amount on multiple benchmark datasets. \n\nComparison performed on multiple action recognition benchmark datasets with ablation study as well. \n\nThe writing, figures, and explanation are reasonable and effective.", "weaknesses": "The novelty of the approach is limited, as-in a new network to extract unique embedding features is not what is being proposed here. This seems more like add-on that could be used with other solutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3VY1EItKjT", "reviewer": "Reviewer_SoaL", "summary": "The current study investigates the effect of hue variance on video recognition and proposes a data augmentation method called Motion Coherent Augmentation (MCA) that introduces appearance variation to encourage the model to prioritize motion patterns over static appearances. This approach is based on the observation that static appearances are less important in videos with motion information. The proposed method uses an operation called SwapMix to modify video samples' appearances efficiently and Variation Alignment (VA) to resolve distribution shifts caused by SwapMix, enforcing the model to learn appearance-invariant representations. Comprehensive experiments on different architectures and datasets demonstrate the effectiveness and generalization ability of MCA, achieving an average performance gain of 1.95% on the Something-Something V1 dataset compared to the competing method Uniformer.", "questions": "Whether the MCA data augmentation method provides gains in video domain adaptation or video domain generalization settings, where videos from different domains are involved, is not explicitly discussed in the given context. To assess whether MCA is beneficial in such video migration scenarios, an evaluation of its performance in tasks that involve domain adaptation or generalization would be necessary. Comprehensive experiments comparing MCA to other data augmentation techniques and assessing its effect on domain adaptation or generalization metrics would provide insights into its potential usefulness in video migration settings.\n\n\nThe potential of integrating MCA with existing video self-supervised methods to further enhance the performance is not explicitly discussed in the given text. However, it is worth exploring the compatibility of MCA with other video self-supervised methods and investigating whether their combination can lead to improved results. By combining the strengths of MCA in introducing appearance variation and prioritizing motion patterns with other self-supervised techniques, it is possible to enhance the overall effectiveness of video representation learning. Further research and experimentation are needed to explore the potential synergies and benefits of integrating MCA with existing video self-supervised methods.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed Motion Coherent Augmentation (MCA) method addresses the issue of overfitting in video recognition by introducing an effective data augmentation strategy. The key advantages of MCA are:\n\n- Hue Jittering: MCA leverages Hue Jittering, which is usually overlooked in object recognition, to generate new appearances for videos. This operation helps the model prioritize the motion pattern over static appearance.\n\n- SwapMix: An efficient operation called SwapMix is introduced to modify the appearance of video samples in the RGB space. This operation helps simulate the effect of appearance variation while preserving important color attributes like saturation and lightness.\n\n- Variation Alignment (VA): VA is used to construct training pairs with different appearances and encourage their predictions to align with each other. This helps the model learn appearance-invariant representations.\n\n- Compatibility: MCA can be seamlessly integrated into existing video recognition approaches with minimal code modifications and provides consistent improvement. This indicates that the method is compatible with existing data augmentation techniques.\n\nExperimental validation: Comprehensive experiments on various architectures and benchmarks demonstrate the effectiveness and generalization ability of MCA. The method demonstrates excellent performance and can further enhance the results of competing approaches like Uniformer.\n\nIn summary, the Motion Coherent Augmentation method addresses the issue of overfitting in video recognition by providing an effective data augmentation strategy that leverages Hue Jittering, SwapMix, and Variation Alignment. MCA achieves consistent improvement while being compatible with existing techniques, making it a promising approach for video recognition.", "weaknesses": "It is a valid point that the paper lacks a discussion on the generalization capabilities of MCA and how it could potentially improve the transferability of models. Introducing such a discussion could indeed help increase the impact of the work. A possible approach could be to analyze the performance of MCA on tasks that are different from the benchmark datasets used during training, and compare it to other data augmentation methods. Additionally, analyzing the learned representations using techniques like clustering or visualization could provide insights into their generalization potential. Overall, a thorough evaluation of the generalization capabilities of MCA could strengthen the paper and make it more impactful.\n\nThe relationship between MCA and existing video self-supervised methods is not further explored in the given text. It would be valuable to investigate how MCA can be integrated with other video self-supervised methods and examine the potential synergies between them. Understanding how MCA complements or enhances existing techniques could provide valuable insights into the effectiveness and generalization abilities of the combined approach. Further research is needed to explore the relationship between MCA and video self-supervised methods and determine how they can be effectively integrated to improve video recognition tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RPWZZqf2Es", "reviewer": "Reviewer_4nis", "summary": "The paper introduces a new augmentation technique for training video recognition networks, referred to as Motion Coherent Augmentation (MCA). The method is a simple but effective modification of existing Hue Jittering method, adapted specifically to video content (e.g. to learn appearance invariant representation, capturing motion patterns). The method is well explained and the experiments confirm the efficacy of the proposed solution.", "questions": "Could authors provide the analysis of \\lambda_{AV} impact on training performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Simple but effective augmentation technique\nCan be easily applied to improve any of video recognition methods\nEvaluated on multiple datasets\n\nOverall the paper is well written and clearly presents the contributions. The proposed augmentation technique (MCA) consists of two parts: SwapMix ( a method, which permutes RGB channels and computes the linear interpolation between original and the permuted image), and Variation Alignment (a method that forces the network to become invariant to SwapMix, ie. the network is explicitly trained to return the same class distributions for both, original image and SwapMix-ed image. The multiple experiments on multiple video datasets confirm the efficacy of the proposed technique. The ablation analysis of different components of the method has also been presented.", "weaknesses": "Slows down the training and requires more GPU resources\nThe analysis of impact of \\lambda_{AV} has not been evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oXJGSrxLE9", "reviewer": "Reviewer_7MY2", "summary": "This paper proposes a method of augmentation for video action recognition. The authors focus on “Hue Jittering”, which is one of common ways to data augmentation for images since it is implemented in common frameworks such as pytorch.\n\nHowever it is computationally inefficient to compute hue and perform  jittering, then the authors propose SwapMix, which is randomly swap RGB channels of a given video frames (Channel Swap), followed by a linear blending with the original frames.\n\nIn addition, the authors propose Variation Alignment (VA) to enhance the invariant to appearance change by SwapMix. It is simply align the  model outputs for the original and SwapMix-ed frames by minimizing cross-entropy of two outputs.\n\nMotion Coherent Augmentation (MCA), the name of the proposed method, is the combination is SwapMix and VA, which is used in experiments.\n\nExperimental results demonstrated that \n- the proposed MCA improved performances of different models (TSM, SlowFast, and Uniformer) for different datasets (SSv1/v2, UCF101, HMDB51),\n- and outperformed other augmentation methods such as Cutout and CutMix.", "questions": "Questions are same as above; comparisons and motivations.\n\nFigure 5 shows TSM with SwapMix, and the performance drops to about 42%. However in Table 5, TSM+SwapMix shows performance over 46%. Why this happens ?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "It is interesting that changing hue of video frames is a pretty simple and channels swap is very fast to compute even on CPU, nevertheless the proposed MCA provides performance improvements on common action recognition datasets.\nIt might become a practical standard for endusers as it is easy to use and implement.", "weaknesses": "The main weakness of the paper is the lack of solid motivation.\n\nInsufficient survey\n\nThe paper says “there has been a lack of studies for data augmentation methods in video recognition” in section 2, however, there a lot of works such as follows.\n\n* Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground, ICCV2023\n* Learning representational invariances for data-efficient action recognition. 2023 CVIU\n* Frequency Selective Augmentation for Video Representation Learning, AAAI 2023\n* Enabling Detailed Action Recognition Evaluation Through Video Dataset Augmentation, NeurIPS 2022\n* Learn2Augment: Learning to Composite Videos for Data Augmentation in Action Recognition, ECCV 2022\n* ObjectMix: Data Augmentation by Copy-Pasting Objects in Videos for Action Recognition, MMAsia22\n* Exploring Temporally Dynamic Data Augmentation for Video Recognition, arXiv 2022\n* VideoMix: Rethinking Data Augmentation for Video Classification, arXiv 2020\n\nTable 3 shows comparisons with other augmentation methods, however, which are for images. It should be better to compare other video augmentation methods for comparisons as the proposed method is for videos.\n\nIn addition, these prior works have tried to tackle the background bias or scene bias (similar to “static appearance” in page 2), which exists in action recognition datasets. It should be better to discuss how the proposed method can contribute to solve the issue.\n\n\n\nWeak motivation for hue jittering: The paper says “Among them, Color Jittering is widely used to transform color attributes like saturation, brightness, and contrast” in the introduction. Indeed it is true, but there are a tens of methods for augmentations implemented in famous libraries such as imageaug, albumentations, and kornia. However, there is no discussions or evidences why hue jittering is so important to explore. \n\nWeak motivation for backbone model: Experiments used TSM, a 3D CNN with shift modules, however it is merely one of a vast amount of models for action recognition. Other models (slowfast and uniformer) are used but not for all datasets. It should be justified why TSM is the best for demonstrating the effect of the proposed method. Otherwise, recent methods (such as ones  referred in the paper) should also compared.\n\n\nMinor comments:\n\nChannel swap: Swapping channels for augmentation is not new and have been implemented in libraries.\nhttps://kornia.readthedocs.io/en/latest/augmentation.module.html#kornia.augmentation.RandomChannelShuffle\nhttps://albumentations.ai/docs/api_reference/full_reference/#albumentations.augmentations.transforms.ChannelShuffle\n\n\nDefinitions: Affinity, Diversity, and high Expected Calibration Error (ECE) are mentioned in the paper, and only Affinity is defined. However, ECE only is shown in experiments (affinity may be from Fig5 indirectly).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)