Temporal Coherent Test Time Optimization for Robust Video Classification
Authors: Chenyu Yi
SIYUAN YANG
Yufei Wang
Haoliang Li
Yap-peng Tan
Alex Kot
Venue: ICLR 2023 poster (2023)
Keywords: Video Classification
Robustness
Test Time Optimization

Abstract: Deep neural networks are likely to fail when the test data is corrupted in real-world deployment (e.g., blur, weather, etc.). Test-time optimization is an effective way that adapts models to generalize to corrupted data during testing, which has been shown in the image domain. However, the techniques for improving video classification corruption robustness remain few. In this work, we propose a Temporal Coherent Test-time Optimization framework (TeCo) to utilize spatio-temporal information in test-time optimization for robust video classification. To exploit information in video with self-supervised learning, TeCo minimizes the entropy of the prediction based on the global content from video clips. Meanwhile, it also feeds local content to regularize the temporal coherence at the feature level. TeCo retains the generalization ability of various video classification models and achieves significant improvements in corruption robustness across Mini Kinetics-C and Mini SSV2-C. Furthermore, TeCo sets a new baseline in video classification corruption robustness via test-time optimization. 

Published as a conference paper at ICLR 2023
TEMPORAL COHERENT TEST-TIME OPTIMIZATION FOR ROBUST VIDEO CLASSIFICATION
Chenyu Yi!
Siyuan Yang!. Yufei Wang' Haoliang Li Yap-Peng Tan'
Alex C. Kot
Schooo of Electrical and Electronic Engineering, Nanyang Technological University, Singapore #Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore Department of Electrical Engineering. City University of Hong Kong, China
} @ e...tu.edu.sg haoliang. li@ cityu.edu.hk
{eyptan,eackot) @ @enuueduu.gg
ABSTRACT
Deep neural networks are likely to fail when the test data corrupted in real- world deployment (e.g., blur, weather, etc.). Test-time optimization is an effec- tive way that adapts models to generalize to corrupted data during testing, which has been shown in the image domain. However, the techniques for improving video classification corruption robustness remain few. In this work, we propose a Temporal Coherent Test-time Optimization framework (TeCo) to utilize spatio- temporal information in test-time optimization for robust video classification. To exploit information in video with self-supervised learning, TeCo minimizes the entropy of the prediction based on the global content from video clips. Mean- while, it also feeds local content regularize the temporal coherence the feature level. TeCo retains the generalization ability of various video classification mod- els and achieves significant improvements in corruption robustness across Mini Kinetics-C and Mini SSV2-C. Furthermore, TeCo sets a new baseline in video classification corruption robustness via test-time optimization.
INTRODUCTION
Deep neural networks have achieved tremendous success in many computer vision tasks when the training and test data are identically and independently distributed (i.i.d). However, the mismatch between them is common when the model deployed in the real world (Wang et al. (2022b); Li eet al. (2220)) For example, weather changes like rain and fog, and data pre-processing like saturate adjustment and compression can corrupt the test data. Many works show that the common corrup- tions arising in nature can degrade the performance of models test time significantly (Hendrycks & Dietterich 2019 Yi et al. 2021 Kar et al. 2022] Geirhos et al. 2018 Yu et al., 2022). In video clas- sification, Yi al. 2021 demonstrates the vulnerability of models against corruptions like noise, blur, and weather variations.
Various techniques have been proposed to improve the robustness of models against common cor- ruptions (a.k.a.corruptio robustness). The most popular direction is increasing the diversity of input training data via data augmentations and applying regularization at training time (Zheng al.,
2016 Wang et al. 2021b Hendrycks et al. 2020: Rusak et al. 2020 Wang et al. 2020). It trains one model and evaluates on all types corruption. However, the training data is often unavailable because of privacy concerns and policy constraints, making it more challenging to deploy models in the real world. In this work, we focus on the direction of test-time optimization. Under such a scheme, the parameters of one model will be optimized by type of corrupted test data specifi- cally. Test-time optimization updates the model to fit into the deployment environment at test time, without access to training data. There are several test-time optimization techniques emerging in image-based tasks (Wang et al., 2021a Schneider et al., 2020 Liang et al. 2020 Sun et al. 2020). However, find these techniques are not able to generalize to video-based corruption robustness tasks well from empirical analysis. We hypothesize the gap between image and video-based tasks comes from several aspects. Firstly, the corruptions in the video can change temporal information,
"Corresponding Author

---

Published as a conference paper at ICLR 2023
which requires the model to be both generalizable and robust Yi et al. 2021 Hence, improv- ing model robustness against the corruptions in video like bit error, and frame rate conversion is more challenging. Secondly, video input data has different format from image data. For example, video data has a much larger size than image data. It is impractical to use a similar batch size as image-based tasks (e.g., batch size of 256 in Tent (Wang et al. (02Ta)), though the batch size is an important Lastly, these techniques ignore the huge information hidden in the temporal dimension.
in test-time optimization (Wang et al. 2021a Schneider et al. 2020).
To improve the video classification model robustness against corruptions consistently, we propose a temporal coherent test-time optimization framework TeCo. TeCo is a test-time optimization tech- nique with two self-supervised objectives. We propose to build our method upon the test-time op- timization which updates all the parameters shallow layers and only normalization layers in the deep layers. It can benefit from both training and test data, and such an optimization strategy re- mains part of model parameters and statistics obtained at training time and updates the unfrozen parameters with test information. Besides, utilize global and local spatio-temporal information for self-supervision. We use uniform sampling to ensure global information in input video data and optimize the model parameters via entropy minimization. By dense sampling, extract other local stream that has a smaller time gap between consecutive frames. Due to the smooth and continuous nature of adjacent frames in the video (Li & DiCarlo 2008: Wood & Wood (016), we apply a temporal coherence regularization as a self-supervisor in test-time optimization on the lo- cal pathway. As such, our proposed technique enables the model learn more invariant features against corruption. As result, TeCo achieves promising robustness improvement on two large- scale video corruption robustness datasets, Mini Kinetics-C and Mini SSV2-C. Its performance superior across various video classification backbones. TeCo increases average accuracy by 6.5% across backbones on Mini Kinetics-C C and by 4.1% on Mini SSV2-C, which is better than the base- line methods Tent (1.9% and 0.9%) and SHOT (1.9% and 2.0%). Additionally, We show that TeCo can guarantee the smoothness between consequent frames at the feature level, which indicates the effectiveness of temporal coherence regularization.
We summarize our contributions as follows:
To the best of our knowledge, we make the first attempt to study the test-time optimiza- tion techniques for video classification corruption robustness across datasets and model architectures.
We propose a novel test-time optimization framework TeCo for video classification, which utilizes spatio-temporal information in training and test data to improve corruption robust- ness.
For video corruption robustness, TeCo outperforms other baseline test-time optimization techniques significantly and consistently Mini Kinetics-C and Mini SSV2-C datasets.
RELATED WORKS
2.1 PROBLEM SETTING
Corruption Robustness. Deep neural networks are vulnerable common corruptions generated in real-world deployment (Geirhos et al. 2017 Hendrycks & Dietterich, 2019). Hendrycks & Diet- terich 2019) firstly propose ImageNet-C
benchmark the common corruption robustness of deep learning models. It assumes the tested model has no prior knowledge of the corruption arising during test time. The model is trained with clean data while tested on corrupted data. Under such setting, we are able to estimate the overall robustness of models against corruption. In the following, bench- mark studies and techniques across various computer vision tasks are booming (Kar et al.] 2022] Michaelis et al. 2019] Kamann & Rother 2020). These studies corruption robustness bridge the gap between research in well-setup lab environments and deployment in the field. Recently, studies have emerged corruption robustness in video classification (Yi et al., 202T Wu & Kwiatkowskal 2020). In this work, we tap the potential of ppatio-temporal information in video data and improve the corruption robustness of video classification models during testing.
Video Classification. Recently, with the introduction of a number of large-scale video datasets such as Kinetics (Carreira & Zisserman 2017), Something-Somethingg V2 (SSV2) (Goyal et al.)
2

---

Published as a conference paper at ICLR 2023
2017]. and SportsIM (Karpathy et al. 2014), video classification has attracted increasing attention.
Most existing video classification works mainly focus on two perspectives: improving accuracy (Si-
monyan & Zisserman 2014 Wang et al. 2016;; Carreira & Zisserman 2017;Hare et al. 2017 Xie
et al. 2018 Yang et al. 2020 Bertasius et al. [222]] Li et al. 2022) and model efficiency (eeicheen-
hofer al. 2019 Lin et al. 2019 Feichtenhoter (2220] Kondratyuk al. 2021] Wang al. 2021c).
However, as mentioned above, techniques for improving video classification corruption robustness
remain few. In this paper, we focus on the optimization of video corruption robustness and conduct experiments on Mini Kinetics-C and Mini SSV2-C datasets. The Kinetics dataset relies on spatial semantic information for video classification, while SSV2 contains more temporal information. enables us to evaluate the effectiveness of our proposed optimization method on these two different types video data.
2.2 TEST-TIME OPTIMIZATION
Optimizing the model with unlabeled test data has been used in many machine learning tasks (Shu
al. 2022; Huang et al.] 2020; Shocher al. 2018 Azimi al.] [022] Niu et al. 2022] Wang
et al. 2022a). These methods only require the pretrained model and test data for optimization in the
inference. Teme-Time Training (TTT) (Sun et al. 2020) and TTT++ (Liu et al. 2021) update models
at test-time, but they jointly optimize the supervised loss and the sell-supervised auxiliary loss in training. Though these methods demonstrate promising improvement in image- based corruption robustness, there is little progress made in video- based test-time optimization. We make the first attempt to utilize the temporal information for model robustness under the test-time optimization setting.
2.3 TEMPORAL COHERENCE IN VIDEO
The concept of temporal coherence makes an assumption that adjacent frames in video data have se- mantically similar information (Goroshin et al. 2015). The assumption is based on the phenomenon that the visual world is smoothly varying and continuous. The stability of the visual world in the temporal dimension is significant for many studies in biological vision (Li & DiCarlo 2008 Wood 2016 Wood & Wood 2016). With the assumption, the invariance between neighboring frames in the video can provide self -uperviiinn for many computer vision tasks (Jayaraman & Grauman 2015
Wiskott & Sejnowski 2002 Dwibedi et al. 2019 Wang et al.) 2019). For example, Jayaraman & Grauman (2015) exploits motion signal in the egocentric video to regularize image recognition task; Wang et al. (2019) utilizes the cycle consistency in time to learn visual temporal correspondence.
In the corruption robustness problem, we use temporal coherence as self-supervisor to regularize models to be more invariant against corruption.
METHOD
We propose an end-to-end training framework to improve the robustness of video classification models against corruption at test time. In such a framework, we optimize pre-trained deep learning models during the test time and minimize two objectives without label information. These two ob- jectives correspond to two pathways. The first pathway uses a global stream as input and minimizes the entropy of prediction; the second pathway leverages a local stream as input and regularizes the temporal coherence among consecutive video frames. We integrate these two pathways and named our framework as TeCo. Figure outlines our temporal coherent test-time optimization framework. The backbone in Figure i initialized by pre-trained model. Any deep video classification model pretrained under a supervised learning scheme can fit into it.
3.1 TEST-TIME OPTIMIZATION STRATEGY
Though BN (Schneider et al. 2020) and Tent (Wang et al.) (021a) achieve promising performance on image-based lest-time optimization tasks, they only show minor improvements on video data and sometimes degrade the robustness corruptions in our empirical analysis. BN only adapts the mean and variance of normalization layers by test data partially, limits the searching space of optimization. Tent removes the training time batch normalization statistics fully, while video classification tasks usually have smaller batch size. It can only capture sub-optimal statistics when just relying test-time data.
3

---

Published as a conference paper at ICLR 2023
Inpul Video
Local Pathway
Stallow Layers
Alternion Module IK))
TxCBWW
Temporal Coherence
Regularization
Shared
Global Pathway
Stallow Layers
Deep Layers g())
Sulonas
Entropy
Minimization
FC
1u
Update all parans
Update layers
Figure 1: The proposed framework TeCo at the test-time optimization stage. TeCo consists of two pathways: the local pathway module h to generate an attention-basee feature map z. Then applies temporal coherence regularization on the feature map. We update all the parameters in the shallow layers in this pathway; the glohal pathway uses a sparse video stream L.. which captures the long-term information. TeCo minimizes the prediction entropy in the global pathway by updating all the parameters in the shallow layers and only normalization parameters in the deep layers.
dense video stream Ia input. It passes through the shallow layers and attention
We perform simple but effective mechanism by grafting the advantage of these two classic tech- niques. Following (Schneider et al. 2020), we initialize mean μ ap,, + (1 a))l and variance x² = ao² + - a)o where stands for training, and t stands for testing. a is the that controls the weights of training and test time statistics. At the same time, we update the affine transformation parameters in the normalization layers, which helps the model utilize both training and test-time statistics after adaptation.
Apart from normalization layers, we also fine-tune other network parameters (e.g.. Conv Layers) in the shallow layers, which are the initial (lower) layers in the network. The shallow layers of robust models generate smoother feature tensors that are better aligned with human perception (Xie al. 2019 Tsipras et al. 2018). They play the role of denoiser when encountering natural corruptions arising in input data. Inspired by this phenomenon, and only normalization layers in the deep layers, as shown in Figure It gives more flexibility to the shallow layers while not interrupting the higher-level semantic information. The effect of different divisions of the shallow and deep layers is explored in the ablation study.
update all the parameters in the shallow layers
3.2 GLOBAL AND LOCAL PATHWAYS
In the test-time optimization of TeCo, we use two pathways (i.e, global pathway and local path- way) and two objectives to optimize model. The global pathway uses uniform sampling to extract frames from the input video f,,. The uniform sampling splits the video into multiple clips with equal length. Then we randomly take one frame from each clip. As result, the selected frames by uni- form sampling can capture the global information effectively (Chen et al.) 202I). In this pathway, optimize the overall parameters of shallow layers and the transformation parameters in deep normalization layers by minimizing the entropy of prediction. The objective is:
Lent = ([((ooftmar(((((T.)))
where g(-) represents the deep layers, f(-) stands for the shallow layers, H is the Shannon entropy
of prediction ý with N classes: H (ğ) = - p(Om) log p().). The Shannon entropy serves as the optimization of the global pathway.
The local pathway leverages dense sampling to take consecutive frames La, random location of the input video, which randomly chooses a location as the center of the sampled sequence. Com- pared to the large time gap in frames by uniform sampling, the time gap in the frames by dense sam- pling can be as low as one. In this pathway, the temporal coherence between frames is much stronger. Because the visual world is smoothly varying, the neighhoring frames in video data will not change
4

---

Published as a conference paper at ICLR 2023
abruptly in the temporal dimension. When common corruptions interrupt the natural structure of video data, we apply a temporal-coherent regularization to the output feature of the shallow layer. The regularization can be achieved by enforcing the feature similarity in consecutive feature maps of corrupted data. We explicitly represent with the loss function: Lecoeerncce = E. || - -'-1111- We use I distance measure the similarity of feature tensors since it penalizes more outliers and benefits the robustness (Alizadeh et al. 2020 [Bektay & Şişman 2010). x' is the tensor at time which is the output after f(()). i the time gap between two neighboring tensors. Leoherence encourages the layers to generate smoother features. Correspondingly, the feature extractor will be more invariant to the perturbation, corruption, and disruption in the input data. It leads more robust video classification models.
3.3 ATTENTIVE TEMPORAL COHERENCE REGULARIZATION
When we apply Leoherence on feature tenors, we simply assume the frames are varying in uniform speed because of the same weights along the time dimensions. However, the changes in neighboring frames have different scales in the real world. The example of bungee jumping in Figure I shows that the change small at the beginning the video because the background fixed. has only the blue sky and jumping platform. When the person is reaching the ground, the background changes rapidly, and there are houses and other people appearing. Hence, we propose to assign different weights to different frames. To encourage temporal consistency in the static background, the slower changes in temporal dimension correspond to higher weights in regularization. The weights can be generated by attention module h(-). In this work, we use the time dimension-only 1D non-local operation (Wang et al., 2018) to obtain the weights along the time dimension. Hence, the attentive temporal coherent regularization be written as:
Latt-co
Σ
(2)
where h(.) is the attention module.
3.4 OVERALL ORJECTIVE FUNCTION
To summarize, the global pathway uses Lu as input and entropy minimization objective, and the local pathway leverages Ia as input and attentive temporal coherence regularization as objective. The overall objective function for TeCo can be represented
L (Ja, f.9) + BLott-co (Ia, (la.f.l. f. h).
(3)
where 3 >> is balancing hyper-parameter.
EXPERIMENTS
We evaluate the corruption robustness of TeCo Mini Kinetics-C and Mini SSV2-C. The mean performance on corruption is the main metric for robustness measurement. We also compare it with other bascline methods across architectures.
V2 (SSV2) Goyal al. 2017) are two most popular large-scale datasets in video classification community. Kineties extracted from the Youtube website and it relies spatial information for classification. As com- plementary, SSV2 is first-view video dataset constructed systematically in the lab environment, which has more temporal changes. We use their variants Mini Kinetic-C and Mini SSV2-C (Yi et al., (02T) to evaluate the robustness of models against corruptions. The mini-version datasets randomly sample half of the classes from the original large-scale datasets. Hence, Mini Kinetics-C contains around 10K videos in the test dataset; Mini SSV2-C has a test dataset with a size around 13K videos. These two datasets apply 12 types of corruptions arising in nature the original clean test datasets. Each corruption has 5 levels of severities. The samples of Mini Kinetics-C C and Mini SSV2-s are shown in supplementary.
Dataset. Kinetics (Carreira & Zisserman 2017) and
Metrics. The classification accuracy on corrupted data is the most common metric to measure the corruption robustness of models. In our experiments, the corrupted datasets have 12 types of
5

---

Published as a conference paper at ICLR 2023
comuptiinn and each has 5 levels of severities. We use mPC (stands for mean performance corruptions) to evaluate the overall robustness. We compute mPC by averaging the classification accuracy as: mPC = Ees E CAc,s, where CA is the classification accuracy on corruption c at level s.
The test-time optimization techniques can usually be applied across architectures. We use several classic 2D, 3D CNN-based architectures and transformers as the backbone show the superiority of our method TeCo. For 2D CNN, we choose vanilla ResNet18 8 and its variant TAM- ResNet18 (Fan al., 2019) as the model architecture. The vanilla ResNet18 treats single image frames in the video as the input data. It has only an average fusion at the output stage for the over- all video classification. TAM-ResNet18 integrates temporal module the ResNet architecture. It helps the model store more temporal information for classification. Hence, it obtains improve- ments clean accuracy, especially on the SSV2 dataset. For 3D CNN, we use the 3D version 8 (Hara et al. 2017) for training and evaluation. The 3D convolution layers enable it to capture sufficient spatio-tempors temporal features in an end-to-end training manner. Since TeCo is model agnostic in principle, we use the state-of of the-art art transformer MViTv2-S (Li 2022) the back- bone as well. Different from CNN-based architecture, the transformer uses Layer Normalization (Ba
Models.
faate-of-the-ane
al. 2016).
Baseline. Though the video classification area lacks research in test-time optimization, we im- port the baseline methods in image- based test-time optimization. BN (test-time batch normaliza- tion) (Schneider et al.] 2020) is a simple method that adapts the batch normalization statistics with test-time data. Tent (Wang al., 202 extends from BN by updating the transformation param- eters in the normalization layers, with an objective of entropy minimization. SHOT (Liang et al., 2020) is well-known for combining entropy minimization and pseudo-labeling. TTT (Sun et al., 2020) optimizes the model with a self-supervised auxiliary task at both the training and test stages. Since we only conduct optimization at test stage and have no access to training data in our setting, we denote TTT as TTT*, We choose rotation prediction (Hendrycks et al. 2019) as the self-supervised auxiliary task in TTT*, All the methods can improve the corruption robustness of image-based deep learning models. We re-implement them on the video classification frameworks based on the public available codes.
Trainig and Evaluation Setup. We separate the setup of the experiment into three stages: pre- training. test-time optimization, and test-time evaluation. In the pretraining stage, we use uniform sampling create 16-frame-length input data. Then we use multi-scale cropping the input for data augmentation and resize the cropped frames to a size of 224. We train the models with an initial learning rate of 0.01 and a cosine annealing learning rate schedule. In test-time optimization, we use the pre-trained model for network weight initialization and update the model parameters for one epoch. For standard (without test-time optimization). BN, Tent, and SHOT methods, we use uniform sampling to extract input from corrupted test data. We apply both uniform and dense sam- pling to create input for TeCo. In all the test-time optimization experiments, we follow the offline adaptation setting in Wang et al. (2021a) Ta). Because the baseline methods have not been implemented in video classification previously, we follow their implementations in image classification and tune the hyper-parameter settings to obtain the best result:] For the optimization, we use SGD with momentum. On Mini Kinetics-C, we use a batch size of 32 and a learning rate of 0.001; while for Mini SSV2-C, we the same batch size but learning rate of 0.00001. After adapting the models with test data, we freeze the model weights and evaluate on the corrupted data. The inference is only based on the global pathway.
4.1 EXPERIMENTAL RESULTS
mPC across architectures and datasets. Benefiting from the temporal coherence, our method TeCo consistently outperforms image-based test-time optimization methods across architectures and datasets. Table I shows the mPC on Mini Kinetics-C of various methods. Compared with the stan- dard inference, BN improves the mPC by up to 2% because it only adapts BN statistics; Tent, SHOT and TTT* increase the robustness by a larger margin, due to their updates on more parameters. However, the improvements are not consistent across architectures. For Tee-R,,, the Tent degrades the performance. We hypothesize that Tent fully removes the training time BN statistics,
'The detailed implementation details can he found in the appendix.

---

Published as a conference paper at ICLR 2023
MINI KINETICS-C
MINT SSV2-C
70.0
Clai Azurary - 702
45.0
Clain Arruracy 68.7
58.7
6022
10.0
38.4
80.0
35.0
25.0
30.0
Standard
BN
Tent
SHOT
TIT*
TeCo
Standard
BN
Taat
SHOT
TIT*
TeCo
METHODS
Mean Robustness Mini Kineties-C and Mini SSV2-C of methods. TeCo reduces the gap hetween model accuracy on clean and corrupted data significantly.
Figure 2:
Table 1:
mPC across architectures on Mini Kinetics-C and Mini SSV2-C. TeCo outperforms other baseline methods different architectures and datasets. Clean Ace is the accuracy of model tested clean data.
Backbone
Clean Ace
Standard
BN
Tent
SHOTT
TTT
TeCo
3D ResNet18
61.7
49.4
50.6
53.9
52.6
54.6
56.9
ResNet18
66.0
51.6
53.0
55.6
53.2
59.5
60.8
Mini Kinetics-C
TAM-ResNNtll
68.5
55.9
57.1
53.8
58.4
62.2
63.4
MViTv2-S
84.4
77.9
78.0
79.2
78.2
78.0
80.1
3D ResNet18
52.2
39.3
40.0
39.9
42.4
41.5
45.7
ResNet18
30.2
20.0
20.3
22.5
23.8
22.2
24.5
Mini SSV2-C
TAM-R ResNet18
55.5
44.2
45.0
44.5
45.2
46.7
49.4
MViTv2-S
56.8
48.1
48.1
48.4
48.2
48.1
48.5
but the test-time data is not sufficient to make the parameters reach the global optimum. Though there is smaller gap between clean and corruption accuracy MVVTvv-S, TeCo is still able to im- prove the mPC by 2.2%. As a result, our proposed TeCo achieves the best performance, enhancing the mPC by 2.2~.9.2% 9.2% on various architectures. In Figure tures, including 3D ResNet18, ResNet18, TAM-ResNet 8 and MVīTv2-S. There is small gap of 5% between the accuracy testing on clean data and the mPC results achieved by TeCo.
show the mPC various architec-
On Mini SSV2-C, our method TeCo surpasses other baseline methods significantly. T demon- strates the overall mPC of methods on Mini SSV2-C. We find the ResNet18- (--kett8-bbeed method has an obvious degradation in the robustness. Because this method has no module in capturing tempo- ral information except output fusion, their performance on Mini SSV2 relatively poor when the Mini SSV2 replies more temporal information for classification. Similar to the phenomenon on Mini-Kinetics- C, BN increases the mPC by up to 1%; Tent, SHOT and TTT show diverging enhancement across architectures, to 3.8%. TeCo consistently improves the corruption robust- ness on various architectures from 0.4~6.4%. Horizontally comparing the performance on Mini Kinetics-C and Mini SSV2-C, there is larger gap between clean accuracy and mPC Mini SSV2- from absolute and relative aspects, as shown in Figure 2 We hypothesize that the corrupted temporal information is hard be compensated, but the spatial semantics can be extracted from the neighboring frames with the nature of temporal coherence.
Accuracy w.r.t corruption and severity. When digging deeper into the robustness of methods, we find that TeCo achieves superior performance on corruptions at different levels of severities. Figure shows the complete results on various corruptions from Mini Kinetics-C, with backbone of 3D ResNet18. The horizontal axis indicates the severity of corruption. When the value is O, corresponds to the accuracy on clean test data. We find the Standard and BN methods have much worse performance compared to Tent, SHOT, TTT* and TeCo, when data is corrupted by shot noise, fog, contrast, saturate change, and rain. We hypothesize that simply adapting the batch normalization statistics is not able to correct the shifts raised by these corruptions. Tent, SHOT, TTT* and TeCo demonstrate a similar trend when they encounter corruptions at various severity levels. However, TeCo always lies the top of the trend lines, especially at severity level of 5. For instance, TeCo maintains the accuracy of 46.7% at level-5 contrast changes, while the accuracy of standard and BN methods drops to 12%. It shows that TeCo benefits from the temporal coherence in video data, apart from an end-to and-t--end test-time optimization scheme. Appendix, we also show TeCo consistently outperforms other methods various corruptions Mini SSV2-C.

---

Published as a conference paper at ICLR 2023
Shot Noise
Motion Blur
Contrast
Standard
22
TeCojO.m)
Saverity Level
Severity Lovel
Sovertyy Lovel
Severity Lovel
Brightness
Saturate
Roin
CRF Compression
26
Severity
Severit
Severity Level
Severity
ABR Compress on
Bit Error
Pac set Loss
Frame Rate Conversion
Severity
Severity
Severity
Figure 3: Full results Mini Kinetics-C, with backbone of 3D ResNet18. The accuracy of standard and BN methods drops sharply when the models encounter severe corruptions like noise, fog, rain, and contrast change. TeCo remains robust against corruptions at even level-5 severity.
Table Ablating components of TeCo on Mini Kinetics-C and Mini SSV2-C. The test-time optimization strategy, local pathway (1-path), and attentive temporal coherence regularization improve robustness.
model, 3D R18
Standard
SHOT
TeCo (w/o I-path)
TeCo (uniform)
TeCo
Mini Kinetics-C
49.4
52.6
55.9
56.5
56.9
Mini SSV2-C
39.3
42.4
44.4
44.5
45.7
4.2 ABLATION STUDIES
Disentangle Modules in TeCo. Table 2 shows the utility of three key components in TeCo: test- time optimization strategy, global and local pathways integration, and attentive temporal coherence regularization with local content. The test-time optimization strategy balances training and test- time parameters and statistics, which is more effective than other baseline methods improving video classification corruption robustness. Global and local pathways provide complementary infor- mation for optimization. If we only use uniform sampling for entropy minimization and temporal coherence regularization, the improvement is as low as 0.1% Mini SSV2-C. When apply the regularization on local content, the mPC increases by 1% and 1.3% respectively.
Study Hypeerparammetrr Sensitivity. Table 3a demonstrates the impact of in Equation We obtain the best result at 3=11 Table 35 shows the comparison results of the attentive temporal co- herence regularization module added after different blocks of 3D ResNet18. The performances after -e are similar, while there is a slight drop after res4. We hypothesize the deeper layer will gen- erate more distinguishable features. It benefits from variety instead of consistency for classification. Table 3c studies the impact of batch size on the performance of TeCo. The mPC keeps improving when increase the batch size from to 32, while drops when we use the batch size of 64. Because we optimize the model for one epoch for fair comparison, larger batch size will lead to fewer iterations of updates. The batch size of 32 balances the amount of data in one batch and the number of iterations in one epoch, which enables models to obtain the best performance. Hence, TeCo can improve robustness reliably without tuning the hyper- parametrs carefully.
Balance training and test statistics with Figure shows the mPC Mini Kinetics-C w.r.t optimization iteration, with different a and the backbone of 3D ResNet18. The a in Section 3.1 controls the weights of training and test statisties in normalization layer initialization. When set a to a smaller value (c.g., 0.2), the normalization layer relies more on test data initially. Hence, converges faster in test-time optimization. After 40 iterations of test-time optimization, the mPC is
8

---

Published as a conference paper at ICLR 2023
Table 3: TeCo ablation experiments with 3D ResNet18 Mini Kinetics- C. If not specified, the default setting is: the beta that control the weight temporal coherence regularization the regularization is applied after res2 stage of model, the test time optimization uses a batch size of 32 and a is 0.4. The default setting has bold text.
TeCo applies a balanced weight to temporal co- herence regularization.
(a) in Equation 3
(b) Block stage. The temporal co- herence regularization module is added after certain block stage.
(c) Batch size. TeCo batches of data test time optimization.
Batch Size
mPC
mPC
Stage
mPC
56.6
0.1
56.4
rc81
56.6
16
56.7
0.5
56.7
rC82
56.9
32
56.9
1
56.9
rC83
56.6
64
56.6
5
56.6
r684
56.2
[-i
t-2i
56
Standard
55.3
54
mPC
BN
52
alpha=0.2
alpha=0.4
Tent
50
alpha=0.6
alpha=0.8
48
40
100
200
300
TeCo
iteration
Figure 4: Iteration in test-time optimization vs mPC, on various a. The mPC improves rapidly in the begin- ning, then the improvements slow down when more data is used. Smaller which enables performance converge faster.
Figure 5: We visualize resi feature maps of consec- utive frames. The time gap between frames TcCo generates smoother feature maps in both spatial and temporal dimensions.
more test data statisties,
enhanced from 49.6% 55.3%. When the iterations increases, the performance improves slower and gets closer for different a. In summary, the model robustness is more sensitive to a when less data used in test-time optimization.
Visualize Space-Time Feature Map. Figure 5 shows the feature maps of noise corrupted video after resi stages. Horizontally, we visualize the features of different frames. The time gap between neighboring frames is i = 1. Standard and BN methods fail to recover buman-recognizzlle ties from the norse-corrupted frames. Tent obtains noisy outlines along the temporal dimension. TeCo removes the noise and generates temporal coherent feature maps, indicating that TeCo helps model obtain smoother and more consistent features.
seman-
5 CONCLUSION
TeCo is test-time optimization technique for improving the corruption robustness of video clas- sification models, which updates model parameters during testing by two objectives with self- supervised learning: TeCo uses global information for entropy minimization, and it also applies attentive temporal coherence regularization on local information. Benefiting from long-term and short-temn spatio-temporal information, TeCo achieves significant corruption robustness improve- ment on Mini Kinetics-C and Mini SSV2-C. We hope TeCo becomes a new baseline method that makes video classification models more reliable and robust in real-world deployment.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENTS
This work was done at Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University. This research is supported in part by the NTU-PKU Joint Research Institute (a collaboration be- tween the Nanyang Technological University and Peking University that is sponsored by donation from the Ng Teng Fong Charitable Foundation). This work is also supported by the Research Grant (RGC) of Hong Kong through Early Career Scheme (ECS) under the Gran121200522, CityU Applied Research Grant(ARG) 9667244,and Sichuan Sci-ence and Technology Program 2022NS- FSCO551.
REFERENCES
Milad Alizadch, Arash Behboodi, Mart van Baalen, Christos Louizos, Tijmen Blankevoort, and Max Welling. Gradient 11 regularization for quantization robustness. In International Conference on Learning Representations, 2020.
Fatemeh Azimi, Sebastian Palacio, Federico Raue, Jörn Hees, Luca Bertinetto, and Andreas Den- gel. Self-supervised test-time adaptation on video data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1439-3448, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.064550, 2016.
Sebahattin Bektaş and Yasemin Şişman. The comparison 1l and 12-norm minimization methods. International Journal of the Physical Sciences, 5(11):1721-1727, 2010.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, pp. 4, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Chun-Fu Chen, Rameswar Panda, Kandan Ramakrishnan, Rogerio Feris, John Cohn, Aude Oliva, and Quanfu Fan. Deep analysis of enn-based spatio-temporal representations for action recogni- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Min-Hung Chen, Zsolt Kira, Ghassan AIRegib, Jaekwon Yoo, Ruxin Chen, and Jian Zheng. Tempo- ral attentive alignment for large-scale video domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6321-6330, 2019.
Tianlong Chen, Zhenyu Zhang, Pengjun Wang, Santosh Balachandra, Haoyu Ma, Zehao Wang, and Zhangyang Wang. Sparsity winning twice: Better robust generaliztion from more efficient training. arXiv preprini arXiv:2202.09884 2022.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Tem- poral cycle-consistency learning. In Proceedings of the IEEE/CVF Conference Computer Vi- sion and Pattern Recognition (CVPR), June 2019.
Quanfu Fan, Chun-Fu Fu Richard Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More is less: Learning efficient video representations by big-little network and depthwise temporal aggrega- tion. Advances in Neural Information Processing Systems, 32, 2019.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 203-213, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 6202-6211, 2019.
10

---

Published as a conference paper at ICLR 2023
Robert Geirhos, David HJ Janssen, Heiko H Schitt, Jonas Rauber, Matthias Bethge, and Felix A Wichmann. Comparing deep neural networks against humans: object recognition when the signal gets weaker. arXiv preprint 1706. 170.006999.9 2017.
Robert Geirhos, Carlos R Medina Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In Proceedings of the Neural Information Processing Systems (NIPS), 2018.
Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learn- ing of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pp. 4086-4093, 2015.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference Computer Vision, pp. 5842-5850, 2017.
Yiwen Guo, Chao Zhang, Changshui Zhang. and Yurong Chen. Sparse dnns with improved adver- sarial robustness. Advances in neural information processing systems, 31, 2018.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d
residual networks for action recognition. In Proceedings of the IEEE international conference computer vision workshops, pp. 3154-3160, 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learn- ing can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations, 2020.
Yujia Huang, James Gornet, Sihui Dai, Zhiding Yu, Tan Nguyen, Doris Tsao, and Anima Anand- kumar. Neural networks with recurrent generative feedback. Advances in Neural Information Processing Systems, 33:535-545, 2020.
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In Proceedings of the IEEE International Conference Computer Vision, pp. 1413-1421, 2015,
Christoph Kamann and Carsten Rother. Benchmarking the robustness of semantic segmentation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 8828 8838, 2020.
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18963-18974, 2022.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014.
Dan Kondratyuk; Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Bo- qing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16020-16030, 2021.
Haoliang Li, YuFei Wang, Renjie Wan, Shiqi Wang. Tic-Qiang Li, and Alex Kot. Domain gener- alization for medical imaging classification with linear-dependency regularization. Advances in Neural Information Processing Systems, 33:3118-3122 2020.
Nuo Li and James DiCarlo. Unsupervised natural experience rapidly alters invariant object repre- sentation in visual cortex. science, (5895 2008.

---

Published as a conference paper at ICLR 2023
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- 2022.
nition, pp.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the 37th International 2020.
Conference on Machine Learning, pp.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7083-7093, 2019.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tu+++: When does self-supervised test-time training fail or thrive? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 2180022880. Curran As- sociates, Inc., 2021. URL /ttp///sss//proceiisssseeeee https//wwwwa.s....ppppap///////.
cc httppppprr/222//11eee
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustmess object detec- tion: Autonomous driving when winter is coming. arXiv preprint arXiv: (900999074599 2019.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. arXiv preprint arXiv:2204.0220 02610, 2022.
Evgenia Rusak, Lukas Schott, Roland Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against image corruptions by playing the game of noise. arXiv preprint arXiv:2001.06057, 2020.
Steffen Schneider, Evgenia Rusak, Luisa Bck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in Neural Information Processing Systems, 33:1 153115199115,, 2020.
Assaf Shocher, Naday Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. Proceedings of the IEEE conference on computer vision and puttern recognition, pp- 5118-3126, 2018.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. arXiv preprint 17X1v:2209.05511, 2022.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 92299922999222 9248. PMLR, 2020.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint 1805. 2018.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference Learning Repre- sentations, 2021a.
Haotao Wang. Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. Advances in neural information processing systems, 34:237-250, 2021b.
12

---

Published as a conference paper at ICLR 2023
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Limin Wang. Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1895-1904, 2021c
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7201-7211, 2022a.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2566-2576, 2019.
Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 3622-3626. IEEE, 2020.
Yufei Wang, Haoliang Li, Hao Cheng. Bihan Wen, Lap-Pui Chau, and Alex Kot. Variational dis- entanglement for domain generalization. Transactions on Machine Learning Research, 2022b. ISSN 2835-8856. URI
net/ forum Fod I TMIZ
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 14(4):715-770, 2002.
Justin N Wood, A smoothness constraint on the development of object recognition. Cognition, 153: 140-145, 2016.
Justin N Wood and Samantha MW Wood. The development of newborn object recognition in fast and slow visual worlds. Proceedings of the Royal Society B: Biological Sciences, 283()829): 20160166, 2016.
Min Wu and Marta Kwiatkowska. Robustness guarantees for deep neural networks on videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 501-509, 2019.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV), pp. 305-321, 2018.
Siyuan Yang, Jun Liu, Shijian Lu, Meng Hwa Er, and Alex C. Kot. Collaborative learning of gesture recognition and 3d hand pose estimation with multi-order feature analysis. In Proceedings of the European conference on computer vision (ECCV), pp. 769-786, 2020.
Chenyu Yi, Siyuan Yang. Haoliang Li, Yap-peng Tan, and Alex Kot. Benchmarking the robustness of spatial-temporal models against corruptions. In Advance in Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Yi Yu, Wenhan Yang, Yap-Peng Tan, and Alex C Kot. Towards robust rain removal against adversar- ial attacks: A comprehensive benchmark analysis and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6013-6022, 2022.
Stephan Zheng, Yang Song. Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4480-4488, 2016.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 SOURCE-FREE DOMAIN ADAPTATION
We benchmark the video domain adaptation for shift from UCF to HMDB and vice versa. In our setting, we have only access to test data and pre-trained model. In the domain adaptation problem, 12 common classes of 3209 video data in UCF and HMDB are used, following the official split provided Kinetics penetics-pretrainee 3D ResNet50. Then
by (Chen al.] 2019). For the pre- pre-trained models, we fine tune the model with training data in UCF and HMDB.
Table reports reports the target accuracy of standard and test-time optimization methods. We denote Tent as Tent* because we remain the training statistics in normalization layer, otherwise the performance will drop significantly. All the methods can help model adapt video data with domain shift. TTT* obtains competitive improvement of 2.7% on UCF-HMDB. Tent also increases accuracy on HMDB-UCF by 2.4%. We find that TeCo consistently outperforms other baseline methods. When we train model with UCF and optimize on HMDB at test time, it improves the accuracy by 3.3%.
Similarly, on HMDB-UCE the accuracy is enhanced by 2.8%.
Table Adapting models to target domain without access to source data. We use Kinetics for model pre- training, and fine tune the model with UCF and HMDB training data respectively. UCF-HMDB means training on UCF and testing on HMDB data.
model, 3D R50
Standard
BN
Tent*
SHOT
TTT*
TeCo
UCF-HMDB
74.2
75.6
76.4
76.7
76.9
77.5
HMDB-UCF
80.7
82.6
83.1
82.7
81.9
83.5
A.2 FULL RESULTS ON MINI SSV2-C
Shat Noise
Motion Blur
Fog
Contrest
IIN
Stensiand
40-
40-
Thrt
33
SHDT
TTT*
Severily Level
Sevvrily Level
Severit Level
Severily Level
Brightness
Saturate
Rain
CRF Compression
20-
52
15
10
13
10
Severity Level
Severity Leval
Sevarity Level
Severity Level
Compression
Error
Pocket
Frame Rate
Accaracy
20
20
20
Severtty
Severity
Severity
Severtty Level
Figure Full results Mini SSV2-C, with backbone of 3D ResNet18. Tent becomes vulnerable motion blur and bit error, while TeCo lies above other methods on most types of corruptions.
Fihws the complete results on Mini CSV2-C. Apart from the corruptions which cause severe accuracy degradation in Mini Kinetics-C, C, the corruptions like motion blur, bit error, and frame rate conversion also lead to significant drops in accuracy in Mini SSV2-C. Intuitively, these three types
14

---

Published as a conference paper at ICLR 2023
of corruption have more impact temporal information. For example, bit error propagates the perturbation in the consecutive frames, and frame rate conversion changes the speed of motions in the video clips. The performance of Tent is devastating when facing motion blur and bit emror. In contrast, TeCo, which uses a balanced test-time optimization framework, guarantees performance though the test-time data has large shift from the clean data. As result, TeCo achieves consistent robustness improvements on different corruptions across datasets.
A.3 BASELINE IMPLEMENTATION AND
Because all the baseline methods are model-agnostic, we follow the implementation the original papers and code. BN simply controls the weights of training and test statisties in the normalization layers. We denote the hyper-parameter as a in Section 3.1 as well. Tent and TTT* have no specific nyper-parameeter but they are sensitive to the learning rate. SHOT contains two objectives. One pseudo-label classification loss and the other is entropy loss. We use to indicate the weight of the pseudo-label classification loss. For fair comparison, we use batch size of 32 in all the implementations.
Table Implementation detail for baseline methods
Dataset
Baseline
Hyper- Parameter
Learning Rate
BN
weight of training statisties a=0..5
Tent
lc-3
Mini Kinetics-C
SHOT
weight of pscudo- label classification loss X=0.3
le-3
TTT*
lc-4
BN
weight of training statistics a=0.66
Tent
le-5
Mini SSV2-C
SHOT
weight of pseudo-labe label classification loss = 0.3
le-5
TTT*
le-5
A.4 ABLATING L1/L2 DISTANCE AND PARTIAL/FULL PARAMETER UPDATE
In the default setting of TeCo, use 1 distance measure the similarity of features along the time dimension. The /1 loss will penalize small perturbations and encourage sparsity in the feature maps, which enhances model robustness (Guo et al., 2018 Chen et al.. 2022). 12 loss is also widely used in de-noising. It usually leads to smoother change. We find that 12 distance also achieves promising performance on corruption robustness. We obtain mPC of 56.8% when TeCo follows the default settings. It remains open question to explore the optimal metric for measuring similarity.
We also conduct an ablation study on Mini Kinetics-C and UCF-HMDB datasets to understand the impact of network parameter updates. TeCo updates part of network parameters mentioned Section 3.1. Table 6 shows that fully updating network parameters may hurt the performance. Especially for small datasets like UCF-HMDB, the accuracy drops by 1.1% on UCF-HMDB and by 1.3% HMDB-UCE Limiting the parameters which can be updated can improve both optimization stability and efficiency (Wang et al.! 2021a).
Table 6: PartiallyFuully update network parameters. Comparison of experiments with updating all/partial network parameters. All the experiments in this table followed the default hyper- parameearsmesees.
Network Parameters
Mini Kinetics-C
UCF-HMDB
HMDB-UCK
Partially Fully
56.9
77.5
83.5
56.2(-0.7)
76.4(-1.1)
82.2(-1.3)
15


Reviews:
[{"review_id": "b-83fQhpjD", "reviewer": "Reviewer_bEhU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method of test-time adaptation for video action classification. In specific, the paper introduces two regularization losses. One is the temporal coherence regularization on the local pathway to guarantee the smoothness of adjacent frames, while the other is the entropy minimization to enhance the robustness of the shallow layers. The experiments show that the proposed method achieves the best results on two corruption robustness datasets (Mini Kinetics-C, Mini SSV2-C).\n", "summary_of_the_review": "Overall, it is a borderline paper with clear motivation and a reasonable method. The downside is the experimental analysis, which could be more informative and highlight the contribution. ", "strength_and_weaknesses": "Strength:\n+ The two regularization is neat and makes sense. Especially, the attention module to compute the temporal affinity smooths the feature and constrains the temporal coherence, which implements the motivation precisely.\n+ The method achieves strong results and conducts a thorough ablation study.\n+ The paper writing is good and easy to understand.\n\nWeakness:\n+ Figure 3 and Figure 4 are surplus. Leaving one figure in the main paper would be enough. \n+ The ablation study of tuning the whole parameters of the model is missing. Otherwise, only tuning shallow layers and BN layers in the deep layers sounds unreasonable to me. It would be better to provide some experimental evidence.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M_1CdIHH1Jk", "reviewer": "Reviewer_rshF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a test-time training (TTT) method called TeCo, which uses not only a standard classification loss, but also a temporal consistency loss, for TTT on video data. On Mini Kinetic-C and Mini SSV2-C, the authors compared TeCo with a number of baseline methods and show that TeCo outperforms prior work on the evaluated settings.", "summary_of_the_review": "Overall, my main concern is regarding experiments. I find the experiment settings limited, and I'm not convinced by the experiments that the proposed method is effective. A more realistic setting (including real datasets, stronger models) would have been more convincing.\n\n\n--- post rebuttal ---\nThanks for providing the rebuttal. The additional experiments comparing with TTT* and the experiments using MViT are helpful and clarify many of my concerns. I also find the domain shift experiments on HMDB and UCF101 very nice and convincing. Regarding the dataset scale, my concerns are still not fully addressed. One potential way to understand the effect of pretraining data scale is to evaluate the proposed method under increasingly larger \"pretraining dataset size\", to see if the improvement still holds when pre-training dataset is large. \n\nOverall, with the new results, I find the paper greatly improved, and I adjusted the rating accordingly.", "strength_and_weaknesses": "Strength:\n+ Advancing TTT can potentially have a big impact on real-world computer vision applications.\n+ The authors evaluate TeCo on more than one dataset, and demonstrate consistent improvement on both.\n\nWeakness:\n- The datasets are not commonly used ones in video research or applications. Mini Kinetic-C and Mini SSV2-C are small and the corruptions are artificially generated. Regarding the size, the larger the pretraining dataset is, the less important TTT could be. Thus, instead of experimenting on small scales, it might be more realistic to understand the methods on commonly used larger datasets (e.g. full Kinetics or full SSv2). \nRegarding the corruptions, I'm not convinced that dealing with these artificially generated corruptions are a great way to study generalization. A better way can be to use full Kinetics and SSV2 to pre-train the model, and then use some \"real-world datasets on other domains, potentially with corruptions\" to evaluate the TTT. After all, if one cannot easily find video datasets with \"corruptions\", maybe that suggests the setting isn't very practically useful.\n\n- The models used are small, proposed in ~2017-2019, and perform significantly worse than todays' state-of-the-art. Video architectures have made significant improvements over the last 3-5 years, so one might observe different observations with newers models (e.g. Swin, MViT, etc.).\n\n- There are a number of video TTT models such as, Azimi et al. 2022, but they are not compared.\n\n- The authors note that \"For standard (without test-time optimization), BN, Tent, and SHOT methods, we use\nuniform sampling to extract input from corrupted test data. We apply both uniform and dense sampling to create input for TeCo.\". I wonder if using \"dense sampling\" contributes the higher performance (because effectively the model sees more diverse inputs and can be seen as a type of data augmentation. )\n\n- I wonder how the baselines are implemented and how are the hyperparameters picked. Since the original baseline papers didn't perform these experiments, I assume the authors implemented these experiments. I wonder if the hyperparameters picked would affect the results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DiSLIONTdv", "reviewer": "Reviewer_naa6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method that deal with input domain shifts in videos during test time (I.e., it deals with corruptions not observed in training data, like different weather conditions).  They propose two self-supervised objectives that are applied to the test-data: one minimizes the entropy of predictions across frames, the other penalizes features/representations that change too strongly during different frames of the video. Results on Mini Kinetics C and Mini SSV2-C are provided.", "summary_of_the_review": "The authors introduce a method to optimize a network for test-time domain shift. Empirical results look promising, but in its current state the authors need to adress some clarity and reproducibility concerns.\n\nUPDATE: After seeing the updated version by the authors, I feel like most of my weak points have been adequately addressed. I'm not familiar enough with the problem it is addressing to judge it's relevance to the community. But both motivation and empirical validation seem sound to me, so I would like to raise my rating to a 7 (i.e., \"good paper, but unclear about practical relevance\").", "strength_and_weaknesses": "Strenghts:\n* The method seems applicable to any video model\n* The empirical results look good\n\nWeaknesses:\n* The method is not evaluated on Transformer-based models. It would be nice (though not necessary) if this would be added.\n* The authors do not discuss runtime implications: how expensive is it to run this method? How long do you need to tune this? How much data do you require to obtain reasonable results? It would be nice if these points could be adressed.\n* There are no explanations/ablations on some of the hyperparameters that the metod introduced (See \"Clarity\" below)\n* The authors need to work on Clarity/Reproducibility (see further down in the review)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster