Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method that deal with input domain shifts in videos during test time (I.e., it deals with corruptions not observed in training data, like different weather conditions).  They propose two self-supervised objectives that are applied to the test-data: one minimizes the entropy of predictions across frames, the other penalizes features/representations that change too strongly during different frames of the video. Results on Mini Kinetics C and Mini SSV2-C are provided.
strength_and_weaknesses: Strenghts:
* The method seems applicable to any video model
* The empirical results look good

Weaknesses:
* The method is not evaluated on Transformer-based models. It would be nice (though not necessary) if this would be added.
* The authors do not discuss runtime implications: how expensive is it to run this method? How long do you need to tune this? How much data do you require to obtain reasonable results? It would be nice if these points could be adressed.
* There are no explanations/ablations on some of the hyperparameters that the metod introduced (See "Clarity" below)
* The authors need to work on Clarity/Reproducibility (see further down in the review)
clarity,_quality,_novelty_and_reproducibility: In general, the exposition was not always clear, some parts of the paper were hard to follow. The authors introduce hyperparameters but never explain them, or even state how they are set during training, which makes it impossible to reproduce the results in the paper.

* The authors use several symbols without introducing them. For example, what are the g and f functions in Section 3.2? They also appear in Figure 1, but are never introduced in the text.

* The hyperparemters alpha and beta are introduced in the beginning Section 3.1, but it’s never discussed how these variables are set during the experiments, or how sensitive the method is to changing this variable.  There is (an apparently different) beta turning up in Eq. 3.4. Again, there is no discussion about how to set this hyperparameter or how sensitive results are to changing this.

*  Eq (1): Why L1 distance? Wouldn’t L2 distance make more sense? (i.e., do not penalize small pertubations?)


Minor remarks:
We propose to build our method upon the test-time optimization which updates all the parameters in shallow layers and only normalization layers in the deep layers. => What are "shallow layers" and what are "deep layers"? From Figure 1, I think you mean the initial (lower) layers are shallow and the layers deeper in the network are the "deep layers". But this formulation is ambiguous and not clearly defined.

“TeCo increases 8% and 5.4% average accuracy across backbones on Mini Kinetics-C and Mini SSV2-C” => TeCo increases accuracy by 8% on across backbones on Mini Kinetics-C and by 5.4% on Mini SSV2-C.

“Recently there emerge studies” => Recently, studies have emerged

“they have subtle enhancement on video data” => they only show minor improvements on video data


summary_of_the_review: The authors introduce a method to optimize a network for test-time domain shift. Empirical results look promising, but in its current state the authors need to adress some clarity and reproducibility concerns.

UPDATE: After seeing the updated version by the authors, I feel like most of my weak points have been adequately addressed. I'm not familiar enough with the problem it is addressing to judge it's relevance to the community. But both motivation and empirical validation seem sound to me, so I would like to raise my rating to a 7 (i.e., "good paper, but unclear about practical relevance").
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a test-time training (TTT) method called TeCo, which uses not only a standard classification loss, but also a temporal consistency loss, for TTT on video data. On Mini Kinetic-C and Mini SSV2-C, the authors compared TeCo with a number of baseline methods and show that TeCo outperforms prior work on the evaluated settings.
strength_and_weaknesses: Strength:
+ Advancing TTT can potentially have a big impact on real-world computer vision applications.
+ The authors evaluate TeCo on more than one dataset, and demonstrate consistent improvement on both.

Weakness:
- The datasets are not commonly used ones in video research or applications. Mini Kinetic-C and Mini SSV2-C are small and the corruptions are artificially generated. Regarding the size, the larger the pretraining dataset is, the less important TTT could be. Thus, instead of experimenting on small scales, it might be more realistic to understand the methods on commonly used larger datasets (e.g. full Kinetics or full SSv2). 
Regarding the corruptions, I'm not convinced that dealing with these artificially generated corruptions are a great way to study generalization. A better way can be to use full Kinetics and SSV2 to pre-train the model, and then use some "real-world datasets on other domains, potentially with corruptions" to evaluate the TTT. After all, if one cannot easily find video datasets with "corruptions", maybe that suggests the setting isn't very practically useful.

- The models used are small, proposed in ~2017-2019, and perform significantly worse than todays' state-of-the-art. Video architectures have made significant improvements over the last 3-5 years, so one might observe different observations with newers models (e.g. Swin, MViT, etc.).

- There are a number of video TTT models such as, Azimi et al. 2022, but they are not compared.

- The authors note that "For standard (without test-time optimization), BN, Tent, and SHOT methods, we use
uniform sampling to extract input from corrupted test data. We apply both uniform and dense sampling to create input for TeCo.". I wonder if using "dense sampling" contributes the higher performance (because effectively the model sees more diverse inputs and can be seen as a type of data augmentation. )

- I wonder how the baselines are implemented and how are the hyperparameters picked. Since the original baseline papers didn't perform these experiments, I assume the authors implemented these experiments. I wonder if the hyperparameters picked would affect the results.

clarity,_quality,_novelty_and_reproducibility: Clarity is good. I can understand the proposed method easily.
The novelty seems on the weaker side to me, because temporal consistency is not new (e.g., Azimi et al. also uses some sort of temporal consistency).
The experiments are not entirely convincing (pls see discussions above).
summary_of_the_review: Overall, my main concern is regarding experiments. I find the experiment settings limited, and I'm not convinced by the experiments that the proposed method is effective. A more realistic setting (including real datasets, stronger models) would have been more convincing.


--- post rebuttal ---
Thanks for providing the rebuttal. The additional experiments comparing with TTT* and the experiments using MViT are helpful and clarify many of my concerns. I also find the domain shift experiments on HMDB and UCF101 very nice and convincing. Regarding the dataset scale, my concerns are still not fully addressed. One potential way to understand the effect of pretraining data scale is to evaluate the proposed method under increasingly larger "pretraining dataset size", to see if the improvement still holds when pre-training dataset is large. 

Overall, with the new results, I find the paper greatly improved, and I adjusted the rating accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method of test-time adaptation for video action classification. In specific, the paper introduces two regularization losses. One is the temporal coherence regularization on the local pathway to guarantee the smoothness of adjacent frames, while the other is the entropy minimization to enhance the robustness of the shallow layers. The experiments show that the proposed method achieves the best results on two corruption robustness datasets (Mini Kinetics-C, Mini SSV2-C).

strength_and_weaknesses: Strength:
+ The two regularization is neat and makes sense. Especially, the attention module to compute the temporal affinity smooths the feature and constrains the temporal coherence, which implements the motivation precisely.
+ The method achieves strong results and conducts a thorough ablation study.
+ The paper writing is good and easy to understand.

Weakness:
+ Figure 3 and Figure 4 are surplus. Leaving one figure in the main paper would be enough. 
+ The ablation study of tuning the whole parameters of the model is missing. Otherwise, only tuning shallow layers and BN layers in the deep layers sounds unreasonable to me. It would be better to provide some experimental evidence.
clarity,_quality,_novelty_and_reproducibility: The paper writing is good and implementation details are specific, which benefits reproducibility.
summary_of_the_review: Overall, it is a borderline paper with clear motivation and a reasonable method. The downside is the experimental analysis, which could be more informative and highlight the contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Test-time optimisation is an effective way to increase model robustness to corrupted data without having to retrain the model. While the problem was studied in the context of images, it received less attention in the context of videos. In this work the authors propose a method to improve the test-time robustness to distribution shifts in video classification models. In particular, two self-supervised objectives are coupled with the standard classification loss, namely entropy minimisation and temporal coherence/consistency regularisation. The results are evaluated on two corrupted datasets (Mini Kinetics-C, mini SSv2-C) where the method outperforms prior work. 

The reviewers appreciated the clarity of writing, the significance of the problem, and the fact that the suggested regularisers match the motivation laid out by the authors. The rebuttal did a good job convincing the reviewers in the empirical results. I will recommend the acceptance of this work. Please add all the clarifications and ablations from the rebuttal in the revised submission as they played a critical role during the discussion phase.
justification_for_why_not_higher_score: - Main datasets are not commonly used in video classification (artificially generated Mini-Kinetics and Mini-SSv2), relatively small dataset sizes
justification_for_why_not_lower_score: - Important problem with real-world applications
- Thorough empirical evaluation (including the rebuttal phase)
Decision: Accept: poster