MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer
Authors: Minghao Zhu
Zhengpu Wang
Mengxian Hu
Ronghao Dang
Xiao Lin
Xun Zhou
Chengju Liu
Qijun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Recognition
Vision-Language Model
Transfer Learning
Primary area: machine_vision
TL;DR: This paper presents MoTE, a knowledge transfer framework that addresses the generalization and specialization trade-off problem when transferring VLMs to the video domain, achieving an optimal balance between close-set and zero-shot performance.

Abstract: Transferring visual-language knowledge from large-scale foundation models for video recognition has proved to be effective. To bridge the domain gap, additional parametric modules are added to capture the temporal information. However, zero-shot generalization diminishes with the increase in the number of specialized parameters, making existing works a trade-off between zero-shot and close-set performance. In this paper, we present MoTE, a novel framework that enables generalization and specialization to be balanced in one unified model. Our approach tunes a mixture of temporal experts to learn multiple task views with various degrees of data fitting. To maximally preserve the knowledge of each expert, we propose Weight Merging Regularization, which regularizes the merging process of experts in weight space. Additionally with temporal feature modulation to regularize the contribution of temporal feature during test. We achieve a sound balance between zero-shot and close-set video recognition tasks and obtain state-of-the-art or competitive results on various datasets, including Kinetics-400 \& 600, UCF, and HMDB. Code is available at https://github.com/ZMHH-H/MoTE.

MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer
Minghao Zhu Xiao Lin
Zhengpu Wang Mengxian Hu Chengju Liu' Tongji University, Shanghai, China {zmhh_hr h, wangzhengpu, humengxian, dangronghao, linx_xx, zhouxun, liuchengju, qjchen)8tonnji edu. cn
Ronghao Dang Qijun Chen
Xun Zhou
Abstract
Transferring visual-language knowledge from large-scale foundation models for video recognition has proved to be effective. To bridge the domain gap. additional parametric modules are added to capture the temporal information. However, zero-shot generalization diminishes with the increase in the number of specialized parameters, making existing works a trade-off between zero-shot and close-set performance. In this paper, we present MoTE, a novel framework that enables generalization and specialization to be balanced in one unified model. Our approach tunes mixture of temporal experts to learn multiple task views with various degrees of data fitting. To maximally preserve the knowledge of each expert, we propose Weight Merging Regularization, which regularizes the merging process experts in weight space. Additionally with temporal feature modulation to regularize the contribution of temporal feature during test. We achieve sound balance between zero-shot and close-set video recognition tasks and obtain state- of-the-art or competitive results various datasets, including Kinetics-400 & 600, UCF, and HMDB. Code is available at https: // //g thub. com/ com/ZMHH- H/MoTE.
1
Introduction
With the advent of large-scale Vision-Languaag Models (VLMs), adapting such foundation models (e.g., CLIP [33], ALIGN [13], Florence [54]) for downstream tasks has become emerging paradigm of intense scrutiny. Their semantic visual concepts empowered by aligning large-scale image-text pairs can be transferred
wide range of downstream tasks, such as open-vocabulary classification [55, 56], detection [11, 51], and segmentation [21, 52].
The crux of an adaptation procedure for foundation models lies in injecting specialized knowledge of the domain of interest. For video recognition tasks, this necessity is reflected in the fact that the dynamic nature of video data requires effective comprehension of context and temporal correlation by the model. Therefore, to condition the adapted model on the video-specialized knowledge, one general and effective way is to incorporate additional parameters in the form of well-designed prompts [43, 45], adapters [30, 53), and temporal modules [28, 49, 50]. However, we observe that while the increased model capacity brought by more additional parameters enables the better fitting of video- specific inductive bias, it comes at the cost of catastrophically forgetting the generalization knowle ledge of the original VLM. To better illustrate this phenomenon, we present overview of existing methods in Figure I. A clear trade-off problem between zero-shot and close-set performance emerges in existing works, which correlates the scale of additional parameters. The former relies more on the generalization capability inherent in VLMs while the latter requires intensive addo-speciallied knowledge, but no method achieves the best of both worlds. On one hand, introducing new knowledge
"Corresponingg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Video
00
00
Harreck TRIE stheea
Encoder
VLM Visual Ercoddr
Moan Pooting
Wiee
STAtector
SAN 231
SCL
Pensama Glayer
Tomspora
NoTE (Ours!
VLM Viual Encoder
VLM Visual Ercoder
Acrel Acople
TORRE
Lapees Clayart
Tersicn Modie
Video
...
Video
Tap 1000 Cosseet
Zero-shot Close-set performance
Performance trade-off temporal layer numbers.
(c) Feature space of various VLM methods
Figure Overview of existing VLM knowledge transfer methods. (a) Trade-off plots between zero-shot (Harmonic mean of UCF, HMDB, and K600) and close-set (K400) performance of recent CLIP-based methods (ViT-B/16), (b) As the number of temporal layers increases, the generalization of the standard Transformer layer severely degrades while our proposed MoTE consistently improves the zero-shot and close-set performance. (c) Our proposed MoTE seeks to construct a reconciled feature space between the optimal generalized and specialized manifolds.
while preserving existing knowledge continually is desirable and significant for the broader adaptation of the foundation model. The rapidly evolving real-world applications also require both specialization and generalization capabilities. However, how to manage the generalivationsseciillaaaooo trade-off of additional parameters in transfer learning remains under-explored.
This paper addresses the aforementioned challenge by delving into two inquiries: (i) How can the generalization capability of additional parameters be enhanced? Given the substantially smaller scale of the fine-tuning dataset compared the pre-training dataset of VLMs, the newly added parameters risk overfitting the fine-tuning data bias, thereby constraining the model generalization. Addressing this question enables steering parameters towards high generalization. Nevertheless, generalization and specialization have proven to be somewhat conflicting [18, 29, 36] in model training, prompting us to further explore (ii) How can generalization and specializution coexist in one unified model? This question is crucial for balanced model to preserve both new and existing knowledge, yet has received limited exploration. We investigate these two questions with a widely employed temporal module [14, 49, 50] (i.e. N-layer transformer), which already features high specialization degree but is less performant on zero-shot tasks.
With this in mind, we present MoTE, generalization and specialization. We offer novel perspective in addressing the first question: constructing more generalized model using multiple data bias views. In contrast to conventional temporal modules that encode patterns with a single feedforward network (FFNN in each Transformer layer, MoTE uses multiple FFN experts to capture various data bias views. During fine-tuning, incoming frame token sequences are routed to one of the temporal experts, keeping the computational cost the same as using single FFN. To enlarge the discrepancy in knowledge Icarned by each expert, we devise a routing algorithm based on the multinomial distribution. During inference, we apply weights merging to collapse multiple experts into one module, enabling the patched model to aggregate the generalized knowledge of each expert. For the second question, we propose Weight Merging Regularization which regularizes the merging process of experts in the weight space. The proposed regularizer drives a range of the merged parameters optimal with respect to the task-specific objective, allowing more effective aggregation of generalized and specialized knowledge in the patched model. To further alleviate the overfitting at test time, we devise plug-and-play module that modulates the contribution of temporal features by measuring the semantic association between the proxy text features from the fine-tuning and the test datasets.
approach with well-balanced
Our main contributions can be summarized as follows:
We propose MoTE a knowledge transfer framework from visual-language to video domain. MoTE tackles the fitting trade-off challenge posed by additional parameters, an aspect largely overlooked by previous works.
We provide new insights for enhancing parameter generalization from the perspective of data bias fitting, all while keeping the computation cost and final structure constant ($ 3. 3.2).
We propose Weight Merging Regularization, novel regularizer for more effective knowledge aggregation, realizing the coexistence of generalization and specialization in weight space ($ 3.3). Together with Temporal Feature Modulation to further improve the generalization of MoTE ($ 3.4).

---

Extensive experiments demonstrate MoTE achieves an optimal trade-off between zero-shot and close-set performance with one unified model. Thorough ablation studies show the scalability and effectiveness of our proposed method ($ 4).
2 Preliminary: Transferring CLIP for Video Recognition
Recent works have adapted CLIP [33] to video datasets and obtained superior results [28, 41]. We briefly describe the typical cross-modal video recognition pipeline. Consider a video V with T frames and the corresponding text label C' described in a set of textual prompts. Each frame is encoded independently by the CLIP visual encoder [(-10,) with parameter 0g and produces frame-level embeddings fee E IR y The text embedding y E RD is generated by the CLIP text encoder g(-10g) with parameter ee, where D is the embedding dimension,
===1-
e1, er = J(V (o), y = o(C)..).
The CLIP visual encoder captures rich spatial information. To bridge the domain gap between image and video, we apply a commonly used temporal module h- h--10emm for cross cross-frame frame communication [14, 49, 50], which is parameterized by several Transformer layers. The final video emhedding E R consists of spatial and temporal embeddings connected in a residual form:
AvgPool((e1. erl + h(e1, "errr
(2)
During optimization, the text encoder typically frozen. We tune the visual encoder and the temporal (x.y)
module to maximize the similarity yi | between the video embedding z and the text embedding if V and C' are matched, otherwise minimize it. The training objective can be formulated as:
C(O,, ftem: D) = Div.c)-o([(ssm(2.y)) onchot(C))].
(3)
where D is the fine-tuning dataset, is the cross-entropy function with softmax operation.
Methodology
overview our proposed method presented in Figure 2. This section first analyzes the parameter generalization from the perspective of data bias fitting ($ 3.1). Then we detail the structure, routing policy, and inference of MoTE (8 3.2). Finally, we present Weight Merging Regularization ($ 3.3) and
Temporal Feature Modulation ($ 3.4) towards the coexistence of generalization and specialization.
3.1 Intuition and Motivation
Typically, more additional parameters allow for better fitting of the training data distribution, leading to better close-set performance. However, steering the model specialized on the target distribution potentially makes it sensitive to out-of-distribution shifts, resulting in the generalization drop on downstream data distributions with unseen video categories. Although one could enlarge the training data distribution to cover as many potential unseen categories as possible, the costly computation and collection of video data make this infeasible, indicating the need for more generalized architecture.
Neural network optimization has many solutions in different loss basins due to the non-convexity of the loss landscape. Models optimized with different configurations (e.g. initialization, optimizer, and data) have different optimization trajectories and may converge to separate local minima, thereby capturing various feature patterns. This inspires our method to construct more generalized model using multiple data bias views. Instead of improving generalization by searching for flatter minimum in the loss landscape [22, 48], we expect the aggregation of diverse knowledge from multiple minima can provide more comprehensive representation. Intuitively, diverse temporal patterns can better facilitate recognizing an unseen video category. For example, aggregating information on 'player movement" and 'racket-ball interaction' can help better recognize "playing tennis'. Our architecture design takes inspiration from fixture-of-Exxperr experts. While MoE was originally proposed for building large pre-trained models, we extend to the context of transfer learning and demonstrate its effectiveness in improving parameter generalization.
[37] to learn multiple data bias views with set of
3

---

Video
Visual Encoder
MoTE
Twd wnttedding
ombbodding
FFN
FFN
FFN
FFN4
Merqed FFN
Router
Layer Norm
Sampling
Layer Norm
CLIP Text Encoder
cestoodding
Simbarty Calculaton
Self SeffAAttantion
Sel-Allention
Freme-iavel
taken sequiince
CLIP Visual Encoder
Textual Prompl
*Playing Tennis"
Text catepory
Pree-trting category
1+1 1-2
I-T-1
Figure 2: An overview of the MoTE framework. (Left):: We independently extract the feature of each frame with the CLIP visual encoder. Then, the frame token sequences from a given batch are routed to an activated expert for temporal pattern encoding. To regularize the merging process, we sample the temperature from discrete set and use it to collapse multi-experts into one merged FFN. (Right): Temporal feature modulation. We modulate the contribution of the temporal feature with the semantic association, which is measured by the similarity between the proxy text features retrieved from the fine-tuning and the test categories. The modulated embedding is used for inference.
3.2 Mixture-of- Teciuur-of-Temppor---EE-rre
Architecture Consider the temporal module h(-10tem) as repeated Transformer layers which consist of self-attention layer and fully-connected feed-forward network (FFN). For clarity, the temporal module's parameter 0tem is factorized into parameter Out for attention layers and parameter 0 for FFNs, where Otem = fatt U Of. Recent studies [5, 9] suggest that factual knowledge is mainly stored in the FFN, which consists of two learnable projection matrices and an activation function. Thus, we replace the FFN in each Transformer layer with N temporal experts and obtain set of experts { 1 N 1 j=1- where E represents the 7th expert the jth layer and has the Ti=1
same structure as the FFN. Each expert starts training from different random initialization to ensure different optimization trajectories, which
experimentally show as critical for learning distinct knowledge (i.e. data bias views). Denote the parameter of the expert Eq 0 and the activated expert's 's index the tth layer E(j). The training objective becomes:
LTF = C(0v, Oatty (0 (o D).
Routing Policy The routing algorithm of MoE determines which experts process inputs. Classic routing mechanisms (i.e. top-k gating [37]) make decisions using learnable gate network and require auxiliary losses to handle load balancing. Instead, we adopt the stochastic routing algorithm [59] considering the architecture brevity, as it avoids the need for extra network, computation, and loss.
Recall that we expect each expert to learn distinct feature patterns. To further enlarge the discrepancy in knowledge learned by each expert, we present stochastic routing algorithm based on the multino- mial distribution. By assigning different activation probabilities to experts, we can control the training data volume of each expert, empowering their knowledge with different degrees of generalization and specialization. Formally, set A = [a]] to be a random vector in the 1thh layer, where of E {0,
indicates whether the ith expert is activated and Σi of 1. We specify the variable A to follow the multinomial distribution where the activation probability of each expert is given as
exp(i)
P(a =
(5)
Σ exp(1))
where exp(-) is the natural exponential function. This allows experts with greater index to be activated more likely and therefore receive larger volume of data during training. Once the expert selected, all inputs in given batch are processed by the same expert.

---

Knowledge Aggregation in Inference During inference, knowledge across experts can be ag- gregated by either merging weights [44, 47] or ensembling logits [16]. We adopt merging weights with the following benefits: (i) More effective knowledge aggregation evidenced in Table 2. (ii) Constant computation cost, while the latter increases linearly with the expert number. (iii) Consistent parameter structure with its initial state. Specifically, for the experts {E) in the t layer, we
average the weights of all the experts obtain the final parameters A for inference, where
0
(6)
3.3 Weight Merging Regularization
While the introduction of MoTE notably improves model generalization, it still incurs decrease in close-set performance as presented in Table I, which again proves the conflicting nature of generalization and specialization in weight space. We attribute this phenomenon to the difficulty of simultaneously preserving the generalized and specialized knowledge during expert merging process.
In this regard, we propose Weight Merging Regularization to reconcile and promote the aggregation efficacy of expert knowledge. To maximally aggregate the specialized knowledge of experts, the final merged model necessitates explicit optimization with respect to the task-specific objective. As for the preservation the generalization knowledge, previous works [17, 22, 27] observe that the flatness (i.e. sharpness) of local minima in the loss landscape is strongly correlated with the model generalization. They suggest that models with flatter local minima in the loss landscape generalize better unseen data, where the flatness refers to the robustness of the loss value to perturbations in the parameters. Within a flat basin the loss landscape, moderate variations of parameters do not lead to significant increases in loss. Inspired by this observation, we propose to facilitate preserving the generalized knowledge of experts by explicitly constructing alaa region around the loss landscape of the final merged model. We optimize the constructed region with the task-specific loss objective and demonstrate that region construction benefits the coexistence of specialization and generalization in one unified model. Formally, for the experts (E J in the 1 layer, we first merge the weights as
N
exp(i/T)
exp(U//) 0
where r is temperature parameter used control the softness of the distribution. By varying the value of r, we can build region of the merged parameters in the weight space. Considering that continuous space may lead difficulties in model convergence, we sample from discrete set given as (+:2% -3)a U (oo), where 3 is a hyper-parameter of the candidate set. With parameters ((h) } merged using different 7 in each layer, the regularizer defined as:
sampling
LWMR L(Ov, Oatey
J D).
(8)
Denote the generated video feature when computing LWMR as Zy We further slightly regularize the consistency between z, and the pooled spatial feature e from the CLIP visual encoder:
LUSE Bevsey=pllzz, e|
where |l-1|2 is the L2 norm. This regularizer encourages MoTE to model the temporal dynamics with a light temporal feature, which potentially reduces the model complexity and improves generalization. The overall learning objective can be written as:
CALL Lre + ALWMR + L
(10)
where we set À = 0.5 and 7 = 0.1 by default.
3.4 Temporal Feature Modulation
While applying MoTE to downstream recognition tasks, potential risk is that its semantic space is limited the category names of the fine-tuning dataset. To demonstrate this, we collect the category names of Kinetics-400, UCF-101, and Kinetics-600 datasets and manually remove duplicate ones. We
5

---

conduct tests with the mixed category names and observe 2.9% closed-set performance drops, 29.8% and 31.1% zero-shot performance drops on Kinetics-400, UCF-101, and Kinetics-600, respectively. That is, the model tends to classify all videos into fine-tuning categories.
In light of this, propose test-time adaptation strategy for networks where the temporal module is separated from the CLIP encoder, to measure the confidence of temporal features by means of CLIP's text space and modulates the contribution of temporal features to the prediction. Given the pooled feature from the CLIP visual encoder, we generate fine-tuning and test proxy features Y/.Yz E RXXD by retrieving the K nearest text features in the fine-tuning and the test dataset categories. We estimate the semantic association p between the two proxy features as
p eexp((t -Myy))))/>))
(11)
where 7 stands for a scale hyper-parameter parameter and (M()) is sequential maximum and average pooling operation. Denote the pooled temporal feature from MoTE as t, the final video embedding used for inference becomes z = e + p.t.
4 Experiments
4.1 Experimental Setup
Architecture. We employ the CLIP [33] pre-trained ViT-B/16 and ViT-L/14 in our experiments. On top of the visual encoder, we add layers MoTE for ViT-L/14 and 4 layers MoTE for ViT-B/16, with 4 temporal experts per layer by default.
Implementation Details. We fine-tune our model using the Kinetics-400 [15] dataset as in previous works [28]. During fine-tuning, sparsely sample T (e.g. 8 or 16) frames as the video imput. Each input example is randomly cropped and resized to the size of 224 x 224 and then undergoes random horizontal flip and random grayscale. We adopt Adam W [25] as the optimizer with weight decay of 0.2, following half-period cosine learning rate decay. The initial learning rate is set to 5 x 10-5 - with total batch size of 1 144. Furthermore, scale parameter for temporal feature modulation to 0.05, and K to 5. We apply temporal feature modulation only in evaluation. Please supplementary for more details.
set the candidate
3 for Lwmn 0.6 and the
Evaluation Protocols. We thoroughly evaluate our method with close-set, zero-shot, and few-shot
video recognition. Close-ser: We evaluate the close-set set performance on Kinetics-400 [15], using one single clip with a center crop (i.e. 1x 1 views) or 4 clips with 3 crops (i.e. 4 x 3 views) per video [28]. Each view contains 8 or 16 sparsely sampled frames. Zero-shot: Following previous works [28, 34], we evaluate zero-shot performance on UCF-101 [38]. HMDB-51 [19], and Kinetics-600 [3]. For K600, we adopt the three splits provided by [4]. Each split contains 160 categories out 220 new
categories. In zero-shot setting, we test using 3 x views with 8 frames per view. Few-shot: We consider standard K-shot setting and evaluate on UCF-101, HMDB-51, and v2 [10]. We adopt single view for evaluation.
4.2 Ablation Studies
Component-wiss analysis of MoTE Table I, we perform in in-depth ablations of the proposed components with the ViT-L/14 network. We adopt Text4Vis [49] our baseline, which serves prevalent CLIP adaptation framework in the video domain. Text4Vis uses a 6-layer Transformer for temporal modeling, featuring a high degree of specialization but low generalization capability. We observe that adopting the temporal experts boosts zero-shot performance significantly, validating our idea of improving generalization with multiple data bias views. We then introduce LWMR for achiev- ing the coexistence of generalization and specialization, which facilitates a more efficient aggregation of generalized knowledge while achieving the same level of specialization as Text4 Vis. Adding LASE further improves the zero-shot performance. Moreover, we find that zero-shot performance benefits from modulating the temporal features for unseen categories during evaluation, especially for HMDB51. In summary, MoTE achieves sound balance between generalization and specialization.
Expert-wise performance of MoTE, To better understand the reconciliation effect of MoTE, show the performance of each expert and the final merged model in Figure 3. In particular,

---

Negel Made
Nerped Madel
Nerged Mebe
Table 1: Ablation study on various compo- nents of MoTE with ViT-L/14 network.
Sepatte
cus-mean
PaperLo
Sqperi
towet
Expent
Eyprt
Method
K400
UCF
HMDB
K600
Baseline (Tex4Vis [49])
86.7
82.6
52.1
72.8
K400 UCF
(b) K400 HMMD
K3O0 Koo0
+Temporal Experts
85.5
86.6
56.3
78.3
86.8
87.2
56.6
78.3
Figure 3: Expert-wise performance of MoTE. CLIP-
EuSE
86.8
78.9
Mean denotes a fine-tuned CLIP model with mean
Temp. Feature Modulation
86.8
88.7
61.4
79.0
pooling for temporal modeling.
Table 2: Ablation studies on key details. We report close-set accuracy on K400 and zero-shot accuracy on UCF-101 and K600 splitl, using the ViT-L/14 network. Default settings are colored in gray.
(a) Effect of initialization and train-
(b) Varying numbers of temporal ex-
(c) Various types of knowledge ag- gregation.
ing data across experts.
perts in each MoTE layer.
Init.
Data
K400
UCF
K600
Numbers
K400
UCF
K600
Type
Param. K400 UCF K600
Same
All
86.7
82.6
72.8
86.6
86.7
77.4
Random rouling
127.6M 85,3 85.4 76.0
Same
Rouled
86.3
86.4
77.9
86.8
87.5
78.9
Ensembbling logits
127.6M 86.0 86.3 77.5
Different
All
86.8
87.2
79.0
86.7
87.6
78.5
Merging weights
42.6M 86.8 87.5 78.9
Different
Rouse
86.8
87,5
78.9
86.7
87.1
78.3
(d) Different routing policies during
(e) Different types of LWMR and (f) Varying scale parameters Y for the temporal feature modulation.
fine-tuning.
candidate set parameter 33.
Policics
K400
UCF
K600
Type Point
K400
UCF
K600
7
K400
UCF
K600
Fixed
86.7
72.8
86.1
78.0
0.04
88.6
78.9
Random
86.8
86.9
78.1
0.5
86.8
87,5
78.9
0.05
86.8
88.7
79.0
Muliinomial
86.8
87.5
78.9
Rogion
0.8
86.8
87.2
78.7
0.06
86.8
88.6
78.8
1.0
86.7
87,5
78.5
0.07
86.8
88.2
78.6
CLIP-Mean denotes a fine-tuned CLIP model with mean pooling for temporal modeling. We do not apply temporal feature modulation in this study, As shown in the figure, each expert exhibits distinct zero-shot and close-set performances. suggesting that diverse knowledge is learned across experts, as expected in Section 3.2. We find that the zero-shot performance of each dataset presents different sweet spots terms of the generalization degree. For example, Expert_ achieves the best results on UCF-101, while Expert_0 performs best on HMDB-51. After weight merging using Equation 6, the merged model yields better zero-shot results than any expert and CLIP-Mean, and the close-set performance slightly outperforms Vis. It demonstrates that our method effectively preserves the generalized and specialized knowledge of every expert, which can be attributed LWMR.
Effect of distinct optimization trajectories. In Table 2a, we show that learning diverse knowledge from distinct optimization trajectories is critical for improving generalization. "Data" indicates whether the input data is sent to all experts routed to one expert for encoding. In the first row, the optimization trajectories across experts are identical since all experts share the same initialization and input data. We find that applying different initialization and training data across experts can significantly improve generalization, respectively. When comparing row with row 4, using the routing strategy achieves similar generalization performance while keeping the computation cost low.
Varying numbers of experts and temporal layers We ablate the number of experts in Table 2b. We observe that the zero-shot performance gains stabilize after proper number of experts, and 4 temporal experts per layer are sufficient to capture various data bias views. As for the number of temporal layers, we obtain consistent gains with different numbers of layers shown in Figure I (b), demonstrating the scalability of our method.
Various types of knowledge aggregation. As shown in Table 2c, the inferior performance of random routing indicates the necessity of leveraging diverse knowledge across experts. Compared with ensembling logits, merging weights aggregate knowledge more efficiently with fewer parameters. Meanwhile, the computation cost of ensembling logits is 4 x merging weights.
Effect of different routing policies. Table 2d ablates different routing policies. The fixed rout- ing policy sends all inputs to a single expert. Our multinomial routing policy achieves the best performance since each expert learns more discrepant knowledge than random routing.
Weight Merging Regularization. We study different types the merging regularizer LWMR as well as candidate set parameters (3 in Table 2e. "Point" indicates the regularizer with the average merged
7

---

Table 3: Zero-shot video recognition performance compared with the state-of-the-arr methods
UCF-101, HMDB-51, and Kinetics-600. i denotes reproduced results with our implementation.
Method
Venue
Encoder
Frames
UCF-101
HMDB-51
Kinetics-600
ActionCLIP [42]
arXiv'21
VIT-B/16
58.3+3344
40.8+5.4
67.7+1.1
A5 [14]
2CCV'22
VIT-B/16
32
69.3+t4.2
44.3+2.2
X-CLIP [28]
ECCY-22
VIT-B/16
32
72.0#2.33
44.6:1.2
65.21:0,4
ST-Adapterf [30]
NcurIPS'22
VIT-B/16
8
76.9=
51.5#0.6
60.2+
Vita-CLIP [45]
CVPR'23
VIT-B/16
8/32
75.0+0.6
48.6+0.6
67.4+0.5
ViFi-CLIP [34]
CVPR'23
VIT-B/16
76.8.10.7
51.310.6
71.241.00
OTI [58]
ACMMM
VIT-B/16
83.3+0.3
54.24:1.
66,9+100
Open-VCLIP [46]
ICML'23
ViT-B/16
83..11.2
53,911.2
73.010.8
MAXI [23]
ICCV'23
VIT-B/16
16/32
78.2.10.8
52.310.7
71.54008
MoTE (ours)
VIT-B/16
83.4+0.7
55.8+0.9
70.2+00.6
X-Florence [28]
BCCV'22
Florence
32
73.2.14.2
48.414.99
68.810.9
Vixt [49]
AAAI223
VIT-J114
82.6+0.7
52.4+0.4
72.1+0.9
OTI [58]
ACMMM223
VIT-I/14
8
88.1+1.00
59.3+117
70.6+
Open-VCLIP VCLIP [46]
ICML'23
VIT-L/14
87.6.1.22
59.010.6
81.1.0.8
DiSTi [32]
ICCV'23
VIT-L/14
74.9+
57.5+1.6
75.0++0.7
MoTE (ours)
VIT-I/14
88.7+00.6
61.41133
78.4+0.9
parameters (i.e. without region construction). Due to the conflicting nature of generalization and specialization in optimization, the regularizer without region construction canmot adequately aggregate expert knowledge. Instead, constructing the region in the loss landscape enables generalization and specialization to coexist in one model. The parameter 3 is used to construct the candidate set of the weights merging temperature. As shown in the table, the performance gain is robust this parameter.
Varying numbers of y for Temporal Feature Modulation. The parameter y is used to scale the semantic association an appropriate interval. In general, 0.05 yields better results. We find that different datasets exhibit varying sensitivities to this parameter and relatively small contribution of temporal features generalizes to the unseen categories. Please see supplementary for more ablations.
4.3
Main Results
Zero-shot video recognition. In Table 3, we compare our method with the state-of sate--o-the-ase tet-the-rrt art results under the zero-shot setting. Our method achieves new tfuteof-the-wee art-art results on UCF-101 and HMDB- 51, with only 8 frames as input. The remarkable performance can be scaled up as the network size and the input frame number increase, demonstrating the great scalability of our method. For K600, although significant improvements are achieved over the baseline, our method still underperforms some methods that not apply any additional parameters such as Open- VCLIP [46] and MAXI [23]. Since the categories of Kinetics-600 are much more complex, for randomly initialized parameters to generalize well on such complex unseen categories. Note that our method still outperforms any method that employs additional parameters K600. Overall, our method presents superior generalization performance.
argue that it is still challenging
Close-set and zero-shot performance trade-off. Table 4 presents comparisons with the -f the-art methods under the close-set setting. We also list the harmonic mean of the zero-shot results on UCF, HMDB, and K600 as indicator of the generalization capability, denoted HMzs. To evaluate the holistic performance in close-set and zero-shot settings, we define the "Trade-off" score as the harmonic mean of Top-1 and HMzs. we consider the specialization and generalization capabilities to be equally important for a balanced model.
As shown in the table, our method presents strong close-set results competitive with SOTAs and ofte--of-the-art zero-shot performance. More importantly, while existing methods can perform well on only one task setting, we achieve superior performance on both settings simultaneously with one unified model. Our method consistently exhibits significant trade-off performance advantages across different networks, evidencing the effectiveness and scalability of MoTE in reconciling generalization and specialization capabilities. Even against ViFi-CLIP [34] which uses different training hyperparameters for close-set and zero-shot settings and takes more frames as the input, our method still shows better balanced performance (74.7% vs. 72.9%). Noteworthy that FROSTER [12] is strong zero-shot action recognition model but performs less well in the sete-set setting, which

---

Table 4: Close-set and zero-shot performance trade-off compared with the state-of-the-ee art methods. We report close-set results on K400. "HMzs" indicates the harmonic mean of zero-shot results on UCF, HMDB, and K600. "frade-off" is defined the harmonic mean of Top-- 4000 and HMzs- "Unified model" indicates whether the method is evaluated using the same model in both settings.
Unified model
K 400
Method
Eneoder
Inpul Size
GFLOPs
Views
Puram (M)
HMps
Trude-off
Top-1
Top-5
A6 [14]
ViT-B/1G
16x224
91.2
76.9
93.5
X-CLIP [28]
VIT-B/16
8x224
145x1x1
131.5
82.3
95.8
58.1
68.1
Vita-CIIP [45]
VII-B/16
8x224
/x1x1
125.0
80.5
95,9
61.5
69.7
Open-VCLIP [46]
VIT-B/1G
8x224
Ixlx1
86.2
78.9
67.8
72.9
FROSTER [12]
ViT-B/16
8x224
94.8
73.7
MoTE (ours)
VIT-B/16
8x224
141x1x1
98.8
95.9
74,2
ActionCLIP [42]
VIT-B/16
16x224
282x10x3
141.7
82.6
96.2
53.2
64.7
SiF [43]
VIT-B/16
8x224
142x4x3
143.9
77.4
93.6
X-CLIP [28]
VIT-B/16
8x224
145x4x3
131.5
83.8
96.7
58.1
68.6
ST-Adapter [30]
VIT-B/I6
8x224
152x3x1
93.0
82.0
95.7
61.2
70.1
Text4Vis [49]
VIT-B/16
8x224
141x4x3
105.2
82.9
59.1
69.0
Vita-CIIP [45]
VIT-B/16
8x224
1x4x3
125.0
81.8
95.9
70.2
ViFi-CLIP [34]
ViT-B/16
16x224
241x
149.6
96.3
72.9
MoTE (ours)
ViT-B/16
8x224
141x4x3
98.8
83.0
96.3
67.9
74.7
X-Florence [28]
Florence
32x224
2822x4x3
86.5
96.9
61.4
71.8
AIM [53]
VIT-L/14
8x224
934x3x1
341.0
86.8
97.2
Text4Vis [49]
VIT-1/114
8x224
649x4x3
346.6
86.7
97.4
66.6
75.3
Opcn-VCLIP [46]
VIT-L/14
8x224
1x3x1
304.0
83.9
96.5
73.7
78.5
DiST [32]
VIT-L/14
8x224
710x3x1
343.0
97.6
68.1
76.4
MoTE (ours)
VIT-L/14
8x224
649x4x3
346.6
97.5
80.1
MoTE (ours)
VIT-L/14
16x224
x299x4x3
346.6
87.2
97.7
80.5
Table 5: Few-shot results compared with the state-of-the-art methods HMDB, UCF, and SSv2. All methods directly fine-tune on CLIP, except MAXI (Tuning after pre-training on K400.).
HMDB-51
UCF-101
SSv2
Method
K=2
K=1
K=8
K=16
K=2
K=4
K=8
K=16
K=2
K=1
K=8
K=16
CLIP [33]
47.2
47.2
47.2
47.2
75.1
75.1
75.1
75.1
2.9
2.9
2.9
2.9
[14]
39.7
56.0
62.4
79.9
9.7
X-CLIP [28]
53.0
57.3
62.8
64.0
76.4
83.4
88.3
91.4
3.9
4.5
6.8
10.0
ActionCLIP [42]
47.5
57.9
57.3
59.1
70.6
71.5
73.0
91.4
4.1
5.8
8.4
11.1
ViFi-CLIP [34]
57.2
62.7
64.5
66.8
80.7
85.1
90.0
92.7
6.2
7.4
8.5
12.4
MAXI [23]
58.0
65.0
66.5
86.8
89.3
92.4
93.5
9.3
12.4
MoTE (ours)
61.3
67.2
68.2
88.8
7.3
8.5
12.2
+3.3
+1.2
+2.2
+1.4
+2.0
+1.7
+0.1
+0,2
+0.1
+0,2
can be attributed to the the application the weights ensemble and the distillation supervision from the Frozen CLIP limits the model's ability to learn video-specialized knowledge.
Few-shot video recognition. We perform all-way few-shot video recognition in Table 5, which requires both specialization and generalization to rapidly adapt to novel set of categories with limited samples. Our method presents consistent improvements across sample shots in all datasets, demon- strating the strong learning capacity and transferability. Our method even outperforms MAXI [23] which uses K400 fine-tuning weights initialization and more diverse textual knowledge.
4.4 Category- performance visualization.
To better understand the specific video categories that each expert excels at recognizing, we visualize the Top-1 classification accuracy for categories sampled from the UCF-101 dataset in Figure 4. The experiment is conducted in the zero-shot setting. We observe that different experts show distinct performance across video categories, likely due to the diverse generalization knowledge learned by each expert. The merged expert benefits from the aggregation of expert knowledge, particularly for categories where temporal information crucial, such as "Handstand Pushups' and 'Wall Pushups'. This suggests that each expert can capture various temporal patterns within the same category.

---

1.0
Expert_3 3
Expert_
Expert 1
Expert 0
Merged, Expert
Aguly Lpsttia
Balanres Beam
Band Ma Marching Backerba Basketthal Dunk
Body Weigit Squaats Strlla Cricket Bowlng Cricktt
Diving
Handstand Pushupss Horse Hala Hoop
Halrcur
jump Lunges
Milttry Parace Mieing
Pizza Diall Playtnng Funch Rope Climbling
Saccer Purrely
Swing Tenno Soetnn solley spiking writing OR Board 1oto
Roor Gymnur front
Figure 4: Visualization of the Top-1 accuracy for each video category sampled from UCF-101 with respect to the merged expert and each individual expert.
Related Work
Video recognition. In the era deep learning, early works explored diverse variants convolu- tional networks for joint spatiotemporal modeling, such 3D convolution [2, 39] and factorized spatial and temporal convolution [40]. The advent of Transformer then attracted the attention of
researchers and has been widely applied for video recognition [1, 6J. In addition, self-supervised learning methods learn transferable representations from large-scale unlabeled data and achieve promising performances [7, 57].
Transferring VLMs for videos. Transferring VLMs for video recognition task has been proven to be effective. ViFi-CLIP [34] suggests that direct fine-tuning process generalizes well on various settings. Open-VCLIP [46] constructs an open-voeeee vocabulary video model by interpolating the model weights and its optimization trajectory. Vita-CLIP [45] extracts discriminative information using Iulti-level prompts. X-CLIP [28] proposes cross-frame attention and multi-frame frame integration modules for temporal modeling. Text4Vis [49] and BIKE [50] explore the way for more effective knowledge transfer and use multi-transformer layers to capture temporal cues. FROSTER [12] mitigates catastrophic forgetting by ensuring the learned features do not diverge too far from the frozen ones through distillation. This divergence comes from the variations of both CLIP and additional trainable parameters. Differently, we believe the main cause of the forgetting problem is the overfitting of additional parameters regardless of whether the CLIP parameters are tuned. Note that FROSTER can also potentially improve the generalization of additional parameters through knowledge distillation, but our method presents a more explicit way to achieve this goal. Existing methods face trade-off of introducing knowledge. Our work addresses this challenge and presents superior results on both close-set and zero-shot settings.
specialized knowledge preserving
generalized
The sparsely activated MoE structure [37] enables the model capacity (i.e. number of parameters) to vastly scaled up while keeping the computation cost per sample basically unchanged. This technique has been widely investigated in building large-scale pre-trained models in various fields [20, 26, 35). Several works employ MoE on large-scale language models for parameter- efficient tuning to improve their specialized performance [8, 44J. In this paper, we demonstrate its effectiveness in balancing generalization and specialization in VLM knowledge transfer.
Mixture-of-Eppereee
Conclusion
In this paper, we present MoTE, an effective Visual- Visual-Language to video knowledge transfer framework that enjoys both superior generalization and specialization. MoTE leverages mixture of temporal ex- perts to enhance performance in both close-set and zero-shot video recognition, all while maintaining conventional temporal module's structure and computational efficiency. With the proposed weight merging regularization and temporal feature modulation, we achieve the coexistence of generalization and specialization one unified model. Extensive experiments validate MoTE's ability to strike an optimal trade-off between close-set and zero-shot performance.
10

---

Acknowledgements
This paper is supported by the National Natural Science Foundation China under Grants (62073245, 621 62233013), Shanghai Municipal Science and Technology Major Project (2021 SHZDZX0100) and Innovation Action Plan (22511104900), the Fundamental Research Funds for the Central Universities.
References
[1] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid. Vivit; video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6836-68888 6846, 2021.
[2] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6299-6308, 2017.
[3] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv: 1808.01340, 2018.
[4] Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[5] Nicola De Cao, Wilker Aziz, and Ivan Titov, Editing factual knowledge in language models. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6491-6506, 2021.
[6] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6824-6833 6235., 2021.
[7] Christoph Fcichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming Hc. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), pages 3299-3309, 2021.
[8] Ze-Feng Gao, Peiyu Liu, Wayne Xin Zhao, Zhong-Yo Yi Lu, Ji-Rong Wen. Parameter-efficient mixture-of- experts architecture for pre- trained language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1263-3273, 2022.
[9] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-valuc memories. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
[10] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017.
[11] Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations (ICLR), 2022.
[12] Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. Froster: Frozen clip is strong teacher for open- vocabulary action recognition. In International Conference Learning Representations (ICLR), 2024.
[13] Chao Jia, Yinfei Yang. Xia, Yi- Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun Hsuan Sung, Zhen Li. and Tom Duerig. Scaling up visual and rision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICMI.), pages 4904-4916, 2021.
Yun-Hsuan
[14] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual- langual models for efficient video understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 105-124, 2022.
[15] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vjayanarasimhan. Fabio Viola, Tim Green Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 17X.v:1705.06955 2017.

---

[16] Guolin Ke. Qi Meng. Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
[17] Nitish Shirish Keskar, Dhcevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations (ICLR), 2016.
[18] Simon Korbblith, Ting Chen, Honglak Lec, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[19] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In Proceedings of the IEEE/CVF International Conference Compuder Vision (ICCV), pages 2556-2563, 2011.
[20] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[21] Boyi Li, Kilian Q Weinberger, Serge Belongic, Vladien Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations (ICLR), 2021.
[22] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
[23] Wei Lin, Leonid Karlinsky, Nina Shvelsova, Horst Possegger, Mateusz Kozinski, Rameswar Fanda, Rogerio Feris, Hilde Kuehne, and Horst Bischof. Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[24] Xiao Lin, Minghao Zhu, Ronghao Dang, Guangliang Zhou, Shaolong Shu, Feng Lin, Chengju Liu, and Qijun Chen. Clipose: Category-level object pose estimation with pre-trained vision-language knowledge. IEEE Transactions on Circuits and Systems for Video Technology. 2024.
[25] Ilya Loshchilov and Frank Hutter. 17 2017.
Decoupled
weight decay regularization. arXiv preprint
[26] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. In Advances in Neural Information Processing Systems (NeurlPS), pages 064-9576,,2 2022.
[27] Behnam Neyshahur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep lcarning. In Advances in Neural Information Processing Systems (NeurlPS), 2017.
[28] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xi- ang. and Haibin Ling. Expanding language-image -inaggee--ageeeeee pretrained models for general video recognition. In Proceedings of the European Conference Computer Vision (ECCV), pages 1-18, 2022.
[29] Changdae Oh, Mijoo Kim, Hyesu Lim, Junhyeok Park, Euiseog Jeong, Zhi-Qi Cheng, and Kyungwoo Song. Towards calibrated robust fine-tuning of vision-language models. arXiv preprint urXiv:2311.01723. 2023.
[30] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image- to-video transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 2022.
[31] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[32] Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Yingya Zhang, Changxin Gao, Deli Zhao, and Nong Sang. Disentangling spatial and temporal learning for efficient image-to-video transfer lcarning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 7748-876.
2021.
12

---

[34] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz. Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 6545-6554, 2023.
[35] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems (NeurlPS), pages 8583 8595, 2021.
[36] Mert Bulent Sariyildiz, Yannis Kalantidis, Karteek Alahari, and Diane Larlus. No reason for no supervision: Improved generalization in supervised models. In International Conference on Learning Representations (ICLR), 2022.
[37] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint 1rXivv1701.06538, 2017.
[38] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 12Xiv:122.00400 2012.
[39] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4489 4490-444772200 2015.
[40] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 6450-6459, 2018.
[41] Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, and Yu-Gang Jiang. Implicit temporal modeling with learnable alignment for video recognition. In Proceedingy of the IEEE/CVF International Conference Computer Vision (ICCV), 2023.
[42] Mengmeng Wang. Jiazheng Xing, and Yong Liu. Actionelip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021.
[43] Qiang Wang, Junlong Du, Ke Yan, and Shouhong Ding. Seeing in flowing: Adapting clip for action recognition with motion prompts learning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5339-53477 2023.
[44] Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan, and Jianfeng Gao. Adamix: Mixture-of adapp-tttiottssi for parameter-efficient model tuning. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3744-5760, 2022.
[45] Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz. Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 23034-23044, 2023.
[46] Zejia Weng. Xitong Yang. Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-vclip: an open-vocabulary video model via interpolated weight optimization. In International Conference on Machine Learning (ICML), pages 15978-36989 2023.
Transforming clip to
[47] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphael Gontijo-Lopes, Ari Morcos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), pages 23965-23998, 2022.
[48] Lei Wu, Zhanxing Zhu, et al. Towards understanding generalization of deep learnigg: Perspective of loss landscapes. arXiv preprint arXiv: 1706. 10239, 2017.
[49] Wenhao Wu, Zhun Sun, and Wanli Ouyang. Revisiting classifier: Transferring vision-language models for video recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2847-2855, 2023.
[50] Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. Bidirectional cross modal knowledge exploration for video recognition with pre- trained vision- languaee models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6620- 6630, 2023.
13

---

[51] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open- vocabuaayy detection with region prompting and anchor pre-matching. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 7031-7040, 2023.
[52] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang- Groupvit: Semantic segmentation emerges from text supervision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 18134-18144 1814,, 2022.
[53] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Li. Aim: Adapting image models for efficient video action recognition. In International Conference on Learning Representations (ICLR), 2022.
[54] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint 21111: 1432, 2021.
[55] Renrui Zhang, Rongyao Fang. Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[56] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision lisnuallagguage models. International Journal of Computer Vision (ICCV), 130(9):2337 2348, 2022.
[57] Minghao Zhu, Xiao Lin, Ronghao Dang, Chengju Liu, and Qijun Chen. Fine- fine-grained spatiotemporal motion alignment for contrastive video representation learning. In Proceedings of the 31 ACM International Conference on Multimedia, pages 4725 4736, 2023.
[58] Yan Zhu, Junbao Zhuo, Bin Ma, Jiajia Geng. Xiaoming Wei, Xiaolin Wei, and Shuhui Wang. Orthogonal temporal interpolation for zero-shot video recognition. Proceedings of the 31st ACM International Conference on Multimedia, pages 7491-7501, 2023.
[59] Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. Taming spansely activated transformer with stochastic experts. In International Conference on Learning Representations (ICLR), 2021.
14

---

Supplemental material
A Limitation and Broader Impact
Limitation and future work While method yields superior results in various settings of video recognition, there are still several aspects for improvement. First, the text space of MoTE is limited to video category names, which do not provide discriminative and semantically rich textual representations. Thus, exploring how to extend the semantic space with large-scale generative models may further enhance the performance of MoTE. Besides, additional parameters take various forms when adapting VLM to downstream tasks. In future work, we will further extend our approach to other forms of parameters to explore the generality and versatility of our method.
Broader Impact Adapting foundation models to downstream tasks has become dominant paradigm in machine learning [24, 30J, we believe that it is worthwhile and important to explore how to preserve the existing knowledge of the foundation model when introducing We hope this work brings new insights for the broader and long-term adaptation of foundation models. Besides, our work focuses video recognition tasks, which have wide range of real- wollworld application scenarios, such as surveillance. However, this work requires careful elimination of privacy and rights concerns before real-world deployments.
knowledge.
Table 6: Hyper-parameter details during fine-tuning.
Value
Optimization details
Batch size
144
Optimizer
Adamm
Weight decay
0.2 0.9, 0.999
Adam $1.92
Learning rate (Base)
Se-5
Learning rate (CLIP layers)
3e-6
Learning rate decay
Cosine schedule 30 (ViT-B), 20 (ViT-L) 5
Training epochs
Linear warm-up epochs
Augmentation
RandomResizeiCCop
Area
[0.08, 1.00]
Aspect ratio
[3/4 4/3]
Crop size
224
Random Horizontal Flip
0.5
Random Gray scale
0.2
B More Implementation Details
Close-set and zero-shot video recognition. In Table 6, present the hyper-parameters set for optimization. Note that we train one model for both close-set and zero-shot tasks, rather than training for each task separately. The parameters of all projection matrices in MoTE are initialized with values drawn from the normal distribution (mean=0, std=0.02). All bias terms are initialized as zero. We conduct experiments with 3 NVIDIA GeForce RTX 4090.
For zero-shot evaluation, the methods are evaluated on three official splits of UCF-101 and HMDB-51. For Kinetics-60l -600, adopt the three splits provided by [4]. Each split contains 160 categories out of 220 new categories that do not exist in K400. We report the average Top-1 1 accuracy and the standard deviation three splits.
Few-shot video recognition. We consider the standard K-shot setting [28, 34], where 2, 4. 8, and 16 video data are randomly sampled for each category for constructing the training dataset. Following previous work [49], we repeat the constructed training dataset to maintain the same size as the
15

---

Pr
Pr dh
dn
Pr on
Pri
Pr dr
Pri dn
Rosjer
up
Bouter
FFN
Pp Pri dn
Layer Nor
Solf-Atention
Route
Standard Transformer Layer
(c) Prj_up
(d) Prj dn
Figure 5: Illustration of optional architecture designs for the temporal expert. We omit the activation function between the projection matrices for brevity.
original full dataset. In few-shot settings, fine-tune models for 2 epochs SSv2 and 10 epochs on UCF-101 and HMDB-51. We adopt a single view for evaluation.
Table 7: Additional ablation studies. Default settings are colored in gray.
(a) Optional designs for the tempo-
(b) Varying neighbor numbers K for (c) Different types of Weight Merg- the temporal feature modulation.
ral expert.
ing Regularization.
Experts
K400
UCF
K600
K
K400
UCF
K600
Types
K400
UCF
K600
Pqi BillupPPP_dd dn
86.7
84.2
74.1
86.8
88.9
78.9
LwMn
86.8
87,5
78,9
Puj_up
83,3
73.6
86.8
LWMR KL
dn
86.3
83.1
73.2
10
86.8
88.5
78.7
AWMR MSE
86.3
86.2
76.3
FFN
87.5
78.9
(d) Ablation study on the training
(e) Results of fine-tuning on UCF (l) Effect of temperature selection and zero-shot evaluating K400.
costs of MoTE.
schemes.
Method
GPU-days GFLOPs
Method
UCF
K400
Type
K400
K600
Baseline
4.35
649
CLIP
80.9
59.2
Discrete set
86.8
78.9
Temppral Experts
4.35
+0.34
Baseline
95.6
56.6
Continuous normal dist.
86.5
77.4
LWMR
4.50
+0.34
MoTE
96.1
66.3
Contimous uniform dist.
86.0
77.9
LMSE:
C Additional Ablations
Optional architecture of temporal expert. By default, replace the whole FFN with experts of the same structure as the FFN. Optionally, we can replace one or both of the projection matrices the FFN with experts of the same structure, as illustrated in Figure We study the optional architecture design of the temporal expert in Table 7a. As shown in the table, we observe degraded performance with projection-level expert designs. We argue that the shared projection matrix or two-stage routing design of the projection-leve expert allows knowledge communication between experts, such implicit communication may lead mutual fitting between the experts. Therefore, preventing the knowledge exchange across experts is critical for learning diverse generalized knowledge.
Varying neighbor numbers for Temporal Feature Modulation. Section 3.4 of the main manuscript, we generate fine-tuning proxy features f and test proxy features by retrieving the K nearest text features in the fine-tuning and the test dataset categories. We study the influence of various K in Table 7b. We find that large K leads to slight performance degradation as the retrieved text features may not represent the semantic information of the video properly. Overall, our performance gains are stable for this parameter.
Various Loss Types of Weight Merging Regularization. Section 3.3 the main manuscript, we design the weight merging regularization LWMR based on the cross-entropy loss function. In this study, we use other loss function types to redesign weight merging regularization LWMR- Note that our training procedure requires two forward passes of the temporal module, one when
16

---

RGB
Merged Expert
Expert
Expert 4
RGB
Merged Expert
Expert 0
Expert 4
"Hugging baby
"Longboarding"
"Photobombing"
"Tying shoe
"Preparing salad"
"Playing maracas"
"Playing beer pong""
*Putting eyeliner"
Figure 6: Visualization of attention maps. We show the RGB image and the attention maps of the merged expert, expert O, and expert_4. 4.
optimizing the activated expert with LTE and the second in calculating weight merging regularization Lwmn- When optimize LWMR using cross entropy.
the ground truth label of the video as the supervised signal. We try to redesign LWMR with other supervisions.
(i) KI. Divergence: We adopt the logits generated during the computation of Le the supervision and optimize with KL divergence loss, denoted as LWMR_K, KL- (ii) Mean Square Error: We use the output feature when computing LTe as the supervision and optimizing with MSE loss, denoted LWMR MSE. The results are presented in Table 7c, Interestingly, we find that the MSE loss achieves notable results in weakly supervised manner, which indicates the potential scalability of our proposed approach.
WEMMRMMEE-
Training cost analysis of MoTE We report the actual training time of our method with respect to the baseline in Table 7d. The wall-clock time of training is benchmarked on 4090 GPUs with batch size of 144. GPU days are calculated by the number of GPUs multiplied by the training time in days. As shown in the table, applying the mixture of temporal experts does not introduce additional training overhead over baseline, which can be attributed to the use of the routing strategy. Adding LWMR and LMSE brings a +0.16 days training time increase since it requires an additional forward pass of the temporal module.
Transferring CLIP with small-scale fine-tuning data. In the main text, our experimental setting follows previous works in fine-tuning on large-scale K400 and then evaluating zero-shot performance on relatively small downstream datasets. To further investigate MoTE's capabilities when the fine- tuning data are limited, we train the ViT-L/14 network UCF-101 and evaluate it on K400, results are shown Table MoTE yields a 0.5 improvement over the Baseline on UCF-101, while zero- shot performance on K400 significantly outperforms both raw CLIP and Baseline. This demonstrates the applicability of MoTE to small-scale datasets and ability to learn generalized knowledge from limited data.
Effect of temperature selection schemes. In this study, we compare our discrete temperature sampling strategy with the continuous space selection schemes. We implement two continuous space selection schemes. (1) Sampling from a continuous standard normal distribution (mean=0, variance=1). (2) Sampling from a continuous uniform distribution. As shown in Table 7f, the continuous space sampling strategy results in a notable performance degradation.
D Textual Prompt Templates
In our work, we adopt a set of hand-craft textual prompt templates to generate text embeddings. Following CLIP [33], we perform prompt ensembling over the 28 templates order to provide comprehensive semantics. The templates are listed in Table 8.
17

---

Table Textual prompt templates of MoTE.
Templates
"a photo of (category)."
photo of person [category): ).'
' photo of a person using [category] )
"a photo of a person doing [category J.
"a photo of person during (category):
photo of person performing (category):
'a photo of person practicing (category):
video of [category).
aa video of person {category)."
a video of person using {category].
a video of person doing (category):
a video of a person during (category]: ).
a video of person performing (category).
a video of a person practicing (category).
example of (category)."
aa example of a person (category):
'a example of person using {category): ).
a example of a person doing [category ).¹
example of person during {category).
a example of a person performing [category)."
example person practicing (category). a demonstration of {category)."
demonstration of person [category J.
a demonstration of a person using [category]] J.¹
demonstration of a person doing (category):
a demonstration of a person during [category]: )
demonstration of person performing [category]:' aa demonstration of a person practicing [category).
E
Qualitative Analysis
E.1
Model Attention
To better understand what knowledge the experts capture, we present the visualization of the model attention in Figure 6. We show the attention map of the merged expert, expert_ O, and expert 4 to explore whether diverse knowledge is learned across experts. Since MoTE only requires the input of frame-level tokens from the CLIP visual encoder, we calculate the attention map between the temporal video features output by MoTE and the image patch tokens from the CLIP encoder. As shown in the figure, experts 0 and experts_ 4 always focus different regions, indicating that they capture various temporal patterns. Besides, we observe that the merged expert is able to focus on more precise and broader foreground areas. This phenomenon suggests that the merged expert sufficiently aggregates and leverages the knowledge from different experts.
E.2 Representation similarity across experts.
We visualize the representation similarities across each expert and the final merged model in Figure 7. The representation similarities are averaged on 100 randomly sampled data of unseen categories from the K600 dataset. As show in the affinity map, the different similarities demonstrate that each expert learns distinctive knowledge from different optimization trajectories. Besides, we observe that the similarities between the merged model and each expert are relatively stable, indicating that the merged expert efficiently leverages the knowledge contained in each expert.
18

---

1.0
Expert 3
0.9
Expert_ 2
0.8
Expert_ 1
0.7
Expert_0
-0.6
Merged Expert
0.5
Merged Expert
Expert 0
Expert_* 1
Expert_2 2
Expert_3
Figure 7: Visualization of representation similarities across each expert and the final merged model.
Dataset Details
large-scale dataset in the video domain. The dataset contains ~400 training videos and 20k validation videos 400 human action categories, with an average length of 10 seconds. The high quality of the dataset makes it the most popular benchmark for video recognition
Kinetics-400 [15]
Kinetics-600 [3] is an extension of Kinetics-400, consisting of ~392k training videos. ~30k validation videos, and ~600 test videos in 600 human action categories. The dataset contains an additional 220 new action categories over Kinetics-40k We evaluate the zero-shot performance on 220 new categories and adopt three splits provided by the previous work [4]. We use its test set for evaluation and report the average performance three splits.
UCF-101 [38] is action recognition dataset that contains 13,320 videos in 101 action categories, collected from YouTube. There are three official splits of training data and validation data.
HMDB-51 [19] contains 7k videos in 51 action categories, collected from movie clips and web videos. There are three official splits of the dataset, each with 3,570 training data and 1,530 validation data. collection of realistic videos from various sources, including movies and web videos. The dataset comprises 7,000 video clips from 51 action categories.
Somethin -Srreehhnn-Smtthigg V2 [10] a temporal-heavy dataset that requires the fine-grained temporal understanding capability of the model. It contains 220,000 videos in 1 174 action categories.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contributions are detailed in lines 6-14 of the abstract and lines 67-77 of the introduction.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We analyze the limitations of this paper Section A of the Appendix.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: As an applied paper, we aim to provide more generalized and effective VLMs knowledge transfer framework for video recognition tasks. We do not include theoretical results in this paper. Note that all assumptions and conclusions mentioned in this paper are precisely cited.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detail the implementation details of the paper in Section 4. of the main manuscript and Section B of the Appendix. Besides, we provide our code in the supplemen- tary zip file. The code will also released when the paper accepted.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide our code in the supplementary zip file with a detailed instruction to reproduce our method. The code will also be released when the paper is accepted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all training and evaluation details of our method in Section 4. of the main manuscript and Section of the Appendix. We also provide statisties details for the dataset used in this paper in Section F of the Appendix.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
20

---

Justification: For zero-shot experiments, following the previous works, we evaluate our method on three splits of UCF-101, HMDB-51, and K600. We report the mean accuracy and the standard deviation of performance of the three splits on each dataset, respectively. As for close-set and few-shot experiments, considering the computational effort, we do not include error bars following previous works.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We report computational resources and training costs in Section B and Section C. of the Appendix.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https /ttp//uurrrirre. http::///oorrr...ccoooa///////aa
Answer: [Yes]
Justification: All authors have read the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential broader impacts in Section A of the Appendix.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All references and datasets in this paper are precisely cited and licensed with corresponding licenses.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide detailed instructions on submitted assets. The details of the dataset and model are reported in our paper.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
21

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper poses such risks.
22


Reviews:
[{"review_id": "ri6UHLEETR", "reviewer": "Reviewer_KXzA", "summary": "To preserve the generalization ability of the model trained on general visual-language model (VLM) with task-specific data, while boost the performance on specific task, this paper propose a new framework and training strategy to learn a unified model with specific performance and generalization ability. Three techniques are introduced. Mixture temporal experts to avoid overfitting on the task-specific data. A weight merging regularization to enlarge the loss flat region such that optimization on generalization ability will not introduce perturbation that drops the close-set performance. A temporal feature modulation to reuse the feature of VLM model when the target category label is not fitted during task-specific finetuning. The proposed method is evaluated on four benchmark datasets. K400 for close-set finetuning and UCF-101, HMDB-51and K600 for zero-shot evaluation.", "questions": "1.\tThe second question in the weakness section\n2.\tThe VLM Clip is actually trained on noisy data, and there are also VLM trained with selected data to boost its cross-modality alignment [1]. Therefore, the selection of K in line189 may have influence on the final performance. Besides, for different text-query, the influence of noisy data is diffrerent, and one fixed K may not be optimal. Is there any solution for this issue. Is the selection of K have large influence on the performance?\n[1]  Bulat, Adrian, Yassine Ouali, and Georgios Tzimiropoulos. \"FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "limitations": "The limitation has been discussed in the suplemental material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tTo train a model with both task-specific performance and zero-shot generalization ability is a interesting topic, and it is less explored in the community. \n2.\tThe proposed method achieves competitive performance compared with the similar methods.\n3.\tBalancing between the zero-shot and the task-specific ability is always hard to handle. Considering the wide application of general VLM, this method bears practical value in the industry.", "weaknesses": "1.\tThe experimental setting may hide the weakness of the proposed method. The method is only trained on K400 and evaluated its zero-shot ability on UCF-101, HMDB-51and K600. Considering K400 is already a large-scale dataset, the MoTE may still have good performance on UCF-101 and HMDB-51. Besides, K600 is an extension of K400, therefore they may have similar data distribution. It would be great to also finetune the model on small-scale dataset and evaluated generalization ability on large-scale dataset, for example, train the model on UCF-101 and evaluate it on K400.\n2.\tA simple solution to handle the zero-shot / task-specific balancing issue is to use a finetuned model such as Text4Vis for specific task and to use its temporally mean-pooled clip feature when facing out-of-distribution task. This baseline is missing in the comparison. If the performance of this baseline is acceptable, is it really necessary to train a unified model with such much cost?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and the time you have dedicated to this paper. Please find the responses to your comments below.\n***\n**1. The experimental setting may hide the weakness of the proposed method. It would be great to also fine-tune the model on the small-scale UCF-101 and evaluate it on K400.**\n\nThanks for your thoughtful and reasonable suggestion! Our experimental setting follows previous works (Methods listed in Table 3) in fine-tuning on large-scale K400 and then evaluating zero-shot performance on relatively small downstream datasets. Following your suggestion, we train the ViT-L/14 network on UCF-101 and evaluate it on K400, results are shown in the table below.\n\n|Method|UCF (close-set)|K400 (zero-shot)|\n|-|:-:|:-:|\n|Raw CLIP|80.9|59.2|\n|Baseline|95.6|56.6|\n|MoTE|96.1|66.3|\n\nMoTE yields a 0.5 improvement over the Baseline on UCF-101, while zero-shot performance on K400 significantly outperforms both Raw CLIP and Baseline. This demonstrates the applicability of MoTE to small-scale datasets and its ability to learn generalized knowledge from limited data.\n***\n**2. Comparison with the baseline variant. Is it really necessary to train a unified model with such much cost?**\n\nWe adopt the recommended baseline and present the results in the table below. Our method still shows notable advantages in zero-shot performance.\n\n|Method|K400 (close-set)|UCF|HMDB|K600|\n|-|:-:|:-:|:-:|:-:|\n|Baseline Variant|86.7|87.1|58.2|77.4|\n|MoTE|86.8|88.7|61.4|79.0|\n\nWe would like to claim that the most significant value of our work is its ability to simultaneously introduce new specialized knowledge while keeping original generalized knowledge, rather than solely achieve the best performance on a particular dataset. Although the recommended baseline performs fairly on close-set and zero-shot tasks, it does not really achieve our goal. This can be evidenced when facing a middle-ground task between close-set and zero-shot tasks, for example, the few-shot task. It requires both specialization and generalization capability to rapidly adapt using limited samples. We present the results of the few-shot task in the table below. Our method significantly outperforms the recommended baseline.\n\n|Method|UCF-2 shot|UCF-4 shot|UCF-8 shot|UCF-16 shot|HMDB-2 shot|HMDB-4 shot|HMDB-8 shot|HMDB-16 shot|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Baseline (clip feature)|78.6|81.6|82.3|83.0|53.7|54.2|55.2|55.8|\n|Baseline (temporal feature)|86.4|88.9|90.6|91.2|55.9|61.2|65.4|66.3|\n|MoTE|88.8|91.0|92.3|93.6|61.3|63.9|67.2|68.2|\n\nFrom the perspective of model structure, the recommended baseline can only be applied when the additional parameters are separated from the CLIP encoder. Our proposed method can be applied to various forms of additional parameters (e.g. adapter) that are integrated into the CLIP encoder, demonstrating the versatility of our method. The details of this part please refer to the response of the second question to the reviewer s5Zs. Moreover, in real-world applications, the data distribution shift is often hard to identify. The recommended baseline faces the choice of whether to use the temporal feature when facing a moderate or unknown distribution shift, which limits its practicality. This further demonstrates the necessity of training a unified model.\n\n***\n**3. For different text-query, the influence of noisy data is diffrerent, and one fixed K may not be optimal. Is there any solution for this issue. Is the selection of K have large influence on the performance?**\n\nThanks for your interesting question! We agree with your opinion and conduct an experiment. In this experiment, we employ the K-means algorithm to cluster the fine-tuning and test category text features and compute the semantic association by retrieving the most similar cluster-centered feature. In this case, the retrieved cluster-centered feature may represent different numbers of data points. We hope this strategy can better represent the semantic associations between fine-tuning and test categories. We also test the effect of the number of K. The results are shown in the table below (TFM indicates the Temporal Feature Modulation).\n\n|Method|TFM|UCF|K600|\n|-|:-:|:-:|:-:|\n|MoTE|-|87.5|78.9|\n|MoTE (clustering)|√|88.8|78.9|\n|MoTE (K=1)|√|88.9|78.9|\n|MoTE (K=5)(default)|√|88.7|79.0|\n|MoTE (K=10)|√|88.5|78.7|\n\nWe find that applying the clustering algorithm does not lead to performance improvement. The reason lies in that using the semantic association to measure the confidence of the temporal feature is actually a rough estimation process. Since this process is inherently somewhat noisy, better-selected data may not largely affect it. This also explains why the selection of the K-value does not have a significant influence on the performance. However, we still observe performance degradation when K is too large, indicating that the noise introduced from the selection of the K dose has a negative influence on the performance.\n***\nWe sincerely hope that this response will address the reviewers' concerns. We will supply the above responses in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fwN1fUtu6Z", "reviewer": "Reviewer_s5Zs", "summary": "This paper introduces MoTE (Mixture-of-Temporal-Experts) to improve the generalization and specialization capabilities of visual-language models (VLMs) when adapting to video tasks. MoTE addresses two main questions: how to enhance the generalization of additional parameters during fine-tuning, and how to balance generalization and specialization in a unified model. The approach uses multiple feedforward network (FFN) experts in each Transformer layer to capture various data bias views, improving generalization. A routing algorithm based on multinomial distribution maximizes knowledge diversity among experts, while Weight Merging Regularization effectively combines generalized and specialized knowledge in the final model.\n\nTo further improve generalization at test time, MoTE incorporates a Temporal Feature Modulation module. Notably, the approach maintains the same computational cost and final structure as conventional methods. The paper contributes to the field by offering a new perspective on enhancing parameter generalization and balancing it with specialization in the context of adapting VLMs to video tasks. Extensive experiments demonstrate that MoTE achieves an optimal trade-off between zero-shot and close-set performance, with thorough ablation studies showing the scalability and effectiveness of the proposed method.", "questions": "Please refer to the weaknesses.", "limitations": "The authors have not sufficiently addressed the limitations of their methodology, as it has been applied exclusively to a specific type of adapted network without demonstrating broader applicability. It would be advantageous to see an exploration of the method’s versatility across different network architectures.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The manuscript is well-written and easy to follow.\n\n- It is interesting to observe that the introduction of a mixture of experts can enhance the balance between acquiring generalizable knowledge and learning video-specific features. The motivation is intuitive, and the extensive experiments effectively validate the method’s efficacy.\n\n- The design of weight merging regularization and temporal feature modulation harmonizes the pursuit of the two learning objectives. The temporal feature modulation is particularly noteworthy, as it takes into account the categorical relationships between the training and test sets to inform the integration of features.", "weaknesses": "- The primary motivation for this study stems from two objectives: (1) mitigating the catastrophic forgetting that emerges with the integration of trainable parameters, and (2) striking a balance between generalizable knowledge and video-specific learning within one single model. However, these objectives bear considerable resemblance to the work presented in the paper FROSTER (ICLR 2024), which has not been discussed by the authors. While I acknowledge that the current paper and FROSTER employ distinct methodologies to address these issues, their close relevance necessitates a thorough discussion and a direct performance comparison.\n\n- According to the description in the paper, the baseline model utilizes a clip encoder equipped with several temporal transformer layers. This leads me to question whether the model can be effectively integrated with alternative network architectures, such as adapter-based networks, X-CLIP, and ST-adapter, particularly given their noted efficiency in training.\n\n- I would also request that the authors provide details regarding the additional computational and training time costs associated with implementing their method in conjunction with the baseline model.\n\n- I believe it would be beneficial to delve deeper into the specific types of actions that each expert excels at recognizing. Providing a more detailed analysis in this area would enhance our comprehension of the distinct roles played by various experts, as well as the unique temporal knowledge they contribute in comparison to one another.\n\n[1] FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and the time you have dedicated to this paper. Please find the responses to your comments below.\n***\n**1. Discussion on FROSTER.**\n\nThanks for your kind reminder! Our motivation does resemble FROSTER's in some way but differs in the following aspects.\n\n*Motivation:*\n\n* FROSTER aims to mitigate the catastrophic forgetting caused by **fine-tuning the model**, while our method focuses on eliminating the catastrophic forgetting caused by **the integration of the additional trainable parameters**. The two motivations stem from different observations and perspectives.\n\n* FROSTER aims to strike a balance between generalized knowledge and video-specific learning **specifically for zero-shot video recognition**, while our work pursues this objective **in more general and various task settings** (close-set, zero-shot, and few-shot recognitions). Since the close-set setting requires much more video-specialized knowledge than the zero-shot setting, our objective is more challenging and practical.\n\n*Contribution:*\n\n* Our method and FROSTER address the objective with different methodologies. FROSTER focuses on knowledge distillation from the frozen CLIP, while our method concentrates on the model structure design and loss landscape regularization. We offer distinct contributions to the community.\n\nWe will add the above discussion and performance comparison in the revised manuscript.\n***\n**2. Whether the model can be effectively integrated with alternative network architectures, such as adapter-based networks?**\n\nThanks for your constructive suggestion! To further demonstrate the generality and scalability of MoTE, we apply our method to ST-Adapter [1]. For simplicity, we remove the depth-wise Conv3D operation in our adapter. In this case, our baseline is slightly lower than ST-Adapter, but it doesn't affect justifying the effectiveness of our method. The results are shown in the table below.\n\n|Method|K400 (close-set)|UCF|K600|\n|-|:-:|:-:|:-:|\n|ST-Adapter|81.1|77.3|61.1|\n|ST-Adapter+MoTE|81.2|79.4|66.7|\n\nOur method delivers a notable generalization performance improvement while maintaining the close-set result, indicating the effectiveness of MoTE applied to alternative networks.\n\n[1] ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning. NeurIPS 2022.\n***\n**3. Additional computational and training time costs associated with implementing the method in conjunction with the baseline model.**\n\nThanks for your comment! We show the training time costs in Table 8 of the supplementary material. Below, we include the table for your convenience and additionally add GFLOPs for reference. GPU days are calculated by the number of GPUs multiplied by the training time in days.\n\n|Method|GPU-days|GFLOPs|\n|-|:-:|:-:|\n|Baseline|4.35|649|\n|+MoTE|4.35|+0.34|\n|+$L_{WMR}$|4.50|+0.34|\n|+$L_{MSE}$|4.51|-|\n\nAs shown in the table, applying our method on the baseline introduces slight computational and training time costs, illustrating the efficiency of our method.\n***\n**4. More detailed analysis of the specific action types that each expert excels at recognizing.**\n\nThanks for your constructive suggestion! We perform a category-wise performance analysis of the merged experts and each individual expert. We visualize the Top-1 classification accuracy for each video category, which is sampled from the UCF-101 dataset. The experiment is conducted on UCF-101 under the zero-shot setting. **The visualization is provided in the attached global response PDF.**\n\nWe find that different experts exhibit distinct performance for each video category. This is because diverse generalization knowledge is learned across experts, as we demonstrate in the paper. The merged experts benefit from the aggregation of knowledge across experts, especially for some video categories where temporal information is particularly needed, for example, 'Body Weight Squats', 'Handstand Pushups', and 'Wall Pushups'. This suggests that each expert can learn various temporal patterns for the same category. For more visualizations, please refer to Figure 5 and Figure 6 in the supplementary material, which shows more intuitive visualizations.\n\n***\nWe sincerely hope that this response will address the reviewers' concerns. We will supply the above responses in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UESTGeMUK2", "reviewer": "Reviewer_7UgE", "summary": "This paper addresses the issue of Video-Language Models (VLMs), such as CLIP, experiencing reduced generalization performance to unseen categories when learning domain-specific knowledge for video understanding tasks. The authors propose the MoTE framework, which introduces temporal experts and employs a Mixture of Experts (MoE) approach to effectively learn domain-specific knowledge for videos. Additionally, a soft stochastic routing policy is utilized to further enhance the learning efficiency of the experts. To guarantee the discrepancy in knowledge learned by different experts while maintaining a flat loss landscape, the paper incorporates weight merging regularization, which improves the generalization performance of the learned features. Moreover, the paper presents a temporal feature modulation method that leverages the semantic relevance confidence of proxy text features to modulate features.", "questions": "1. What is the design basis for the candidate set of the temperature hyperparameter in weight merging? The paper does not provide a reference for the design of this candidate set, nor does it further validate its superiority over continuous space selection schemes in the experimental analysis.\n\n2. What is the connection between the modulation method proposed in Section 3.4 and the paper's overall motivation? The issue of constrained semantic space it addresses does not seem to be related to the MoE method or the maintenance of feature generalization.\n\n3. What is the specific idea behind the trade-off metric mentioned in Section 4.3? Considering the balance between the two, the arithmetic mean does not seem to be a good metric. If a model achieves 100% ZS performance but 0% close-set performance, its trade-off metric result would be the same as if both values were 50%. How is this issue addressed?", "limitations": "The paper does not adequately explain the connection of data bias view and MoE in Section 3.2. For reading-friendly, there should be additional descriptions of the relationship between experts and data bias views, and how the MoE approach leverages multiple data bias views to improve model performance.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces the Mixture of Experts (MoE) approach in zero-shot video classification tasks based on Video-Language Models (VLMs). By utilizing weight merging regularization and other methods, the approach ensures effective learning of domain-specific knowledge in videos while maintaining strong model generalization.\n\n2. The study effectively combines temporal modeling of visual content with the MoE approach. During downstream task adaptation, it leverages multi-perspective data bias learning to avoid overfitting, thus enhancing the learning effectiveness of domain-specific knowledge in videos.\n\n3. The paper analyzes model generalization from the perspective of loss landscape flatness. By improving the flatness, weight merging regularization enhances the generalization performance of the learned features.", "weaknesses": "1. There is ambiguity in the use of certain symbols within the paper. For example, the symbol L is used to represent both the loss function of CLIP and the number of layers in the Transformer introduced in MoTE. This issue is particularly evident in Equations (4) and (7). The paper should consider adjusting the usage of these symbols to avoid confusion.\n\n2. There seems to be a problem with the calculation in Equation (5). The notation \"exp\" typically represents the exponential function of e, but this is not clearly explained. According to the equation, the probability of selecting an expert increases with i, which seems to contradict the intended randomness of stochastic. This requires clarification or correction.\n\n3. In the Introduction and Section 3.4, the paper emphasizes the plug-and-play characteristic of the modulation module. However, the subsequent experiments only demonstrate the improvement in model performance without introducing additional training parameters (Play). They do not showcase the flexibility and usability of the module regardless of the upper model structure (Plug). Therefore, it would be beneficial to add experiments validating the plug-and-play effect or adjust the relevant descriptions in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and the time you have dedicated to this paper. Please find the responses to your comments below.\n***\n**1. Ambiguity in the use of certain symbols.**\n\nThanks for your kind reminder, and sorry for the confusion. We double-checked the usage of all symbols and will correct the mistakes in the revised manuscript.\n***\n**2. Missing the notation of the operation exp(). The probability of selecting an expert increases with i, which seems to contradict the intended randomness of stochastic.**\n\nThanks for your comments! The randomness of expert selection can be described and realized under different probability distributions. For example, the vanilla stochastic routing algorithm follows the uniform probability distribution, while ours follows the multinomial probability distribution. Compared to the uniform distribution, the multinomial probability distribution is less random but more controllable (by assigning different activation probabilities to each expert as in Equation (5)). This allows experts with a greater index to be activated more likely and therefore receive a larger volume of data during training. As a result, each expert can learn knowledge with different degrees of generalization and specialization.\n\nWe will include the explanation of the notation exp() and the above discussion in the revised manuscript.\n\n***\n**3. It would be beneficial to add experiments validating the plug-and-play effect or adjust the relevant descriptions in the paper.**\n\nThanks for your thoughtful suggestion! We agree that the plug effect should be further validated. Actually, Temporal Feature Modulation can be applied when the temporal module is separate from the CLIP encoder. Considering this constraint, we decided to adjust the relevant descriptions to make them more accurate and rigorous:\n\nOriginal:\n\n\"We propose a plug-and-play module that measures the confidence of temporal features by means of CLIP’s text space.\"\n\nRevised:\n\n\"We propose a test-time adaptation strategy for networks where the temporal module is separated from the CLIP encoder, to measure the confidence of temporal features by means of CLIP’s text space.\"\n***\n**4. What is the design basis for the candidate set of the temperature hyperparameter in weight merging? Missing comparison with the continuous space selection schemes.**\n\nThanks for your valuable question! The candidate set of the temperature hyperparameter {$±2^n · β$} $_{n=0}^4$ ∪ {∞} is designed by ourselves. We set the candidate temperature parameters to grow exponentially so that we can explore a larger range in the weight space (i.e. the coefficients calculated from Equation 7 with different temperatures vary more). {∞} is a special case where all the experts are averagely merged according to Equation 7.\n\nThe comparison with the continuous space selection schemes is presented below. We implement two continuous space selection schemes. (1) Sampling from a continuous standard normal distribution (mean=0, variance=1). (2) Sampling from a continuous uniform distribution. The continuous space sampling strategy results in a notable performance degradation.\n\n|Type|K400 (close-set)|UCF|K600|\n|-|:-:|:-:|:-:|\n|Discrete set (default)|86.8|87.5|78.9|\n|Continuous normal dist.|86.5|87.1|77.4|\n|Continuous uniform dist.|86.0|85.9|77.9|\n\n***\n**5. The connection between the modulation method, the issue of constrained semantic space, and the paper's overall motivation.**\n\nThanks for your constructive suggestion! This part is correlated with the second objective of our paper: *How can generalization and specialization coexist in one unified model (line 45-46)?* The additional parameters can model the temporal information quite well when dealing with the fine-tuning categories. However, since **the semantic space is constrained** during fine-tuning, the additional parameters may not model the temporal information properly when facing unknown categories. This concern grows as the test categories are less semantically correlated with the fine-tuning categories. Thus, we propose the **modulation module** that measures the confidence of the temporal feature by the semantic association of proxy fine-tuning and text categories. This strategy allows us to **improve the model generalization while keeping its specialization performance**. \n\nWe will refine the above discussion and add it to the revised manuscript.\n\n***\n**6. Improper trade-off metric.**\n\nThanks for your constructive question! We agree that the reviewers' concern is thoughtful and reasonable. We change the trade-off metric to the harmonic mean of the zero-shot performance A and close-set performance B, calculated as $\\frac{2AB}{A+B}$. This metric is not sensitive to extreme values and provides a more accurate indication of the overall performance.\n\n***\n**7. The relationship between experts and data bias views, and how the MoE approach leverages multiple data bias views to improve model performance.**\n\nThanks for your suggestion, and sorry for the confusion. ''Data bias views'' indicates the diverse knowledge learned from various optimization trajectories. We set different optimization trajectories for each expert and construct a more generalized model using distinct knowledge learned across experts. We will add the above description to the revised manuscript.\n***\nWe sincerely hope that this response will address the reviewers' concerns. We will supply the above responses in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8xG8m2Ft9T", "reviewer": "Reviewer_XfMx", "summary": "The paper introduces a novel framework called MoTE. This framework addresses the trade-off between zero-shot generalization and close-set performance in video recognition tasks by tuning a mixture of temporal experts. The key contributions include:\n\n- Introducing Weight Merging Regularization to balance generalization and specialization.\n- Proposing temporal feature modulation to improve generalization during inference.\n- Demonstrating state-of-the-art or competitive results on various video datasets such as Kinetics-400, Kinetics-600, UCF-101, and HMDB-51.", "questions": "- Can you provide more details on how expanding the text space with large-scale generative models might improve the model's performance?\n- How does the performance of MoTE vary with different numbers of temporal layers and experts? Are there optimal configurations for specific tasks?\n- What measures can be taken to reduce the computational overhead introduced by the additional components such as Weight Merging Regularization?", "limitations": "The following work is recommended for citation & discussion:\n\nOh, C., Lim, H., Kim, M., Han, D., Yun, S., Choo, J., Hauptmann, A., Cheng, Z.-Q., & Song, K. (2023). Towards calibrated robust fine-tuning of vision-language models. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models.\n\nTu, S., Dai, Q., Wu, Z., Cheng, Z.-Q., Hu, H., & Jiang, Y.-G. (2023). Implicit temporal modeling with learnable alignment for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 19936-19947).", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The introduction of Weight Merging Regularization and temporal feature modulation provides a novel approach to balancing generalization and specialization in video recognition.\n- The experimental results are thorough, demonstrating the effectiveness of the proposed methods on multiple datasets.", "weaknesses": "- The framework's text space is confined to video category names, which limits the richness of textual representations. Expanding the semantic space using large-scale generative models could enhance performance.\n- The method currently explores limited forms of additional parameters. Extending the approach to other forms could improve generality and versatility.\n- While results on certain benchmarks are promising, the model's performance on more diverse and challenging datasets needs further validation.\n- The additional complexity from Weight Merging Regularization and other components can slightly increase training time, which may be a barrier for real-time applications.\n- Extensive fine-tuning required for different tasks can be computationally expensive and time-consuming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments and the time you have dedicated to this paper. Please find the responses to your comments below.\n***\n**1. Expanding the semantic space with large-scale generative models.**\n\nThanks for your constructive suggestion! Following your suggestion, we replace the category names with rephrased detailed descriptions, which are generated by a large generative language model GPT3.5 as used in FROSTER [1]. The generated descriptions expand the semantic space by providing more details about the action and the scene. For example, \"*Ice climbing*\" is rephrased to \"*A person is seen climbing up a wall of ice using specialized equipment like crampons and ice axes*\". The results are presented in the table below.\n\n|Method|K400 (close-set)|UCF|K600|\n|-|:-:|:-:|:-:|\n|MoTE + category name|86.8|88.7|79.0|\n|MoTE + rephrased description|86.8|89.2|80.2|\n\nWe can see that the enriched description leads to better generalization performance. We will actively explore this direction in future work.\n\n[1] FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition. ICLR 2024.\n\n***\n**2. Extending the approach to other forms could improve generality and versatility.**\n\nThanks for your constructive suggestion! To further demonstrate the generality and versatility of MoTE, we apply our method to ST-Adapter [2]. For simplicity, we remove the depth-wise Conv3D operation in our adapter. In this case, our baseline is slightly lower than ST-Adapter, but it doesn't affect justifying the effectiveness of our method. The results are shown in the table below.\n\n|Method|K400 (close-set)|UCF|K600|\n|-|:-:|:-:|:-:|\n|ST-Adapter|81.1|77.3|61.1|\n|ST-Adapter+MoTE|81.2|79.4|66.7|\n\nOur method delivers a notable generalization performance improvement while maintaining the close-set result, indicating the effectiveness of MoTE applied to alternative networks.\n\n[2] ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning. NeurIPS 2022.\n\n***\n**3. The model's performance on more diverse and challenging datasets needs further validation.**\n\nThanks for your suggestion! We have evaluated our method on the Something-Something V2 (SSv2) dataset under the few-shot setting in Table 5, which requires intensive temporal understanding capability of the model. We find an inappropriate hyperparameter setting during fine-tuning and present the fixed results in the table below.\n\n|Method|SSv2-2 shot|SSv2-4 shot|SSv2-8 shot|SSv2-16 shot|\n|-|:-:|:-:|:-:|:-:|\n|ViFi-CLIP|6.2|7.4|8.5|12.4|\n|MAXI|7.1|8.4|9.3|12.2|\n|MoTE|7.4|9.5|13.3|16.7|\n\nWe also evaluate our method on SSv2 under the zero-shot setting, as shown in the table below\n\n|Method|SSv2-zero shot|\n|-|:-:|\n|CLIP|2.7|\n|ViFi-CLIP|4.5|\n|MoTE|6.4|\n\nIn both settings, our method outperforms the previous works by a notable margin, demonstrating the effectiveness of our method in facing more challenging datasets.\n\n***\n**4. The additional complexity from Weight Merging Regularization and other components can slightly increase training time.**\n\nThanks for your thoughtful comment! We show the training time costs (Table 8 of the supplementary material) and the corresponding GFLOPs in the table below. GPU days are calculated by the number of GPUs multiplied by the training time in days.\n\n|Method|GPU-days|GFLOPs|\n|-|:-:|:-:|\n|Baseline|4.35|649|\n|+MoTE|4.35|+0.34|\n|+$L_{WMR}$|4.50|+0.34|\n|+$L_{MSE}$|4.51|-|\n\nApplying our method on the baseline introduces slight computational and training time costs, and this cost can be effectively reduced when training with the Distributed Data Parallel framework of PyTorch, demonstrating the efficiency of our method.\n\n***\n**5. Extensive fine-tuning required for different tasks can be computationally expensive and time-consuming.**\n\nThanks for your comment! We would like to clarify that our method is evaluated on various datasets under zero-shot and close-set settings using **one unified model**, avoiding the need for multiple fine-tuning processes. Our method can also be applied to parameter-efficient training frameworks, such as adapters, which exhibit impressive training efficiency.  The details of this part please refer to the response of the second question. We will actively explore this direction in future work.\n\n***\n**6. How does the performance of MoTE vary with different numbers of temporal layers and experts? Are there optimal configurations for specific tasks?**\n\nThanks for your comment! We have ablated the number of temporal layers and experts in **Figure 1 (b)** and **Table 2 (b)** in the main paper. Please refer to the figure and table in the main paper for the results.\n\nAccording to the results,  we observe that 4 temporal experts per layer are sufficient to learn various knowledge for both close-set and zero-shot tasks. More experts do not lead to further performance gains. As for the number of the temporal layers, our method consistently outperforms the vanilla Transformer layer and 6-layer MoTE achieves the best trade-off between close-set and zero-shot performance. Therefore, we conclude that the 6 layers MoTE with 4 experts per layer are generally the optimal configuration for various tasks.\n\n***\n**7. What measures can be taken to reduce the computational overhead introduced by the additional components?**\n\nThanks for your thoughtful question! Our work is applied to on the standard Transformer structure in the paper. Replacing it with some efficient structure (e.g. flash attention) can effectively reduce the computational overhead of the additional modules. Besides, applying our method to efficient structures (e.g. adapters) can also reduce the computational cost of the additional components.\n\n***\n**8. Relevant works**\n\nThanks for your suggestion! We find the recommended papers relevant to our work. We'll cite and discuss them in the revised manuscript.\n\n***\nWe sincerely hope that this response will address the reviewers' concerns. We will supply the above responses in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all reviewers for their valuable and constructive comments. We have carefully considered the points raised by each reviewer and provided comprehensive responses to each question. Besides, we attach an additional PDF file containing a detailed analysis of the category-wise performance across experts.

Meta-review: This paper receives unanimous weak accept recommendations. After a careful check of paper, comments, and rebuttal, the AC agrees with the reviewers and think the authors has well addressed the reviewer concern. The paper introduces a interesting idea to balance the generalization and speicalization ablity during transfer learning of VLM. The results demonstrate the effectiveness of proposed method.Thus, the AC makes a accept recommendation.
Decision: Accept (poster)