FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition
Authors: Xiaohu Huang
Hao Zhou
Kun Yao
Kai Han
Venue: ICLR 2024 poster (2024)
Keywords: Open-Vocabulary Action Recognition
Action Recognition
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We adopt the frozen CLIP as a teacher model to distil its pre-trained generalizable knowledge into the tuned models for open-vocabulary action recognition.

Abstract: In this paper, we introduce \textbf{FROSTER}, an effective framework for open-vocabulary action recognition. The CLIP model has achieved remarkable success in a range of image-based tasks, benefiting from its strong generalization capability stemming from pretaining on massive image-text pairs. However, applying CLIP directly to the open-vocabulary action recognition task is challenging due to the absence of temporal information in CLIP's pretraining. Further, fine-tuning CLIP on action recognition datasets may lead to overfitting and hinder its generalizability, resulting in unsatisfactory results when dealing with unseen actions.
To address these issues, FROSTER employs a residual feature distillation approach to ensure that CLIP retains its generalization capability while effectively adapting to the action recognition task. Specifically, the residual feature distillation treats the frozen CLIP model as a teacher to maintain the generalizability exhibited by the original CLIP and supervises the feature learning for the extraction of video-specific features to bridge the gap between images and videos. Meanwhile, it uses a residual sub-network for feature distillation to reach a balance between the two distinct objectives of learning generalizable and video-specific features.
We extensively evaluate FROSTER on open-vocabulary action recognition benchmarks under both base-to-novel and cross-dataset settings.  FROSTER consistently achieves state-of-the-art performance on all datasets across the board. 
Project page: \url{https://visual-ai.github.io/froster}.

Published as a conference paper at ICLR 2024.
FROSTER: FROZEN CLIP IS A STRONG TEACHER FOR OPEN-VOCAAAE VOCABULARY ACTION RECOGNITION
Xiaohu Huang! Hao Zhou²
Kun Yao²
Kai Han
I Visual Lab, The University of Hong Kong
2 Department of Computer Vision Technology (VIS), Baidu Inc.
bhanggiiahhuconnee cooonect.hkk.hie hk, 2 zhouh15 6 60mail.......am.maammmaammeaammm .sw..udu.cc.... yaokun01@baidu, com, kaihanx@hku.c
ABSTRACT
In this paper, we introduce FROSTER, effective framework for open- vocabulary action recognition. The CLIP model has achieved remarkable success in range of image- based tasks, benefiting from its strong generalization capa- bility stemming from pretaining on massive image-text pairs. However, applying CLIP directly to the apen-vocabulary action recognition task is challenging due to the absence of temporal information in CLIP's pretraining. Further, fine-tuning CLIP action recognition datasets may lead to overfitting and hinder its general- izability, resulting in unsatisfactory results when dealing with unseen actions. To address these issues, FROSTER employs residual feature distillation approach to ensure that CLIP retains its generalization capability while effectively adapt- ing the action recognition task. Specifically, the residual feature distillation treats the frozen CLIP model as teacher to maintain the generalizability exhib- ited by the original CLIP and supervises the feature learning for the extraction of video-specific features to bridge the gap between images and videos. Mean- while, it uses a residual sub-network for feature distillation to reach a balance between the two distinct objectives of learning generalizable and video-specific features. We extensively evaluate FROSTER on open-vocabulary action recogni- tion benchmarks under both base-to-novel and cross-datasel settings. FROSTER consistently achieves state-of-the-art performance on all datasets across the board. Project page: https: /vvvvvviall... ai., sii.gittin github. io/ froster.
INTRODUCTION
Open-vocabulary action recognition aims to recognize action categories that may not have been seen during training. The study of action recognition was dominated by pure vision based methods, such as (Ji et al., 2012; Carreira & Zisserman, 2017; Wang et al., 2016; Feichtenhofer et al., 2019; Lin et al., 2019), which assumes closed-set setting where the models are trained and evaluated on set of fixed and predefined action categories. Recently. the emergence of large-scale vision- language models, such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), has enabled image classification on an open vocabulary of categories. Inspired by this success, attempts have been made to apply the CLIP model to action recognition by processing the video frame-br soune-by-frame, as CLIP was pretrained on image-text pairs. However, directly utilizing pretrained CLIP models yields sub-optimal performance, as these models lack access to video-text data during pretraining. To bridge the domain gap between images and videos, methods have been proposed fine-tune the CLIP model (Wang et al., 2021; Liu et al., 2023; Rasheed et al., 2023) or utilize efficient adap- tion/prompting techniques (Chen et al., 2022a; Yang al., 2023; Ju et al., 2022; Wasim al., 2023; Ni et al., 2022) to extract video-specific knowledge. While these methods have achieved success in the closed-set setting. their performance is inferior when it comes to unseen categories (see Tab. 1).
In Fig. 1, compare the frozen CLIP and three CLIP-adapted video models, namely Action CLIP (Wang et al., 2021), AIM (Yang et al., 2023) and STT-Adapter (Pan et al., 2022), in an open- vocabulary setting. These models fine-tuned (except for the frozen CLIPP) on Kinetics-400 (Car- reira & Zisserman, 2017) but evaluated on UCF-101 (Soomro et al., 2012), HMDB-51 (Kuchne
1 CCoresponding author.

---

Published as a conference paper at ICLR 2024.
Frozen CLIP
Action CLIP
AIM
ST
Ensemble w/ Fruzen CLIP
UCF-101 (0.805)
HMMDB-5((0.71))
Kiinetis--000 (0.721)
53.3
72
72.1
1 80
81
78
79
+3.3%
+2.0%
+4.2%
77.9
52
51
50.6
+3.0%
Trpp Arraay
68
20
60.1
+7.0%
69.5
+5.4%
66.7
77.4
+2.6%
64
+8.9%
74
75
* 74.2
47.6
62.5
60.2
73
Frozen
ST-Adopter
Frozen CLIP
Action
AIM
ST-Adapter
Fruees CLIP
Action
ADI
ST-Adapter
CLIP
CLIP
CLIP
(c)
Figure 1: Performance comparison (Top-1 Ace (%)) under the open-vocabulary evaluation setting where the models are tuned on Kinetics-400, but evaluated on UCF-101, HMDB-51, and Kinetics- 600. Note that shared categories between Kinetics-600 and Kinetics-400 are excluded when testing on Kinetics-600. The numbers in the brackets denote the semantic distance between training and evaluation datasets, which is measured by Hausdorff distance text features of category names. Larger numbers denote higher similarities.
al., 1), and Kinetics-600 600 (Carreira et al., 2018) (shared categories with Kinetics-400 cluded). Compared to the frozen CLIP, we observe that the tuned models achieve higher perfor- mance UCF-101 and HMDB-51, but achieve inferior performance on Kinetics-600. This ob- servation indicates that the generalization capacity of tuned models varies across different datasets. Further, using text features of category names, we measure the semantic distances between the train- ing set (Kinetics-4000) and testing sets (UCF-101, HMDB-51, and Kinetics-600), We find that, for testing sets that are more semantically similar the training set (i.e., UCF-101 and HMDB-51), the tuned models exhibit improvements over the frozen CLIP. However, when it comes to the se- mantically less similar dataset, namely Kinetics-600, which demands stronger generalizability, the performance of all tuned models deteriorates compared to the frozen CLIP. This suggests decline in generalizability after fine-tuning. Based on these observations, we believe that CLIP-based video model should possess the following properties: (1) The CLIP model should acquire video-specifi knowledge to bridge the gap between images and videos, effectively addressing the challenges posed by the image-to-vine videe- domain transition. This entails adapting the model understand and extract meaningful information from video data, taking into account temporal dynamics and context. (2) crucial preserve the strong generalization capability of the pretrained CLIP model within the video model. This ensures that the video model can effectively generalize across different video datasets and tasks, leveraging the learned representations from the pretraining phase.
To validate the hypothesis made above, evaluate the performance of each adapted model by sembling it with the frozen CLIP by summing up their respective outputs. The results in Fig. demonstrate that the ensemble models exhibit significant improvements across all datasets, indicat- ing their efficacy as open-vocabulary classifiers. However, the naive approach of ensembling models results in notable additional computational costs due to inferring two models simultaneously. This limitation hinders practical applications in real-world scenarios. Therefore, it is imperative to ex- plore methods for consolidating the ensemble model's 's knowledge into single model, mitigating the computational burden.
Input Video
Frozen CLIP
zf
Residual
Feature Distillation
A video of "Brushing teeth."
2
Clussification Loss
Tuned Model
Input Texts
g
Figure 2: Overall idea of our FROSTER framework. It effectively learns feature representation that is both video-specific (via simple action-based fine-tuning) and generalizable (via our proposed residual feature distillation).
2

---

Published as a conference paper at ICLR 2024.
To this end, we propose FROSTER, simple yet effective framework for open open-vocabulary action recognition, which effectively learns feature representation that is both video-specific and gener- alizable. As shown in Fig. 2. 'video-specific' is achieved through common classification- finetuning, while "generaizzabee is achieved by using frozen CLIIP as a teacher to impart pretrained knowledge to the tuned model, inspired by knowledge distillation techniques (Hinton et al., 2015; Romero et al., 2014). The distillation process is akin to regularization term that ensures the tuned features do not diverge too far from the frozen ones. Having two distinct objectives, we need to bal- ance the feature learning between them. For instance, if we enforce the tuned features to be overly close to the frozen features, it may hinder the video-specific learning to fit the video data, Con- versely, if we overemphasize video-specific learning, the generalizable capacity in the tuned model might be lost. To address this issue, we propose a residual feature distillation approach to balance feature learning between the two joint objectives.
Taking inspiration from the model patching method (Ilharco et al., 2022), a prior study (Weng et al., 2023) develops an open-vocabulary action recognition model through weight interpolation between the fine-tuned model and the frozen CLIP. This approach bears some resemblance of motivation to ours. Nevertheless, the weight interpolation technique necessitates the fine-tuned model and frozen CLIP to possess identical weight dimensions, thereby restricting its applicability across di- verse network architectures (e.g. adapter-based CLIP models). Also, methods (Zenke et al., 2017; Kirkpatrick et al., 2017) that penalize the updates of the pretrained layers are not applicable to the adapter- based methods since the pretrained layers are frozen.
We thoroughly evaluate the effectiveness of FROSTER on the two open-vocabulary settings, namely cross-dataset and base-to-novel, using the Kinectics-400 (Carreira & Zisserman, 2017), Kinetics- 600 (Carreira et al., 2018), UCF-101 (Soomro et al., 2012), HMDB-51 (Kuehne et al., 201 1), and Something-te to- Something V2 (Goyal et al., 2017) datasets. The cross-dataset evaluation protocol tests models on a different dataset, while the base-to-novel evaluation protocol evaluates their ability to recognize unseen action categories within the same dataset. We couple FROSTER with different video recognition networks. In all cases, FROSTER demonstrates superior performance, showcasing its effectiveness and versatility.
The main contribution of this paper can be summarized as follows: Firstly, we introduce FROSTER, a simple yet effective framework for open- vocabuaryyy action recognition. It can effectively learn fea- ture representation that is both video-specific and generalizable. Secondly, we introduce residual feature distillation approach to balance feature learning in both objectives. This technique mitigates potential conflicts and enables the model to achieve both goals simultaneously. Thirdly, we demon- strate the superiority of FROSTER through extensive evaluations on cross-dataset and base-to-novel settings across Kinectics-400, Kinetics-600, UCF-101, HMDB-51, and Something- t-Soothhing V2 datasets with various network architectures.
2
RELATED WORK
2.1 CLIP-BASER ACTION RECOGNITION
Inspired by the strong representation of the pretrained CLIP model, many video recognition ap- proaches have been proposed based on CLIP. These works can generally be categorized into two types: full fine-tuning (Wang et al., 2021; Liu et al., 2023; Rasheed et al., 2023) and partial fine- tuning (Chen et al., 2022a; Ni et al., 2022; Pan et al., 2022; Yang et al., 2023; Park et al., 2023). For full fine-tuning methods, ActionCLIP (Wang et al., 2021) is the first to introduce CLIP into video recognition, fine-tuning the CLIP model and applying additional temporal layers to model motion. Similarly, STAN (Liu et al., 2023) uses an auxiliary network to extract temporal features based on CLIP. ViFi-CLIP (Rasheed et al., 2023) shows that simply fine-tuning both the vision and the text encoders are also effective for action recognition. For partial fine-tuning, a common practice is to freeze the pretrained model parameters and introduce extra parameters for training. Inspired by the parfmeter-efficient fine-tuning in Natural Language Processing (Houlsby et al., 2019; Hu et al., 2021), Adaptformer (Chen et al., 2022a), X-CLIP (Ni et al., 2022), ST-Adapter (Pan al., 2022). AIM (Yang et al., 2023), and DUALPATH (Park et al., 2023) employ lightweight adapters to transfer knowledge from the image to the video domain. Similarly, VPT (Ju et al., 2022) and Vita-CLIP (Wasim et al., 2023) leverage learnable prompts to improve the recognition performance.
Although these two types of methods have shown promise in the closed-set setting, their per- formance is not satisfactory in open-vocabulary settings, as shown in Fig. 1. Recently, Open-
3

---

Published as a conference paper at ICLR 2024.
VCLIP (Weng et al., 2023) targets the open-vocabulary setting, yet its reliance weight inter- polation constrains its applicability to networks having exactly the same weight dimensions. In contrast, proposed framework distills knowledge from CLIP by enforcing feature consistency, without constraining the network architectures, thereby being compatible with diverse networks.
2.2 FEATURE-BASER KNOWLEDGE DISTILLATION
Knowledge distillation a technique that involves transferring knowledge from teacher model to a student model. The mainstream distillation methods can be broadly categorized as logits- (Hin- ton et al., 2015), similarity-based (Tung & Mori, 2019), and feature-based (Romero et al., 2014). Feature-based distillation has been shown to provide clearer optimization targets and outperforms the other two approaches (Chen et al., 2022b).
Recently, the feature-based distillation method has been applied in CLiP-based models, egg., CLIP- PING (Pei et al., 2023) and VLKD (Dai et al., 2022). CLIPPING focuses on model compression, which aims to fully transfer the knowledge from Clip4clip (Luo et al., 2022) (a large teacher model) to MobileViT-v2 (Mchta & Rastegari, 2022) (a small student model). The method is tailored for the aforementioned architectures and can not generalize to other architectures. VLKD concentrates on aligning the features of the language model the CLIP model and subsequently integrating them to be a multi-modal generator. All of these methods aim to distill the same knowledge from one model to another. Differently, in our case, have two objectives: maintaining the generalization capability of a pretrained CLIP and effectively adapting from image to video tasks.
To achieve our goals, we propose novel approach, called residual feature distillation, which allows flexibly adapting features from images to videos, while not sacrificing the generalization capability of the pretrained CLIP.
3 METHOD
The overall pipeline of FROSTER consists of two key components, namely, model finetuning to bridge the gap between image and video tasks, and knowledge distillation maintain the gener- alizability of the pretrained CLIP. In the following, we first introduce the preliminary in Sec. 3.1, followed by our method in Sec. 3.2.
3.1 PRELIMINARY
3.1.1 ACTION RECOGNITION WITH CLIP
Owing the strong representation stemming from pretraining massive image-text pairs, CLIP has become a foundation model, which has been increasingly applied to various downstream tasks. Utilizing CLIP for video recognition is straightforward: process each frame individually and then average their outputs for prediction. Consider the CLIP model with ViT (Dosovitskiy et al., 2020) architecture. Given a video æ; x with T frames and each frame is with the spatial dimension of H x W, we can directly feed it into the visual encoder fo of CLIP, which can be written as:
can simply use the CLIP model to
2 (a())
where the visual feature 2 v.S TXC and C' denotes the feature dimension of the [CLS] token in VIT. The feature of (CLS) token embeds representation of the whole frame. To obtain the video-level representation, we employ simple average pooling on all 2 2, from different frames to
obtain the global embedding vector z/I E RC, Regarding text processing, a common practice is to embed the action categories in pre-defined templates (e.g., "A video of [J") as the input. Consider an embedded action tj. we can obtain the text features 2// with the text encoder ft as:
filt),
where 2// E C.C. During training, the learning objective is to maximize the image-text representa- tion similarity between zy and 2/ corresponding to the same class. However, the text data may be scarce when fine-tuning the text encoder if we just use the template-embeddde action names as text inputs. To alleviate this issue, following Xiang et al. (2023); Momeni et al. (2023); Zhao et al. (2023), we adopt GPT3.5 to enrich the action names with more auxiliary descriptions, which can be viewed as text data augmentation during training. The implementation details are given in Sec. 4.
4

---

Published as a conference paper at ICLR 2024.
zf
Frozen CLIP
Frozen CLIIP
Frozen CLIIP
LED
LFD
MIP
Tuned Model
Lee
LLE
Tuned Model
Las
Zg
Zg
(a) Feature Distillation w/o projector. (b) Feature Distillation projector. (c) Residual Feature Distillation.
Figure 3: Illustration of feature distillation approaches. LCE and LED denote cross-ents coos-entroppy loss and feature distillation loss, respectively. (a) Directly minimizing the feature distance between the student and teacher model (Chen et al., 2022b). (b) Using a two-layer MLP to map the feature from the student space the teacher space (Deng & Zhang, 2021; Yang al., 2020). (c) The proposed residual feature distillation, employing modified residual network on the student model to achieve balance in feature learning. Our method aims to simultancously optimize vidco-specific knowledge and generalizability, enhancing the overall performance of the model.
3.1.2 FEATURE-BASED DISTILLATION
The feature-based distillation (Romero et al., 2014; Deng & Zhang, 2021; Yang et al., 2020; Chen al., 2022b) achieved by enforcing the feature consistency between the teacher and student models, with loss functions such 1.2 loss:
1 LFD N Σ (2) 13
- (a)))l
(3)
where f( f!()) and N denote the teacher model, student model, and batch size respectively. In this paper, the objective of feature distillation is maintain generalizability in the fine-tuned model, which is crucial for open-vocabulary recognition.
3.2 FROSTER
Video-specific fine-tuning. For model fine-tuning, following Eq. (1) and Eq. (2), by feeding the video-text inputs into the tuned model g. obtain the tuned visual features ag. Rº and textual features 24, RCC
"g
2 (x;),
(4)
2/ ge(t)),
where and denote the visual and textual encoders of the tuned model, respectively. By calculating the feature similarities between each video and texts of all categories, we can predict the category y! for each video. Given the ground truth ÿt, we use cross-entropy loss for video- specific learning:
K
LCE
Di, log Vi.j.
(5)
N
where K denotes the total number of classes.
Residual feature distillation. Taking the frozen CLIP as teacher, two common ways to realize feature-based distillation are illustrated Fig. 3 (a) (Chen et al., 2022b) and Fig. (b) (Deng & Zhang, 2021: Yang et al., 2020). Here, for simplicity, we use and 2 to denote z (zt) and zÿ (z²). As shown in Fig. (a), since the output feature dimensions of the tuned model and the frozen CLIP stay the same, we can directly conduct feature distillation between them without feature projection. However, this supervision expects the tuned features 2g to remain the same as the pretrained ones 21, which prohibits =n from learning video-specific knowledge. Another possible way (as in Fig. (b)) is to apply a projector to map 2g from the student space to the teacher space. This can relax the constraints 2g for better fitting the video data. However, under such condition, this distillation loss would too loose for 2, to keep close with Zy. thereby limiting its generalizability. Therefore, need to find trade-off between the above methods, considering both learning objectives.
5

---

Published as a conference paper at ICLR 2024.
Inspired by the residual design of ResNet (He et al., 2016) for reducing optimization difficulty in deep networks, we propose a modified residual network for balancing the two learning objectives when conducting distillation. The intuition behind the design is to allow the tuned features to effec- tively receive supervision from generalized ones while also being video-specifie. As in Fig. 3 (c), apply modified residual network h on 2g transform its representation with two-layer MILP projector and an identity mapping:
2g h(zg) 2g + W2(o((W(<g))),
(6)
where W1 E RCXC, W2 E RCC, denotes GELU (Hendrycks & Gimpel, 2016) function, and a balancing coefficient. this way, our design enjoys three major benefits:
(1) Since there exists identity mapping in the transformation Eq. (6), the generalizable target 2f can directly guide the generalizable learning of 2g. which is similar to Fig. 3 (a). But differently, given the projected term W2(o(WW (zg))) we do not enforce 2g to be the same as 2,, which makes it flexible for to fit the video data.
(2) a is an important factor in balancing the learning in two objectives. If we set it as small number, the learned embedding space for 2g is largely constrained by the teacher model, otherwise Eg may overfit the video data and impair generalizability. In experiments, we find that setting a as relatively small number (e.g., 0.1) leads to better performance than a large one. This phenomenon suggests that the pretrained CLIP already possesses strong representation, thus we only need to slightly adjust it to transfer from images to videos.
(3) Inspired by the initialization strategy in Hu et al. (2021), to make sure &g is learned starting from the pretrained status, we initialize the parameters the second fully connected layer W2 as zeros. Therefore, the beginning of fine-tuning, 2g only contains 2g and gradually gets updated. This notably improves the training stability.
Loss function. After obtaining the transformed vectors 2y and ay. we can use the frozen CLIP as the teacher distill knowledge. The distillation loss for vision and text features can be written as:
N
LED =
Σ -e
K L'p - (t(o(())))l
Σ
3
LFD = LID + L'FD-
The overall learning objective is then the combination of classification and distillation losses:
LCE + BLFD,
(8)
where ( is a balancing coefficient.
4 EXPERIMENT
Experimental setting. Following the common practice in the literature, we adopt two experimen- tal settings: base-to- no-evll- and cross-dataset evaluation. Base-to-novel: Under this setting, for each dataset, we divide the class vocabulary into two non-overlapping sets, i.e., the base set Yn and novel YN. where Yn n YN 0. The models are trained on samples from Yn. and evaluated ing samples from YB U YN- Cross-datasel: Under this setting, we train the models on a source dataset with the class vocabulary set as Yg and evaluate them on a target dataset with another vo- cabulary set as Yr. where IYs U Yrl > |Yss n Yr!. We evaluate our method using the common UCF-101 dataset (Soomro al., 2012), HMDB-51 dataset (Kuehne al., 2011), Kinetics-400 (K- 400) dataset (Carreira & Zisserman, 2017), Kinetics-600 (K-600) dataset (Carreira et al., 2018), and Seweehhing-t-SSmmtthee
test-
V2 (SSv2) dataset (Goyal et al., 2017). K-400 and K-600 are large- scale action recognition datasets with 400 and 600 action classes, respectively. UCF-101 consists of 13,320 video clips from 101 action classes, and HMDB-51 includes 6,849 videos from 51 action classes. SSv2 contains 174 fine-gn fine-grained action classes. We follow the literature (e.g., Rasheed et al. (2023); Ni et al. (2022); Weng et al. (2023)) to conduct experiments under the two settings and report the average top-1 accuracy.

---

Published as a conference paper at ICLR 2024.
Performance comparison (Topl-Ace (%)) with the CLiP-based methods using ViT-B/16 under the base-to-novel setting. "HM" denotes the harmonic mean of the accuracy from the base and novel sets. The results of most other papers are taken from ViFi-CLIP. i denotes the results with our implementation. The best results are bolded, and the second-best results are underlined.
Table 1:
K-400
HMDB-51
UCF-101
SSv2
Method
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Frozen CLIP (Radford et al., 2021)
62.3
53.4
57.5
53.3
46.8
49.8
78.5
63.6
70.3
4.9
5.3
5.1
ActionCIIIP (Wang al., 2021)
61.0
46.2
52.6
69.1
37.3
48.5
90.1
58.1
70.7
13.3
10.1
11.5
XCLIP (Ni et al., 2022)
74.1
56.4
64.0
69.4
45.5
55.0
89.9
58.9
71.2
8.5
6.6
7.4
VPT (Ju ct al., 2022)
69.7
37.6
48.8
46.2
16.0
23.8
90.5
40.4
55.8
8.3
5.3
6.4
AIM i (Yang et al., 2023)
74.6
62.5
68.0
64.0
51.6
57.1
89.8
76.4
82.6
8.5
7.9
8.2
ST-Adapier i (Yang et al., 2023)
73.6
62.0
67.3
65.3
48.9
55.9
85.5
76.8
80.9
9.3
8.4
8.8
ViFi-CLIP (Rasheed et al., 2023)
76.4
61.1
67.9
73.8
53.3
61.9
92.9
67.7
78.3
16.2
12.1
13,9
Open- VCLIP t (Weng et al., 2023)
76.5
62.6
68.9
70.3
50.4
58.7
94.8
77.5
85.3
16.0
11.0
13.0
FROSTER
77.8
64.3
70.4
74.1
58.0
65.1
95.3
80.0
87.0
18.3
12.2
14.6
Architectures. We use CLIP with ViT-B/16 vision encoder for all experiments. As for the tuned model, we adopt VCLIP (Weng et al., 2023), fully fine-tuned CLIP, Action CLIP (Wang et al., 2021), Adaptformer (Chen et al., 2022a), AIM (Yang al., 2023), and ST-Adapter (Pan et al,, 2022) to verify the effectiveness of our method. Unless stated otherwise, we use VCLIP for our experiments.
Text augmentation. To enrich the category names with more details for training, we adopt GPT3.5 to generate helpful descriptions using the instruction: "Please describe this action in the video IP" We find that such simple instruction is effective in providing extra text for training. For instance, the action "brushing teeth" is augmented to "The video teaches the process of brushing teeth, which involves using toothbrush and toothpaste to clean and maintain oral hygiene".
More experimental details can be found in Appendix A.1.
4.1 COMPARISON WITH STA METHODS
Base-to-novel. In Tab. 1, we compare our method with the previous methods under the base-to- novel setting. From the results, three notable findings summarized: (1) Compared with some adapted models (Action CLIP, X-CLIP and VPT), the frozen CLIP is still competitive, especially on the novel sets of K-400, HMDB-51, and UCF-101, indicating its strong generalizability. (2) FROSTER achieves the best performance over all datasets for both the base and novel sets, validating its capacity to be video-specific and generalizable. (3) Compared to the baseline (VCLIP), our method can achieve consistent achievements on all base and novel categories, which further verifies its effectiveness. (4) On K-400, HMDB-51, and UCF-101, FROSTER achieves larger improvements on the novel sets over the base sets. This demonstrates its great potential to be applied in open- world applications. Besides, we observe that the performance gains achieved on the novel set of SSv2 are relatively modest compared to other datasets. Given the fine-gre fine-grained nature of SSv2, it requires a stronger temporal awareness of the model to perform well on this dataset. As no cross- frame temporal information generalize well on the fine greined actions. This could potentially also explain why the performance of all other methods is significantly lower SSv2 compared to other datasets.
used during the CLIP pretraining, it is still challenging for CLIP to
Catssedataset. In Tab. 2, we compare our method with the previous methods under the cross- dataset setting. We notice that, on the most generalizability demanding dataset, i.e., K-600, the frozen CLIP outperforms most of the other methods, further demonstrating its superior general- izability over the adapted video CLIP models. FROSTER achieves the best performance over all datasets compared with the fully-finetunin based methods (X-CLIP, AIM, and ST-Adapter), prompting method (VPT), and weight interpolation method (Open-VCLe VCLIP), demonstrating its strong generalizabilty.
methods (ActionCLIP and ViFi- ViFi-CLIP), adapter-
4.2 DIAGNOSTIC STUDY
Effectiveness with different networks. In Tab. 3, we apply FROSTER with two types of CLIP- based video models, i.e., fully-tuned and adapter-based. We notice that: (1) For all the networks, FROSTER can effectively improve performance, highlighting its broad applicability. Empirically, find that FROSTER achieves larger improvements on K-600, which is more generalizability-
7

---

Published as a conference paper at ICLR 2024.
Table 2: Performance comparison (TopI-Ace (%)) with the previous approaches under the cross- dataset setting. All methods are based on CLIP ViT-B/16, except for ER-ZASR (TSM (Lin et al., 2019) pre-trained on ImageNet- 1k) and Text4Vis (ViT-L/144. UCF* and HMDB* indicate evalu- ating the full validation set, while UCF and HMDB denote evaluating across the three validation splits, The results of most other papers are taken from Open-VCLIP and ViFi-CLIP. i denotes
results are produced with our implementation.
Method
UCF
UCF
HMDB *
HMDB
K-600
ER-ZASR (Chen & Huang. 2021)
51,8
35.314.6
42.1
Frozen CLIP { (Radford et al., 2021)
74.2
73.8+0.6
46.3
4..9
68.1
ActionCLIP i (Wang et al., 2021)
77.4
77.5+0.8
48.0
1..2
1.2
X-CLIP (Ni et al., 2022)
72.0+2.33
44.6+t5.2
65.2++0.4
VPT (Ju al., 2022)
69.3+44.2
44.3++2.2
55.8+0.7
Tixt4Vis (Wu et al., 2023)
79.6
49.8
68.9+1.0
AIM f (Yang et al., 2023)
79.0
7..411.0
49.5
5..310.8
66.7.10.5
ST-Adapter i (Pan et al., 2022)
77.9
77.610.77
50.3
51.110.6
60.2
Vita-CLIP (Wasim et al., 2023)
75.0110.6
48.610.6
67.410.5
ViFi-CLIP (Rasheed et al., 2023)
76.810.7
51.310.6
71.21:10
Open- Open-VCLIP (Weng ct al., 2023)
83.5
83,4+112
53.2
53.9 1.2
73.0+00.8
FROSTER
85.0
84.8+1.11
54.5
54.8 1.3
74.8
Table 3: Performance comparison (topl-Ace (%))
when using FROSTER with different networks.
Method
UCF*
UCF
HMDB*
HMDB
K-600
Fine-tuned CLIP
80.0
80.0+0.8
48.3
19.8+1.66
66.1
Fine-tuned CLIP w/RROSTER
83.5
83.3+0.8
53.6
53.811.5
73.6+1.0
Fully-Tuned
Action CLIP (Wang et al., 2021)
77.4
77.5+0.88
48.0
48.2+
6..5++122
Action CLIP w/ FROSTER
84.0
83.7 +0.8
54.0
73.8+0.9 54.1=008
VCLIP (Weng al., 2023)
79.7
7..8
49.8
50.3.10.8
65.9
VCLIP w/RROSTER
85.0
84.8+
54.5
74.8+00.9 5..
AdapterFormer (Chen el 2022a)
80.5
80.
50.5
$1.0.0.88
67.0.10.4
AdapterFormer w/FROSTER
81.0
8.2
53.1
53.5#1.3 74.0
Adapter-Based
AIM (Yang et al., 2023)
79.0
79.4
49.5
50.310.8
66.710.55
AIM w/FROSTER
81.2
81.0
53.2
53.5+1.3
74.1+0.8
ST ST-Adapter (Pan et al., 2022)
77.9
77.6=0077
50.3
51.1
60.2+1..8
ST-Adapter w/FROSTER
79.5
79.3+11.1
52.0
+2, 73.1+00.9
demanding dataset compared with UCF-101 and HMDB-51. (2) Combining with FROSTER, fully fine-tuned models can generally achieve better results than the adapter-based methods. This im- provement might be attributed to the fitting capacity of more trainable parameters, which afford greater flexibility for attaining both video-specific and generalizable learning objectives than the adapter-based models.
Effectiveness of residual feature distillation. In Tab. 4, we conduct experiments to verify the effectiveness of our distillation design (see Fig. 3). is notable that: (1) For both the vision and text encoders when applying distillation methods (see Exp. B D and Exp. F- H), they can effectively improve recognition performance compared with not using them (see Exp. A and Exp. E). This phenomenon verifies the useful generalizable knowledge in the frozen CLIP. (2) By comparing Exp. and (or Exp. and G), we find that using a projector for feature distillation achieves inferior performance than not using it, which indicates that the projector may hinder the distillation supervi- sion, which degrades the generalizable capacity. (3) Using the proposed residual feature distillation method achieves the best performance (see Exp. D and Exp. H), verifying its superiority over the other two widely used methods for balancing video-specific and generalizable learning. (4) By comparing Exp. I and J, we can see that the simple class name enriching technique can effectively improve the action recognition performance.
Attention visualization. As shown in Fig, 4, we notice that frozen CLIP tends to focus the background regions instead of the moving objects. In contrast, the fine-tumed VCLIP exhibits at- tention towards the key semantic elements (e.g., bottle and cup). Further, method attends to the moving parts more (e.g., hands), while considering also the helpful background information (e.g., sink and table).
8

---

Published as a conference paper at ICLR 2024.
Table Effects of frozen CLIP distillation methods and the augmented text under the cross-dataset evaluation setting. "RFD" denotes Residual Feature Distillation.
Distillation
Exp.
Tuned Encoder
UCF
HMDB
K-600
w/o MLP w/ MLP
RFD
A
79.7
49.8
65 6..9+1.0
B
81.8
70.1+00.9
C
Vision
80.5
49.8
67.9+0.99
D
82.9
52.7
71.1=100
E F G
77.2
69.3++0.9
78.3
51.7
70.6=10.9
Text
78.2
48.7
69.7.110
H
78.6
52.3
71.0+1099
I
Vision + Text Vision Text (Text Augmented)
84.4
73.0++0.7
J
85.0
54.5
74.8
(0.77L) IKK-000 (8.721)
Irputs
Reladre Performaee Inprevccent
20
Frow CLIP
VCLP
ST-Adaple
Adapter
FROSTER
CLEP
Figure 5: Relative improvements of different methods when combined with FROSTER. The number next to the legend for each dataset indi- cates the dataset's similarity with K-400 using Hausdorff distance (cosine similarity).
Figure 4: Attention visualization of attention cor- relations between [CLS] and image tokens of the action "pour". Our method focuses on moving ob- jects and informative backgrounds.
Dataset semantic distance v.s. performance improvements. To study the relationship between dataset semantic distances and performance improvements, we measure the semantic distances be- tween the training and testing datasets by comparing text features of action class names between datasets. Figure presents the relative improvements on different testing datasets by different mod- els. We observe that greater improvements are attained on datasets that require more generalizability, i.e., K-600. This finding serves as additional evidence the efficacy of FROSTER in enhancing generalizable capacity.
Please refer to the appendix for additional experiments and visualizations.
CONCLUSION
In this paper, we present FROSTER, simple yet effective framework designed to tackle the open- vocabulary video recognition task. FROSTER achieves the dual objective of being video-specific and generalizable. To accomplish this, we introduce frozen CL.IP model teacher model to facilitate generalization. Additionally, we propose residual feature distillation method balance feature learning in both objectives. FROSTER is compatible with various network architectures and has been extensively evaluated on base-to-novel and cross-dataset evaluation settings using large- scale video datasets, demonstrating its effectiveness.
ACKNOWLEDGMENTSS
This work is supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27208022), National Natural Science Foundation of China (Grant No. 62306251), and HKU Seed Fund for Basic Research.

---

Published as a conference paper at ICLR 2024.
REFERENCES
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
Joao Carreira, Eric Noland, Andras Banki- Horrit, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600) Arxiv e-prints, 2018.
Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition. In ICCV, 2021.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurlPS, 2022a.
Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. In NeurIPS, 2022b.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation clip via vision-language knowledge distillation. Arxiv e-prints, 2022.
Xiang Deng and Zhongfei Zhang. Comprehensive knowledge distillation with causal intervention. In NeurlPS, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition at scale. Arxiv e-prints, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski. Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). Arxiv e-prints. 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. Arxiv e-prints, 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efffcient transfer learning for nlp. In ICML, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. Arxiv e-prints, 2021.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Si- Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpo- lating weights. In NeurlPS, 2022.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE TPAMI, 2012.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vasion-language representation learning with noisy text supervision. In ICMI., 2021.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, 2022.
10

---

Published as a conference paper at ICLR 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Graassa--arwwine et al. Overcom- ing catastrophic forgetting in neural networks. In National Academy of Sciences, 2017.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In ICCV, 2011.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module efficient video understanding. In ICCV, 2019.
Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, and Thomas Hi. Revisiting temporal modeling for clip-based smage-to-vides knowledge transferring. In CVPR, 2023.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. 2022.
Sachin Mehta and Mohammad Rastegari. Separable self-attention for mobile vision transformers.
Arxiv e-prints, 2022.
Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. Arxiv e-prints, 2023.
Bolin Ni. Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng. Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recog- nition. In ECCV, 2022.
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. In NeurlPS, 2022.
Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Dual-path adaptation from image to video trans- formers. In CVPR, 2023.
Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songcen Xu, Peng Dai, Juwei Lu, and Youliang Yan. Clipping: Distilling clip-based models with a student base for video-language retrieval. In CVPR, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Fine-tuned clip models are efficient video learners. In CVPR, 2023.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. Arxiv e-prints, 2014. 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. Arxiv e-prints, 2012.
Frederick Tung and Greg Mori. Similarity knowledge distillation. In ICCV, 2019.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. Arxiv e-prints, 2021.
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. CVPR, 2023.
Zejia Weng. Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-velip: Transforming clip to open-vocabulary video model via interpolated weight optimization. In ICML, 2023.
11

---

Published as a conference paper at ICLR 2024.
Wenhao Wu, Zhun Sun, and Wanli Ouyang. Revisiting classifier: Transferring vision-language models for video recognition. In AAAI, 2023.
Wangmeng Xiang, Chao Li, Yuxuan Zhou, Biao Wang, and Lei Zhang. Generative action description prompts for skeleton-based action recognition. In ICCV, 2023.
Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. In ICLR, 2020.
Taojiannan Yang. Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. Arxiv e-prints, 2023.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICMI, 2017.
Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar, Learning video representations from large language models. In CVPR, 2023.
12

---

Published as a conference paper at ICLR 2024.
A APPENDIX
A.1 MORE EXPERIMENTAL DETAILS
Datasets. We evaluate our method using the common UCF-101 (Soomro et al., 2012), HMDB- 51 (Kuehne et al., 2011), Kinetics-400 (K-400) (Carreira & Zisserman, 2017), Kinetics-600 (K- 600) (Carreira et al., 2018), and Something- to-mmmihhnn- V2 (SSv2) (Goyal et al., 2017) datasets. (1) K-400 & K-600 are large-scale action recognition datasets, containing 400 and 600 action classes, respectively. K-400 includes 240k training and 20k validation samples, while the K-600 dataset is an extension of the K-400, which has 410k training and 29k validation samples. K-600 contains 220 non-overlapped categories compared to K-400. Each sample a YouTube video clip representing a human action. (2) UCF-101 consists of 13,320 video clips from 101 action classes, where 9,537 samples are used as the training set and the remaining 3.783 samples are used for testing. Offi- cially, there are three training/testing splits. The dataset includes variety of human activities and widely used for benchmarking in human action recognition research. The videos vary in length but are generally short. (3) HMDB-55 (Kuehne et al., 2011) includes 6,849 videos from 51 action classes and has more than 101 samples per class. Similar to UCF-101, there are also three official training/testing splits. The dataset comprises a diverse range of sources, including movies, public databases, and YouTube videos, offering a broad range of human actions and interactions. (4) SSv2 contains 174 fine-grained action classes, which are mostly related actions performed with daily objects. In total, there are 168,915 video clips for training and 24,777 video clips for testing. This dataset is unique in its focus human-object dynamics, thereby is known temporal--caallenging dataset.
Evaluation metrics. We follow the literature (e.g., Rasheed et al. (2023); Ni al. (2022); Weng et al. (2023)) to conduct experiments under the two experimental settings. For the base-to-novel setting, we conduct experiments on UCF-101, HMDB-51, K-400, and K-600, and report the aver- age top-1 accuracy. The action classes are separated into base and novel classes for each dataset. Frequent classes are used as the base classes while less frequent classes are used as novel classes. The models are trained on samples from base classes in the raw training split in each dataset and evaluated on samples from novel classes in the raw validation split. 16 video clips are sampled for each base class for training. Three different base sets are constructed for each dataset and used to train the model separately. The resulting models are evaluated on the same novel set and we report the average results using models trained on three different base sets. During testing, HMDB-51 1 and UCF-101 datasets have three validation splits in the raw data. However, in the base-to-novel setting being used here, only the samples from novel classes in the first split are used for evaluation. On the other hand, K-400 and SSv2 datasets have only one validation split in the raw data, and the samples from the novel classes in the entire split are used for evaluation here. For the cross-datesel setting, the models are trained on K-400 (Carreira & Zisserman, 2017), and evaluated on UCF-101 (Soomro et al., 2012). HMDB-51 (Kuehne et al., 2011), and K-600 (Carreira et al., 2018). For HMDB-51 and UCF-101, the methods are evaluated using their respective three validation splits in the raw data, and we report the average top-1 accuracy on these splits as well as the performance variance. For K-600, the methods are evaluated on the 220 categories that do not exist in K-400. We report the average 1op-1 accuracy over three randomly sampled splits used in Ni et al. (2022); Rasheed et al. (2023); Weng et al. (2023), with each split containing 160 categories.
Training configurations. The initial learning rate set to 3.33 x 10-6 and is decayed using the cosine scheduler. For base-to-novel evaluation, we train each model for 12 epochs and set the first 2 epochs for warming up. Differently, for cross-dataset evaluation, since we have larger training data, train the models for 22 epochs with the first 2 epochs as warm-up. The hyper- pper-pparaettrss and 3 are set as 0.1 and 2. During training, each video is uniformly sampled with 8 frames. During testing, we sample 3 video clips (8 frames per clip) with crop ("3 x 1" views) of each video and ensemble the outputs with an average summation. Following Open-VCLI (Weng et al., 2023), when testing, average the models learned in different epochs to improve generalizability for the cross-dataset setting. We use 8 x A100 GPUs to conduct all the experiments.
13

---

Published as a conference paper at ICLR 2024.
A.2 DIAGNOSTIC EXPERIMENTS
Impacts of the zero initialization. As shown in Tab. 5, notice that initialization can improve the performance since it enables a starting point just like the pretrained CLIP which gets updated smoothly.
Impacts of the weight a. As shown in Tab. 6, we conduct experiments by setting a to 1.0, 0.5, 0.1, and 0.05, respectively. We find that smaller a generally obtains better results, which indicates that the video-specific learning should be well-constrained. For the performance trade-off on different datasets, we set a 0.1.
Table 6: Impacts the value of a.
Table 5: Impacts of the zero initialization W2-
UCF*
HMDB*
K-600
Method
JUFF*
HMDB*
K-600
1.0
84.7
53.1
73.2+1.0
Ours w/o Zero Init. 84.7
54.1
1..4
0.5
84.8
53.9
74.0 :00.9
Ours w/ Zero Init. 85.0
54.5
74.8+-0.9
0.1
85.0
54.5
74.8+0099
0.05
85.1
54.8
74.
Table 7: Impacts of the balancing coefficient /3 in the loss function.
UCF*
HMDB*
K-600
84.7
54.3
7..0:
85.0
54.8
74.8+:0.9
85.0
54.1
7..0+
Impacts of the coefficients 3. In Tab. 7, we experiment with different values for 3 by setting it to 1, 2. and 3, respectively. We can see that the variance among different values is not large and choose to be our default choice.
14

---

Published as a conference paper at ICLR 2024.
A.3 VISUALIZATION RESULTS
In Fig. 6-Fig. present more qualitative comparison. Overall, model attends informative regions related to the action for more reliable recognition. For example, in Fig. 6, our method effectively captures the movements of the woman's mouth extract the motion features relevant to "chew"; in Fig. 7, our model attends to the human legs when recognizing "push".
Inputs
CLIP
VCLIP
FROSTER
Ches
Figure 6: Attention maps for "chew". (1) The frozen CLIP attends to the background. (2) The tuned
VCLIP attends to the face and forehead. (3) FROSTER attends to the mouth, which is more relevant to the action.
Imputs
Froeen CLIP
Push
Figure 7: Attention maps for "push". (1) The frozen CLIP attends to the surrounding stuff around the baby. (2) The tuned VCLIP attends to the clothing of the baby. (3) FROSTER attends the legs of the baby, which are more relevant to the action.
15

---

Published as a conference paper at ICLR 2024.
Inputs
Frosen
CLIP
FROSTER
Fexing
Figure 8: Visualization of attention maps for action "fencing". (2) The frozen CLIP attends the background, egg. the ground. (2) The tuned VCLIP learns to focus the moving body parts. (3) FROSTER attends to the arms and legs, which are more relevant to the action.
Imputs
U
LOLL
L&B
Frum CLIPP
VCLIP
FROSTER
Hit
Figure 9: Attention maps for "hit". (1) The frozen CLIP attends to the ground and sky. (2) The tuned VCLIP attends to the person's 's pelvic region and the ground. (3) FROSTER attends to the person's hands and the hammer, which more relevant to the action.
16


Reviews:
[{"review_id": "6WSt8HPEEw", "reviewer": "Reviewer_ZdtY", "summary": "The paper addressed the problem of adapting image-based CLIP models to open vocabulary action recognition in videos. The starting point of this paper is an observation that adapting CLIP on a particular video dataset will likely reduce its performance on other distinct datasets. Based on this observation, the authors proposed a method that combines a variant of feature distillation (Residual Feature Distillation) with standard cross entropy loss for adapting CLIP to videos. The proposed method can be attached to existing adaptation methods, and demonstrated strong empirical results on public benchmarks.", "questions": "I am a bit confused by how the results were reported for the proposed method in Table 1. Sec 4.2 (implementation details) states that “Otherwise stated, we use VCLIP for conducting experiments.” Is the proposed method (FROSTER) also built on VCLIP in Table 1? If so, VCLIP should be included as a baseline here. If not, a description should be included.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper addressed an important and trendy topic on adapting CLIP models for video recognition tasks. \n\n* The paper is fairly well written. Technical details are clearly described and easy to follow.\n\n* The proposed method is well motivated, and offers a reasonable solution.\n\n* The experiments are extensive. The results are solid.", "weaknesses": "My main concern lies in the technical innovation of the paper. The proposed feature distillation falls into the framework of knowledge distillation, which has been extensively explored for adapting pre-trained models. While I agree that many prior works are insufficient for the objective of this paper, there are a few very recent papers that share the key idea of using feature distillation to adapt CLIP models for video tasks (see e.g., [a]). With this prior work, the technical innovation of the paper seems a bit incremental.  \n\n[a] Pei et al., CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval, CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a5B72u5A9Y", "reviewer": "Reviewer_zWh1", "summary": "This paper proposed FROSTER as an open-vocabulary action recognition framework. Due to the gap between image and video domains, it is difficult to fine-tune CLIP model on video datasets without hurting its generalization ability. To this end, FROSTER employs the knowledge distillation strategy by treating the frozen CLIP as a teacher. The proposed method is evaluated on both base-to-novel and cross-dataset settings, and achieves better accuracy than existing methods without knowledge distillation on all datasets.", "questions": "Please see the weaknesses. The authors should clarify the difference from the previous knowledge distillation works and add the corresponding subsection in the related work.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper has a good motivation to explore how a good open-vocabulary classifier transfers CLIP to the video domain, which is clearly shown in Figure 1.\n2. The experiment is conducted comprehensively on 5 popular video datasets in terms of different network architectures.\n3. The proposed method is practically useful and should be not difficult to implement.\n4. The authors promise to release the source code.", "weaknesses": "1. I have concerns on the novelty. Knowledge distillation on CLIP is not new in this field. The authors didn't discuss related references as follows. Besides, the authors almost follow the classic knowledge distillation framework except for residual feature distillation.\n[a] CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval, CVPR 2023.\n[b] Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation, ACL 2022.\n2. The authors didn't employ compact student model as in normal knowledge distillation. I am wondering if the performance is better if we use ViT-B/32 as teacher and ViT-B/16 as student.\n3. In Fig. 1, why CLIP performs more stable on larger Kinetics dataset than HMDB51 or UCF101?\n4. What is the contribution of text rephrasing in Section 4.2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTTH7pqSy6", "reviewer": "Reviewer_FMt3", "summary": "This paper presents a CLIP-based framework for open-vocabulary action recognition, which distills the CLIP features by treating it as a strong teacher.", "questions": "Please refer to the Weakness. In addition, I still have a few questions:\n\n1. The results in Table 1 are a little bit confusing, as the \"Top1 accuracy\" is only 77.8 on base categories. It would be better to further clarify the training and evaluation protocol, and the number of clips/crops used.\n\n2. As the paper employs rephrased action descriptions, instead of action names, it would be interesting to show the video-text retrieval results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. \n\n2. The proposed Residual Feature Distillation is interesting and inspiring and outperforms the previous methods by a large margin (As shown in Table 1 and 2).", "weaknesses": "1. It would be better to report the finetuned results on all datasets (including K-400) for a better comparison.\n\n2. The performance gains compared to baseline methods (in Table 1 and 2) are indeed surprising, and it would be better to visualize the learned embeddings on both base and novel datasets to see the effects of RFD on feature learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AGrYWgpD6V", "reviewer": "Reviewer_RLyA", "summary": "In this paper, the authors present FROSTER, a clip-centric approach tailored for open vocabulary action recognition. They propose a simple but effective residual feature distillation and the proposed method achieves state-of-the-art results on five datasets in both base-to-novel and cross-dataset scenarios.", "questions": "Please refer to 'weakness'.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed residual feature distillation is simple but effective. \n\n2. The proposed method achieves promising performance on five datasets under both cross-dataset and base-to-novel settings.", "weaknesses": "My major concerns are on the experimental part. \n\n1. The authors touch upon \"Text rephrasing\" in Sec. 4.2, yet there's an absence of any ablation study related to it.\n\n2. In the caption of Table 1, the authors mentioned that 'The results of most of the other papers are taken from Open-VCLIP and\nViFi-CLIP.', where are the results for Open-VCLIP? From Table 2 and Table 3, if my understanding is correct, the reported results use the Open-VCLIP as the backbone. Hence, showcasing Open-VCLIP's results in Table 1 becomes critical to underscore the effectiveness of the suggested method.\n\n3. It would be beneficial to incorporate columns for \"encoder\" and \"pretrained dataset\" within the table for clearer comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)