BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Authors: Fei Long
Xiaoou Li
Jiaming Lv
Haoyuan Yang
Xianjun Cheng
Peihua Li
Venue: ICML 2025 poster (2025)
Keywords: Action recognition
video-language alignment
Brownian distance covariance
CLIP
Primary area: general_machine_learning->representation_learning
TL;DR: We adapt CLIP for action recognition based on Brownian distance covariance (BDC) using all visual and textual tokens. It addresses limitations of previous arts relying on cosine similarity, highlighting significance of advanced metrics like BDC.

Abstract: Bridging contrastive language-image pre-training (CLIP) to video action recognition has attracted growing interest. Human actions are inherently rich in spatial and temporal contexts, involving dynamic interactions among people, objects, and the environment. Accurately recognizing actions requires effectively capturing these fine-grained elements and modeling their relationships with language. However, most existing methods rely on cosine similarity--practically equivalent to the Pearson correlation coefficient--between global tokens for video-language alignment. As a result, they have limited capacity to model complex dependencies and tend to overlook local tokens that encode critical spatio-temporal cues. To overcome these limitations, we propose BDC-CLIP, a novel framework that leverages Brownian Distance Covariance (BDC) to align visual and textual representations. Our method can capture complex relationships--both linear and nonlinear--between all visual and textual tokens, enabling fine-grained modeling in space, time, and language. BDC-CLIP achieves state-of-the-art performance across zero-shot, few-shot, base-to-novel, and fully supervised action recognition settings, demonstrating its effectiveness and broad applicability.

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Fei Long") Xiaoou Li Jiaming Lv
Haoyuan Yang'
Xianjun Cheng
Peihua Li
Abstract
Avg Pool
Cerrclakes
Bridging contrastive language-image pre-training (CLIP) to video action recognition has attracted growing interest. Human actions are inherently rich in spatial and temporal contexts, involving dynamic interactions among people, objects, and the environment. Accurately recognizing actions requires effectively capturing these fine-grained elements and modeling their relationships with language. However, most existing methods rely on cosine piruclrityypructically equivalent to the Pearson correlation ceeffccientteetweee global tokens for video-language alignment. As re- sult, they have limited capacity model com- plex dependencies and tend to overlook local to- kens that encode critical spatio-temporal cues. To overcome these limitations, we propose BDC- CLIP, a novel framework that leverages Brown- ian Distance Covariance (BDC) to align visual and textual representations. Our method can capture complex relationships-bee both linear and nonlinear-between all visual and textual tokens, enabling fine-ge mne-grained modeling in space, time. and language. BDC-CLIP achieves statc-of-the- art performance across zero-shot, few-shot, base- to-novel, and fully supervised action recognition settings, demonstrating its effectiveness and broad applicability.
Temeocal
[CLS]
(FOSS
por tokeess
(POSS
Parca sotune
Wood tocome
Prach rolira
Word tokire
Visaa
Vitzal
Doceder
Uncreerr
Eaowder
(playing palle)
palko
in villo
is vidoo
(a) Prior arts.
(b) Our BDC-CLIP.
Figure I. Key differences between prior arts (a) and our BDC- CLIP (b). Previous methods align vidco and language based on cosine similarity between average of frame-level [CLS] tokens in video and sentence-level [EOS] token. This limits the alignment to coarse semantic matching. In contrast, BDC-CLIP aligns the two modalities via Brownian distance correlation using all visual tokens and all textual tokens, which can capture fine-gre ine-grained spatio- temporal cues crucial for action recognition.
recognition capabilities. In particular, adapting CLIP to image-centric downstream tasks has driven significant ad- vances (Gu et al., 2022; Zhou et al., 2022). Motivated by this success, researchers have turned their attention to similarly powerful models for video-language alignment. However, training such models from scratch remains impractical (Ni et al., 2022; Rasheed al., 2023) due the limited availabil- ity of large-scale video-text datasets and the prohibitive com- putational costs of video processing. Consequently, recent efforts focus on repurposing image-pretrained CLIP models for video-centric tasks, yielding promising progress (Mo- meni et al., 2023; Huang et al., 2024; Kim et al., 2024).
1. Introduction
Multimodal foundation models, such as contrastive language-image pre-training (CLIP) (Radford al., 2021), have demonstrated remarkable zero-shot and few-shot
'These authors contributed equally to this work. SScoool of In- formation and Communication Engineering, Dalian University of Technology, Dalian, China. "School of Artificial Intelligence, Bei- jing University of Posts and Telecommunications, Beijing, China. Correspondence to: Peihua Li <peihuali @ diut.edu.cn.>.
Human actions exhibit rich contextual information when captured in video and described by human language, in- vol volving dynamic interactions among people, objects, and the environment (Rasheed et al., 2023; Chen et al., 2024a). For example, the action (playing polo} features players swinging mallets on horseback, galloping across a grass field, with the scene continuously evolving (see Figure 5).
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
These fine-grained elements, which are encoded as image regions within frames (Dosovitskiy et al., 2021; Caron et al., 2021) and key words in textual descriptions, are mapped via CLIP to a shared embedding space alongside corresponding textual descriptions. Effectively capturing and distinguish- ing these elements while modeling their relationships with language is crucial for robust video action recognition.
We introduce Brownian Distance Covariance (BDC) for multimodal alignment in foundation models such as CLIP, going beyond the limitations of cosine similarity, BDC enables the model to capture complex statistical dependencies in the video-language embedding space.
We propose a temporal BDC attention that captures patch- wise importance and temporal dynamics, along with a language-side BDC representation derived from all tex- tual tokens. This enables fine-grained multimodal con- text modeling across space, time, and language.
However, most existing methods suffer from two primary limitations in modeling video-language relations. First, they rely on cosine similarity to align video and language. Since cosine similarity is practically equivalent to the Pearson correlation coefficient (Zhelezniak et al., 2019), it can only capture linear relationships and fails to model more complex, nonlinear dependencies. Second, these methods typically focus on global tokens-that is, the average of frame-level [CLS] tokens for video and the sentence-level [EOS] to- ken for text-while neglecting local tokens (i.e., patch to- kens and word tokens) that encode fine-grained elements in video frames and textual descriptions. As a result, these approaches are limited to coarse semantic matching and lack the capacity to capture the detailed spatio-temporal cues that are essential for understanding actions in videos.
Our method achieves strong performance across a range of video recognition tasks, including zero-shot, few-shot, base-to-novell and fully supervised recognition, demon- strating its ability to capture subtle spatio-temporal cues critical for video action understanding.
2. Related Works
Recently many methods have been proposed for adapting image-pretrained CLIP to action recognition. They can be roughly grouped into two categories: the methods based on frozen CLIP encoders and those fine-tuning CLIP encoders.
To address these limitationss, we propose BDC-CLIP, novel framework for video-language alignment based on Brown- ian distance covariance (BDC) (Székely& & Rizzo, 2009). BDC overcomes the limitations of existing methods in two key ways. First, unlike cosine similarity, BDC can cap- ture both linear and nonlinear correlations, enabling it to model the complex dependencies between video and lan- guage embeddings. Second, BDC naturally models rela- tionships between sets of embeddings by treating them as random vectors. allowing it to fully leverage both global tokens and local tokens. This makes BDC particularly well- suited for fine-grained video-language alignment, where understanding the relationships between detailed visual and linguistic elements essential.
Aaatiin frozen (*) CLIP encoders This research di- rection focuses on designing learnable prompts or adapters attached to frozen CLIP encoders. Vita-CLIP (Wasim et al., 2023) learns frame-level and video-level prompts alongside summary video prompt for the vision encoder, as well as linguistic prompts for the text encoder. ST-Adapter (Pan et al., 2022) plugs light spatial-temporal adapters into the visual encoder. DUALPATH (Park et al., 2023) extends the visual encoder with a dual-branch architecture and intro- duces lightweight adapters to perform spatial and temporal modeling independently. CAST (Lee et al., 2023) also uses two-stream architecture, with the CLIP visual encoder as the spatial expert and VideoMAE (Tong et al., 2022) as the temporal expert. It introduces a bottleneck cross-attention to enable interaction between the spatial and temporal streams. However, all these methods require backpropagation (BP) through entire encoders, making them To improve efficiency, BP-free methods have been proposed. A5 (Ju et al., 2022) connects frame-level temporal adapter to the visual encoder, and meanwhile learns textual prompts. EVL (Lin et al., 2022) designs an efficient transformer de- coder connected to visual encoders for capturing temporal cues. DiST (Qing et al., 2023) disentangles temporal learn- ing via a light temporal encoder and spatial learning via CLIP encoder. MoTED (Qing et al., 2023) aligns the spa- tial embedding output by the frozen visual encoder with text description of categories, and also aligns temporal em- beddings output by the light motion encoder with motion descriptions of categories. RIVA (Qian et al., 2024) employs slot attention to extract compact object tokens from frozen
Figure 1 illustrates the key differences between our approach and prior methods. BDC-CLIP introduces two core com- ponents: a video BDC adapter and a text BDC adapter. In the video BDC adapter, we leverage all visual tokens (i.e., [CLS] and patch tokens) to compute a BDC matrix as a frame-wise representation and design a temporal attention mechanism to model frame-to-frame dynamics. The video representation is then obtained by averaging these frame- wise features. On the textual side, we exploit all textual tokens (i.e., [EOS] and word tokens) to compute BDC matrix as the text representation. Finally, we align the video and text representations using Brownian distance correlation (BDCorr), a normalized metric that is invariant to orthogo- nal, translational, and scaling transformations.
The primary contributions of this paper are summarized as follows:
2

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Word token
[EOS]
BDC Manri
LN Linear for Dim Roduction
Token Weighting
BDC Matrices Computing
Encoder
(M
Describe the action pplaying polo) video.
Text BDC Adapter
Browitian Distunce
Carrstaton
OF
Linear
dooc f
Visual Envoder
IN Linear Ror Dim Reduction
Arg
Linear
Token Weighting
BDC Matrices
Patch
[CLS]
tokens
7 DV i X
Compuring
Video BDC Adapter
7x
BDC Matricus
Figure 2. Illustration of BDC-CLIP adapting image-pretrained CLIP for action recognition in videos. Our key idea is Brownian Distance Covariance (BDC) for video-language alignment. Specifically, we develop video BDC adapter that models temporal relation of frame-wise BDC matrices computed from weighted visual tokens. Similarly, we introduce a text BDC adapter that computes BDC matrix using weighted textual tokes as language representation. To decrease computations, we use linear layer for dimension (dim) reduction hefore the BDC adapters. As in previous arts, we use L.MMs to augment simple category names. Besides the BDC alignment, we also introduce a separate vision classifier (not shown for simplicity), attached to the video BDC adapter for further improvement.
pretrained models and performs temporal reasoning through object-time interactions.
2014) in tasks such as network comparison, disentangled representation learning, and improving robustness in adver- sarial learning. DeepBDC (Xie et al., 2022) applies the BDC metric to measure similarities between query images and support classes for few-shot image classification. Similarly, BDC-tda (Zhang et al., 2023) utilizes a BDC-based nearest neighbor classifier for pure vision prediction, which is integrated with vision-language prediction for final clas- sification. Our work is inspired by both DeepBDC and BDC-Adapte but is distinguished by two key points: (1) we extend the BDC metric to match two distinct modalities (i.e., vision and language), rather than limiting it to uni- modal tasks (i.e., vision only); and (2) we focus on video action recognition, introducing a novel BDC attention mech- anism model temporal cues, whereas prior methods focus image recognition that requires temporal modeling.
Fine-tuming (0) CLIP encoders This category focuses on fine-tuning CLIP's vision and text encoders for video tasks. ViFi-CLIP (Rasheed et al., 2023) simply finetunes both encoders with minimal design changes. It efficient and strong, even better than previous methods with specific adapters model temporal cues, e.g., ActionCLIP (Wang et al., 2)2 and XCLIP (Ni et al., 2022). As such, the follow-up works are often based on the fine-tuned encoders. FROSTER (Huang et al., 2024) distills the fine-tuned mod- els with the frozen CLIP models to enhance generalization ability. OST (Chen et al., 2024a) augments action class Descriptors, which
names with LLMs into aligned frame-level representations based on optimal transport. TC-CLIP (Kim et al., 2024) proposes method of temporal contextualization. which summarizes informa- tive tokens for infusing temporal cues and also serves as video-conditional text prompts. Open-VCLIP VCLIP (Weng et al., 2023) introduces regularization via weight interpolation (II- harco al., 2022) to balance adaptation and generalization capabilities.
3. From Cosine Similarity to BDC
To facilitate analysis, we first introduce some notations. Let P = [Po...., pN] E [d denote the N + visual token embeddings of d dimensions for an image, where Po indicates the [CLS] token and the remaining entries indicate patch tokens. Similarly, let W= w.]] E pdx(M+1) denote the textual token embeddings, where Wo indicates the [EOS] token and the rest indicate word tokens.
BDC for deep learning The concept and theory of BDC were established in (Székely & Rizzo, 2009) to measure the dependence between sets of random vectors. BDC is computationally efficient, capable of modeling both linear and non-linear correlations, and provides complete charac- terization of the independence of random variables. These favorable properties have motivated researchers explore its applications in deep learning. Zhen et al. (2022) inves- tigate the use of BDC and partial BDC (Székely & Rizzo,
3.1. A Statistical Perspective on Cosine Similarity
As described in (Zhelezniak et al., 2019), the similarity between tokens can be interpreted from statistical per- spective. Specifically, each d-dimensional token embedding can be regarded as a sample of d observations from a ran-
3

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
where Bdc-M(-) computes symmetric matrix whose (i, th entry is the Fuclidean distance between the i-th and j-th rows of P. adjusted by subtracting row mean, column mean, and the global mean. The BDC matrix for textual tokens, Bix, is computed analogously. The BDC metric is then given by:
Frequency
0.08
Fraquency
0.06
0.04
0.04
0.92
0.02
0. 0.3
011 00
01 02 03
0.00
0.
03
0.2-00.-000 011
0.0
01 02 D4
Textual token embeddings
Visual taken embeddings
1 BDC BDug Bui)
tr trug Bua),
(3)
Figure Distributions of token embeddings.
d2
where tr(-) denotes the trace of matrix. The BDC metric can capture arbitrary statistical dependencies both linear and nonlinearwwithout requiring distributional assumptions.
dom variable. When these samples are nearly zero-centered, cosine similarity (CS) becomes approximately equivalent to the Pearson correlation coefficient (PCC). This property holds for the CLIP model, as evidenced by the near-zero mean distributions in Figure 3.
In the next section, we extend this framework to align videos with textual descriptions. By modeling complex dependen- cies across all visual tokens (across frames) and all tex- tual tokens, our method effectively captures fine-gra fine-grained semantics such as salient regions and key words critical for video understanding. This is further supported by Grad- CLIP (Zhao et al., 2024) visualizations in Figures 5 and 7.
Most existing CLIP adaptations rely on computing the CS between the [CLS] token Po and the [EOS] token Wo, which can be interpreted as samples drawn from two random vari- ables, Ring.o and Rus,o, respectively. Thus, we have:
E E Rest BBR
CS CS Wo) 2 PCC Wo)
4. Proposed BDC-CLIP
ing. D (B(RRa.)
The framework of BDC-CLIP, as shown in Figure 2, mainly consists of video BDC adapter and text BDC adapter. The two adapters output respectively video and language representations, aligned by Brownian distance correlation.
where E(-) denotes expectation over the random variable. However, PCC is limited to capturing linear relationships, and is optimal only when the joint distribution is Gaussian. It cannot capture more complex, nonlinear dependencies. As illustrated in Figure 4, the correlations between visual and textual tokens are highly nonlinear and non-Gaussian. Con- sequently, methods that rely solely on CS or PCC are fun- damentally limited in modeling such dependencies. More- over, these methods often ignore fine- ane-ggrieed information encoded in patch and word tokens across both modalities.
4.1. Video-Language Alignment with BDC Metric
Consider a video sample consisting of T frames and its cor- responding textual description indicating an action category. These inputs are processed by the visual and text encoders, respectively. Let P' = [pó p'vl E Idd x(N+1) be the visual token embeddings of the t-th frame output by the visual encoder. Let W = [Wo.... wM] E R x(M++) be the textual token embeddings from the text encoder.
3.2. BDC for Modeling Statistical Dependency
To overcome these limitations, we propose to use BDC to match vision and language modalities by considering all to- kens jointly. We treat each visual (resp. textual) embedding p: (resp. W,) as sample of d observations drawn from random variable Ring, (resp. R. We then define the random vectors:
Video representation To decrease computational cost, introduce linear layer for dimension reduction:
P = LLNee (.))))
(4)
where Pp. P .......,''] (R) LN denotes layer normalization, and Linear denotes a linear transformation reducing the feature dimension from d' to d (d < d').
Rang Rmim.. Rmmg. I RRix,, ..... M
We construct the video representation using all visual to- kens per frame. Notably, different patch tokens contribute differently video recognition tasks. For instance, certain patch tokens may capture essential parts of the subjects (e.g.. persons or objects), while others may represent background or irrelevant regions. To quantify token importance, we introduce a weighting mechanism based on the similarity of each token to the [CLS] token. Since the [CLS] token is designed to encapsulate the holistic representation of a
As formalized in Szěkely & Rizzo (2009), BDC measures the discrepancy between the joint distribution of Rang and Rum and the product of their marginals. For the discrete case, the BDC metric can be computed in closed form using the so-called BDC matrices. For the visual modality, the BDC matrix B Bing E dd is computed as:
B Bing Sdc-M(PPo.... .NN)),,,
4

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
frame, is reasonable to assign greater importance to tokens with higher similarity the [CI.S] token. The weighted embedding of patch token is computed as:
By leveraging all visual and textual tokens, BDC-CLIP captures fine-grained relationships across spatial, temporal, and linguistic domains. In contrast, cosine similarity-based methods capture only coarse, linear correlations between global tokens.
P P
w - Softmax
pb pl/Va),
(5)
where denotes the dot product of two vectors, and Sof tmax the softmax function. The BDC matrix for the f-th frame computed as:
4.2. Training Objective
In addition to aligning video and text BDC representations, we introduce a vision classifier built upon the video BDC adapter, leveraging ground-trute labels for supervision. Con- sistent with prior works (Chen et al., 2024a; Zhang et al., 2024), we retain the backbone loss, i.e., CLIP's original contrastive loss, which directly aligns the embeddings from the visual and textual backbone encoders.
B ing = (dc .. .Pi)),
(6)
For efficient processing, we perform salf-vectorizzation (Vech) for B' by stacking the entries on and below its di- agonals, obtaining compact vector b Ding Vech (Bing) Id(d+1)/2
Let Bua(c) be the textual BDC matrix of the c-th category, and let (c) (000=BDCoor(BBBr Bua()). Additionally, let
Then, we learn temporal relationships across frames with an attention mechanism. Let the values be V ibg b bimg: ... bing). We feed the frame-wise [CLS] to- kens [P)... ... P6 to two separate linear layers, achieving the queries Q E [R and keys K E gdxT, respectively. The attention module outputs:
y = [Mc]e= be the ground-truth probability vector, where C is the number of categories. The Vision-Language Con- trastive (VL-Ctr) loss for the BDC adapters defined as:
exp(SBoc(c)/T)
L
VL.Cr Uudupter
Σ Vo log
e==1 exp (xp(Ssoc(c')/T) ( (11)
Attention K, VJ.
where is learnable temperature parameter.
For the vision classifier, the half-vectorized video represen- tation Bvid is passed through fully mollecconnected layer param- eterized by R for softmax classification. The classification loss is given by:
Finally, we perform average (Avg) pooling across frames and obtain the video-level representation:
Buid Sym-h(Avg hcAvgg-ool(((() (V)),
where Sym-h h reconstructs a symmetric matrix from a vector.
L
V-Cls ulapter
Σ yo LoogSSofttmaxeecceBBBiiiii
(12)
Text representation The computation for text representa- tion mirrors that of video representation, but without tem- poral attention. Given textual tokens W, we first achieve the embeddings of reduced dimension W E dxx through a layer normalization and a linear layer, which are then weighted in terms of their similarities to the [EOS] token Wo for obtaining W;, i = 0, .. The BDC matrix for text representation is then computed as:
where (+)c denotes the e-th component of the softmax out- put. This supervised loss encourages learning of more dis- criminative and generalizable features.
The backbone loss is based on cosine similarity (Rasheed et al., 2023):
exp(Scos(c)/1) (xpp()on00)/1))_) (13)
I backhoone
Σ ye log
o' (c)()7)
1ix = Bdc-M(( M(woM ... Wad).
(9)
where (eos(C) cos(Avg pool cos(Avg-pool ((o](())]T_.))) the [EOS] token of the e-th category, and 7" is learnable temperature parameter.
Wo(c)), wo(c) is
lasguala alignment and text representations is measured using Brownian Dis- tance Correlation (BDCorr):
The similarity between video
The overall loss function combines the three components:
tr (Bvid Brx)) BDCorr Brx)= /t / tr CTrB Brx)
Liotal
L
adapter + L adupter + L Lackbone
(14)
(10)
5. Experiments
The values of BDCorr lie in the interval [0, 1]. where a value of zero indicates that the two modalities are independent of each other. is worth mentioning that BDCorr is invariant to orthogonal, translational, and scaling transformations.
We first describe the experimental setup ($5.1). Then we compare to sfate-of-the-art methods in light of perfor- mance ($5.2) and cost ($5.3). We finally conduct ablation study ($5.4) and provide qualitative analysis ($5.5).
5

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
5.1. Experimental Setup
Table I. Comparison to previous methods zero-shot setting using 14000pretrained models. WSE indicates Weight-Space Ensem- bling: CLIP Backbone Encoders (BEs) are frozen (+8) or fine- tuned ). The best results are bold and the second-best ones are underlined. indicates results reproduced by us.
Datasets and task setting We conduct experiments on five widely used action recognition datasets, i.e., Kinetics- 400 (K400) (Carreira & Zisserman, 2017). Kinetics-600 (K600) (Carreira et al., 2018), HMDB-51 (Kuehne et al.,
2011), UCF-101 (Soomro et al., 2012) and Something Some- thing V2 (SSv2) (Goyal et al., 2017). Following (Rasheed et al., 2023; Kim et al., 2024; Chen et al., 2024b; Li et al., 2024), we pretrain on K400 and then evaluate on down- stream tasks in zero-shot, few-shot, base-to-novel general- ization and super-super settings. Few-shot and base-to- novel generalization tasks directly fine-tuned from CLIP are reported in Section B.2 of Appendix.
Method
BEs HMDB-51 UCF-101
K600
K600
(Top-1)
(Top-5)
44.3+22
69.3+42
55.8
81.4aa
Vita-CLIP Wainetdd
48 48.6+a6
75.0
75,0+us
67.4+u
DiST oiugetal
$
55.4
72.3
we WSB
MoTEED Zang et
$
58.2+11
78.3
69.9
ActionC'LII Wangetal.
49.1is4
68.0ine
56.1
83.2
X-CLIP sictal
44.6++5.2
72.0+23
65.2+04
86.1+as
ViFi-CLIP Rashsod ct.al.
52.3++a2
78.9+11
70.7+as
92.1+03
TC-CLIP' et
56.82n9
83.0
75.400
94.7
Implementation We utilize CLIP models with a ViT-B/16 visual encoder and the corresponding text encoder through- out this paper. For text augmentation, following the ap- proach in (Huang al., 2024; Kim et al., 2024), we query GPT-40 to generate enriched textual descriptions for each category name using the prompt: "Describe the action of {category name} in a video." Both temperature parameters T in Eq. 11 and r" in Eq. 13 in the training objective are initialized by 0.07. All experiments are conducted using GeForce RTX 4090 GPUs with the PyTorch framework.
BDC-CLIP (Ours)
59.4
85.9
76.5
95.0
ActionCLII Wang std..
51.9+as
74.2+11
67.5+11
90.7++a
ViFi-CLIPP Rasbood al
52.2++a7
81,0+a9
73.9+us
93.3+aa
Open VCLIP Weng
53.9
81A41a
73.0aus
93.22n1
WSSE
MAXI
52..3.07
78.22ns
71.5
92.5
OST chmmenl at
55.9.11
79.7
75.1
94.6192
FROSTER mronggraa
54.8211
84.8
74.8
TC-CLIP Kin al.
56.0++a
85.4+us
18.1
95.7
BDC-CLIP (Ours)
60.7.aa
78.9
95.9.a2
by 1.2%, 1.5%, 2.3% and 4.3% for 2-, 8- and 16- shot tasks, while performing better on UCF-101 (~1.0%% in each of K Khshot setting. We provide results without pretraining on K400 (i.e., directly finetuned from CLIP) in Table 8.
Detailed experimental setup is provided in Section A.
5.2. Comparison to State-of-The-Ar Methods
(iii) Base-to-novel generalization All methods pretrained K400 are fine-tuned and tested on hase classes, along with evaluation novel classes in zero-shot manner. Ta- ble 3 shows Top-1 accuracies for base and novel classes and their harmonic mean (HM). We can see that BDC- CLIP achieves state-of-the-art performance across all three datasets, with HM exceeding TC-CLIP by 2.6% on HMDB- 51, 2.0% on UCF-101, and 2.1% on SSv2. See Appendix Table for the results without pretraining on K400.
(i) Zero-shot recognition For this task, all models are trained on K400 and then directly evaluated on down- stream datasets in zero-shot manner. Note that recent works (Kim et al., 2024; Huang et al., 2024; Lin et al., 2023; Weng et al., 2023) have exploited weight-space ensembling (WSE) (Wortsman et al., 2022) to boost performance of zero-shot task; see Section A.2 for implementation details. For fair comparison, we report the results with (w/) and without (w/o) WSE. Table 1 presents the comparison results. The results of ActionCLIP and ViFi-CLIP w/ WSE are du- plicated from TC-CLIP. For the case of w/o WSE, the top-1 accuracies of BDC-CLIP are better than the second-best ones by 1.2%, 2.9% and 1.1% in top-1 accuracy on HMDB- 51, 10F-101 and K600, respectively. All methods w/ WSE improve over their counterparts without WSE, particularly on UCF-101 and K600, while our BDC-CLIP still stands out, outperforming the rumners-up by 4.7%, 1.5% and 0.8% on the three datasets. These results suggest that BDC-CLIP has better generalization ability than the competitors.
(iv) suplyisup recognition Following the common practice for closed-set setting, we conduct experiment with standard splits on K400. As in ViFi-CLIP, we sample 16 frames per video clip and conduct inference with video clips and 3 spatial crops (4x3 views). We also apply WSE to this setting. From Table 4, we can see that, without WSE, BDC-CLIP surpasses the previous best performer (TC-CLIP) by 0.4% in top-1 accuracy. Moreover, BDC- CLIP with WSE further improves, achieving top-1 accu- racy of 86.5%.
(ii) Few-shot recognition In this setting, all methods are first trained on K400 and subsequently are evaluated for all-way classification provided with K training examples per class. The comparison results are presented in Table 2. TC-CLIP), our BDC-CLIP outperforms on HMDB-51 by more than 2.7% in each K- shot task, and improves on SSv2
5.3. Computational Cost Analysis
Following TC-CLIP, we analyze the computational cost in terms of parameters, GFLOPS, and throughput (per view), as reported in Table 5. All metrics are normalized to the baseline (ViFi-CLIP), and measured on a single GeForce RTX 4090 GPU. We separately list the costs for BDC-
6

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
HMDB-51
UCI-101
SSv2
Mothod
BEs
Table Comparison to previous methods in all-way K K-shot set- ting with pretraining on K400.
K=2
K=4
K=8
K=16
K=2
K=4
K=8
K=16
K=2
K=4
K=8
K 16
CLIP Kallendara
41.9
41.9
41.9
41.9
63.6
63.6
63.6
63.6
2.7
27
2.7
2.7
46.7
50.4
61.3
65.8
76.3
84,4
90.7
93.0
4.5
6.7
7.2
9.5
CLIP Backbone Encoders (BEs) are frozen (#8) or finetuned (O).
ActionCLIP at
54,3
56.2
59.3
66.1
76.7
80,4
87.6
91.8
4.8
6.9
9.1
12.3
X-CLIPNietd
60,5
66.8
69.3
71.7
89.0
91.4
94.7
96.3
6.6
7.8
9.9
13.7
The best results bold and the
VIFi-CLIP Rushed ct
63.0
65.1
69.6
72.0
91.0
93.7
95.0
96.4
6.7
7.9
10.2
13.5
second-ber ones are underlined.
OST aot
64.8
66.7
69.2
71.6
90.3
92.6
94.4
96.2
8.0
8.9
10.5
12.6
ALT at
64.3
66.7
70.4
74.5
93.2
95.3
96.4
97.3
6.6
7.7
9.4
12.9
The results of directly fine tie-ttning from CLIP are provided in Ta-
TC-CLIP Kin atd.
65.3
68.5
714
73.0
94.1
95.6
96.6
97.3
8.7
10.1
12.1
15.2
BDC-CLIP (Ours)
68.0
71.5
75.9
77.3
94.9
96.7
97.5
98.5
9.9
11.6
14.4
19.5
ble 8.
Table Comparison to previous methods in base-to-novel generalization setting with pretraining on K400. HM indicates the harmonic mean. The results of directly fine-tuning from CLIP are given in Table 9.
Table 4. Comparison to previous methods in filly- supervised setting on K400. i: result with WSE.
Method
BEs
Views
Frames
Top- I
Top-5
Method
BEs
HMDB-51
UCF-101
S8v2
DIST gugera MuTED MoreeDDoreeal Dongeral
*
3
32
85.0
97.0
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
*
3 1
32
86.2
97.5
AIM Yagerd.
3 1
32
84.7
96.7
CLIP Ralladdd
53.3
49.8
78.5
63.6
4,9
5.1
ALT Chas
3
32
85.5
AS jo
70.4
51.7
59.6
95.8
71.0
81.6
12.9
5.7
7.9
Vita CLII etin
4
16
82.9
96.3
ActionCLIP stl.
69.0
57.2
62.6
85.6
75.3
80.1
8.1
8.7
8.4
X-CLIP st
84.7
X-CLIP
75.8
52.0
61.7
95.4
74.0
83.4
14.2
11.0
12.4
ViFi-CLIP Rusbood
4
16
83.9
96.3
ViFi-CIIP et..
77.1
54,9
64.1
95.9
74.1
83.6
15.8
11.5
13.3
TC-CLIP eim
4.x 3
16
85.2
96.9
TC-CLIP Kin
79.4
58.3
67.2
92.5
84,5
90.5
19.6
15.6
17.4
BDC-CLIP (Ours)
4.x
16
85.6
96.9
BDC CLII (Ours)
81.0
61.3
69.8
97.5
88.0
92.5
20.9
18.2
19.5
TC-CLIP Kizs
4.x
16
85.7
97.1
BDC CLIP (Ours)
4 4 3
16
86.5
97.4
Table 5. Comparison of computational cost to previous methods. Throughput measured on single GeForee RTX 4090 GPU.
coders (Rasheed et al., 2023). Based on this, we analyze the role of BDC VL alignment and BDC vision (V) classifica- tion, as shown in Table 6a. It can be seen that, combination of BDC VI. alignment improves significantly over the base- line by 1.8% on K600, 4.1% and 3.5% for 16-shot task on HMDB-51 and SSv2, respectively. These big gains indicate our BDC alignment can more effectively mitigate the do- main gap between the image and video, and have stronger ability for video-language alignment. By further integrating separate BDC vision classifier, the performance improves non-trivially. This suggests that the additional vision super- vision is helpful in learning generalized representations.
Method
Params (M)
GFLOPS
Throughput
ViFi CLIP at
124.3 1ox
285 1,0x
m
ActionCTIIP Wangetal
143.7 1,5x
567
X-CLIP sitaa.
169.7 1.31
288 x
42
Vita-LIPP Wsdmeral
161.8 1030
307 nxx
34030x
OST chereta.
124.3 100x
287 x
42a9ix
TC-CLIP Kinetal.
127.5 x
304 x
33azx
BDC (ZeroS, Ours)
126.9 1
316 L
37mmx
BDC-CLIP (FullS, Ours)
132.0 1,6x
316 L
370m0
CLIP in zero-shot (ZeroS) and fully supervised (FullS) settings. Notably, BDC-CLIP requires only a modest in- crease over ViFi-CLIP in parameters 1.02x-1.06xx and GFLOPS (1.11x), with throughput at 0.80x. Compared to TC-CLIP, a previous top-performing method, BDC-CLIP in- curs slightly higher GFLOPS but achieves faster throughput, demonstrating efficient trade-offs in computational cost.
Metric and representation We compare different metrics and representations. For the BDC, the vanilla method is simple application of DeepBDC (Xie et al., 2022) to videos, where the average of frame-wiss wise BDC matrices is used as a video representation. For the cosine similarity (CS), besides the scheme of global tokens (GlobalT), the local tokens (i.e., patch tokens) can be combined, where averagely pool all patch tokens along with the global [CLS] token as pramfe representation. We also compare to bilinear pooling (Lin et al., 2015) that uses normalized second moment of all tokens as the representations and Frobenius (Frob) distance as the metric. The results are shown in Table 6b.
5.4. Ablation Analysis
To facilitate fast ablation, pretrain on K400- tiny (Rasheed et al., 2023) where each class has 100 training videos, and evaluate zero-shot recognition on K600 and K-shot (K- = 2, 16) recognition on HMDB-51 and SSv2.
We first note that all BDC--ased methods superior to CS-based ones, suggesting that the BDC metric is more suitable for video-language matching. For the CS, the Glo-
Component analysis Our baseline is the original CLIP ladeo-language (VI.) alignment attached to the backbone en-
7

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
LocaL scheme is better than the scheme of single GlobalT, which indicates that usage of local tokens benefits alignment of the two modalities. For the BDC metric, our BDC-CLIP outperforms DeepBDC by 1.1%-1.3% for zero- or 2-shot setting and 2.1 2.1% -300% for 16-shot setting. In the end, we note that bilinear pooling is better than CS while being significantly inferior to BDC-CLIP.
Table 6. Ablation analysis of BDC-CLIP.
(a) Role of components.
K600
HMDB-5 S1
SSv2
Zem-shot
2-shot 16-shor 2-shoe 16-shot
Backbone vision-language align
71.720.0
63.2
69.3
7.7
12.9
+ BDC vision-langunge align
73.5
65.9
73.4
8.7
16.4
+ +BDC vision classifter
73.8
66.1
73.9
8,9
16.8
Dimension reduction (DR) The size of BDC matrices is quadratic of the dimension of token embeddings. To de- crease computations, we introduce linear layer for DR before the BDC adapters. We assess performance as func- tion of the dimension d. As seen in Table 6c, on the whole the accuracies increase consistently as the dimension grows until 320 and then decrease. To trade off the perfor- mance and cost, we adopt d= 192 across the paper.
(b) Effect of metric and representation.
K600
HMDB-51
SSv2
Metric
Representation
Zero-shot
2-shor 16-shor
2-shot 16-shot
Drep
72.5
64.9
13.8
BDC
BDC-CLIP
73,8+0.8
66.1
16.8
GlobalT
71.7+0.9
63.2
69.3
7.
12.9
CS
GlobalT-LoalTT
71.
63.6
13.5
Text augmentation We compare the vanilla prompt tem- plate hand-engineered by CLIP, i.e., "A video of {)", with the prompts generated by LLM. From Table 6d, we can see that LLM enhanced prompts improve non- trivially over the for different settings across
Froh
Bilincariaan
12.1
63.9
71.2
7.6
13.5
(c) Effect of dimension (dim).
Dim d
K600
HMDB-51
S8v2
Zero-sbot
2-shot 16-shot
2-shot 16-shot
vanilla prompts by 0.3%-0.75 the datasets. As such, throughout the paper we adopt the text augmentation via LLM for the category name, unless otherwise specified.
128
72.7+0.8
65.7
73.3
8.4
17.0
192
73.8
66.1
73.9
16.8
256
73.720.8
66.3
73.7
8.4
17.1
320
73.810.a
67.1
74.5
8.8
17.1
384
73.910.7
65.7
73.2
8.7
16.9
Additional ablation study is provided in Section B.I.
(d) Influence of text augmentation.
5.5. Qualitative Analysis
Prompts
K600
HMDB-51
SSv2
Zero shot
2-shot 16-shox
2.shot 16-shox
Why cosine similarity is insufficient Most CLIP adapta- tion methods (e.g., Rasheed er al., 2023; Kim et al., 2024) rely on cosine similarity (CS)-empirically equivalent to the Pearson correlation coefficient (PCC)-for cross-modal matching. However, since PCC only measures linear depen- dence, it cannot capture the higher-order, non-monotonic relations that often arise between vision and language. To investigate this, present scatterplots and density contours for some representative visual- -esuull token pairs in Figure 4. We observe that mroal token relationships are non- linear and the joint distributions are non-Gaussian. In such scenarios, CS collapses disparate patterns to similar scores, whereas BDC cleanly separates matched from mismatched pairs. Recall that in Table 6b we give the lataset-level, apples-to-apples accuracy gains of BDC over CS.
Vanilla
73.810.8
66.1
73.9
16.8
LLM
74.521.0
66.6
74.6
17.3
tively to the key words including 'players' and 'mallets'. This confirms that BDC-CLIP captures fine-grained cues across space, time, and language, leading to better action recognition.
Section B.4 provides 1-SNE visualization of video-language representation and additional heatmap visualizations.
6. Conclusion
We propose BDC-CLIP, a novel framework for adapting image-pretrained CLIP models to video action recognition. By leveraging Brownian Distance Covariance (BDC) and utilizing all visual and textual tokens, our approach captures complex dependencies between video and language in high- dimensional embedding space. This allows BDC-CLIP to effectively exploit fine-grained cues-such as salient regions in video frames and key words in textual descriptions- that are crucial for accurate action recognition. In doing so, it overcomes the limitations of prior methods that rely solely on classical cosine similarity. The strong performance of BDC-CLIP across a range of action recognition benchmarks highlights the value of advanced statistical metrics like BDC for multimodal learning.
Heatmap visualization of video-text pairs To enhance understanding, we visualize attention maps for {playing polo} from the K600 validation set. The models are eval- uated in zero-shot manner using augmented text prompts generated by LLMs for better fine-gran -neigrdi alignment. Grad- ECLIP (Zhao et al., 2024) is employed to visualize the attention maps of both visual and textual modalities. As shown in Figure 5, BDC-CLIP demonstrates a stronger fo- cus on critical regions in the dynamic frames, such as the horse and players. In the meantime, it attends more effec-
8

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
0.15
0-2766
CS: 0.1913
EDC: 0.3268
0.15
8BC: 0.1927
0.15
BDC: 0.1006
0.05
0.05
Visual [CLS] tokeen (Playing
0.00
-0.10
-0.10
-0.10
-0.05 0.00
0.15
-0.35 -0.10 -0.05
0.30
-0.15 -1.1--0.0-0003
0.00
0.05
0.15
0.15
0.05
0.05
-0.30
-0.10
-0.10
-0.15
-0.30
-8.65 6.00
6.30
0.35
-0.15 -0.10 -865
6.00
6.05
0.10
0.36
-6.15
2.100-..000 0.00
0.05
6.10 6.15
Textual [EOS] token of (Playing polo)
Textual [EOS] token of (Playing field hockey)
Textual [EOS] token of (Arguing)
BDC: 0.1605
0.2772
0.10
BDC: 0.1386
C5:
0.2706
0.15
8DC: 0.1031
C5: 0.1880
0.03
token (Playing
0.00
-0.05
-0.05
-0.15
-0.10
-0.15-0.10-0.05 6.05 0.15
-0.15-8.10-0.05 6.05 0.15
-0.10 -0.05 0.00
0.05
0.30 0.15
0.35
0.15
0.30
0.09
0.89
0.80
0.05
-0.05
0.00
-0.30
-0.10
-0.65
-9.15-0.10-0.05 6.55 0.15
-0.15-0:10-0.05 0.00 6os 0.10 0.35
-0.10 -0.05 0.00
0.05
0.10
0.35
Word token of (Ploying polo)
Word token of (Playing field hockey)
Word token of (Arguing)
Figure Scatterplots and density contours of example visual vs, textual tokens (top: global tokens, bottom:
local tokens). The tokens of vision and lanuage exhibit complex relations and their distributions non Non-Gausiian. Compared to CS/PCC, BDC effectively distinguishes matching vs. non-man non-matching text-video tokens the top panel and attenuates irrelevant background tokens in the bottom panel. See Table 6b for dataset-level, apples to-apples accuracy gains of BDC over CS/PCC.
Original Frames
ViFi-CLIP
Blaying anta mvolwes hones mowing
Aleying Bolm mvnless Nonke mow fist playing Bolm meolees horses noving playing Belel Insclves hocsos noving bas
While players using long
while playersinit iit Lusing long.
players ball handied mailees.
while players ball usingl
handledd mallets.
handleed
hand ed mallets.
BDC- 3DC-CLIP(Ours)
Stayingg Bala meoives honssammviing
Blaying Botal menoivss horsesmavinn fast
plasing Sola mnvdeees honsed moxingg while playershitiche bailusing
plasing Bolo insolvess horsas mowing while Bltseers thitch beil
while hitth balllusing long
while alavershiitthe usin long
mallers
Figure 5. Visualizations of {playing polo} using ECLIP Compared to ViFi-CLIP, our BDC-CLIP better focuses on important image regions such the horses players across different frames, while attending more effectively to key words such as 'player' 'mallets'. Best viewed by zooming in.

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Acknowledgments
Chen, T., Yu, H., Yang, Z., Li, Z.,, Sun, W., and Chen, C. OST: Refining text knowledge with optimal spatio- temporal descriptor for general video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18888-18898, 2024a.
This work was supported by the National Natural Science Foundation of China (NSFC Nos. 62471083 and 61971086).
Impact Statement
Chen, Y., Chen, D., Liu, R., Zhou, S., Xue, W., and Peng, W. Align before adapt: Leveraging entity-to- align- ments for generalizable video action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18688-18698, 2024b.
We propose BDC-CLIP, a novel framework for video- language contrastive learning. Our method can capture complex dependencies between video and text in a shared embedding space. BDC-CLIP holds potential for a range of video understanding tasks such as zero-shot action recog- nition and video-text retrieval. We hope this work inspires further exploration of BDC and other advanced statistical distances for fine-grained video-language matching.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Since BDC-CLIP is built upon the pre-trained CLIP model, it inevitably inherits some of CLIP's ethical concerns, such as the amplification of pre-existing biases (e-g-. gender or racial bias) and the potential for malicious use (Radford al., 2021). Moreover, our method relies on LLMs for textual augmentation, it may also be affected by the inherent biases present in these models. We hope acknowledging these potential risks will contribute to development and deployment of more responsible and ethical models.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vi- sion and pattern recognition, pp. 248-255. 2009.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenbomn D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
References
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R. D. MINE: Mutual infor- mation neural estimation. In Internatioonal Conference on Machine Learning, volume 80, pp. 531-540, 2018.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested 101 object categories. In Conference computer vision and pattern recognition workshop, pp. 178-178, 2004.
Bossard, L.,, Guillaumin, M., and Van Gool, L.. Food-101- mining discriminative components with random forests. In European conference on computer vision, pp. 446-461 2014.
Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yiani- los, P., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I., and Memisevic, R. The 'something something" video database for learning and evaluating visual common sense. Proceedings of the IEEE/CVF International Confer- ence on Computer Vision. pp. 5842-5850, 2017.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Gu, X., Lin, T.-Y., Kuo, W., and Cui, Y. Open-vocabulary object detection via vision and language knowledge dis- tillation. In International Conference on Learning Repre- sentations, 2022.
Carreira, J. and Zisserman, A. Quo vadis, action recogni- tion? A new model and the kinetics dataset. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4723-4733, 2017.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-222 2019.
Carreira, J., Noland, E., Banki-Horvathh A., Hillier, C., and Zisserman, A. A short note about Kinetics-600, arXiv preprint Arxiv:1808 0/040,2000 2018.
Chen, S. and Huang, D. Elaborative rehearsal for zero- shot action recognition. In Proceedings of the JEEE/CVF International Conference on Computer Vision. pp. 13638- 13647, 2021.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
10

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Huang, X., Zhou, H., Yao, K., and Han, K. FROSTER: Frozen CLIP is strong teacher for open-vocabulary ac- tion recognition. In International Conference on Learning Representations, 2024.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-gre Fine-grained visual classification of aircraft, arXiv preprint urXiv:1306.1133 2013.
Momeni, L., Caron, M., Nagrani, A., Zisserman, A., and Schmid, C. Verbs in action: Improving verb understand- ing in video-language models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 15579-15591, 2023.
Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, L. Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems, pp. 30618-30630, 2022.
Ni, B., Peng, H.,, Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., and Ling, H. Expanding language-image pre- trained models for general video recognition. In European Conference on Computer Vision, pp. 1-18, 2022.
Ju, C., Han, T. Zheng, K., Zhang, Y., and Xie, W. Prompting Disual-language models for efficient video understanding. In European Conference on Computer Vision, pp. 105- 124, 2022.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Pro- cessing, pp. 722-729, 2008
Kim, M., Han, D., Kim, T., and Han, B. Leveraging tem- poral contextualization for video action recognition. In European Conference on Computer Vision, pp. 74-91, 2024.
Krause, J.. Stark, M., Deng, J., and Fei-Fei, L. 3d object rep- resentations for fine-grained categorization. In Proceed- ings of the IEEE international conference on computer vision workshops. pp. 554 561, 2013.
Pan, J., Lin, Z., Zhu, X., Shao, J., and Li, H. ST-Adapter: Parameter-efficien image-to-video to-video transfer learning. In Advances in Neural Information Processing Systems, pp- 14400-26413, 2022.
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. HMDB: A large video database for human motion recognition. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2556-2563, 2011.
Park, J., Lee, J., and Sohn, K. Dual-path adaptation from image to video transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2203-2213, 2023.
Parkhi, O. M., Vedaldi, A.. Zisserman, A., and Jawahar,
Lee, D., Lee, J., and Choi, J. Cast: cross-attention in space and time for video action recognition. In Advances in Neu- ral Information Processing Systems, pp- 093999-79225, 2023.
Cats and dogs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3498-3505 2012.
Qian, R., Ding, S., and Lin, D. Rethinking image-to-video adaptation: An object-centric perspective. In European Conference on Computer Vision, pp. 329-348, 2024.
Li, Y., Chen, G., Abramowitz, B., Anzellotti, S., and Wei, D. Learning causal domain-invariant temporal dynamics for few-shot action recognition. In International Conference on Machine Learning, pp. 77499-275111 2024.
Qing, Z., Zhang, S., Huang, Z., Zhang, Y., Gao, C., Zhao, D., and Sang, N. Disentangling spatial and temporal learning for efficient image-to-video transfer learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13934-13944, 2023.
Lin, T.-Y., RoyChowdhury, A., and Maji. S. Bilinear enn models for fine-gres the-ggained visual recognition. In Proceed- ings of the IEEE International Conference Computer Vision (ICCV), pp. 1449-1457. IEEE, 2015.
Lin, W., Karlinsky, I.., Shvetsova, N., Possegger, H., Kozin- ski, M., Panda, R., Feris, R., Kuchne, H., and Bischof, H. Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 2851-2862 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 7748-8763, 2021.
Rasheed, H., Khattak, M. U., Maaz, M., Khan, S., and Khan, F. S. Fine-tuned CLIP models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6545 6554, 2023.
Lin, Z., Geng, S., Zhang, R., Gao, P., de Melo, G., Wang, X., Dai, J.. Qiao, Y., and Li, H. Frozen CLIP models are efficient video learners. In European Conference on Computer Vision, pp. 388 384-404, 2022.

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Soomro, K., Zamir, A. R., and Shah, M. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: arXiv:
Yang. T., Zhu, Y., Xie, Y., Zhang, A., Chen, C., and Li, M. AIM: adapting image models for efficient video action recognition. In International Conference on Learning Representations, 2023.
2012.
Székely, G. J. and Rizzo, M. L. Brownian distance co- variance. Annals of Applied Statistics, 3(4):1236-1265, 2009.
Zanella, M. and Ben Ayed, I. Low-rank few-shot adapta- tion of vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1593-1603, 2024.
Székely, G. J. and Rizzo, M. L. Partial distance correlation with methods for dissimilarities. The Annals of Statistics, 42(6))2382-241) 2412, 2014.
Zanella, M., Gérin, B., and Ayed, Boosting vision- language models with transduction. In Advances in Neu- ral Information Processing Systems, pp. 62223-62256. 2024.
Tong, Z., Song, Y., Wang, J., and Wang, L. Videomae: Masked autoencoders are data-efficient learners for self- supervised video pre-training. In Advances in Neural Information Processing Systems, 2022.
Zhang, W. Wan, C., Liu, T., Tian, X., Shen, X., and Ye, J. Enhanced motion-text alignment for image-to-video transfer learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 18504-18515, 2024.
Wang, M., Xing, J., and Liu, Y. ActionCLIP: A new paradigm for video action recognition. arXiv preprint arX14:2109.08472, 2021.
Wang, Y., He. Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al. Internvid: A large- scale video-text dataset for multimodal understanding and generation. In International Conference on Learning Representations, 2023.
Zhang, Y., Zhang, C.,, Liao, Z., Tang, Y., and He, Z. BDC- Adapter: Brownian distance covariance for better vision- language reasoning. In Proceedings of British Machine Vision Conference, pp. 182-186, 2023.
Zhao, C., Wang, K., Zeng, X., Zhao, R., and Chan, A. B. Gradient-based visual explanation for transformer-based CLIP. In International Conference Machine Learning, pp. 61072-61091, 2024.
Wasim, S. T., Naseer, M., Khan, S. H., Khan, F. S., and Shah, M.
Video and text adaptive CLIP via multimodal prompting. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 23034-23044, 2023.
Zhelezniak, V., Savkov, A., Shen, A., and Hammerla, N. Y. Correlation coefficients and semantic textual similarity. In Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 951-962, 2019.
Weng. Z., Yang, X., Li, A., Wu, Z., and Jiang, Y.-G. Open- VCLIP: Transforming CLIP to an open-vocabulary video model via interpolated weight optimization. In Inter- national Conference on Machine Learning, pp. 36978- 36989, 2023.
Zhen, X., Meng. 7.,, Chakraborty, R., and Singh, V. On the versatile uses of partial distance correlation in deep learning. In European Conference on Computer Vision, pp. 327-346, 2022.
Wortsman, M., Ilbarco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959- 7971, 2022.
Zheng, Z., Wei, J., Hu, X., Zhu, H., and Nevatia, R. Large language models are good prompt learners for low-shot image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28453-28462, 2024.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485- 3492, 2010.
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Jour- nal of Computer Vision, 130(9):2377-2348, 2022.
Zhu, B., Niu, Y., Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 15659-15669, 2023,
Xie, J., Long, F., Lv, J., Wang. Q. and Li, P. Joint distri- bution matters: Deep brownian distance covariance for few-shot classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7972-7981, 2022.
12

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Appendix
the average accuracy and standard deviation.
A. Detailed Experimental Setup
Few-shot recognition We evaluate all-way K-shot tasks on downstream datasets including HMDB-51, UCF-101, and SSv2, where K = 2, 4, 8, 16. We randomly sample K videos per category for training, while testing on the first validation split for HMDB-51 and UCF-101, along with the full validation split for SSv2. We experiment in two different few-shot settings: (i) pre-training on K400 followed by fine-tuning on downstream datasets, and (ii) directly fine-tuning from CLIP without pretraining on K400.
A.I. Datasets
We conduct experiments on five widely used action recog- nition datasets, i.e. Kinetics-400 (Carreira & Zisser- man, 2017), Kinetics-600 (Carreira et al., 2018), HMDB- 51 (Kuchne et al., 201 UCF- 101 (Soomro et al., 2012), and Something-Sometting v2 (Goyal et al., 2017).
Kinetics-400 (K400) K400 is a large-scale action dataset that spans 400 human action classes each with at least 400 videos collected from YouTube. It provides approximately 240K training videos, 20K validation videos and 40K test videos, capturing a broad spectrum of human actions.
Base-to-nove generalization This task involves training models on base (known) classes in few-shot manner while testing on novel (unknown) classes. We evaluate on K400, HMDB-51, UCF-101, and SSv2 using base/novel category splits prescribed by ViFi-CLIP. Three training splits per dataset are constructed. On HMDB-51 and UCF-101, only the first training split is used for training and validation; on K400 and SSv2, evaluations are performed on the full validation set. We also experiment in two different settings: (i) pretraining on K400, and (ii) directly fine-tuining from CLIP without pretraining on K400.
Kinetics-600 (K600) This is an extension of K400 that broadens the scope to 600 categories and ~ 650K videos. It has 410K training and 29K validation samples.
HMDB-51 This dataset contains 51 action categories and approximately 7K videos manually curated from di- verse sources such movies and YouTube. Three train- ing/validation splits are predefined, each of which contains 3,570 training and 1,530 validation videos.
recognition This is the classical closed- set evaluation. As in ViFi-CLIP, we use standard splits for training and testing. In addition, we employ WSE technique for further performance improvement, where we interpolate ViFi-CLIP zero- model with our BDC-CLIP modes with a ratio of 0.2.
UCF-101 It consists of 101 action classes distributed across over 13K videos sourced primarily from YouTube. It defines three splits for training and validation, each contain- ing roughly 9.5K training and 3.7K validation videos.
v2 (SSv2) SSv2 contains more than 100,000 videos across 174 fine-grained action cate- gories, providing about 168K training and 24K validation videos. It emphasizes human-object interactions, and is highly temporal biased compared to other datasets.
A.3. Hyper-parametts Settings
To achieve better practice of VICLIP (Wang et al., 2023) by computing atten- tion over the tokens of all frames in video together at the 4th, 8th, and 12th layers of the CLIP vision encoder.
modeling, we follow the
A.2. Task Settings
Pretraining on K400 For zero-shot pretraining, we use the Adam W optimizer with $1 = 0.9, 32g = 0.98, a weight decay of le-3 and a batchsize of 256. The base learning rate (LR) of the backbone is 8c-6 with a cosine schedule in 10 epochs. The LRs of adapters and vision classifier are 100x and 50x base LR, respectively. We adopt the augmentation techniques and label smoothing as in ViFi-CLII. For few- shot pretraining, the hyper-parameter setting is consistent with zero-shot pretraining, except that the base LR is 4e-6. We sample 32 frames per video and conduct inference with temporal clip and 1 spatial crop (1 x view).
As in (Rasheed et al., 2023; Kim et al., 2024; Chen et al., 2024b), we benchmark on zero-shot & few-shot recognition, base-to-novel generalization, and supey-suee recog- nition. For these tasks, we mainly follow the evaluation protocols of ViFi-CLIP (Rasheed et al., 2023).
Zero-shot recognition We train our models on K400 and benchmark on HMDB-51, UCF-101, and K600. On HMDB- 51 and UCF-101, evaluations are conducted on the three of- ficial test splits. On K600, as prescribed by Chen & Huang, we use three splits randomly selected from 160 out of the 220 novel categories not present in K400. Following TC- CLIP (Kim et al., 2024) and OST (Chen et al., 2024a), we adopt weight-space ensembling (WSE) technique (Worts- man et al., 2022) for improving zero-shot performance, in which the original CLIP parameters are linearly combined with the fine-tuned parameters with a ratio of 0.3. We report
Downstream tasks with K400 pretrained models The few-shot and base-to-novel settings utilize a batch size of 64, a learning rate of 2e-6 with cosine schedule in 60 epochs, and linear warmup over first 5 epochs. We set the learning rate of adapters and vision classifier to 200 and 100x base LR, respectively. The other hyper-parameters align with those used in zero-shot pretraining. We use 32
13

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
sampled frames and conduct 1 xx] I view inference.
Table 7. Additional Ablation Study about BDC-CLIP.
Fully-supereised training on K400 The paree--aaraeeters in supervise setting are largely consistent with those in the setting of zero-shot pretraining, with a few notable differences. Specifically, the model is trained for 30 epochs that includes 5 linear warmup epochs with a batch size 512; the base LR is set to 2.2c-5 with a cosine schedule. We sample 16 frames per video and conduct inference with 12 views consisting of 4 temporal clips and 3 spatial crops.
K600
HMMDB-51
Vision
Text
zero-shoo shot
2-shot
16-shot
2-shot
16-shot
Eflect Token Ratio
4/4
4/4
73.8
66.1
73.9
8.9
16.8
3/4
4/4
73.740.7
64.5
73.7
8.6
16.5
4/4
73.4+0.7
74,9
16.5
1/4
4/4
73.640.77
64.7
73.8
8.4
16.4
4/4
2/4
73.420.7
65.5
73.2
8.6
16.4
Visual and/or Textual BDC
X
73.6
65.2
72.3
7.9
16.0
B. Further Experiments
73.8
64.3
71.5
7.5
14.3
73.840.9
66.1
73.9
8.9
16.8
B.1. Additional Ablation Study about BDC-CLIP
Effect Token Weighting
x
1.1
65.7
73.2
7.6
16.2
To better understand our BDC-CLIP, we further conduct ablation study on the ratio of tokens used for computing BDC, the impacts of individual visual and textual BDC, and of Token Weighting.
74.010.7
66.1
73.4
8.1
16.1
73.840.9
66.1
73.9
8.9
16.8
novel classes in zero-shot manner. Our BDC-CLIP mod- els are trained in 12 epochs, with the base LRs of 2e-6 on HMDB-51 & UCF-101, 4e-6 on K400 and 5e-6 on SSv2. Table 9 reports the Top-1 accuracy for the base and novel classes, along with their harmonic mean (HM). The results indicate that BDC-CLIP achieves sfate-of-the -ffte--rrt perfor- mance across all three datasets. Specifically, in terms of HM, BDC-CLIP surpasses the previous best-performing method TC-CLIP by 0.5% on K400, and 11.0% on all other three datasets.
As shown in Table 7, subsampling patch tokens at ratios 3/4, 2/4 and 1/4-or taking only half the word tokens-leads to performance degradation, suggesting retaining all tokens is important for fine- ane-grained alignment. For the impact of applying BDC the visual and textual branches, we observe that using BDC on only one branch-either vision or text- leads to a clear performance drop, especially on HMDB-51 and SSv2, while applying BDC to both branches achieves optimal results across all benchmarks. Similarly, applying Token Weighting to both branches consistently outperforms using it on a single branch, with weighting only the text branch generally being more effective than weighting only the vision branch.
It is worth noting that most methods in this directly fine- tuned setting perform remarkably inferior to their individ- ual counterparts in the setting of pretraining on K400, for either few-shot recognition (Table 2 vs. Table 8) or base-to- novel generalization (Table 3 vs. Table 9). This comparison suggests that pretraining on large-scale K400 is pivotal in bridging the image and video gaps for CLIP, significantly benefiting downstream video action recognition tasks.
B.2. Directly Fine-tuned from CLIP
Here we compare to prior arts in the setting of directly fine-tuned from CLIP, i.e., without pre-training on K400.
(i) Few-shot recognition We conduct experiment HMDB-51, UCF-101 and SSv2 in all way K Khohot setting without pretraining on K400. We train BDC-CLIP mod- els with 60 epochs with the base LR of 2e-6 on all three datasets. All methods are directly fine-tuned on the down- stream datasets, using K training examples per class for all-way classification. The results, summarized in Table 8, demonstrate that BDC-CLIP achieves superior performance across all three datasets. Specifically, BDC-CLIP consis- tently surpasses the second bbst methods by 2.1%-2.89 on HMDB-51 and 1.0%-1.5% .55 on SSv2 for each of K K-shot set- tings. Additionally, BDC-CLIP outperforms the runners-up by 0.6%-1.0% across all settings on UCF-101.
B.3. BDC-CLIP for Few-shot IMAGE Recognition
Our BDC-CLIP can be extended few-shot image recog- nition. To adapt to image data, we remove the temporal attention module from the visual encoder. Given the limited number of training images, we adopt murameter--fffciiee approach similar to CLIP-LoRA (Zanella & Ben Ayed, 2024) by attaching a LoRA (Hu et al., 2022) module (rank 2, alpha 1) to each transformer block in both textual and visual encoders. We use CLIP model with ViT-B/16 as im- age encoder, and conduct experiments under the 16-shot setting. For the training hyperparameters, we follow the same settings as CLIP-LORA. Following standard prac- tice, we evaluate our approach on ImageNet (Deng et al., 2009), Aircraft (Maji et al., 2013), Food (Bossard et al., 2014), DTD (Cimpoi et al., 2014), UCF101 (Soomro et al., 2012), Cars (Krause et al., 2013), OxfordPets (Parkhi et al.,
(ii) Base-to-novel generalization In this part of experiment, we compare the competing methods four datasets in base-to-novel generalization setting, Every method is di- rectly fine-tuned on the base classes and then evaluated on
14

---

(#f or fine-tuned (0). The hest BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Method
HIDBB-5 51
UCF-101
SSv2
Table Results of few-shot
K=2
K=4
K=8
K=16
K=2
K=i
K=8
K=16
K=2
K=l
K=8
K=16
VIDEO recognition without pretzaining on K40C.
CLIP Ratrod
41.9
41.9
41.9
41.9
63.6
63.6
63.6
63.6
2.7
2.7
2.7
2.7
al
39.7
50.7
62.4
79.9
methods are directly fine-tuned from
ActionCLIP Waaginal
47.5
57.9
57.3
59.1
70.6
71.5
73.0
91.4
4.1
5.8
8.4
CLIP K-shot settings. CLIP 10.0
11.1
X-CLIP Niveia
53.0
57.3
62.8
64.0
76.4
83.4
88.3
91.4
3.9
4.5
6.8
VIFi-CLIP Radhood
57.2
62.7
64.5
66.8
80.7
85.1
90.0
92.7
6.2
7,4
8.5
12.4
Backbone Encoders (BEs) are frozen
Chas
.9.1
TC. CLIP
58.6
63.3
65.5
68.8
86.8
90.1
94.3
7.3
8.6
14.0
results are bold and the second-hest
BDC-CLIP (Ours)
61.4
65.6
67.9
70.9
87.4
90.8
93.0
95.1
8,3
9.6
10.6
15.5
ones are underlined.
K400
HMDB-51
UCF-101
SSv2
Method
Bis
Novel
Base
Novel
Novel
HM
Table 9. Results of base-to-novel
CLIP Iualioni
62.3
53.4
57.5
53.3
46,8
49.8
78.5
63.6
70.3
4.9
5.3
5.1
VIDEO recognition without pretzaining on K40C. All methods are directly fine-tuned from
Juuuell.
69.7
37.6
46.2
16.0
90.5
55.8
6.4
ActionCLIP Warg
61.0
46.2
52.6
69.1
37.3
48.5
90.1
70.7
13.3
10.1
11.5
X-CLIP Ni
74.1
56.4
64.0
69.4
45,5
55.0
89.9
58.9
71.2
8.5
6.6
7.4
ViFi-CI et
76.4
61.1
67.9
73.8
53.3
61.9
92.9
67.7
78.3
16.2
12.1
13.9
CLIP on base classes, and then
Open-VCLIP Wene
76.5
62.6
68.9
70.3
50.4
94.8
77.5
85.3
16.0
11.0
evaluated on validation set of base 13.0
FROSTER Haxss at
77.8
64.3
70.4
74.1
58.0
65.1
95.3
80.0
87.0
18.3
12.2
14.6
classes along with novel classes
TC-CLIP
79.1
65.4
71.6
73.3
59.1
65.5
95.4
81.6
88.0
17.5
13.4
15.2
BDC-CLIP (Ours)
79,9
65.6
72.1
75.0
59,5
66.4
95.6
83.7
89.1
18.8
14,9
16.6
in zero-shot manner.
Table 10. Results of few-shot IMAGE recognition. All methods compared using CLIP models with VIT-B/16 image encoder in the 16-shot setting. CLIP Backhone Encoders (BEs) are frozen (*8) or finetuned
Method
Bis
ImageNet
Aircraft
Food101
DTD
UCF101
Cars
Pets
SUN397
Flowers102
Caltech101
EuroSAT
Average
BDC-Adupter Zlusg et
*
66.5
39,5
80,5
71.1
86.5
78.8
92.0
72.7
97.0
93.9
85.2
78.5
TransCLIP Zunzll
*
71.8
38.6
86,9
65.1
82.1
79.8
92.4
74.7
94.4
94.0
83.0
78.4
ProGrad Zhe
72.1
43.0
85.8
68.8
82.7
71.9
36.8
75.1
96.6
95.9
83.6
79.9
L.MMP Zieng etal.
+
73.5
56.1
87.6
74.2
86.8
86.1
94.2
77.0
98.1
47.1
91.3
83.8
CLIP-LoRA Zunella & Ben Aved
73.6
54.7
84.2
72.0
86.7
86.3
92.3
76.1
98.0
96.4
92.1
83.0
BDC-CLIP (Ours)
75.0
57.3
88.1
76.5
87.7
86.5
94.4
78.3
98.4
97.3
93.9
84,9
2012), SUN397 (Xiao et al., 2010), Flowers 102 (Nilsback & Zisserman, 2008), Caltech101 (Fei-Fei et al., 2004), Eu- roSAT (Helher al., 2019).
On HMDB-51 (Figure 6a), BDC-CLIP exhibits better inter- class separability compared to the strong baseline of ViFi- CLIP, and significantly outperforms vanilla CLIP. Similar trends observed on K600 (Figure 6b). We further eval- uate clustering quality using Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI), where BDC- CLIP achieves higher scores for both metrics, consistent with the visual results. These comparisons suggest that the BDC metric, leveraging all visual and textual tokens, can learn representations with stronger generalization capabili- ties compared to the cosine similarity relying only global tokens.
As shown in Table 10, BDC-CLIP surpasses CLIP- LoRA (Zanella & Ben Ayed, 2024), a strong baseline, by 1.9%, and outperforms the second-best method, i.c., LLaMP (Zheng et al., 2024), by 1.1%. Notably, BDC-CLIP achieves the highest performance across all 11 datasets. The comparison suggests that BDC-CLIP that uses BDC and all visual and textual tokens is effective for both video and image recognition tasks.
B.4. Extra Visualizations
Visualization of video-text pairs We compare models pretrained on K400 and evaluated in a zero-shot setting on representative clips from K600. Figure shows that BDC- CLIP consistently highlights salient image regions and words more accurately than ViFi-CLIP. For (Bulldozing). ViFi-CLIP is distracted by most of irrelevant regions (i.e., ground) while BDC centers on the machine and material without distraction; meanwhile, BDC-CLIP better attends to key words such as 'scraping' For (ull fighting). BDC-
t-SNE visualization of video-language representation We compute prediction probability vectors with K400- pretrained models for video samples from the validation sets of HMDB-51 and K600. To isolate the influence of LLMs, we use the CLIP prompt template for zero-shot clas- sification. These probability vectors are projected into a 2D space using t-SNE. Figure presents the results, with different categories represented by distinct colors.
15

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
CLIP better focuses the image regions of the bull, mata- dor, and cape across the frames; in the meantime, BDC- CLIP pays more attention to key words such 'bull fight- ing" and 'rushes' in the textual description. These examples demonstrate BDC-CLIP's ability to capture rich spatial, temporal, and linguistic context. We attribute this to the BDC metric and the inclusion of all local tokens, which to- gether enable the modeling of complex, ane-grained video- language relations.
BOC-CLIP (Dung
MM:
ARI: 0.40 MMi 0.70
(a) HMDB-51
C. Limitations and Future Work
BDC-CLIP exhibits somewhat higher GFLOPs and lower throughput compared to state-of-the-art methods. However, as shown in Table 5, this overhead remains moderate. While BDC effectively captures nonlinear correlations, other met- rics, such mutual information (MI), offer similar capabil- ities. However, MI faces computational challenges in high- dimensional spaces due to density estimation requirements and the difficulty of lower-bound estimation (Belghazi et al., 2018). Exploring such alternatives presents an exciting av- enue for future work. Currently, BDC-CLIP tailored for video action recognition. Extending it to other tasks, such video-text retrieval open-world object detection in videos, represents a promising direction for future research.
CUP 4RE MMO
VIF--LIP
BDC-CLPP
ARL
(b) K600
Figure 6. t-SNE visualizations of joint video-language representa- tions. K4400-pretrained models are used for zero-shot recognition HMDB-3 51 and K600. BDC-CLID shows better separability than ViFi-CLIP and vanilla CLIP; the quantitative results of the two clustering quality measures (i.e., ARI and NMI) are consistent with visualization results. Best viewed by zooming
16

---

BDC-CLIP: Brownian Distance Covariance for Adapting CLIP to Action Recognition
Original Frames
ViFi-CLIP
millooiinghhwwee the blade a machine Srrapingaoong the ground, movingloose
bulldocing shews the blade a machine Bulldozingshoor the blade of machine
bulldering chows the of a machine
scrapingllong the ground, moving loose material one side.
crapingalong the ground, moving loose material to one side
loose
scraping Phonngngolongggooooo the ground, moxing loose material one side
material one side.
BDC-CLIP(Ours)
hullcozing chows the blade machine
Buildozing shows the blade of machine bulldozing the blade of machine aulldozing shows the bf machine
scapingllong plong ground, movingloose
Sempinnglong the ground, moving material
scraping ground. moving material to side.
crrppingolong the ground, moving locse material one side.
Action category (Bulldozing)
Original Frames
ViFi-CLIP
fighting fighter
hughiighting shows fighter bent
lighting shows the figtar cape entendled as the buil lowerss hesd and rushes forward.
ghiing shows fighterbent
extended as exteeededded the lowers it's head and rushes forward.
Cape eeweeeetteddd..co the lowerss Its head and rushes forward
nushes
BDC-CLIP(Ours)
cape fighter
cape fighter
fighting fighter bentwith eapelextended
fighter bent
lowers
owers
and nushes
forward.
rushes
(b) Action category (Bull fighting
Figure 7. Heat- maps for representative video-text pairs produced with Grad-ECLIP using the K400- prattiee BDC-CLIP in zero-shot setting. Compared with ViFi-CLIP, BDC-CLIP more sharply highlights both the action-critical image regions and the corresponding salient words they evolve time, demonstrating its ability to capture fine-grained multimodal context space, time, and language. Best viewed by zooming in.
17


Reviews:
[{"review_id": "9iefGAMaH6", "reviewer": "Reviewer_GB1s", "summary": "This paper introduces BDC-CLIP, a framework designed to adapt CLIP for video action recognition by using Brownian Distance Covariance (BDC). The authors claim that traditional methods, relying on cosine similarity on global tokens, lack the capacity to capture complex spatio-temporal relations in video data. Their proposed solution overcomes these limitations by leveraging BDC, which captures both linear and nonlinear relationships amongst all visual and textual tokens. Through extensive experimentation, the authors demonstrate state-of-the-art performance across various scenarios.\n\n## update after rebuttal\nI agree with the observations made by the other reviewers. The authors have clearly described their proposed method and provided detailed experimental results. However, the theoretical explanation and the visual evidence supporting the benefits of the proposed approach remain relatively weak.\n\nThat said, the rebuttal addressed many of my initial concerns, and I appreciate the authors’ efforts in clarifying their contributions and presenting additional analysis. If the authors can incorporate stronger theoretical insights and more comprehensive visualizations in the final version of paper, it would significantly strengthen the paper.\n\nAt this stage, I maintain my Weak Accept recommendation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GB1s,\n\nWe sincerely thank you for providing constructive and insightful comments. Especially, we are grateful for your positive feedback including **\"The *novel* integration of BDC into the CLIP framework \"**, and **\"Comprehensive evaluation compared with previous methods and ablation study.\"** alongside **\"The authors demonstrate state-of-the-art performance across various scenarios.\"**\n\n\n> ### Q1: What exactly do linear and nonlinear correlations refer to in this context? How do they specifically manifest in video action recognition tasks? What qualitative or quantitative evaluation methods can be used to assess them?\n\nThanks for raising these insightful concerns. \n\n1) **Statistical perspective on linear & nonlinear correlations**\n* **Cosine Similarity (CS) $\\approx $ Pearson Correlation Coefficient (PCC).** As formalized in (Zhelezniak et al., 2019), a token embedding can be viewed as a sample of observations from some scalar random variable (RV) and CS is practically equivalent to PCC that can capture only  linear correlations between RVs. \n* **Current CLIP-Based Methods.** Most existing adaptations apply CS on global tokens ([CLS]/[EOS]) across frames, only modeling linear dependencies between two scalar random variables (representing each token). This coarse alignment often overlooks more complex, fine-grained cues.\n* **BDC-CLIP.** We measure the similarity between the set of all textual tokens $S_{\\text{txt}}=\\\\{\\mathbf{w_{0}},\\ldots, \\mathbf{w_{M}} \\\\}$ and all visual tokens of one frame $S_{\\text{img}}=\\\\{\\mathbf{p_{0}},\\ldots, \\mathbf{p_{M}} \\\\}$. Extending from the statistical framework of Zhelezniak et al. (2019), we view textual embeddings $S_{\\text{txt}}$ as a set of samples of $d$ observations from some theoretical set of scalar RVs $R=\\\\{W_0,\\ldots, W_M\\\\}$. BDC can quantify all kinds of statistical dependency between two sets of RVs $R_{\\text{txt}}$ and $R_{\\text{img}}$ (Szekely & Rizzo,2009). \n2) **Manifestation in Video Action Recognition**\n* Human actions exhibit rich contextual information across both spatial and temporal dimensions, involving dynamic interactions among people, objects, and the environment.\n* CS coarsely measures linear associations between the two modalities via global tokens, which  CS may miss crucial details unless they lie in a single linear dimension of the embedding space.\n* BDC, however, captures subtler interactions among fine-grained elements between language and vision, which can be highly nonlinear in the common embedding space.\n3)\t**Qualitative Evaluation**\n* We give **scatterplots and density contours** of some example visual vs. textual tokens; **kindly see [Figure R1](https://anonymous.4open.science/r/rebuttal-D15F/Figure_R1.jpg).** The plots show complex, nonlinear relations and clear non-Gaussian distributions, revealing nonlinear relation modeling is crucial and BDC can better capture these complex dependences. \n* We provide **heatmaps of example text-video frame pairs** for some action categories. These heatmaps highlight both key words (in shades of green) and salient spatial regions, evolving over time; **kindly see [Figure R2](https://anonymous.4open.science/r/rebuttal-D15F/Figure_R2.jpg).** The visualizations suggest BDC-CLIP can learn complex fine-grained multimodal context in space, time, and language.\n\n**Kindly refer to 2nd paragraph of Section 1 in our main paper for analysis on why nonlinear relation modeling is crucial for video action recognition.**\n\n\n> ### Q2: Marginal Empirical Improvements Relative to Computational Overhead.\n\nWe appreciate the concern. We would like to emphasize that BDC-CLIP achieves substantial improvements over SOTA methods while maintaining a competitive computational profile. Specifically, BDC-CLIP significantly outperforms TC-CLIP: for zero-shot recognition, it achieves +4.7% on HMDB-51 and +1.5% on UCF-101; for base-to-novel task the gaps in light of HM are 2.6%, 2.0% and 2.1% on HMDB-51, UCF101 and SSv2, respectively. Meanwhile, BDC-CLIP uses fewer parameters (0.99x) and achieves higher throughput (1.12x), with only a slight increase in GFLOPs (1.04x). Overall, these gains justify BDC-CLIP’s use in real-world scenarios.\n\n> ### Q3: First, a Preliminary Knowledge section is needed to introduce Brownian Distance Covariance (BDC). Following that, a thorough analysis of the theoretical motivation and insights should be provided. \n\nThanks for the thoughtful suggestions.\n\nIn the revised paper, we will add a separate section, *in the Appendix (due to page limit in the main paper),* introducing background knowledge on BDC (Szekely & Rizzo, 2009). Then, we will highlight BDC’s theoretical advantages over PCC  that is practically equivalent to cosine similarity. Also, we will integrate the response to Q1, making clear what linear and nonlinear correlations mean in CLIP-based video recognition, meanwhile providing qualitative evaluation showcasing the importance of learning complex relations.", "claims_and_evidence": "The primary claim of this paper revolves around the effectiveness of using Brownian Distance Covariance (BDC) for video-language alignment. This claim is supported by the ablation study in Table 6.\n\nHowever, some important details lack sufficient evidence.\nIn the original statement:\n“BDC can capture both linear and nonlinear correlations, enabling it to model the complex dependencies that exist between video and language embeddings.”\n\nIt is necessary to clarify: What exactly do linear and nonlinear correlations refer to in this context? How do they specifically manifest in video action recognition tasks? What qualitative or quantitative evaluation methods can be used to assess them?", "methods_and_evaluation_criteria": "Yes, the method and evaluation criteria make sense.", "theoretical_claims": "The authors leverage known theoretical foundations of Brownian distance covariance. However, their usage remains heuristic; no novel theoretical contributions or proofs are provided. Therefore, no theoretical analysis was performed or required for validation.", "experimental_designs_or_analyses": "The experimental designs make sense. They follow previous works.", "relation_to_broader_scientific_literature": "The paper situates itself within recent trends of adapting image-language pretrained models for video understanding tasks. The use of BDC for video-language alignment is indeed novel in this context.", "essential_references_not_discussed": "Essential relates works are cited and discussed.", "other_strengths_and_weaknesses": "Strengths:\n1. Comprehensive evaluation compared with previous methods and ablation study\n2. The novel integration of BDC into the CLIP framework\n\nWeakness:\n1. Marginal Empirical Improvements Relative to Computational Overhead:\nThe improvements shown over state-of-the-art baselines, while consistent, remain modest. Given the additional computational complexity and overhead introduced by integrating and computing BDC matrices, the incremental improvement may not be sufficient justification for adoption in real-world scenarios where computational efficiency is crucial.\n\n2. Theoretical insights or deeper conceptual understanding are marginal or missing. First, a Preliminary Knowledge section is needed to introduce Brownian Distance Covariance (BDC). Following that, a thorough analysis of the theoretical motivation and insights should be provided, explaining the advantages of using BDC in video action recognition, how it captures non-linear relationships, and why non-linear relationship modeling is crucial for video action recognition tasks.", "other_comments_or_suggestions": "Deeper analysis focused on interpretability, explaining why exactly BDC works better practically, is strongly recommended.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jYH48Wnz5v", "reviewer": "Reviewer_oB1H", "summary": "This paper proposes BDC-CLIP, a novel framework for video-language alignment based on Brownian Distance Covariance (BDC). Unlike cosine similarity, BDC can capture both linear and nonlinear correlations. BDC-CLIP leverages all visual and textual tokens to model both linear and nonlinear relationships in the multimodal embedding space, thereby capturing rich contextual information across spatial, temporal, and linguistic dimensions. The framework also introduces a temporal BDC attention mechanism that integrates patch-wise spatial cues and frame-wise temporal dynamics.\n\n## update after rebuttal\nI tend to give a borderline score (between 2 and 3), but since ICML only has a weak accept option, I have raised the score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oB1H,\n\nWe sincerely thank you for your constructive and insightful comments, especially your positive feedback that **\"The paper is well-written and easy to understand\"** and   **\"The proposed method achieves state-of-the-art results on multiple datasets. \"**\n\n\n\n> ### Q1: However, I have two concerns regarding the novelty of the paper: (1) The idea of adding adapters after the CLIP encoder is not entirely new. Many existing works have explored similar techniques, including approaches related to relationships between sets of embeddings, global tokens, and local tokens. These are common strategies for processing CLIP tokens. (2) The use of Brownian Distance Covariance primarily originates from image-based few-shot classification tasks and has been adapted for video action recognition in this work.\n\nThank you for sharing these concerns.\n\n1) **BDC Integration into CLIP.** \nAs noted in Section 1 of our main paper, **our primary contribution is introducing BDC for vision-language alignment in foundation models like CLIP.** While past works such as DeepBDC [Xie et al., 2022] and BDC-Adapter [Zhang et al., 2023] apply BDC in single-modality, image-based few-shot classification, **none** address the cross-modal alignment challenge of video and text. Our approach goes beyond the predominantly used cosine similarity, demonstrating how BDC effectively captures statistical dependence between two distinct modalities. We also appreciate that **Reviewer GB1s recognized this novelty**, calling it the ***“novel integration of BDC into the CLIP framework.”***\n\n2) **BDC-Based Temporal Adapter for Local and Global Tokens.**\nOur second contribution is **a temporal adapter tailored for BDC representation** that leverages both local (patch/word) tokens and global ([CLS]/[EOS]) tokens for adapting CLIP to video recognition. To the best of our knowledge, no prior work has applied BDC to modeling relationships among local tokens for CLIP-based video recognition. If such related work exists, we would welcome further references to ensure comprehensive coverage.\n\nGiven these points, **we respectfully request a reconsideration of our paper’s novelty,** which combines BDC-based cross-modal alignment with a temporal adapter that captures fine-grained token interactions for video action recognition.\n \n \n\n> ### Q2: The motivation of this paper is not clearly articulated. The authors claim that previous methods rely on cosine similarity and only use global tokens. However, this is not entirely accurate, as many existing works already explore different similarity computations and incorporate local tokens. The paper does not sufficiently clarify this issue.\n\nThank you for noting this issue.\n\nAs we state in our Abstract (Lines 21-23) and Section 1 (Lines 57-71), most existing methods still rely on cosine similarity and focus on global tokens. To our best knowledge, **for CLIP-based video recognition,** *OST (Chen et al., 2024) is the only approach that departs from cosine similarity by using optimal transport*, but it aligns **frame-level [CLS] tokens with sentence-level [EOS] tokens rather than local (patch/word) tokens,** and its primary goal is to enhance textual descriptors. We have not encountered prior CLIP-based video recognition works that leverage local tokens for cross-modal alignment, but would appreciate any pointers if they exist. It is worth noting  **Reviewer LE28 affirmed that our *“motivation is clear and intuitive.”***\n\n\n> ### Q3: The paper states that BDC can capture both linear and nonlinear correlations, but it is unclear what exactly is meant by \"linear\" and \"nonlinear\" in this context. How do \"linear\" and \"nonlinear\" correlations correspond to specific methods in the proposed framework? There is a lack of experimental validation to demonstrate the advantage of capturing both linear and nonlinear correlations.\n\nThank you for raising these points.\n\n**We address them in detail in (and kindly refer to) our response to Q1 from Reviewer GB1s,** where we explain *from the statistical perspective* the distinction between linear and nonlinear correlations (i.e., cosine similarity vs. BDC) and provide both theoretical rationale and qualitative evaluations (e.g., scatterplots and density contours and heatmaps) to illustrate BDC’s ability to capture more complex dependencies. Our extensive experiments in Section 4 further show that capturing complex dependencies--including linear and nonlinear correlations--has yielded superior performance, compared to strong baselines (i.e., VIFI-CLIP and TC-CLIP) based on cosine similarity that can only model linear correlations.", "claims_and_evidence": "Previous methods align video and language based on the cosine similarity between the average of frame-level [CLS] tokens in the video and the sentence-level [EOS] token. This limits the alignment to coarse semantic matching. In contrast, BDC-CLIP aligns the two modalities using Brownian Distance Covariance (BDC), which considers all visual and textual tokens. This approach captures fine-grained spatio-temporal cues crucial for action recognition.", "methods_and_evaluation_criteria": "BDC-CLIP introduces two core components: (1) a video BDC adapter and (2) a text BDC adapter, which are aligned using Brownian Distance Correlation. (1) Video BDC Adapter: By leveraging all visual tokens (i.e., [CLS] and patch tokens), BDC-CLIP computes a BDC matrix as a frame-wise representation and designs a temporal attention mechanism to model frame-to-frame dynamics. (2) Text BDC Adapter: BDC-CLIP exploits all textual tokens (i.e., [EOS] and word tokens) to compute a BDC matrix as the text representation. Finally, BDC-CLIP aligns the video and text representations using Brownian Distance Correlation.", "theoretical_claims": "The idea of using Brownian Distance Covariance (BDC) for aligning video and language is theoretically reasonable.", "experimental_designs_or_analyses": "The paper conducts experiments on five widely used action recognition datasets: Kinetics-400, Kinetics-600, HMDB-51, UCF-101, and SSv2. The model is evaluated across various downstream tasks, including zero-shot, few-shot, base-to-novel generalization, and fully-supervised settings. The proposed BDC-CLIP achieves state-of-the-art performance on these tasks.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\nThe paper is well-written and easy to understand.\nThe proposed method achieves state-of-the-art results on multiple datasets.\n\nWeaknesses:\n1, The main innovations of BDC-CLIP include:\n(i) A video BDC adapter and a text BDC adapter, which serve as post-processing modules for the visual and text encoders to enhance alignment.\n(ii) The use of Brownian Distance Covariance to strengthen alignment.\nHowever, I have two concerns regarding the novelty of the paper:\n(1) The idea of adding adapters after the CLIP encoder is not entirely new. Many existing works have explored similar techniques, including approaches related to relationships between sets of embeddings, global tokens, and local tokens. These are common strategies for processing CLIP tokens.\n(2) The use of Brownian Distance Covariance primarily originates from image-based few-shot classification tasks and has been adapted for video action recognition in this work.\n2, The motivation of this paper is not clearly articulated. The authors claim that previous methods rely on cosine similarity and only use global tokens. However, this is not entirely accurate, as many existing works already explore different similarity computations and incorporate local tokens. The paper does not sufficiently clarify this issue.\n3, The paper states that BDC can capture both linear and nonlinear correlations, but it is unclear what exactly is meant by \"linear\" and \"nonlinear\" in this context. How do \"linear\" and \"nonlinear\" correlations correspond to specific methods in the proposed framework? \n There is a lack of experimental validation to demonstrate the advantage of capturing both linear and nonlinear correlations.", "other_comments_or_suggestions": "None.", "questions_for_authors": "The paper states that BDC can capture both linear and nonlinear correlations, but it is unclear what exactly is meant by \"linear\" and \"nonlinear\" in this context. How do \"linear\" and \"nonlinear\" correlations correspond to specific methods in the proposed framework? Which components in the method explicitly capture these correlations? There is a lack of experimental validation to demonstrate the advantage of capturing both linear and nonlinear correlations. It would be beneficial to include ablation studies or quantitative analyses to verify this claim.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3rpHVFoRCj", "reviewer": "Reviewer_LE28", "summary": "This paper proposes BDC-CLIP, a framework that introduces Brownian Distance Covariance (BDC) to address the limitations of current CLIP-based video models. BDC-CLIP can leverage all the visual and textual embeddings and construct non-linear relations for vision-language modeling. BDC-CLIP achieves state-of-the-art performance on multiple video benchmarks under various settings.\n\n## update after rebuttal\nThanks for your rebuttal. The concerns have been solved. I will increase the score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LE28,\n\nWe sincerely thank you for providing constructive and insightful comments. In particular, we appreciate your positive comments including  **\"The motivation is clear and intuitive\"** as well as **\"The achieved performance is great on a range of benchmarks and experimental settings.\"**\n\n\n> ### Q1: The technical contribution appears somewhat limited, as the core technique, Brownian Distance Covariance, was already introduced in prior work.\n\nThanks for your concern. \n\nTo our best knowledge, our work is the first attempt that introduces BDC into both text and visual encoders of CLIP for vision-language alignment. **Our work highlights, *in foundation models such as CLIP,* potential of advanced metric (e.g., BDC) over ubiquitous cosine similarity.** In contrast, DeepBDC (Xie et al., 2022) concerns no CLIP framework, while BDC-adapter (Zhang et al., 2023) only uses BDC in single vision modality. Besides, we design **temporal attention mechanism for  BDC representations that previous arts lack** as they focus on image recognition. *kindly see Lines 151-164 in the paper for detailed discussion on differences from the previous works.*\n\nIn terms of the above clarifications, we would be grateful if you could re-evaluate our technical contributions.  \n\n\n> ### Q2: Additional ablation experiments should be added. In the paper, I cannot see how the performance gradually improves on top of a vanilla baseline with all the proposed modules. The authors should elaborate more on this.\n\nThank you for the comment.\n\nKindly note that we present the **performance variation on top of a vanilla baseline *in Table 6a.*** Following your suggestion, we **add additional ablation experiments** for further illustrating the effect of proposed modules. Specifically, we evaluate BDC-CLIP for the following settings: 1) not employ temporal attention (TA), 2) using 3/4, 1/4 of patch tokens or half of the word tokens, 3) turn off text or visual BDC matrices, and 4) turn off text and visual tokens weighting (TW).  ***Kindly refer to our response to Q3 of Reviewer sjZc for the results and discussion.***\n\n \n\n> ### Q3: What features will the model use for inference: global features, local features or BDC matrices?\n\nOur inference relies primarily on the BDC matrices produced by the two adapters for both video-language classification and purely visual classification. In addition, as in prior works, we also incorporate global features from the backbone encoder to enable a standard CLIP-like classification branch.\n\n> ### Q4: The proposed method should be compatible with image-only tasks, e.g., few-shot image classification. I suggest the authors conducting some experiments on that for further demonstrating the effectiveness of the paper.\n\nThanks for your comment. \n\n**As suggested, we extend BDC-CLIP to few-shot image recognition.** Specifically, we remove temporal attention module in the visual encoder to fit for image recognition task; as the training images are scarce, we adopt parameter efficient technique as in CLIP-LoRA, attaching a LoRA module (rank 2, alpha 1) to each transformer block for both textual and visual encoders. Following previous arts, we conduct comparison on 11 datasets with ViT-B/16 as the visual encoder in 16-shot setting. From the table below, we see our BDC-CLIP improves over the strong baseline of CLIP-LoRA by 1.9\\%, while outperforming the second-best (i.e, LLaMP) by 1.1\\%. Notably, BDC-CLIP stands out across all 11 datasets. **The comparison suggests that BDC-CLIP that uses BDC and all patch tokens is general, effective for both video and image recognition tasks.**\n\n|Method|ImageNet|Aircraft|Food| DTD|UCF|Cars|Pets|SUN|Flowers|Caltech|EuroSAT|Avg|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|TransCLIP (Zanella et al.)|71.8|38.6|86.9|65.1|82.1|79.8|92.4|74.7|94.4|94.0|83.0|78.4|\n|ProGrad (Zhu et al.)|72.1|43.0|85.8|68.8|82.7|71.9|36.8|75.1|96.6|95.9|83.6|79.9|\n|CLIP-LoRA (Maxime et al.) |73.6|54.7|84.2|72.0|86.7|86.3|92.3|76.1|98.0|96.4|92.1|83.0|\n|LLaMP (Zheng et al.)|73.5|56.1|87.6|74.2|86.8|86.1|94.2|77.0|98.1|97.1|91.3|83.8|\n|BDC-CLIP (ours) |75.0|57.3|88.1|76.5|87.7|86.5|94.4|78.3|98.4|97.3|93.9|84.9|\n\n* Zanella M, Gerin B, Ayed I. Boosting vision-language models with transduction. In NeurIPS, 2024.\n* Zhu B, Niu Y, Han Y, et al. Prompt-aligned gradient for prompt tuning. In CVPR, 2023.\n* Maxime Z, Ismail B A. Low-rank few-shot adaptation of vision language models. In CVPRW, 2024.\n* Zheng Z, Wei J, Hu X, et al. Large language models are good prompt learners for low-shot image classification. In CVPR, 2024.", "claims_and_evidence": "The claims are well-supported in the paper.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, in Sec. 4.", "relation_to_broader_scientific_literature": "The proposed method can achieve state-of-the-art performance on a wide range of video benchmarks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strength\n\n- The motivation is clear and intuitive, which aims to address an intrinsic limitation for current video models.\n\n- The achieved performance is great on a range of benchmarks and experimental settings.\n\n### Weaknesses\n\n- The core technique, i.e., Brownian Distance Covariance, is proposed by a previous paper. Therefore, the technical contribution of this paper seems a bit limited.\n\n- Additional ablation experiments should be added. In the paper, I cannot see how the performance gradually improves on top of a vanilla baseline with all the proposed modules. The authors should elaborate more on this.\n\n- What features will the model use for inference: global features, local features or BDC matrices?\n\n- The proposed method should be compatible with image-only tasks, e.g., few-shot image classification. I suggest the authors conducting some experiments on that for further demonstrating the effectiveness of the paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yvdc9TWkP0", "reviewer": "Reviewer_sjZc", "summary": "In this work, the authors propose an adapter-based action recognition method built on top of CLIP visual and text encoders. To better capture local details, they propose to utilize all visual patch tokens and word tokens and employ Brownian Distance Covariance (BDC) as a similarity metric between video and text representations. They validate the proposed method on several public datasets and tasks.\n\n\n## update after rebuttal\nThe additional evidence is somewhat weak as it is based on only a couple of samples or one action class. Still, ~20% accuracy on SSv2 is far from being useful, similar to other CLIP-based methods. So I would say this is still a downside of this work. Therefore, I would like to keep my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer sjZc,\n\nWe sincerely thank you for your constructive and insightful comments, particularly your positive feedback & decision.\n\n> ### Q1: The paper lacks evidence demonstrating that BDC enables modeling of complex dependencies in the video-language embedding space.\n\nThanks for this concern. \n\nIn BDC-CLIP, we compare two sets of $d$-dimensional tokens from language and vision. According to Zhelezniak et al. (2019), this setup can be viewed as modeling dependencies between $d$-observation samples from two random variable sets (textual vs. visual tokens). Furthermore, **Szekely & Rizzo (2009) show that BDC captures all types of statistical dependencies--linear and nonlinear--without assumptions on joint distributions.** Hence, by applying BDC to video-language alignment, BDC-CLIP can effectively model complex dependencies in the shared embedding space. *The scatterplots and token distributions (kindly see [Figure R1](https://anonymous.4open.science/r/rebuttal-D15F/Figure_R1.jpg)) illustrate that our model can effectively learn nonlinear relationships.*\n\n\n>  ### Q2: The paper does not provide evidence to support the claimed fine-grained multi-modal context modeling across spatial, temporal, and language dimensions.\n\nThe set of textual tokens naturally encodes key nouns and verbs describing people and objects, while the set of patch tokens captures crucial spatial regions in each video frame. As BDC can measure any form of statistical dependency between textual and visual tokens, **it provides a principled way to capture rich relations across spatial and language dimensions.** Further, by averaging BDC matrices over consecutive frames, our method models temporal evolution of the token-level correspondences. \n\nThe heatmaps (kindly see [Figure R2](https://anonymous.4open.science/r/rebuttal-D15F/Figure_R2.jpg)) show how the model focuses on horses and players in the {playing polo}--highlighting the relevant objects and interactions in both text and video frames. **These results suggest that BDC-CLIP indeed learns fine-grained, action-centric context spanning space, time, and language.**\n \n\n>  ### Q3: BDC-CLIP needs to be further validated by more experiments: 1) without (w/o) temporal attention (TA), 2) using 3/4, 2/4, 1/4 of patch tokens (PT) or half of the word tokens (WT), 3) turn off text or visual BDC matrices, and 4) turn off text and visual tokens weighting (TW). \n\nThank you for the suggestions.\n\n 1)$\\ $W/O TA, BDC-CLIP’s performance drops consistently across 3 datasets, indicating TA provides valuable temporal modeling. 2) Subsampling patch tokens at ratios 3/4 and 1/4--or taking only half the word tokens--also degrades results, suggesting retaining all tokens is important for fine-grained alignment. 3) Removing either text or visual BDC significantly hurts performance, with a more pronounced drop from removing visual BDC, suggesting more importance of the visual adapter. 4) Disabling TW for either text or vision negatively impacts performance, underscoring how focusing on more informative tokens benefits our approach.\n\n\n|Method|K600_Zeroshot|HMDB51_2shot|HMDB51_16shot| SSv2_2shot|SSv2_16shot|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|BDC-CLIP|73.8$\\pm$0.8|66.1|73.9|8.9|16.8|\n| w/o TA|73.3$\\pm$0.8|65.5|73.4|8.3|16.2|\n|3/4 PTs|73.7$\\pm$0.8|64.5|73.7|8.6|16.5|\n|1/4 PTs|73.7$\\pm$0.9|64.7|73.8|8.5|16.5|\n|1/2 WTs|73.4$\\pm$0.8|65.5|73.2|8.6|16.4|\n|w/o text BDC|73.6$\\pm$0.8|65.2|72.3|7.9|16.0|\n|w/o visual BDC|73.8$\\pm$0.8|64.3|71.5|7.5|14.3|\n|w/o text TW|73.2$\\pm$1.1|65.7|73.2|7.6|16.2|\n|w/o visual TW|74.0$\\pm$0.7|66.1|73.4|8.1|16.1|\n\n>  ### Q4: Similar to prior works using CLIP, BDC-CLIP shows limitations in modeling temporal dynamics as demonstrated on SSv2 in Table 2 and 3, achieving ~20% accuracy while SOTA method with similar backbone (ViT-B) shows >70% accuracy.\n\nKindly note our reported ∼20% or lower accuracy in Tables 2 and 3 are from few-shot and base-to-novel settings, while the mentioned SOTA results (>70%) rely on fully supervised (FullS) training. **These differing settings are not directly comparable**, and it remains unclear how those FullS approaches would fare in few-shot or base-to-novel settings. \n\n \n\n> ### Q5: Essential References Not Discussed.\n\nThanks for highlighting the three works that will be cited and discussed in our revision. Briefly, **they focus on adapting pure vision-pretrained models for fully-supervised (FullS) action recognition**. In contrast, **BDC-CLIP relies on multimodal alignment between vision and language,** achieving strong performance in, beside FullS setting, zero-shot, few-shot and base-to-novel settings.\n\n>  ### Q6: How to construct $\\mathbf{b_t}$ in L190-192?\n\n$\\mathbf{B_t}$ is a symmetric $d\\times d$ matrix, so half-vectorization (Vech) collects the elements on or below its diagonal into a $d(d+1)/2$-dimensional vector $\\mathbf{b_t}$. Kindly see the [Wikipedia article](https://en.wikipedia.org/wiki/Vectorization_(mathematics)#Half-vectorization) for details on Vech.", "claims_and_evidence": "Some of the claims made are not fully supported by evidence.\n- “BDC captures both linear and nonlinear relations, enabling it to model the complex dependencies in the video-language embedding space” -> There is no evidence of BDC enabling complex dependency modeling in the video-language embedding space. The paper only shows favorable task performance, some feature space visualization and attention map visualization.\n- “…enabling fine-grained multi-modal context modeling in space, time, and language” -> Similar to the first claim above, there is no evidence of *fine-grained multi-modal context modeling* in space, time, and language in the paper.", "methods_and_evaluation_criteria": "The proposed method is sensible as we can reuse a vision-language pre-trained model with relatively little amount of fine-tuning effort. Employing BDC as similarity metric to model non-linear relation between tokens make sense and is proven effective in action recognition to some extent. Downside of the method is the limited capability of temporal dynamics modeling, similar to prior CLIP-based works such as Action-CLIP, X-CLIP, TC-CLIP, etc, as shown in SSV2 experiments in Table 2 and 3.", "theoretical_claims": "There is no theoretical claims made.", "experimental_designs_or_analyses": "The proposed method needs to be further validated by more experiments.\n- What happens if we do not employ temporal attention in eq (4)? (effect of the proposed temporal modeling method) \n- What happens if we use 3/4, 2/4, 1/4 of the patch tokens or half of the word tokens instead of using them all?\n- What happens if we turn on and off (or replace it with cosine similarity) text BDC matrices computing and visual BDC matrices computing? Which BDC computing is more important?\n- What happens if we turn on and off text token weighting and visual token weighting?", "relation_to_broader_scientific_literature": "Since this work is built on top of CLIP [Radford et al., ICML 2021], it is related to the vision-language alignment and multi-modal learning field. Similar to prior works using CLIP, the proposed method also shows limitations in modeling temporal dynamics as demonstrated in the Something-Something-V2 experiments in Table 2 and 3. All the CLIP-family including the proposed method show unfavorable performance on SSV2: ~20% accuracy while SOTA method with similar backbone (ViT-B) shows >70% accuracy on this dataset.", "essential_references_not_discussed": "Please consider citing the following works and discussing the relation to the proposed method.\n\n- [Park et al., Dual-path Adaptation from Image to Video Transformers, CVPR 2023]\n- [Lee et al., CAST: Cross-Attention in Space and Time for Video Action Recognition, NeurIPS 2023]\n- [Qian et al., Rethinking Image-to-Video Adaptation: An Object-Centric Perspective, ECCV 2024]", "other_strengths_and_weaknesses": "Please address my concerns on the unsupported claims, temporal modeling capability of the method, missing empirical validation.", "other_comments_or_suggestions": "- L195: “Let the values be V=[…” -> where does V come from?\n- L211-213: “we first achieve the embeddings of reduced dimension …” -> awkward sentence\n- L417: “... models are valuated ...” -> evaluated?", "questions_for_authors": "Please address my concerns on the unsupported claims, temporal modeling capability of the method, missing empirical validation. \n1) I do not understand how to construct $\\textbf{b}^t=Vech(\\textbf{B}^t) \\in \\mathbb{R}^{d(d+1)/2}$ in L190-192. Can the authors elaborate on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a contrastive vision-language pre-training method for action recognition by using Brownian Distance Covariance for video-language alignment. Reviewers initially raised concerns regarding motivations and novelty, theoretical insights, and insufficient ablations of the proposed method.

After the rebuttal, the authors addressed most of the concerns, and all reviewers agreed to accept this paper, although some of the issues still remain (e.g., weak theoretical explanation and visual evidence). By taking the overall contributions and experimental results into consideration, AC believes most significant concerns have been alleviated and this paper is above the acceptance threshold.
Decision: Accept (poster)