Federated Multimodal Fusion for Action Recognition Leveraging Vision-Language Embeddings and Spatio- Temporal CNNs
Authors: Aditi Palit
Kalidas Yeturu
Venue: Accepted by TMLR ()

Abstract: Federated learning (FL) for Video Action Recognition (VAR) faces significant challenges in balancing privacy preservation, communication efficiency, and model performance.  This paper introduces FLAMeST (Federated Learning for Action Recognition with Multimodal embeddings and Spacio-Temporal Fusion), a FL framework that synergizes Vision-Language Models (VLMs) and spatiotemporal CNNs to address these challenges.  Unlike existing works that use BLIP (VLM) solely for caption generation, FLAMeST leverages BLIP in a dual manner.  To enhance temporal modeling, complementary spatiotemporal features are extracted using a pre-trained 3D CNN (Slow network). These semantic (BLIP) and motion (Slow) embeddings are concatenated into a unified representation to train a lightweight Multi-Layer Perceptron (MLP).  Within the FL paradigm, only the MLP parameters are shared with the server, ensuring raw video data and generated captions remain local.  FLAMeST employs the FedAvg algorithm for model aggregation, achieving 99%(↓) lower communication overhead compared to full-model training.  Experiments on UCF101 and HMDB51 datasets demonstrate the framework’s robustness, achieving improved accuracies of 5.13%(↑) and 2.71%(↑), respectively, against the baseline.

Published in Transactions Machine Learning Research (12/2025)
Federated Multimodal Fusion for Action Recognition Leveraging Vision-Language Embeddings and Spatio- Temporal CNNs
Aditi Palit
cs21 d001 Giilip.ac.in
Department of Computer Science Indian Insiitute of Technology Tirupali
Kalidas Yeturu
Mrilidas@iitp..ac.
Department of Computer Science Indian Institute of Technology Tirupoti
Reviewed OpenReview: https: // openreview. net/ forum? Adbaodzdgiiee
Abstract
Federated learning (FI.) for Video Action Recognition (VAR) faces significant chal- lenges in balancing privacy preservation, communication efficiency, and model performance. This paper introduces FLAMeST (Federated Learning for Action Recognition with Multimodal embeddings and Spacio- Temporal Fusion), FL framework that synergizes Vision-Language Models (VLMs) and spatiotemporal CNNs to address these challenges. Unlike existing works that use BLIP (VLM) solely for caption generation, FLAMeST leverages BLIP in dual manner. To enhance temporal modeling, complementary spatiotemporal features are extracted using a pre-trained 3D CNN (Slow network). These semantic (BLIP) and mo- train
tion (Slow) embeddings are concalenated into a unified representation lightweight Multi-Layer Perceptron (MLP). Within the FL paradigm, only the MLP paramcters are shared with the server, cnsuring raw video data and gener- ated captions remain local. FLAMeST employs the FedAvg algorithı for model aggregation, achieving 99%(L) lower communication overhead compared to full- model training. Experiments on UCF100 and HMDB51 datasets demonstrate the framework's robustness, achieving improved accuracies 5.13%(t) and 2.71%(1), respectively, against the baseline.
1 Introduction
Video Action Recognition (VAR) plays a vital role in computer vision, with applications spanning human-computer interaction, surveillance, healthcare, and autonomous systems (Al-Faris et al., 2020; Javaid et al., 2024; Gumbs al., 2022). Traditional VAR approaches leverage deep learning architectures such as 3D Convolutional Neural Networks (3D-CNNs) (Ji al., 2012; Tran et al., 2015), Recurrent Neural Networks (RNNs) (Ji et al., 2012; Tran et al., 2015; Yang et al., 2022),

---

Published in Transactions Machine Learning Research (12/2025)
and Transformers (Ulhaq et al., 2022), to model spatiotemporal dynamics from large-scale datasets like UCF101 (Soomro et al.; 2012) and Kinctics (Carrcira & Zisserman, 2017). However, these centralized training paradigms face limitations due to data privacy laws, e.g., GDPR (General Data Protection (eguuation,(Buroeean Union, 2016)), data decentralization, and communication bottlenecks (AbdulRahman et al., 2020).
In real-world deployments, video streams from surveillance or wearable devices often contain sensi- tive user data, rendering inter-institutional sharing infeasible (Posner, 2008).
To address these challenges, Federated Learning (FL) has emerged as a promising solution by en- abling decentralized training across clients without transferring raw data (Mammen, 2021; Kairouz et al., 2021). Clients update shared global model through local training and only communicate parameter updates (McMahan et al., 2017). While this paradigm preserves privacy, most exist- ing FI.-based VAR methods rely on unimodal visual features and overlook the semantic richness of LaaguuLL Models (VLMs) (Zhang et al., 2024). VLMs such as CLIP (Radford et al., 2021) and BLIP (Li al., 2022) have shown superior performance in zero-shot and few-shot tasks by aligning visual and textual modalities through joint. representations (Saha et al., 2024). These models offer rich semantic context-c.28 captions like "a person opening a door" can enhance action understanding when fused with visual features.
To address this issue of crivacy-preserving VAR without compromising efficiency, we propose a framework for VAR in a cross-silos FL. environment called FLAMeST, which stands for Federated Learning for Action Recognition with Multimodal embeckinngs and Spacio-Temport Fusion, that integrates spatiotemporal CNNs with VLM-based embeddings. In FLAMeST as shown in Figure each client uses pre-trained BLIP model to generate coption from sampled video frame and then derives cross-modal embeddings by reprocessing the image-caption pair through BLIP's cross- allention module. This dual use of BLIP, beyond standard caplion generation, yields stronger semantic representations. In parallel, a SlowFast-3D CNN1 (Feichtenhofer et al., 2019) extracts temporal features from keyframes and combines them with the VLM embeddings to form rich visual-level representations. The embeddings are concatenated and used to train a lightweight Multi-Tayer Perceptron (MLP). To ensure communication efficiency, only the MLP weights are exchanged with the central server during aggregation, which drastically reduces communication overhead (Section 5.1, Table 1).
Our contributions are as follows:
Multimodal Embedding Fusion: We introduce a novel hybrid embedding that com- bines BLIP-generated vision languaee features with Slow-3D CNN outputs capture both semantic and motion dynamics.
Communication Efficiency: By training only the MLP in FL, our framework reduces communication overhead by 99.4% compared to full-model FL.
FLAMeST achieves improvements 5.13% and 2.71% over FL-based knowledge distillation (Jain et al., 2021) the UCF101 and HMDB51 datasets, respectively.
Although VLMs are large-scale models, we assume clients can execute them in inference mode (Zhuang et al., 2023), enabling practical deployment while leveraging their strong semantic priors.
For Fast we used Slow pathway with frames input.
2

---

Published in Transactions Machine Learning Research (12/2025)
Our primary objective is explore the underutilized potential of VLMs in FL environments. While we instantiate FLAMeST for video action recognition on UCF101 and HMDB51, the underlying design is substantially more general. By freezing multimodal foundation backbones (BLIP and 3D CNN) and federating only a lightweight prediction head, FLAMeST aligns with the broader trend in federated learning toward parameter-efficint tuning where small heads, adapters, or prompts are collaboratively optimized while large backbones remain fixed across clients. This decoupled design is not tied to any specific dataset or architecture and can, in principle, extend to a range of vision or multimodal tasks that map pro-trained emboddings downstream predictions.
2 Related Work
Human Action Recognition (HAR) tasks, particularly those relying on wearable sensors, generated considerable attention both academia and industry (Gani et al., 2019; Wang et al., 2020; Hassan al., 2018; Kalabakov al., 2023), subsequently motivating the extension of such techniques to video-based applications.
Early VAR methods primarily relied on hand-crafted features such as motion-energy images (MEI) and motion-history images (MHT) to capture mntiotemporal dynamics (Bobick & Davis, 2001) ).
Despite being computationally efficient, these methods exhibited limited robustness to variations in viewpoint and complex motion patterns (Bobick & Davis, 2001; Zhao et al., 2024).
Centralized Action Recognition: With deep learning, 2D CNNs like AlexNet and VGGNet (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014) enabled robust spatial feature extraction from frames but lacked temporal modeling. To address this, two-stream networks emerged, com- bining RGB frames (spatial stream) and optical flow (temporal stream) (Alomar al., 2024; Le et al., 2022). While these improved accuracy, they required separate optical flow computation, thus increasing complexity.
Subsequently, 3D CNNs such as C3D, 13D and SlowFast networks (Tran et al., 2015; Carreira & Zis- serman, 2017: Feichtenhofer et al., 2019) enabled end-to-end spatiotemporal learning. Transformer- based models have also shown promise for VAR. Vision Transformers (ViTs) and Video Swin Trans- formers capture long-range temporal dependencies via attention mechanisms (UThaq et al., 2022; Liu et al., 2022), while hybrids like PSO-ConvNet Transformers combine CNN and transformer fea- tures for improved accuracy (Nguyen & Ribeiro, 2023). VideoMAE extends masked autoencoding to videos using spatiotemporal tube masking (Tong et al., 2022).
InternVideo2.5 (Wang et al., 2025) builds on the InternVideo2 architecture by introducing long- context modeling and hierarchical visual-token compression to efficiently process extended video sequences. It integrates video encoder (for spatial-temporal features) with language model backbone via a unified mross-modal adapter, enabling frame-level reasoning over long clips. The training follows three-stage pipeline progressive pretraining image text and short video text pairs, followed by long-video alignment- to improve temporal coherence and tross-modal under- standing.
However, these models are trained in centralized seltings and focus on unimodal visual features, raising concerns over privacy, data sharing, and regulatory compliance (Kazakos al., 2021; Akbari et al., 2021).
3

---

Published in Transactions Machine Learning Research (12/2025)
Federated Learning for Action Recognition: FL addresses privacy by enabling decentralized training. Several works have applied FL to HAR using wcarable sensor data (Gani et al., 2019; Hassan et al., 2018). For video, most methods simplify the problem to image-level classification (Doshi & Yilmaz, 2022) or apply self-supervised learning to improve generalization (Rehman et al., 2022). Personalization and heterogeneity have been addressed through techniques like Meta-HAR (Li et al., 2021), FedCLAR (Presotto et al., 2022), and FedMAT (Shen et al., 2022), which balance shared and user-specific learning. An activity recognition framework in FT. is proposed (Yang al., 2024), where both global (modality-agnostti learned collaboratively across clients. This design effectively separates shared and unique modality characteristics through adversarial training. Architecture-aware FL approaches like FedConv (Xu et al., 2023) study CNN configurations under heterogencity have also been explored. Addressing the limited computational capacity of edge devices, (Jain et al., 2021) proposed a knowledge dis- tillation (KD) strategy involving two teacher models to facilitate efficient model deployment in an FL environment. This hierarchical distillation pipeline introduces a teaching assistant model as an intermediary, effectively bridging the gap between the complex teacher and the constrained student. For few-shot, learning under FL., (Tu el al., 2024a) employed CNNs to caplure spatiotemporal cues and applied meta-learning for improved generalization. FedVision enabled FL for object detection with YOLOv3 (Liu et al., 2020), while recent transformer-based FL work targets video anomaly detection (Doshi & Yilmaz, 2023).
and private (modality-spccifie) classificrs are
Although these methods support video or multimodal learning in FL,, they do not incorporate vision-language models (VLMs), missing the benefits of semantic context.
Vision-Language Models and FL Integration: CLIP and BLIP (Radford al., 2021; et al., 2022) demonstrate strong image-text aligmment for zero-shot tasks, but their application in vidco tasks remains limited. A few recent efforts addressed this gap for example, (Zhuang et al., 2023) analyzes how foundation models can be incorporated into FL, outlining both opportunities for collaboration and the associated technical hurdles. ActionCLIP (Wang et al., 2023), which adapts VLMs like CLIP for video VAR by leveraging the semantic alignment between visual and textual modalities. Unlike traditional methods that treat action labels as discrete classes, ActionCLIP treats them as rich textual descriptions. JoVALE (Son al., 2024) combines BLIP with person detection and audio-visual fusion, but the added modalitics increase computational overhead.
To address these gaps, we propose FLAM framework that combines bhse- vision- language embeddings with spatiotemporal CNN features for efficient, privacy-preserving FL in VAR.
Our approach achieves competitive accuracy on UCF101 and HMDB51 while significantly reducing communication overhead.
3 FLAMeST Framework
The pipeline for the FLAMeST framework shown in Figure 1. We consider an FL set-up com- prising N clients {Cr, ........ CN), }, where each client C' holds a private labeled video dataset. Vj represents the jth video clip of the ith device.
4

---

Published in Transactions Machine Learning Research (12/2025)
D CNN
K-means
Feature Extraction
Input Video
Keyframes
Raudem frame selection
Cuption Gemeration
D2
Aligned Embedding
Frozen
Prompt: Describe
Trainable
the action
Image
LLM
FL
Image enender
LLM
VLM
CROSS AA
Coucalenate
AA small boy riding cycle.
Figure 1: Overview of the FeST pipeline. (A) Keyframes are selected using k-means clustering. (B-C) A keyframe is sampled and captioned using a VLM. (D1) All keyframes are passed through a 3D CNN; (D2) the sampled keyframe-caption pair is reprocessed for cross-modal embeddings. (E-F) CNN and VLM features are concatenated train lightweight MLP, which participates in the FL.. Only the MLP parameters updated and sent to the server, which aggregates via FedAvg and redistributes the global MLP (step G). The foundation models remain frozen and are not transmitted, reducing communcattion overhead (Supplementary Figure A-1).
D' D' ((V) ,[V}, where D = M
(1)
Each video sample consists of n/ RGB frames of spatial dimension H x W where the associaled action class o belongs in label space of K action classes.
Vj x
(2)
2 E (1....,K) K}
3.1 Key Frame Selection via Clustering
To reduce the temporal redundancy in video sequences, we apply k-Means based clustering in Icarned feature space to identify a reduced set of informative koyframes. Across all devices and their data elements, we set m k; where k is the number of clusters. We experimented with & E 32 5, 8) and selected k=3 using the elbow method and silhouette confidence intervals. This makes the keyframe selection systematic and reproducible, rather than heuristic. Method details
5

---

Published in Transactions Machine Learning Research (12/2025)
are presented in Supplementary Section A.1, Algorithm A-1,
F Key. [f]...... fmy where f E x3 1 (1,...,mi. m)))
(3)
3.2
Random Key Frame Sampling
From the set of key frames
Ficy, key. a single representative frame f3 is sampled uniformly at random.
fj ~ ( k)y,j) where P(C)) mi
1
(4)
3.3 Multimodal Embedding Generation via VLM
The sampled frame fj is first passed through a VLM PYLM to generate a descriptive caption tj.
tj (f))
(5)
Subsequently, the caption tj and the frame fj are fed into the VLM cross-attention module to produce a joint Dision-language embedding, i.e, the [CLS] token embedding of the last hidden layer is extracted (Algorithm 1). Without loss of generality, we consider the BLIP VLM.
eLLM, = VLM VLM(() 1t) E R
(6)
where d. 1 768.
3.4 Visual Embedding via CNN Backbone
In parallel, the set of keyframes F Teey. is processed using 3D CNN-basod backbone &CNN (e.g., ResNet-3D, 13D, Slow) to obtain visual-only embedding. Without loss of generality, consider the Slow -3D CNN model.
CCNN, PCNN (Fi (Fioy.) Rd.
(7)
where dy 1 x 2048.
The embeddings are obtained from the last dense layer of the CNN model (Algorithm 1).
3.5 Joint Feature Representation
The final multimodal feature vector is formed by concatenating the VLM-based and CNN-based embeddings. We also experimented with baleddaateeniion--aaedd fusion technique. However, the performance is not any better than simple concatenation (The results are provided in Section 6.5).
ej e E Rd.
(8)
where dimensionality eaj x 2816.
6

---

Published in Transactions Machine Learning Research (12/2025)
Each client prepares its transformed data set,
DVLM.CNN {(e), 3
(9)
The CNN (eÇNN,)) and VLM (e(LM)) embeddings often differ in dimensionality, making element- wise operations like the dot product infeasible. While projecting them to common space is possible, it may introduce complexity and risk of information loss. To retain the full representation of both modalities without additional transformations, we adopt simple concatenation for fusion.
A Algorithm 1 High-level steps for obtaining VLM and CNN embeddings on each client
1: Determine keyframes (FQ. 3)
2: Determine representative frame (EQ, 4)
3: Obtain caption for the frame (EQ. 5)
4: Obtain text-visual cross-embedding for the caption text and the corresponding frame (FQ. 6)
5: Obtain CNN embedding for the representative frame (EQ. 7)
6: Obtain combined embedding (EQ. 8)
7: Prepare transformed data set (EQ, 9)
3.6 Local Training and Federated Aggregation at Server
On each client O train local multi-layer perceptron (MLP) classifier g²(+) parameterized on w 2 on the transformed dataset DVLM.CNN
g'(w'): Bdotd. + RR
(10)
Each client trains g'(+) by minimizing the cross-entropy loss using Stochastic Gradient Descent (SGD) as,
J(w') D
Σ Lcr(s(e),s)
(e.v)ED
(11)
w/41 w V wi Iww))
where D DVLM.CNN: g is (w'), LoE(..) denotes the cross-entropy loss and r is the learning rate. After local training, cach client transmits its updated parameters w (EQ. 11) to a central server. The server aggregates these received models by weighted averaging of the models' parameters (EQ. 12, Algorithm 2).
DDi
((-1))
Σ
a
w (0) a'
E |Di
(12)
Where corresponds to the data proportion of each client. It denotes the weight given to each client during model aggregation. The aggregated global model Wict)) is broadcast to all clients, and the training proceeds for T communication rounds.
Pw is wt time
7

---

Published in Transactions Machine Learning Research (12/2025)
Each client has a pre-trained VLM and a CNN model, both frozen during training and inference. The only trainable component is the MLP, which participates in the FL cycle.
An alternative is to employ recent video-language captioners such as Qwen-VT, Cosmos- Reasoning, However, deploying multi-billion parameter captioners on each FL client is infeasible due to memory and latency constraints, and server-side caplioning would require uploading raw video, violating privacy. Our FLAMeST pipeline is lightweight, ddient-fiennee and Exploring distilled captioners in FL an avenue for future work.
Algorithm 2 Federated Learning with federated averaging
1: Initialize server model random weights wi
for I [] do 3: for E (1, N] do
4:
Initialize client weights (EQ. 10) w² = w(t-1)
5: Prepare (or update) client data set (Algorithm 1)
6: Minimize client specific loss (EQ. 11)
w argmin (w)
7: end for
8: Perform federated averaging server (EQ. 12) to obtain w(o)
9: end for
4 Experimental Set-Up
We design a comprehensive experimental framework to evaluate the performance FLMeST for VAR. Table shows the details regarding cach model and embedding size. All the experiments done on NVIDIA A100. All references to BLIP embeddings in this work pertain to the cross- attention embeddings derived from the visual-semantic alignment within the BLIP model.
Benchmark Datascts and Data Partitioning Strategy: The experiments in this study are conducted on two widely recognized benchmark datasets: UCF101 (Soomro et al., 2012) and HMDB51 (Kuehne et al., 2011). UCF101 comprises 13,320 video clips spanning 101 action cate- gories. The dataset split into 10,619 training and 2,701 testing samples. HMDB51 consists 6,766 video clips categorized into 51 action classes. The dataset partitioned with 5,413 videos for training and 1,353 videos for testing, following 80:20 train-test. split. To simulate real-world Non-IID conditions,
partition data across clients using Dirichlet distribution Dir(3), where smaller indicates higher label skew. For baseline comparisons, we use IID splits (B = 1), while Non-lID settings use 3 = 0.6 to evaluate robustness under data heterogeneity.
Benchmarking Methods and Feature Extractors: We use two popular vision-language models (VLMs): BLIP-2 (Li et al., 2022) and CLIP-ViT /I-14 (Radford et al., 2021). ). For benchmarking, we evaluate four spatiotemporal models: (1) ResNet-3D (Tran al., 2015) as 3D CNN baseline; (2) I3D (Carreira & Zisserman, 2017), which inflates 2D kernels into 3D; (3) SlowFast (Feichtenhofer
8

---

Published in Transactions Machine Learning Research (12/2025)
al., 2019) using an 8-frame Slow pathway to capture appearance and motion; (4) VideoMAE (Tong et al., 2022), masked mutoencoder-basee ViT for video reprosentation, and InternVideo2.5 (Wang et al., 2025), a VLM for video QA. All models use public code and pretrained weights from PyTorchVideo (Fan et al., 2021), Torchvision, and HuggingFace. Feature vectors are extracted from the final convolutional layer before classification to capture spatiotemporal semantics. In addition to the feature extractors discussed above, we benchmark our method against the work of (Jain al., 2021)3 We have also considered the recent method ActionCI.IP by (Wang et al., 2023) a non-FL sctting that uses a CLIP model for action recognition on unseen classes. We do not compare our method with existing FL-based VAR approaches such as FL for Driving Action Recognition (DAR) (Doshi & Yilmaz, 2022), which targets driver-specific actions using 2D CNNs and T (He et al., 2020) for communication efficiency. Their task-specific focus differs from our objective of general-purpose action recognition. Similarly, FSAR (Guo et al., 2023) is another VAR method in the FI. setting, but it operates on skeleton-based datasets rather than video data, making direct comparison with our approach inappropriate. Few-shot FL methods (Tu et al., 2024b) are also excluded, as they address different problem setting. However, we consider the extension of our framework to few-shot FL-based VAR as an interesting direction for future work.
Client-Side Training Protocol: In the proposed FL framework, foundation models remain frozen during training and inference and do not participate in FL communication. Each client trains a local MLP classifier, whose parameters are shared with the server during aggregation. By default, the MLP comprises two hidden layers (512 and 256 ncurons) and an output layer matching the number of action classes. The input layer aligns with the dimensionality embeddings. Clients train locally for five epochs per round using a batch size of 128. Optimization is performed using Adam with a learning rate of 0.01 and weight decay of 1 x 10-1, with cross- entropy loss as the loss objective. Additional MLP configurations are explored in Section 6. Unless stated hthewwiee, experiments use four clients in a cross-silo IID-FL setting over 80 rounds.
the frozen backbone
Model Aggregation, Client Update and Evaluation Metrics: The model aggregation is accomplished via FodAvg (McMahan et al., 2017), which simply averages model weights across participating clients. For client-side model update, we have considered two methods, FedProx (Li et al., 2020) and FedDyn (Acar et al., 2021), apart from simple gradient update, which were compared in ablation studies (Section 6).
Performance is assessed using two primary metrics. The top-1 classification accuracy of the global model and communication efficiency are measured as the number of model parameters exchanged per communication round.
Resource Usage and Storage Requirements: We use the VLM BLIP4 model for generating the caption and subsequently visual-semantic cross embeddings. For the 3D-CNN backbone, we used the Slow-3D CNN model which is part of Facebook Al's PySlowFast framework. An analysis of resource usage during inference mode shows that the SLDw-3D CNN model recorded a maximum memory allocation of 319.51 MB per point, while the BLIP model utilized 1023.97 MB per point as inspected using watch and torch." 7 The execution times for Slow-3D CNN and BLIP are 1.315
Thee code for this paper was not publicly available. Therefore, the accuracy values reported in work are those
quoted directly from the original paper.
http://rrrrbii--mma---aaapiiioii "https://pytorchh.. arg/hb// www.:a/h.rggguubbbbbbboeeeeeee /aaeebbbrooreerrcchccccccacaaaaa pytorchvideo resnet
"watch I nvidia-smi
"torch.cuda.nk max memory allocated((/10021**))
9

---

Published in Transactions Machine Learning Research (12/2025)
Parameter reduction by using only the MLP in F
Parameter reduction compared to Knowledge
Distillation (KD):
279,88 16,,868,172
KD Our
Our MLP
BLIP + Slow + MLP) MLP
KD
(BLIP + Slow MLP)
11, 689, 512 1, 1,573, 889
(14)
(279, 868, 172 + 1,573, 889) - 183, 279,868, 172 1,573, 889
11, 689, 512
0.865 or 86.5%
0.994 or 99.4%
(13)
seconds and 1.362 seconds, respectively, when processing single input point. The storage require- ment for BLIP is approximately 941.44 MB, the Slow-3D CNN model is 131.85 MB, and the MLP model is 6.02 MB (Refer to Table 1).
5 Results and Evaluation
In this section, conduct quantitative evaluation of FLAMeST, focusing on its communication overhead and performance in both FT, and centralized learning setups. We compare FLAMMeST against various foundation model extractors to assess its effectiveness and efficiency in these different learning environments.
5.1 Communication Overhead
We assume that each participating client possesses sufficient storage and computational resources host pre-trained foundation models, such as BLIP, CLIP, and 3D-CNN. Table outlines the parameter count of the models employed. We propose lightweight communication strategy to overcome the significant communication overhead typically associated with federated training of large-scale models. Instead of exchanging the full parameter sets of these massive foundation models, we freeze the pre-trained model's backbone during local training and extract fixed embeddings from it. A lightweight MLP classifier trained on these embeddings. This is the only component that solely participates in the FL optimization loop. This decouples the representation learning phase from the federated aggregation step, substantially redueing communication costs. As shown in Table 1, the number of parameters involved in FL is smaller than the size of the underlying foundation models. For instance, while the BLIP+Slow model compriscs approximately 279 million parameters, only 1,573,889 888 parameters from the MLP are involved in FL communi- cation, resulting in parameter transfer reduction of approximately 99.4% (EQQ. 13). Similarly, compared to the baseline KD, the parameter transfer reduction is approximately 86.5% (EQ. 14). This selective participation significantly enhances the communccation efficiency of the proposed framework without compromising representation quality. Freezing the foundation models strikes balance between performance and efficiency, preserving the strong semantic priors of these models while avoiding the prohibitive cost of fine-tuning and communicating billions of parameters across clients.
9The Floating Point (FP) 32
10

---

Published in Transactions Machine Learning Research (12/2025)
Table 1: Model Comparison: Parameters and Input Dimensions for Feature Extractors and FL Models, where the last column refers to the learnable parameters participating FL. Details are presented in Section 4.
Feature
Input Frame
Base Model
Input Dimension 768
Parameters
S.No
Extractor Models
Parameters
BLIP (L. al., 2022)
247,414,076
525,311
2
CLIP (Radford et al., 2021)
1
151,277,311 ,313
1024
656,3855
3
KD (Baseline) (Jain et al., 2021)
8
49,482,360 360
11,689,512
4
ResNet3D (Tran et al., 2015)
3
33,371, 1,72
512
394,24 241
5
13D (Carreira & Zisserman, 2017)
8
28,043,472
20488
1,180,673
6
Slow (Feichtenhofer al., 2019)
8
34,566, 488
2048
1,180,673
VidcoMAE (Tong al., 2022)
16
86304869
786
525,313
BLIP (Text Only)
1
247,114 4470614
768
525,313
BLIP 13D
(1+88
275,457,544 548
2816
1,573,889
10
BLIP + ResNet3D
(1++3)
280,785,548
797,451
11
BLIP + Slow (FLAMeST)
9 (1+-8)
279,868,172
2816
1,633,899
12
BLIP (Text Ouly) + ResNet3D
4 (1+-3)
280,785,55 548
1280
797 797,457 ,377
13
BLIP (Text Only) I3D
9 (118)
275,457 5488
2816
1,573,888
14
BLIP (Text Only) + Slow
9 (1+8)
279,868,172
2816
1,573,889
15
ActionCI.IP (non-FL. (Wang et al., 2023)
3
150,000,000
16
InteraVideo2.6 (Wang et al., 2025)
8
L,775,422,770
1024
682,085
5.2
Comparison of PLAMeST and other Feature Extractors
Table 2-(A, B) presents a comparative study of different. VLMs, various CNN and transformer-based feature extractors in FL training settings on the UCF101 and HMDB51 datasets. For UCF101, the highest accuracy is achieved by the combination of embeddings extracted from the Slow model along with the cross-attention embeddings obtained from the BLIP model. Our method achieves an improvement of 5.13% against the bascline (Table 2-A, row 14, row 3).
The Slow model, being a 3D CNN, elfectively captures the spatial-temprral aspects of video sequences. In contrast, the BLIP model integrates visual and textual informa- tion through its vision-language alignment mechanism. Fusing these embeddings provides richer video content representation, allowing the classifier to make more informed predictions. The integration of BLIP embeddings with the 3D CNN features leads to significant improvement in accuracy. Specifically, there is an 18.05% increase (Table 2-A, row 1- row 14) in accuracy for BLIP embeddings when combined with the Slow model, while the Slow model benefits from an improvement of 3.47% (Table 2-A, row 5, row 14). This trend underscores the advantage of utilizing muul timodol embeddings that encapsulate textual and visual semantics. The socond-best performing combination comprises the I3D model coupled with BLIP alignment embeddings. This pairing results in an accuracy improvement of 4.23% (Table 2-A, row 6 row 13) and 17.38% (Table 2-A, row 1, row 13) for the 13D) model and the BLIP embeddings, respoctively. Similarly, the ResNet-3D model, when fused with cross-altention embeddings from BLIP, exhibits an increase of accuracy of 16.15% (Table 2-A, row 4, row 12) for UCF101. The highest accuracy achieved the HMDB51 datasct.
Sciu Hold welo Aiheenee MECCIANN
May
v M NWW wwyga
Figure 2: Accuracy achieved by the global model (FLAMeST) over 50 rounds (cycles).
11

---

Published in Transactions Machine Learning Research (12/2025)
is significantly lower compared UCF101, which can be attributed to the inherent complexity and challenging nature of HMDB51.
Even in a centralized learning setup, the best performance achieved was approximately 71% (Ta- ble 3-B, row 15) ). In the FL setting, the highest accuracy of 67% (Table 2-B, row 14) was attained using the BLIP model and the Slow architecture. Our method achieves an improvement of 4.58% (Table 2-B, row 3, row 14) against the baseline. The BLIP and Slow models individually contributed to performance gains of 11.63% (Table 2-B, row 1, row 14) and 2.59% (Table 2-B, row 5, row 14), respectively, over their standalone performance. Furthermore, when BLIP embeddings were fused with the I3D model, an improvement of 4.89% (Table 2-B, row 6, row 13) in accuracy was observed, while the integration with ResNet-3D resulted in 20.91% (Table 2-B, row 4, row 12) performance gain for the 3D CNN models.
These findings suggest that incorporating semantic-visual embeddings alongside spaliia--empporu em- beddings enhances model performance by enriching feature representations with complementary con- tertual information. Both the VLM and the CNN model benefit from the collaboration. Regarding the training accuracy of FLAMeST (Figure 2), the training accuracy improves and stabilizes with the subsequent communication cycle. We also caleulate the cross- valsdaanion accuracy on UCF101 and HMDB51 over 20 folds across 80 rounds to get the average estimate over different train-test splits. The mean accuracy for UCF101 was approximately 94% with standard deviation of 0.32 (Table 2-A, row 15), whereas for HMDB51, the mean calculated was 70% with a standard deviation of 0.98 (Table 2-B, row 15).
The InternVideo2.5 framework attains superior recognition performance, yielding an accuracy im- provement of 1.32% over the proposed BLIP Slow embeddings the UCF101 dataset (Table 2-A, rows 16 and 14) and 3% the HMDB51 dataset (Table 2-B, rows 16 and 15). The excellent quality of InternVideo2. embeddings is expected given its scale as the variant evaluated is 8B parameters, trained using a mixture of visual-text and long-video data (e.g., 3.7 M video-text pairs, 3.5 M image descriptions, and 0.7 M long vidcos). In contrast, our caption-basee embedding backbone (BLIP) operates at an order of magnitude smaller size (hundreds of millions of parameters) and trained on fewer image-caption pairs.
Though the quality of embeddings is very good with InternVideo2.5, it imposes stringent hard- ware requirements, thereby constraining its feasibility for deployment platforms with limited computational or memory resources.
In addition to action recognition, we evaluated the effectivnnes of the BLIP cross-modal embeddings for image classification on the CIFAR-10 dataset. Our experiments achieved an accuracy of 90% after 30 communication rounds. Details are provided in Supplcmentary Section E.
5.3 Poor Performance of Static Text Embeddings Generated by CLIP
Unlike BLIP, which can generate captions directly for given image, CLIP operates in a retrieval- based manner. To facilitate this process, we first generate set of action-specific captions for each action in the UCF101 and HMDB51 datasets using large-scale language models such as ChatGPT, Each caption is then encoded using the text encoder of the CLIP model, and the resulting text embeddings are stored in a dictionary for efficient retrieval during training and inference (More details in the Supplementary A.3). Though computing embeddings in prior reduces the computa- tional cost due to their static nature, the quality of embeddings is observed to be poor. As shown
12

---

Published in Transactions Machine Learning Research (12/2025)
Table 2: Federated Learning: Accuracy Comparison of Different Feature Extractors on UCF101 and HMDB51 Datasets. The bold-faced text denotes baseline and above accuracies. The baseline and FLAMeST values are indicated by * and ** prefixes, respectively.
(a) UCF101 Dataset
(b) HMDB51 Dataset
S.No
Feature Extractor
Accuracy (%)
S.No
Feature Extractor
Accuracy (%)
BLIP
76.38
BLIP
51.75
2
CLIP
60,98
2
CLIP
30.98
3
KD (Baseline)
*89.30
3
KD (Baseline)
$61.80
4
ResNet-3D
64.01
4
ResNet-3D
38.95
5
Slow
90.96
5
Slow
*63.79
I31)
89.53
131)
59.62
Video MAE
65.04
Video MAE
33.45
8
BLIP (Text Only)
30.28
8
BLIP (Text Only)
27.18
9
BLIP (Text Only] + Net-33)
64.00
9
BLIP (TText Only) + Nes-31)
40.04
10
BLIP (Tex Ouly] + Slow
91.03
10
BLIP (Tex Ouly) + Slow
63.19
BLIP (Text Ouly] + 13D
89.92
BLIP (Tex Only) + 13D
60.13
12
BLIP ResNet-3D
80.16
12
BLIP ResNet-3D
59.86
BLIP I3D
93.76
BLIP I3D
64.51
BLIP Slow (FLAMeST)
**94.43
BLIP Slow (FLAMeST)
**66.38
15
FLAMeST (Cross Validation)
91:0.332
15
FLAM&ST (Cross Validation)
70:
16
InterVVideo2.5
95.75
16
InternVine InternViden2.5 iden2.5
73.45
in Table 2-A, for UCF101, the accuracy obtained is 61% (Table 2-A, row 2), and for HMDB51, it is 33% (Table 2-B, row 2). Since these captions are generated before training, they are inherently generic and may fail to reflect individual video frames' unique visual and contextual attributes. As a result, the fixed tertual representation may not align well with the dynamic and heterogencous nature of the visual embeddings, leading to suboptimal mullimoddal fusion and reduced classification accuracy.
As the CLIP-generated embodding was not well refined, did not conduct further studics it, most of the improvement would have come from the embeddings of the CNN model rather than the VLM in this joint learning.
5.4 Significance of Textual Embeddings
An additional set of experiments was conducted using only the textual embeddings generated by the VLM. The MLP model, in this case, was only trained on the text embeddings obtained from the VLM. Our findings resonate with recent analyses of embedding based retrieval limits (Weller et al., 2025). They highlight weaknesses of single-vector representations, consistent with our observation that BLIP-only text embeddings are weak. For the BLIP model, when only text embeddings (captions generated) used to train the MLP model, the accuracies obtained for UCF101 and HMDB51 are only 30.28% (Table 2-A, row 8) and 27.18% (Table 2-B, row 8), respectively. Aligning the text embeddings with the visual embeddings of the BLIP model improved accuracy by more than 27.57% for the HMDB51 dataset (Table 2-B, row 8 to row 1) and by 46.1% (Table 2-A, row 8 to row 1) for the UCF101 dataset. From this observation, we conclude that textual information alone camnot fully capture the dynamic nature of actions in video sequences. While textual descriplions can introduce supplementary contertual details, they do not encompass the full range of spotial and temporal dependencies in videos. However, textual information when used with visual features can
13

---

Published in Transactions Machine Learning Research (12/2025)
Table 3: Centralized Learning: Accuracy Comparison of Different Feature Extractors on UCF101 and HMDB51 Datasets. The bold-faced text denotes baseline and above accuracies. The baseline and FLAMeST values are indicated by * and ** prefixes, respectively.
(a) UCF101 Dataset
(b) HMDB51 Dataset
S.No
Feature Extractor
Accuracy (%)
S.No
Feature Extractor
Accuracy
BLIP
85.01
BLIP
62.36
2
CLIP
62.80
2
CLIP
55.87
3
KD (Baseline)
*91.10
3
KD (Baseline)
64.10
4
ResNet-3D
78.05
4
ResNett-D
53.87
5
Slow
91.37
5
Slow
63.17
I3D
92.71
I3D
65.24
Video MAE
66.38
Video MAE
39.38
BLIP (Texl Only)
34.73
BLIP (Texl Only)
35.50
9
BIJI (Text Only) + Nes-3tt
79.30
9
BIJ (Text Only) + ResNett3
52.92
10
BLIP (Text Only) + Slow
90.67
10
BLIP (Text Only) + Slow
66.64
11
BLIP (Text Only) + I3D
88,78
11
BLIP (Text Only) + I3D
65.98
12
BLIP ReaNet-3D
85.82
12
BLIP ReaNet-3D
64.21
13
BLJP + 131)
93.97
13
BLJP 131)
68.04
ActionCLIP
95
ActionCLIP
BLIP Slow (FLAMcST)
*94.30
BLIP Slow (FLAM+ST)
**71.4
IndemnVide2..5
96.75
InternVideo2.3
74.20
enhance the overall representative quality (Tables 2-(A, B), row 14). An illustration of the text captions generated by BLIP are shown in Supplementary C.2.
5.5 Comparison with Centralized Training
Table 3-(A, B) reports results under a centralized training setting, where all data is aggregated at single site, effectively eliminating the challenges posed by data decentralization in FL. In this scenario, single client possesses the entire dataset, and the MLP model is trained for 80 epochs using the same optimizer and learning rate as in the FL setup (Section 4). As expected, centralized training consistently outperforms federated learning across most feature extractor combinations for a single cycle (Tables 2 and 3). This is primarily because, in FL, the model undergoes incremental updates over multiple communication rounds rather than maturing a single training cycle. Cou- sequently, in FL, the learning process more gradual, and convergence takes longer compared to centralized setting where the model has access to the complete dataset at all times. Evaluating models in centralized setup provides vatuable baseline for assessing the effectiveness of different feature extractors in a non-FL encironment. The results help determine whether feature extractor is inherently strong or is hindered by federated constraints such as non-llD data distribution and communication limitations. The highest accuracy is achieved by the Internvidco2.5 model whereas the highest accuracy obtained with BLIP variation is 94.30% (Table 3-A, row 15) by the Slow and BLIP model when taken in conjunction with UCF101 and 71% (Table 3-B, row 15) for the HMDB51 dataset.
14

---

Published in Transactions Machine Learning Research (12/2025)
Table 4: Comparison of ActionCLIP and FLAMeST across different datasets and performance
metrics in Centralized Non-FL Setting.
Accuracy (%6)
Train Time
Test Time
Trainable
Dataset
Method
(soc)
(soc)
Parameters]
UCF101
ActionClip
95
242.993
96.51
150-155 million
FLAMeST
93.34
5.862
0.595
million
HMDB51
ActionClip FLAMEST
75.89
139.170
15.001
150-155 million
73.36
3.637
0.379
million
Training Accuracy for UCF101
Training Accuracy HMDB51
1080
30
Training Accuracy
Training Accurocy
22.5
30
33.0
ActionCLP FLAMeST
Number of Epochs
Epochs
Figure Centralized training accuracy trends of ActionCLIP and FLAMeST over 80 epochs. The graph illustrates how the models perform across training, highlighting the stability and convergence behavior of each method.
5.6 Comparison of ActionCLIP and FLAMeST
We trained the ActionCLIP* model from scratch, utilizing its default parameter settings, for total of 80 epochs on our specific data partition (Section4). The results of this training are presented in Table 4, which provides a comparative analysis between FLAMeST and ActionCLIP. The com- parison reveals that ActionCLIP outperforms FLAMeST by achieving 2% higher accuracy on both UCF101 and HMDB51. However, PLAMeST demonstrates notable advantages in terms of efficiency, with significantly reduced training and inference times compared to ActionCLIP. Fur- thermore, MeST operates with substantially lower number of training parameters, making more lightweight and computationally efficient option relative to ActionCLIP. Figure 3 shows that over the epochs, FLAMeST also acquires comparable training accuracy to ActionCLIP, leading to the model convergence.
5.7 Understanding Embedding Quality through UMAP Projections
To assess the quality of the embeddings generated by the BLIP and Slow 3D CNN, we utilize UMAP widely used dimensionality reduction and visualization technique for high-dimensional data (MeInnes et al., 2018). For this analysis, randomly select two classes from each dataset,
15

---

Published in Transactions Machine Learning Research (12/2025)
PLAMeST Embesdiing for UCT101
FLAMEST Emoeddingg for HMCB51
Ckss
12.
143
13.
UMAP Llimension 1
(a) FLAMeST
(b) FLAMeST
Figure 4: UMAP visualization of different embedding strategies on selected classes from the UCF101 and HMDB51 datasets. For UCF101, consider the classes 'Apply Lipstick' and 'Archery'. For HMDB51, the selected classes are 'Catch' and 'Cartwheel.'
UCF101 and HMDB51, and visualize the corresponding embeddings in a two-dimensional space. As illustrated in Figure (a,b), the FLAMeST embeddings for UCF101 and HMDB51 form well- seporaled and compact clusters, indicaling that the combined representation effectively distinguishes between the two selected classes, demonstrating the discriminative power of the embeddings. More UMAP Figures A-3 are provided in Supplementary C.I.
6 Ablation Studies
The experimental configuration for the ablation study, including the model architecture, hyperpa- rameters, and data partitioning strategies, remains consistent with the sctup outlined Scction 4. The results quoted are the best accuracy obtained by the global model over 80 rounds.
6.1 Client-Side Model Update
Table 5 evaluates federated optimization algorithms-Feddae (McMahan et al., 2017), FedProx (Li et al., 2020) and FedDyn (Acar et al., 2021) under both IID and Non-IID data distributions. FedAvg (EQ. 11) is the baseline aggregation method, where client updates are averaged without accounting for data distribution differences. In contrast, FedProx introduces a proximal term in the local objective function to address client data heterogeneity (FQQ. A-1). Whereas FedDyn uses dynamic update of the local objective loss function to ensure consistency of the local model update with the global model update (EQ. A-2).
Table 5 shows that all three methods perform comparably under the I1D setting, Whereas for the Non-IID case, FedProx outperforms FedAvg and FedDyn for the UCF101 and HMDB51 datascts. This improvement highlights the effectiveness of FedProz handling the stolistical challenges posed by heterogeneous data encironments, making more suitable choice in world scenarios where data is often non-uniformly distributed across clients.
16

---

Published in Transactions Machine Learning Research (12/2025)
Table 5: Comparison of client update methods on IID and Non-IID settings across datasets (4 clients).
Table 6: Performance of different MLP archi- tectures on the UCF101 dataset.
S.no
Hidden Layers
Train Acc. (%)
Test Acc. (%)
Dataset
Method
IID (%)
Non-IID (%)
[51g, 256/
99.07
91.89
(256, 256]
98.50
91.30
UCF101
FedAvg
90
84
[1024, 512)
99.37
96.22
FedProx
93
86
[1024, 512, 266
98.48
94.
FodDyn
91
84
[1024, 1024, 1024]
98.80
96.30
HMDB51
FodAvg
65
FodProx
67
FodDyn
69.15
6.2
Classifier Network Architecture
As shown in Table 6, increasing the width of hidden layers improves test performance, with the [1024, 1024, (row] 5) model achieving the highest accuracy of 96.%.. However, deeper networks do not always yield better results results-[1085 [1024, 512, (ro 4) underperforms compared to the simpler (1024, 512] (row 3) model, suggesting that additional depth may introduce optimization difficulties or lead to diminishing returns. All architectures achieve high training accuracy (>98%), indicating minimal overfitting. moderulely deep and wide MLP is often sufficient achieve strong generalization, while excessively deep models may offer significant gains.
6.3 Effect of Epochs on Model Performance
Table 7 presents the impact of training epochs on model accuracy. For UCF101 under IID settings, accuracy begins at 96% with epochs and stabilizes near 94.8% by epoch 15, indicating early convergence. In contrast, Non-IID performance fluctuates. It rises from 84% to peak of 86% at epoch 10, then declines to 83%, suggesting potential overfitting under data heterogeneity. On HMDB51, the IID model improves consistently from 70% to 73%, whereas the Non-lID model shows modest gains from 58% to 61%, further highlighting the challenges of learning from skewed distributions
While increasing the number of epochs helps improve or stabilize performance under IID settings, the same trend does not hold under NDn-IID conditions.
6.4 Scalability with Increasing Clients
For Scalability Approaches 1A, 1B, and 2, the total dataset size remains constant, so as the number of clients increases, the amount data per client decreases. contrast, for Approaches 3A and 3B, the data assigned each client remains fixed, mcaning the per-client dataset size does not decrease when the number of clients increases.
6.4.1 Scaling Approach: 1A & 1B
Table 8 presents the performance FLAMeST as the number of clients increases a cross-silo FL setting, where each client typically represents an institution. We limit the client count to 10 reflect realistic deployment scenarios. The data is divided among the clients unequally, i.e., one client may get more data and the other may get much less. Experiment 1A corresponds to
17

---

Published in Transactions Machine Learning Research (12/2025)
IID distribution, and Experiment 1B corresponds to a non-llD distribution among the clients. As the number of clients increases, classification accuracy declines for both the UCF101 and HMDB51 datasets in the IID (rows and 2) and Non-IID scenarios (rows and 4). The degradation is primarily due to reduced data per client as the number of clients increases, which weakens local training and limits global model generulization.
6.4.2 Scaling Approach: 2
We investigate label-skewness scaling under the constraint that the per-class-per-cline dataset sizes remain fixed. Each client is first assigned a balanced base dataset containing cqual number of samples from every class. Class-specific heterogencity is then introduced through probabilistic allocation mechanism (Supplementary Section A.4). Specifically, class proportions are drawn from a Beta distribution to determine the relative representation of each class, while a Bernoulli distri- bution identifies the subset of clients (takers) associated with that class. Once the participating clients for a given class are selected, the corresponding sample quotas are evenly distributed among them, thereby cnsuring fairness in allocation while preserving skewness in label distribution. As observed in Table 8 (rows and 6), an increase in the number of clients leads to gradual decline in classification accuracy for both UCF101 and HMDB51, primarily due to the heightened data heterogeneity and decrease in the data per client. Interestingly, for HMDB51, scaling results in an accuracy improvement that even surpasses the performance achieved under the IID setting.
6.4.3 Scaling Approach: 3A & 3B
To assess the scalability of foderated learning (FL) under realistic distributed conditions, we perform a single Dirichlet partition of the dataset with concentration parameter 3 0.6 (Non-IIID) and 3 =0 10 (IID) into ten clients. This partition is created once and reused across all experiments (Supplementary Section A.5). Each client permanently retains its assigned local dataset, ensuring that the per-client data remains fixed irrespective of the number of participating clients. No data reshuflligg or redistribution is performed when varying the number clients participating in FL. Initially, 4 randomly selected clients participate in FL, and we compute the metrics. The next 8 clients participate in FL, including the previously selected ones and newly randomly selected; however, all models are reinitialized from scratch. The next 10 clients participale in FL, including the 8 previously selected ones and 2 newly randomly selected; however, all models are reinitialized from scratch.
The nested design i.e. incremental client addition is essential to maintain experimental consistency as it guarantees that any observed performance variation arises solely from the increase in the number of participating clients, rather than from changes data composition. Experiments 3A and 3B correspond IID and Non-IID distributions, respectively. As shown in the Table 8 (rows 7,8,9,10), the average improves as the number of participating clients increases. For ITD there is an increases from 89.18% (4 clients) to 91.40% (10 clients) for UCF101 and 59.70% (4 clients) to 64.68% (10 clients) for HMDB51 respectively (row 7,8) The average global accuracy in the case of Non-lID also increases from 82.91% % (4 clients) to 87.63% (8 clients) and 88.04% (10 clients) for UCF101 (row 9) and 49.67% (4 clients) to 59.68% (10 clients) for HMDB51 (row 10).
This trend indicates that adding more clients contributes to broader data coverage and richer inter-client diversity, allowing the global model to generalize better. A similar pattern is observed across different random seeds, where incrcasing the federation size enhances model performance.
18

---

Published in Transactions Machine Learning Research (12/2025)
Table 7: Accuracy (%) across varying training epochs on UCF101 and HMDB51 over 80 cycles for 4 clients.
UCF101
UCF101
HMDB51
HMDB51
S.no
Epochs
ITD
Non-IID
IID
Non-IID
96.0
84.0
70.0
58.0
94.7
86.0
71.0
59.0
3 15
94.8
83.0
73.0
61.0
Table 8: Accuracy (%) across varying client counts on UCF101 and HMDB51 over 80 cycles with
5 local epochs. Rows are partitioned based on scalability experiments 1A, 1B, 2, 3A, and 3B.
S.no
Setting
Clients 8 Clients
10 Clients
Scalability 1A
UCF101 (IID)
96
93
91
2
HMDB51 (IID)
67
63
57
Scalability 1B
UCF101 (Non-1ID)
86
82
77
4
HMDB51 (Non-IID)
55
47
37
Scalability 2 Fixed Class per Client
UCF101
84.93
83.93
82.49
6
HMDB51
74
72.96
72
Scalability 3A Fixed Dataset per Client (IID)
UCF101 (IID)
89.18
90.74
91.40
8
HMDB51 (IID)
59.70
63.30
64.68
Scalability 3B Fixed Dataset per Client (Non-IID)
UCF101 (Non-IID)
82.91
87.63
88.04
10
HMDB51 (Non-IID)
49.67
58.44
59.68
Overall, these results confirm that the proposed FT. framework effectively scales with distributed data growth, demonstrating stable convergence and improved accuracy the federation expands.
6.5 Fusion by Gated-Attention
order to study the effectiveness of a more advanced fusion techuique than plain concatenation. we have experimented with residual gated cross fusion (RGCF) between image and text embed- dings (Refer to Supplementary section A.6). Table shows that simple concatenation consistently outperforms RGCF on both UCF101 and HMDB51, even when RGCF is trained for more epochs. However, the accuracy of RGCF does improve with increased training- foniing-fromm 75% to 89.5% on UCF101 and from 56% to 58.8% on HMDB51, indicating that RGCF may benefit from longer training to stabilize. We suspect the average performance of RGCF is due to the addod complexity and parameterization of RGCF, leading to overfitting on limited local data. In contrast, simple concatenation preserves the full representational capacity of CNN and VLM embeddings without additional transformations.
19

---

Published in Transactions Machine Learning Research (12/2025)
Table 9: Performance comparison of Residual Gated Cross Fusion (RGCF) and simple concatena- tion method on UCF101 and HMDB51 datasets over 80 cycles and clients.
S.no Dataset
RGCF (5 epochs)
RGCF (50 epochs)
Simple Concatenation
UCF101
75
89.5
94.43
HMDB51
56
58.8
70
7 Failure Analysis
We observe that broadly, the error cases fall into three main categories: (A) selection of uninfor- mative frames, (B) high inter-class similarity, and (C) noisy or incomplete caption VLM. Detailed discussion of failure cases is presented in (Supplementary D) and their mitigation strategies are discussed (Supplementary D.1).
Uninformative Frame Selection: In FLAMeST, frames are sampled randomly from the video clips. Consequently, the selected frame may not adequately capture the action being performed. The frames corresponding to the "kayaking" and 'haircut'' classes fail to depict critical visual cues such as a kayak or scissors objects that are central to recognizing the activity (Figures A-5-A and A-6-A).
High Inter- Class Similarity: Certain action classes exhibit substantial visual overlap, particularly when temporal context omitted. For example, "kayaking" and "rafting" both involve similar water-based environments and the presence of boats, making static frame-based distinction challenging (Figures A-5-B and A-6-B).
Noisy or Incomplete Captions: The VLM's inability to generate accurate and de- scriptive captions for the selected frames (Supplementary Figures A-5-C and A-6-C) along with the qualitative analysis of the captions generated by VLM shown in Supplementary Section B suggests that captions alone often insufficient for complex actions as text can be noisy. Our fusion with Slow CNN embeddings mitigates this, but reliance on captions remains limitation.
8 Broader Applicability
Although our experimental study focuses on video action recognition on UCF101 and HMDB51, the FLAMeST design is more general. The key idea of using frozen multimodal backbones to com- pute local embeddings and federating only a lightweight prediction head, matches recent trends parameter-efficient federated learning with foundation models. principle, the same template can be instantiated for other video and multimodal problems where task can be formulated as mapping from pro-trained embeddings to outputs, such multi-label event detection, gesture recognition, video retrieval, egocentric activity understanding, audio visual classification. Sim- ilarly, the BLIP + 3D CNN backbone can be replaced by stronger task-specific encoders (e.g., alternative VLMs or video models like InternVideo-style architectures). and even heterogeneous backbones across clients, as long as their reprosentations are projected into a compatible feature space. In all these cases, federating only a small head (or other light adaptation modules) keeps communication and compute costs manageable while allowing the framework to benefit from more diverse datascts and model configurations.
20

---

Published in Transactions Machine Learning Research (12/2025)
9 Conclusions and Future Work
This study introduces MEST, an approach for integrating foundation models within an FL framework enhance VAR performance. We leverage embeddings from the BLIP model along- side features extracted by 3D CNN model called Slow. These combined representations train lightweight MLP in the federated cycle, substantially reducing communication overhead compared to transmitting the full foundation model. The fusion of semantic and visual embeddings yields notable accuracy gains on challenging benchmarks such as HMDB51 and UCF101. Ablation studies confirm FLAMeST's robustness across diverse update schemes and its scalability under both IID and Non-IID data distributions. Moreover, the MeST embeddings form well-clustered class representations, highlighting their discriminative richness. Comparative analyses against al- ternative strategies further demonstrate how FLAMeST effectively addresses key challenges in VAR tasks in collaborative set-up.
As part of future work, we intend to investigate distilled variants of these foundation models to re- duce storage and computation costs, thereby improving the practicality of edge-device FL. Another direction involves enabling client-side customization, wherein users can record and label video clips. This introduces new challenges around server-side training and privacy preservation, which merit deeper investigation. As future work, we plan to explore distilled lightweight video-language cap- tioners that could be feasible for deployment in federated learning without compromising privacy.
Acknowledgement
The authors acknowledge support from the Department of Computer Science and Engineering for providing access to a high-performance computing facility based on NVIDIA DGX equipment. The authors also thank the Government of India (Gol) ANRF/SERB project CRG/2023/007223 for partial financial support.
References
Sawsan AbdulRahman, Hanine Tout, Hakima Ould-Slimane, Azzam Mourad, Chamseddine Talhi, and Mohsen Guizani. A survey on federated learning: The from centralized to distributed on-site learning and beyond. IEEE Internet of Things Journal, 8(7):5476 5497, 2020.
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. CoRR, abs/2111.04263, 2021. URL http::///xxxvv.or/abbbb httgg/ab//111..2225 042663.
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in neural information processing systems, 34:24206-2422 222211, 2021.
Mahmoud Al-Faris, John Chiverton, David Ndzi, and Ahmed Isam Ahmed. A review on computer vision-based methods for human action recognition. Journal of imaging, 6(6):46, 2020.
Khaled Alomar, Halil Ibrahim Aysel, and Xiaohao Cai. Runs, cnns and transformers in human action recognition: A survey and hybrid model. arXiv preprint 07000440,.06162, 2024.
21

---

Published in Transactions Machine Learning Research (12/2025)
Aaron F. Bobick and James W. Davis. The recognition of human movement using temporal tem- plates. IEEE Transactions on pattern analysis and machine intelligence, 23(3):257-2677 2001.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 629966999900 6308, 2017.
Keval Doshi and Yasin Yilmaz. Federated learning-based driver activity recognition for edge devices. In Proceedings of the IEEE/CVF Conference on computer Vision and Pottern Recognition, pp.
3338 3388-336,44 2022.
Keval Doshi and Yasin Yilmaz. Privacy-preservnng video understanding via transformer-basee federated learning. In 2029 IEEE Conference on Dependable and Secure Computing (DSC), pp. 1.8. IEEE, 2023.
European Union. Regulation (eu) 2016/679, 2016. URL https://eur-lex.eurre
Haoqi Fan, Tullic Murrell, Heng Wang, Kalyan Vasudev Alwala, Yanghao Li, Yilci Li, Bo Xiong, Nikhila Ravi, Meng Li, Haichuan Yang, al. Pytorchvideo: A deep learning library for video understanding. In Proceedings of the 29th ACM international conference on multimedia, pp-
3783 3786, 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202 6211, 2019.
Md Osman Gani, Taskina Fayezeen, Richard Povinelli, Roger O Smith, Muhammad Arif, Ahmed Kattan, and Sheikh Tqbal Ahamed. A light weight smartphone based human activity recognition system with high accuracy. Journal of Network and Computer Applications, 141:59 72, 2019.
Andrew A Gumbs, Vincent Grasso, Nicolas Bourdel, Roland Croner, Gaya Spolverato, Isabella Frigerio, Alfredo Manes,, Mohammad Abu Hilal, Adrian Park, and Eyad Elyan. The advances in computer vision that enabling more autonomous actions in surgery: systematic review of the literature. Sensors, 22(13):4918, 2022.
Jingwen Guo, Hong Liu, Shitong Sun, Tianyu Guo, Min Zhang, and Chenyang Si. Fsar: federated skeleton-based action recognition with adaptive topology structure and knowledge distillation. In Proceedings of the IEFE/CVF International Conference Computer Vision, pp. 10400-10410, 2023.
Mohammed Mehedi Hassan, Md Zia Uddin, Amr Mohamed, and Ahmad Almogren. A robust human activity recognition system using smartphone sensors and deep learning. Fulure Generation Computer Systems, 81:307-313, 2018.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cans at the edge. Advances in neural information processing systems, 33:1406s 14080, 2020.
Pranjal Jain, Shreyas Goenka, Saurabh Bagchi, Biplab Banerjee, and Somali Chaterji. Federated action recognition heterogeneous embedded devices. arXiv preprint arXiv:2107.12147, 2021.
22

---

Published in Transactions Machine Learning Research (12/2025)
Mohd Javaid, Abid Haleem, Ravi Pratap Singh, and Mumtaz Ahmed. Computer vision to enhance healtheare domain: An overview of features, implementation, and opportunities. Intelligent Pharmacy, 2024.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions pattern analysis and machine intelligence, 35(1):221-231, 2012.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurelien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning, Foundations and trends@ in machine learning, 14(1-2):1-21) 210, 2021.
Stefan Kalabakov, Borche Jovanovski, Daniel Denkovski, Valentin Rakovic, Bjarne Pfitzner, Orhan Konak, Bert Arnrich, and Hristījan Gjoreski. Federated learning for activity recognition: A system level perspective. IEEE Access, 11:64442-6 -4447,, 2023.
Evangelos Kazakos, Jacsung Huh, Arsha Nagrani, Andrew Zisserman, and Dima Damen. With little help from my temporal context: Multimodal egocentric action recognition. arXic preprint orXii:2111.01022 2021.
Alex Krizhevsky, Tlya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Hildegard Kuehne, Hueihan Jhuang, Estíbalix Garrole, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on com- puter vision, pp. 2556-2563. IEEE, 2011.
Le, Kiet Tran- Trung, and Vinh Truong Hoang, A comprehensive review of recent deep learning techniques for human activity recognition. Computational Intelligence and Neuroscience, 2022(1): 2022.
Chenglin Li, Di Niu, Bei Jiang, Xiao Zuo, and Jianming Yang. Meta-har: Federated representation learning for human activity recognition. In Proceedings of the web conference 2021, pp. 912 912-922, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-inage pre- training for unified languae understanding and generation. In International conference on machine learning, pp. 12888-12900 PMLR, 2022.
Tian Li, Anit Kumar Sahu, Manzil Zahoor, Maziar Sanjabi, Amoct Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-4510 2020.
Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tianjian Chen, Han Yu, and Qiang Yang. Fedvision: online visual object detection flatform powered by federated learning, In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13172-13179, 2020.
23

---

Published in Transactions Machine Learning Research (12/2025)
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2202-3211. 2022.
Lloyd. Least squares quantization pem. IEEE Transactions on Information Theory, 28(2): 129-137, 1982. doi: (0.1109/TIT.198 1956480.
Priyanka Mary Mammen. Federated learning: Opportunities and challenges. arXiv preprint arXio:2101.05428, 2021.
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint orXiv:1802, 800000000,,, 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agucra y Arcas. Communication--fficcee learning deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273 1282. PMLR, 2017.
Huu Phong Nguyen and Bernardete Ribeiro. Video action recognition collaborative learning with dynamics via pso-convnet transformer. Scientific Reports, 13(1):14424, 2023.
Richard A Posner. Privacy, surveillance, and law. U. Chi L. Rev., 75:245, 2008.
Riccardo Presolto, Gabriele Civitarese, and Claudio Bettini. Fedclar: Federated clustering for personalized aanso-baased human activity recognition. In 2022 IEEE international conference on pervasive computing and communications (PerCom), pp. 227-236. IEEE, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PmLR, 2021.
Yasar Abbas Ur Rehman, Yan Gao, Jiajun Shen, Pedro Porto Buarque de Gusmao, and Nicholas Lane. Federated self-supervised learning for video understanding. In European Conference on Computer Vision, pp. 506-30 522. Springer, 2022.
Oindrila Saha, Grant Van Horn, and Subhransu Maji. Improved zero-shot classification by adapting vims with text descriptions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 17542-17552, 2024.
Qiang Shen, Haotian Feng, Rui Song, Stefano Teso, Fausto Giunchiglia, Hao Xu, et al. Federated multi-task attention for cross-individual human activity recognition. In IJCAI, pp. 3423 3429. International Joint Conferences Artificial Intelligence, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. XrXiv preprint or Xin: 1409. 4,09.1556 2014.
Taein Son, Soo Won Seo, Jisong Kim, Seok Hwan Lee, and Jun Won Choi. Jovale: Detecting human actions in video using audiovisual and language contexts. arXiv preprint arXiv:2412. 181008, 2024.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from vidcos in the wild. orXiv preprint ar 1r1:::2212.0402, 2012.
24

---

Published in Transactions Machine Learning Research (12/2025)
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang, Videomae: Masked autoencoders are data- efficient learners for self-supervieed video pre-training Advances in neural information processing systems, 35:10078-10093 2022.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 1899-4497, 2015.
Nguyen Anh Tu, Assanali Abu, Nartay Aikyn, Nursultan Makhanov, Min-Ho Lee, Khiem Le- Le-Huy, and Kok-Seng Wong. Fodfslar: A federated Icarning framework for few-shot action recognition. In Proceedings of the IEEE/CVF Winler Conference on Applications of Compuler Vision, pp. 270-279 2024a.
Nguyen Anh Tu, Nartay Aikyn, Nursultan Makhanov, Assanali Abu, Kok-Seng Wong, and Min-Ho Leo. Benchmarking federated few-shot learning for video-based action recognition. IEEE Access, 2024b.
Anwaar UTaaq, Naveed Akhtar, Ganna. Pogrebna, and Ajmal Mian. Vision transformers for action recognition: survey, arxiv preprint urXiv:220.057000 2022.
Mengmeng Wang, Jiazheng Xing, Jianbiao Mci, Yong Liu, and Yunliang Jiang. Actionclip: Adapt- ing language-image pretrained models for video action recognition. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Qu Wang, Haiyong Luo, Hao Xiong, Aidong Men, Fang Zhao, Ming Xia, and Changhai Ou. Pedes- trian dead reckoning based on walking pattern recognition and online magnetic fingerprint tra- jectory calibration. JEEE Internet of Things Journal, 8(3):2011 2026, 2020.
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2.5: Empowering video milmns with long and rich context modeling. Xixiv preprint ar arXi::2001. 12386, 2025.
Orion Weller, Michael Boratko, Iftekhar Naim, and Jinhyuk Lee, On the theoretical limitations of embedding- retrieval. arXiv preprint ar Xrxi::2008. 21038, 2025.
Peiran Xu, Zeyu Wang, Jieru Mei, Liangqiong Qu, Alan Yuille, Cihang Xie, and Yuyin Zhou. Fedconv: Enhancing convolutional neural networks for handling data heterogeneity in federated learning, Xixiv preprint ar arXio:2310.0441 2023.
Jiewen Yang, Xingbo Dong, Liujun Liu, Chao Zhang, Jiajun Shen, and Dahai Yu. Recurring the transformer for video action recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14063-14773 2022.
Xiaoshan Yang, Baochen Xiong, Yi Huang, and Changsheng Xu. Cross-modal federated human activity recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Lanfei Zhao, Zixiang Lin, Ruiyang Sun, and Aili Wang. A review of state-of-the-art methodologies and applications in action recognition. Electronics, 13(23):4733, 2024.
25

---

Published in Transactions Machine Learning Research (12/2025)
Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motivations, challenges, and future directions. arXie preprint arXiv:2306.1554 15546, 2023.
26


Reviews:
[{"review_id": "str40V1Cm1", "reviewer": "Reviewer_zJ5z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes FLAMeST, a federated learning framework for video action recognition that integrates BLIP and spatiotemporal CNNs to balance privacy, efficiency, and performance. Unlike prior works that use BLIP only for captioning, FLAMeST fuses semantic and motion embeddings to train a lightweight MLP, sharing only MLP parameters during aggregation with FedAvg, thereby preserving privacy and reducing communication overhead by 99%. Experiments on UCF101 and HMDB51 show significant performance gains over baselines (+5.13% and +2.71% accuracy).", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most of claims are accurate", "audience_yn": "Yes", "audience_explanation": "Yes. People working in the Video Action Recognition domain may be interested in this paper.", "requested_changes": "1. Figure 1: The FL procedure should be explicitly illustrated. Since Figure 1 presents the FLAMeST pipeline, it should clearly show how federated learning integrates with the video action recognition (VAR) component.\n\n2. Table (1): The comparison methods are outdated. The authors should consider including more recent approaches from 2024 and 2025 to provide a stronger baseline.\n\n3. Tables (2) and (3): These appear to be ablation studies. However, the direct comparison between the proposed FLAMeST framework and the baselines listed in Table (1) is missing and should be added.", "broader_impact_concerns": "N/A.", "additional_comments": "No.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GB9r3XRXDn", "reviewer": "Reviewer_ioup", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a new method for federated learning for video-based action recognition (VAR) that leverages modern visual-language models (e.g., BLIP) to generate more detailed and discriminative embeddings. At its core, the method combines several ideas: 1) extracting embeddings from given video clips with pre-trained foundational models can dramatically decrease the number of weights that need to be exchanged during the federated learning step, and 2) in contrast to previous VAR approaches that leveraged VLMs, using only captions for training an action classifier is not optimal; therefore, combining both text-based caption embeddings and visual embeddings yields better performance. The authors benchmark their method on a number of popular action recognition datasets and demonstrate the effectiveness and efficiency of the proposed approach.\n\nWeaknesses: \n\n* The contribution of the method is mostly interesting from an engineering perspective, as it primarily demonstrates that modern pre-trained foundational and vision-language models can already produce semantically rich and useful embeddings that can be successfully utilized in downstream tasks such as video action recognition. While this is a valuable observation from a practical standpoint, the paper does not provide substantial novel theoretical insights or methodological innovations beyond adapting existing models, which may limit its impact from a research perspective. Nevertheless, acknowledging this engineering angle, the paper remains valuable in highlighting the applicability and efficiency of such embeddings in federated learning setups.\n\n* The method contains a step where one selects key frame(s) from the video and generates captions for them (followed by generating embeddings with a VLM cross-encoder). This particular step feels somewhat ad hoc. One potential alternative would be to use video-language models (such as Qwen-VL or Cosmos-Reasoning variants) that generate textual descriptions for the entire video and also support embedding generation with cross-attention. It would be very interesting to see how such an approach performs, since the (semi)manual selection of key frames might severely underperform, while a caption for the entire video should describe the action much more precisely.\n\nStrengths: \n\n* The point about using text captions for describing complex video actions not being sufficient is interesting and useful for a broader audience. In a nutshell, with this claim the authors suggest that simple text-based classification/retrieval for videos is underperforming and that additional information from the visual component is necessary.\n\n* The idea of using pre-trained models that can be shared across different clients is both practical and useful, as it significantly reduces the total amount of weights that need to be transferred during training. This not only lowers communication costs in federated learning setups but also makes the approach more scalable and applicable in real-world scenarios where bandwidth and efficiency are critical.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by empirical evaluation on several standard video action recognition datasets, which demonstrates both the effectiveness and efficiency of the proposed approach. The authors clearly show that combining visual embeddings with caption-based embeddings outperforms using captions alone, and they provide benchmarks to substantiate this claim. The results also convincingly illustrate the practicality of reducing communication costs in federated learning through the use of shared pre-trained models.", "audience_yn": "Yes", "audience_explanation": "The paper will be of interest to members of the TMLR audience working on federated learning, video-based action recognition, and the integration of large pre-trained models into downstream tasks. The findings highlight practical ways to reduce communication costs in federated learning by leveraging pre-trained embeddings and show that relying solely on text captions for video understanding is insufficient (which popular approaches to the task do).", "requested_changes": "* The current choice of selecting key frames manually (or semi-manually) for caption generation feels ad hoc. The authors should provide a clearer rationale for this choice or evaluate an alternative—e.g., using video-language models (such as Qwen-VL or Cosmos-Reasoning variants) that generate captions for entire videos—to ensure this step is not a weak link in the method.\n\n* While modern VLMs generally produce good captions, for very complex behaviors their descriptions may lack sufficient detail to accurately represent the action, which can lead to weaker text embeddings (especially when using only one frame for captioning). This limitation should be explicitly discussed.\n\n* In addition to the previous comment, the current evaluation does not clearly show how accurate the generated captions for the key frames are. Do they truly describe the observed action in the video? Demonstrating this would make it clearer that the method works as intended.\n\n* One of the recent papers [1] suggests that using only text-based retrieval is suboptimal and that additional strategies—such as multi-vector retrieval—are needed. These findings are aligned with the observations in the current paper (e.g. using both visual + text-based embeddings) and should be explicitly discussed to place the work in the context of ongoing research.\n\n[1] Weller, Orion, et al. \"On the Theoretical Limitations of Embedding-Based Retrieval.\" arXiv preprint arXiv:2508.21038 (2025).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u11L1drQV9", "reviewer": "Reviewer_VnGY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a Federated Learning (FL) framework for Action Recognition (AR) models, leveraging cross-modality embeddings (semantic, image, and temporal) and partial parameter aggregation to enhance efficiency. The experiments validate the effectiveness of the proposed embeddings, the computation efficiency, and effectiveness of the FL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The paper is well written, with clear exposition of methodology and objectives.\n- Empirical validation is thorough, featuring abundant experiments and ablation studies that robustly support the authors’ claims.\n- The technical approach is novel in combining cross-modality embeddings with partial FL parameter aggregation, addressing a gap in efficient FL for video understanding.", "audience_yn": "Yes", "audience_explanation": "The paper represents a solid contribution to efficient FL for AR, with rigorous experimentation.\n\nHowever, concerns persist about its practical relevance: \n- The work risks positioning FL as an end in itself. Given the rise of high-performing open-source foundation models, the paper does not sufficiently justify why FL is a superior or complementary direction for privacy-sensitive video understanding.\n- Leveraging private data via FL may not outperform strategies that enhance zero-shot performance using public data, especially given the flexibility of prompting/in-context learning in modern models. The absence of evidence positioning FL as a necessary tool for cutting-edge video understanding weakens its perceived significance.", "requested_changes": "- I am curious on how does FL-based AR fare against scalable open-source models (e.g., Internvideo2 [1]) that achieve strong performance without privacy constraints. The paper mentioned ACTION-CLIP only, which is a little bit out-of-date.\n- A key FL advantage lies in distributed data scaling. However, experiments fix the overall dataset size while varying client numbers. To better assess scalability, analyzing performance as client count increases (with per-client data fixed) would clarify whether FL’s benefits scale under realistic distributed settings.\n- The framework is framed as a general approach for video understanding, validation is limited to AR. Expanding discussions—or experiments—on its applicability to broader tasks (e.g., event recognition, video captioning) would strengthen claims of generality.", "broader_impact_concerns": "No Specific Concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)