MV2MAE: Self-Supervised Video Pre-Training with Motion-Aware Multi-View Masked Autoencoders
Authors: Ketul Shah
Robert Crandall
Jie Xu
Peng Zhou
Vipin Pillai
Marian George
Mayank Bansal
Rama Chellappa
Venue: Accepted by TMLR ()

Abstract: Videos captured from multiple viewpoints can help in perceiving the 3D structure of the world and benefit computer vision tasks such as action recognition, tracking, etc. In this paper, we present MV2MAE, a method for self-supervised learning from synchronized multi-view videos, built on the masked autoencoder framework. We introduce two key enhancements to better exploit multi-view video data. First, we design a cross-view reconstruction task that leverages a cross-attention-based decoder to reconstruct a target viewpoint video from source view. This helps in effectively injecting geometric information and yielding representations robust to viewpoint changes. Second, we introduce a controllable motion-weighted reconstruction loss which emphasizes dynamic regions and mitigates trivial reconstruction of static backgrounds. This improves temporal modeling and encourages learning more meaningful representations across views.
    MV2MAE achieves state-of-the-art results on the NTU-60, NTU-120 and ETRI datasets among self-supervised approaches. In the more practical transfer learning setting, it delivers consistent gains of +2.0 -- 8.5% on NUCLA, PKU-MMD-II and ROCOG-v2 datasets, demonstrating the robustness and generalizability of our approach. Code: https://github.com/kshah33/mv2mae

Published in Transactions on Machine Learning Research (03/2026)
MV2MAE: Self-Supervised Video Pre- Training with Motion- Aware Multi-View Masked Autoencoders
Ketul Shah Johns Hopkins University
kefuks27@gmail. com
Robert Crandall
verandal@Bmazzon.com
Amazon
Jie Xu*
jiemwwjBgmaillcom
Sony AI
Peng Zhou
pengshox@terrmaii.com und. edu
Amazon
Vipin Pillai
piliipin@omazzo.com
Amazon
Marian George"
mariang@google.com
Google
Mayank Bansal
maybons@emazon.com
Amazon
Rama Chellappa
rehella4@jhur edu
Johns Hopkins University
Reviewed on OpenReview: https: openreview. net/ forum? id- ngt35.JJkk
Abstract
Videos captured from multiple viewpoints can help in perceiving the 3D structure of the world and benefit computer vision tasks such as action recognition, tracking, etc. this paper, we present MV2MAE, method for self-supervised learning from synchronized multi- view videos, built on the masked autoencoder framework. We introduce two key enhance- ments to better exploit multi-view video data. First, design cross-view reconstruction task that leverages cross-atiention--aseel decoder to reconstruct target viewpoint video from source view. This helps in effectively injecting geometric information and yielding representations robust to viewpoint changes. Second, we introduce a controllable motion- weighted reconstruction loss which emphasizes dynamic regions and mitigates trivial recon- struction of static backgrounds. This improves lemporal modeling and encourages learning more meaningful representations across views. MV2MAE achieves state-of-the-art results on the NTU-60, NTU-120 and ETRI datasets among self-supervised approaches. In the more practical transfer learning setting, it delivers consistent gains of +2.0 8.5% on NUCLA, PKU-MMD-IT and ROCOG-v2 datasets, demonstrating the robustness and gemeralizability of our approach. Code: https:/ http:/gttttbccooo com/ htt///hhhhhhheeeeee my2mae
Work completed during internship Amazon. Work completed while Amazon.

---

Published in Transactions on Machine Learning Research (03/2026)
1 Introduction
Multiple viewpoints of the same event are crucial to its understanding. Humans move around and obtain different viewpoints of objects and scenes, and develop a representation robust to viewpoint changes (lsik) et al.) 2018). Different viewpoints often have very different appearance, which can help address challenges due to occlusion, lighting variations and limited field-of-view. In many real world scenarios, we have videos captured from multiple viewpoints, e.g. sports vidcos (Saito et al.) 2004], elderly care (Jang et al.) 2020). self-driving (Yogamani et al., 2019h; complex robotic manipulation lasks (Seo et al.] (023a) and securily videos (Corona et al. (021). Learning a robust pre-trained model from large amounts of unlabeled synchro- nised multi-view data. is of significant value for these applications. Such a model which is aware of the 3D geometry will be robust to changes in viewpoint and can be effectively used as foundation for downstream finetuning on smaller datasets for different tasks.
There has been significant progress in video self-supervised learning (Schiappa et al. 2023 (SSL) for the single-view case, í.e. where synchronized multi-view data is not available. Recently, Masked Autoencoders (MAEs) as a paradigm for self-supervised learning has seen growing interest, and it has been successfully extended video domain (Feichhennhofrr et al. 2022 Tong et al., 2022 Wang et al. 2023a). MAE-
based methods achieve superior performance (Tong et al. p standard datasets such as Kinetics-400
(Kay et al., 2017) and Sometting-Sommetiingg- Something-v2 v2 [Goyal et al. 2017,, compared to contrastive learning methods (Feichtenhofer et al. 2021). However, existing MAE-based pro-training approaches are not explicitly designed to be robust to viewpoint changes. View-invariant learning from multi-view videos has been widely studied using NTU (Shahroudy et al.) 2016 Liu et al. 2020a) and ETRI (Jang et al.] 2020) datasets. However, most of these methods use 3D human posc, which is difficult to accurately capture for in-the-wild scenarios. There has been a growing interest in RGB-based self-supervised learning approaches leveraging
multi-view videos (Parameswaran & Chellappa 2006 Das & Ryoo| 2023 Vyas et al. 2020 Li al. 2018).
Liu et al. 2020a Jang ct al. 2020). ViewCLR (Das & Ryoo 2023). which achieves state-of-the-art results among SSL methods,
facilitated by the availability of large-scale mulli-view datasets (Shahroudy et al. 2016
introd introduces uces a latent viewpoint generator as a learnable augmentation for generating positives in a rontrastive learning (Chen et al., 2020 framework. However, this method is memory intensive as it requires storing two copies of the feature extractor and two queues of features, while also requiring multi-stage training. In contrast, the recent success of MAEs for video SSL motivates us to explore its potential in the multi- view video SSL scenario.
In this paper, we aim to learn self-supervised video representations that are robust to viewpoint shifts. Humans learn representation robust to viewpoint variations for tasks such as action recognition and are able to visualize how an action looks from different viewpoints (lsik et al. 2018). Motivated by this, we design the task of using one viewpoint to predict the appearance from a different viewpoint, and integrate it in the MAE framework. More specifically, given a video of an activity from one viewpoint, it is converted to patches and a high fraction of the patches are masked out. The visible patches are encoded, which the decoder uses (along with MASK tokens for missing patches) to reconstruct the given video. We introduce an additional view decoder, which is tasked with reconstructing the masked patches of target viewpoint by using the visible regions from source view. This requires the model to understand the geometric relations between different views, making the pre-trained model robust to viewpoint shifts. Another challenge with MAE in videos is temporal redundancy, which makes it casier to reconstruct the static, background regions by simply copy pasting from adjacent frames where those are visible. Fxisting solutions for this problem involve specialized masking stratogics using extra learnable modules (Bandara et al., 2023; Huang et al., 2023) or tube masking (Tong et al., 2022 Wang et al., 2023a), which are not effective in certain scenarios, e.g. when motion is localized in a small region of the frame. We propose a simple solution, without introducing additional learnable parameters, by modifying the reconstruction loss to focus on moving regions. We can control the relative weights of moving and static regions using a temperature parameter.
We perform pre-training experiments on three multi-view video datasets: NTU-60 (Shahroudy et al., 2016) NTU-120 (Liu et al., 2020a), ETRI (Jang et al. 2020). Our method achieves SOTA accuracy on these action recognition benchmarks in the full fine-tuning protocol. More notably, the robustness of our representation is shown in the transfer learning results on smaller datasets. We achieve SOTA results on NUCLA (Wang)
2

---

Published in Transactions on Machine Learning Research (03/2026)
Source View (S)
0. K.
Self-View (S (S-S)
(I,, 18
Source View (S)
motion weights
Cross-View Decoder (T-+S)
(I 1 12-18
K,
Encoder
K,
Cross-View Decoder GT)
Cm( 1
Targel weights
Target View
Self-View Decoder (T (T-+T)
CI" CI" 17
Figure 1: Multi-Vice Video Masked Autoencoder (MV2MAE). Given synchronized videos from source (S) and larget (T) viewpoint, we first tokenize and encode visible patches using a shared encoder. We introduce a cross-view decoder to reconstruct one viewpoint from the other, while the self-view decoder reconstructs each view independently. The proposed motion-weighted reconstruction loss Lm (I, I), computed using the input clip I and reconstructed clip I, emphasizes dynamic regions and reduces the effect of trivial reconstruction static background regions. The motion weights are derived from the input clip. Decoders in the samee color share weights.
et al. 2014 ROCOG-v2 (Reddy et al. 2023 and PKU-MMD-II (Liu al. 2017) datascts in the transfer learning selting.
Our main contributions be summarized follows:
We present MV2MAE, self-supervised pre-training approach explicitly designed for synchronized multi-view videos, achieving state-of-the-art results among SSL methods on NTU-60, NTU-120, and ETRI benchmarks, as well as strong transfer gains (+2.0- 8.5%) on NUCLA, PKU-MMD-IL, and ROCOG-v2. We also show synthetic multi-view data as scalable alternative for pre-training data.
We introduce dedicated cross-view decoder that reconstructs target viewpoint video from source view, forcing the encoder to capture 3D geometry and learn viewpoint-robust representations.
We design simple yet effective motion-aware loss that emphasizes moving regions while down- weighting static backgrounds, mitigating trivial reconstruction and substantially improving temporal representation learning.
2
Related Work
2.1 Supervsue Learning from Videos
Pretext Learning. Many pretext tasks have been proposed for learning self-supervised video representa- tions, initially inspired from the progress in SSL for images. Tasks such as video rotation prediction (Jing et al. 2018), solving spatio-temporal jigsaw (Ahsan et al. 2019), predicting motion and appearance statis- tics Wang et al.) 2019) were direct extensions their image counterparts, and showed impressive per- formance. Methods leveraging the temporal order vidcos for constructing pretext tasks such as frame ordering (Xu al. 2019) and odd-one-out learning (Fernando et al.) 2017 were also proposed. These methods were outperformed by contrastive learning approaches.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Contrastive Learning. These methods create augmented versions of the input (positives) which preserve the semantic content of the input. The contrastive loss is used to pull these closer together in the feature space, while simultaneously pushing them away from other samples (negatives). Numerous ways of gener- ating positive pairs were proposed such as using random clips from the same vidco, clips of different frame rales [Wann et al. 2020), choosing nearby clips (Qian et al. 2021), and using optical flow (Han et al. 2020 among others.
Masked Video Modeling. Recently, masked video modeling has emerged a promising area for SSL.
Methods such as BEVT (Wang et al. 2022), , MaskFeat (Wei et al. 2022), VideoMAE (Tong et al. 2022
MAE-ST (Feichtenhofer et al. 2022] show superior performance on the standard video self-supervised ing benchmarks. Different reconstruction targets have been studied, such as MVD (Wang et al. which uses distillation from pre-trained features, and MME (Sun et al. 2023) which reconstructs motion trajectories. To tackle trivial reconstruction solution via copy-paste in videos, which becomes an issue due to high redundancy, different masking strategics have been proposed. MGMAE (Huang et al. 2023) uses motion-guided masking based on motion vectors, VideoMAE (Tong et al. 2022) proposed using tube mask- ing, AdaMAE (Bandara et al. 2023) introduces neural network for mask sampling. Orthogonal to these, we propose to tackle the issue by using a motion-weighted reconstruction loss. Morcover, unlike the proposed approach, existing MAE pre- pre-training approaches are not explicitly designed to be robust to viewpoint shifts, Beyond action recognition, the masked modeling paradigm has also been explored in adjacent fields such as robotics, where multi-view masked world models (Seo et al. 2023b) have been utilized for visual robotic manipulation.
learn-
2023b)
2.2 Multi-View Action Recognition
Early works in this area designed hand-crafted features which are robust to viewpoint shifts Paramessaaan & Chellappa 2006 Rao al.) 2002 Xia et al.) 2012). Many unsupervised learning approaches have been proposed for learning representations robust to changes in viewpoint. A large number of methods leverage
2D/3D human pose information (Shah et al.) 2022 Zhou et al. 2025), which greatly aids in achicving
robustness to viewpoint variations. Methods based on RGB modality (Das & Ryoo 2023 Li et al.] 2018
Vyas et al. 2020) have gained increasing popularity. These can be broadly divided into two categories:
One trend is to enforce the latent representations of diffcrent viewpoints to be close. Along this line, (Zeng et al. 2012) follows a dictionary learning approach and encourages videos of different views to have the same sparse representation. (Rahmani & Mian 2015) fits a 3D human model to a mocap sequence and generates vidcos from multiple viewpoints, which are forced to predict the same label. More recently, methods based contrastive learning have been proposed such as ViewCLR (Da & Ryoo 2023) which achieves remarkable performance. They add latent viewpoint generator module which is used to generate positives in the latent space corresponding to different views.
Another line of work uses one viewpoint to predict another. Li et al. 2018 uses cross-view prediction in 3D flow space by using depth as an additional input to provide view information. Their approach also uses gradient reversal layer for achieving robustness to view changes. (Vyas et al., (020)) uses the encoded source view features to render same video from unseen viewpoint and random start time. Their approach hence needs to be able to predict across time and viewpoint shifts. They leverage view embedding which requires information of camera. height, distance and angle. In contrast to these approaches which rely on view embedding or depth for providing viewpoint information, the view information is inherently available in the visible patches of the source and target viewpoints in our approach. While CrossMAE (Guo et al.] 2024] also uses cross-attention in the decoder, it does not explicitly address synchronized multi-view geometry, and RGB-based viespoint-agnostic methods such as 3D TRL (Shang et al. (202) rely on temporal relational learning rather than cross-vies view reconstruction, which is central to our approach.
4

---

Published in Transactions on Machine Learning Research (03/2026)
3 Method
3.1
Preliminary: Masked Video Modeling
Here we revisit the MAE framework for videos. Given a video, we first sample frames with stride T to get the imput clip: I E RC Here, W is the spatial resolution, T denotes the number of frames sampled, and C is the number of input (RGB) channels. The standard MAF architecture has three main components: tokenizer, encoder & decoder.
Tokenizer.
The imput clip is first converted into N patches using patch size of x h x w, where N = A w. The tokenizer returns N tokens of dimension d by first linearly embedding these N patches.
I x H W. This is implemented practice using a strided 3D convolution layer. Next, we provide position information to these tokens by adding positional embeddings (Vaswani al. 2017).
Encoder. A high fraction of these N tokens are dropped with masking ratio p E (0, Different masking strategies (Tong al. 2022. Bandara al. 2023 Huang al. 2023) have been explored for choosing which tokens to mask out. Next, the remaining small fraction of visible tokens are passed through the encoder (Fenc) to obtain latent representations. The encoder is vanilla ViT (Dosovitskiy et al. 2020) with joint space-time attention TTng et al. 2022). These latent representations need to capture the semantics in order to reconstruct the masked patches.
Decoder.
The encoded latent representations of the visible patches are concatenated with learnable [MASK] tokens corresponding
masked-out patches, resulting in combined tokens Z. The positional embeddings are then added for all tokens, and passed through light- decoder (Deec) to get the predicted pixel values i = aec (Z).
The loss function is the mean squared error (MSE) between the reconstructed values and the normalized pixel values (Feichtenhofer et al. 2022 Tong et al. 2022 for masked patches f2
1
C(I,î)
Σ 1, i PN iEf
(1)
3.2 Cross View Reconstruction
The goal of cross-view reconstruction is to predict the missing appearance of video in target viewpoint using the visible patches of video from source viewpoint. Being able to extrapolate across viewpoints requires understanding the geometric relations between different viewpoints, making il elfective task for learning representations robust to viewpoint variations.
As shown Figure 1 consider two synchronized videos an activity, Is and Is,, from source view (S) and target view (7) respectively. We first lokenize, mask and encode the visible tokens for each video separately using a shared encoder Penc- We then introduce cross-view decoder (Deress-riey which additionally uses dec
the visible tokens in the source view to reconstruct the target viewpoint video, 1 crose--view (z"" Z.S.). More specifically, each block of the cross-view decoder consists of cross-attention and self-attention layers, followed by ferd-forward layer. The tokens from the larget view atlend to the visible source view tokens using cross-attention, and then to each other using self-attention. Moreover, the standard decoder (Dase) is used to reconstruct video from each viewpoint independently "/ Psac(Z") for E {S,T). Figure 5 visualizes the cross-vies view reconstruction quality and cross-attention maps, which demonstrates that the model learns to focus on relevant regions across viewpoints.
&
A key aspect of methods using the cross-view prediction paradigm is how the viewpoint information is provided: (Vyas et al.) 2020) conditions the decoder on a viewpoint embedding, while some approaches (Li ct al.] 2018) use extra modalities such as depth to provide information about the target vicwpoint. contrast to these, in our approach, the visible patches provide the required target viewpoint information. The amount of view information want to provide can be easily varied by changing the masking ratio.
5

---

Published in Transactions on Machine Learning Research (03/2026)
3.3 Motion-Weighted Reconstruction Loss
A given vidco can be decomposed into static and dy- namic regions. Static regions typically involve scene background and objects which do not move through- out the video. Patches from such regions are triv- ial to reconstruct (Sun et al. 2023 Bandara et al.) 2023) due to temporal redundancy in videos. In or- der to deal with this, we offer a simple solution by re-weighting the reconstruction loss of each patch proportional to the amount of motion within that patch. The motion weights used for re-weighting are obtained using frame difference for simplicity. Note that other motion features such as optical flow, mo- tion history image, etc can used place of frame difference, but frame difference is extremely fast compute. In order to get the final weights, take the norm of frame difference within each patch, and apply temperature softmax over all tokens. We can control the extent to which to focus on the moving regions by controlling the temperature parameter. The higher the temperature value, the more uniform the resulting weights. Examples of motion weights overlaid on the original frames for different temper- ature values
Frame
Frame
Frame 5
Frame 8
20
60
200
Figure 2: Motion weights with varying temper- ature. Each row shows motion weights overlaid on the input frames for a different temperature. Higher temperature increases the weight on static and back- ground regions.
shown in Figure 2 PyTorch-style code for computing the motion weights for clip is provided in the supplementary. The final motion- reconstruction loss (Lm) is given below, where (T( is the weight for tth patch with temperature T:
C. (I,
1 P.N
Σ
(T( I, -
(2)
3.4 Pre-training and Evaluation
Pre-training. To sum we first sample two videos from a set of synchronized videos of an activity, I from source view and 1" from target view. The encoder, self-view decoder and cross-view decoder are pre-trained using the overall loss given below:
(IS, C. ( T-TT (IT ( + XL (IS, (IS,1" T-8
self-view reconstruction
cross-view reconstruction
Here, 8T denotes reconstructed video from target view using source view, and on. À is the weight for cross-view reconstruction loss and is set to 1.
Evaluation. Following prior work (Li et al. 2018 Kim et al. 2022 Vyas et al. 2020: Das & Ryoo 2023
we evaluate our pre-trained models on the task of action recognition using two settings: 1) full fine-Luning on the same datasets and 2) transfer learning on smaller datasets. We discard the decoders, and attach classifier head which uses the global average pooled features for classification. For testing, we sample 5 temporal clips, and use 10 crops from each following (Das & Ryoo 2023). taking their average for the final prediction.

---

Published in Transactions on Machine Learning Research (03/2026)
4 Experiments
We evaluate our approach on several common multi-view video datasots: NTU60 Shahroudy et al. 2016); NTU120 (Liu et al. 2020a). ETRI (Jang et al. 2020). NUCLA (Wang et al. 2014). PKU-MMD (Liu et al. 2017j, and ROCOG (Redyy et al., 2023). For NTU and ETRI, we achieve state-of-the-at results among SSL methods by pre-training and fine-tuning on the target domain. On NUCLA, PKU-MMD, and ROCOG, we demonstrate excellent transfer learning performance by pro-training only on NTU, and fine-tuning on the target dataset.
4.1 Datasets
NTU RGB+D 60. (Shahroudy et al. 2016) large-scale multi-view action recognition dataset, con- sisting of 56,880 videos from 60 distinct action classes. These vidcos were recorded from 40 subjects using Kinect-v2. Each activity instance is simultaneously captured from three viewpoints. The dataset consists of two benchmarks outlined in (Shahroudy et al. 2016): (1) Tross-Subject (xsub) and (2) Cross-Vice (xview). In the cross-subject benchmark, the 40 subjects are divided into training and testing sets, with 20 subjects in each. In the cross-view scenario, videos from cameras 2 and 3 are used for training, while testing is performed on videos from camera 1. This corresponds to using the front view and the +990 views for training, whereas using the intermediate =45 views for testing.
NTU RGB+D 120. (Lin et al. 2020a) is an extended version of the NTU-60 dataset containing 114,480 videos spanning 120 action categories. Our evaluation follows the established protocols outlined in {Liu
et al. 2020a): (1) Cross-Subject (xsub) and (2) Cross-Setup (xset). In the cross-subject scenario, subjects are partitioned into training and testing groups, while in the cross-sctup setting, the data is divided into training and Lesting subsets based on the setup ID.
ETRI. (Jang et al. 2020) is another large-scale multi-view action recognition dataset consisting of activities of daily living for elderly care. It has 112,620 videos captured from 55 action classes. All activity instances are recorded from 8 synchronized viewpoints. (Jang et al. 2020) describes a cross-subject benchmark which we use to evaluate our approach.
4.2 Implementation Details
We sample a clip of 16 RGB frames with a stride of from each vidco. We downsample the resolution of frames to 128x 128 following (Das & Ryoo 2023). During pre-training, we only apply random resized crops augmentation. We use temporal patch size of 2 and a spatial patch size of 16x16, which results in 512 tokens. A masking ratio of 0.7 is used unless otherwise specified. We choose fixed sinusoidal spatio-temporal positional embedding following (Tong et al.) 2022 Bandara et al. 2023). All of our experiments use the vanilla ViT-S/10 16 (Touvron et al. 2021) | Hutter 2017). The pre- pre-training is carried out for 1600 epochs. Please refer to the supplementary material for additional details.
architecture as the encoder, trained using AdamW optimizer (Loshchilov &
4.3 Comparison with state-of-the-art
We compare our approach with prior supervised and self-supervised approaches on the cross-subject (xsub) and cross-view (xview) benchmarks of the commonly used NTU-60 and NTU-120 datasets. We also present our results on the ETRI datasot, which only has a cross-subject benchmark.
Table [ and Table 2 show results on the NTU-60 and NTU-120 datasets. We outperform all previous self- supervised methods based on RGB, Flow or Pose modality on both cross-view and cross-subject benchmarks of the two datascts. In the xsub sctting, we see an improvement of +0.3% and +1.2% on NTU-60 and NTU- 120 respectively, and in the xview setting, we observe an improvement of +1.8% and +0.9% respectively. ViewCLR (Das & Ryoo 2023) uses a MoCo [Chen et al. 2020) framework which requires large batch-sizes for convergence (Chen et al. 2021a He et al. 2022). Vyas et al. (Vyas ct al. 2020 uses a cross-view prediction paradigm but underperforms (86.3% vs 95.9% on xview and 82.3% vs 90.0% on NTU60 xsub) despite using more parameters (~7 vs 22M) Unlike their approach which relies only on learnt viewpoint embeddings
7

---

Published in Transactions on Machine Learning Research (03/2026)
NTU-60 (%)
Method
Modality
Resolution
Frames
Xview
xsub
Supervised Methods
STA-Hands Baradel et al.
2017
RGB- Pose
299
20
88.6
82.5
Separable STA (Das
2019
RGB Pose
64
94.6
92.2
ESE-FN (Shu et al. 2022
RGB- Pose
64
96.7
92.4
VPN Das al. 2020
RGB Pose
64
96.2
VPN- Das el al. 2021
RGB - Pose
64
99.1
96.6
3DA Kim et al. 2023
RGB- Pose
224x224
12
97.9
94.3
PoseC3D (Dua et al.
2022
RGB-Post
32
99.6
97.0
DA-Net (Wang al.
2018
RGB
75.3
Zhang et al. (Zhang et al.
2018
RGB
70.6
63.3
Glimpse Clouds (Baradel
2018
RGB
224x221
8
93.2
86.6
DMCL Gaarcaa et al. [2021]
RGB
224x224
8
83.6
Debnath al. (Debnath et
2021
RGB
87.2
FSA-CNN (Jang et al.
2020
RGB
92.2
88.1
Piergiovanui el
PPergiovanni Ryoo 2021
RGB
93.7
ViewCon (Shah et al.
2023b
RGB
224x224
32
98.0
91.4
DVANet [iddiiqui 2024
2024
RGB
98.2
93.4
T- V-T
Reilly & Das
RGB
224x224
16
97.9
94.0
Sepe-Superoised Methods
Li et al. [Li et al. 2018
Flow
6
83.4
80.9
GL- -L--rrnsformer
Kim et al.
2022
Pose
300
83.8
76.3
AimCLR (Cuo et al.
Pose
89.2
83.0
HaLP (Shah et al.
2023a
Pose
64
88.6
82.1
Vyas el al. Vyas al.
2020
RGB
112x112
86.3
82.3
VideoMAE Tong el al.
2022
RGB
128x 128
16
92.7
85.2
ViewCLR Das Ryoo
2023
RGB
128x128
32
94.1
89.7
MV2MAE (Ours)
RGB
128x
95.9
90.0
Table 1: Comparison with state-of-the-art cross-view and cross-subject action recognition benchmarks of NTU-60 dataset. Top: supervised methods using multiple modalities (RGB+Pose), and RGB only, Bottom: self-supervised methods using any modality. We report top-1 accuracy after finetuning as in (Li
et al. 2018 Das & Ryoo 2023 Vyas et al. 2020 Kim et al. 2022 We perform multi-crop testing with 5
clips and 10 crops for each, following (Das Ryoo 2023).
t: using their publicly available implementation.
for information of the target viewpoint, we implicitly use the view information from the visible patches of target viewpoint. We also compare with VideoMAF where we use the same amount of data and match the number of optimization steps, and observe consistent gains compared to this single-view bascline. Morcover, it noteworthy that most the supervised methods (in Table 1 2 6 use a resolution of 224x224 or higher, and despite using a much lower resolution of 128x 128, MV2MAE shows strong performance.
On the ETRI dataset (Table 3), MV2MAE improves the action classification accuracy by +1.4% compared to (Dokkar el al. 2023). ETRI does not include official cross-view benchmark, and thus cross-view generalization is not directly evaluated on this dataset.
4.4 Transfer Learning Results
Transfer learning is an important setting for evaluating the generalization capabilities of pre-trained models. The model is initialized using pre-trained weights, and fine-tuned on smaller datasets. We perform transfer learning experiments on three action recognition datasets: 1) NUCLA, 2) PKU-MMD-II, and 3) ROCOG-v2.
NUCLA Wang et al. 2014) is a multi-view action recognition dataset consisting of 1493 videos spanning 10 action classes. Each activity has been captured from three viewpoints, and we follow the cross-view protocol
8

---

Published in Transactions on Machine Learning Research (03/2026)
NTU-120 (%)
Method
Modality
Resolution
Frames
xset
xsub
Supervised Methods
Hu et al.
(Hu et al.
2018
RGB+-Depth
14.9
36.3
Hu al.
(Hu
2015
RGB+Depth
54.7
50.8
Separable STA
Das et al.
2019
RGB-Pose
64
82.5
83.8
VPN
Das et al. 2020
BCGB+PPose
87.8
86.3
VPN
Das el al. 2021
RGB 1-1B--PP Pose
64
92.5
90.7
3DA
Kim et al. 2023
RGB+ Pose
224x224
91.4
90.5
Pose
Duan al.
2022
RGB+Pose
32
96.4
95.3
PEM Liu & Yuan
2018
Pose
66.9
64.6
2s AGCN Lei et al. 2019
Pase
300
84.9
82.9
MS-G3D Net LLu al.
2020b
Pose
300
88.4
86.9
CTR-GCN (Chen et a.. b021b
Pose
64
90.6
88,9
ProtoGCN Liu et 2025
Pose
92.2
90.9
Hyper-GCN Zhou el al. 2025
Pose
300
92.0
90.9
Two-streams (Simonyan & Zisserman Liu et al. (Liu et al. 22020a
2014
RGB
54.8
58.5
RGB
all
54.8
58.5
13D Carreira & Zisserman
2017
RGB
224x224
250
80.1
77.0
DMCL Garcia et al.
202
RGB
224x224
84.3
ViewCon (Sahh
2023b
RGB
224x221
87.5
85.6
DVANet
Siddiqui et al.
202
RGB
91.6
90.4
V-V RRilly & Das
2024)
RGB
224xX244
16
92.9
91.9
Self- Supercised Methods
GL- Tr-ansssorrmrr Kim et al.
2022
Pose
300
68.7
66.0
AimCLR (Guo et 2022)
Pose
76.7
76.4
HaLP
(Shah et al. 2023a
Pose
64
73.1
72.6
VideoMAF (Tong al. ViewCLR Das & Ryoo
2022
RGB
128x
16
82.4
79.7
2023
RGB
128 x228
32
86.2
84.5
MV2MAE (Ours)
RGB 128 128
16
87.1
85.3
Table 2: Comparison with state-of-the-art on cross-setup and cross-subject action recognition benchmarks of NTU-120 dataset. Top: supervised methods using multiple or single modality, Bottom: self-supervised methods using any modality. We report top-1 accuracy after finctuning as in (Das & Ryoo 2023 Vyas al. 2020 Kim et al. 2022), elc. We perform multi-crop lesting with clips and 10 crops for each, following (Das & Ryoo 2023 using their publicly available implementation.
ETRI (%) xsub
Method
Modality
Resolution
Frames
Supervised Methods
ESE-FN Shu et al. 2022
RGB+Pose
64
95.9
FSA-CNN
Jang el al.
2020
RGB
90.6
COnVIVIT
Dokkar et al.
2023
RGB
95.1
Self Superiiedd Methods
VideoMAE Tong et al. MV2MAE (Ous)
2022
RGB
128x x 128
16
93.4
RGB
128x 128
16
96.5
Table 3: Comparison with state-of-the-att cross-subject action recognition benchmark of ETRI dataset. MV2MAE performs better than prior work, which are all supervised approaches. We perform multi-crop testing with clips and 10 crops for cach. t: using thoir publicly available implementation.
for experiments. PKU-MMD-II [Liu et. al. 2017 is another dataset for 3D action understanding,

---

Published in Transactions on Machine Learning Research (03/2026)
NUCLA (%)
Method
Modality
Resolution
Frames
xview
Supervised Methods
STA
Das et al.
2019)
RGB++Pose
64
92.4
VPN
Das a..
2020
RGB+Pose
93.5
VPN+ + Das et al.
2021
RGB+Pose
64
93.5
DA-Net WWag et al.
2018
RGB
86.5
Glimpse Cloud Baradel et al.
2018
RGB
224x224
90.1
I3D
Carreira & Zisserman 2017
RGB
224x224
250
88.8
ViewCon (Shah et al. 20236
RGB
224x224
32
91.7
DVANet (Siddiqui et al. 2024
RGB
96.5
Hyper-GCN Zhou
2025
Pose
97.6
Self-Superrised Methods
MS'L (Lin et al.
2020
Pose
200
86.8
Li et al. Colorization Yang et a.
Li et al. 2018
Depth Depth
62.5
2021
50
94.0
Vyas et al.
Vvas et al.
2020
RGB
112x112
83.1
ViewCLR Das & Ryuo
2023
RGB
128x128
89.1
MV2MAB (Ours)
RGB
128x 128
16
97.6
Table 4: Transfer learning on NUCLA. Self-supervised methods are pre-trained on NTU-60 dataset.
MV2MAE significantly outperforms other methods showing remarkable transfer capability of our repre- sentations.
PKU-MMD (%)
Method
Modality
Resolution
Frames
xsub
Sapp-Suped Methods
CrosSCLR-B (Zolfagaari et al.
2021
Pose
52.8
CMD
Mao al.
2022
Pose
64
57.0
HaLP
Shah et al.
2023a
Pose
64
57.3
MV2MAE (Ours)
RGB
128 128
16
60.1
Table 5: Transfor lcarning on PKU-MMD. All mothods use NTU-120 datasct for pro-training. MV2MAE surpasses other unsupervised methods, all which use Pose modality.
consisting of 6945 videos from 51 activity classes Following prior work (Shah et al. 2023a). use the phase 2 of the dataset and evaluate our approach on the cross-subject setting. ROCOG-v2 (Reddy et al. 2023) is gesture recognition dataset consisting of 304 ground viewpoint videos from 7 gestures.
As shown in Table our method achieves better performance than prior supervised and unsupervised methods on the NUCLA dataset. MV2MAE improves the action classification accuracy by +8.5% upon the previous RGB-based SOTA SSL approach (Das & Ryoo 2023) and by +1.1% compared to best supervised pre-trained approach (Siddiqui et al. 2024b. On the PKU-MMD-II dataset (Table our method shows an improvement of +2.8% compared to prior work, all of which are based on Pose modality. Finally, we show results on the ROCOG-v2 dataset in Table 0 where we gain an improvement of +2.0%.
These transfer learning results clearly demonstrates that the representations learnt using our approach generalize well. is noteworthy that although the Pose modality shows superior performance in supervisod setting (Table 2 lags behind when used for sule-supe learning in both in-domain fine-tuning (Table and transfer learning (Table and settings.
10

---

Published in Transactions on Machine Learning Research (03/2026)
Method
Modality
Resolution
# Frames
ROCOG (%)
Reddy al. (Reddy et al. 2023)
RGB
256x256
16
87.0
MV2MAE (Ours)
RGB
128 128
16
89.0
Table 6: Transfer learning on ROCOG ground dataset.
4.5 Synthetic Multi-View Data for Pre- Training
Pre-training using real vs synthecic data
Real multi-view videos can be difficult to acquire and can pose privacy concerns. As an alternative, we investigate the use of synthetic multi-view action recognition data. In experiments, we pre-train models using synthetic data from SynADL (Hwang et al. 2021 dataset and fine-tune and evaluate on real data from ETRI (Jang et al.) (020)) dataset. We compare synthetic pre-training (green) with real pre-training (orange). We observe that the amount of syuthetic data used is same (Ix) as the amount of real data, there is performance drop due to the domain difference. However, increasing the amount of synthetic data used for pre-training allows synthetic pre-training to surpass real pre- training, as seen in Figure g These results suggest that synthetic data can serve as a scalable and effective alternative for pre-training.
58
synact
96
55
82
91-
pre-training
Figure 3: Pre-training using synthetic data. Pre- training using more (4x) synthetic data beats pre- training using real data on the same real test set.
4.6 Ablation Study and Analysis
We perform ablation experiments the cross-subject benchmark of the NTU-120 dataset. For ablations regarding viewpoints (Table an
and 11], we use the ETRI datasct as it contains four synchronised viewpoints compared to NTU-120 which has three.
How much emphasis to place on reconstructing moving patches? The motion weights in MV2MAF can be adjusted to modulate the emphasis moving patches, using the temperature parameter. As shown in Figure 2 a lower (emperature places more focus on reconstructing patches with more motion, and increasing the temperature increases the weight given to the background pixels. Figure ! shows the influence of the temperature parameter on accuracy. From the plot, we see that a temperature value of 60 performs best, which is used in all our experiments.
Increasing the weights of background patches by increasing temperature degrades the performance. This is because it is trivial to reconstruci the back- ground patches by copy-paste from nearby frames. The performance degrades significantly to 82.46% if each patch in the vidco weighted equally, since the the number of static pixels are more than those with motion. Note that these experiments are trained for 1200 epochs.
How much emphasis on patches with motion?
85.2
85.0
84.8
Acc@1
84.6
Masking Ratio. We study the impact of mask- ing ratio in Table 7 Note that the optimal masking ratio is lower in our multi view setting than the stan- dard MAE (Tong et al. 2022), which we hypothesize
84.4
with motion weights without motion weights
82.46 %
84.2
20
40
60
80
100
200
Temperature
11
Temperature parameter 7 of motion weights modulates the focus on static vs moving re-
Figure 4:

---

Published in Transactions on Machine Learning Research (03/2026)
is because more information is needed from individ- ual views to effectively infer cross-view geometry.
Model Scaling. To study how the performance scales with models of different capacilies, pre- train using VIT-T, VIT-S, and ViT-B encoders and compare the fine-tuning accuracy on the NTU-120 cross-subject setting in Table [ MV2MAE can ef- fectively pre-train larger models using the same amount data. All model scaling experiments perform pre-training for 800 epochs as opposed to the default 1600 due to computational constraints.
Masking ratio (p)
0.6
0.7
0.8
0.9
Backbone
ViT-T
ViT-S
ViT-B
NTU-120 xsub (%)
83.6
85.3
84.3
83.4
NTU-120 xsub (%) 82.0
83.4
85.1
Table 7: Masking Ratio. MV2MAE performs best with a masking ratio of 0.7. gray
Table 8: Model Capacity. Our approach scales effectively with bigger models. Default. in gray
Default in
Ablation of Cross-view Decoder. The cross-view decoder significantly boosts performance in both xsub and xset settings, with more improvement on the xview benchmark (+2.9%) than on xsub benchmark (+2.2%), showing effectivenoss when evaluating on un- known viewpoints. In the experiment without cross- view decoder, the model is trained using single-viewpoint videos ignoring synchronization, on the same amount of data.
Cross- View Decoder
NTU-120 xsub (%)
NTU-120 xset (%)
x
83.1
84.2
85.3 (+2.2)
87.1 (+22.)
Table 9: Cross- View Decoder leads to substantial improvements on both xview and xset benchmarks.
Visualizing Cross-Attention Maps and Reconstructions. Here, we analyze the cross-view decoder by visualizing the cross-attention maps and the cross-vics view reconstruction quality. The cross-attention maps are visualized in Figure 5 The first and second rows show the input and masked input frames from the target viewpoint, with a masked query token circled in red. The third shows the reconstructed larget view from the cross-view view decoder. The last row shows the cross-attention map corresponding to the query token overlaid on the source view frames. We can see that model is able to find matching regions in the source viewpoint, demonstrating the learnt geometry.
How many source views to use? For the cross- view decoder, we study the effect of number of source viewpoints used in Table 10 For these experiments, all viewpoints used are chosen randomly from available synchronized views. The performance is similar when using one or two source viewpoints. We observe that the fine-tuning accuracy drops we use more source viewpoints for reconstructing the target viewpoint, by making the reconstruction task easier.
# Source Views
1
2
3
ETRI xsub (%)
94.0
93.9
93.1
Table 10: Number of Source Views. Using more source views makes reconstruction task easier and de- grades performance. Default in gray
How different should the views be? natural question that arises when creating such datasets is which viewpoint to capture? Given target viewpoint, we study how far should the source view be. We study this by fixing the target view be Viewl (shown in Figure G and varying the source view to be View2, View3 or View4. The results are reported in Table [1] which shows that the performance drops slightly if the chosen target and source viewpoints are separated by a lot. For all experiments in the paper, we do not fix the target view, and choose source and target views randomly for added diversity.
Songle-Clip Bangle-Crop Inference We compare single-clip, single-crop setting the full multi-crop evaluation with 5 clips and 10 crops from each in the cross-subject setting of NTU-60 and NTU-120 datasets.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Frame 1
Frame
Frame 3
Frame 4
Frame 5
Frame
Frame 7
Frame 8
larget
Input
Target Masked
Source Target View
Recon
Figure 5: Cross- View Decoder Qualitative Analysis. We visualize the reconstructions and cross- allention maps from the cross-view decoder. First row: Target viewpoint input frames, Second row: Masked imput frames from target viewpoint, Third row: Reconstruction of target view from the cross-view decoder, Last row: Cross-attention maps visualized on source view frames. The red circle indicates a query token in the larget viewpoint whose allention maps are visualized in the last row. Green circles shows that the model is able find and leverage matching regions across viewpoints for view reconstruction.
On NTU-60, the performance drops from 90.0% 82.1%, and on NTU-120, the performance drops from 85.3% to 77.3%.
Source View
View2
View3
View4
View1
View2
View3
View4
ETRI xsub (%)
94.7
94.4
94.3
Which views to choose? View2 is the closest and View4 the farthest from the target. Per- formance degrades slightly if the two views are very different.
Table 11:
Figure 6:
Example of synced views from ETRI
dataset.
5 Limitations
While MV2MAF demonstrates strong performance controlled datasets with static backgrounds (e.g.. NTU, ETRI), its efficacy on unconstrained, in-the-wild multi-view videos with complex camera motions and dynamic backgrounds (e.g, EgoExo4D (Grauman el al. 2024), LEMMA (Jia el al. 2020)) remains to be fully explored. To address this, future work could investigate pre-training on larger-scale datasets featuring more realistic motions, or explore robustness strategics such as artificially introducing background motion during fine-tuning.
6 Broader Impact
While multi-view action recognition advances fields like elderly care, its potential application in surveillance raises privacy concerns regarding the capture of detailed geometric information. To mitigate these risks, we investigate synthetic data as a scalable, synthetic data allows models to surpass those trained real-world footage, further research into domain
alternative for pre-training. Although scaling
13

---

Published in Transactions on Machine Learning Research (03/2026)
adaptation is essential to better bridge the synthetic-to-real gap, offering a viable path to minimize reliance sensitive data without compromising robustness.
7
Conclusion
We propose a self-supervised learning approach for harnessing the power of multi-viow videos within the masked autoencoder framework. Our method integrates cross-view reconstruction task, leveraging a dedi- cated decoder equipped with rrossaattentio mechanism to instill geometry information into the model. The introduction of motion-focused reconstruction loss further enhances temporal modeling. We empirically show that our SSL method enables learning robust generalizable multi-view features contributing to better performance when used for full fine-tuning and transfer learning.
References
Unaiza Ahsan, Rishi Madhok, and Irfan Essa. Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 189. IEEE, 2019.
Amurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CT /VF international conference on computer vision, pp. 6836-6846, 2021.
Wele Gedara Chaminda Bandara, Naman Patel, Ali Gholami, Mehdi Nikkhah, Motilal Agrawal, and Vishal M Patel. Adamae: Adaptive masking for efficient spatiotemporal learning with masked autoen- coders. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
14507 14517, 2023.
Fabien Baradel, Christian Wolf, and Julien Mille. Human action recognition: Pose-based allention draws focus to hands. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 604-613, 2017.
Fabien Baradel, Christian Wolf, Julien Mille, and Graham W. Taylor. Glimpse clouds: Human activity recognition from unstructured feature points. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Joao Carreira. and Andrew Zisserman. Quo vadis, action recognition? a model and the kinctics dataset. In 2017 IEEE Conference Compuler Vision and Pattern Recognition (CVPR), pp. 4724 4733. IEEE, 2017.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive lcarning. arXie preprint arXiw:2003.0420 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9640-99009 (0096949,46 2021a.
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel-wise topology refinement graph convolution for skeleton-based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13359 13368, 2021b.
Kellie Corona, Katic Osterdahl, Roderic Collins, and Anthony Hoogs. Meva: A large-scale multiview, multimodal video dataset for activity detection. In Proceedings of the TEEE/CVF Winler Conference on Applications of Computer Vision (WACV), pp. 1060- 10081068, January 2021.
Srijan Das and Michael Ryoo. Viewelr: Learning sclf-supervisod vidco representation for unseen viewpoints. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5573 5583, 2023.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome: Real-world activities of daily living. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), October 2019.
Srijan Das, Saurav Sharma, Rui Dai, Francois Bremond, and Monique Thonnat. VPN: Learning video-pose embedding for activities of daily living. In European Conference on Computer Vision, pp. 72-90. Springer, 2020.
Srijan Das, Rui Dai, Di Yang, and Francois Bremond. VPN++: Rethinking video-pose embeddings for
understanding activities of daily living. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):9703-9711 2021.
Bappaditya Debnath, Mary O'brien, Swagat Kumar, and Ardhendu Behera. Attentional learn-able pooling for human activity recognition. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13049 13055. IEEE, 2021.
Rachid Reda Dokkar, Falen Chaieb, Hassen Drira, and Arezki Aberkane. Convivil a deep neural net- work combining convolutions and factorized self-attention for human activity recognition. arXiv preprint arXiv:2310. 14411441 2023.
Alexcy Dosovitskiy, Lueas Beyer, Alexander Kolcsnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa, Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2969-2978, 2022.
Christoph Reichtenhofer, Haoqi Fan, Bo Xiong, Ross B. Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. 2021 IEEE/CVF Conference on Computer Vi- sion and Pallern Recognilion (CVPR), pp. 3298 3308, 2021. URL https::/tpppi www.ntccscoolar.rco.ommm org/
ID
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Aduances in neural information processing systems, 35:35946-35958, 2022.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 3636 3645-6655544 2017.
Nuno Cruz Garcia, Sarah Adel Bargal, Vitaly Ablavsky, Pietro Morerio, Vittorio Murino, and Stan Sclaroff. Distillation multiple choice learning for multimodal action recognition. In Proceedings of the IEEE/CV /CVF Winter Conference on Applications of Computer Vision, pp. 2755-27655 2021.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, He- una Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, al. The' something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exold: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEFE/CV Conference on Computer Vision and Pattern Recognition, pp. 19383 19400, 2024.
Tianyu Guo, Hong Liu, Zhan Chen, Mengyuan Liu, Tao Wang, and Runwei Ding. Contrastive lcarning from extremely augmented skeleton sequences for self-supervised action recognition. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp. 762-770, 2022.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Yuxin Guo, Siyang Sun, Shuailei Ma, Kecheng Zheng, Xiaoyi Bao, Shijie Ma, Wei Zou, and Yun Zheng. CrossMAF: Cross-modality masked autoencoders for region-aware audio-visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26721 26731, 2024.
Tengda Han, Weidi Xie, and Andrew ZZisseraan. Self-supervised co-training for video representation learning. Advances in Neural Information Processing Systems, 33:5679 5690, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked auloencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Jian-Fang Hu, Wei-Shi Zheng, Jianhuang Lai, and Jianguo Zhang. Jointly learning heterogeneous features for RGB-D activity recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5344-5352, 2015.
Jian-Fang Hu, Wei-Shi Zheng, Lianyang Ma, Gang Wang, Jianhuang Lai, and Jianguo Zhang, Early action prodiction by soft regression. IEEE transactions pattern analysis and machine intelligence, 41(11): 2568 2583, 2018.
Bingkun Huang, Zhiyu Zhao, Guozhen Zhang, Yu Qiao, and Limin Wang. MGMAE: Motion guided masking for video masked autoencoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13493 13504, 2023.
Hochul Hwang, Cheongjae Jang, Geonwoo Park, Junghyun Cho, and Ig-Jae Kim. Eldersim: A synthetic data generation platform for human action recognition in eldercare applications. IEEE Access, 2021.
Leyla Isik, Andrea Tacchelti, and Tomaso Poggio. A fast, invariant representation for human action in the visual system. Journal of neurophysiology, 119(2):631-6))) 640, 2018.
Jinhyeok Jang, Dohyung Kim, Cheonshu Park, Minsu Jang, Jaeyeon Lee, and Jaehong Kim. ETRI- activity3D: A large-scale rgb-d dataset for robots to recognize daily activities of the elderly. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2020.
IEEE,
Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-Chun Zhu. LEMMA: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivitics. In European Conference on Computer Vision, pp. 767-786. Springer, 2020.
Longlong Jing, Xiaodong Yang, Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning via video rotation prediction. arXiv preprint orX1::1811.11987, 2018.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. ar preprint Xrx:::1705 06950, 2017.
Boeun Kim, Hyung Jin Chang, Jungho Kim, and Jin Young Choi. Global-local motion transformer for unsupervised ddeleton-based action learning. In Europeon conference on computer vision, pp. 209 222. Springer, 2022.
Sangwon Kim, Dasom Ahn, and Byoung Chul Ko. Cross-modal learning with 3d deformable attention for action recognition. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10265-10275, 2023.
Shi Lei, Zhang Yifan, Cheng Jian, and Lu Hanqing. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In CVPR, 2019.
Junnan Li, Yongkang Wong. Qi Zhao, and Mohan S Kankanhalli. Unsupervised learning of view-invariant
action representations. Advances in neural information processing systems, 31, 2018.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Lilang Lin, Sijie Song, Wenhan Yang, and Jiaying Liu. Ms21: Multi-task self-supervised learning for skele- ton based action recognition. Tn Proceedings of the 28th ACM International Conference on Multime- dia, MM '20, pp. 2490 2498, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379885. doi: 10.1145 10.1145/3394171.11 1339411113413544 URL https:t//oi..orre
1145/3394171 3443333 3413548
Chunhui Liu, Yueyu Hu, Yanghao Li, Sijic Song, and Jiaying Liu. PKU-MMD: A large scale benchmark for continuous multi-modal human action understanding. arXiv preprint Xiv:: 773. 00080.0775 75, 2017.
Hongda Liu, Yunfan Liu, Min Ren, Hao Wang, Yunlong Wang, and Zhenan Sun. Revealing key details to see differences: A novel prototypical perspective for skeleton-based action recognition. In Proceedings of the Computer Vision and Pottern Recognition Conference, pp. 19948-292557 2025.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. NTU RGB+D 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(10))2684 2701, 2020a.
Mengyuan Liu and Junsong Yuan. Recognizing human actions as the evolution of pose estimation maps. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. Disentangling and unifying graph convolutions for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference Compater Vision and Pallern Recognilion, pp. 132-1 2020b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Xie preprint orXis:1711.0510: 2017.
Yunyao Mao, Wengang Zhou, Zhenbo Lu, Jiajun Deng, and Houqiang Li. Cmd: Self-supervised 3d action representation learning with cross-modal mutual distillation. In European Conference on Compuler Vision (ECCV), 2022.
Vasu Parameswaran and Rama Chellappa. View invariance for human action recognition. International Journal of Computer Vision, 66(1):83-101, 2006.
AJ Piergiovanni and Michael Ryoo. Recognizing actions in videos from unseen viewpoints. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4124 4132, June 2021.
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuaa Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In 2021 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 6960-6970. TEFF, 2021.
Hossein Rahmani and Ajmal Mian. Learning a non-linear knowledge transfer model for cross-view action recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2458- 2466, 2015.
Cen Rao, Alper Yilmaz, and Mubarak Shah. View-invariant representation and recognition of actions. International Journal of Computer Vision, 50:203-226, 2002.
Arun V Reddy, Ketul Shah, William Paul, Rohita Mocharla, Judy Hoffman, Kapil D Katyal, Dinesh Manocha, Celso M de Melo, and Rama Chellappa. Synthetic-to-real domain adaptation for action recogui- tion: A dataset and baseline performances. In IEEE International Conference on Robotics and Automation (ICRA). IEFE, 2023.
Dominick Reilly and Srijan Das. Just add?! pose induced video transformers for understanding activities of daily living. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18340 18350, 2024.
H. Saito, N. Inamoto, and S. Iwase. Sports scene analysis and visualization from multiple-vicw vidco. In 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No. No.04 TH8763),, volume pp. 1395 1395-1398 Vol.2, 2004. doi: /1110///MM..200..1333
17

---

Published in Transactions on Machine Learning Research (03/2026)
Madeline C Schiappa, Yogesh S Rawat, and Mubarak Shah. Self-supervised learning for videos: A survey. ACM Compuling Surveys, 55(13s):1-371 2023.
Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, and Pieter Abbeel. Multi-view masked world models for visual robotic manipulation. arXiv preprint arxi2:2302 02408, 2023a.
Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, and Pieter Abbeel. Multi-view masked world models for visual robotic manipulation. In International Conference on Machine Learning, pp. 30613-30632. PMLR, 2023b.
Anshul Shah, Shlok Mishra, Ankan Bansal, Jun-Cheng Chen, Rama Chellappa, and Abhinav Shrivastava. Pose and joint-aware action recognition. In Proceedings of the IEEE/CVF Winler Conference on Appli- cations of Computer Vision, pp. 3850-330 3860, 2022.
Anshul Shah, Aniket Roy, Ketul Shah, Shlok Mishra, David Jacobs, Anoop Cherian, and Rama Chellappa. HaLP: Hallucinating latent positives for skeleton-based self-supervised learning of actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18846-18856, 2023a.
Ketul Shah, Anshul Shah, Chun Pong Lau, Celso M de Mclo, and Rama Chellappa. Multi-view action recognition using contrastive learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3381 3391, 2023b.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. NTU RGB+D: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on compuler vision and pattern recognition, pp. 1010-1019, 2016.
Jinghuan Shang, Srijan Das, and Michael Ryoo. Learning viewpoint-agnostic visual representations by recovering tokens in 3d space. Advances in Neural Information Processing Systems, 55:31031-310444 2022.
Xiangbo Shu, Jiawen Yang, Rui Yan, and Yan Song. pppaussion-qqueez--ecccaaatttttd fusion network for elderly activity recognition. IEEE Transactions on Circuits and Systems for Video Technology, (28))5521155292
2022.
Nyle Siddiqui, Praveen Tirupattur, and Mubarak Shah. Dvanet: Disentangling view and action features for multi-view action recognition, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4873 4881, 2024.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Aduances in neural information processing systems, pp. 568-576, 2014.
Xinyu Sun, Peíhao Chen, Liangwei Chen, Changhao Li, Thomas Li, Mingkui Tan, and Chuang Gan. Masked motion encoding for self-supervised video representation learning. In The IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), 2023.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 10078-10093, 2022.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Shruti Vyas, Yogesh S Rawat, and Mubarak Shah. Multi-viow action recognition using cross-view vidco prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pp. 427 444. Springer, 2020.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Dongang Wang, Wanli Ouyang, Wen Li, and Dong Xu. Dividing and aggregating network for multi-view action recognition. In Proceedings of the Europeon Conference on Compuler Vision (ECCV), pp. 451-467, 2018.
Jiang Wang, Xiaohan Nie, Yin Xia, Ying Wu, and Song-Chun Zhu. Cross-view action modeling, learning and recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2649-2656, 2656, 2014.
Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, and Wei Liu. Self-supervised spatio- temporal representation learning for videos by predicting motion and appearance statistics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4006-4015, 2019.
Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu. Self-supervised video representation learning by pace predic- tion. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pp. 504-521. Springer, 2020.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang. Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 14549-14560, 2023a.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengehen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. BEVT: BERT pretraining of vidco transformers. In Proceedings of the IEEE/CVE conference on computer vision and pallern recognilion, pp. 14733 14743, 2022.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengehen Liu, Lu Yuan, and Yu- Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In Proceedings of the IEFE/CT Conference on Computer Vision and Pattern Recognition, pp. 6312 6322, 2023b.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wi,, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVE Conference Computer Vision and Pallern Recognilion, pp. 14668 14678, 2022.
Lu Xia, Chia-Chih Chen, and Jake K Aggarwal. View invariant human action recognition using histograms of 3d joints. In 2012 IEEE compuler sociely conference on compuler vision and pallern recognition workshops, pp. 20-27. 27. IEEE, 2012.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10334-10343, 2019.
Siyuan Yang, Jun Liu, Shijian Lu, Meng Hwa. Er, and Alex C. Kot. Skeleton cloud colorization for unsu- pervised 3d action representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13423- -14433, October 2021.
Senthil Yogamani, Ciarán Hughes, Jonathan Horgan, Ganesh Sistu, Padraig Varley, Derek O'Dea, Michal Uricár, Stefan Milz, Martin Simon, Karl Amende, et al. Woodscape: A multi-task, multi-camera fisheye dataset for autonomous driving. In Proceedings of the IFFE/CVF International Conference on Compuler Vision, pp. 9308-9318, 2019.
Pengfei Zhang, Jianru Xue, Cuiling Lan, Wenjun Zeng, Zhanning Gao, and Nanning Zheng. Adding at- tentivencss to the neurons recurrent neural networks. In proceedings of the European conference on computer vision (ECCV), pp. 135 151, 2018.
Jingjing Zheng, Zhuolin Jiang, Jonathon Phillips, and Rama Chellappa. Cross-viow action recognition via a transferable dictionary pair. In brvve volume 1, pp. 7, 2012.
Youwci Zhou, Tianyang Xu, Cong Wu, Xiaojun Wu, and Josef Kittler. Adaptive hyper-graph convolution network for skeleton-based human action recognition with virtual connections. In Proceedings of the IEFE/CVP International Conference on Computer Vision, pp. 12648 -2658, 2025.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. Crossclr: Cross-modal contrastive learning for multi-modal video representations In Proceedings of the TEEE/CVE International Conference on Computer Vision, pp. 1450 1459, 2021.
A PyTorch code for motion weights
Given the input video frames, patch-size and the temperature parameter, we provide PyTorch code below for motion weights based on frame difference used in the paper.
Algorithm 1: PyTorch code for motion weights.
frames input
shape [,,
# size: si_t,m,, p_height, p_width) temperature paraneter
fdiff franes[:, ir
framen[;,
fdiff torch.cat (ffiffff ;;:,; :l, fdiffl, dim-2)
Idiff rearrange (Idiff,
po) (h p1) (N p2) -> (t (p0 p1 p2 e)', po-patch_eize(01 pi-patch_eizeeAi [1],
p2-patch_ [2])
fdiff = torch, afs
forch.linalg... vector www.teeeeernneeeieeeeeeee norn(fdiff, din-2, keepdin-True)
notion_weights torch. an. functional. wow.eiionn...otmaxxleoooeoo (fdiff/t, dim-1)
B Architecture Details
The detailed asymmetric architecture of the encoder and decoders shown in Table 12 Table 13 and Table 14 We have two decoders in our architecture: 1) self-view decoder and 2) cross-view decoder. The self-view decoder only uses self-attention reconstruct same view whereas the cross-view decoder uses cross- attention in addition self-attention for reconstructing target viewpoint while also using source viewpoint. These decoders discarded during fine-tuning. We use 16 frame input and choose ViT-S/16 our default encoder. We adopt the joint space-time attention (Araa et al. 2021 for the encoder.
Stage
Vision Transformer (Small)
Output Sizes
data
stride 4x1x1
3x 3x16x 128x 128
2x16x 16, 384
cube
384x8x64
stride 2x 1x1661
random mask
mask
184x8x'64x(1-))
mask ratio p
MHA(384) MLP((136)
encoder
x12
384x8x (1-P)
MLP(192)
projector
192x8x61
concat learnable tokens
Table 12: Encoder of MV2MAE.
The encoder processes 16-frame input clips from source and target
views, and the encoded representations of the visible tokens are combined with the learnable mask tokens, before passing through the decoder.
C Additional Implementation Details
The pre-training and fine-tuning hyper-parameter settings for NTU-60, NTU-120 and ETRI datasets are given in Table 15 and Table 16
Single-View vs Multi-View Inference
At test time, multiple viewpoints of an activity are available in the cross-subject setting. However, evaluation in prior work is carried out using single-view at time, following the original benchmark (Shahroudy et. al.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Stage
Transformer
Output Sizes
Stage
Transformer
Output Sizes
MHCA(192))
(HA(
cross-view decoder
MHA((19)
X4
192x8x64
self-view decoder
(76))
x4
192x8x64
MLP((68)
projector
(13()
1536x8x64
projector reshape
MLP((153)
1536x8x64
reshape
from 1536 to 3x2x16x 3x16x 128x
from 1536 to 3x2x16x16 3x16x128x125
Table 13: Self-view decoder MV2MAE. It takes the source and target view tokens and recon- structs both the views independently.
Cross-view decoder of MV2MAE. The cross-view decoder uses the visible Lokens from the source view to reconstruct the missing patches in the target view.
Table 14:
coulig
NTU60
NTU120
ETRI
optimizer
AdammW
base learning
1e-3
weight decay
config
NTU600
NTUU20
ETRI
optimizer
8, 0.9.. 0.999
optimizer
momentum
learning rate
batch
1024
weight. deray
learning
decay
optimizer
91, 11,9,,0.9.00.
schedule
batch size
1024
warmuu epocis
5
10
learning schedule
cosine docay
training epochs
35
120
120
warmup epochs
320
repeated
6
total epochs
3200
1600
1600
augmentation
augmentation
MultiScaleCrop
augmentatiin
yes
RandAug
0.5)
Table 15: Pre-training setting.
label smoothing
0.1
drop path
0.1
layer wise Ir deray
0.9
Table 16: End-to-end fine-tuning setting.
2016). Though in most practical scenarios, it would be natural to combine the predictions from available synchronized viewpoints for a given activity. We show this comparison of single-view and multi-view inference in Table 17 For multi-view inference, the predictions are combined using late fusion strategy.
Table 17: SV vs MV inference. We perform late fusion for multi-view inference.
Cross-Subject (%) NTU-60
Method
NTU-120
Single-Vier View Inference
90.0
85.3
Multi-View Inference
91.9
87.9
E
Parameter Count Comparison
21

---

Published in Transactions on Machine Learning Research (03/2026)
Method
Modality
Backbone
Params (M)
MV2MAE (Ours)
RGB
ViT-S/16 (encoder only)
22.1
VideoMAF.
RGB
VIT-S/16
22.1
VideoMAF.
RGB
ViT-B/16
86.6
VideoMAE
RGB
VIT-L/16
304.4
Vyas et. al.
RGB
3D Conv + Conv-LSTM
72
ViewCLR
RGB
S3D (encoder only)
AG-AGCN
Pose
2s-AGCN
6.9
MS-G3D
Pose
MS-G3D
2.7-3.2
CTR-GCN
Pose
CTR-GCN
1.4
AimCLR
Pose
AimCLR
0.85
Table 18: Model parameter counts for MV2MAE and baselines reported in our main comparisons. For MV2MAE, we report encoder-only parameters since docoders are discarded at fine-tuning. For VidcoMAE, parameter counts depend on the backbone size used.
22


Reviews:
[{"review_id": "rEY9iZT0ww", "reviewer": "Reviewer_PaoQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes MV2MAE, a masked-autoencoder-style pretraining approach fpr synchronized multi-view videos. The core idea is to augment standard masked video modeling with a a dedicated cross-view decoder uses cross-attention from a masked target view to visible tokens from a source view, encouraging representations that capture geometric/viewpoint relationships. In addition, the paper introduces a simple motion-weighted reconstruction loss to reduce the dominance of easily reconstructed static/background regions. The overall pretraining objective combines self-view reconstruction on both views plus cross-view reconstruction in both directions.\nThe authors evaluate on standard multi-view action recognition benchmarks (NTU-60/120, ETRI) and transfer settings (e.g., NUCLA/PKU-MMD/ROCOG). They report strong performance among SSL methods; e.g., on NTU-60 they report **95.9% (xview)** and **90.0% (xsub)** with RGB at 128×128 and 16 frames. Ablations show that the cross-view decoder is a meaningful contributor. The paper also includes a synthetic-data study suggesting that sufficiently scaling synthetic multi-view data can match or surpass real-data pretraining, but still requiring much more sinthetic samples.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main algorithmic claims are supported by (i) a clear method description, (ii) multiple benchmark evaluations, and (iii) targeted ablations.\n\n-  Cross-view reconstruction is clearly defined and motivated (learning viewpoint-robust representations by requiring cross-view extrapolation), and the architectural mechanism is described concretely.\n    \n-  The full pretraining loss is explicit and includes both self-view and cross-view terms, making it easy to understand what is optimized.\n    \n- Ablations support key components: enabling the cross-view decoder yields consistent improvements across NTU-120 splits.The motion-weighting temperature study indicates an optimum around \\tau=60 and shows a notable drop when weights become uniform.\n    \n- Experimental protocol details are present (e.g., 16 RGB frames with stride 4; 128×128 input; ViT-S/16; 1600 epochs), and evaluation uses 5 temporal clips x 10 crops.\n\nThat said, a few aspects weaken how clear the evidence is for some of the paper’s comparative and \"practicality\" claims:\n\n- Some comparisons mix backbones, resolutions, and evaluation settings; the paper notes supervised methods often use ≥224 resolution while MV2MAE uses 128, but the main tables still leave room for ambiguity about fairness (e.g., crops/clips, compute, model size).\n    \n- The paper states MV2MAE is faster/more memory efficient than ViewCLR because ViewCLR uses MoCo-style components, but it would be more convincing with a simple quantitative comparison (GPU memory, training throughput).\n\nOverall, the evidence supports the core claims, but I think the paper would benefit from clarifications and a couple of additional baselines to make the comparisons more rigorous.", "audience_yn": "Yes", "audience_explanation": "Multi-view video is common in real deployments (multi-camera rooms, assisted living, robotics, sports), and learning viewpoint-robust representations without requiring pose/depth is a relevant problem. The approach is appealing because it is simple and modular: it stays close to the MAE paradigm while injecting cross-view structure via a cross-attention decoder, and discards decoders at fine-tuning time (so downstream cost remains similar to a standard encoder). The inclusion of synthetic multi-view pretraining analysis also makes the work interesting for audiences thinking about privacy, scalability, and data sourcing.", "requested_changes": "### Major / high-priority\n\n1.  **Add/clarify a “single-view MAE” baseline (compute-matched)**\n    -   You already include an ablation disabling the cross-view decoder (Table 9), which is helpful, but it is still a multi-view training regime (two synchronized views available).\n    -   Please add an explicit baseline where you ignore synchronization and pretrain a standard VideoMAE/MAE on the same underlying videos (treat each view independently as its own sample), matched for total compute (e.g., same number of tokens/updates). This would directly answer: Do we need cross-view, or do gains come mostly from \"more data/augmentation\" by having multiple views?\n\n        \n2.  **Make evaluation protocol explicit in each main results table (clips/crops)**\n    \n    -   The paper states test-time evaluation uses 5 temporal clips and 10 crops, which is a substantial inference-time budget.\n        \n    -   Please add a column/footnote in the main result tables specifying:\n        -   number of temporal clips\n        -   number of spatial crops  \n        -   whether results are single-crop vs multi-crop\n            \n    -   If prior work uses different evaluation budgets, consider adding a “single-clip, single-crop” evaluation for MV2MAE (and maybe ViewCLR if feasible) to contextualize inference cost.\n        \n3.  **Clarify backbone choice and comparison fairness (ViT-S, params/FLOPs)**\n    \n    -   The default encoder is ViT-S/16. Please clarify whether the main SSL baselines you compare against use the same backbone capacity, or if not, include parameter count and/or FLOPs for each method (or at least for the key SSL baselines).\n    -   You mention parameter counts for one comparison (e.g., Vyas et al. uses more parameters); extending this systematically would strengthen the SOTA claim.\n        \n4.  **Explain and/or ablate the 128×128 resolution choice**\n    \n    -   You state you downsample to 128×128 following ViewCLR. This is reasonable, but many ViT/video works commonly use ≥224.\n    -   Please either:\n        -   justify the choice in terms of compute/memory constraints (especially since pretraining uses two synchronized views), and/or\n        -   provide a small ablation showing how performance scales with resolution (e.g., 128 vs 160/224) for at least one dataset.\n            \n5.  **Clarify viewpoint geometry and train/test camera relationship (esp. NTU)**\n    \n    -   The paper states NTU cross-view trains on cameras 2&3 and tests on camera 1, but it would help readers to know how those cameras are positioned.\n        \n    -   In the NTU dataset documentation, the three cameras correspond to horizontal angles −45°, 0°, +45°, and “camera 1” corresponds to the 45° views while cameras 2/3 capture front/side views. Adding a one-sentence clarification/diagram would prevent confusion (e.g., whether the test camera is \"far\" or \"intermediate\" relative to training cameras).\n        \n    -   You already do a nice viewpoint-distance analysis on ETRI (View2 closest vs View4 farthest) with only slight degradation when views are very different. A similar clarification for NTU would align with this analysis.\n        \n6.  **ETRI: explain “cross-subject only” and consider adding a cross-view protocol**\n    \n    -   The paper says ETRI is evaluated using the cross-subject benchmark from the dataset paper, which likely explains why there is no cross-view result.\n        \n    -   Still, since ETRI has 8 synchronized viewpoints, it would be valuable to either:\n        -   define a simple cross-view split (train on a subset of cameras, test on held-out cameras), or\n        -   explicitly discuss why that is not feasible/standard, and how your method would be expected to behave.\n            \n7.  **Deepen the analysis of masking ratio vs number of views (“information leakage”)**\n    \n    -   You find the optimal masking ratio is lower (0.7) than in standard VideoMAE-style settings (around ~0.9) and hypothesize this is because more info is needed per view to infer geometry.\n        \n    -   You also show that using more source views can make reconstruction too easy and hurts downstream accuracy.\n        \n    -   I suggest adding one compact experiment/plot: accuracy as a function of (number of source views, masking ratio), to show the tradeoff between task difficulty and representation quality. This would make the “view leakage” story much more concrete.\n        \n\n### Minor / presentation and clarity\n\n1.  In the compiled PDF, Figures 3 and 4 appear rasterized/blurry. Please export plots as vector (PDF) or increase resolution so the paper is readable when zoomed.\n        \n2. In the viewpoint-distance discussion, the text references “Table 7” while the view-distance result is says Figure 7; please correct this.\n        \n3. Since you argue ViewCLR is heavier due to MoCo queues and double networks, a small table with peak GPU memory and iterations/sec (or GPU-days) would substantiate this nicely.", "broader_impact_concerns": "- Multi-view action recognition methods can be applied to surveillance or monitoring scenarios. Since the work is explicitly about leveraging synchronized multi-camera video, it can amplify capabilities in settings where privacy concerns are significant.\n   \nI suggest adding a short discussion on: (i) privacy considerations for collecting synchronized multi-view data, and (ii) whether synthetic multi-view generation (simulation, rendering, etc.) could be a safer alternative and what tradeoffs remain.", "additional_comments": "You might also briefly mention adjacent multi-view masked modeling work outside action recognition (e.g., multi-view masked autoencoders/world models used in robotics) to contextualize the broader relevance [Seo et. al.](https://proceedings.mlr.press/v202/seo23a/seo23a.pdf).", "extra_scores": {}, "extra_text": {}}, {"review_id": "4yz9GqdmeC", "reviewer": "Reviewer_sUVt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces MV2MAE, a self-supervised pre-training strategy designed for synchronized multi-view videos. The method builds upon the MAE paradigm and makes two primary contributions: 1) a cross-view reconstruction task where a cross-attention decoder predicts a target viewpoint’s masked patches from visible source-view tokens to force the encoder to learn view-invariant features; 2) a motion-weighted loss that emphasizes reconstruction of dynamic (high motion) regions to reduce trivial background reconstruction and improve temporal understanding. Their method achieves SOTA  performance among other self-supervised approaches on NTU60, NTU120, and ETRI, and also demonstrates strong transfer learning gains on smaller datasets such as NUCLA, PKU-MMD, ROCOG. As multi-view videos can be difficult to obtain, the authors perform a preliminary study on synthetic multi-view data as a scalable alternative.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The results support the claims made in the submission. On NTU60, MV2MAE surpasses ViewCLR (+1.8 points on xview, +0.3 on xsub), and on NTU120 (+0.9 on view, +1.2 on sub). Accuracy on the ETRI dataset surpasses ConViViT by +1.4. The transferability of the method is validated on NUCLA, PKU, and ROCOG, where MV2MAE outperforms representative baselines. The effect of each contribution is validated through ablation studies: removing the cross-view decoder reduces NTU120 accuracy by 2-3%, equal weighting instead of motion weighting reduces accuracy by ~2.5%. Additional analysis (masking ratio, temperature sensitivity, source-view count) validate the design choices of their method.", "audience_yn": "Yes", "audience_explanation": "This work addresses the task of self-supervised learning in videos in the case where multiple time-synchronized views are available, and is likely to interest readers working on tasks such as multi-view representation learning and cross-view knowledge transfer. Given the requirement of multiple synchronized views, it can also be of interest to readers working on video surveillance with security cameras, as multiple cameras often monitor the same scene.", "requested_changes": "The reviewer would like to see how MV2MAE performs on videos with more complex and realistic motions, or for the authors to provide a limitations section regarding this. The datasets explored in this paper are in very tightly controlled environments: the background scenes are static, the actors are always centered in the frames, and the motions that define each action is emphasized. This is not the case for real-world multi-view data (datasets such as LEMMA, EgoExo4D, Toyota Smarthome), which may limit the success of the proposed method.\n\nFrom Figure 4, MV2MAE seems highly sensitive to the motion-weighted loss as without it the performance drops to 82%. The current method used to compute motion is frame differencing, which suggests that the method likely fails if videos contain motion unrelated to the action being performed (such as outdoor scenes, or homes with windows or a TV). Can the authors examine how their method performs on videos containing background motion?\n\nCan the authors present the action classes where MV2MAE improves performance? This can make the paper more convincing if the actions being improved are highly dependent on understanding motions", "broader_impact_concerns": "N/A", "additional_comments": "Clarifying question: in Figure 4, why does the result at $\\tau=60$ not match the result in Table 2 for xsub (85.3)? It seems like this ablation might have been trained for less epochs (similar to the Model Scaling analysis), which is fine but isn’t mentioned", "extra_scores": {}, "extra_text": {}}, {"review_id": "OTN0wdyWQn", "reviewer": "Reviewer_eiaw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a self-supervised method for viewpoint-agnostic video representation learning, extending VideoMAE’s masked autoencoding framework. The approach adds (i) a cross-view decoder that reconstructs pixels in one camera view from features derived from a time-synchronized multi-view input, and (ii) a motion-aware, pixel-wise loss that weights reconstruction by estimated motion across frames. Ablations demonstrate that both cross-view reconstruction and motion-aware weighting contribute significantly to performance. The method is validated on three datasets and further transfers to two additional datasets, showing consistent, promising gains for downstream adoption.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in this paper are well supported with experimental validation on sufficient number of datasets.", "audience_yn": "Yes", "audience_explanation": "The task of learning viewpoint-agnostic video representations is challenging. Many existing methods rely on pretraining with large-scale Internet videos, assuming ample viewpoint diversity; however, this assumption often fails in indoor settings with limited camera perspectives. Self-supervised approaches that exploit synchronized multi-view data provide a principled alternative: they learn viewpoint-agnostic features directly and mitigate a key limitation of current neural networks, whose operations are not inherently viewpoint-invariant.", "requested_changes": "1. The paper does not discuss several recent works. For example, CrossMAE (Guo et al., CVPR 2024) is a self-supervised method whose decoder uses cross-attention resembling the cross-view decoder proposed here. RGB viewpoint agnostic method - 3D TRL (Shang et al., NeurIPS 2022).\n\n2. Relevance of Table 3 / Section 4.5 - The SynADL model is pre-trained on ETRI and evaluated on ETRI, i.e., same-domain. A more informative setup would select common actions from other datasets (e.g., NTU RGB+D, Toyota Smarthome (Das et al., ICCV 2019)) and perform action recognition there. The baseline should be a model trained only on single-view data from the target dataset. This would better demonstrate the value of the method and of multi-view data, even for synthetic data.\n\n3. VideoMAE comparisons are quite important in this paper. Only a single result is presented and not in a table. Please include a full table reporting results across all datasets.\n\n4. Indicate that decoders shown in the same color share weights to avoid reader confusion in Figure 1.\n\n5. Another baseline could have been a VideoMAE trained on Kinetics-400. For fairness, randomly sub-sample Kinetics to match the sample count of NTU RGB+D. This directly tests whether multi-view data is necessary, addressing the counterargument that web-scale data already captures diverse viewpoints; if not, it strengthens the case for MV2MAE.", "broader_impact_concerns": "There is no broader impact section in this paper. I think the study could be leveraged by the vision community to leverage multiview data, perhaps could be explored for ego-exo learning as well. I don't see any ethical concerns in this paper, so having no broader impact section should work fine.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)