Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space
Authors: Jinghuan Shang
Srijan Das
Michael S Ryoo
Venue: NeurIPS 2022 Accept (2022)
Keywords: Representation Learning
Viewpoint-Agnostic
Transformer

Abstract: Humans are remarkably flexible in understanding viewpoint changes due to visual cortex supporting the perception of 3D structure. In contrast, most of the computer vision models that learn visual representation from a pool of 2D images often fail to generalize over novel camera viewpoints. Recently, the vision architectures have shifted towards convolution-free architectures, visual Transformers, which operate on tokens derived from image patches. However, these Transformers do not perform explicit operations to learn viewpoint-agnostic representation for visual understanding. To this end, we propose a 3D Token Representation Layer (3DTRL) that estimates the 3D positional information of the visual tokens and leverages it for learning viewpoint-agnostic representations. The key elements of 3DTRL include a pseudo-depth estimator and a learned camera matrix to impose geometric transformations on the tokens, trained in an unsupervised fashion. These enable 3DTRL to recover the 3D positional information of the tokens from 2D patches. In practice, 3DTRL is easily plugged-in into a Transformer. Our experiments demonstrate the effectiveness of 3DTRL in many vision tasks including image classification, multi-view video alignment, and action recognition. The models with 3DTRL outperform their backbone Transformers in all the tasks with minimal added computation. Our code is available at https://github.com/elicassion/3DTRL.

Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space
Jinghuan Shang'
Srijan Das Michael S. Ryoo
Department of Computer Science
Stony Brook University, UUniversity of North Carolina at Charlotte [jishang, mryoo>&cs www.ssrosooooooooo.. edu, "sdas24@uncc. edu
Abstract
Humans are remarkably flexible in understanding viewpoint changes due to visual cortex supporting the perception of 3D structure. In contrast, most of the computer vision models that learn visual representation from a pool of 2D images often fail to generalize over novel camera viewpoints. Recently, the vision architectures have shifted towards convolution-free architectures, visual Transformers, which operate on tokens derived from image patches. However, these Transformers do not perform explicit operations to learn viewpoint-agnostic representation for visual understanding. To this end, we propose a 3D Token Representation Layer (3DTRL) that estimates the 3D positional information of the visual tokens and leverages it for representations. The key elements of 3DTRL include a
learning niewpoint-agnosee pseudo-depth estimator and learned camera matrix to impose geometric transfor- mations on the tokens, trained in an unsupervised fashion. These enable 3DTRL to recover the 3D positional information of the tokens from 2D patches. In practice, 3DTRL is easily plugged-in into a Transformer. Our experiments demonstrate the effectiveness of 3DTRL in many vision tasks including image classification, multi- view video aligmment, and action recognition. The models with 3DTRL outperform their backbone Transformers in all the tasks with minimal added computation. Our com/ el wwassinnnsDTa
code is available at httts:: pestps://get
Introduction
to accurately classify objects in image, align image frames among video pairs, classify actions in videos, and more. Despite their successes, many of the models neglect that the world is in 3D and do not extend beyond the XY image plane 200 While humans can readily estimate the 3D structure
Over the past few years, computer vision models have developed rapidly from CNNs
26 S8
now Transformers I6 24 59 With these models,
scene from 2D pixels an image, most of the existing vision models with 2D images do not take the 3D structure of the world into consideration. This one the reasons why humans are able to recognize objects in images and actions in videos regardless of their viewpoint, but the vision models often fail to generalize over novel viewpoints [1]
Consequently, in this paper, we develop approach to learn viewpoint-agnostt representations for robust understanding the visual data. Naive solutions to obtain newwoont--ggossee representation would be either supervising the model with densely annotated 3D data, or learning representation from a large scale 2D datasets with samples encompassing different viewpoints. Given the fact that such high quality data expensive acquire and hard to scale, approach with higher sample efficiency without 3D supervision is desired.
To this end, we propose a 3D Token Representation Layer (3DTRL), incorporating 3D camera transformations into the recent successful visual Transformers 7 36 59 3DTRL first recovers camera-centered 3D coordinates of each token by depth estimation. Then 3DTRL estimates a camera
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Tokemizaai contiittionn
Trensforme
Layers
3DTRL
Trensformes
Viewpoint- Agnostic Representation
Tokenss
Deptt
SEI Prodieees Enaoddi
Trensom
Camera Pa amener Essimanor
[[R(
XB
KD
Input
30TRL
Figure I: Overview of proposed 3DTRL. Left: 3DTRL is module inserted in between Transformer layers. Right: 3DTRL has three parts, Pseudo-Depth Estimator, Camera Parameter Estimator, and 3D Positional Embedding layer. Within Pseudo- Depth Estimator, we first estimate depth of each token and then calculate 3D coordinates from 2D locations and depth.
matrix to transform these camera-centered coordinates to a 3D world space. In this world space, 3D locations of the tokens are absolute and view-invariant, which contain important information for learning viewpoint-agnoste representations. Therefore, 3DTRL incorporates such 3D positional in- formation in the Transformer backbone the form of 3D positional embeddings, and generates output tokens with the 3D information. Unlike visual Transformers only relying on 2D positional embedding. models with 3DTRL are more compliant with learning viewpoint-agnostic representations.
We conduct extensive experiments on various vision tasks to confirm the effectiveness of 3DTRL. Our 3DTRL outperforms the Transformer backbones 3 image classification, multi-view video aligmment, and 2 multi-view action recognition datasets in their respective tasks. Moreover, 3DTRL is leggt-weighted, alug-and-play module that achieves the above improvements with minimal (2% computation and 4% parameters) overhead.
In summary, we present learnable, differentiable layer 3DTRL that efficiently and effectively learns viewpoint-agnostic representations.
2 Background: Pinhole Camera Model
3DTRL is based the standard pinhole camera model widely used in computer vision. Thus, first briefly review the pinhole camera model. In homogeneous coordinate system, given a point with world coordinate world camera projects pixel p image by:
p == K
R p world
(1)
where K is the intrinsic matrix and R|t is the extrinsic matrix. K is further represented by
C
K
0
00
1
where e is the focal length and (uo, Up) are offset. In this work, we explore visual understanding in multi-view setting, thus aiming at learning jiewpoint-annosti representations. In this setting, a scene may be captured with different cameras positioned Here, the world coordinate morid is the same across all the cameras while pixel projection is different across cameras. We focus on how to estimate the world coordinates p world from their corresponding image pixels at p. Estimating p world from involves two transformations that correspond to the inverse of K and R/tt which might not be known beforehand. Thus, we learn to estimate them from image patches instead, which is key procedure in 3DTRL.
non-identical locations and viewing angles.
3D Token Representation Layer (3DTRL)
In this section, we detail how 3DTRL estimates 3D positional information in a Transformer and is integrated. We will introduce 3DTRL for image analysis, then we adapt to video models.
3.1 Overview
3DTRL is a simple yet effective plug-in module that can be inserted in between the layers of visual Transformers (Figure T left). Given set of input tokens S, 3DTRL. returns a set of tokens with 3D information. The number of tokens and their dimensionality are kept unchanged. Within the

---

module (Figure (nguueerrghtt,, 3DTRL first performs 3D estimation using two estimators: (1) pseudo-depth estimator and (2) camera parameter estimator. Then the tokens are associated with their recovered world 3D coordinates, which are transformed from estimated depth and camera matrix. Finally, 3DTRL generate 3D positional embeddings from these world coordinates and combine them with input S to generate the output of 3DTRL..
In the aspect that we insert 3DTRL in between the Transformer model, 3DTRL implicitly leverages the Transformer layers before it to be a part of the 3D estimators, and layers after that to be the actual 3D feature encoder (Figure ! left). This avoids adding large number of parameters while resulting in reasonable estimations. We empirically find that placing 3DTRL at a shallow-meding the network yields better results (see Section 4.6). which is a trade-off between model capacity for estimation and 3D feature encoding.
layer of
3.2 3D Estimation of the Input Tokens
3DTRL first estimates both the camera- centered 3D coordinates of each token using depth estimation and camera matrix shared by all the tokens of an image. Then, the camera-centered 3D coordinates are transformed to the "world" coordinates by the camera matrix.
Pseudo-depth Estimation. Given input tokens S = {sl...... SN) RR 3DTRL first performs pseudo-depth estimation of each token Sn. The pseudo-depth estimator a function f R + I that outputs the depth d, f(s,) of each token individually. We implement using 2-layer MLP. We call this pseudo-depth estimation since it is similar to depth estimation from monocular images but operates at very coarse scale, given that each token corresponds to an image patch rather than single pixel in Transformer.
After pseudo-depth estimation, 3DTRL transforms the pseudo-depth map to camera-centered 3D coordinates. Recall that in VIT [16], an image X is decomposed into N patches {X1.... XN) E XXXXXxX3 where P x P is the size of each image patch and the tokens S are obtained from linear projection of these image patches. Thus, each token is initially associated with a 2D location on the image plane, denoted as (u, v). By depth estimation, 3DTRL associates each token with one more value d. Based on the pinhole camera model explained in Section 2 3DTRL transforms u, v, d to camera-centered 3D coordinate (I. 2. by:
Ut 2m Un /c where 2,, dp,
Pn
V/
Zn
in
Since we purely perform the aforementioned estimation from monocular images and the camera intrinsic matrix is unknown, we simply set e to a constant hyperparameter. To define coordinate system of 2D image plane (u, v), we set the center of the original image is the origin (0, 0) for convenience, so that the image plane and camera coordinate system shares the same origin. We use the center of the image patch to represent its associate (u, v) coordinates.
We believe that this depth-based 3D coordinate estimation best leverages the known 2D geometry, which is beneficial for later representation learning. We later confirm this in our ablation study (in Section 4.6). where we compare it against a variant of directly estimating (1r, y: =) instead of depth.
Camera Parameter Estimation. The camera parameters are required to transform the estimated camera-centeree 3D coordinates paam the world coordinate system. These camera parameters are estimated jointly from all input tokens S. This involves estimation of two matrices, a 3 x 3 rotation matrix R and
1 and translation matrix through an estimator g. We implement g using MLP. Specifically, we use a shared MLP stem to aggregate all the tokens into intermediate representation. Then, we use two separated fully connected heads to estimate the parameters in and respectively. We note the camera parameter estimator as a whole: R.,t.] = g(S). To ensure R. is mathematically valid, we first estimate the three values corresponding to yaw, pitch and roll angles of the camera pose, and then convert them into a 3 x 3 rotation matrix. Research has shown that the discontinuity occurs at the boundary cases in rotation representation 63 however, such corner cases are rare.
In case of generic visual understanding tasks like object classifications, we expect the camera parameter estimation to perform an "object-centric canonicalization" of images with respect to the "common" poses of the class object. This is qualitatively shown by Figure I0 in the Appendix.
Transform to World Coordinates. Now, with the camera parameters, 3DTRL transforms estimated camera-centered coordinates aa into the world space, 3D space where 3D coordinates of the
3

---

tokens are absolute and viewpoint- invariint Following the pinhole camera model, we recover p
world
R R R p.
Pn
3.3 Incorporating 3D Positional Information in Transformers
The last step of 3DTRL is to leverage the estimated 3D positional information Transformer backbone. For this, we choose to adopt typical technique of incorporating positional embedding that is already used in Transformers [16] 600. contrast to 2D positional embedding in ViTs [[6],
3DTRL learns a 3D embedding function h R + R" transform estimated world coordinates p world to positional embeddings pdD This 3D embedding function h is implemented using a two-layer MLP. Then, the obtained 3I) positional embedding is incorporated in the Transformer backbone by combining it with the token representations. The outcome is the final token representations {s²D):
Sp Sn + p where DdD hhppp"pm )
JD
After 3D embedding, the resultant token representations associated with a 3D space, thus enabling the remaining Transformer layers to encode viewpoint-agnosse token representations. We ablate other ways of incorporating the 3D positional information of the tokens Section 4.6
3.4 3DTRL in Video Models
Notably, 3DTRL can be also easily generalized to video models. For video models, the input to 3DTRL a set of spatial-temporal tokens {Si,..., ST} corresponding to a video clip containing T frames, where S, = {......., S,N } are N tokens from t-th frame. We simply extend our module to operate on an additional time dimension, where depth estimation and 3D positional embedding done for each ppatial-temporal tokens Stn individually: de, f(tn), Pmn h(pporld) Camera parameters are estimated per input frame (S₁) in a dissociated manner, namely [R|t ] = g(s)). resulting in total of T' camera matrices per video. We investigate another strategy of camera estimation in the supplementary, where only one camera matrix is learned for all frames. However, studies have substantiated the effectiveness of learning dissociated camera matrices per frame.
Experiments
We conduct extensive experiments demonstrate the efficacy of viewpoint-agnoske representations learned by 3DTRL in multiple vision tasks: (i) image classification, (ii) multi-view video alignment, and (iii) video action classification. We also qualitatively evaluate the pseudo-depth and camera estimation to confirm 3DTRL works.
4.1 Image Classification
In order to validate the power of 3DTRL, we first evaluate VIT 16 with 3DTRL for image classifica- tion task using CIFAR-10 B41 CIFAR- 100 34 and ImageNet-IK 15] datasets.
Training We use the training recipe of DeiT 59 for training our baseline Vision Transformer model on CIFAR and ImageNet datasets from scratch. We performed ablations find an optimal location in ViTs where 3DTRL should be plugged-in (Section 4.6). Thus, in all our experiments we place 3DTRL after Transformer layers, unless otherwise stated. The configuration of our DeiT-T, DeiT-S, and DeiT-B is identical to that mentioned in [59]. All our transformer models are trained for 50 and 300 epochs for CIFAR and ImageNet respectively. Further training details and hyper-parameters can be found in the supplementary material.
Results From Table 1 3DTRL with all DeiT variants shows consistent performance improvement the baseline the CIFAR datasets, with only ~2% computation overhead and ~4% more parameters. Despite fewer training samples, 3DTRL significantly outperforms the baseline in CIFAR, showing the strong generalizability of 3DTRL when limited training data available. We argue that multi-view data is not available in abundance, especially in domains with limited data (like in medical domain), thus 3DTRL with its ability to learn viewpoint-agnoste representations will be crucial in such domains. We also find the performance improvement on ImageNet is less than that on CIFAR. This is because ImageNet has limited viewpoints in both training and validation splits, thus reducing the significance of performing geometric aware transformations for learning view agnostic representations. In contrast, CIFAR samples present more diverse camera viewpoints, so it is more suitable dataset for testing the quality of learned wewwoinn--gnostti representations.

---

Table I: Top I classification accuracy (%) on CIFAR-I0 and 100, ImageNet- IK (IN-1K), viewpoint-pertubed IN-1K (IN- IK- and ObjectNet. We also report the number of parameters (#params) and computation (in MACs). Note that the MACs are reported w.r.t IN-IK IK samples.
Method
Apurams
MACs
CIFAR-10
CIFAR-100 IN-KK
IN-IKpp
ObjectNet
DeiFT
5,72M
1,00
74,1
51,3
73,4
61.3
21.3
+3DTRL
5.95M
1.10G
78.8 (447)
(+2.4
73.6
(40.2)
(+.3)
22.4 (+1.1)
DerT-S
22,1M
4,246
77.2
54.6
79.4
21.1
25.8
Figure 2: Top: original IN-IK sam-
+3DRRL
23.0M
4,33G
80,7 (+3.)
(16.)
79.7
(+0.3)
72.7
(-16) (+1.))
DeiT.B
76.6
81.0
ples. Bottom: viewpoint-perturbee IN-1K samples.
+3DTRL
90.1M
17.20
82.8 (+6.2)
61.8 (+9.9)
81.2 (+0.3)
74,7 (-4.0) 27.3 (40.3)
Robustness on Data with Viewpoint Changes In order to emphasize the need of learning aiemosint-agmostie representations, further test the models trained on IN-IK on test datasets: ObjectNet 2 and ImageNet- IK perturrbe (IN-IK-p). ObjectNet 2 is a test set designed to intro- duce more rotation, viewpoint, and background variances in samples compared to ImageNet. Overall, ObjectNet is a very challenging dataset considering large variance in real-world distributions. IN-1K- p is hetppoinn--perturber IN-1K validation set by applying random perspective transformations to images, constructed by our own. Example images are shown in Figure - We note that perspective transformation these static images is not equivalent to real viewpoint changes. Nonetheless, it is a meaningful for a proof-of-concept experiment. Results in Table [
show that models with 3DTRL consistently outperform their corresponding Transformer baselines in these two test sets, suggesting 3DTRL is trained to generalize across viewpoint changes.
4.2 Multi-view Video Alignment
Video alignment 18 2 41 is task learn frame-to-fran fraaee mapping between video pairs with close semantic embeddings. In particular, we consider a multi-view setting that aligns videos captured from the same event but different viewpoints, which could further facilitate robot imitation learning 56). Here, video pairs from the same event are temporally
from third-person views synchronized.
40
51
Pouring
MC
Pick
Can
Lift
Figure 3: Examples video alignment datasets. Each dataset has synchronized videos of at least 2 viewpoints. All datasets except Pouring have one ego-centric view, highlighted in red boxes. More details are available in supplementary material.
Datasets We use 5 multi-view datasets from a wide range of environments: Minecraft (MC) video game, Pick, Can, and Lift from robot simulators (PyBullet [9] and Robomimic 39) and Pouring from real-world human actions 49 Example video frames are provided in Figure 3 Each dataset contains synchronized videos from multiple cameras (viewpoints). There one ego-centric
camera per dataset except Pouring, which are continuously moving with the subject. These ego-centric videos make the alignment more challenging. Detail dataset statisties available in supplementary.
Training We follow common video alignment methods 40 train an encoder that outputs frame- wise embeddings. We still use DeiT 59 as baseline model (DeiT+TCN) and apply 3DTRL to it (43DTRL), similar to image classification. During training. we use the cime-contrastive loss 3y to encourage temporally closed embeddings to be similar while temporally far-away embeddings to be apart. Then, we obtain alignments via neirest- such that an embedding u; from video is being paired to its nearest neighbor vj in video 2 in the embedding space. And similarly uj from video 2 is paired with its nearest neighbor U2 in video 1. We use ImageNet-1KK pre-trained weights for experiments on Pouring, but we train from scratch for other datasets considering that simulation environments are out of real-world distribution.
5

---

Alignment Error
Cycle Error
Kendall's Tau
1.00
8.20
0.00
Seen
0.10
1.10
0.96
Pouring Pick
MC
Can
Pouring Pick
MC
Can
Lift
Pouring Pick
MC
Can
Lift
0.40
0.30
0.75
0.30
0.22
026
Unseen
0.20
0.00
0:10
-9
0.00
0.00
Pick
MC
Can
Pick
MC
Can
Lift
DeIT+TCN
+3DTRL
Figure 4: Results on video alignment in Seen and Unseen protocols. t indicates a higher metric is better and 1 is otherwise. Blue bars are for DeiT+TCN without 3DTRL and green bars are with 3DTRL, 3DTRL outperforms the baseline consistently in both settings. We note Unseen is applicable to Pouring dataset because only two cameras are available in this dataset.
Evaluation We evaluate the alignment by three metrics: Alignment Error 49 Cycle Error, and [3) Let the alignment pairs between two videos be (us 05) and (1y, ux). In brief, Aligmment Error measures the temporal mismatching i - of (u,, 0y). Cycle Error based on where pairs (#z, U2) and (uj, u&) called consistent when k. Thus, Cycle Error measures the inconsistency based on distance metric l ---k, kl.. Kendall's Tau (r) measures ordering pairs. Given pair of embeddings from video (ui, uj) and their corresponding nearest neighbors from video (Up. ti),, the indices tuple (i, q) is concordani when < j and p q j and p > q- Otherwise the tuple is discordant. Kendall's Tau computes the ratio of concordant pairs and discordant pairs over all pairs of frames. Let N be the number of frames in video, then the formal notations of the three metrics are:
Kendall's Tau
cycle-consistency IS
Alignment Error = E, ب
Cycle Error E, N
# concordant discordant N -1/2
We establish two evaluation protocols: (a) Seen and (b) Unseen. In Seen, we train and test models on videos from all cameras. However, in Unseen, we hold out several cameras for test, which is representative scenario for validating the effectiveness of 3DTRL. Detail of experimental settings are provided in supplementary.
Results Figure illustrates the evaluation results of 2 viewpoint settings over datasets, compared to the DeiT baseline. 3DTRL outperforms the baseline consistently across all datasets. In particular, 3DTRL improves the baseline by large margin in Pouring and MC, corroborating that 3DTRL adapts to diverse unseen viewpoints. The improvements of 3DTRL Pick and MC also suggests the strong generalizability when learning from smaller datasets. With enough data (Lift & Can), 3DTRL still outperforms but the gap is small. When evaluating in Unseen setting, both methods have performance drop. However, 3DTRL still outperforms Pick, MC, and Can, which suggests the representations learned by 3DTRL are able generalize over novel viewpoints. MC has the largest viewpoint diversity so it is hard to obtain reasonable align results in the unseen setting for both the models.
Table 2: Video alignment results compared with SOTA methods. Values are alignment errors.
DDei++TNN +3DTRL
Alignment Error
Method
Backbone
Input
Pouring
Pick MC
TCN 49
CNN
1 frame frame
0.180
0.273
0,286
0.10
Disentanglement
51
CNN
0.155 0.233
0.55
mITCN
3DCNN
frames
0,143
mfTCN
3DCNN
frames
0.088
0.00
15
DeiT 4TCN
Transformer
frame
0.155
0.216
0.292
Number Training Videos
+3DTRI
Transformer
frame
0,080
0.116 0.202
Figure 5: Alignment error w.r.t # training videos.
Red dashed line indicates 3DTRL using 10 videos outperforms DeiT using 45 videos.
In Table
we further compare 3DTRL with previous methods on certain datasets. Note that Disentanglement [5] uses extra losses whereas others time-contrastive loss only. We find that

---

3DTRL with only single-frame input is able to surpass the strong baselines set by models using extra losses [5T or multiple imput frames [77. We also vary the number of training videos in Pouring dataset, and results from Figure 5 show that 3DTRL benefits from more data. Meanwhile, 3DTRL can outperform the baseline while only using 22% of data the baseline used.
4.3 Quantitative Evaluations on Recovering 3D information
We perform quantitative evaluations on how well 3DTRL recovers 3D information. We emphasize once more that the 3D recovery in our approach is done without any supervision; it is optimized with respected to the final loss (e.g.. object classification), without any access to the ground truth 3D information. We first discuss 3D estimations focusing on pseudo-depth estimation (Section 4.3. then we evaluate camera estimation (Section 4.3.2).
4.3.1 3D Estimation Evaluation
The key component of our 3D estimation is pseudo-depth estimation. In order to evaluate the 3D estimation capability, compare the pseudo-depth map with ground truth depth map, using NeRF 40 dataset. We test the pseudo-depth with DeiT--+3DTRRI trained on IN-IK.
Depth Correlation. Since our estimated pseudo-depth (d') and ground truth (d) are in different scales, we measure their relative correspondence, i.e., correlation of two sets of data. We use Pearson'sr: r: r= correlation (d, d'), where we regard two depth maps as two groups of data. Note that 3DTRL operates on visual Transformers, so the pseudo-depth map is a very coarse scale (14x 14). We also resize the given depth map to 14x 14 to perform the evaluation. We report the average of r across all evaluation subsets
Metric:
Results. F the evaluation results. We find the estimated pseudo-depth highly correlates with the ground truth (r = 0.7). Compared to the random prediction baseline, the depth from 3DTRL shows much higher correlation to the ground truth. We also find that the model learns the estimated depth with higher correlation in the earlier stage of the training (e.g., 20 epochs) and then drops a bit in the later training epochs (~ 0.07 drop).
Camera Poalion Dipparty Kondalls
Canmun Orteration bi Konda Tau Disparty given
-epph Comsintion Randor
1.52
30
0.
1.00
0.25
4.12
035
07
0.35
03
0.00
08
0.00
200
200
40
Tratning Spodio
Figure 6: Depth correlation eval- uation results. We find our esti- mation has high (~00.) corre- lation to the ground truth.
Figure 7: Quantitative evaluations camera position (left) and camera orientation (right) respectively. Overall, we show the estimated cameras from 3DTRL have acceptable mapping to the ground truth (both give < 0.5 disparity).
4.3.2 Camera Estimation Evaluation
In this evaluation, we mainly answer how well 3DTRL estimates camera position and orientation. To do this, we evaluate DeiT+3DTRL trained in previous video-alignment task (Section 4.2) using Can dataset. Recall that we train the model from scratch for the video aligmment, without access to ground truth 3D information. We use first-person view videos for our evaluation the camera moves together with the robot and our objective is to estimate its pose. We introduce two metrics.
Metricl: Camera Position Disparity: For each video, get the estimated camera positions (p') and ground truth camera positions (p). which are 3-D vectors. Since estimated and ground truth cameras in two different coordinate systems, we need to measure how estimated camera positions map to the ground truth in a scale, translation and rotation-invariant way. The existing metrics
like AUROC 30 48 are not applicable. Therefore, we use Procrustes analysis [21] and report the disparity metric. The disparity value ranges [O. 1J, where a lower value indicates that two sets are more similar. We take the average disparity of all videos.
Metric2: Camera Orientation Disparity: Each camera has its orientation, i.e. "looking-at" direction, described by a 3-D vector. We note the estimated camera orientations {o'} and ground truth (o).
'oo do so, convert to Fisher's z, take the mean value across all subsets, and convert back to Pearson's
7

---

Appe
Louoor
Chairias dloodin
Facc
Xoot
Figure 8: Top: IN-1K samples with corresponding class label. Bottom: Estimated pseudo-depth map, interpolated from 16 x 16 to 224 x 224 for better understanding. Depth increases from blueish to yellowish region. More examples are in Figure I4 in Appendix.
Similar as the camera position disparity mentioned above, we use the disparity given by Procrustes analysis 21 to measure how {o'} and {o} align. We report the mean disparity from all the videos.
Results. Results are shown in Figure 7 Both position and orientation disparity of the final model are relatively small (< 0.5), showing that the camera estimation from 3DTRL is partially aligned with the ground truth. We also observe that the disparity decreases over the training epochs.
4.4 Qualitative Evaluations
We visualize some qualitative results for an intuitive understanding of 3DTRL's effectiveness. The visualization contains estimated pseudo-depth maps which are from the key intermediate step of 3DTRL. We use ImageNet- IK trained DeiT-T+3DTRL and its validation samples for visualization. In Figure S we observe a fine separation of the object-of-interes and the background in most of the depth maps. Thus, these predicted pseudo-depth maps are sufficient to recover the tokens corresponding to object-of-intereet in 3D space. For a qualitative evaluation on camera pose, please refer to Figure T0 in the Appendíx, where show the images with the same/similar object pose (in different environments) result in similar estimated extrinsics regardless of the background. We believe our estimated extrinsics are doing object-centric canonicalization of images with respect to their object poses, in order to optimize the representations for the final losses.
4.5 3DTRL on More Transformer Architectures
3DTRL is designed to be plug-and-play model for Transformers. We test 3DTRL with more Transformer architectures on CIFAR and two multi-view video alignment datasets. We use Swin (36] (Tiny) and TnT [241 (Small). Results are provided in Table 3 We find that 3DTRL generally improves the performance of two Transformer architectures in all the datasets. This confirms 3DTRL is applicable to different Transformer architectures. The relative small improvement over the Swin backbone is due to the strong inductive bias from its local window.
Table 3: Results of using 3DTRL in more Transformer architectures, on CIFAR and multi-view video alignment datasets. Reported numbers accuracy for CIFAR and Kendall's tau for video alignment, both are the higher the better. We show 3DTRL generally improves the performance in all tasks.
Model
CIFAR-10
CIFAR-100
Pouring
Pick
Swin-T
50.11
21.53
0.584
0.623
+3DTRL
50.29 (40.18)
(-0.02)
0.683 (40.099)
0.640 (+0.017)
TuT-S
81.25
54.07
0.740
0.640
+3DTRL
82.43 (+1.18)
56.00
(-1.93)
0.792 (+0.052)
0.671 (+0.031)
4.6 Ablation Studies
We conduct our ablation studies on image models mostly using CIFAR for image classification and Pick for multi-view video alignment.
MLP vs. 3DTRL. 3DTRL is implemented by several MLPs with required geometric transforms in between. In this experiment, we replace 3DTRL with the similar number of fully-connected layers with residual connection, to have comparable parameters and computation as 3DTRL, Results are provided in Table 4 We find that MLP implementation is only comparable with the baseline performance despite the increase in parameters and computation. Thus, we confirm that the geometric transformations imposed on the token representations is the key to make 3DTRL effective.
Token Coordinates Estimation. In this ablation, we show how estimating only depth compared to estimating set of 3 coordinates zyz differs in 3DTRL. We find at Line 4 and 5 in Table 4 that
8

---

Table 4: Ablation study results. For CIFAR, we test on models based on tiny (T), small (S) and base (B) back- bones (DeiT) and report accuracy(%)) ()). For Pick we only test base model and report alignment error.
CIFAR 10 Pick
80
0.65
Kendall's Tau
Accuracy
78
0.63
77
0.60
0.58
Method
CIFAR-10 (1/S/8) CIEAR-100 (1/S/8) Pick
DeiT
74.1/ 76.2/7666
51.3/546/51.9
0.216
1/6
1/3
1/2
2/3
DeiT MLP 3DTRL
74,2 /
47.9/54.7755
0.130
Relative Position of 3DTRL Inserted
78.8
53.7/61.5 61.8
Depch Estimation - zy= Estimation
76.7/78.2 78.2/7774
48.3/54.1/525
0.134
Figure 9: Results on inserting 3DTRL at different locations.
Embedding Concat.
80.7 83.7 / /84,9 53./66.8/00
0.133
depth estimation is better because it uses precise 2D image coordinates when recovering tokens 3D space, whereas estimating zy² regresses for the 3D coordinates without any geometric constraints. Also, we find that estimating zyz hampers the performance in image classification task more than video alignment. This is because estimating 3D coordinates is harder when the training samples are in unconstrained scenarios. In contrast, video pairs in an alignment dataset share the same scene captured from different view angles, which facilitates the recovery of token in 3D space.
How to incorporate 3D positional information in Transformers? By default, we use Equation 5 to incorporate the 3D positional information in Transformer backbone through learned positional embedding p3D In this experiment, we directly infuse the estimated 3D world coordinates woodd within the token representation by concatenating them across the channel axis. The (m + 3)-d feature is then projected back to m-d by a MLP. We keep parameters and computation comparable to the default 3DTRL. We test this 3DTRL variant (Embedding - Concat.) and results are presented in Table E We find that the concatenation variant outperforms the default variant of 3DTRL on CIFAR-IO, but comparable and worse results in CIFAR- 100 and Pick. This observation substantiates the instability of using raw 3D coordinates. In comparison, the use of 3D positional embedding is generalizable more challenging and diverse scenarios.
Comparison with perspective augmentation We compare 3DTRL with vanilla data augmentation on perspective transforms video alignment task and results are in Appendix We confirm 3DTRL does better than applying perspective augmentations.
Where should we have 3DTRL? We vary the location of 3DTRL to empirically study the optimal location of 3DTRL in a 12-layer DeiT-T backbone. In Figure ! we find that inserting 3DTRL at the earlier layers (after 1/3 of the network) yields the best performance consistently on both datasets.
4.7 3DTRL for Video Representation Learning
In this section, we illustrate how 3DTRL can be adapted for video models. Since we aim at learning viewpoint-agnostic representations, as natural choice validate the effectiveness of 3DTRL on video datasets with multi-camera setups and cross-view evaluation. Consequently, we conduct our experiments on two multi-view action recognition datasets: Toyota Smarthome 12 (Smarthome) and NTU-RGB+D (50) (NTU) for the task of action classification. For evaluation on Smarthome, we follow Cross-View 2 (CV2) and Cross-Subject (CS) protocols proposed in [12.. whereas on NTU, we follow Cross- View (CV) protocol proposed in [50]. In cross-view protocols, the model is trained on set of cameras and tested on different set of cameras. Similarly for cross-subject protocol, the model is trained and tested on different set of subjects. More details on these datasets are provided in Appendix[] !]
Table 5: Results on action recognition on Smarthome and NTU. Ace is classification accuracy (%) and mPA is mean per-class accuracy. In methods using Kinetics-400 (K400) pre-training, TimeSformer backbone is always initialized with pre-trained weights, and 3DTRL w/, w/o K400 denotes 3DTRL is randomly initialized and is initialized from pre-trained weights respectively.
Smarthome (CV2)
Smarthome (CS)
NTU (CV)
Method
Acc
mPA
Acc
mPA
Acc
TimeSTormer
59,4
27.5
75.7
56.1
86.4
+ 3DTRL
62.9 (43.5)
34.0 (+6.5)
76.1 (+0.4)
(40.9
87.9 (+1.5)
Kinetics-400 pre-trained
TimeSformer 33
69.3
37.5
77.2
57.7
87.7
+ 3DTRL wfo K400
69.5 (+0.2)
39.2 (+1.7)
77.5 (+0.3)
58.9 (+1.2)
88.8 (+1.I)
3DTRL w/ K400
71.9 (+2.6)
41.7 (+4.2)
(+0.6)
61.0 (+2.3)
88.6 (+0.9)

---

Network architecture & Training / Testing TimeSformer B is straightforward extension of ViT [6] for videos which operates on spatio-temporal tokens from videos, so that 3DTRL can be easily deployed to TimeSformer as well. Similar to our previous experimental settings, we place 3DTRI, after 4 Transformer blocks TimeSformer. Please refer to Appendix for detailed setrings.
Results In Table 5 we present the action classification results on Smarthome and NTU datasets with 3DTRL plugged in TimeSformer.
3DTRL can easily take advantage of pre-trained weights because it does not change the relying backbone Transformer -jus juust being added in between blocks. In Table [] we present results for two fine-tuning scenarios: (a) 3DTRL w/o K400 and (b) 3DTRL w/ K400. For the first scenario (a), TimeSformer initialized with K400 pre-training weights and leave the parameters in 3DTRL ran- domly initialized. Then in the fine-tuning stage, all the model parameters including those of 3DTRL is trained. In the second scenario (b), all parameters in TimeSformer and 3DTRL are pre-trained on K400 from scratch and fine-tuned on the respective datasets.
We find that all the variants of 3DTRL outperforms the baseline TimeSformer results. Our experiments show that although there is an improvement with 3DTRL compared to the baseline for different fine-tuning strategy, it is more significant when 3DTRL is pre-trained with K400. However, when large-scale training samples are available (NTU), 3DTRL does not require K400 pre- pre-training To sum up, 3DTRL can be seen as a crucial ingredient for learning diewpoint-agnoste video representations.
5 Related Work
There has been remarkable progress in visual understanding with the shift from the use of Transformers have shown substantial improvements
CNNs 6 26
35
58 to visual Transformers 16
over CNNs in image [16
24
BI
36
59
analysis and video I| 37
44
47 understanding
tasks due to its flexibility in learning global relations among visual tokens. Studies also combine CNNs with Transformer architectures to leverage the pros in both the structures [C
38
52
53
In addition, Transformer has been shown to be effective in learning 3D representation
68 How-
ever, these advancements in architecture-tye types have not addressed the issue of learning viewpoint- agnostic representation. Viewpoint-agnose
representation learning is drawing increasing atten- tion in the vision community due to its wide range of downstream applications like 3D object-
detection 46 video alignment [8] 18, [9], action recognition 54 55
pose estimation
25 robot
learning
51 56
and other tasks.
There is a broad line of work towards directly utilizing 3D information like depth 25 pose B3 14,
and point clouds A 35 in some cases deriving 3D structure from paired 2D inputs 601. However, methods rely on the availability of multi-modal data which is hard to acquire are not scalable.
Consequently, other studies have focused on learning 3D perception of the input visual signal in order to generalize the learned representation to novel viewpoints. This is done by imposing explicit 46 64]. without the requirement of any 3D supervision. contrast to these existing works, our Transformer-based 3DTRL imposes geometric transformations on visual tokens to recover their representation in a 3D space. To the best of our knowledge, 3DTRL is the first of its kind to learn 3D positional embedding associated with the visual tokens for viewpoint-agnosti representation learning in different image and video tasks.
geometric transform operations in CNNs [. 28 42
6 Conclusion
In this work, we have presented 3DTRL,, plug-and-play module for visual Transformer that leverages 3D geometric information learn wewpoint-agnostii representations. Within 3DTRL, by pseudo- depth estimation and learned camera parameters, it manages to recover positional information of tokens in 3D space. Through our extensive experiment, we confirm 3DTRL is generally effective in a variety of visual understanding tasks including image classification, multi-view video alignment, and cross-view action recognition, by adding minimum parameters and computation overhead.
A Acknowledgment
We thank insightful discussions with members of Robotics Lab at Stony Brook. This work is supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Ministry of Science and ICT (No.201 8-0-00205, Development of Core Technology of Robot Tasellnnell for Improvement of Labor Condition. This work also supported by the National Science Foundation (IIS (IS-2104404 and CNS-2104416).
10

---

References
[1] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid. Vivit: A video vision transformer. In Proceedings of the International Conference on Computer Vision (ICCV), pages 6836-6846, 2021.
[2] A. Barbu, Mayo, Alverio, W. Luo, C. Wang, D. Gutfreund, Tenenbaum, and Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models, Advances in neural information processing systems, 32, 2019.
[3] G. Bertasius, H. Wang, and L. Torresani. Is space-time attention all you need for video understanding? Proceedings of the International Conference on Machine Learning (ICML), July 2021.
[4] R. Burgert, J. Shang, X. Li, and M. Ryoo. Neural neural textures make sim2real consistent. arXiv preprint arXiv:2206. 13500, 2022.
[5] A. Cao and R. de Charette. Monoscene: Monocular 3d semantic scene completion. arXiv preprint arXiv:21 12.0222...0.22. 2021.
[6] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In The British Machine Vision Conference (BMVC), 2014.
[7] C.-F.R. R. Chen et al. Crossvit: Cross-attention multi-scale vision transformer for image classifi- cation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[8] M. Chen, F. Wei, C. Li, and D. Cai. Frame-wise action representations for long videos via sequence contrastive learning. arXiv preprint arXiv:2203. 14957, 2022.
[9] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. httt:://pybull/
org 2016-2019.
[10] 7. Dai, H. Liu, Q. V. Le, and M. Tan. Coatnet: Marrying convolution and attention for all data sizes. In Advances Neural Information Processing Systems (NeurIPS), 2021.
[11] S. Das and M. S. Ryoo. Viewelr: Learning self-supervised video representation for unseen viewpoints. Arxiv, 2112.03905, 2021.
[12] Das, R. Dai, M. Koperski, L. Minciullo, Garattoni, Bremond, and G. Francesca. Toyota smarthome: Real-world activities of daily living. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
[13] S. Das, S. Sharma, R. Dai, F. Bremond, and M. Thonnat. Vpn: for activities of daily living. In Proceedings of the European Conference on Computer Vision (ECCV), pages 72-90. Springer, 2020.
Learning video-pose embedding
[14] S. Das, R. Dai, D. Yang, and F. Bremond. Vpn++: Rethinking video-pose embeddings for understanding activities of daily living. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[15] J. Deng, W. Dong, R. Socher, Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255 2009.
[16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, Gelly, Uszkoreit, and N. Houlsby. An image worth 16x words: Transformers for image recognition at scale. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[17] D. Dwibedi, J. Tompson, C. Lynch, and P. Sermanet. Learning actionable representations from visual observations. IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1577-1584, IEEE, 2018

---

[18] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. Temporal cycle- consisennss learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1801-1810, 2019.
[19] Gao, M. Chen, and C. Xu. Fine-grained temporal contrastive learning for weakly-supervised temporal action localization. arXiv preprint arXiv:2203. 16800, 2022.
[20] J. Georgia Gkioxari, Jitendra Malik. Mesh r-cnn. Proceedings of the International Conference on Computer Vision (ICCV), 2019.
[21] J. C. Gower. Generalized procrustes analysis. Psychometrika, 40(1):33-51, 1975.
[22] B. Graham, A. El-Nouby, H. Touvron, Stock, A. Joulin, H. Jégou, and M. Douze. Levit: vision transformer in convnet's 's clothing for faster inference. In Proceedings of the International Conference on Computer Vision (ICCV), pages 12259-12269, 2021.
[23] I. Hadji, K. G. Derpanis, and A. D. Jepson. Representation learning via global temporal alignment and sycle-consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10668-1077,, 2021.
[24] K. Han al. Transformer in transformer. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[25] A. Haque, B. Peng, Z. Luo, A. Alahi, S. Yeung, and L. Fei-Fei. Towards viewpoint invariant 3d human pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 160-177. Springer, 2016.
[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[27] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn. Vision-based manipulators need to also see from their hands. Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[28] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015.
[29] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters (RA-L), 2022.
[30] L. Jin, S. Qian, A. Owens, and D. F. Fouhey. Planar surface reconstruction from sparse views. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12991-13000, 2021.
[31] K. Kahatapitiya and M. S. Ryoo. Swat: Spatial structure within and among tokens. arXiv preprint arXiv:2 13677, 2021.
[32] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, Green, T. Back, Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 2017.
[33] M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81 1938.
[34] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Y. LeCun, Y. Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
[36] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the International Conference Computer Vision (ICCV), 2021.
12

---

[37] Z. Liu, J. Ning. Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu. Video swin transformer. arXiv preprint arXiv:2206.132 13230, 2021.
[38] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and Xie. A convnet for the 2020s. arXiv preprini arXiv:2201.03544 2022.
[39] A. Mandlekar, D. Xu. J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv: 108. 03298, 2021.
[40] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. 'T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference Computer Vision (ECCV), pages 405-421. Springer, 2020.
[41] I. Misra, C. L.. Zitnick, and M. Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 527-544. Springer, 2016.
[42] A. Piergiovanni and M. S. Ryoo. Recognizing actions in videos from unseen viewpoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4124 4132, June 2021.
[43] A. Piergiovanni, Casser, M. Ryoo, and A. Angelova. 4d-net for learned multi-modal alignment. In Proceedings of the International Conference on Computer Vision (ICCV), pages 15435-15445, 2021.
[44] K. Ranasinghe, M. Naseer, S. Khan, F.S. Khan, and M. Ryoo. Self-supervised video transformer. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[45] D. Robert, B. Vallet, and L. Landrieu. Learning multi-view aggregation in the wild for large- scale 3d semantic segmentation. arXiv preprint arXiv:2204.07548, 2022.
[46] D. Rukhovich, A. Vorontsova, and A. Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2397-2406, 2022.
[47] M. S. Ryoo, Piergiovanni, A. Arnab, M. Dehghani, and Angelova. Tokenleamer: What can 8 learned tokens do for images and videos? Advances in Neural Information Processing Systems (NeurIPS), 2021.
[48] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedingg of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020.
[49] Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang. S. Schaal, S. Levine, and G. Brain. Time- contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), pages 1 1134-1141, IEEE, 2018
[50] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[51] J. Shang and M. S. Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 214-221, 2021. doi: 10. 10.11099/ROSSS 168.16882 00221.666633.
[52] J. Shang, K. Kahatapitiya, X. Li, and M. S. Ryoo. Starformer: Transformer with state-action- reward representations. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[53] J. Shang, X. Li, K. Kahatapitiya, Y.-C. Lee, and M. S. Ryoo. Starformer: Transformer with state-action-reward representations for robot learning. IEEE transactions on pattern analysis and machine intelligence, 2022.
13

---

[54] G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari. Actor and observer: Joint modeling of first and third-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7396-7404, 2018.
[55] G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari. Charades-ego: A large-scale dataset paired third and first person videos. arXiv preprint 1rXiv:180.009226 2018.
[56] B. C. Stadie, P. Abbeel, and I. Sutskever. Third-person imitation learning. ArXiv, abs/1703.01703, 2017.
[57] J. Sun, J. Zhao, L.-C. Chen, F. Schroff, H. Adam, and T. Liu. View-invariant probabilistic embedding for human pose. In Proceedings of the European Conference on Computer Vision (ECCV), pages 53-70. Springer, 2020.
[58] C. Szegedy, Vanhoucke, loffe, Shlens, and Z. Wojna. Rethinking the inception architec- ture for computer vision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818 2818-2826, 2016. doi: 10000//VVR..00....0
[59] H. Touvron, M. Cord, M. Douze, Massa, A. Sablayrolles, and H. Jegou. Training data- efficient image transformers: Distillation through attention. In Proceedings of the International Conference on Machine Learning (ICML), volume 139, pages 10347-10357, July 2021.
[60] A. Vaswani et al. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[61] S. Vijayanarasimhan, Ricco, C. Schmid, Sukthankar, and K. Fragkiadaki. Sfm-net: Learning of structure and motion from video. arXiv preprint arXiv:1704.07880 2017.
[62] X. Wang, A. Jabri, and A. A. Efros. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2566-2576, 2019.
[63] S. Xiang. Eliminating topological errors in neural network rotation estimation using self- selecting ensembles. ACM Transactions on Graphics (TOG), 40(4):1-21, 2021.
[64] X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
[65] L. Yuan, Y. Chen, T. Wang. W. Yu, Y. Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens- to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the International Conference on Computer Vision (ICCV), pages 558-567, 2021.
[66] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the International Conference on Computer Vision (ICCV), pages 6023-6033 2019.
[67] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[68] H. Zhao, L. Jiang. J. Jia, H. Torr, and V. Koltun. Point transformer. In Proceedings of the International Conference on Computer Vision (ICCV), pages 16259-16268, October 2021.
[69] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5745-5753 2019.
14


Reviews:
[{"review_id": "pBeJiy5VXoR", "reviewer": "Reviewer_6XtT", "summary": "In this paper, the authors claim that deep learning-based models applied to 2D images lack 3D information that is easily understood by human vision. In turn, the authors focus on transformer-based architectures. The authors propose a 3D Token Representation Layer (3DTRL), a plug-and-play module for transformer-based architectures to learn viewpoint-agnostic representation for the visual data. Authors do this by lifting 2D token locations to 3D by a pseudo depth estimator, and a pseudo camera parameter estimator. Through experiments on Image Classification datasets and multi-view video alignment datasets, authors show the effectiveness of 3DTRL modules being robust to viewpoint changes.\n", "questions": "How did you come up with eq (3)? This is not a standard pinhole model\nHow would 3DTRL compare with doing some perspective augmentation while training? Isn’t that also learning viewpoint agnostic representations?\nDid you try 3DTRL with the new transformer architectures such as Swin [3]? Does 3DTRL still help with such architectures?\n", "limitations": "The authors have not mentioned any limitations or negative social impact of the work. The clear limitation of the 3DTRL module I see is it is not generic module, and mostly applicable for data with multiple views of the same scene. (i.e. multiview video alignment)\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Authors' motivation for a need to embed 3D information in tokens is clear\n2. Proposed 3DTRL is a plug-and-play module, that can be embedded into any transformer architecture\n3. Authors show experiments on multiple datasets to show the effectiveness of the module\n\nWeakness:\n\nWhile the authors' motivation for the need to embed 3D information into learning visual representation is clear, the proposed approach is not convincing for multiple reasons, \n1. Authors have proposed a pseudo depth estimator and pseudo camera parameter estimator to lift 2D token locations to 3D. While I can understand 2D image information being used to learn depth, in a free-view image analysis scenario, I don’t think camera parameter estimation makes sense. I.e. We don't even know the reference, and how the R,t of one image is related to the R, t of the other image in CIFAR-10? In 3D multi-view datasets there are approaches that directly regress camera poses and scene coordinates ([1], [2]). However, in this case, the authors are trying to embed 3D scene information in model weights, while in free-viewpoint images that don't have a common reference frame, the proposed approach is most likely not valid.\n\n2. The transformers are effective with large amounts of data, and the minor improvement of 3DTRL on the ImageNet-1K dataset shows that transformers already become effective without 3DTRL. I understand that authors have shown a validation set on view-point perturbed data with the perspective transformation of the validation-set of ImageNet-1K, but this is not a valid viewpoint change, and we cannot trust that this will really work in multi-view scenarios.\n\n3. While  I can understand this approach being suitable for multi-view video alignment, since there we are fine-tuning it for each task and 3DTRL can build implicit representation, I cannot see this as a generic module for learning viewpoint-agnostic representation.\n\n4. Also, from the depth-map visualization (Fig. 6), I think it is just learning to pay attention to the primary parts of the class (e.g. for each of the dog images, the eyes, ears, and noses have very low value). I cannot see the depth being learned here.\n\n5. Authors mention that this is a generic plug-and-play (L14, L48), but all the experiments are only with DeiT. More experiments with different architecture are needed to support the claim.\n\nThe notations and math are also unclear at many places:\n1. E.g. in eq (2) authors represent (u,v) as center pixel coordinates. While in eq (3), (u,v) are token 2D locations.\n\n2. The camera model presented in eq (2) is not a standard pinhole camera model. In standard pinhole camera if dn is a depth, then xn = dn*un/c , yn = dn*vn/c , zn=d (considering camera frame and image frame matches at (0,0)) \n\nSome typos/writing mistakes\n\nL55: (u, v) are pixel coordinates: (u, v) are not any pixel coordinates, they are coordinates of the center pixel of the image.\nL104: which is beneficial for later representation learning → which is beneficial later for representation learning (or did you mean, which is beneficial for better representation learning)\nL133: raw → yaw\nFig4: 3DTPL → 3DTRL\n \n\nReferences:\n[1] Li, Xiaotian, Juha Ylioinas, and Juho Kannala. \"Full-Frame Scene Coordinate Regression for Image-Based Localization.\" Robotics: Science and Systems Conference. University of Queensland, 2018.\n\n[2] Brachmann, Eric, and Carsten Rother. \"Learning less is more-6d camera localization via 3d surface regression.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[3] Liu, Ze, et al. \"Swin transformer: Hierarchical vision transformer using shifted windows.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nrW0xbfFwZw", "reviewer": "Reviewer_xnSs", "summary": "This paper presents a method for transformers to upgrade their 2D image inputs to pseudo-3D, and shows quite convincingly that this contributes a performance boost. The basic idea is a layer that goes somewhere in the middle of the transformer: estimate per-token depth and also a camera pose (pitch yaw roll), then unproject coordinates to 3D, encode these coordinates into embeddings, then add these with the existing embeddings, and proceed with the rest of the transformer. This gives performance boosts on a variety of tasks. \n", "questions": "When you use multiple 3DTRL locations (in the supp), do you share the depth and pose estimates across all of them? \n", "limitations": "I think the authors forgot to write a limitations section, but I think they could resolve this easily. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am quite amazed by this paper. The idea is simple and intuitive, it adds negligible computation, it requires no extra supervision, and yet it improves results by a few points on a variety of tasks. The ablation studies indicate that it is indeed the 3d lifting, and not the additional parameters, that leads to the performance benefit. I think this is a significant result and worth publishing. Also the paper is easy to read. \n\nAs for weaknesses: I wish the paper had a more thorough analysis, even if qualitative, to show why exactly this is working. The depth maps of the dogs in Figure 6 are somewhat helpful, but why so few, and why just dogs? I would like to see a random sample of depth maps, to get a feel for the normal behavior. (I found a few more in the supplementary, but for me the 3D visualizations there are less clear than the depth maps.) Also, analysis of the camera poses seems to be completely missing. I may have missed it, but I'm not sure the pitch/yaw/roll estimation was shown to be helping. Maybe the model is only estimating some basic \"intrinsics\", to help with the scaling as the points go to 3D. I think we expect the poses (extrinsics) are doing some canonicalization of the input imagery -- registering them closer to a common pose. Besides adding the basic ablations, it would be great, for example, to get a dataset with pose annotations, do some self-supervised learning, and then show that the estimated camera poses have some mapping (even if weak) to the real pose distribution. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iAONypffKBW", "reviewer": "Reviewer_sPDR", "summary": "This paper proposes 3DTRL, a 3D-aware component that can be plugged into transformer-based supervised/selfsupervised learners. In particular, 3DTRL utilizes the tokens generated by the previous layers, and uses per-token MLP to predict psuedo depth values for each token. Then, a MLP-based camera pose estimator uses all the tokens to predict the euler angles and camera translation. In the end, a 3D embedding is used to generate embeddings for each token given their predicted camera-space coordinates. The following transformer layers will therefore be 3D aware. \nTo support the validity of this proposed module, extensive experiments are conducted. The three main tasks are image classification under prespective transform, video alignment under different viewpoints, and activity classification. The proposed module only induces slight increase in parameter counts and memory usage, but improves the final result by a sigificant margin. ", "questions": "1. I simply find it interesting that imposing a structural prior is already effective with a significant margin. Does the choice of camera pose parameterization affect the results at all? Euler-angles have gimbal locks and therefore induce ambiguity, which might be harmful.\n\n2. The camera pose results visualized in the supp. is quite interesting. Even though it might be a strech, is it possible to compare how well the camera poses improved over the course of training? As argued in the paper, precise depth and camera pose is not required to make good classification decision, but how well shoud they be to make the 3D embedding meaningful?", "limitations": "The authors do not discuss the limitation of the method. It is important to have a up-front discussion of the limitations, and potential bells and whistles. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Originality:\n\nThe proposed module is novel from the reviewer's perspective. It's simple in nature, imposing camera transformations as a structual prior into the transformers. \n\n2.Quality:\n\nI find this paper of good quality. It presents all the necessary details and experiements to support the calim, and addresses the design choices well through ablation studies. \n\n3. Significance:\nI find this paper significant, and hopefully to the community as well. The idea itself is simple, yet proves to be effective. In addition to the applications showed in this paper, it may also be helpful for areas such as unsupervised depth/pose estimations, especially how they could benefit from large image/video datasets without annotations. \n\n4. Clarity:\nI find this paper easy to follow and well written.\n\n\nWeakness:\nThis is not a major issue, but I'm not convinced that the predicted depth/camera poses are meaningful in terms of explaing the image geometrically. The circular tokens in Fig.1 doesn't seem to expain the image, and there's less visualization on the predicted pose. I do not find this very concerning though, since the prior is really weak as they are only structrual in nature. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZvUcLa9Lu2Q", "reviewer": "Reviewer_FjVH", "summary": "The paper proposes 3DTRL, a module in ViT which helps the learning of viewpoint-agnostic representations. After ViT extracts image patches to tokens, 3DTRL estimates a depth for the whole patch, as well as camera extrinsics. Then the features are transformed into 3d tokens, and feed into the next transformer layers. Experiments are conducted on both image classification and multi-view video alignment. 3DTRL improves the performance on more viewpoints. ", "questions": "- Is it possible to evaluate the learned camera extrinsics R and T? At least in simulators the ground truth camera pose should be available.\n- Do you notice any failure modes introduced by 3DTRL? \n- The paper writes the focal length c is set as a constant hyperparameter. What do you finally use? I guess you might use different c for different datasets since c is available for synthetic datasets?", "limitations": "Limitations are not discussed in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- It is a neat idea to add an additional module to transformer so that transformer can learn viewpoint-agnostic features. Pseudo depth estimation on patches makes a lot of sense.\n- My primary concern after reading the idea is whether pseudo depth estimation and camera pose estimation can really learn any meaningful information. Especially, the pseudo depth estimation is just 2 MLP layers. And it is addressed well in Fig 6 and supp Fig 7. I appreciate it!\n\n\nWeaknesses:\n\n- The pseudo-depth prediction seems to have scale ambiguity -- how does the model learn metric depth without supervision? Do you enforce scale-invarance somewhere?\n- The learned camera pose also seems to have ambiguity. Take Fig 7 as an example. It seems multiple camera trajectories are reasonable in both cases. We only care about the relative camera pose so the absolute camera pose does not matter. I'm not sure how networks learn an absolute camera pose in this case. \n- The improvement on large datasets such as ImageNet is relatively small (around 0.2%). It probably because ImageNet is too diverse and does not benefit from viewpoint-agnostic representations. At the same time, it works well on ImageNet-perturbed. However, these perturbed images may have additional cues to learn geometric transformations based on the black boundary.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "401wiYJa0bV", "reviewer": "Reviewer_kBS4", "summary": "This work presents 3DTRL, a plug-and-play module aiming to learn viewpoint-agnostic representations by incorporating 3D geometric constraints in the learning process. The module predicts both the (pseudo) depth estimate and the camera parameters such that it can project the learned tokens into 3D world coordinates. The authors have shown that the module, when used together with visual Transformers, can achieve better performance in a range of visual understanding tasks, including image classification, multi-view video alignment, and cross-view action recognition.", "questions": "Please see the questions in the weaknesses section.", "limitations": "The authors did not explicitly discuss the limitations of the proposed method, but more discussions/evaluations about the model's ability to infer 3D would be needed.\n\nI do not see any potential negative societal impact from this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\nI like the idea of incorporating 3D geometric information into the learning pipeline. The added module is simple to use and only casts negligible overhead regarding the parameter count and computation time but is seemingly effective in boosting the performance of existing Transformer architectures.\n\nThe authors have conducted extensive evaluations on a range of tasks involving images and videos. The proposed module is effective in all evaluated tasks and has consistently improved over the existing baselines. The authors have also shown a range of ablation studies justifying the design of the added module regarding the architecture, the type of 3D information to infer, and the way to incorporate 3D geometric constraints. These are all important lessons that can be helpful for the readers to have a deeper understanding of the method.\n\n\n[Weaknesses]\n\nThe primary concern I have about this paper is that I do not fully understand how well the added module can infer the 3D information.\n\nFor example, the proposed module predicts the depth information and the camera parameters from a single camera, which will for sure have a lot of ambiguities (i.e., different combinations of depth and camera parameters could explain the same visual observation). It is unclear how the module resolves the ambiguities and how such ambiguities factor in the final performance.\n\nI'm also not convinced about the module's 3D inference capability. It is great that the authors show the 3D prediction results in Figure 8 of the supplementary materials. However, the results do not seem to be very satisfying. I know that this is a very hard task, but no matter the size of the object, the distance from the camera to the object, and the distance between the foreground and the background, the 3D prediction results all seem similar without much differences from each other. If the predicted 3D information does not reasonably reflect the underlying 3D contents, it is a bit hard for me to justify its benefit to the downstream task performance.\n\nCorrect me if I am wrong, but the authors seem to predict only the camera extrinsic parameters but ignore the intrinsic parameters. However, the dataset will for sure contain images taken by cameras with different intrinsic parameters, which will greatly impact how the 3D points in the camera frame are projected to the 2D image plane; thus, I'm curious about how much the negligence of the intrinsic parameters influences the final performance. \n\nThe authors have also ablated the model by using different numbers of 3DTRLs in the architecture and have shown improved performance when using multiple modules. It would be thus quite interesting to see how the inferred 3D information changes as the model use different numbers of 3DTRLs.\n\nI'm also not entirely sure how the authors specify the origin of the world coordinates; Or is the origin purely learned in an unconstrained and unsupervised way (i.e., whatever coordinates produced by the neural network)?\n\nDue to the reasons above, I strongly suggest the authors show results in scenarios where we have access to the ground truth 3D information (e.g., simulation or some 3D scanned scenes) and systematically evaluate how well the proposed module can recover the underlying 3D contents.\n\n\n[Minor]\n\nLine 222: Figure 3 should be Table 3, but however, Table 3 currently is a figure. There might be some inconsistencies in the LaTeX code.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept