Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method for transformers to upgrade their 2D image inputs to pseudo-3D, and shows quite convincingly that this contributes a performance boost. The basic idea is a layer that goes somewhere in the middle of the transformer: estimate per-token depth and also a camera pose (pitch yaw roll), then unproject coordinates to 3D, encode these coordinates into embeddings, then add these with the existing embeddings, and proceed with the rest of the transformer. This gives performance boosts on a variety of tasks. 

strengths_and_weaknesses: I am quite amazed by this paper. The idea is simple and intuitive, it adds negligible computation, it requires no extra supervision, and yet it improves results by a few points on a variety of tasks. The ablation studies indicate that it is indeed the 3d lifting, and not the additional parameters, that leads to the performance benefit. I think this is a significant result and worth publishing. Also the paper is easy to read. 

As for weaknesses: I wish the paper had a more thorough analysis, even if qualitative, to show why exactly this is working. The depth maps of the dogs in Figure 6 are somewhat helpful, but why so few, and why just dogs? I would like to see a random sample of depth maps, to get a feel for the normal behavior. (I found a few more in the supplementary, but for me the 3D visualizations there are less clear than the depth maps.) Also, analysis of the camera poses seems to be completely missing. I may have missed it, but I'm not sure the pitch/yaw/roll estimation was shown to be helping. Maybe the model is only estimating some basic "intrinsics", to help with the scaling as the points go to 3D. I think we expect the poses (extrinsics) are doing some canonicalization of the input imagery -- registering them closer to a common pose. Besides adding the basic ablations, it would be great, for example, to get a dataset with pose annotations, do some self-supervised learning, and then show that the estimated camera poses have some mapping (even if weak) to the real pose distribution. 

questions: When you use multiple 3DTRL locations (in the supp), do you share the depth and pose estimates across all of them? 

limitations: I think the authors forgot to write a limitations section, but I think they could resolve this easily.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes 3DTRL, a module in ViT which helps the learning of viewpoint-agnostic representations. After ViT extracts image patches to tokens, 3DTRL estimates a depth for the whole patch, as well as camera extrinsics. Then the features are transformed into 3d tokens, and feed into the next transformer layers. Experiments are conducted on both image classification and multi-view video alignment. 3DTRL improves the performance on more viewpoints. 
strengths_and_weaknesses: Strengths:

- It is a neat idea to add an additional module to transformer so that transformer can learn viewpoint-agnostic features. Pseudo depth estimation on patches makes a lot of sense.
- My primary concern after reading the idea is whether pseudo depth estimation and camera pose estimation can really learn any meaningful information. Especially, the pseudo depth estimation is just 2 MLP layers. And it is addressed well in Fig 6 and supp Fig 7. I appreciate it!


Weaknesses:

- The pseudo-depth prediction seems to have scale ambiguity -- how does the model learn metric depth without supervision? Do you enforce scale-invarance somewhere?
- The learned camera pose also seems to have ambiguity. Take Fig 7 as an example. It seems multiple camera trajectories are reasonable in both cases. We only care about the relative camera pose so the absolute camera pose does not matter. I'm not sure how networks learn an absolute camera pose in this case. 
- The improvement on large datasets such as ImageNet is relatively small (around 0.2%). It probably because ImageNet is too diverse and does not benefit from viewpoint-agnostic representations. At the same time, it works well on ImageNet-perturbed. However, these perturbed images may have additional cues to learn geometric transformations based on the black boundary.

questions: - Is it possible to evaluate the learned camera extrinsics R and T? At least in simulators the ground truth camera pose should be available.
- Do you notice any failure modes introduced by 3DTRL? 
- The paper writes the focal length c is set as a constant hyperparameter. What do you finally use? I guess you might use different c for different datasets since c is available for synthetic datasets?
limitations: Limitations are not discussed in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes 3DTRL, a 3D-aware component that can be plugged into transformer-based supervised/selfsupervised learners. In particular, 3DTRL utilizes the tokens generated by the previous layers, and uses per-token MLP to predict psuedo depth values for each token. Then, a MLP-based camera pose estimator uses all the tokens to predict the euler angles and camera translation. In the end, a 3D embedding is used to generate embeddings for each token given their predicted camera-space coordinates. The following transformer layers will therefore be 3D aware. 
To support the validity of this proposed module, extensive experiments are conducted. The three main tasks are image classification under prespective transform, video alignment under different viewpoints, and activity classification. The proposed module only induces slight increase in parameter counts and memory usage, but improves the final result by a sigificant margin. 
strengths_and_weaknesses: Strengths
1. Originality:

The proposed module is novel from the reviewer's perspective. It's simple in nature, imposing camera transformations as a structual prior into the transformers. 

2.Quality:

I find this paper of good quality. It presents all the necessary details and experiements to support the calim, and addresses the design choices well through ablation studies. 

3. Significance:
I find this paper significant, and hopefully to the community as well. The idea itself is simple, yet proves to be effective. In addition to the applications showed in this paper, it may also be helpful for areas such as unsupervised depth/pose estimations, especially how they could benefit from large image/video datasets without annotations. 

4. Clarity:
I find this paper easy to follow and well written.


Weakness:
This is not a major issue, but I'm not convinced that the predicted depth/camera poses are meaningful in terms of explaing the image geometrically. The circular tokens in Fig.1 doesn't seem to expain the image, and there's less visualization on the predicted pose. I do not find this very concerning though, since the prior is really weak as they are only structrual in nature. 
questions: 1. I simply find it interesting that imposing a structural prior is already effective with a significant margin. Does the choice of camera pose parameterization affect the results at all? Euler-angles have gimbal locks and therefore induce ambiguity, which might be harmful.

2. The camera pose results visualized in the supp. is quite interesting. Even though it might be a strech, is it possible to compare how well the camera poses improved over the course of training? As argued in the paper, precise depth and camera pose is not required to make good classification decision, but how well shoud they be to make the 3D embedding meaningful?
limitations: The authors do not discuss the limitation of the method. It is important to have a up-front discussion of the limitations, and potential bells and whistles.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors claim that deep learning-based models applied to 2D images lack 3D information that is easily understood by human vision. In turn, the authors focus on transformer-based architectures. The authors propose a 3D Token Representation Layer (3DTRL), a plug-and-play module for transformer-based architectures to learn viewpoint-agnostic representation for the visual data. Authors do this by lifting 2D token locations to 3D by a pseudo depth estimator, and a pseudo camera parameter estimator. Through experiments on Image Classification datasets and multi-view video alignment datasets, authors show the effectiveness of 3DTRL modules being robust to viewpoint changes.

strengths_and_weaknesses: Strengths:

1. Authors' motivation for a need to embed 3D information in tokens is clear
2. Proposed 3DTRL is a plug-and-play module, that can be embedded into any transformer architecture
3. Authors show experiments on multiple datasets to show the effectiveness of the module

Weakness:

While the authors' motivation for the need to embed 3D information into learning visual representation is clear, the proposed approach is not convincing for multiple reasons, 
1. Authors have proposed a pseudo depth estimator and pseudo camera parameter estimator to lift 2D token locations to 3D. While I can understand 2D image information being used to learn depth, in a free-view image analysis scenario, I don’t think camera parameter estimation makes sense. I.e. We don't even know the reference, and how the R,t of one image is related to the R, t of the other image in CIFAR-10? In 3D multi-view datasets there are approaches that directly regress camera poses and scene coordinates ([1], [2]). However, in this case, the authors are trying to embed 3D scene information in model weights, while in free-viewpoint images that don't have a common reference frame, the proposed approach is most likely not valid.

2. The transformers are effective with large amounts of data, and the minor improvement of 3DTRL on the ImageNet-1K dataset shows that transformers already become effective without 3DTRL. I understand that authors have shown a validation set on view-point perturbed data with the perspective transformation of the validation-set of ImageNet-1K, but this is not a valid viewpoint change, and we cannot trust that this will really work in multi-view scenarios.

3. While  I can understand this approach being suitable for multi-view video alignment, since there we are fine-tuning it for each task and 3DTRL can build implicit representation, I cannot see this as a generic module for learning viewpoint-agnostic representation.

4. Also, from the depth-map visualization (Fig. 6), I think it is just learning to pay attention to the primary parts of the class (e.g. for each of the dog images, the eyes, ears, and noses have very low value). I cannot see the depth being learned here.

5. Authors mention that this is a generic plug-and-play (L14, L48), but all the experiments are only with DeiT. More experiments with different architecture are needed to support the claim.

The notations and math are also unclear at many places:
1. E.g. in eq (2) authors represent (u,v) as center pixel coordinates. While in eq (3), (u,v) are token 2D locations.

2. The camera model presented in eq (2) is not a standard pinhole camera model. In standard pinhole camera if dn is a depth, then xn = dn*un/c , yn = dn*vn/c , zn=d (considering camera frame and image frame matches at (0,0)) 

Some typos/writing mistakes

L55: (u, v) are pixel coordinates: (u, v) are not any pixel coordinates, they are coordinates of the center pixel of the image.
L104: which is beneficial for later representation learning → which is beneficial later for representation learning (or did you mean, which is beneficial for better representation learning)
L133: raw → yaw
Fig4: 3DTPL → 3DTRL
 

References:
[1] Li, Xiaotian, Juha Ylioinas, and Juho Kannala. "Full-Frame Scene Coordinate Regression for Image-Based Localization." Robotics: Science and Systems Conference. University of Queensland, 2018.

[2] Brachmann, Eric, and Carsten Rother. "Learning less is more-6d camera localization via 3d surface regression." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[3] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
questions: How did you come up with eq (3)? This is not a standard pinhole model
How would 3DTRL compare with doing some perspective augmentation while training? Isn’t that also learning viewpoint agnostic representations?
Did you try 3DTRL with the new transformer architectures such as Swin [3]? Does 3DTRL still help with such architectures?

limitations: The authors have not mentioned any limitations or negative social impact of the work. The clear limitation of the 3DTRL module I see is it is not generic module, and mostly applicable for data with multiple views of the same scene. (i.e. multiview video alignment)
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work presents 3DTRL, a plug-and-play module aiming to learn viewpoint-agnostic representations by incorporating 3D geometric constraints in the learning process. The module predicts both the (pseudo) depth estimate and the camera parameters such that it can project the learned tokens into 3D world coordinates. The authors have shown that the module, when used together with visual Transformers, can achieve better performance in a range of visual understanding tasks, including image classification, multi-view video alignment, and cross-view action recognition.
strengths_and_weaknesses: [Strengths]

I like the idea of incorporating 3D geometric information into the learning pipeline. The added module is simple to use and only casts negligible overhead regarding the parameter count and computation time but is seemingly effective in boosting the performance of existing Transformer architectures.

The authors have conducted extensive evaluations on a range of tasks involving images and videos. The proposed module is effective in all evaluated tasks and has consistently improved over the existing baselines. The authors have also shown a range of ablation studies justifying the design of the added module regarding the architecture, the type of 3D information to infer, and the way to incorporate 3D geometric constraints. These are all important lessons that can be helpful for the readers to have a deeper understanding of the method.


[Weaknesses]

The primary concern I have about this paper is that I do not fully understand how well the added module can infer the 3D information.

For example, the proposed module predicts the depth information and the camera parameters from a single camera, which will for sure have a lot of ambiguities (i.e., different combinations of depth and camera parameters could explain the same visual observation). It is unclear how the module resolves the ambiguities and how such ambiguities factor in the final performance.

I'm also not convinced about the module's 3D inference capability. It is great that the authors show the 3D prediction results in Figure 8 of the supplementary materials. However, the results do not seem to be very satisfying. I know that this is a very hard task, but no matter the size of the object, the distance from the camera to the object, and the distance between the foreground and the background, the 3D prediction results all seem similar without much differences from each other. If the predicted 3D information does not reasonably reflect the underlying 3D contents, it is a bit hard for me to justify its benefit to the downstream task performance.

Correct me if I am wrong, but the authors seem to predict only the camera extrinsic parameters but ignore the intrinsic parameters. However, the dataset will for sure contain images taken by cameras with different intrinsic parameters, which will greatly impact how the 3D points in the camera frame are projected to the 2D image plane; thus, I'm curious about how much the negligence of the intrinsic parameters influences the final performance. 

The authors have also ablated the model by using different numbers of 3DTRLs in the architecture and have shown improved performance when using multiple modules. It would be thus quite interesting to see how the inferred 3D information changes as the model use different numbers of 3DTRLs.

I'm also not entirely sure how the authors specify the origin of the world coordinates; Or is the origin purely learned in an unconstrained and unsupervised way (i.e., whatever coordinates produced by the neural network)?

Due to the reasons above, I strongly suggest the authors show results in scenarios where we have access to the ground truth 3D information (e.g., simulation or some 3D scanned scenes) and systematically evaluate how well the proposed module can recover the underlying 3D contents.


[Minor]

Line 222: Figure 3 should be Table 3, but however, Table 3 currently is a figure. There might be some inconsistencies in the LaTeX code.
questions: Please see the questions in the weaknesses section.
limitations: The authors did not explicitly discuss the limitations of the proposed method, but more discussions/evaluations about the model's ability to infer 3D would be needed.

I do not see any potential negative societal impact from this work.
review_title: null

Meta-review: metareview: This paper presents a method for transformers to upgrade the 2D image input to pseudo-3D. It proposes a neural layer that estimates per-token depth and also a camera pose (pitch yaw roll), then unproject token coordinates to 3D, encodes these coordinates into embeddings, then adds these with the existing embeddings, and proceeds with the rest of the transformer. This gives performance boosts on a variety of tasks, such as video alignment.
The reviewers raised concerns regarding the depth maps inferred looking more like saliency maps, but also, the depth scale ambiguity, and the ambiguity of absolute camera pose inference. The rebuttal submitted by the authors included additional results that showed that the inferred depth maps and camera poses correlated with the correct ones. All reviewers appreciated the additional experiments contributed by the authors, and suggested them to be included to the main paper. 

confidence: Less certain
Decision: Accept