Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses problems that involve learning a geometric transformation (e.g., rotation) on the input data. Examples of the problem are Abstraction and Reasoning Corpus. The core idea of the proposed method is to inject lattice symmetry prior as soft masks to modulate the attention weights in a transformer. The paper shows that the various lattice symmetry actions can be realized with attention masks (such as translation, rotation, and reflection). The results on ARC tasks show that using the proposed method with infused lattice symmetry prior can perform well (Table 2 and Figure 5) and is sample-efficient. 
strength_and_weaknesses: Strength: 
+ Interesting idea of infusing lattice symmetry prior to a problem into an end-to-end trainable transformer model. 
+ The progress on ARC datasets using deep learning models is promising. 
+ The method is technically sound. The CNN-based mask prediction (Lattice Mask Expert) is interesting and supports compositions of multiple actions (e.g., reflection and rotation).

Weakness:
- From the examples in the paper and supplementary material, it seems that rotation is only 90 degrees. Not sure if this makes the problem trivial to solve (if we know the symmetric group of testing input data).
- No visual examples and results on the cross-modal image registration tasks. I am not familiar with the tasks so it's a bit hard for me to judge the significance of the results and how the results validate the proposed method. 
- I wasn't able to follow some of the details in Section 3.3 and Section 4. But it may be because I don't have the background on related problems. 
clarity,_quality,_novelty_and_reproducibility: Clarity:
- I can get the general idea of the paper but I was not able to follow the details without a relevant background in these problems.

Quality:
- I think the evaluation quality is somewhat weak since there are no real baseline competitors (e.g., neural-symbolic visual reasoning work). The paper only compares with end-to-end trainable models such as CNN and transformers.

Novelty:
- I find infusing these priors using soft attention masks novel and interesting.

Reproducibility:
- The supplementary material provides more implementation details of the architectures and theorem proof. 
summary_of_the_review: I think this is an interesting paper that shows end-to-end trained models can perform well on abstract reasoning tasks. I don't know the practical implication of solving these lattice symmetry problems and how the ideas in the paper could be used to solve some other tasks. The paper did show an image registration task. But it does not provide sufficient details (e.g., are these only translations to rotation and translation? what's the task? what metrics were used to evaluate the success in Table 3?) for me to understand the significance. For registering images from different modalities, one should also try stronger descriptor-matching methods beyond SIFT, e.g., SuperGlue.
Given these concerns, I am thus on the fence about this paper but leaning slightly positive.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, a LATFORMER model is introduced for learning effective attention. It incorporates both lattice geometry and topology priors for learning attention masks. The paper is demonstrated on ARC and on synthetic visual reasoning tasks, showing improved performances compared to standard attention and transformer on these tasks. 
strength_and_weaknesses: Pros:
+ The idea of introducing lattice geometry and topology priors for attention design is interesting
+ The paper is clearly written.

Cons:
- The experiments on ARC and synthetic datasets and tasks are not sufficient to demonstrate the effectiveness and generalization of the design. It would be much more interesting if the method can be demonstrated on main-stream vision and NLP tasks to show the effectiveness of the designed attention mechanism.
- The ablation study should be extended to show the proposed attention mechanism with the lattice geometry and topology priors. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The novelty of the paper is not significant. 
summary_of_the_review: The experiments could be enhanced. More challenging tasks and datasets could be considered to better validate the proposed attention mechanism. The novelty is moderate.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to incorporate lattice priors into attention mask for deep learning. The theoretical results show this prior can be formulated as convolutions on an identity matrix. The proposed approach improves the sample efficiency of the model to solve the ARC tasks (few-shot geometric transformation learning) compared to vanilla transformers. 

strength_and_weaknesses: Strength
* The idea of introducing structural priors in the form of attention masks for geometric transformation tasks is intuitive and well-motivated
* The theoretical results show that such priors can expressed as a convolution on an identity matrix, which motivates the design of LatFormer
* Empirical results show remarkable sample efficiency gains over vanilla transformers in ARC tasks and outperform baselines on real image registration

Weakness
* It’d be valuable to show more experiments on real data. How about showing the results on standard image registration/correspondence learning benchmarks?
* In Table 3, are there other baselines to compare with? I’d imagine a vanilla transformer would be a reasonable one.
* In Figure 5, it seems that the transformer is barely learning the task. I’m curious how many samples would it take for a transformer to learn this task perfectly (how far should we extend the x-axis).



clarity,_quality,_novelty_and_reproducibility: The paper is clear and the ideas appear to be novel. It seems reasonably easy to reproduce. Many experimental details are provided in the appendix.
summary_of_the_review: The idea of the paper is reasonable. Theoretical results and experiments on synthetic data show the proposed approach is superior to vanilla transformer. However, the experiments on real data seem relatively weak and it’d be valuable to see more evidences of LatFormer outperforming transformers and more existing approaches than SIFT/alpha-AMD on image registration benchmarks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on helping deep learning models learn fundamental geometric transformations efficiently. Specifically, LatFormer is proposed to incorporate lattice symmetry biases into attention mechanisms by modulating the attention weights using learned soft masks. Experiments are conducted on both synthetic tasks and image registration tasks to show that the proposed model can generalize better than the same attention modules without masking and Transformers. Besides, LatFormer also shows potential on ARC tasks that incorporates geometric priors. 
strength_and_weaknesses: Strength:

+ The proposed method presents an interesting way to infuse inductive biases and knowledge priors in deep neural networks, which is an important open problem for next step's developments of AI and neural networks. 

+ The idea of using a convolutional neural network to generate soft attention masks to scaled attention weights is reasonable. 

+ Experimental results on synthetic tasks and ARC tasks show that the proposed LatFormer can generalize better and from fewer
examples than transformers and unmasked attention modules. Experiments on image registration task further validates the applicability of LatFormer on natural images.


Concerns:

- LatFormer shows potential of inductive biases and knowledge priors. I'm wondering if LatFormer also performs better than learning transformations invariant representations by applying augmentations? Do we have such studies on the current experimental settings/tasks?

-  For image registration experiments in Section 5.3, how about reuslts of using the baseline Transformer?

- How about the application of the proposed LatFormer to other reasoning tasks, e.g. Physical Reasoning in [1] 

[1] Physical Reasoning Using Dynamics-Aware Models, ICML 2021
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The code is not provided. The experiments on ARC tasks require extra annotations which is not public available. 
summary_of_the_review: This is an interesting paper about learning geometric transformations with deep neural networks, with experimental evidences from both synthetic tasks, visual reasoning tasks as well as applications on natural images.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes LAtFormer, which incorporates lattice geometry and topology priors in attention masks. The proposed architecture is a modification of the standard attention mechanism where attention weights are scaled using soft attention masks generated by a convolution neural net. Experiments on ARC and synthetic visual reasoning tasks show that LatFormer requires 2 orders of magnitude fewer data compared to standard attention and transformers.
strength_and_weaknesses: Strengths:
1. The paper is evaluated on a number of tasks.
2. Results show that Latformer can learn geometric transformations with fewer training data.

Weaknesses:
1. Table 3 only compares the proposed method to alpha-AMD and SIFT which are old methods. Since this is an image registration task, the proposed method should be compared to more recent methods, e.g., COTR: Correspondence transformer for matching across images. ICCV 2021.
2. The paper should also discuss failure modes, limitations of the proposed method and future work.
clarity,_quality,_novelty_and_reproducibility: The paper reads well. The proposed method makes sense. Code is not submitted and hence reproducibility is not guaranteed.
summary_of_the_review: Please see the comments in the boxes above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, the authors propose a new method for learning a geometric transformation to aid problems of abstraction and reasoning. In particular, the authors propose the addition of a lattice symmetry prior to adjust the attention weights of a Transformer. The authors first provide some theoretical results showing proof of existence and use these theoretical results to demonstrate that modifications to a standard attention layer through a mask generated by a convolutional network are sufficient to achieve a geometric prior. The authors demonstrate the efficacy of their results on the ARC and LARC datasets (i.e. synthetic visual benchmarks) providing good performance in a sample efficient manner. In particular, the authors show that their method may achieve comparable results to standard self-attention and Transformers but with two orders of magnitude fewer data.

The reviewers commented positively on the novelty of the idea, the empirical success on the ARC dataset, and the technical soundness of the approach. The reviewers did express some concerns about the generality of the approach, the clarity of some of the sections and the lack of qualitative examples. The two largest concerns though were (1) the lack of strong baselines and (2) the connection to real world problems. 

Addressing (1), the authors responded with additional baselines showcasing the performance of models with attention as well as auto-regressive models (e.g. PixelCNN). In addition, the authors performed notable ablations with data augmentations to see if such a simple suggestion might suffice to match their proposed method. Their proposed method outperformed all of these benchmark comparisons indicating that their method provided non-trivial gains over standard methods. Addressing (2), the authors took the reviewer’s suggestions and benchmarked their method on an image registration/correspondence task. The authors performed additional experiments on image registration which they include in the Appendix. The authors compared their method against SIFT and alpha-AMD. Their method again showcased superior performance.
justification_for_why_not_higher_score: Given the borderline reviews for the paper I took the opportunity to read the paper in depth and see if I can find justification for passing the bar for acceptance at the conference. First of all, I agree with the reviewers comments that strong baselines are warranted and are of central importance to the experiments. I am somewhat concerned though that several experiments including baseline comparisons were presented during the rebuttal period and not provided at the time of submission. There are two reasons for my concerns: (1) not having a complete paper at submission time and doing necessary experiments after submission time presents an unfair advantage with respect to other papers and submissions. (2) new experiments presented during rebuttals can not be as thoroughly vetted as other experiments.

With regard to the issues surfaced in (1) and (2), I have several concerns mirroring several of the reviewer comments:
* Given that the ARC results are central to the paper, it is critical to see more discussion about previous work on the ARC dataset, and how these results compare against previously presented work. I see minimal discussion about previous approaches, and the successes and limitations of the previous methods. This seems critical for placing the quality and strength of the results in context. For instance, there was a [Kaggle competition](https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/) 3 years ago on the ARC dataset which included almost 1000 submissions. How do the numbers presented compare with those reported on their leaderboard? What methods did these submissions use? Did the authors benchmark against these methods? Why or why not?
* I recognize that the image registration experiments are not considered central to the paper by the authors, however showing some potential applicability of the proposed method to real world problems is an important criteria for judging the quality of the proposed methods. That is, a synthetic task serves us a useful purpose in our field to provide a probe or proxy for a more difficult problem that is hard to measure or assess. If the authors are not able to demonstrate the application of their method to a real world problem, then it is incumbent on the authors to provide ready discussion and citations arguing why performing better on the synthetic task (e.g. ARC) would lead to better performance on real world tasks. 
* A discussion about how a synthetic task provides a valuable proxy for real world tasks does not remove the responsibility of authors to attempt to demonstrate the applicability of their method on a real world task. So long as the method may be readily applied to a real world problem, it is incumbent upon the authors to make a good faith effort to test out their method on this task. I believe that several of the reviewers implicitly reflected this sentiment, and would rather see the results on image registration not be demoted in the manuscript. Accordingly, it is important to show solid results on this real world task that this method is amenable to, and if not, be able to provide some explanation for the limitations of the method. Delving into these experiments, I see several items of concern remaining: Namely, the selection of using SIFT as a baseline appears quite dated. SIFT was a mainstay of computer vision before the advent of deep learning and has been superseded by many methods. For instance, in a simple search I just performed I see many methods employing deep learning that were not compared against ([Deep Image Homography Estimation](https://arxiv.org/abs/1606.03798) (2014); [Multi-Temporal Remote Sensing Image Registration Using Deep Convolutional Features](https://ieeexplore.ieee.org/document/8404075) (2018); [COTR: Correspondence transformer for matching across images](https://arxiv.org/abs/2103.14167) (2021)). The selection of the selected baselines need to be better justified and I would like to see the authors compare against more modern methods.

Given these issues, I think more work is needed to properly describe the baselines and experiments the authors investigated and place these in the context of previous work. Additionally, I would suggest that all of these results be incorporated into the main body of the paper. For these reasons, this paper will not be accepted into this conference at this time. I suspect that given all of these recent experiments, this is quite achievable and a reformulated version of this paper would make a good submission to a future machine learning venue.

justification_for_why_not_lower_score: n/a
Decision: Reject