Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper studies the relevance and benefits of using SE(2) equivariant models in Q-learning in robotic problems. The paper identifies the conditions under which the Q function is SE(2) equivariant (Proposition 4.1). The paper, via experimental evaluation on various robotic manipulation problems, demonstrates the sample efficiency of the equivariant models over other models considered in the literature. 
strengths_and_weaknesses: Strengths: The paper studies an important problem. The study of equivariant/invariant models for Q-learning and imposing structure gleaned from the problem, is a great direction to pursue. This paper presents a good step in that direction. Prior works have not used equivariant architectures to model Q functions and thus this paper is novel and makes a worthy contribution. The experimental results are thorough and convincing.

Weaknesses: The paper could have been better written.
summary_of_recommendation: The paper studies the relevance and benefits of using SE(2) equivariant models in Q-learning in robotic problems. The paper identifies the conditions under which the Q function is SE(2) equivariant (Proposition 4.1). The paper, via experimental evaluation on various robotic manipulation problems, demonstrates the sample efficiency of the equivariant models over others considered in the literature. 
Prior works have not used equivariant architectures to model Q functions and thus this paper is novel and makes a worthy contribution. The paper is not very well written.

issues: Following are some of the issues, but a major re-drafting of the paper write-up would help the paper make its case even better. 

- Figure 3 should be placed close to page 2 where it is first referred to. 

- In Sec 3, the paper states “learning policies that are invariant to translation and rotation.” This is misleading and confusing. The policies need to be invariant to translation and rotation to state space is perhaps what they mean.

- The abstract, and the write-up in general, are very verbose. Examples:
** “we identify the conditions under which the optimal Q function is SE(2) equivariant, and equivariant Q learning is therefore applicable.”
** Sentences 3-5 in the abstract.

It would have been good to briefly describe sample efficiency earlier in the Introduction as it is referred to so many times in the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to bake in the equivariance constraints in the state-action space in deep Q-learning methods for robotic manipulation problems. The method is to utilize equivariant convolutions in addition to other symmetries (such as gripper symmetries) and additional state fed into the network. These new q-function architectures are then evaluated on a number of problems and are shown to be better.
strengths_and_weaknesses: Strengths:
1. The paper proposes to actually utilize symmetries in the Q-function training, which is interesting since it can be done by just using a different architecture.
2. The empirical sample-efficiency results seem quite better compared to the baselines on a number of domains.

Weaknesses:
1. The paper does not compare to some obvious baselines such as enforcing these symmetries via data augmentation. While this is discussed in related work, a comparison on at least some domains is needed.
2. I would suggest reorganizing the paper to have a clearly defined method section with a summarized table of various options and then experiments. 
summary_of_recommendation: The idea proposed in the paper is interesting, but I am not sure of the baselines (e.g., data augmentation) and how important the method will be in the long run -- is it likely that we will be able to just use simpler techniques without having to modify the architectures for Q-functions and obtain similar sample efficiency?  
issues: Atleast the data augmentation baseline must be added.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents an architecture for learning rotation and translation equivariant policies in order to learn object manipulation policies more efficiently. The method also provides a channel for non-equivariant information. The experiments are on 6 simulated pick-and-place tasks with real world evaluations. 
strengths_and_weaknesses: Strengths:
- The topic is relevant to CoRL. Increasing learning efficiency is an important problem in robot learning. Structural biases such as rotational and translational equivariance are sensible and applicable to many manipulation problems. 
- The experimental results show the method is able to perform well on a variety of pick and place tasks.
- The soft-equivariant baseline is a nice comparison of data enforced equivariance vs model enforced. Does " augmented three fold for rotations every 90 degrees" match the number of rotations in the model? If not, then having an experiment where they do match would be the best comparison.

Clarity:
- The method is very confusing to read. The architecture is based of off ASR [29], but ASR is barely explained. It would be easier to understand if the equivariant aspect were first explained for a normal FCN policy, and then the ASR was added. My understanding is that with ASR, each dimension of the action space is predicted sequentially. From the video, the policy looks like a simple FCN applied to each rotation of the image, but this is not what I understood from reading the paper. I don't understand how this architecture differs from the one in Form2Fit, which outputs embeddings for each of 20 rotations (I do understand that algorithmically, this paper outputs Q values and Form2Fit outputs embeddings, is that the only difference?).
- The theoretical backing for the method is overwrought for what seems to be in the a fairly simple method, consider cleaning it up to keep only what is necessary. 

Experiments:
- To disentangle the contributions of Q learning vs this approach to equivariance, the comparison to transporter could be made more fair by either evaluating a BC version of the proposed model or a Q-learning method of the transporter networks. 

Question:
- Is the model always applied to images where the gripper is not visible?
summary_of_recommendation: The paper's clarity needs to be improved. It is difficult to figure out exactly what the method is and how it differs from prior work. This also makes the novelty of the method difficult to determine. I find the general direction of the work to be impactful, although the method seems limited to non-dynamic pick and place tasks.


-----------

Update after author response and improved paper
I find the paper clarity improved and the new experiments convincing. I am in favor of accepting this paper.
issues: See above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper aims at exploiting equivariances under SE(2) present in robotics manipulation tasks with spatial state and action spaces in order to improve sample efficiency of image-based deep Q-learning. 
In particular, the paper proposes to exploit the equivariance of the Q-function wrt. simultaneous symmetry transformations in the state and action spaces. For that purpose the authors introduce two new architectures for the network parametrizing the Q-function, which are equivariant wrt SE(2) transformations of (spatial components of) the state and action space.
The authors compare the performance of their proposed architectures to several baselines, which either do not take into account equivariances or try to learn these via data augmentation, in several robotics manipulation tasks in simulation. Furthermore, the authors demonstrate the applicabilty of their approach in a real world setup.  
strengths_and_weaknesses: Strenghts: 
The paper is well written and mathematically sound. 
It clearly outlines the pre-conditions under which the considered equivariance properties apply. 
It proposes two novel architectures that directly encode the SE(2) equivariance into the structure of the network of the Q-function.
The experiments  are well chosen and results are properly documented and discussed. 

Weaknesses:
see detailed issues below
summary_of_recommendation: The paper discusses an the topic of equivariances, which is very important for robotics research and applications. The proposed approach to encode the equivariance directly into the network of the Q-function is promising and presumably easy to implement and might therefore have an impact on robotics research. 
The paper is well written, mathematically sound and the experiments are well chosen and properly discussed. 
There are some minor issues, which the authors should address. Inspite of these I would recommend the paper for acceptance to CORL, since the positive aspects clearly outweigh the negatives. 
issues: - line 29: The authors state that other methods cited in previous lines improve generalization but "generally make policy learning harder". I think this statement is too strong and not properly justified. 

- experiments: 
-- I wonder how the angles for the outpur of basesline 1 (line 216) are chosen. Do they have the same granularity and respect the gripper symmetry as the equivariant method (corresponding to a choice [0, 30, 60, 90, 120, 150])? The authors should state that. Also, if the authors chose a coarsere granularity (i.e. [0, 60, 120, 180, 240, 320]) I believe the comparison is not fair.  Similarly I wonder, why the authors chose to rotate the input of the baseline 2) only in terms of 90 degree, leading to a much coarser angel granularity. I believe this comparison is not really fair. 
-- The authors do not discuss the runtime of their approach. How much does the runtime suffer from the sequential approach the authors propose in section 4.3?
--  I think, that the assumed SE(2) equivariance may be slightly violated in the considered experiments due to the fact that the robot is fixed to the workspace at some point. Some state-action pairs might therefore be more difficult to attain for the robot than an equivariant pair. Did the authors observe such effects?
-- section 4.4: The architecture and approximation mentioned in lines 339ff is not sufficiently discussed. The reader has no chance to understand it without the appendix. I think that such important "details" should be part of the main paper and not be transferred to the appendix, otherwise a page limit would become meaningless.
review_title: null

Meta-review: The authors propose in this paper a Q function neural architecture that is invariant to rotations. The consensus among reviewers is the following.

Pros:
- The problem is important and applicable to many tasks.
- The robotics task is well chosen and results are encouraging.

Cons:
- Concerns regarding the fairness of the experimental setup, especially the data augmentation baselines (R-H (Reviewer HHEb), R-G, R-B).
- The Q function architecture beyond the equivarient convolutions not well explained (R-G).

The authors are encouraged to actively engage with the reviewers and revise their submission if they see fit. Concerns regarding the fairness of the comparisons w.r.t. data augmentation (R-H, R-G, R-B) are major and should be properly addressed so as to improve the quality of the submission. Authors should justify their choice of a more limited set of rotations for the data augmentation baseline (R-G, R-B) and ideally provide a baseline with matching settings. As suggested by R-3, a clearer definition of sample efficiency in the related work section (e.g. when discussing data augmentation), and experiments that support the claims made in that section---that appear sometimes too strong (R-B), would greatly improve the submission.

I thank the authors for their updates and the new baseline results. The rebuttal addressed most of the reviewers' concern and I thus recommend the paper for acceptance. If possible, in the final version the authors could also discuss the computational complexity, especially since data augmentation alternatives would not increase computational cost at test time.
Decision: Accept (Poster)