Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an approach for inducing invariance to absolute object positions in visual imitation learning policies by determining the policy’s regions of attention in the input image and using a discriminator to ensure the learned visual representations do not encode information about their position. In experiments on abstract block pushing tasks, ProcGen and a real robot block pushing task the introduced regularization leads to better policy performance than a number of previous regularizers.
strength_and_weaknesses: # Strengths
- the approach is well-explained and easy to follow
- it is demonstrated across three different environments, including a real robot
- Fig 2 makes it quite easy to understand the approach
- the discussion of related work is comprehensive

# Weaknesses

There are several major concerns regarding the problem formulation, approach and experimental evaluation:

(A) **limited definition of structural generalization**: The paper defines structural generalization as generalization to scenes with different absolute positions of objects (and thus argues that in order to allow policies to perform structural generalization, we need to remove absolute position information form the learned representations). However, this is a very limited definition of structural generalization: there are many other aspects of such generalization, e.g. generalization to new instances of objects, new distractors, new scene layouts (that also involve relative location changes), more instances of the objects (eg pick up two cups after being trained on picking up one cup). As a result the scoping of the paper seems overly general for the actual invariances that the model introduces. Alternative approaches mentioned in the related work, e.g. the object centric approach by Zhou et al. 2022, can achieve a broader range of the aforementioned structural policy generalization.

(B) **objective can encourage information-less representations**: The introduced objective uses a discriminator to decrease mutual information between the regions of attention of the policy (computed via GradCAM) and the learned representation. There are two ways to reduce this mutual information: by removing position information from the representation or by uniformly attending to all parts of the image (so that the GradCAM attention visualization is uninformative). The latter solution is undesirable and does not align with the intuition of the proposed method, but it seems that the objective does not explicitly encourage the first over the second solution.

(C) **experiments don’t describe how structural generalization is tested**: The description of the experimental evaluation does not explicitly outline how structural generalization is tested. What is the structural difference between the training demonstrations and the test tasks for the different environments? This information is crucial and without it understanding the experimental results is hard. (For example in the MAGICAL environment experiments the demonstrations already use randomized object positions, so it is hard to understand what unseen structural generalization is required at test time)

(D) **missing baselines: random crop data augmentation**: The proposed method is an approach for inducing a certain invariance in the policy (absolute position invariance). As mentioned in the related work, an alternative for inducing invariances is data augmentation. While the experimental evaluation compares to multiple augmentation methods, it surprisingly omits random image cropping. This data augmentation has been shown to be the most effective for visual policy learning in prior work (DrQ, RAD) and also aims to induce positional invariance. At the same time it is substantially easier than the proposed approach. Thus, a comparison to this baseline and a qualitative comparison of the learned representation is crucial.


There is also a number of smaller points of critique:

(E) qualitative analysis does not investigate positional information in representation: the presented qualitative analysis (Fig 4) does not actually investigate the learned representation or the core property the paper tries to induce in it, positional invariance. It merely shows that the policy learns to attend to the relevant objects, but I would expect any functioning policy to show such patterns. Instead, some measure of the mutual information between the learned representation and the absolute object position should be reported and compared to baselines.

(F) dropout model attends to distractor, but not relevant objects: following the previous point, it is unclear to me why the dropout policy in Fig 4 would learn to attend to the distractor. I would expect any policy that properly learns to solve the task to attend to the relevant objects, not the distractor. The main qualitative difference between the baselines and the proposed approach should be in the degree to which the learned representation contains absolute position information. This could suggest that the baselines had some training issues unrelated to positional information.

(G) no ablation studies: the paper does not perform ablation studies of the proposed approach — for example it could be good to ablate the simplified vs full loss function (eq. 3 vs 4) and the introduced target network.

(H) not enough details about the robot experiment: the description of the robot experiment is too short, not enough details are provided eg about the action space of the agent, about why only the workspace camera encoder is regularized or how structural generalization is tested between training and test (see point (C) above).

# Questions

- How would the method perform for tasks with moving cameras where the 2D GradCAM projection might not accurately reflect the 3D position of the attended objects?

- Why does the proposed regularization, which aims to remove absolute object position, make the policy more robust to distractors? This is not intuitively clear to me.

- How would the method perform in tasks where absolute object position is important?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and introduces a novel approach. However, it lacks some important qualities in terms of scoping and experimental evaluation. It also lacks quite a few details important for reproducability, in particular with regards to the robot experiments.
summary_of_the_review: I have a number of concerns with regards to the proposed method and it’s experimental evaluation. In particular, I believe the paper is too broadly scoped for the proposed method, its not clearly explained how the experiments test the desired generalization capabilities and vital baselines are missing. Thus, I cannot recommend acceptance of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for learning 2D visual representations which explicitly discourage the model from memorizing absolute positions of objects on the scene, while still enabling the model to reason about their relative arrangement in order to derive action policies.
strength_and_weaknesses: Strengths:
- interesting approach to solving the problem, and solid demonstration that solving this particular 'structural generalization' problem matters to improving BC policies.
- paper is thorough, with a very good description of the method, and a sensible set of experimental results that provide convincing evidence.
- code is provided
- real-world experiments are provided, which greatly enhances the value of the experimental validation for robotics problems.

Weaknesses:
- The language used in the paper is sometimes getting in the way of clarity. Calling relative position invariance 'structural generalization' and expressing everything under that lens makes things unnecessarily difficult to read at first glance.
[edit post rebuttal: addressed]
- The level of complexity thrown at the problem in order to (partially) solve it is a definite concern: first you need to provide a saliency model, which are notoriously brittle to all but the simplest real-world images, and then you need to learn a complex discriminator model on top of it. This likely makes the approach prohibitive in practice, and means its usefulness is limited to being one purely academic step in addressing the problem.
[edit post-rebuttal: Acknowledging that http://gradcam.cloudcv.org/ was shown to be robust to real-world scenarios in prior works. Validation of its use in real-world downstream tasks is still missing (not a huge concern) and excessive complexity-to-outcomes ratio argument still stands]
- A number of works (https://transporternets.github.io/ comes to mind) bypass the issue of learning position-invariant representations by decoupling the policy representation from the visual representation learning, and having the policy only reason about the 'delta' between visual representations, while still enabling end-to-end differentiability of the architecture. I would have loved to see more discussion of such approaches.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, thorough in its experimentation, and the provided code should make the experiments reproducible.
summary_of_the_review: I am a bit torn. While I can't find any major flaw in the paper, it remains the case that it's the scientific equivalent of using a nuclear bomb to hammer in a nail, and only succeeding in inserting it halfway. There is scientific relevance to such study, but little that would generalize beyond the narrow setting of the problem. My confidence level in this assessment is relatively low.
[edit-post rebuttal: some concerns well-addressed. 5->6]
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a position-invariant regularizer to remove the absolute information from features while preserving the robust relational information. The effectiveness of the proposed method in improving generalization is evaluated on MAGICAL, Procgen benchmark and a real-world robot manipulation problem.
strength_and_weaknesses: Strength:
* The paper is in general well written.
* The idea of leveraging GradCAM to handle the challenges (Sec. 4.2) is interesting and clever. 

Weaknesses and Comments:
* The experiments on Procgen are a little bit limited. It would be better to evaluate the proposed method on all 16 games.
* Apart from BC, does the position-invariant regularizer also work for other (maybe a SOTA one) imitation learning methods?
* Compared to the baselines, how much extra cost (computation, time) does the adversarial training bring?
* I suggest the authors using the metrics recommended in [1] to compare different methods.

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.
clarity,_quality,_novelty_and_reproducibility: * Clarity: Good.
* Quality and Novelty: The idea is interesting and the performance, though I do have some concerns (see review above).
* Reproducibility: The authors provide the source code. I would assume the results are reproducible.
summary_of_the_review: Overall, I think this paper is ok but requires some improvements to meet the acceptance bar.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper had received two negative reviews and one slightly positive one and was  heavily leaning towards the rejection. While the general idea was found interesting, even the slightly positive review had a mixed touch to it and reviewer efvU described the method as overly complex for a simple problem, which was not entirely solved. While the authors could answer and clear up some minor issues, the main problems remained:

- Limited definition of structural generalization and also identical distribution between training and testing. The answers the authors provided on this important point were considered as unconvincing.
- Missing baselines and ablation studies.
- Lack of details.
- Overly complex formulation to address a quite simple problem.

The AC considers that the paper is not yet ready and recommends rejection.
justification_for_why_not_higher_score: Explained in the meta-review
justification_for_why_not_lower_score: -
Decision: Reject