Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces the Channel-consistent LOcal Permutations (CLOP), which randomly swap adjacent features spatially after the last convolution layer in a neural network. Empirical evaluations show that CLOP can improve the generalization of convolutional networks on both supervised and reinforcement learning settings. 
main_review: ## Strengths
- The empirical results are comprehensive with great ablation studies. The presentation of the method and the intuition behind it is clear and easy to follow.

## Weaknesses
- Presentation issues
  - The acronyms for the baselines in Table 2 need to be connected to the paper references.
  - Figure 1: it is unclear to me how many pairs of $(h, w)$ are being sampled from $P$. As far as I could tell, this piece of information is also not provided in the rest of the paper. By looking at Figure 2, it seems likely that the for loop is looping over all spatial location of the feature map. It would be good to get clarification from the authors on this. 

- Weak experimental results
  - Compared to previous baselines, the proposed method is only better on 7 out of 16 Procgen games. IDAAC is quite competitive as it performs the best on 4 of 16 Procgen games. The gap is even smaller when we are looking at the number of tasks that these approaches get top 2 on. IDAAC is top 2 for 9 out of 16 games and CLOP is top 2 for 10 out of 16 games. 
  - The authors mentioned that "Compared to these methods, the CLOP layer offers a direct and easy way to augment the RL agent’s ability to generalize to unseen environments, with the benefit of being entirely complementary with each of them." The combination of the CLOP layer and the previous approach could potentially be quite promising but unfortunately there is no experiment that demonstrates whether CLOP can *actually* bring benefits to the previous approaches. 

## Other comments
- In Section 3 last paragraph, the authors stated that "Our method approaches the augmentation of data by noise injection at the feature level, thus avoiding the computationally costly operations of high-dimensional image transformation in regular data augmentation" - From my understanding, data augmentations usually cost a negligible amount of time compared to running the image through the network itself. 


summary_of_the_review: The empirical evaluations of the paper are thorough and informative. Although the experimental results are not the strongest (marginal improvements over baselines), they do reflect some effectiveness of the approach despite its simplicity. Because of that, I believe the paper deserves a spot in the conference.

=======================================================================
UPDATE:

Thanks the authors for their response. All of my concerns have been thoroughly addressed. Even though I still think that the improvements of CLOP over previous baselines are not that large, the insights provided by additional experiments and empirical analyses can be quite valuable to the community. Because of that, I have raised my score to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work, a novel regularization method for deep neural networks is introduced. By locally swapping dimensions of intermediate feature maps, the authors report generalization improvements in supervised learning and on several reinforcement learning benchmarks. The method is evaluated against several other common regularization techniques and was found to lead to better results.
main_review: STRENGTHS

I want to start by acknowledging that the authors provided a version of the code used in their experiments which was very easy to navigate through and understand.

The main strength of the regularization method introduced in this work is in its simplicity. Swapping dimensions of an intermediate feature map while keeping channel-wise consistency does not require heavily engineered functions  and is potentially applicable besides images to other visual inputs like videos. The only introduced parameter in the method, alpha, is the one that controls the probability of randomly swapping an element of the feature map with a random neighbour. Generalization seems not to be sensitive to the value of alpha as long as it is non-zero, implying the lack of necessity for heavy hyperparameter tuning.

Another important contribution of this work is in the improved generalization performance obtained in a supervised learning as well as a RL setup. In the former, CLOP improves accuracy on two vision dataset. For the latter, the authors validate CLOP on a set of procedurally generated RL environments. Compared to other commonly used regularizations, their method ranked first or second in terms of generalization performance on 10 out of 16 environments.


WEAKNESSES


Throughout the paper, CLOP computational complexity is defined as negligible without any explanation. Does CLOP have to process each element of the intermediate feature map, thus scaling with a quadratic cost assuming a constant cost for a swap operation, and is considered a negligible overhead given that features maps usually have low dimensions?
Without a proper cost analysis it is difficult to judge the tradeoff of using CLOP and I am not in favor of accepting this work without any discussion of the reportedly negligible cost.

I agree with the claim that the type of spatial regularization that CLOP tries to inject works at higher layers of neural networks. However, I’m questioning the robustness of using CLOP across different layers. Someone wishing to use CLOP in an RL setup would greatly benefit from an analysis of its performance across different parts of the model. Did the authors try applying CLOP on a deep layer that is not the last one in the model architecture? 
With evidence of the robustness of the proposed method I am in favor of vouching for the acceptance of the paper.

In this supervised learning results (section 5.1, table 1), CLOP seems to be similarly beneficial when using MNIST as well as Imagenette, with the latter being a visually richer dataset. 
Moreover, Imagenet(te) pictures are iconic, with the target object often clearly visible at the center of the image. For this reason, being able to achieve things like background invariance, possibly induced by CLOP, should be more beneficial for Imagenette compared to MNIST/USPS. This makes me wonder about the scalability of CLOP to richer vision datasets. Did the authors try to validate their results on a larger-scale experiment (e.g. ImageNet or CIFAR100). While this is not a fundamental weakness, I think the paper would benefit by the inclusion of this experiment. On this topic, why is the paper presented mainly as a reinforcement learning regularization method if supervised learning results are also reported?

----

Minor Suggestions or enhancements:


The generalization taxonomy in section 2 is rather confusing, especially given that the paper later focuses on the “simplest” type of generalization among those presented. This is the problem of generalizing within a single MDP and is defined as observational overfitting (OO) at the end of section 3. However, when first introduced quoting Song et al. (2019), OO is defined as the generalization to a distribution over MDPs and, even if presented in the text, it is not clear how the generalization phenomenon studied in this work departs from Song’s formulation. 

In the code, I noticed that the Imagenette experiments used a stochastic flipping function to preprocess the input. This should be mentioned in the paper, possibly in a footnote.

While I appreciate the authors explicitly reporting standard deviation values in their results, I did not find mentions of the number of repeated runs for each experiment until table 6 in the appendix. I suggest the authors briefly report this in the main text or point to the appendix for a better interpretation of the presented results.

In the “observational overfitting in RL paragraph”, OO is mentioned (“Therefore, preventing OO in RL remains the ability etc.)” without being properly defined until later in the text (“Song et al. call the problem of OO etc.”).

There could be more consistency in the choice image encoding: in section 4, an image is described as C x H x W, then in section 5.2 an image is described as 64 x 64 x 3.

Across plots and tables, except for the supervised learning results, the “no regularization” setup is defined as PPO and compares to CLOP and several others regularization/augmentation strategies. I find it clearer to define said setup as something like “no regularization” or “plain setup” or something of the sort.

I was able to grasp the main message behind table 3 but I found the results on the CLOP gap slightly confusing. It might be worth mentioning in the caption how the gap is computed.

summary_of_the_review: Overall, the paper is clearly written and easy to follow and I find the idea behind CLOP simple and effective. However, concerns regarding scalability and robustness of the proposed method should be addressed by the authors. I am happy to raise my score and vote for the inclusion of this work at ICLR provided that authors report evidence of a deeper empirical analysis.


===============

UPDATE: I raised my score from 5 to 8. I think that the analysis conducted by the authors have drastically improved the paper. I believe it meets the criteria to be included in the conference. The main contributions of this work are the introduction of a new regularization method (CLOP) for image-based RL and supervised learning tasks as well as a detailed empirical analysis and evaluation on several standard benchmarks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This submission proposes a simple new regularization technique for convolutional neural networks: "channel-consistent local permutations" (CLOP): at a deep layer in a CNN, it randomly swaps some pixels with their neighbors in the forward pass during training. This is shown to outperform other approaches for regularization, both in supervised learning and in reinforcement learning settings, showing particular benefits for generalizing across levels of procedurally generated games.
main_review: Strengths:
- A novel technique (to my knowledge) for CNN regularization, a topic that is well-studied for many years now
- Strong experimental design and results, spanning supervised and reinforcement learning settings, particularly, generalization across levels.

Weaknesses and clarification questions:

- The supervised learning results are a bit superficial, and don't evaluate against state-of-the-art approaches like Mixup, and on more standard tasks like the full ImageNet task. I do appreciate that that is not the aim of the authors, but since the proposed method itself doesn't have much to do with RL, and the impact of the paper would be broader if it added more thorough supervised learning experiments too, to establish the proposed approach as the state of the art for CNN regularization in general vision tasks.

- Fundamentally, augmentation is a way of expressing known invariances to "label-preserving transformations" in the data. We normally perform augmentation in the input space because this is where we know the invariances. In a latent feature space many layers deep into the CNN, we don't know what is represented. What does it mean to do augment the data here? How do we know that moving the features won't change the optimal action? For example, in a driving scenario, if CLOP moves a pedestrian in a scene from the pavement onto the road, shouldn't the driving agent respond significantly differently? Have the authors observed systematic ill effects from CLOP in some settings?

- How does the layer to which CLOP is applied affect performance? Besides the prescription of applying it to the last convolutional layer, some empirical results addressing this would be useful.

- The writing is sometimes a bit convoluted. The text in Sec 2 Generalization in RL, I found more confusing than illuminating. 
summary_of_the_review: An interesting and neat approach to regularize CNNs, motivated by the need to learn task-relevant representations invariant to background and other distractions for visual RL tasks. Strong results, well-designed experiments, and well presented. I am in favor of accepting this paper, but await the author responses and other reviewers' comments to form a stronger opinion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an augmentation technique: channel-consistent local permutations (CLOP), to help address observational overfitting issue in RL. The proposed method is evaluated on Procgen benchmark and outperforms other methods.
main_review: **Strengths:**

* The proposed technique is simple and effective.
* The paper is easy to follow.

**Weaknesses:**

* The paper aims at generalization in RL. However, the proposed method seems more like a general data augmentation technique for visual input and CNN. The motivations in section 4 are only loosely connected to RL. For example, "the important information is often spatially scarce and very localized" also applies to image classification.
* It would be better to use the same experiment protocol as in the Procgen paper (full distribution of levels for testing instead of 1000, for hard difficulty). Also, the choice of the three environments (Dodgeball, Miner, Chaser) seems arbitrary. In addition, it would be better to include comparison with other methods that also use hard difficulty such as mixreg.
* The results of IDAAC and CLOP in table 2 seem comparable. Though the improvement may be complementary, it is better to justify with experiments (i.e., IDAAC + CLOP)
* To enable direct comparison, it would be better to compute a normalized score, as suggested by the Procgen paper.
* Why table 3 only includes the comparison of CLOP and PPO? The training and testing results for IDAAC / mixreg / UCB-DraC are available in IDAAC paper.
* To make the results more convincing, it would be better to conduct ablation experiments (Figure 5 and Figure 7) on all environments.
summary_of_the_review: Overall, I think the proposed augmentation technique is simple and effective when compared to original PPO and other data augmentation techniques, but it does not significantly outperforms state-of-the-art IDAAC. I have some concerns on its connections to RL and some experiment details. Therefore, at this moment I lean towards rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new regularization scheme, which involves randomly permuting nearby (in terms of spatial locations in the downsized feature tensor) depth blocks. The authors show its effectiveness in supervised learning tasks and establish a SOTA-like result over Procgen generalization, and also provide several empirical ablations and visualizations over the method to understand its inner workings. The paper shows that regularization which involves the locality of objects is particularly effective.
main_review: Pros:
* As the authors have mentioned, this is a very simple regularization method which achieves good empirical results. From the point of view of judging it alongside other image-based regularization techniques (e.g. cutout, data augmentation) and even RL-specific upgrades, this method seems particularly effective.
* The paper is quite comprehensive in its showing of results (over classification, Procgen + baselines, saliency, visualizations), and via ablations over locality of permutations, show that the method does take advantage of the locality of objects in images strongly. I have a good empirical understanding over the method.

Weaknesses:
* One incredibly small detail that I caught (but I think may matter a lot) in Figure 9, Appendix A.2 is the fact that the CLOP layer isn't used in the image torso shared by both the value and policy heads, but rather only the policy head itself. This makes me somewhat suspicious that one hypothesis is maybe CLOP just affecting the policy's action outputs, leading to random actions during training and thus an exploratory behavior/"data augmentation" that's leading to the higher test performance. This is even more confounded because of the separation between forward passes (for data collection) vs backward passes (where the value head is still affected by CLOP since the image encoder's gradient is affected by CLOP via the policy head). Could you please provide an explanation for why you only used CLOP on the policy head, or add a few extra ablations to refute my proposed hypothesis? For example, what if CLOP is applied at the very end of the image encoder torso, but still shared by both heads? It is not a fatal issue if my proposed hypothesis is true, but analyzing this direction would substantially clarify why CLOP is effective. Apologies if I may be overreacting.
 
* Since this is a vision-based method, it is naturally difficult to provide more fine grained conceptual analysis on the CLOP regularization method, and thus this paper might not be as impactful to say, someone in the theoretical field as someone in the computer vision field. From the point of view of computer vision, this is a good contribution, but it seems slightly unsatisfying from a theoretical perspective. The method relies on the inductive bias where objects in images are localized, which is true for a large distribution of images. How would this method work on scenarios where this is not the case, especially ones where data other than RGB is used? What would happen if the data also involves e.g. [1] 2D xy coordinates appended into the depth channel, or standard positional embeddings for images sent to a Vision Transformer [2]? It is not obvious to me if CLOP would work here. I wonder if it may be possible to learn the distribution of random permutations during training instead for these non-RGB scenarios.

[1] https://eng.uber.com/coordconv/

[2] https://arxiv.org/abs/2010.11929
summary_of_the_review: The proposed CLOP method is simple and effective, although I'm afraid it can (to some readers) seem like a (somewhat motivated) "hack", which isn't a huge flaw, but might be limited in impact scope. It is strong to the eyes of an experimentalist, but raises many questions to someone who may be more theoretical.

I am willing to increase the score if the authors answer my main question about the policy head, which made me somewhat alert/suspicious.


EDIT: I have increased my score to an 8 after the rebuttal. The authors have answered my questions thoroughly. Previously, I was concerned that the CLOP method might be a bit too "hacky" (for lack of a better word), but I think the paper's simplicity and strong empirical results may also lead to how we redefine RL generalization. Perhaps Procgen tests too much on "visual generalization" and thus this paper might be providing some evidence for new benchmarks.
review_title: null

Meta-review: This paper presents a novel regularization technique for CNNs based on swapping feature vectors in the final layer. It is demonstrated that this simple technique helps with generalization in supervised learning and RL with image inputs.
Following the author rebuttal, all reviewers agreed that the simplicity of this method and the nice empirical performance it obtains is important to report to the community. In this respect, I agree with the reviewers, and recommend acceptance.

One important issue that came up during the discussion is how much this work is related to RL, and the authors SL experiments helped to put the contribution in a broader context. Indeed, one way to see the results of this work is that if such performance improvement is obtained in the Procgen benchmark with just image-based regularization, perhaps this benchmark is not very suitable for studying generalization in RL (where we expect that more sophisticated techniques would be required). In addition, I can think of RL domains (e.g., Tetris, which was mentioned in the discussion) where I would not expect the proposed method to help. It would be good if the authors discuss these issues in some capacity in their final version.

Please take all reviewer comments into account when preparing the final version.
Decision: Accept (Poster)