Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper target the task of automatically determining the best augmentation method to obtain improved accuracy. While the previous related studies focus on the image-level augmentation and ignore the semantic information of the augmented images, the proposed algorithm augments the grid-wise patches of the given input with the preserved semantic information. To overcome the enlarged number of combinations to consider all the patches, the algorithm utilizes the MARL algorithm with the unified reward function. By MARL algorithm, the number of parameters can be reduced and the training speed can be much improved, compared to the previous auto-augmentation methods. Through the image classification and fine-grained image recognition tasks, the proposed algorithm was validated, and it shows the state-of-the-art performance among the compared methods.
main_review: Strengths
+ It seems effective to utilize the MARL algorithm to reduce the computational complexity for the multiple patches.
+ As a result, the computation cost of the proposed algorithm becomes much reduced when it is compared to the previous auto-augmentation algorithms. Furthermore, the algorithm can select the optimal combinations at every epoch without any additional computations, which can improve its generality across the various tasks.

Weaknesses & Questions
- About state and observation. In the proposed algorithm, the state and the observation are obtained by the deep feature from the trained model. My question is the image to obtain the deep feature. When the image is the original image before the determined augmentation, the state and the observation are not affected by the selected action. Then, the best action can be determined just by one simple simulation, and no reinforcement learning algorithm is necessary to find the optimal augmentation. If the augmented images are fed into the model again, the computation cost cannot be reduced as shown in the experiments. What is the detailed sequence to obtain the deep feature?
- Since the deep feature is utilized, its invariant property can affect the determination of the optimal augmentation. In other words, when the trained model is robust to the translation shift, the deep features from the model would be similar even after various translation shifts. Then, the training loss becomes also similar, which results in no selection of translation augmentation. This means that the algorithm would select only the augmentation method that is not considered in the initial model, and this can work as one of the biased optimizations for the optimal augmentation methods. Is there any analysis about the robustness of the initial model for the deep features?
- In the experiments, the proposed algorithm is only compared with the out-of-date algorithms, so its performance cannot be validated well. I recommend the authors add the state-of-the-art studies related to auto-augmentation and the learning-based augmentation algorithms, which include Saliency-mix[1] and Co-mixup[2].
- What would be the limitation of the proposed algorithm?

[1] Uddin et al. "SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization", ICLR2021
[2] Kim et al. "Co-Mixup: Saliency Guided Joint Mixup with Supermod-ular  Diversity", ICLR2021

summary_of_the_review: As I said above, I have some questions about the necessity of the reinforcement learning-based algorithm in the framework. In addition, the comparison with the state-of-the-art algorithms should be added for the validation. I will change my score when I solve the above issues.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a fine-grained automated data augmentation approach, Patch AutoAugment (PAA), which tries to increase diversity in local regions by divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. The proposed PAA considers the task as a multi-agent reinforcement learning problem,  and adopt a multi-agent reinforcement learning
algorithm to automatically search for the optimal augmentation policies by considering the contextual relationship between the patches. They verify the proposed method on many classification and fine-grained recognition dataset(CIFAR10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Aircraft). The experiments show a good result and visualization results provide some insights that the PAA  help the target network to localize more class-related cues.

main_review: Over all, this paper is well written and technically sound, the experimental evaluation is comprehensive and the results are compelling, but I have several concerns:
1. The most important contribution of this paper in my mind is the new search space focus on image patches for data augmentation. It seems like an incremental job relative to mixup series or cutout series.
2. The proposed PPA formulate the task as a multi-agent reinforcement learning problem, and the results show it can find better policy than random, but [1] demonstrate that only use reinforcement learning can not get better policy than random baseline. The two experimental results are contrary.
3.The calimed SOTA results is inappropriate, methods with better experimental results were not compared. For example, the paper  compare with AdvAA on the  computational cost, but not coapre on the accuracy.
4.Another question is that recent work [2] with careful designed augmentation policy and advanced training strategy can get much better results, It is better to compare with it to give more insights.

[1] Cubuk E D, Zoph B, Shlens J, et al. Randaugment: Practical automated data augmentation with a reduced search space
[2] Wightman R, Touvron H, Jégou H. ResNet strikes back: An improved training procedure in timm
summary_of_the_review: According to above analysis and concerns, I suggest borderline. If the author can solve my concerns, I will consider change the score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:   The paper proposes an evolution of the traditional
  pipeline of image data augmentation used to reduce ML
  model overfitting.

  Instead of applying transformations such as shear,
  rotate, CutOut, etc. at the image level, the proposed
  technique divides the images into a fixed grid and
  applies a potentially different transformation to each
  cell. The problem of selecting a transform for each cell
  is cast as a multi-agent RL (MARL) task, and the agents
  learn as the main network trains within a (multi-agent)
  Advantage Actor Critic framework.

  The agents use a shared reward mechanism, with the reward
  defined as the difference between the loss on the
  augmented sample and the original loss.

  The regular grid is fixed for a dataset and
  hyperparameters like the magnitude of the augmentations
  follow a fixed schedule; the agents only pick _which_
  augmentation to apply on a patch.

  Experiments performed on CIFAR-{10,100}, ImageNet,
  CUB-200-2011, Stanford Cars, and FGVC-Aircraft show
  relatively small but very consistent improvements in
  terms of image classification accuracy. Different design
  choices (MARL vs. single-agent vs. random, grid size,
  etc.) are ablated and discussed in detail.

main_review: ## Strengths

  - S1: Ablation studies and theoretical explanations
    motivate the choice for multi-agent reinforcement
    learning reasonably well.
  - S2: The efficiency of the proposed method is
    investigated very clearly in Section 4.3. It seems
    considerably more efficient than many other
    learning-based DA methods.
  - S3: The patch-wise data augmentation is a powerful idea
    which can spark a large body of interesting follow-up
    research.


## Limitations

  - L1: While this is not the main focus of the paper, the
    empirical improvement over RL baselines like Fast
    AutoAugment and non-RL baselines like RandAugment is
    relatively small.
  - L2: (Minor) While multiple datasets are used for
    evaluation, only two Resnet variants are investigated
    for ImageNet. It would be nice to see an evaluation of
    the proposed data augmentation procedure on other NN
    architectures as well on ImageNet, but I also
    understand that the computational costs of this can be
    quite large, so it may not always be possible.
  - L3: (Minor) The agents rely on a pre-trained ResNet to
    produce features to be used as observations.


## Suggestions (Per-Section)

### Related Work

  - This is minor, but it may be worth briefly mentioning
    [hong2019patch]. While their work doesn't do patch
    level data augmentation, and instead "patches" objects
    from an existing bank onto training samples, it still
    has connections to things like CutMix that make it
    relevant.
  - Some recent papers on automatic augmentation are
    missing and would be worth discussing. This includes
    [DADA], [DDAS], [hendrycks2019augmix] and
    [mueller2021trivialAugment]. In particular, Mueller and
    Hutter's paper highlights an interesting limitation of
    their otherwise robust baseline, which is that it
    doesn't work too well for object detection, an area
    where the current method may have the upper hand!
  - Similarly, there are some further papers, such as […]
    that perform a kind of "local" (though not strictly
    patch-based) data augmentation by "pasting in" objects
    into the 3D point cloud.


### Method

  - "directly applying reinforcement learning (RL)
    algorithm to …" => "algorithms"
  - "In MARL configuration" => "In the MARL configuration"
  - Towards the end of Section 3.2, it may make more sense
    to replace the "Until all patches" to "Once all
    patches…" or maybe "After".
  - Q1: Can an agent output more than one augmentation for
    a patch? Would it be possible for a patch to, e.g., get
    rotated AND colorized?
  - Q2: Given the more "local" nature of this augmentation
    method, have you considered evaluating it on more
    "local" tasks, such as object detection and semantic
    segmentation. Perhaps it could make a bigger impact
    there.
  - Q3: Given that the parameters of actor networks of all
    agents are shared (see above Eq. (5)), does that mean
    that each actor's policy is identical, and the only
    difference between the actors is what observation each
    one gets (the local patch features)?


### Experiments

  - At the start of Section 4.1, "Exactly, we focus" =>
    "Specifically, we focus"
  - In "Policy Visualization", "important levels" ->
    "importance levels"
  - In Section 4.3 I would say the "main methods for
    reducing the computational cost", since what you next
    describe are methods for reducing the cost, not what
    one would typically call reasons.
  - Table 3 is a little hard to follow because of the
    different hardware employed by each column. After
    staring at it for a while it's becoming a bit clearer,
    but at first glance it was a bit confusing.
  - Q4: Do the times from Table 3 include the inference
    time required to compute the agent state and
    observations with the auxiliary ResNet-18?


### Conclusion

  - The parallels to vision transformers are very
    interesting and help give the reader a broader
    perspective on possible applications of the current
    paper.
  - (Minor) The word "Furthermore" is repeated for two
    almost consecutive sentences. Using something like
    "additionally" to start one of the sentences can
    improve the flow of the paragraph a little.


## References

  - [hong2019patch]: Hong, Sungeun, Sungil Kang, and
    Donghyeon Cho. "Patch-level
  augmentation for object detection in aerial images."
  Proceedings of the IEEE/CVF International Conference on
  Computer Vision Workshops. 2019.
  - [cubuk2020randaugment]: Cubuk, Ekin D., et al.
    "Randaugment: Practical automated
  data augmentation with a reduced search space."
  Proceedings of the IEEE/CVF Conference on Computer Vision
  and Pattern Recognition Workshops. 2020.
  - [DDAS] Liu, Aoming, et al. "Direct Differentiable
    Augmentation Search." arXiv preprint arXiv:2104.04282
    (2021).
  - [DADA] Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy
    M. Hospedales, Neil Martin Robertson, and Yongxing
    Yang. DADA: differentiable automatic data augmentation.
    CoRR, abs/2003.03780, 2020.
  - [mueller2021trivialAugment]: Müller, Samuel G., and
    Frank Hutter. "TrivialAugment: Tuning-free Yet
    State-of-the-Art Data Augmentation." ICCV 2021.
  - [hendrycks2019augmix]: Hendrycks, Dan, et al. "Augmix:
    A simple data processing method to improve robustness
    and uncertainty." arXiv preprint arXiv:1912.02781
    (2019).

summary_of_the_review:   The paper proposes a new way of looking at the classic
  task of image augmentation for classification. Instead of
  learning an image-level augmentation policy using RL, the
  proposed approach uses multi-agent RL to learn
  patch-specific augmentations. While the benefit of
  multi-agent RL is relatively small, it seems significant
  and the general idea of learning to apply different
  augmentations to different parts of the image is
  interesting and, to the best of my knowledge, new. The
  idea to apply image augmentations at a local level is the
  core contribution of this paper, in my opinion.

  While it would be interesting to see how this technique
  works on tasks which, unlike image-level classification,
  are themselves more "local", such as semantic
  segmentation or detection, overall the proposed approach
  is evaluated thoroughly.

  The writing is clear and while the authors don't
  explicitly mention an intent to publish their source
  code, it should be possible to implement the proposed
  approach just based on the description from the paper. As
  such, I recommend accepting the paper to ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an automatic data augmentation approach. Different from existing works, they proposed to augment patches in the image rather than the whole image. The approach is formulated as a multi-agent reinforcement learning problem. They empirically show the effectiveness of their approach across several image classification datasets.
main_review: ### Strengths:
- 1.	The paper conducted a comprehensive evaluation of their approach over several datasets. They achieve gains over the compared baseline method validating the efficacy of their proposed approach.
- 2.	The approach considers patched based augmentation, which is sufficiently different/novel from existing work.

### Weaknesses:
- 3.	The motivation of the approach is somewhat lacking. For example, it is unclear why MARL on patches is much faster. I think the paper could benefit from motivating more on why MARL and what is its benefit with respect to the task. 
- 4.	The paper did not reference or compared to a key related work, “Differentiable automatic data augmentation” [A]. Based on its relevance, the paper should discuss and contrast with this work, as they also have minimal search time. Specifically, comparison in Table 2 and Table 3 would strengthen the paper.
- 5.	How is the model architecture of the actor/critic network determined? Also, what ranges of hyperparameters for MADDPG are considered? I saw the hyperparameters for the target model but not MADDPG. RL is known to be sensitive to hyperparameters, I wonder if the authors will release the code and pre-trained models for reproducibility. 
- 6.	Ablation with independent agents as a baseline would be interesting. It would further motivate the use of MADDPG and the multi-agent setting.
 
### References
[A] Li, Yonggang, et al. "Differentiable automatic data augmentation.", Proc. ECCV, 2020.



summary_of_the_review: The main concern with the paper is with the motivation and missing reference/comparison with a key existing work. 
Currently, I recommend a weak reject.
review_title: null

Meta-review: The paper proposed an approach to search image augmentation policies. The paper formulates this problem as a cooperative multi-agent decision-making problem, which is interesting. The paper received 3 borderline accept and 1 borderline reject ratings. The reviewers originally had multiple concerns regarding the necessity of RL-based approach, lacking references, and additional experiments, and the authors responded to some of the concerns of the reviewers reasonably. However, none of the reviewers ended up strongly supporting the paper, staying with their ratings.

The RL formulation of the problem is interesting, but it requires multiple rounds of the target network training due to its nature (i.e., it is not an end-to-end approach). The paper misses some details on how exactly the patch-wise RL-based augmentation works and it requires additional hyperparameters for the selection of patch size and shape. It is also unclear how this RL-based method is conceptually superior to previous augmentation approaches and the empirical results are not strong enough, as some of the reviewers also pointed out.

Although the paper has interesting ideas and the AC also think the paper has some merit, the senior AC finds the technical contribution of the paper weaker than the others. We unfortunately need to recommend the rejection of the paper.
Decision: Reject