Local Patch AutoAugment with Multi-Agent Collaboration
Authors: Shiqi Lin
Tao Yu
Ruoyu Feng
Xin Li
Xin Jin
Zhibo Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: Automatic Data Augmentation
Multi-Agent Reinforcement Learning

Abstract: Data augmentation (DA) plays a critical role in improving the generalization of deep learning models. Recent works on automatically searching for DA policies from data have achieved great success. However, existing automated DA methods generally perform the search at the image level, which limits the exploration of diversity in local regions. In this paper, we propose a more fine-grained automated DA approach, dubbed Patch AutoAugment, to divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. We formulate it as a multi-agent reinforcement learning (MARL) problem, where each agent learns an augmentation policy for each patch based on its content together with the semantics of the whole image. The agents cooperate with each other to achieve the optimal augmentation effect of the entire image by sharing a team reward. We show the effectiveness of our method on multiple benchmark datasets of image classification and fine-grained image recognition (e.g., CIFAR-10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Aircraft). Extensive experiments demonstrate that our method outperforms the state-of-the-art DA methods while requiring fewer computational resources.

Under review as a conference paper at ICLR 2022
LOCAL PATCH AUTOAUGMENT WITH MULTI-AGENT COLLABORATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Data augmentation (DA) plays critical role in improving the generalization of deep learning models. Recent works on automatically searching for DA policies from data have achieved great success. However, existing automated DA meth- ods generally perform the search at the image level, which limits the exploration of diversity in local regions. In this paper, we propose more fine-grained auto- mated DA approach, dubbed Patch AutoAugment, to divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. We formulate it as a multi-agent reinforcement learning (MARL) problem, where each agent learns an augmentation policy for each patch based on its content to- gether with the semantics of the whole image. The agents cooperate with each other to achieve the optimal augmentation effect of the entire image by sharing a team reward. We show the effectiveness of our method on multiple benchmark datasets of image classification, fine-graing grained image recognition and object de- tection (e.g., CIFAR-10, CIFAR-100, ImageNet, CUB-200-288 FGVC-Aireraft and Pascal VOC 2007). Extensive experiments demonstrate that our method outperforms the state-of-the-art DA methods while requiring fewer computational resources.
Stanford Cars,
INTRODUCTION
Data Augmentation (DA) has been widely used to alleviate the overfitting risk in training deep neu- ral networks by appropriately enriching the diver- sity of training data (Shorten & Khoshgoftaar 2019 Gontijo-Lopes et al. 2020). Notable DA methods improve the performance and robustness of the neu- ral networks, such as rotation, Mixup (Zhang et al., 2017) and Cutmix (Yun et al. 2019). But these ap- proaches are typically handcraft and require human prior knowledge, which causes weak transferability of DA across different datasets. To relieve the de- pendence on manual design and further explore more adaptive augmentation, Auto/ugment (AA) (Cubuk) et al. 2018). as new DA paradigm, is proposed to automate the search of the optimal DA policies (i.e., DA operation, probability and magnitude) from the training dataset. To be specific, AA trains proxy model with the augmentation policy generated by a controller, which updated through reinforcement learning using validation accuracy as the reward signal. AA, its optimization procedure is computationally intensive due to the need to evaluate thousands of policies. Therefore, to reduce the search costs, multiple automated DA approaches (Lim et al.) 2019 Ho et al. 2019 Hataya et al., 20199] Zhang et al. 2009b] Lin et al. 2019 have been proposed. For example, (Lim al. (2019) employs density matching search method to accelerate the pol- icy search, and (Zhang et al. 2019b) introduces adversarial learning to organize the target network training and augmentation policies search in online manner.
Original Images
Image Wise Auto-Augment
Panch- Wise AutoAugment
Figure 1: Illustration of different automated augmentation policies. We show the ex- amples processed by image-wise automated DA, e-g., AutoAugment (middle row) and processed by our proposed Patch AutoAug- ment (bottom row).
In spite of the superior performance of

---

Under review as a conference paper at ICLR 2022
Policy Network
Opecadin Noy
Joint Adtion
Tanker
Braud
Agenl
Agent
Aatton
Toan
Tarpe
Networks
Actor
Tranlate
Stear
Agint x
O Pat
Randen Srere
Class
Cutra
Festure
Saae
State Valte
Obarratoz
Update Policy Network
Team
Traising
Catiic
Reward
Losa
Figure 2: The framework of Patch AutoAugment (PAA). We divide an image into a grid of patches and assign agent to each patch for selecting the optimal augmentation operation according the patch content together with the whole image semantics. The agents cooperate with each other to achieve the joint optimal DA results by sharing team reward through multi-agent reinforcement learning (MARL). Specifically, each agent outputs Ox(:: PG; m.) operation performed on patch i, which includes two parameters: probability of calling the operation pt and the magnitude my, Note that PAA is co-trained with the target network.
Yet the aforeementooned automated DA methods all search for policies at the image level. They ig- nore the exploration of diversity in local regions, which may result in insufficient diversity of dataset and limit the benefits of DA (Gontijo- augmentation, they may damage key semantic information and introduce ambiguity into the training process (see a simple example in Figure | row 2, column 2). With those in mind, it is necessary to automatically search for the optimal augmentation policies for different regions by taking re- gional diversity into account. One straightforward idea is to directly apply image-wise automated DA methods in different regions. But such an intuitive solution ignores the contextual relationship between regions, which may lead to the non-globally optimal effectiveness of DA policies across the entire image. Besides, it may encounter an extremely high computational cost due to the need of optimizing multiple policies for regions respectively.
et al. 2020). In addition. due to this coarse- gaanse-rrained
In this paper, we propose a new approach, named Patch AutoAugment (PAA) (see the last row of Figure D. to address the above-mentioned problems. We first divide image into grid of patches to increase the flexibility of representations for different regions and take "patch" as the basic control unit. Then, we model the search for the augmentation policies of patches a fully cooperative multi- agent task, and we leverage a multi-agent reinforcement learning (MARL) algorithm where agents cooperatively learn the policies. Specifically, based on the content of each patch and the semantics of the entire image, the agent searches for a policy in terms of choosing which transformation to apply of the pre-defined DA operations. To encourage our policy networks to adaptively learn the beneficial augmentation policies for the target network, we use the feedback of the target network as the team reward signal to guide the policy networks to learn on the fly. All agents wind up benefiting from two mechanisms (i.e., parameter sharing and centralized training with decentralized execution) in MARL. In this way, all agents collaboratively and parallelly learn policies to further achieve the joint optimal DA policy across the whole image and alleviate the computational cost.
In summary, our contributions can be summarized as follows: 1) We pinpoint that exploring diversity in local regions is important for automated learned DA approaches. To our best knowledge, are the first to propose a more fine- ane-inaieed automated DA approach, that searches for the optimal policies for the patches according to the content of the patch and the semantics of the entire image. 2) To further achieve the joint optimal policy across the image, we model the DA policy search of patches as a fully cooperative multi-agent task, and adopt multi-agent reinforcement learning algorithm for Patch AutoAugment by considering the contextual relationship between the patches.
3) Our visualization results provide some insights the DA community on which augmentation operation should be chosen for patches with different content during the whole training process.
2 RELATED WORK
2.1 DATA AUGMENTATION
Despite the remarkable performance of deep learning models in computer vision tasks, they often suffer from overfitting. Data augmentation (DA) effective technique has been proved im-
2

---

Under review as a conference paper at ICLR 2022
prove the generalization ability of deep learning models (Shorten & Khoshgoftaar 2019). Previous works (Cubuk et al., 2020, GontijoLLpes et al., 2020) indicate that the main benefit of DA arises from increasing the diversity of images. Popular techniques, such as rotation, flipping, color trans- formation, have been performed as commonly used augmentation methods. Recently, thanks to the skillful design of human experts, many DA methods (e-g.. Cutout Vries & Taylorr 2017]. Mixup (Zhang et al.) 2017). CutMix (Yun et al.) (019)) have been proposed and show significant perfor- mance. However, these manually designed methods require additional human prior knowledge on the dataset and sometimes they are limited to certain datasets and target tasks. Naturally, auto- matically finding DA methods from data have emerged to overcome the limitations of dependence on manual cumbersome exploration. Some works use generative adversarial networks (GANs) to directly generate training data (Shrivastava et al. 2017 Tran et al. 2017).
Furthermore, recent studies aim to automate the search for augmentation policies that choose the optimal transformations for training images. AutoAugment (AA) (Cubuk et al., 2018) adopts a controller to generate augmentation policy which is used to train proxy network, then gets the validation accuracy as the reward signal to update the controller using reinforcement learning. Unfortunately, the evaluation of thousands of policies makes AA computationally expensive. There- fore, multiple automated DA approaches focus on reducing the huge complexity and have achieved great progress. For example, PBA (Ho et al., | 2201) employs hyperparameter optimization, Fast AA (Lim et al. 2019) uses a density matching algorithm, Adv AA (Zhang et al., 2019b) proposes an adversarial framework to jointly optimize the target network and the augmentation network, and DADA (i et al. 2020) and DDAS (Liu et al. 2021a) target relaxing the discrete DA policy selec- tion differentiable problem. Besides, Randent (Cubuk et 2020 removes the separate search on proxy task in the offline automated DA methods (e.g.. AA) while also outperforming them. However, these automated DA methods perform the search at the image level, i.e., they use the same policy on the whole image. It inevitably ignores the diversity of different regions in an image, which limits the diversity of data increased by data augmentation, and sometimes causes the damage of critical semantic information. In contrast, our method takes diversity in local regions and contextual relationships into account to search for augmentation policies for multiple regions.
Our proposed method is conceptually orthogonal to most region-based DA methods where DA trans- formations are performed in a non-automated way. For example, RandomErasing (Zhong et al.) replacement operation
2020), CutMix (Yun et al. 2019) perform cropping lected rectangle region, Hard chips (Hong et al. 2019) extracts the object patches from an object pool and pastes them an image to train the detector, and SECOND (Yan et al.. 2018) performs DA by pasting the objects into the 3D point cloud. Some works further exploit class activation map (Zhou al.] 2016) or saliency map (Zhou et al. 2015) select representative regions which are
randomly
augmented by a randomly selected DA operation (e.g.. SaliencyMix (Uddin et al. 20200) SnapMix
(Huang et al., 2020p). AugMix (Hendrycks et al.) 2019). KeepAugment (Gong al., (020) and TrivialAugment (Müller & Hutter|[221]) Yet our proposed method automatically searches for the augmentation transformations based on the given datasets and tasks.
2.2 MULTI-AGENT REINFORCEMENT LEARNING
The most significant characteristic of MARL is the cooperation between agents (Tampuu et al. 2017] Ma & Cameron 2008 Foerster et al. 2017) which is distinct from directly applying reinforcement learning (RL) al aggoritmms to multi-agent systems. Due to the limited observation and action of a single agent, cooperation is necessary in the reinforeed multi-agent system achieve the common goal. Compared with independent agents, cooperative agents can improve the efficiency and robust- ness of the model (Neto) 2005) Zhang et al.] 2019a] [Busoniu et al., (2008). Many vision tasks use MARL to interact with the public environment to make decisions, with the goal of maximizing the expected total return of all agents, such as image segmentation (Liao et al. 2020 Han et al. 2019 Ma et al. 2020), image processing (Furuta et al. 2019).
PROPOSED METHOD
As above mentioned, we aim to search augmentation policies for the patches to explore more aug- mentation diversity. Learning the optimal DA policies often needs to interact with the target network, which aligns with the nature of RL (i.e., RL agents learn to make decisions through interacting
with the environment Sutton & Barto 2018 Arulkumaran et al. 2017). In addition, it is of great
3

---

Under review as a conference paper at ICLR 2022
necessity to take the patch content and the contextual relationship between patches into account. Therefore, we use multi-agent reinforcement learning (MARL) algorithm. Specifically, the search for patch policies is based on the content of the patch together with the semantics of the image, and policies are encouraged to coordinate to achieve the joint optimal DA policy across the whole image. In this section, we first describe the preliminaries of MARL,, then elaborate our aug- mentation policy formulation and modelling. Furthermore, we summarize the framework of Patch AutoAugment.
3.1 PRELIMINARIES OF MULTI-AGEN REINFORCEMENT LEARNING
We first introduce the preliminaries of reinforcement learning (RL). RL models the decision- problem a Markov decision process (MDP) which is presented with a tuple (S. A, P. R. T). In RL framework, given the state $ E S, the agent takes an a E AA according to its policy aia (s) S x A - [0, 1] and then receives reward r S x A - IR. The environment moves to the next state with a transition function denoted as P S x A x S > [o, 1]. + E (0. 1] is discount factor and 7' is a time horizon. The agent aims maximize the long-term reward R over I steps to learn the optimal policy π².
Furthermore, multi-agent reinforcement learning (MARL) considers a group of N agents, de- noted as .N, operating cooperatively in shared environment towards a common goal. It can be formulated as a multi-agent MDP (MAMDP) (Boutilier [1996) represented with a tuple (S, , (A P. R, 7: T). Here, S describes the shared state space and {O1, ON) set of observations for agents.
the MARL. configuration, each agent receives private ob- servation correlated with part of state, ie., 0, :S - O. According to the global state s and its observation og, the agent takes its action a; E A, based on its policy т. (a; lo,, 8), A, is the action space for the i-th agent and A= {A1,-.. ...,., AN) denotes the joint action space. Then, the joint action a = 01 X ... XaN produces next state according to transition function P: S x A S - [0, 1] and the environment gives the team reward
S x A- R to agents A The objective of each agent is to maximize the total accumulative reward A to cooperatively learn the globally optimal policy " { "> vy that consists of the optimal policy of each agent. In this paper, mentioned before, we employ MARL to search optimal policy for each patch, and achieve the optimal effec- tiveness of DA across the entire image further improve the performance of the target network as possible.
3.2 PATCH AUTOAUGMENT
In our proposed Patch Auto.Au (PAA), we formulate the task of policy search for the patches as a cooperative multi-agent decision-making problem and adopt multi-agent reinforcement learning (MARL) to solve it. In the following, We clarify the detailed formulation i) the state, observation and action modeling for the policy, it) an effective team reward function design and wii the detailed MARL algorithm for policy learning) of Patch Autmnn..
Policy Modeling, As illustrated in Figure 2 given the original input batch æ and the corresponding label y (i.e., {a, y} {z,, 8/ 1 and is the batch size), we divide an image Ij into N equal- sized and non-overlapping patches, denoted as x, (P) where p is the i-th patch of the image Ty. In our proposed method, we aim to search the augmentation policy for each patch. Therefore, in MARL formulation, the augmentation policy of the patch pi is controlled by an agent e W and we detail the state, observation and action for the augmentation policy as below.
State. As aforementioned. the selection of augmentation operation for patch is closely bound up with the contextual relationship between regions. Therefore, the augmentation policy needs to perceive the image semanties and we take the deep features of the whole image extracted by a backbone (e.g., ImageNet pre-trained ResNet-18 (He et al., 2016)) as the global state which is visible to all agents. We analyze the impact of feature extractor initialization in Appendix F
Observation. Apart from capturing state (i.e., the global information), the agent only use their own observation (i.e., the local information) which is invisible to other agents. The augmentation policy seeks to choose augmentation operation based the content of the patch and the observation is generally part of the state. Considering all these factors, we utilize the deep features of the i-th patch observation og which are extracted by the same backbone the state extractor.
4

---

Under review as a conference paper at ICLR 2022
Action. The augmentation policy is responsible for choosing which transformations to apply from pre-defined operations. Following the previous automated DA methods (Cubuk et al. 2018 Lim et al., 2019), we define the fifteen operation functions (i.e., ShearX/Y, TranslateX/Y, Rotate, In- vert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, RandomErasing, Cutout, Mixup, Cutmix) construct the action space A. Given the state s and the observation og, according to policy on the patch. Each operation is associated with two hyperparameters: probability p and magnitude m. In order to dramatically and effectively reduce the action space for augmentation policy, sim-
each agent determines an action (-) E A, which is the operation performed
ilar with (Cubuk et al., 2020 HH et al., [019 Lim et al.) 2019). we take a fixed probability and magnitude schedule. Among them, the probability of applying the operation is sampled from the uniform distribution (i.e., pi ~ (0, 1). Following (Ho et al. (019)), we employ the same linear scale to be the magnitude schedule. In summary, the processed 1-th patch in image zy is denoted as Pj = ag (Pj) with the probability p; and the magnitude of (-) is m,, otherwise Pj P: Note that most operations are label-invariant, except Mixup (Zhang et al. 2017) and CutMix (Yun et al.) 2019) are tabel-disturbing operations that combine different patches as well as their labels. We take Mixup as an example, then Pi = XP; + (1 - >)P; with probability Pi. where Pú is a patch from another image 24, t * j. À ~ Beta(a, ( a), for a E (0, oo), and the one-hot label is modified as 30 - ,, ( N y/: Once all patches are processed by the corresponding operations chosen by the augmentation policies, we obtain the image augmented by our PAA, denoted as ã. = (P]. (P....,PN PNN, and the final label Øy More operation details are shown in Appendix A.
Reward Function. The reward function is of importance to guide the agents to learn so that they follow desired behaviors. The previous work, AdvAA (Zhang et al.) 2019b), attempts to increase the training loss of the network to generate harder augmentation policies and explore the weakness of the target network. Inspired by AdvAA, we reformulate the reward design appropriately under configuration. In our proposed PAA, the common objective of all agents is to improve the performance of the mainstream target task through enhancing the benefits of DA. Therefore, we compare the feedback of target network o6-) on the augmented data processed by our proposed PAA E with the original data æ and take their difference on the training losses as the reward for the policy in MARL, as in Eq.
= 1(6(#) 7) - 1()(x).))
(1)
where and denote raw inputs and labels in supervision tasks. In our PAA model, all agents are encouraged to cooperate to achieve the common goal, thus we adopt the team reward function design (i.e., the shared reward mechanism in MARL) Eq. (1) for all agents to make the joint augmentation policy achieve the optimal effectiveness.
Policy Learning. Here, we introduce the training for the augmentation policies mentioned above. Considering that the action space is discrete, we adopt multi-agent Advantage Actor-Critic algorithm (Lowe et al., 2017) to learn the augmentation policies and encourage the coordination behaviors. In MARL, the framework of centralized training with decentralized execution (Foerster et al.) 2018
Rashid al.] 2018) generally adopted. More concretely, each agent has an actor which is to learn discrete policy (a,aaoo,, s) and agents share common critic which aims to estimate the value of global state V (s). And we use the centralized critic to train decentralised actors. Here, we reformulate it appropriately for our task. We model the centralized action-value Q function that takes the actions of all agents in addition to state information s and outputs the Q-value for the team, formulated as:
QT (s. a) F. Re 88, ",,"",,........................
". .aN]]
where a is the joint action of all agents a = {a.... an) and RS = I'rets e is the long-term discounted reward. Then, the advantage function on the augmentation policy is given as follows:
(s = V(*))))
(3)
The critic network is a Multi-layer Perceptron (MLP). And we use to denote the its parameters. We take the square value of the advantage function A as the loss function update yo:
T(p) (A (s, a))²
(4)
5

---

Under review as a conference paper at ICLR 2022
Table I: Test set accuracy (%) on CIFAR-IO 10 and CIFAR-I00. The results of our proposed PAA is
the average accuracy (#staandrr deviation) over four random runs.
Datuset
Model
Baseline
CuiOut
Mixup
CutMix
Co-Mix
AA
FastÁA
RA
DADA
PAA
WRN-2 10NN-28-11
96.1
97.1
97.2
97.3
97.4
97.3
97.5
SS(26 2x32d)
96.4
97.0
97.2
97.3
97.4
97.5
97.3
97.5
97.3
97.6
CIFAR-II 10
SS(26 2x96d)
97.1
97.4
97.7
97.8
98.0
97.7
97.7
97.8
98.0
98.1 10.1
SS(26 2x1 12d)
97.2
97.4
98.0
98.0
98.0
98.1
98.0
98.0
98.1
Pyramid+SD
97.3
97.7
98.0
98.1
98.2
98.5
98.2
98.5
98.3
98.6
WRN-28 28- 10
81.2
81.6
82.1
82.8
K3.4
82.9
82.7
83.3
K2.5
20.4
CIFAR-100
SS(26 2x96d)
82.9
84.0
85,4
85.6
85.8
85.7
85.1
85.5
85.7
85.9.0.2
Pyramid+SD
86.0
87.8
88.5
88.9
89.0
89.3
88.1
89.2
88.8
89.2 89.2
Table 2: Validation set Top-1 Top-5 accuracy (%) on ImageNet.
Method
Baseline
Mixup
CutMix
Co-Mix
AA
FastAA
RA
DADA
PAA
ResNet-50 50
76.3/33.1
77.0/93.4
77.2/93.5
77.6/9377
77.6/93.8
77.6/93.7
77.6/93.7 80.1/95.0
77.5/93.5
28.3 +4.2
ResNet-220)
78.5/94.2
79.6/94.8
79.9/94.9
80.094.9
80.1/95.0
80.6/5533
79.8/94.8
81.0 9
The global policy network is a fully convolutional network (FCN), denoted as Besides, to further achieve the ability to cooperate, similar to (Foerster et al., 2018 Rashid et al. [2018), the parameters of the actor networks of all agents are shared. The loss function for updating @ is defined as: L( = log To
(5)
Framework Summary. In this part, summarize the overall framework of our proposed PAA. As shown in Figure 2 PAA first divides an image into a grid of patches. Then, we use a feature extractor to obtain the deep features of the whole image as the state. Each agent draws its individual observation, i.e,, the deep features of the patch. According to the global state (i.e., the semantics of the entire image) and the local observation (i.e., the content of the patch), the actor networks output the augmentation operations of patches to further construct the joint operation map performed on the whole image. The augmented images are processed by our proposed PAA and then we input them to the target task network for parameters updating. Moreover, the feedback of the target network is used as the team reward signal to update the policy network.
EXPERIMENTS
4.1 EXPERIMENT OVERVIEW
In this section, to study the effectiveness of Patch AutoAugment (PAA), we conduct experiment on image classification, fine-grained image recognition and local tasks where local information is of importance, e.g., object detection. Specifically, we focus on CIFAR-10, CIFAR- 100 (Krizhevsky) al. 2009) and ImageNet (Deng et al.] 2009) datasets as well as three fine-grained object recog- nition datasets, i.e., CUB-200-2011 (Wah et al. (011)). Stanford Cars (Krause et al. [013), FGVC- Aircraft (Maji et al. 2013) and Pascal VOC 2007 (Everingham & Winn 2009). We describe the datasets in detail in Appendix B We compare PAA with baseline pre-processing, Cutout (De Vries & Taylor 2017). Mixup (Zhng et al. (0017), Cutmix (Yun et al. 2019). Co-Mixup (Co-Mix) (Kim let al. 2021p. AutoAugment (AA) (Cubuk et al. 2018). Fast AutoAugment (FastA.) (Lim et al. 2019). RandAugment (RA) (Cubuk et al. 2020). , DADA (Li et al. 2020) and Adversarial AA (Ad-
vAA). The baseline follows (Zoph et al. 2018; Yamada et al. [018 Gastaldi 2017): standardizing the data, horizontally flipping with 0.5 probability, zero- pero-padding and random cropping. Our pol- icy model architecture design mainly follows (Chu et al. 2016 and we make some adjustments according to our task scenario. More details about our policy network architectures and the hyper- parameters of the target model and our policy model are supplied in Appendix Cland respectively. Moreover, we default the number of patches N to 16 except for CIFAR tasks N 4. To ensure the reliability of our experiments, we run each experiment four times using different random seeds.
4.2 RESULTS AND ANALYSIS
Classification Results on CIFAR-10 and CIFAR-100. For CIFAR-10 and CIFAR-100, we ex- amine on Wide-ResNett-8-111 (WRN-28-10) Zagoruyko & Komodakis 2016), Shake-Shake (SS) (Gstaadi 2017) and Pyramid-Nete Neeld-Net+hhaakaeaae Shakeeaa (Pyramid+SD) (Han et al. 2017 Yamada et al.) 2018) models. The results are reported in Table [ which shows the proposed approach consistently outperforms several state-of-the-an DA methods. We observe that the improvement of performance is relatively slight, due to the small image size of CIFAR which is 32 x 32. In the following, we further apply our proposed PAA datasets with larger image sizes and other networks.

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison of computational cost (GPU hours) between our proposed PAA and other previous automated DA methods. We train Wide-ResNett28-1D1 on CIFAR-10 and ResNet-50 on ImageNet. Search: the time of searching for augmentation policys. Train: the time of training the target network. The training time of PAA includes the inference time required to compute the state and observations. Total: the total time. Except PAA, all metries are cited from (Zhang et al. 2019b Lim et al. 2019).
Dataset
hours
hours
AA
FastAA
AdvAA
PAA
Seurch
5000
3.3
-
-
CIFAR-10
Train
6
7.5
Total
5006
7.5
Search
15000
450
~00
~0
ImageNet
Train
160
160
1280
270
Total
15160
1280
270
Table 4: Test accuracy (%) on various fine- riee-graied classification datasets including CUB-200-20) (CUB) (Wah et al. 2011). Stanford Cars (Cars) (Krause et al. 2013) and FGVC-Aireraft (Aircraft)
Maji et al.) 2013).
Dataset
Model
Baseline
Mixup
CutMix
Co-Mix
AA
FastAA
RA
DADA
PAA
CUB
ResNet-50
83.3
86.2
86.1
87.1
86.8
86.3
86.9
86.8
87.5
ResiNet-10 101
85.6
87.7
87.9
88.2
88.1
87.9
88.0
88.1
88.3:0.2
ResNet-50
93.0
93.9
94.1
93.9
94.2
94.0
94.1
93.7
94,3
Cars
ResNet-101
93.1
94.1
94.2
94.1
94.2
93.8
94.2
93.7
94.5
ResNet-50
91.0
92.0
92.2
92.2
92.3
92.2
92.3
91.8
92.620.2
Aircraft
ResNet- 101
91.6
92.9
92.3
93.1
92.8
92.9
92.6
92.8
93.5
Classification Results on ImageNet. As shown in Table 2 we evaluate our method on ResNet-50 and ResNet-200 (He et al.) 2016) backbone on ImageNet, and our PAA significantly improves the performance of the target networks. The results further demonstrate that our proposed method is an effective DA technique for consistent and expressive benefits for datasets with larger image sizes.
Effectiveness of Fine-grained Classification. Furthermore, we evaluate our proposed method on fine-grained image recognition tasks. According to previous work (Du et al.) 2020 Chen et al.) 2019). we take ResNet-50 and ResNet-101 as the backbones. The results which illustrates that the performance of PAA is consistently better than other methods and PAA has achieved remarkable performance these challenging fine ime-grained tasks.
shown in Table 4
Effectiveness of Local Tasks. In order to further verify the effectiveness of our proposed method, conduct experiments the object detection task. We adopt the mainstream detector Faster R- CNN (Ren et al., 2013) (ResNet-50) on the dataset Pascal VOC 2007 (Everingham & Winn 2009) and show the mAP results on the Table We observe that the proposed PAA achieves consistent improvements and PAA is indeed effective for the local tasks like object detection.
4.3 COMPLEXITY ANALYSIS
In this section, in order to further demonstrate the performance of PAA in terms of complexity, we compare the policy search time and training time of PAA with AA (Cubuk et al., 2018), FastAA (Lim et al.) 2019) and AdvAA (Zhang et al., 2019h), as illustrated in Table 3 As shown in Table 3. compared the previous works, PAA requires the fewest total computational resources, and the search time is almost negligible. As for the parameters, the total number of PAA model parameters (about 0.23M) is less than 1% of the target network (e.g., ResNet50: about 25.5M).
In summary, the main methods to reduce the computational cost lie in three aspects: 1) The augmen- tation policy network is jointly optimized with the target network, similar to (Zhang et al. 2019b] Lin et al. 2019). Namely, our proposed method searches for policies in an online manner, obviating thousands of policies validation on small proxy network and the requirement for retraining the target network. Besides, we use fixed schedules for the two corresponding hyperparameters (i.e., probability and magnitude) of each transformation to effectively reduce the search space, which makes it easier to search for effective policies. By these means, PAA compresses most of the policy search time 2) As mentioned before, a MARL algorithm is adopted, in which all agents parallelly leam the augmentation policies, reduce the training time. 3) Compared with the previous DA methods using image-by- baage-by-imaae by-image sequential transformations, we perform parallel transformations on tensor. Specifically, pick the patches in batch performing the same operation to reconstruct
7

---

Under review as a conference paper at ICLR 2022
Table 5: The mAP (%) results on Pascal VOC. We use the Faster R-CNN as the baseline detector.
Method
Buseline
CutOut
Mixup 78.3
AA
PAA
MAP
76.0
77.2
78.7
79.0
Table 6: Ablation study: performance comparisons (%) of Patch RandomAugment (PRA), Patch Single AutoAugment (PSAA) and our PAA on CIFAR- and CIFAR- 100 (test accuracy) with Wide- ResNet-22-10 (WRN-28-100 and on ImageNet (Top- 1 / Top-5 accuracy) with ResNet-50.
Method
Policy
Dataset
random
SARL
MARL
CIFAR-T TO
CIFAR:
ImageNet
PRA PSAA
97.1
83.0
71.979399
97.3
83.1
78.0/ 93.9
PAA(Ours)
97.5
83,4
78.3/ 94.1
new tensor. And use Kornie realize tensor transformations on GPU to further reduce the processing time which is included in the training time.
4.4 ABLATION STUDY
In this section, we study the effectiveness of MARL and also discuss the design patch numbers, through several ablation experiments.
Performance of random policies. We randomize the augmentation policy for each patch, dubbed Patch RandAugment and compare it with proposed PAA. To be specific, the augmentation policy is randomly selected from predefined transformations with uniform probability. As shown in Table 6. Patch RandAugment (PRA) leads to considerable appreciable improvements. However, Patch AutoAugment with meaningful guidance significantly surpasses the random patch policies.
Performance of the policies searched using single-agent RL. Furthermore, directly the same single-agent RL to search for each patch's policy, termed as Patch SingleAutoAugment (PSAA) and compare it to PAA, where the search task is cooperative multi-agent problem. In short, PSAA ignores the contextual relationship between patches, where the agents are independent and patches are non-cooperative. The results (see Table(6) indicate that taking the contextual information into account with cooperative RL-agents has improved the joint effectiveness of DA.
Impact of the number of patches. We explore the effect of the number of patches on the target model performance. As shown in Table 7 we respectively set the number of patches N {4, 16, 64, 256, 1024). The results show that on ImageNet CUB-200-2011, when patch numbers N increase, the performance accuracy first increases and then decreases. In addition, when N = 16, PAA achieves the best performance. We analyze that the small number of patches may cause PAA to be unable to effectively explore local diversity, and its advantages would be limited. In particular, when N = 1, PAA degenerates into image-wise automated DA. In contrast, under too larger values (e.g., the extreme case is to search for DA policy for each pixel), the local mantic consistency is broken and the benefit brought by the consideration of contextual relationship between patches gradually overtaken.
4.5 VISUALIZATION
Policy Visualization. Grad-CAM (Selvaraju et al. 2017) is used localize the important regions in the image. Therefore we can calculate the importance score of each patch, and we divide the importance scores into four bins, iee, very important, important, normal and not important. Then, we categorize patches into four groups and draw four stacked area charts showing the percentages of operations selected by PAA augmentation policies over time. We take ResNet-50 backbone trained on CUB-200-2011 as an example. Since our proposed PAA searches for patch policies in an online manner, the strategies change dynamically over time as shown in Figure At the beginning of the training process, the selected actions are messy since the MARL network is in the exploratory stage. At the tail end of the training, the target network has converged, causing the percentage of all operations to be almost the same and the percentage to flatten out.
In addition, we have some interesting findings which may provide some insights to the DA commu- nity. i) The optimal augmentation strategies of patches vary by their importance levels. Therefore, it is necessary to take the image content into account when performing data augmentation. ii) In
Kornia (Riba et al. 2020) a differentiable computer vision library for PyTorch. We use it to accelerate augmentation operation on tensors.
8

---

Under review as a conference paper at ICLR 2022
Table 7: Ablation study; performance comparisons (Top-1 accuracy (%)) under five different patch numbers N on CIFAR-10 (image size: 32 x 32) with WRN-28-10 10 and on ImageNet (image size:
224 x 224) together with CUB-200-2011 (CUB) ) (image size: 448 x 448) with ResNet-50 backbone.
N
64
256
1024
CIFAR-I0
97.5 97.3 97.2
97.1
CUB
87.2 87.5
87.3
87.1 86.8
Baseline
Augment
Ours
Roado (a) Very Importan:
00
10 130 T4E
100 150
(b) Importent
103 120 141
150 1S0
Normol
Important
biightnes
controat
cutmis
randomerasing
abear
invert
posterioe
mixup
color
botution
sharpness
cloarize
trunslate
cutout
equalize
Figure 3: Grad-CAM visualization on the examples from CUB with ResNet-50 trained by baseline, Augment and our PAA. PAA tends to help the target network focus task-related features.
Figure 4: We categorize patches into four groups ac- cording to the patch importance calculated through Grad-CAM. We draw four stacked area chart of the operations' percentages over time. The area represents the percentage of each operation.
the middle of the training process, different types of patches prefer to select different augmentation operations. Concretely, as illustrated in Figure 4 for the important patches, color transformation mostly picked. For the unimportant patches, RandomErusing and Cutoul are usually chosen by PAA. Important patches commonly take along semantic information that is related to mainstream tasks. better to choose the mild transformations (e.g., color) for them, which can effectively protect the semantic information from being damaged. In contrast, unimportant patches typically carry unexpected features (Singla et al. 202I) which are causally unrelated to the desired class. Severe transformations (e.g., Cutour, RandomErasing! could be chosen for them, which introduce noise and disturbance to reduce the impact unexpected features on the target network learning.
Grad-CAM Visualization. Here, we adopt Grad-CAM (Selvaraju et al. 2017) to visualize the learned features to intuitively show the impact of PAA, as shown in Figure We take the ResNet-50 the backhone which is trained with dataset processed by 1) Baseline 2) AutoAugment and 3) Patch AutoAugment, respectively. We observe that the model trained with PAA focus on more task-related areas rather than spurious correlations (e.g., the branch where the bird stands) or overemphasized features (e.g., birds" claws). More visualization results are provided in the Appendix
5 CONCLUSION
In this paper, we propose Patch AutoAAument (PAA), more fine-grained automated data augmen- tation approach. Our method adopts multi-agent reinforcement learning to automatically search for the optimal augmentation policies for patches, and encourages agents to cooperate with each other to further achieve the joint optimal policy across the entire image. Extensive experiments demonstrate that PAA improves the target network performance with low computational cost in various tasks. Meanwhile, we use visualization to show that PAA is beneficial for the target network to localize more class-related cues. Furthermore, we hope that our visual observations of policies will be useful for future development. future work, we will investigate different schemes on dividing different regions. Additionally, our method is naturally aligned with the patch token mechanism in the current vision transformers (Dosovitskiy et al. 2020 Touvron et al. 2021 Liu et al. 2021bb and the data augmentation specific to vision transformers has not been extensively studied. Therefore, we leave the automated data augmentation for vision transformers to future work, which may provide some interesting insights to the community.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep rein- forcement learning: A brief survey, IEEE Signal Processing Magazine, 34(6):26-38, 2017.
Craig Boutilier. Planning, learning and coordination in multiagent decision processes. TARK, volume 96, pp. 195-210. Citeseer, 1996.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. comprehensive survey of multiagent rein- forcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-172, 2008.
Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. Destruction and construction learning for fine- grained image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5157-5166, 2019.
Tianshu Chu, Shuhui Qu, and Jie Wang. Large-scale multi-agent reinforcement learning using image-based state representation. In 2016 IEEE 55th Conference Decision and Control pp. 7592-7597. IEEE, 2016.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.090 2018.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, and Nasser M Nasrabadi. Supermix: Super- vising the mixing data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13794-13803, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 1708.04570 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Ruoyi Du Dongliang Chang, Ayan Kumar Bhumia, Jiyang Xie, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Fine-grained visual classification via progressive graai-gray training of jigsaw patches. In European Conference on Computer Vision, pp. 153-168. Springer, 2020.
Mark Everingham and John Winn. The pascal visual object classes challenge 2007 (voc2007) de- velopment kit. 2009.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaming awareness. arXiv preprint (rXXiv:709.043330 2017.
Ryosuke Furuta, Naoto Inoue, and Toshihiko Yamasaki. Pixelrl: Fully convolutional network with reinforcement learning for image processing, IEEE Transactions Multimedia, 2019.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv: 1705.07705 2017.
Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Keepaugment: A simple information-preserving data augmentation approach. arXiv preprint arXiv:2011.11778 2020.
10

---

Under review as a conference paper at ICLR 2022
Raphael Gontijo-Lpess, Sylvia J Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.08973, 2020.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5927-5935, 2017.
Lei Han, Peng Sun, Yali Du, Jiechao Xiong, Qing Wang. Xinghai Sun, Han Liu, and Tong Zhang. Grid-wise control for multi-agent reinforcement learning in video game ai. In International Con- ference on Machine Learning, pp. 2376-25887 PMLR, 2019.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. arXiv preprint arXiv:1911.06987. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:191..02788. 2019.
Daniel Ho, Eric Liang, Xi Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules, In International Conference Machine Learning, pp. 2731-2741, 2019.
Sungeun Hong, Sungil Kang, and Donghyeon Cho. Patch-level augmentation for object detection in aerial images. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pp. 0-0, 2019.
Shaoli Huang, Xinchao Wang, and Dacheng Tao. Snapmix: Semantically proportional mixing for augmenting fine-grained data. arXiv preprint arXiv:2012.0488 040446 2020.
Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. arXiv preprint arXiv:2 102.03660, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Dada: Differentiable automatic data augmentation. arXiv preprint arXiv:2003.0378) 2020.
Xuan Liao, Wenhao Li, Qisen Xu, Xiangfeng Wang. Bo Jin, Xiaoyun Zhang, Yanfeng Wang, and Ya Zhang. Iteratively-refined interactive 3d medical image segmentation with multi-agent rein- forcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9394 9394-9402, 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, pp. 6665-6675, 2019.
Chen Lin, Minghao Guo, Chuming Li. Xin Yuan, Wei Wu, Junjie Yan, Dahua Lin, and Wanli Ouyang. Online hyper-parameter learning for auto-augmentation strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6579-6588, 2019.
Aoming Liu, Zehao Huang, Zhiwu Huang, and Naiyan Wang. Direct differentiable augmentation search. arXiv preprint arXiv:2104 042222 202
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021b.
11

---

Under review as a conference paper at ICLR 2022
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-competttvee environments. arXiv preprint arXiv: htti:::7060022275. 2017.
Chaofan Ma, Qisen Xu, Xiangfeng Wang, Bo Jin, Xiaoyun Zhang, Yanfeng Wang, and Ya Zhang. Boundary-aware supervoxel-level iteratively refined interactive 3d image segmentation with multi-agent reinforcement learning. IEEE Transactions on Medical Imaging, 2020.
Jie Ma and Stephen Cameron. Combining policy search with planning in multi-agent cooperation. In Robot Soccer World Cup, pp. 532-543. Springer, 2008.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, PMLR, 2016.
Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmen- tation. arXiv preprint arXiv:2103.10158 2021.
Gonçalo Neto. From single-agent to multi-agent reinforcement learning: Foundational concepts and methods. Learning theory course, 2005.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference on Machine Learning, pp. 4295-4304. 4304. PMLR, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3674-3683, 2020.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 2107-2116, 2017.
Satinder Singh, Tommi Jaakkola, Michael L Littman, and Csaba Szepesvári. Convergence results for single-step on- on-policy ceiforceeeentleemming algorithms. Machine learning, 38(3):287-308, 2000.
Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12853-12862, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning- PloS one, 12(4):c0172395, 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2022
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. A bayesian data augmenta- tion approach for learning deep models. arXiv preprint arXiv:1710.10544 2017.
AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et al. Salien- cymix: A saliency guided data augmentation strategy for better regularization. arXiv preprint arXiv:2006.01991, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Yoshihiro Yamada, Masakazu Iwamura, and Koichi Kise. Shakedrop regularization. 2018.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings the the IEEE International Conference on Computer Vision, pp. 6023-6032, 2019.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:1605.07146, 2016.
Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0111. 2017.
Kaiging Zhang, Zhuoran Yang. and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. arXiv preprint 0rXhv:/911.10635 2019a.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. arXiv preprint arXiv:1912.11188. 2019b.
2019b.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In AAAI, pp. 13001-13008, 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Li Zhou, Zhaohui Yang, Qing Yuan, Zongtan Zhou, and Dewen Hu. Salient region detection via inte- grating diffusion-basel compactness and local contrast. IEEE Transactions on Image Processing, 24(11):3308-33 3320, 2015.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
13

---

Under review as a conference paper at ICLR 2022
Table 8: We list fifteen kinds of augmentation operations that we use. Additionally, the schedule of magnitude (i.e., the parameters for Kornia (Riba al., 2020) pytorch library) for each operation are shown in the third column. Some transformations do not use the magnitude information (e.g., CutMix and Cutout).
Operation Name
Description
magnitudes
Adjust the brightness of the patch. A magnitud==0 gives a
Brightess
(rightmess=(0.., 0.95)
black patch, whereas magnitude=1 gives the original patch.
Control the contrast of the patch. A magnitude=0 gives gray patch, whereas magnitude=1 gives the original patch.
Contrast
contrast= 0.95)
Replace this patch with another patch (selected at random
CutMix
from the patches which are also performed CutMix).
Cutout
Set all pixels in this patch to the average value of the patch.
Invert
Invert the pixels the patch
Linearly add the image with another image (selected random
Mixup
A ~~ Beta(1, 1)
from the patches which are also performed Mixap).
Posterize
Reduce the number of bits each pixel magnitude bits.
bits=3
Solarize
Invert pixels above threshold value magnitude.
thresholds=0.1
scale=(0.09,, 0.36),
RandomErasing
Erases random rectangle region in patch.
ratio=(0.5, 1/0.5)
Rotation
Rotate the patch magnitude degrees.
degrees=30.0
Adjust the sharpness of the image. A magnitude= gives
Sharpness
sharpness=0.5
blurred image, whereas magnitud==1 gives the original image.
Sheur the image along the horizontal or vertical axis with rate magnitude
Shear(X/Y)
shear=(-30, 30)
Translate the patch in the horizontal vertical direction
Translato(X/Y)
translate=(0.4. 0.4)
absolute fraction of patch length.
Color
Adjust the color balance of the image.
hue=(-0.3, 0.3)
Equalize
Equalize the image histogram.
Table 9: The model architecture of policy network and critic network in PAA. For each convolution layer, we list the input dimension, output dimension, kernel size, stride, and padding. For the fully- connected layer, we provide the input and output dimension. BN is short for batch normalization.
Layer
Policy network
Critic network
FC(1568,256)
2
ReLUU.COoW2DIG44433........
ReLU
3
FC(256,
APPENDIX
A OPERATIONS DETAILS
Following (Cubuk et al. 2018 Ho al. 2019 Lim et al. 2019 Zhang et al. 2019b), we define the
fifteen common augmentation operations to form the action space. Here, we detail the description of these operations, illustrated Table & In addition, we give magnitudes range of augmentation operations corresponding to hyperparameters of the functions in the Kornia PyTorch library. Some operations, such as Cutout and CutMix, have no parameters.
Additionally, in our implementation, we pick out the patches that perform the same operation and put them into a new tensor. Then, we speed up the process by performing parallel transformations on the tensor. Tensor transformations the computational costs. In particular, when performing the label-disturbing operations (i.e., Mixup and CutMix), a patch needs mix with another patch that is randomly selected from the tensor. Namely, a patch is mixed with another patch that performs the same operation.
GPU can be realized by Kornia (Riba et al. 2020) to reduce
B DATASETS
We evaluate Patch AutoAugment (PAA) on the following datasets: CIFAR-10 (Krizhessky et al. 2009) CIFAR-100 (Krizhevsky et al. 2009). ImageNet (Deng et al. 2009) and three fine-grained
14

---

Under review as a conference paper at ICLR 2022
Table 10: The hyperparameters of various target models on CIFAR-10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Aircraft. LR represents learning rate of the target net- work, WD represents weight decay, and LD represents learning rate decay method. If LD is multi- step, we decay the learning rate by 10-fold at epochs 30, 60, 90 etc. according to Lt-step. LR-A2C represents the learning rate of augmentation model.
Dataset
Model
BatchSize
LR
WD
LD
LRstep
LR-A2C
Epoch
WRN-28-11 10
128
0.1
5e-4
cosine
le-3
200
SS(26 2x32d)
128
0.2
le-4
cosine
le-4
600
CIFAR-10
SS(26 2x96d)
128
0.2
le-4
cosine
le-4
600
SS(26 2x1 12d)
128
0.2
le-4
cosine
le-4
600
PD
128
0.1
le-4
cosine
le-4
600
WRN-28-10
128
0.1
5e-4
cosine
le-4
200
CIFAR-100
SS(26 2x96d)
128
0.1
5e-4
cosine
le-4
1200
Pyramid+SD
128
0.5
le-4
cosine
le-4
1200
ResNet-50
512
0.1
le-4
multistep
[30, [30,60,90,, 10,.000000.150)
270
ImageNet
ResNet-200
512
le-4
multistep
[30, (30.60.90.120.1500
le44
ResNet-50
512
lc-3
To-4
multistep
[30,60,90]
le-4
CUB-200-2011
ResNet-101
512
le-3
le-4
multistep
[30,60,90]
le-4
ResNet-50
512
le-3
le-4
multistep
[30,60,90]
le.4
200
Stanford Cars
ResNet-101
512
le-3
le-4
multistep
[30,60,90]
le-4
200
ResNet-50
512
lc-3
lo-4
multistep
[30,60,90]
lc44
200
FGVC-Aireraft
ResNct-101
512
1c-3
lo-4
multistep
[30,60,90]
lc44
200
image recognition datasets FGVC-Aircraft (Maji et al. 2013) and Pascal VOC (Everingham & Winn 2009
(Wah al.) 2011), Stanford Cars (Krause al. 2013)
To be specific, both CIFAR-10 and CIFAR-100 have 50,000 training examples. Each image of size 32 x 32 belongs to one of 10 categories. ImageNet dataset has about 1.2 million training images and 50,000 validation images with 1000 classes. Original ImageNet data have different sizes and we resize them to 224 x 224. In addition, we evaluate the performance of our proposed PAA on three standard fine-grained object recognition datasets. CUB-200-2011 (Wah et al. 201T) consists of 6,000 train and 5,800 test bird images distributed in 200 categories. Stanford Cars (Krause et al,, 2013) contains 16, 16,185 images in 196 classes. The FGVC-Aircraft dataset contains 10,200 images of aircraft, with 100 images for each of 102 different aircraft model variants, most of which are airplanes. The image size the above three datasets 448 x 448. The Pascal VOC dataset consists of a set of images and each image has an annotation file giving bounding box and object class label for each object in one of the twenty classes present in the image. The data has been split into 50% for training and 50% for testing. In total there are 9,963 images, containing 24,640 annotated objects.
C MODEL ARCHITECTURE
Here, we provide the detailed model architecture for each component our PAA augmentation policy model, including feature extractor network, actor network and critic network. We use the pre- trained on ImageNet ResNet-18 backbone (excluding the deep features of the image and the patch, which are denoted as the state and the observation o respectively. As for the global policy network and the critic network, the detailed model architectures shown in Table 9 The policy network is fully convolutional neural network (FCN). The parameters of the actor networks all agents are shared. Although they share parameters, each agent's 's policy not identical. When determining an action, each agent's policy relies on different activations of the global policy network.
final avgpool and softmax layer) to extract
D HYPERPARAMETERS
We detail various target models hyperparameters (e.g., batch size, learning rate and training epochs) on 110AR-10, CIFAR- 100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Airerale in Table
10 We do not specifically tune these hyperparameters, and all of these are consistent with previous
works (Cubuk al. 2018 Lim al. 2019 Dabouei et al. 2021 Du et al. 2020 Chen al. 2019.
15

---

Under review as a conference paper at ICLR 2022
Table 11: Ablation study: impact of the initialization of the feature extractor. We respectively the pretrained ResNet- 18 and non pretrriied ResNet-18 as the feature extractor to get the deep features as the state/observation and show the test accuracy (%).
Dataset & Model
Baseline
Pretrained ResNet-18
Non-pretrained ResNet-
CIFAR- 100 (WRN-28-10)
81.2
83.4
83.2
In addition, the time horizon T determines the times of augmentations performed sequentially patch. Therefore the agent can output more than one augmentation for a patch by setting > 1. In our PAA model, we set time horizon 7 1, i.e., the augmentation policies take actions at every time step for saving computational cost and avoiding over-regularization Even in the single-step decision-making process (i.e., T 1). RL
still effectively search for the optimal policy as illustrated (Mnih et al. 2016 Singh et al., 2000). addition, we use the SGD optimizer with an initial learning rate of Te-4 to train the actor network and the critic network.
E MORE GRAD-CAM VISUALIZATION
In this section, provide more Grad-CAM (Selvaraju al. 2017) results of ResNet-50 models trained using baseline augmented data, AA (Cubuk et 2018 and our proposed PAA, respectively, shown in Figure 5 The visualization results demonstrate that our proposed PAA improves the localization ability of the target network and tends to help the target network focus on more parts of the foreground object. short, our proposed PAA make the target network focus the important or representative regions closely related to the class within an image.
IMPACT OF THE INITIALIZATION OF THE FEATURE EXTRACTOR
We further study the impact of the initialization of the feature extractor as shown in Table I1 We first observe that the performance of the Non-pretrained ResNer-18 scheme (using non-pretrained ResNer-18 as the state/observation extractor) still much better than the baseline. In other words, our agents do not rely on the pre-trained ResNet to produce features to be used as the state or observation. We also notice that the performance the Von-pretruined RexNet-18 scheme is slightly lower than the Pretrained ResNet-18. This may be because that at the very early stage of the training, the features extracted by Non-pretrained ResNet- are not as good as Pretrained ResNer-18 in practice.
G DETAILS OF COMPUTATIONAL COST
The computational cost of our proposed PAA estimated on GeForce GTX 1080 Ti while AA
Cubuk et al. 2018) and AdvAA (Zhang et al. 2019b) are NVIDIA Tesla P100. Besides, FastAA
Lim et al. 2019) is estimated on NVIDIA Tesla VI0O. The computational cost is mainly used for searching policies and training the target networks.
16

---

Under review as a conference paper at ICLR 2022
Baseline
AA
PAA
Baseline
AA
PAA
Figure 5: We give more visualizations of augmentation and Grad-CAM((Selvrajuu et al. 2017) re- sults of baseline. AA(Cubuk al. 2018) and our PAA. PAA performs rich diversity of DA and tends to help the target model focus on the class-related features.
17


Reviews:
[{"review_id": "hVuxmq_u3ec", "reviewer": "Reviewer_ci48", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ It seems effective to utilize the MARL algorithm to reduce the computational complexity for the multiple patches.\n+ As a result, the computation cost of the proposed algorithm becomes much reduced when it is compared to the previous auto-augmentation algorithms. Furthermore, the algorithm can select the optimal combinations at every epoch without any additional computations, which can improve its generality across the various tasks.\n\nWeaknesses & Questions\n- About state and observation. In the proposed algorithm, the state and the observation are obtained by the deep feature from the trained model. My question is the image to obtain the deep feature. When the image is the original image before the determined augmentation, the state and the observation are not affected by the selected action. Then, the best action can be determined just by one simple simulation, and no reinforcement learning algorithm is necessary to find the optimal augmentation. If the augmented images are fed into the model again, the computation cost cannot be reduced as shown in the experiments. What is the detailed sequence to obtain the deep feature?\n- Since the deep feature is utilized, its invariant property can affect the determination of the optimal augmentation. In other words, when the trained model is robust to the translation shift, the deep features from the model would be similar even after various translation shifts. Then, the training loss becomes also similar, which results in no selection of translation augmentation. This means that the algorithm would select only the augmentation method that is not considered in the initial model, and this can work as one of the biased optimizations for the optimal augmentation methods. Is there any analysis about the robustness of the initial model for the deep features?\n- In the experiments, the proposed algorithm is only compared with the out-of-date algorithms, so its performance cannot be validated well. I recommend the authors add the state-of-the-art studies related to auto-augmentation and the learning-based augmentation algorithms, which include Saliency-mix[1] and Co-mixup[2].\n- What would be the limitation of the proposed algorithm?\n\n[1] Uddin et al. \"SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization\", ICLR2021\n[2] Kim et al. \"Co-Mixup: Saliency Guided Joint Mixup with Supermod-ular  Diversity\", ICLR2021\n", "summary_of_the_paper": "This paper target the task of automatically determining the best augmentation method to obtain improved accuracy. While the previous related studies focus on the image-level augmentation and ignore the semantic information of the augmented images, the proposed algorithm augments the grid-wise patches of the given input with the preserved semantic information. To overcome the enlarged number of combinations to consider all the patches, the algorithm utilizes the MARL algorithm with the unified reward function. By MARL algorithm, the number of parameters can be reduced and the training speed can be much improved, compared to the previous auto-augmentation methods. Through the image classification and fine-grained image recognition tasks, the proposed algorithm was validated, and it shows the state-of-the-art performance among the compared methods.", "summary_of_the_review": "As I said above, I have some questions about the necessity of the reinforcement learning-based algorithm in the framework. In addition, the comparison with the state-of-the-art algorithms should be added for the validation. I will change my score when I solve the above issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_7vX2Z-3OAl", "reviewer": "Reviewer_cJ8q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n- 1.\tThe paper conducted a comprehensive evaluation of their approach over several datasets. They achieve gains over the compared baseline method validating the efficacy of their proposed approach.\n- 2.\tThe approach considers patched based augmentation, which is sufficiently different/novel from existing work.\n\n### Weaknesses:\n- 3.\tThe motivation of the approach is somewhat lacking. For example, it is unclear why MARL on patches is much faster. I think the paper could benefit from motivating more on why MARL and what is its benefit with respect to the task. \n- 4.\tThe paper did not reference or compared to a key related work, “Differentiable automatic data augmentation” [A]. Based on its relevance, the paper should discuss and contrast with this work, as they also have minimal search time. Specifically, comparison in Table 2 and Table 3 would strengthen the paper.\n- 5.\tHow is the model architecture of the actor/critic network determined? Also, what ranges of hyperparameters for MADDPG are considered? I saw the hyperparameters for the target model but not MADDPG. RL is known to be sensitive to hyperparameters, I wonder if the authors will release the code and pre-trained models for reproducibility. \n- 6.\tAblation with independent agents as a baseline would be interesting. It would further motivate the use of MADDPG and the multi-agent setting.\n \n### References\n[A] Li, Yonggang, et al. \"Differentiable automatic data augmentation.\", Proc. ECCV, 2020.\n\n\n", "summary_of_the_paper": "This paper proposes an automatic data augmentation approach. Different from existing works, they proposed to augment patches in the image rather than the whole image. The approach is formulated as a multi-agent reinforcement learning problem. They empirically show the effectiveness of their approach across several image classification datasets.", "summary_of_the_review": "The main concern with the paper is with the motivation and missing reference/comparison with a key existing work. \nCurrently, I recommend a weak reject. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3AIGA5R3Ezx", "reviewer": "Reviewer_XdSw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n  - S1: Ablation studies and theoretical explanations\n    motivate the choice for multi-agent reinforcement\n    learning reasonably well.\n  - S2: The efficiency of the proposed method is\n    investigated very clearly in Section 4.3. It seems\n    considerably more efficient than many other\n    learning-based DA methods.\n  - S3: The patch-wise data augmentation is a powerful idea\n    which can spark a large body of interesting follow-up\n    research.\n\n\n## Limitations\n\n  - L1: While this is not the main focus of the paper, the\n    empirical improvement over RL baselines like Fast\n    AutoAugment and non-RL baselines like RandAugment is\n    relatively small.\n  - L2: (Minor) While multiple datasets are used for\n    evaluation, only two Resnet variants are investigated\n    for ImageNet. It would be nice to see an evaluation of\n    the proposed data augmentation procedure on other NN\n    architectures as well on ImageNet, but I also\n    understand that the computational costs of this can be\n    quite large, so it may not always be possible.\n  - L3: (Minor) The agents rely on a pre-trained ResNet to\n    produce features to be used as observations.\n\n\n## Suggestions (Per-Section)\n\n### Related Work\n\n  - This is minor, but it may be worth briefly mentioning\n    [hong2019patch]. While their work doesn't do patch\n    level data augmentation, and instead \"patches\" objects\n    from an existing bank onto training samples, it still\n    has connections to things like CutMix that make it\n    relevant.\n  - Some recent papers on automatic augmentation are\n    missing and would be worth discussing. This includes\n    [DADA], [DDAS], [hendrycks2019augmix] and\n    [mueller2021trivialAugment]. In particular, Mueller and\n    Hutter's paper highlights an interesting limitation of\n    their otherwise robust baseline, which is that it\n    doesn't work too well for object detection, an area\n    where the current method may have the upper hand!\n  - Similarly, there are some further papers, such as […]\n    that perform a kind of \"local\" (though not strictly\n    patch-based) data augmentation by \"pasting in\" objects\n    into the 3D point cloud.\n\n\n### Method\n\n  - \"directly applying reinforcement learning (RL)\n    algorithm to …\" => \"algorithms\"\n  - \"In MARL configuration\" => \"In the MARL configuration\"\n  - Towards the end of Section 3.2, it may make more sense\n    to replace the \"Until all patches\" to \"Once all\n    patches…\" or maybe \"After\".\n  - Q1: Can an agent output more than one augmentation for\n    a patch? Would it be possible for a patch to, e.g., get\n    rotated AND colorized?\n  - Q2: Given the more \"local\" nature of this augmentation\n    method, have you considered evaluating it on more\n    \"local\" tasks, such as object detection and semantic\n    segmentation. Perhaps it could make a bigger impact\n    there.\n  - Q3: Given that the parameters of actor networks of all\n    agents are shared (see above Eq. (5)), does that mean\n    that each actor's policy is identical, and the only\n    difference between the actors is what observation each\n    one gets (the local patch features)?\n\n\n### Experiments\n\n  - At the start of Section 4.1, \"Exactly, we focus\" =>\n    \"Specifically, we focus\"\n  - In \"Policy Visualization\", \"important levels\" ->\n    \"importance levels\"\n  - In Section 4.3 I would say the \"main methods for\n    reducing the computational cost\", since what you next\n    describe are methods for reducing the cost, not what\n    one would typically call reasons.\n  - Table 3 is a little hard to follow because of the\n    different hardware employed by each column. After\n    staring at it for a while it's becoming a bit clearer,\n    but at first glance it was a bit confusing.\n  - Q4: Do the times from Table 3 include the inference\n    time required to compute the agent state and\n    observations with the auxiliary ResNet-18?\n\n\n### Conclusion\n\n  - The parallels to vision transformers are very\n    interesting and help give the reader a broader\n    perspective on possible applications of the current\n    paper.\n  - (Minor) The word \"Furthermore\" is repeated for two\n    almost consecutive sentences. Using something like\n    \"additionally\" to start one of the sentences can\n    improve the flow of the paragraph a little.\n\n\n## References\n\n  - [hong2019patch]: Hong, Sungeun, Sungil Kang, and\n    Donghyeon Cho. \"Patch-level\n  augmentation for object detection in aerial images.\"\n  Proceedings of the IEEE/CVF International Conference on\n  Computer Vision Workshops. 2019.\n  - [cubuk2020randaugment]: Cubuk, Ekin D., et al.\n    \"Randaugment: Practical automated\n  data augmentation with a reduced search space.\"\n  Proceedings of the IEEE/CVF Conference on Computer Vision\n  and Pattern Recognition Workshops. 2020.\n  - [DDAS] Liu, Aoming, et al. \"Direct Differentiable\n    Augmentation Search.\" arXiv preprint arXiv:2104.04282\n    (2021).\n  - [DADA] Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy\n    M. Hospedales, Neil Martin Robertson, and Yongxing\n    Yang. DADA: differentiable automatic data augmentation.\n    CoRR, abs/2003.03780, 2020.\n  - [mueller2021trivialAugment]: Müller, Samuel G., and\n    Frank Hutter. \"TrivialAugment: Tuning-free Yet\n    State-of-the-Art Data Augmentation.\" ICCV 2021.\n  - [hendrycks2019augmix]: Hendrycks, Dan, et al. \"Augmix:\n    A simple data processing method to improve robustness\n    and uncertainty.\" arXiv preprint arXiv:1912.02781\n    (2019).\n", "summary_of_the_paper": "  The paper proposes an evolution of the traditional\n  pipeline of image data augmentation used to reduce ML\n  model overfitting.\n\n  Instead of applying transformations such as shear,\n  rotate, CutOut, etc. at the image level, the proposed\n  technique divides the images into a fixed grid and\n  applies a potentially different transformation to each\n  cell. The problem of selecting a transform for each cell\n  is cast as a multi-agent RL (MARL) task, and the agents\n  learn as the main network trains within a (multi-agent)\n  Advantage Actor Critic framework.\n\n  The agents use a shared reward mechanism, with the reward\n  defined as the difference between the loss on the\n  augmented sample and the original loss.\n\n  The regular grid is fixed for a dataset and\n  hyperparameters like the magnitude of the augmentations\n  follow a fixed schedule; the agents only pick _which_\n  augmentation to apply on a patch.\n\n  Experiments performed on CIFAR-{10,100}, ImageNet,\n  CUB-200-2011, Stanford Cars, and FGVC-Aircraft show\n  relatively small but very consistent improvements in\n  terms of image classification accuracy. Different design\n  choices (MARL vs. single-agent vs. random, grid size,\n  etc.) are ablated and discussed in detail.\n", "summary_of_the_review": "  The paper proposes a new way of looking at the classic\n  task of image augmentation for classification. Instead of\n  learning an image-level augmentation policy using RL, the\n  proposed approach uses multi-agent RL to learn\n  patch-specific augmentations. While the benefit of\n  multi-agent RL is relatively small, it seems significant\n  and the general idea of learning to apply different\n  augmentations to different parts of the image is\n  interesting and, to the best of my knowledge, new. The\n  idea to apply image augmentations at a local level is the\n  core contribution of this paper, in my opinion.\n\n  While it would be interesting to see how this technique\n  works on tasks which, unlike image-level classification,\n  are themselves more \"local\", such as semantic\n  segmentation or detection, overall the proposed approach\n  is evaluated thoroughly.\n\n  The writing is clear and while the authors don't\n  explicitly mention an intent to publish their source\n  code, it should be possible to implement the proposed\n  approach just based on the description from the paper. As\n  such, I recommend accepting the paper to ICLR.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1TQ6xUTdMSn", "reviewer": "Reviewer_CpsG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Over all, this paper is well written and technically sound, the experimental evaluation is comprehensive and the results are compelling, but I have several concerns:\n1. The most important contribution of this paper in my mind is the new search space focus on image patches for data augmentation. It seems like an incremental job relative to mixup series or cutout series.\n2. The proposed PPA formulate the task as a multi-agent reinforcement learning problem, and the results show it can find better policy than random, but [1] demonstrate that only use reinforcement learning can not get better policy than random baseline. The two experimental results are contrary.\n3.The calimed SOTA results is inappropriate, methods with better experimental results were not compared. For example, the paper  compare with AdvAA on the  computational cost, but not coapre on the accuracy.\n4.Another question is that recent work [2] with careful designed augmentation policy and advanced training strategy can get much better results, It is better to compare with it to give more insights.\n\n[1] Cubuk E D, Zoph B, Shlens J, et al. Randaugment: Practical automated data augmentation with a reduced search space\n[2] Wightman R, Touvron H, Jégou H. ResNet strikes back: An improved training procedure in timm", "summary_of_the_paper": "This paper proposed a fine-grained automated data augmentation approach, Patch AutoAugment (PAA), which tries to increase diversity in local regions by divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. The proposed PAA considers the task as a multi-agent reinforcement learning problem,  and adopt a multi-agent reinforcement learning\nalgorithm to automatically search for the optimal augmentation policies by considering the contextual relationship between the patches. They verify the proposed method on many classification and fine-grained recognition dataset(CIFAR10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars and FGVC-Aircraft). The experiments show a good result and visualization results provide some insights that the PAA  help the target network to localize more class-related cues.\n", "summary_of_the_review": "According to above analysis and concerns, I suggest borderline. If the author can solve my concerns, I will consider change the score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed an approach to search image augmentation policies. The paper formulates this problem as a cooperative multi-agent decision-making problem, which is interesting. The paper received 3 borderline accept and 1 borderline reject ratings. The reviewers originally had multiple concerns regarding the necessity of RL-based approach, lacking references, and additional experiments, and the authors responded to some of the concerns of the reviewers reasonably. However, none of the reviewers ended up strongly supporting the paper, staying with their ratings.

The RL formulation of the problem is interesting, but it requires multiple rounds of the target network training due to its nature (i.e., it is not an end-to-end approach). The paper misses some details on how exactly the patch-wise RL-based augmentation works and it requires additional hyperparameters for the selection of patch size and shape. It is also unclear how this RL-based method is conceptually superior to previous augmentation approaches and the empirical results are not strong enough, as some of the reviewers also pointed out.

Although the paper has interesting ideas and the AC also think the paper has some merit, the senior AC finds the technical contribution of the paper weaker than the others. We unfortunately need to recommend the rejection of the paper.
Decision: Reject