MixPro: Data Augmentation with MaskMix and Progressive Attention Labeling for Vision Transformer
Authors: Qihao Zhao
Yangyu Huang
Wei Hu
Fan Zhang
Jun Liu
Venue: ICLR 2023 poster (2023)

Abstract: The recently proposed data augmentation TransMix employs attention labels to help visual transformers (ViT) achieve better robustness and performance. However, TransMix is deficient in two aspects: 1) The image cropping method of TransMix may not be suitable for vision transformer. 2) At the early stage of training, the model produces unreliable attention maps. TransMix uses unreliable attention maps to compute mixed attention labels that can affect the model. To address the aforementioned issues, we propose MaskMix and Progressive Attention Labeling (PAL) in image and label space, respectively. In detail, from the perspective of image space, we design MaskMix, which mixes two images based on a patch-like grid mask. In particular, the size of each mask patch is adjustable and is a multiple of the image patch size, which ensures each image patch comes from only one image and contains more global contents. From the perspective of label space, we design PAL, which utilizes a progressive factor to dynamically re-weight the attention weights of the mixed attention label. Finally, we combine MaskMix and Progressive Attention Labeling as our new data augmentation method, named MixPro. The experimental results show that our method can improve various ViT-based models at scales on ImageNet classification (73.8% top-1 accuracy based on DeiT-T for 300 epochs). After being pre-trained with MixPro on ImageNet, the ViT-based models also demonstrate better transferability to semantic segmentation, object detection, and instance segmentation. Furthermore, compared to TransMix, MixPro also shows stronger robustness on several benchmarks.

Published as a conference paper at ICLR 2023
MIXPRO: DATA AUGMENTATION WITH MASKMIX AND PROGRESSIVE ATTENTION LABELING FOR VISION TRANSFORMER
Qihao Zhao', Yangyu Huang", Wei Hu' Fan Zhang' Jun Liu Beijjing University of Chemical Technology. China Microsoft Research Asia, China "Singapore University of Technology and Design, Singapore
ABSTRACT
The recently proposed data augmentation TransMix employs attention labels to help visual transformers (ViT) achieve better robustness and performance. However, TransMix is deficient in two aspects: 1) The image cropping method of TransMix may not be suitable for vision transformer. 2) At the early stage of training, the model produces unreliable attention maps. TransMix uses unreliable attention maps to compute mixed attention labels that can affect the model. To address the aforementioned issues, we propose MaskMix and Progressive Attention Labeling (PAL) in image and label space, respectively. In detail, from the perspective of image space, we design MaskMix, which mixes two images based on patch-like grid mask. In particular, the size of each mask patch is adjustable and is multiple the image patch size, which ensures each image patch comes from only one image and contains more global contents. From the perspective of label space, we design PAL, which utilizes a progressive factor to dynamically re-weight the attention weights of the mixed attention label. Finally, we combine MaskMix and Progressive Attention Labeling as our new data augmentation method, named MixPro. The experimental results show that method can improve various ViT- based models at scales on ImageNet classification (73.8% top-1 accuracy based on DeîT-T for 300 epochs). After being pre-trained with MixPro on ImageNet, the ViT-based models also demonstrate better transferability to semantic segmentation, object detection, and instance segmentation. Furthermore, compared to TransMix, MixPro also shows stronger robustness on several benchmarks.
INTRODUCTION
Transformers Vaswann et al. 2017) have revolutionized the natural language processing (NLP) field and have recently inspired the emergence of transformer-style architectures in the computer vision (CV) field, such as Vision Transformer (ViT) (Dosovitskiy et al.
2020). These methods design with
competitive results in numerous CV tasks like image classification (Touvron al., 2021a Yuan et al.)
2021; Wang et al. 2021 Liu et al.
2021] 202T] Touvron et af.) 2021b Ali et al. 2021 ). object detection
Dai et al.] 2021 Carion al.) 2022 Zhu et al. (0200) and image segmentation (trudel et al.) 202 Wang et al., 2021 Liu et al. 2021). Previous research has discovered that
FFng et al. [022
ViT-based networks are difficult to optimize and can easily overfit to training data with many images
(Russakovsky et al.) 2015), resulting in significant generalization gap in the test data. To improve the generalization and robustness the model, the recent works (Dosovitskiy et al.] 2020 Touvron et al.
2021a; Yuan et al. 2021 Wang et al., 2021: Liu et al.) [021] Touvron et al. 202 Ali 2021) employ data augmentation (Zhang et al. 2017) and regularization techniques (Szegedy
ct al.
ctt al. 2016) during training. Among them, the mixup-base methods such as Mixup (Zhang et al.
2017). ) CutMix (Yun et al. 2019) and TransMix (Chen et al. 202T) are implemented to improve the generalization and robustness of ViT-based networks. For the CutMix, patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. Furthermore, TransMix based on CutMix considers that not all pixels are created equal.
Corresponding author: huwei @ buct.edu.cn Fan Zhang is with the College of Information Science and Technology and the Interdisciplinary Research Center for Artificial Intelligence, Beijing University of Chemical Technology, Beijing 100029.

---

Published as a conference paper at ICLR 2023
Input
Image Space
Label Space
Alto
À = a A Artn + (1-
area
(a) TransMix
(b) MixPro
Figure 1: Comparison between TransMix (Chen et al. 2021 (a) and our proposed MixPro (b). 1) For image space, TransMix shares the same cropped region with CutMix (Yun et al. 2019), which results in patches containing different regions from the two images (patches colored red). Differently, as shown on the right of the figure, MixPro mixes patches using a patch-like mask. The size of the mask patches is the multiple of the image patches. This enables each patch of the mixed image to come from only one image (patches colored yellow and blue). 2) For label space, TransMix computes À by Aatin- contrast, we propose progressive attention labeling that dynamically re-weights Aarea and Aatta using progressive factor (a).
Then it exploits VIT's attention map to re-assign the weight to the mixed label rather than applying the proportion of the cropped image area.
Nevertheless, TransMix (Chen et al. 2021) has the following drawbacks for vision transformer: (1) TransMix and CutMix share the same region-level cropped images as input. However, ViT-based models naturally have global receptive fields from self-attention (Dosovitskiy al. 2020), so the region-based mixed images may provide insufficient image contents. (More details are Sec. I (2) As shown in Fig. (a)TransMir provides cropped patches with sharp rectangular borders that are clearly distinguishable from the background (viewed as red patches). The ViT-based models may naturally be curious about the cropped patch and then pay attention that patch, resulting in basic weight of attention regardless of whether the patch contains useful information (Chen al. [202T). (3) TransMix employs attention map to re- re-weight the confidence for the mixed targets. However, attention maps may not always be reliable during the training process. For example, the beginning of the training, the model has no representation capability, and the attention maps gained are unreliable. In addition, it is possible to obtain difficult samples using massive data augmentation strategies, and the attention map is also unreliable, At this point, reassigning the mixed labels utilizing low-confidence attention map will generate noisy mixed labels.
To this end, we propose novel data augmentation method, MixPro, to tackle the above issues from the perspective of image space and label space, respectively. Our approach is presented in Fig. (b). In detail, from the perspective of image space, we designed MaskMix, which is inspired by the mask strategy of MAE (He al., 2022). Our MaskMix replaces the masked patches of image with visible patches of another image to create a mixed image. In particular, the scale of each mask patch adjustable and is a multiple of the image patch size. this way, each image patch comes from only one image (viewed as yellow and blue patches in the figure). In addition, the mask decomposition of pictures can take into account both region and global contents. From the perspective of label space, we designed Progressive Attention Labeling (PAL), which utilizes progressive factor (a) to dynamically re- se-wegght the attention weight of the mixed attention label. The progressive factor (a) provides indirect measure of the confidence of the attention map for the mixed sample, to trade-off the attention proportional weight (Aattr) and the area proportional weight (Aarea). In conclusion, we combine MaskMix and Progressive Attention Labeling, namely MixPro, our data augmentation strategy to improve the generalization and robustness of ViT-based models.
In experiments, we demonstrate extensive evaluations of MixPro on various VIT-based models and tasks. MixPro exhibits greater performance gains than TransMix for all listed ViT-based models. Notably, MixPro can further bring an improvement of 0.9% for DeiT-S. Moreover, we demonstrate that if the model is first pretrained with MixPro on ImageNet, the superiority can be further transferred onto downstream tasks including object detection, instance segmentation, and semantic segmentation.
2

---

Published as a conference paper at ICLR 2023
In terms of robustness, compared to TransMix, MixPro also shows stronger robustness on three different benchmarks.
Overall, summarize contributions follows:
We propose new data augmentation method, MixPro, address the shortcomings of TransMix from the perspective of image space and label space, respectively.
From the perspective of image space, MixPro ensures that each image patch comes from only one image and contains more global contents. From the perspective of label space, MixPro utilizes a progressive factor to dynamically re- re-weight the attention weight of the mixed attention label.
In experiments, demonstrate extensive evaluations of MixPro various ViT-based models and downstream tasks. It boosts Deit-T achieving 73.8% and Deit-S achieving 81.3% on ImageNet- K. Furthermore, compared to TransMix, MixPro also shows stronger robustness on three different benches.
RELATED WORK
Vision Transformers (ViTs), Transformers were initially proposed for sequence models such as machine translation (Vaswani al. 2017). Inspired by the success of transformers in NLP tasks, Vision Transformer (ViT) (Dosovitskiy et al. 2020) attempted to apply transformers for image classification by treating an image as sequence of patches with excellent results compared to even state-of-the-att convolutional networks. DeiT (Touvron et al. 2021a) further extended ViT using novel distillation approach and a powerful training recipe. Based on the success of ViT, numerous ViT-based models have emerged (Yuan et al. 20221] Wang et al. [021]] Liu et al. 2021 Touvron et al. 2021b] Ali et al.) 2021 1| Yang et al. (021). PVT (Wang et al. 2021) developed a progressive shrinking pyramid and a reeiial-rddction attention layer to obtain high-resolution and multi-scale feature maps under limited computation/meent
resources. XCIT (Ali et al. 202T) built
their models with cross-cor crosscoovariance attention its core component and demonstrate the effectiveness and generality of our models on various computer vision tasks. Swin (Liu et al., (021)) proposed the shifted window-based self-attention and showed the model problems. These ViT-based models have achieved good results not only for classification tasks (Deng
effective and efficient on vision
al. 2009), but also for object detection (Fang al. 2021 Dai al., [021] Carion et al. [220 Zhu
iet al. 2020) and image segmentation (Strudel et al. 2021 Wang et al. 2021 Liu et al. 2021) tasks.
However, these ViT-based models rely on MixUp-based data augmentation to enhance generalization in case of insufficient data.
Mixup-d data augmentation. Mixup (Zhang et al. 2017) was the first work propose
interpolation of two images and their labels to augment the training data. Subsequent variants of the Mixup appeared (Yun et al.] 2019 Chen et al.) [2021] Verma et al., (2019] Baek et al. 202T Cascante-Bonille et al. 2002 Kim et al. 2020 Walawalkar et al. 2020 Uddin et 2020). They can
be mainly divided into two groups: global image mixture (c.g. Manifold Mixup (Verma et al. 2019). GridMix (Back et al., (2221). TokenMix (Liu et al.) 2022). and PatchMix Cascante-Bonila et al.
2021)), and local/region image mixture (e.g. CutMiix (Zhang et al. 2017). Puzzle-Mix (Kim et al. (Walawalkar et al.) 2020), SaliencyMix (Uddin et al. 2020), and Mix Token (Jiang et al. (021)). Manifold Mixup (Verma et al. 2011)) proposed to train neural networks on interpolations of hidden representations. GridMix (Baek et al. 2021) was composed of two local constraints: local data augmentation by grid-based image mixing and local patch mapping constrained by patch-level labels. In CutMix (Yun al.) 2019), patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. Attentive- CutMix (Walawalkar et al. 2020) enhanced CutMix by choosing the most descriptive regions based on the intermediate attention maps from feature extractor, which enables searching for the most discriminative parts in an image. MixToken (Jiang et al., 202 is a modified version of CutMix operating on the tokens after patch embedding. Among them, Mix and CutMix are most successful in improving the performance of ViTs TTovrron et al. 2021a). Furthermore, TransMix (Chen et al. 2021) based on CutMix continues to improve the generalization and robustness of ViT-based models by reassigning the ground truth labels with transformer attention guidance. TokenMix (Liu et al. 2022) utilizes pre-trained teacher model to generate attention maps for guiding the mixed labels.
2020
3

---

Published as a conference paper at ICLR 2023
All in all, proposed MixPro differs significantly from the above MixUp-based methods. First of all. CutMix, MixToken and TransMix focus on region contents, whereas MixPro mixes images with adjustable mask patches and may contain more global contents. Secondly, TransMix mixes image patches may generate noisy attention maps, while MixPro generates clean attention maps for mixing labels. The last and most important the fact that the model attention map during training is not always reliable. MixPro solves this issue by using progressive factors to dynamically recalculate the attention weight of the mixed attention label without pre- pre-trained model to generate it.
that, from the viewpoint of label space, TransMix neglects
3 METHOD
3.1 BACKGROUND
Multi-head self-Attention. Multi-head self-Attention as introduced by ViTs (Dosovitskiy et al. 2020), firstly divides and embeds an image x RW*H to patch tokens Xpatches E R R where N is the number of tokens, each of dimensionality d. It aggregates the global information by class token X.s E xx Then ViTs operate on the patch embedding z = [xels: Xpotches E (I++N)) Given Transformer with h attention heads and input patch embedding z, the class attention for each head can be formulated as:
Xele W 9s
(1)
k= W ky
A' Softmooti k/ d/hh. A (A [J,
where W are weight matrices for q and k. A E [0, 1J is the attention map from the class token to the image patch tokens. It is summarized as which patches are most valuable for the final classifier. When there multiple heads in the attention, obtain them by simply averaging over all heads of the attention.
TixnsMix. TransMix (Chen et al. 2021) calculates Aattn (the proportion for mixing two labels) with the attention map A and a binary mask M E {0, 1) X from CutMix (Yun et al. 2019):
Xatin = A- (M).
(5)
where 1. (-) denotes the nearest-neighhor interpolation downsampling that can transform the original M from H W into N pixels.
3.2 MIXPRO
Our approach, MixPro, consists of MaskMix and progressive attention labeling to improve the performance and robustness of ViT-based models from the viewpoint of image space and label space, respectively. Next, we will describe these two methods in detail.
MaskMix. Let E R RWxH xC denote training image and let be its corresponding label. ViT (Dosovitskiy et al. 2020) regularly partitioned high-resolution image x into N patches of fixed size of P Pimage X P image pixels (N W/ W/P.a Pimage x H Pimage) and then embeds these to tokens. In our method, we first divide the grid mask M E {0, 13 into S W/Pmask X H/Pmask patches, resulting each mask patch region of size P Pmask x Pmast- particular, to make each image patch come from only one image, we ensure that the size of mask patches (Pmesk) is multiple of the image patch size (Psmage). Then we select [S T regions mask, where
= (e,a(B,
and the value in the selected mask region is set to 1. In all our experiments, we follow CutMix (Yun) al. 2019) set (8 to 1, the is sampled from the uniform beta distribution Beta(1,1). With the mask
4

---

Published as a conference paper at ICLR 2023
M. for samples Tı, 2, and their corresponding labels yes By, utilize the mask M E (0, 1 to generate their mixing samples (i: y/)
ā = + (1 M) 2z.. Xarea + YE + (1 Aarea) @ Fj+
(7)
where Xarea W Σ M(i,j)/(W x H), binary mask filled with ones is represent as 1 and @ is element-wiss wise multilicatiin.. In this way, as illustrated in (b). image patches are aligned with mask patches when mixing images to ensure that each image patch comes from only one image.
Progressive Attention Labeling. However, the attention map A may not always be accurate for each mixed sample. At the early stage of the training, models have poor representational ability,, and the attention map A may not be reliable for some difficult mixed samples. At this point, the mixed labels calculated with Xattn would limit the performance of ViT.
Since blindly using attention maps to reassign mixed labels is not reliable. The focus is on designing a progressive factor (a)) which could provide indirect measure of the confidence of the attention map for the mixed sample, to trade-off the attention proportional weight (Aaun) and the area proportional weight (Narea). For neural networks with ground-trudh distributions, cross-entropy measures the epistemic uncertainty of the model (Kendall & Gal 2017). The more certain the model is, the more reliable the attention map is obtained. There, we employ the cosine similarity of the model output to the ground-truth labels as proxy of cross-entrope to measure whether mixed sample can obtain high-confidence attention map. We calculate the cosine similarity, i.e.,
p.Y |
d(p. y)
|p||
where p is the model's output softmax probability. Since both and y are non-negative vectors, the range of d is E [0, 1]. Ideally, the cosine similarity closer to the samples better the network fits the mixed sample and it can also produce high-confidence attention map. Therefore, we use the cosine similarity as our progressive factor a,
and the À written as:
A= a Aattn + (1 ox) Xarea
where À is the proportion to instead of Xarea in Eq. (7) for mixing two labels. So far, we propose Progressive Attention Labeling (PAL), which employs the progressive factor (a) to dynamically re-weight the attention weight of the mixed attention label.
3.3
DISCUSSION
100
MuXPr
MixPro Tanshii
Top-1 Emro((%)
0.8
0.6
0.6
0.4
0.4
0.2
0.2
20
0.0
300
0.0
100
0
0
100
200
300
100
200
300
Epoch
Index
Epoch
Epoch
(a) Trends a
(b) a mini-batch
(c) Loss
(d) Top-1 Error
Figure 2: Visualization of progressive factor (a) and generalization analysis of our MixPro and TransMix. We employ Mixpro and TransMix based on Deit-S on ImageNet-1k for 300 epochs. Figures (a) and (b), demonstrate the visualization of the progressive factor (a:). Figures (c) and (d) depict their loss and top-1 validation error on ImageNet- 1k. MixPro provides improved generalization compared to TransMix.
Visualization of progressive factor (a). Fig. (a) indicates the trend of the mean progressive factor (a) of all samples with the epoch. We can observe that since the model will gradually fit the training samples, the progressive factor will grow slowly with epoch. Fig. (b) indicates the value of the progressive factor a mini-batch at epoch 300. We can find that since the model learns to different degrees for different samples, the progressive factors obtained by the model will be different, so
5

---

Published as a conference paper at ICLR 2023
method can flexibly adjust the weight of the attention map in the mixed labels according to the progressive factors.
Generalization analysis. We compare and analyze the generalization of MixPro and TransMix. Fig. (c) and (d) depict their loss and top-1 validation error on ImageNet- Ik for 300 epochs based on Deit-S. We can notice that although MixPro has larger loss in training compared to TransMix, has a smaller top-1 error in verification. This demonstrates that Deit-S trained with MixPro can achieve better generalizability.
EXPERIMENT
In this section, we primarily evaluate MixPro for effectiveness, transferability, and robustness. We first study the effectiveness of MixPro ImageNet- Ik classification in Sec. 4.1 Next, we show the transfer ability of a MixPro pre-trained model when it is fine-tuned for downstream tasks ( subTran). In the Sec
also show that Mixpro improve the model's robustness compared to TransMix.
4.1 IMAGENET CLASSIFICATION
Table 1:
Compared TransMix, MixPro provides better performance wide range of model variants, e.g. DeiT, PVT, CaiT, XCIT Swin on ImageNet-IK classification. All the baselines are reported in TransMix (Chen al. 202T)
Models
Params
#FLOPs
Top-1 Acc(%)
Top-1 Acc(%)
Top-1 Acc(%)
+TransMix
+MixPro
DeïT-T Touvron et al.
2021a
5.7M
1.6G
72.2
72.6
73.8+++1.2)
PVT-T Wang et
2021
13.2M
1.9G
75.1
75.5
76.7+(+1.2)
XCiT-I
al.
2021)
2.3G
79.4
80.1
81.2+(+1.1)
CA-Swin T Liu al.
2021
28.3M
4.2G
81.6
81.8
82.8+(+1.0)
CaiT-XXS
17.3M
3.8G
79.1
79.8
80.6+(+0.8)
DeïT-S Touvron et al.
2021a
22.1M
4.7G
79.8
80.7
81.3+(40.6)
PVT-S ( Wang et al.
2021
24.5M
3.8G
79.8
80.5
81.2+(+0.7)
XCIT-S Ali et
2021
26M
4.8G
82.0
82.3
82.9+(+0.6)
CA-Swin-S LLu
2021
49.6M
8.5G
82.8
83.2
83.7+(+0.5)
PVT-M Wang
2021
44.2M
6.7G
81.2
82.1
82.7+(+0.6)
PVT-L
Wang ct al.
202T
61.4M
9.8G
81.7
82.4
82.9+(+0.5)
XCTT-M Ali et al.
202T
84M
16.2G
82.7
83.4
84.1+(+0.7)
DeïT-B
Touvron et al. 2021a
86.6M
17.6G
81.8
82.4
82.9+(40.5)
XCIT-L Ali et al.
2021)
189M
36.1G
82.9
83.8
84.7+(+0.9)
Implementation Details. For image classification, we evaluate on ImageNet-IE IK (Deng et al. 2009). which contains 1.28M training images and 50K validation images from 1,000 classes. We examined various baseline vision Transformer models using in TranMix (Chen et al. 2021) including DeiT (Touvron al. 2021a), PVT (Wang al.) (022). CaiT (Touvron al. 2021b), XCIT (Ali al.) 2021), CA-Swin (Liu et al. 2021 Chen et al. 2021). Specifically, CA-Swin replaces the last Swin (Liu al. 2021 block with classification attention (CA) block without parameter overhead, which makes it possible to generalize TransMix and Mix onto Swin. The training schemes will be slightly adjusted the official papers' implementation. These experiments' settings follow TransMix/Chers et al.) (021). We employ an Adam W optimizer for 300 epochs expect that XCIT (Ali et al.) (021) and CaiT (Touvron et al.) (021b) reported 400 epochs. Following the TransMix, we use cosine decay learning rate scheduler and 20 epochs of linear warm-up. Similarly, batch size of 1024, an initial learning rate of 0.001, and a weight decay 0.05 are used. All baselines have already contained the carefully tuned regularization methods reported in TransMix (Chen et al. 2021 except for repeated augmentation (Hoffer et al.) 2020) and EMA (Polyak Juditsky 1992) which do not enhance performance. Following TransMix(Cher)) al. 2021), the attention map A in Eq. 5 can be obtained as an intermediate output from the multi block.
multi-head head self-attention layer of the last transformer

---

Published as a conference paper at ICLR 2023
4.1.1 COMPARISON WITH TRANSMIX ON A WIDE RANGE OF IIT-BASED MODEL.
As shown in Table 1 MixPro can enhance the top-1 accuracy on ImageNet for all the listed ViT-based models. Compared TransMix, MixPro achieves better performance on all models. In particular, MixPro has better performance on models with fewer parameters. For example, MixPro reaches 73.8% on Deit-T, which is 1.2% higher than TransMix.
4.1.2 COMPARISON WITH SOTA MIXUP VARIANTS
Table 2: -apcurccr.cy. training speed (im/sec) and number of parameters comparison with state-of- the-art Mixup variants on ImageNet-Ie for 300 epochs. For a fair comparison, all listed models are built upon DeiT-S training recipe. Training speed (im/sec) takes account of data mixup, and model forward and backward in train-time. The main results are reported in TransMix (Chen et al. 202T and TokenMix (Liu al. 2022).
Method
Backbone
#Params
Speed (im/sec)
top-1 Acc (%)
Bascline
22M
322
78.6
GridMix
22M
322
79.5 (+0.9)
CutMix Attentive-CutMis
22M
322
79.8 (+1.2)
DeiT-S
46M
239
77.5 (-1.1)
SaliencyMix
22M
314
79.2 (+0.6)
Puzzle-Mix
22M
139
79.8 (+1.2)
TransMix
22M
322
80.7 (+2.1)
TokenMix
22M
322
80.8 (+2.2)
MixPro(Ours)
22M
322
81.3 (+2.7)
We compare many SOTA Mixup-based methods on ImageNet- 1k (Deng et al., 2009) in this section. Following TransMix, we also train based on DeiT-S for fair comparison. MixPro measured in image second (im/sec), training speed (i.e., training throughput), and takes into account data mixup (Zhang et al. 2017). model forward and backward in train-time in an average of five runs for images at resolution 224x224 under 128 batch size with Tesla 100 graphic card. Table shows that MixPro outperforms all other Mixup-based methods.
4.2 TRANSFER TO DOWNSTREAM TASKS
We demonstrate the transfcrability our MixPro-based pre-trained models to the downstream tasks, including semantic segmentation, object detection, and instance segmentation. We observe the enhancements over the vanilla pre-trained baselines and TransMix.
4.2.1
SEMANTIC SEGMENTATION
pretrained
Backbome
ResNet101 ResNec101
Deeplaby3+
47.3
48.5
Backhome
AParams
APh
Object detection
AP APh
AP..
APM A/S
segmentation
DeiT'S +TransMix
49.1
49.6
ResNes50 ResNet101
44.2M
57.1
Tinear
63.2M
40.4
61.1
44.2
364
57.7
38.8
+MixPro
50.3
50.9
44.1M
43.8
60.1
DeiT-S
Transix PVTS
44.1M
40.9
63.8
44.0
38.4
60.7
41.3
+TramsMix DeFFS
Segmenter
50.6
$1.2
+MixPro
Table Following TransMix (Chen et al.
2021 Overhead-
Table 3: Overhead-free impact MixPro transferring to a down- stream semantic segmentation task on the Pascal Context (Mottaghi et al. 2014) dataset. (MS) denotes multi- scale testing. The best results in bold.
free impact of MixPro transterring to downstream object detection and instance segmentation using Mask R-CNN He et al.] 2017) with PVT (Wang et al. 2021) backbone on COCO val2017.A4 detection, and A pmn denotes mask AP for instance segmen- tation.
denotes bounding box AP for object
7

---

Published as a conference paper at ICLR 2023
Settings. We decode the sequence of patch encoding ppatches E RPxd to segmentation map R xx.Hwxxx x in our experiments, where K is the number of semantic classes. Following TransMix decoders. The first one is the linear decoder,
(Chen et al. 2021), we also employ two convolution which is point-wise linear layer DeiT. Patch encoding 2paiches E _2xd is also employed to generate patch-level logits, which are reshaped and bilinearly upsampled to the segmentation map s. decoder, namely the Mask Transformer. We also train and evaluate the models on the Pascal Context (Mottaghi al. 2014) dataset, which contains 4998 images with 59 semantic classes and background class. In addition, the reported mean Intersection over Union () is averaged over all classes as the main metric. The experiments are carried out using MMSeggmenatiin (Contributors 2020). All
The second one is the segmenter decoder (Strudel et al. 2021). which is transformer-
comparative experimental results come from TransMix (Chen et al. 2021
Results. Table shows that the MixPro pre-trained DeiT-SLinear and DeiT-S- Seegmener outperform the TransMix pre-trained baselines of 0.6% and 0.5% mloU, respectively. For multi-scale testing. MixPro also outperforms the TransMix pre-trame pre-trained baselines of 0.6% and 0.4% mloU.
4.2.2 OBJECT DETECTION AND INSTANCE SEGMENTATION
Settings. Object detection and instance segmentation experiments conducted on COCO 2017, which contains 118K images and evaluates 5K validation images. Following TransMix, we employ our method on PVT (Wang et al. 202T) as the detection backbone since its pyramid features make beneficial to object detection. Following TransMix (Chen et al. (202T). we adopt 1x training schedule (i.e., 12 epochs) to train the detector based on mmDetection (Chen et al. 2019) framework.
Results. As indicated in Table we observe that on two downstream tasks, the results of MixPro's pre-trained backbone once again outperformed TransMix's 's pre-trained backbone, which enhanced 0.5% box AP and 0.5% mask AP, respectively.
4.3 ROBUSTNESS ANALYSIS
We also compare MixPro with TransMix in terms of robustness and out-of-distributio performance.
90
80
90
80
80
Top-1 Acc
60
Top-1 Acc
Top-1 Acc
60
40
50
DeiT.S
DeiT-S
50-
DelT
TransMix
20
Deiis
TransMi--Dei-SS
40
MixPro-Deit-S
MixPro-Deit-S
40
MixPro-Deit-S
30.0
8.0
0.2
0.4
0.6
0.8
38.0
0.2
0.6
0.8
Information loss
Information loss
Information loss
(a) Random patch dropping
(b) Salience patch dropping
(c) Non-Salience patch dropping
Figure 3: Robustness against occlusion. The figure shows the robustness of DeiT-S against occlusion with different information loss ratios.
4.3.1 ROBUSTNESS TO OCCLUSION
Naseer et al. (Naseer et al. 202T) investigate whether ViTs perform robustly in the presence of missing partial or substantial image content. particular, ViTs divide an image into N =196 patches on a 14x14 spatial grid. For example, an image of size 224x224x3 is split into 196 patches, each of size 16x16x3. the experiments, "patch dropping" means replacing original image patches with O-value patches. Following TransMix (Chen et al. (02T), we showcase the classification accuracy on an ImageNet-Ik validation set with three dropping settings. (1) Random Patch Dropping, which selects and drops subset of M patches at random. (2) Salient (foreground) Patch Dropping, which studies the robustness of ViTs against occlusions highly salient regions. Naseer et al. (Naseer et al. 2021) thresholds DINO's attention map to obtain salient patches, which are dropped by ratios. (3) Non-salient (background) Patch Dropping, in which the least salient regions an image selected and dropped following the above approach.
8

---

Published as a conference paper at ICLR 2023
Results. As demonstrated in Fig. DeiT-S with MixPro is superior to TransMix and vanilla DeiT-S on all occlusion levels.
4.3.2 NATURAL ADVERSARIAL EXAMPLE
Table The robustness DeiT-S against natural adversarial examples on ImageNet-A and out-of- distribution examples ImageNet-O.
Nat. Adversarial Example Calib-Error
Out-of-Diss AUPRC
Models
Topl-Acc
AURRAC
DeïT-S
19.1%
32.0 %
23.8 %
20.9 %
TransS
21.1%
31.2 %
28.8 %
21.9 %
MixPro-DeiT-S 22.4%
30.3 %
32.4 %
23.1 %
For these experiments, we use the ImageNet-A dataset (Hendrycks et al. (021). which adversari- ally collects 7,500 unmodified, natural but "hard" real-world images. They are drawn from some challenging scenarios ,such fog scenes and occlusion. For evaluating settings TransMix (Chen et al., (02T), we evaluate methods on the top-1 accuracy, Calibration Error (CalibError) (Hendrycks et al.) 2021) which judges how classifiers reliably forecast their accuracy, and the Area Under the Response Rate Accuracy Curve (AURRAC) which is an uncertainty estimation metric.
method, following
Results. As demonstrated in Table MixPro-trained Deit-S outperforms TransMix-trained DeïT'S and vanilla DeiT-S all metrics. MixPro lifts Topl-Acc ImageNet-advvesarial by 1.3%. For AURRAC, MixPro-DeiT-S achieves 32.4%, 3.6% higher than TransMix-DeiT-S.
4.3.3 OUT-OF-DISTRIBUTION DETECTION
The ImageNet-O (Hendrycks al. 2021 is adversarial out-of-distribution detection dataset. It adversarially collects 2000 images from outside ImageNet-IK. The anomalies of unforeseen classes should result in low-confidence predictions. The metric is the area under the precision-recall curve (AUPRC). Table Sshoww that MixPro-trained DeiT-S outperforms MonsMMxx-raiine DeiT-S by 1.2% AUPRC and outperforms DeiT-S by 2.2% AUPRC.
CONCLUSION
In this paper, propose new data augmentation method, MixPro. MixPro addresses the shortcom- ings of the current SOTA data augmentation method, TransMix, from the perspective of image and label space for the vision transformer. From the perspective of image space, we propose MaskMix which is random mask strategy with adjustable scale. MaskMix ensures each image patch comes from only one image and contains more global contents. From the perspective of label space, we propose progressive attention labeling, which utilizes a progressive factor (a) to dynamically re- weight the attention weight of the mixed attention label. Experimental results show that compared with TransMix, our method brings an improvement of 1.2%, 0.6%, and 0.5% for DeïT-T, DeiT-S, and Deit-B on the imagenet, respectively. After being pre-trained with MixPro on ImageNet, the ViT-based models also demonstrate better transferability to three downstream tasks such as semantic segmentation, object detection, and instance segmentation. In addition, compared to TransMix, MixPro also shows stronger robustness three different benchmarks. In the ablation study, detail the effect of each proposed module, the effect of different scales of mask patches, the different strategies of the progressive factor, and so
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China under Grant 62271034, and in part by the Fundamental Research Funds for the Central Universities under Grant XK2020-03.

---

Published as a conference paper at ICLR 2023
REFERENCES
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34, 2021.
Kyungjune Baek, Duhyeon Bang, and Hyunjung Shim. Gridmix: Strong regularization through local context mapping. Pattern Recognition, 109:107594, 2021.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Paola Eesscante-Bonillee Arshdeep Sekhon, Yanjun Qi, and Vicente Ordonez. Evolving image compositions for feature representation learning. arXiv preprint arXiv:2106.00011. 2021.
Jie-Neng Chen, Shuyang Sun, Ju He, Philip Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. arXiv preprint arXiv:2111.0983. 2021.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. 2019.
arXiv preprint
MMSegmentation Contributors. Mmsegmentation: Openmmlab semantic segmentation toolbox and benchmark, 2020.
Xiyang Dai. Yinpeng Chen, Jianwei Yang, Pengchuan Zhang, Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9988-2997, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pultern recognition, pp. 248-255 leee, 2009.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint
2017.
A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. 2020.
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 34, 2021.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262 15271, 2021.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1129-8138, 2020.
Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. Advances in Neural Information Processing Systemx, 34: 115590118022 2021.
10

---

Published as a conference paper at ICLR 2023
A. Kendall and Y. Gal. What uncertainties do need in bayesian deep learning for computer vision? 2017.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pp. 5275-5285. PMLR. 2020.
Jihao Liu, Boxiao Liu, Hang Zhou, Hongsheng Li, and Yu Liu. Tokenmix: Rethinking image mixing for data augmentation in vision transformers. arXiv preprint arXiv:2207.08409, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012-10022, October 2021.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 891-898, 2014.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-88) 855, 1992.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-222 2015.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7262-7272, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 28 18-662,,, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Leurning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 32 42- 2021b.
AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et al. Salien- cymix: A saliency guided data augmentation strategy better regularization. arXiv preprint N.NV:20006.01791, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Pazz -azz and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438-6447. PMLR, 2019.
Devesh Walawalkar, Zhiqiang Shen, Zechun Liu, and Marios Savvides. Attentive cutmix: An enhanced data augmentation approach for deep learning based image classification. arXiv preprint arXiv:2003. /. 13048, 2020.
11

---

Published as a conference paper at ICLR 2023
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 568-578, October 2021.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 558-567, October 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.0971 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
A APPENDIX
A.1 ABLATION STUDY
The effect of our proposed module. demonttteee the effect of removing different compo- nents provide insights into what makes MixPro successful. The experimental results are based on DeiT-S training for 300 epochs and Mixup (Zhang et al.) (017)) is default used. We can observe that combining MaskMix and PAL, our method MixPro can enhance 1.5% compared to the standard setup (using Mixup and CutMix together), reaching 81.3% top-1 accuracy.
Table 6: Topl-accuracy is imagenet- 1K based on DeiT-S. We study the effect of removing different components to provide insights into what makes MixPro successful.
CutMix
TransMix
MaskMix
PAL
top-1 Ace (%)
X
X
79.8
XXX
X
80.7 (+0.9)
X
x
81.0 (+1.2)
81.1 (+1.3)
X
X
81.3 (+1.5)
region
(b) block
(c) random
(d) random with adjustable scale (Ours)
Figure 4: Illustration of different mask strategies.
12

---

Published as a conference paper at ICLR 2023
The effect of different mask strategies. Figure illustrates the different mask strategies. The region-basse based strategy is widely used previous works (Yun et al. 2019 Uddin et al.) [020 Veirsss
& Taylor 2017). Block and random based strategies mainly used in self-supervised learning (He
et al. 2022 Bao et al. 2021). Our MixPro boosts random strategy on an adjustable scale. To better inspect the impact of mask strategies alone, in table 7 directly Aarea to generate mixed labels instead of PAL. Table 7 shows the different performances of the mask strategies. Block and random strategy results come from TokenMix (Liu al. 2022). We can observe that the block and random mask strategy improves significantly compared to the region mask strategy. This also indicates that images with more mixed global content are more effective for the vision transformer. Furthermore, our proposed adjustable scale is necessary for the random mask strategy. Compared with block and random strategies, the mask patches with a 4x scale can improve the accuracy of Deit-T by 0.5% and Deit-S by 0.2%.
Table 7: Ablation mask strategy.
model
region
block
random
random(4x scale)
Deit-T
72.2
72.7
72.7
73.2
Deit-S
79.8
80.6
80.6
80.8
The effect of different scales of mask patches. The scale of mask patches P Pmask is multiple of the scale of image patches. There are several optional scales E (1x, 2x, 4x, 7x}- The Fig. 5 indicates that the evaluate results of MixPro with different scales based on DeiT-S on Imagenet- top-1 error. For all scales considered, MixPro improves upon the baseline (20.2%) significantly. Moreover, it performs best when the scale is multiplied by 4x. We still recommend adjusting the scale size more finely for different models to get better results.
19
18.8
Top-1 Error(%)
19.5
Top-1 Error %)
18.6
19
18.4
18.5
18.2
18
18
4x
0.5
0.8
Scale
β
(a) The different scales of mask patches.
(b) Effect of different B.
Figure 5: Effect of different scales of mask patch and [ on Imagenet- I top-1 error with DeiT-S.
The effect of different 3. In Fig. (b), we demonstrate the top-1 error of MixPro under different Beta distribution, such as 8 E (0.5, 0..., 2). on Imagenet-1K IK with DeiT-S, We can observe that is equal to when our model achieves the best performance.
The different strategies of progressive factor To facilitate the understanding of our proposed progressive attention labeling (PAL), we explore several different strategies to generate the progressive factor a evaluating on Imagenet-1K 1K with DeiT-T. We employ several types of strategies: 1) Progress- relevant strategies adjust with the number of training epochs, e.g., parabolic decay, etc. 2) Progress-irrelevat strategies include the equal weight. 3) learnable parameter.
How the boundary inside the patch influences VITs? TransMix introduces sharp rectangular
borders even within an image patch (see Fig. [ red colored) that are clearly distinguishable from the background. These borders inside the patch may draw excessive attention to the model, because for vision transformers, a patch would form a semantic unit, and the computation of the attention map of the whole image is obtained by summing the attention of each image patch.
13

---

Published as a conference paper at ICLR 2023
Our MaskMix is able eliminate the influence of the internal borders of patches. By aligning the borders of the mask patches with the image patches, there thus will not be sharp rectangular borders within image patches. Here also design three approaches to explore the performance of our MaskMix: A. Aligning mask borders to patch borders for TransMix. B. Misaligning the borders of mask patches with image patches for MaskMix. C. Reducing the scale of each mask unit for our MiskMix to introduce more noises. Experimental results are below:
Table 8: Three approaches to exploring the performance of our MaskMix.
method
TransMix
A
B
C
MaskMix(OUrs)
Top-1 Acc (%)
72.6
73.1
73.1
73.4
73.8
We can observe that our MaskMix introduces more noisy patterns (such as and C) lead worse results, The TransMix eliminates the internal borders of the patches using the approach A and lifting Top-1 Ace by 0.5%. This shows the improvements brought by the design our MaskMix.
Whether introducing more random patterns will further improve the performance? We test three random patterns evaluate the DeïT-T on ImageNet. The patterns and results are below:
Table 9: Evaluated on more noisy patterns.
model
results
Baseline
73.2
Random shuffling on the cropped patches
69.8
Random shuffling the patches of the canvas image Both
69.7
67.3
We can observe that damaging the original geometry the image with these schemes leads to worse results, i.e., simply introducing random patterns (e.g., via simple random shuffling) does not bring performance improvement.
Table 10: Ablation studies of different progressive factor strategies on Imagenet- 1K with on DeiT-T.
strategy
Top-1 Error (%)
Equal weight
0.5 T
27.3
Linear increment
27.1
T T
Parabolic increment
26.9
Tmoa
Learnable parameter
27.0
PAL (Ours)
cosine distance
26.2
As illustrated in Table 10 the rrlerant-relevant strategies (i.e., linear increment, and parabolic increment) for generating a can yield better results than the progress-irellvamt irrelevant strategy. The effects of progress-relevant strategies and learnable parameter strategy equally. These observations prove our motivation that the model gets a low-confident attention map during the early training process, so employing the low-confident attention map for mixed labels may be good choice. Among these strategies, the one with the best performance for generating is our proposed PAL.
A.2 VISUALIZATION
As shown in Fig. 6 we visualize images generated by our method and its corresponding attention map. We can observe that mixed images produced by MaskMix consist of multiple scattered and continuous areas in the image, which thus capture broader parts of the object content of the original images from a global perspective. Furthermore, the attention maps these mixed images also make sense, capturing the feature of objects in mixed images.
14

---

Published as a conference paper at ICLR 2023
chickadee
tench
lemon
pomegranate
lemon
scabbard
scabbard
water bottle
chameleon
rock crab
Aitention maps Mixed images Origital
4
(a)
(b)
(c)
(d)
Figure 6: Visualization. Top: Original images. Medium: Mixed images generated by our MaskMix.
Bottom: Attention maps of mixed images.
A.3 TRAINING DETAILS ON IMAGENET- IK
Our training receipt follows previous works (Touvron et al. 2021a Chen et al. 2021 ). The default
setting is in Table
Table 11: Training settings ImageNet-IK. K.
config
value
optimizer
AdamW W
learning rate
0.001
weight decay batch size
0.05
1024
learning rate schedule
cosine decay
warm-up epochs
20
training epochs
300
augmentation
RandAug(9, 0.5)
label smoothing
0.1
drop path
0.1
MixUp
0.8
CutMix
MixPro
A.4 PSEUDO-CODE
Algorithm provides the codd MixPro in a pytorch-like style. It demonstrates that simply few lines of code boost the performance in the plug-and-play manner.
15

---

Published as a conference paper at ICLR 2023
Algorithm I Pseudo-code of MixPro in PyTorch-like style.
H. W: the height and width of the input image.
# h, w: the height and width of the attention map.
M: 0/1 mask of MaskMix with shape (H.)). # downsample: downsample from (H,W) to (h,w).
for (x, y) in dataloader: # load a mini-batch
= (3,)) # Eq. (6)
M generkkgeerrteet,, Pmaak..Pamaa Pinage)
Aarea = sum(M)
* M x.flip(0) * (1-M) # Eq. (7)
logits, A model())
Aarea y+ (I-Aarea) (0) # Eq. (7)
cos cas__oosssssee similerityylloiiss, 37 # Eq. (8)
M' downsample(M)
Aatin matmul(A, M') Eq. (5)
= a Aattn + (1-a) * Xarea Eq. (9)
y = x (1-A) ()-))x)..fipp(
CrossEnnropyy.oss(loggiss backwad)))))
16


Reviews:
[{"review_id": "fUar9wLiN0N", "reviewer": "Reviewer_ujft", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes MixPro, an improvement of TransMix, a mixup method designed for vision transformers where the label interpolation weights are determined by the attention maps. It contributes two ideas: 1) (MaskMix) in the input space, the mask is random over patches rather than rectangular (as in CutMix). 2) (Progressing Attention Labeling, PAL) The label interpolation weights evolve from area-based (as in CutMix) to attention-based (as in TransMix) as the confidence of the classifier increases during training, thus attention maps becoming more reliable.", "summary_of_the_review": "Overall, this work proposes two simple and interesting ideas and is well implemented and well written, but given its limited novelty and limited benefit, one would expect as compensation at least a wider range of empirical studies that would allow a deeper understanding of the related methods. The current study is not mature enough and I am not sure if it can make it for this ICLR.", "strength_and_weaknesses": "The paper is well written and a pleasure to read. The ideas are very simple and the two improvements make sense. There are extensive experiments and comparisons, mostly following TransMix.\n\nI find the two improvements incremental. They are both of the kind one would expect to find between a conference paper and a journal submission, not in a new paper.\n\nThe most critical missing experiment is an independent evaluation of the two components. As it stands, it is impossible to assess the benefit of either idea (MaskMix and PAL). There is an ablation in Table 6 (Appendix) but only on ImageNet-1k accuracy, where the gains of the two ideas are only 0.3% and 0.4%. Independent evaluation should follow on all experiments.\n\nAnother limitation of the empirical study is the narrow focus on vision transformers. TransMix may have initiated this but there are many ways one may obtain an attention map from a convolutional network, so I see no reason for this narrow focus. Extending to convolutional networks will also allow a comparison with previously published results. Currently, the only comparison with methods prior to TransMix is on ImageNet-1k classification (Table 2). All other comparisons are limited to TransMix and the baseline, which is not sufficient.\n\nMaskMix is motivated by the global receptive field of vision transformers, but I find this motivation insufficient. For one thing, region-level crops appear to be more difficult for the local receptive field of convolutional networks; for another, an independent evaluation of MaskMix and a study on convolutional networks could at least validate the claim.\n\nConcerning PAL:\n- The use of cosine similarity in (8) is not elegant or justified, since the two probability vectors are l1-normalized. More appropriate is the histogram intersection (which is to l1 norm what is dot product to l2 norm). Or any other similarity measure based on a proper distribution metric.\n- The probability vector p of the interpolated input is not a good indicator of classifier confidence. More reliable are the probability vectors of the clean images, which could be compared with the clean labels (resulting in the predicted probability of the ground truth classes). I am not sure how this could be implemented, given that TransMix only processes mixed images. Maybe confidence could be estimated globally by considering some clean images per epoch. This could be at least a baseline for Table 8, even if it does not improve.\n- Since patches are chosen at random from one or the other image, it is not clear if the attention map of a mixed image makes as much sense as that of a clean image or at least that of TransMix (CutMix) style mixing (by rectangular area). At least some visualization is in order. Another baseline is using always the area-based label interpolation as in CutMix.\n- Of course, PAL should be evaluated independently of MaskMix (with rectangular area image mixing).\n\nAnother idea or baseline that relates to both improvements is to use attention maps not only for label but also for input image interpolation, as in SaliencyMix.\n\nOther benchmarks that were considered in prior works and not in TransMix or this work include robustness to adversarial attacks, calibration and out-of-distribution detection on datasets other than ImageNet, where the network is trained. Such experiments are important not just for completeness but because they have margin for considerable gains for a new method. For example, the gain of MixPro over TransMix is in the order of 0.5% in most experiments. Given that MixPro consists of two ideas, the gain of each is even less.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WlgBcytEfA", "reviewer": "Reviewer_qEAs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this submission, the author proposed a data augmentation method for ViT. Specifically, two samples will be mixed in a global level, where patches of an image will be replaced with the patches from another images at corresponding locations. Then, the labels are generated using both the attention map generated by the ViT and the binary mask to avoid the negative effect using only attention map in the beginning of training process. The experimental results have illustrated the effectiveness of the proposed method. The approach is further demonstrated in the downstream applications. ", "summary_of_the_review": "To sum up, the paper is well written and easy to follow. It proposed a data augmentation method for ViT which can further boost the performance of ViT on various tasks. Experiments on different tasks and applications were conducted and the results were well analyzed. I have some questions which can be referred to in Strength And Weaknesses.", "strength_and_weaknesses": "Pros:\n1. The paper is well written and easy to follow.\n2. The experiments are thorough and the results are pretty good.\n3. The proposed method can be generalized to different visual tasks and boost performance.\nCons:\n1. In the downstream tasks, seems like the proposed method is only employed for pretraining. Can it be used for directly finetuning for downstream applications?\n2. Minor: in Generalization analysis: \"Fig2(c) depicts their loss and top-1.....\" should be Fig2(c) and (d) depicts their loss and top-1....\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8FR4qdUR_q", "reviewer": "Reviewer_vD1g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new variant of CutMix, namely MixPro to further improve the previous state-of-the-art (TransMix) for Vision Transformers. The proposed method, including MaskMix and Progressive Attention Labeling, aims to tackle three primary drawbacks they found in TransMix. Extensive experiments show that MixPro can lead to consistent and remarkable improvement on various tasks and datasets.", "summary_of_the_review": "I hold a double feeling of this paper. On the one hand, MixPro demonstrates its strong performance on various tasks and datasets., meanwhile, I like ideas that are simple yet sweet. On the other hand, I cannot agree with the explanation stated in the introduction and related work. I would first leave a borderline here and see how the authors respond.", "strength_and_weaknesses": "### Strength\n\n**1. The proposed method is simple yet effective, which makes it simple to follow.** MixPro basically follows the paradigm designed in TransMix, which only requires several lines of code to implement. Personally I like ideas that are simple and sweet. The major changes in MixPro, including MaskMix and Progressive Attention Labeling, are easy to be implemented in principle. However, the authors did not include the pseudo-code in their manuscript. It would be more clear if they can include the pseudo-code in the appendix.\n\n**2. The proposed method exhibit strong and consistent improvement** for different tasks and models even when compared with a strong baseline TransMix.\n\n**3. The proposed method is fast and applicable to many different Transformer-based networks.** As shown in Table 2, the proposed method can lead to a significant gain with minimal computational introduced.\n\n**4. The ablation studies and experiments are comprehensive and extensive.** The comparison between MixPro and TransMix makes it clear to understand the efficacy of the proposed method.\n\n### Weakness\n\n**1. I am not fully persuaded by the whole story and the explanation given by the author in the introduction.** The authors argue that TransMix has three major drawbacks including region-based mixing, boundary artifact and unreliable attention matrix.\n- 1.1 For region-based mixing, I am a bit confused about why *region-based mixed images may provide insufficient image contents* and why the proposed MaskMix can help to alleviate this problem as MaskMix will also mix just a portion of the entire image, which may be insufficient as well.\n- 1.2 For boundary artifact, the authors argue that it will introduce *sharp rectangular borders that\nare clearly distinguishable from the background.* However, the solution MaskMix will also lead to these sharp artifacts and the patch-wise mixture will spread this kind of noise everywhere in the mixed image, which may even aggravate such problem. Instead of echoing the argument by the authors, I would suggest that MaskMix may introduce more noisy patterns into the image mixture so that the network can be better regularized by these noises. \n- 1.3 For Progressive Attention Labeling (PAL), the authors argue that it is better than TransMix since *attention maps may not always be reliable during the training process.* However, the $\\lambda$ of TransMix is actually the mean of attention weights and the original $\\lambda$ introduced by CutMix (a.k.a. $\\lambda_{area}$ in the manuscript). This indicates that TransMix also considers this problem and uses the area of the cropped image as an extra condition. Meanwhile, TransMix also evaluates the effect of using different attention maps in calculating $\\lambda_{attn}$. However, using a highly-discriminative attention map pre-trained by DINO to calculate $\\lambda_{attn}$ does not result in any improvement in terms of the top-1 accuracy on ImageNet (see in Table 6 in TransMix paper). So I would doubt whether the label noise introduced in the label space would also be helpful.\n\n**2. The illustration for details of the proposed method is a bit unclear to me.** The authors did a good job in describing their overall pipeline and the basic idea. However, I am a bit confused about whether the introduced image will be first shuffled or not before mixing. The cat in Figure 2 does not follow the original geometric structure, but there seems to be no shuffling operation in Eq. (7).\n\n**3. May the authors do more analysis on why and how MaskMix works better than TransMix?** My assumption is that **MixPro creates more complex scenarios in the image mixture such that the network can be better regularized**.  Thereby I am curious about whether introducing more random patterns will further improve the performance. These random patterns may include but not be limited to:\n- random shuffling (or shifting) on the cropped patches\n- random shuffling (or shifting) on the patches of the canvas image\n\n**4. The proposed method is only applicable to Vision Transformers with class token.**", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster