RecursiveMix: Mixed Learning with History
Authors: Lingfeng Yang
Xiang Li
Borui Zhao
Renjie Song
Jian Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Data augmentation
Historical Knowledge

Abstract: Mix-based augmentation has been proven fundamental to the generalization of deep vision models. However, current augmentations only mix samples from the current data batch during training, which ignores the possible knowledge accumulated in the learning history. In this paper, we propose a recursive mixed-sample learning paradigm, termed ``RecursiveMix'' (RM), by exploring a novel training strategy that leverages the historical input-prediction-label triplets. More specifically, we iteratively resize the input image batch from the previous iteration and paste it into the current batch while their labels are fused proportionally to the area of the operated patches. Furthermore, a consistency loss is introduced to align the identical image semantics across the iterations, which helps the learning of scale-invariant feature representations. Based on ResNet-50, RM largely improves classification accuracy by $\sim$3.2% on CIFAR-100 and $\sim$2.8% on ImageNet with negligible extra computation/storage costs. In the downstream object detection task, the RM-pretrained model outperforms the baseline by 2.1 AP points and surpasses CutMix by 1.4 AP points under the ATSS detector on COCO. In semantic segmentation, RM also surpasses the baseline and CutMix by 1.9 and 1.1 mIoU points under UperNet on ADE20K, respectively. Codes and pretrained models are available at https://github.com/implus/RecursiveMix.

RecursiveMix: Mixed Learning with History
Lingfeng Yang!*, Xiang Li Borui Zhao?, Renjie S Jian Yang"
Nanjing University of Science and Technology, Nankai University, Megvi Technology
(yanglfnjuat, csjyang)\njust.edd. en, xiang 1ig..li..o implustnankaiiacccmmmmmm edu. zhaoborui, shabboruigg@@maill.... gmdgmail.coooo com, songrenjiedmegvix.
Abstract
Mix-based augmentation has been proven fundamental to the generalization of deep vision models. However, current augmentations only mix samples from the current data batch during training, which ignores the possible knowledge accumulated in the learning history. In this paper, we propose a recursive mixed-sample learning paradigm, termed "RecursiveMix" (RM), by exploring novel training strategy that leverages the historical input-reediction-label triplets. More specifically, we iteratively resize the input image batch from the previous iteration and paste it into the current batch while their labels are fused proportionally to the area the operated patches. Furthermore, a consistency loss is introduced to align the identical image semantics across the iterations, which helps the learning of scale- invariant feature representations. Based on ResNet-50, RM largely improves classification accuracy by ~~3.2% on CIFAR- 100 and ~2.8% on ImageNet with negligible extra computation/storage costs. In the downstream object detection task, the RM-pretrained model outperforms the baseline by 2.1 AP points and surpasses CutMix by 1.4 AP points under the ATSS detector COCO. In semantic segmentation, RM also surpasses the baseline and CutMix by 1.9 and 1. mloU points under UperNet on ADE20K, respectively. Codes and pretrained models are available at https: // thub. com/cmm//////c itmimmpuuu uup /eee/e/euussreeee siveMxx.
Introduction
Deep convolutional neural networks (CNN) have made great progress in many computer vision tasks such as image classification [26, 29, 62], object detection [25, 71, 42], semantic segmentation [72, 67], etc. Despite achieving better performances, the models inevitably become more complex and incur fett-fitting risks. To overcome this problem, methods like regularization [53, 30] and data augmentation [15,70 70, 68] are developed consequently. Recently, mixed- based data augmentations [70, 68, 59,9 60, 34, 13] have been proven enhance the model generalization capability. Following the popular Mixup [70] and CutMix [68], which optimize networks based on mixed pixels and fused labels of two images, improvement has been made on aspects
mix level [59, 18, 39] and mix
region [34, 13, 60). However, existing methods only mix samples from the current data batch. As result, they fail to capitalize on the potential knowledge in their learning history.
To leverage the historical knowledge, we propose an efficient and effective augmentation approach termed "RecursiveMix" (RM) that makes use of the historical pappu--reddition-laabe triplets. Different
"Corresponding author. Equal contributions. Lingfeng Yang. Xiang Li and Jian Yang with Jiangsu Key Lab of Image and Video Understanding Social Security, and Key Lab Intelligent Perception and Systems for High-Ding Dimensional Information Ministry of Education, Nanjing University of Science and Technology, Nanjing, 210094, P.R. China.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Destettate yd
Dog
Home
Rovioffi
2
21+2
2+12
Hisoorical Prediction
K1.0wrgro
CAP
BusAlggs
TAP
GAP
Historical Input
Ieration-:1
Iteration
1+1
Ieeotion 1+?
Illuttation of the RecursiveMix, which leverages the historical imput prediction label triplets input images are resized and then mixed into current ones where labels are fused proportionaly (A and A) to the area of operating patches.
Figure 1:
paradigm. The bistorical
from the conventional mix-based approaches which only fuse the training images in the current batch, RM further leverages the last mixed inputs and labels to form new ones in a recursive paradigm. aims at continuously reusing the augmented data and supervisions in previous iterations (Fig. 1). In addition, consistency loss is introduced align two spatial representations derived from the identical instance. RM shows sigmificant advantages over its competitive counterparts [68, 70] on classification tasks and its space-aware features can potentially benefit the downstream tasks such as object detection and segmentation as they are more sensitive to the spatial understanding of the image. The advanced performance of RM will illustrated later in our methods and experiments.
Another notable property of the proposed RM is that it hardly increases the training/inference budget (Table 11) but only consumes an extremely negligible amount of stor- age complexity (only mini-batch of the historical input, prediction, and label) and a lightweight additional module (only an Rol [25] operator and an optional fully connected layer). Experimental evidence in CIFAR and ImageNet classification benchmarks demonstrate its effectiveness, Specifically, RM shows gain of absolutely 3.2% points over the baseline model and outperforms CutMix [68] by 1.1% points in the CIFAR- 100 dataset under DenseNet- 161 [29]. In the ImageNet dataset, RM improves by 2.9%
Moup
CuEMi
RM
and 2.7% points based the ResNet-50 [26] and PVTv2-
B1 [61] baselines while outperforming CutMix by 0.6%
and 0.7% points, respectively. In terms of the downstream tasks, RM-pretrained models improve the bascline by 1~~3 box AP points (object detection) and I~2 mask AP points (instance segmentation) on the COCO dataset and I~3 mloU (semantic segmentation) ADE20K while consis- tently outperforming models pretrained by CutMix. No- tably, under the one-stage detector ATSS [71], the RM- pretrained model under ResNet-50 outperforms the base- line and CutMix by 2.1 AP and 1.4 AP points, respectively.
Consistency
Figure 2: Three benefits of the proposed RecursiveMis, which iteratively leverages the historical input-prediction label triplets: 1) enlarged diversity data space and richer supervisions, 2) adequate trining signals for an instance multi scale' space views, and 3) explicit leaming the spatial semantic consistency which can further benefit the downstream tasks.
2 Related Work
Regularization Methods. There are many other types of regularization techniques that focus on the
feature [53, 20, 30], data [15, 8), label [49], data-label pair [70, 68, 34] and feature-label pair [59].
Dropout [53], DropBlock [20] and Stochastic Depth [30] introduce the stochastic feature drop
during training in an element-wise, block-wise, and path-wise way, respectively. Cutout [15] and
GridMask [8] randomlylsystematiially erase regional pixels on images while Label Smoothing [49]

---

tries to prevent the classifiers from being too confident on a certain category by slightly modifying the training labels into soft distribution [27]. Mixup [70] and CutMix [68] both combine two samples, where the corresponding training label is given by the linear interpolation of two one-hot labels. They differ in the detailed combining strategy of the input images: Mixup employs pair- wise linear combination but CutMix adopts a regional crop-and-pase operation. SaliencyMix [58], PuzzleMix [34], Co-Mixup [33], and SuperMix [13] further exploit the salient regions and local statistics for optimal Mixup. Attentive CutMix [60] and SnapMix [31] take the activation map of the correct label as a confidence indicator for selecting the semantically meaningful mix regions. TransMix [5] requires the attention map of the self-attention module to re- reiet the targets. Manifold Mixup [59]. PatchUp [18] and MoEx [39] perform a feature-level interpolation over two samples to prevent overfitting the intermediate representations. Recently, StyleMix [28] employs style transfer to enrich the generated mixed images. The proposed RecursiveMix (RM) shares similarities with CutMix by combining data samples in regional replacement manner, but differs essentially from all the existing regularizers in that RM is the first to exploit the historical information. In the experiment part, we will show the comparisons between the proposed RM and other regularization methods.
Contrastive Learning. Contrastive learning [22] aims to minimize the distances between the positive pairs by consistency losses [22] which measure their similarities in a representation space probability distribution space. In recent years, contrastive learning has made a lot of progress in the fields of computer vision [24, 51, 2, 57, 1, 66, 50, 4, 10, 19, 11, 21, 38] and natural language processing [64, 48, 32]. Positive pairs can be constructed in many ways. In the fields of self- supervised learning and semi-super learning, many works [2. 24, 9, 4, 21] have used two augmented inputs of one instance to form positive pairs. In terms of generating positive pairs with different model structures or weights, [64, 52] employ Dropout [53] twice on model, and [24, 57] make a difference in weights through Exponential Moving Average (EMA) and separate projection heads. The above methods essentially require Siamese networks [3] or the need to pass one instance through a model twice to obtain positive pairs, which involves additional computational costs and large architectural designs. Another way is record historical representations or predictions and compare them with their current outputs [66, 35, 38, 65]. However, using memory bank to store key statistics for all instances consumes a huge memory cost and is not applicable to even bigger datasets. Our method employs the idea of contrastive learning from the learning history while consuming extremely negligible storage and computational complexity.
Learning with History. In the training history, the accumulated elements are a large amount of wealth for the improvement of learning deep models. Existing methods have explored the usage across model parameters [4, 10, 24, 19, 11, 21J. gradients [54, 17, 69, 36, 47, 44], predictions [35, 38], and all-Ievel feature maps [63, 65, 66, 24] applied during the training process. More related works refer to the Supplementary Material. Different from existing practices, we exploit a novel aspect of the historical input-predictte prediction triplets to improve the generalization of vision models.
3 Method
In this section, we introduce the proposed RecursiveMix (RM) approach and discuss its properties.
3.1 RecursiveMix
We attempt to exploit the historical input-prediction- label triplets of learning instance improve the gen- eralization of vision models. A practical idea is to reuse the past paput-prediction-lee versely mixed training samples with the current batch (see Fig. 1). To be specific, the historical input images are resized and then filled into the current ones where the labels are fused proportionally to the area of the mixed patches. Interestingly, such an operation formulates an exact recursive paradigm where each instance can have a gradually shrunken size of view during training. thus we term the method "RecursiveMix" (RM). Specifically at iteration t, the preliminary of RM is to generate a new
Label
triplets to construct di-
Incon
sistency
CUT
Resize
Figure 3: "Cut" operation may lead to the inconsistency be- tween input and label under the proposed historical operation. but "Resize" can correctly preserve the consistency.
3

---

training sample (I'. by combining the current sample (x". and the past historical one (x", vy"). The new training sample (I', 3²) participates in training with the original loss objectives Lov and then updates the historical pair (z", y²) iteratively:
V M @ ResizeFill("", ,M) + (1 M) e Ca',
V¹ y'y + (1 -2)'',
(I)
r²h x², y = J²:
where M {0, 13W*H is the binary mask indicating the place for the historical input to fill, 1 the binary mask filled with ones and @ is the element-wise product. The combination ratio À (X* indicates at moment t) is sampled from the uniform distribution U O0, ax] where a defaults to 0.5. This setting is different from the symmetric beta distribution in Mixup [70] and CutMix [68] because À is not suggested to be quite large for denotes the proportion of the historical images in our case. Otherwise, the information of the new data points has risk of being discarded. The function ResizeFill M)) means resizing the image x" and filling it back exactly into its rectangle sub-region where M = 1. The resized height and width are also determined by the rectangle area. This design is to make the recursive mechanism reasonable since the original cut operation in CutMix [68] could possibly lose former information, which would cause the inconsistency problem with the label (see Fig. 3).
The binary mask M is generated by filling in with inside the sampled bounding box coordinates Th Th),, otherwise defaulting to O. Similar to CutMix [68], the size and position of the B (Ta, Ty Fary rectangular region B are uniformly sampled according to:
Fz ~ U(O, W), Tw = W VA,
(2)
ry ~ UO, H), ra = HVĀ.
As depicted in Fig. 2, the resize-filled part within the current input is semantically consistent with the historical input regardless of proportion and scale. Inspired by contrastive learning [22, 24, 4], we optimize the KL divergence between the cross-iteratis noss-iteration predictions of the two corresponding regions. To be specific, we record the historical prediction pa which is updated by the original outputs after network function F(-), global average pooling GAP(.), and linear layer H((-) in the last iteration:
p²-1
((AA (FT'
(3)
In the current iteration, we obtain the corresponding prediction of local features by 1x1 RolAligm [25] aligning with the computed coordinates of RecursiveMix:
Pro: = 7' (RoLAlign(F("")
(4)
where F. denotes the backbone function and H. Ht' denote the final linear classification layer. By default, the parameters are not shared between layers H and H', which is purely suggested by the experiments in Table 4a. The total loss function to train our model is:
(5) where Los denotes the cross-entropy loss and LKL denotes the consistency loss. Note that p the historical prediction from the last iteration, and the consistency loss weight w is set to 0.1 unless otherwise stated. In addition, we also use the mixed ratio X representing the proportion of the resize-filled historical image to weight LKL. It makes sense that the confidence of Lx relates as a smaller X (smaller image size to be filled) usually causes the Rol feature to lose more spatial semantic information.
L = LCE("', D') + WA LKL
3.2 Discussion
Similar to CutMix/Mixup, RM is simple and hardly introduces additional computational costs (see Table 11). During training, only a mini-batch of the historical input, label, and prediction (i.e., (arh "..""" p²)))) needs to be stored and updated iteratively. The Rol [25] operator and an individual fully connected layer considerably lightweight and free of deployment. The additional storage cost is extremely negligible compared to the entire model, data, and feature maps, thus making RM very efficient to train any network architecture.
As illustrated in Fig. 2, introducing the historical design of input, label, and prediction into augmenting the mixed training pairs recursively and bridging the spatial semantics consistency has three obvious

---

PyramidNet-200, 6=240 epochs)
Top-1 Err
Model (200 epochs)
Type
Top- Err (8)
Baseline
Baseline
Label Smoothing [49]
3.74
Mixup [70]
ResNet- 18 [26]
DrapBlock [20]
3.27
+ CutMix [68]
Stochastic Depth [30]
3.11
+ RM (ours)
18.64
Cutoul [15]
3.10
Baseline
20.62
Mixup (<==1.0 [70]
3.09
+ Mixup [70]
19.19
ResNet 34 [26]
Manifold Mixup (a=1.0) [59]
3.15
CatMix [68]
17.89
CutMix [68]
2,88
RM (ours)
17.15
MoFx [39]
3.44
Baseline
StyleCurMix (auto-r) [28]
2,55
* Mixup [70]
DenseNet-121 [29]
(ours)
2.35
+ CutMix [68]
Tahle Comparison state-of-the regularization
+ RM (ours)
16.22
CIPAR-10 under 300 epochs settings, the same that of the Cut- Mix [68] paper. Results are reported as average over runs.
Baseline
18.78
+ Mixup [70]
16.84
DenseNet [29]
Model
RS
HIS
Top-1 (S))
CatMix [68]
16.64
PyramidNet
16.67
RM (ours)
15.54
+ CutMix [68]
15.59
Baseline
15.36
PyramidNet-166 6=270 (23]
Mixup [70]
+ RM (ours)
1
14.81
CutMix [68]
14.65
+ RM (ours)
Table 2:
Ablation studies on the individual gain by each component with PyramidNet- á=270 under the 200- epach setting. The results are averaged over 3 muns. "RS": Resize strategy. "HIS": Historical mix. "CL": Consistency loss.
Table Performance varieus architecttress in CIFAR- 100 under 200 spochs. The propesed RM (a=0.5, w=0.1) shows consistent im-- provements other competitive approuches. We report average of runs.
RM
CIFAR10
CIFAR100
CIFAR10
CIFAR100
28.4
3.2
1244
Em (N6)
3.3
Top-1 Err 196
19.2
15.5
2.4
19.0
20.8
33
Rayee Hyperrparameter
parammparaetter
Hyper parametrrr
typersparmmter
Figure 4: The effect of hyperparameter CIFAR-10 and CIFAR- 100 datasets. Standard deviation is also plotted and another hyperpu- rameter
Figure The effect of hyperparameter CIFAR-I0 CIFAR- 100 datasets. Standard deviation is also plotted and another hyperpa-
advantages: 1) RM encourages more diversity in data space as it exhaustively explores exponential combinations of multiple instances along with their corresponding richer training signals within single iteration. 2) RM can explicitly provide continuous training supervisions for an instance with multi-scale and spatial-variant views. 3) RM makes use of the identical part of inputs from two adjacent iterations through contrastive learning via the consistency loss, which helps the model learn cpatial-correlative semantic representation. These properties thus lead the better generalization of computer vision models and can potentially benefit downstream tasks, e.g., object detection and semantic segmentation with enhanced spatial representation ability. Interestingly these properties be simultaneously achieved by a simple operation of iteratively using the historical input-prediction- label triplets.
Experiment
In this section, evaluate RM image recognition tasks to show the effectiveness of the usage of historical input- prpuutprediction-tee triplets. More details can be found the Supplementary Material.
4.1 CIFAR Classification
The two CIFAR datasets [37] consist of colored natural scene images, each with 32x32 pixels in total. The train and test sets have 50K images and 10K images respectively. CIFAR-10 has 10 classes and CIFAR-100 has 100.
Dataset.
Setup. We conduct two major training settings: 200-epoch and 300-epoch training, respectively. For 200-epoch training, employ SGD with a momentum of 0.9, weight decay of x 10-4, and
5

---

Top-1 Err (%)
CIFAR-I0
CIFAR-I00
ImageNet
Top-1 I Err (%)
CIFAR-I0
CIFAR-100
ImageNet
Unshared weights
2.35
18.64
20.80
nearest
2.35
18.64
20.80
Shared weights
2.61
18.66
20.92
bilinear
2.58
18.78
20.86
(a) Comparisons of linear layer weights.
(b) Comparisons of different interpolation modes.
Table Ablation studics I) CIFAR-10 with PyramidNet-200, d==200 under 300-epoch training setting, CIFAR-100 with ResNot-18 under 200-epoch training setting and 3) ImageNet with ReaNet- under e0D-e training setting.
ResNet-50 (300 epochs)
Top-1 Eir (%)
Top-5 Err (8)
Baseline
23.68
7.05
Model (300 epochs)
Top-1 Err (%)
Top-S Err (%)
Culout [15]
22.93
6.66
ResNet-50 [70]
23.68
7.05
Stochastic Depth [30]
22.46
6.27
Mixup [70]
22.58
6.40
+ Mixup [70]
22.58
6.40
CutMix [68]
21.40
5,92
Manifold Mixup [59]
22.50
6.21
RM (ours)
20.80
5.42
DropBlock [20]
21.87
5.98
PVTv2-B1 [61]
24.92
8.06
+ Feature CutMix [68]
21.80
6.06
Mixup [70]
23.23
6.65
CutMix [68]
21.40
5.92
CutMix [68]
22,92
6.42
+ PuzzleMix [34]
21.24
5.71
RM (ours)
22.22
6.17
MoEx [39]
21,90
6.10
Tahle 5: Performance on various archilectures in ImageNet under
CuiMix MoEx [39]
20.90
5.70
epochs. The proposed (a=0.5, w=U.5) shows consistent
improvementts over other competitive counterparts.
RM (ours)
20.80
5,42
Table Compurison state-of art regularization methods ImageNet under 300 epochs.
2 GPUs with a mini-batch size of 64 on each optimize the models. The learning rate is set 0.1
with linear warmup [26] for five epochs and cosine decay schedule [46]. For 300-epoch training, we align all the hyperparameters with the official CutMix [68] for fair comparisons. Following [68], the averaged best performances are reported via three trials of experiments.
Comparisons with flate-of-thearre Regularizers. Based PyramidNet-200, Ø=240 [23], Table 1 shows the comparisons against the state-of-the-art data augmentation and regularization approaches under 300 epochs. The proposed RM achieves 2.35% Top-1 classification error in CIFAR-10, 1.5% better than the baseline (3.85%). It outperforms the two popular regularizers Mixup and CutMix by 0.74% and 0.53% points, respectively.
Performance under Various Network Backbones. The effectiveness RM is further validated across variety of network architectures in CIFAR-100 under the 200-epoch training setting, including ResNet [26], DenseNet [29], and PyramidNet [23]. From Table 3, we observe that RM has consistent improvement in accuracy against the baselines (+1.5~3.5%% and other competitive counterparts (+0.3~1.11%) Notably, for DenseNet- 161, RM shows an absolute gain of 3.2% points over the baseline model and outperforms the competitive CutMix by 1.1%
Ablation Study on Hyperparameter a. To reveal the impact of a, we conduct ablation study by varying 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9} in CIFAR-10 with PyramidNet-200, a=240 backbone under 300-cpoch setting and CIFAR- 100 with ResNet- 18 [26] backbone under 200-epoch setting, respectively. As shown in Fig. 4, RM improves upon the baseline (3.85% and 21.70%) for all considered a values. The best performance is achieved when a = 0.5.
Ablation Study on Hyperparameter w. Next, we examine what is the best value for the loss weight w. We adopt the same experimental setting on CIFAR-10 and CIFAR- 100 as that on a. By fixing a to 0.5, we change E {0, 0.01, 0.05, 0.08, 0.1, 0.2, 0.3, 0.4, 0.5, }} and report their performances average value and standard deviation over 3 runs on each. Notably, when w=0, the RM model is optimized without the consistency loss, but only leverages the historical input-label pairs. Fig. 5 shows that the best performance achieved when w = 0.1, while the performance is not sensitive (i.c., around 0.3%) in CIFAR-10 10 to a wide range of w E [0.
A Allaioon Study on RM Components. To illustrate the performance improvement brought by each RM component individually, we first modify CutMix with the same resize strategy of RM, instead of the original cut operation. Then we add the historical mechanism and consistency loss, respectively. It is observed in Table 2 that the resize strategy can have a slightly positive effect (i.e., +0.23% accuracy) on CutMix for necessary to successfully formulate the correct historical (recursive) paradigm (Fig. 3). Further, the historical mechanism and consistency loss individually improve by +0.55% points and +0.16% points, respectively. Above all, RM achieves total of 2.02% gain over the baseline and 0.94% over CutMix.

---

Detector
Pretrain Backbone
AP
APse
APps
Detector
Pretnii Rockhone
Aphs
AP
AP
Ap
AP
APS
ResNes-50 [26]
39.4
57.6
42.8
ResNes-50 [26]
38.2
58.8
41.4
34,7
55.7
37.2
CutMix (68)
58.4
41.4
CatMix [68]
58.9
34.8
56.0
[7]]
RM (ours)
41.5
59,9
45.1
Mask R-CNN (25]
RR (ours)
39.6
60.4
43.1
35.8
57.3
38.2
PVTv2-B1[c1 [6]]
39.3
57.2
42.5
PVTv2-B1 [6]]
38.5
60.7
41.5
36.1
57.7
38.4
CutMix [68]
60.3
45.5
CatMix [68]
40.6
63.1
59.9
40.1
RM (ours)
42.3
61.0
45.6
RM (ours)
41.2
63.5
44,4
38.1
60.5
40.9
ResNet-50 [26]
40.2
58.4
43.3
ResNer-50 [26]
41.9
60.5
45.5
37.1
57.8
40.1
CutMix (68)
41.3
59.5
44.6
4 CatMix [68]
42.2
60.7
46.0
37.4
58.2
40.4
GPL [42)
(ours)
60.2
45.6
HTC [6]
(ours)
61.4
58.5
PVT [61
1022
58.1
43.2
PVTv2-BI [6]]
43,0
62.9
16.1
39.2
60.5
12.1
CutMix [68]
42.1
60.7
45.5
+ CatMix [68]]
45.2
65.0
49.2
40.7
62.4
44.2
RM (ours)
43.0
61.6
46.5
(ours)
49.8
63.0
Table 7: Object detection fine tuned on COCO with the schedule by ATSS [71] and GFL
Table 8: Object detection and instance segmentation fine- funettned on COCO with the Ix schedale by R-CNN [25] and HTC [6].
Ablation Study on Linear Classifiers. As illustrated in Sec. 3, the historical predictions and the aligned current predictions are derived through separate linear classifiers H. and H.. We conducted comparisons on whether to share the parameters of the two linear layers. The training strategy on CIFAR datasets follows the above. Table 4a shows that unshared weights bring slightly better performance for it enhances model diversity which benefits the contrastive learning.
Ablation Study on Interpolation Mode. As depicted in Table 4b, for the resizing operation on historical inputs, we compare two common down-sampling algorithms, i.e., interpolation methods, "nearest" and "bilinear". As result, "nearest" performs slightly better, so we fix the interpolation method to it by default.
4.2 ImageNet Classification
Dataset. The ImageNet 2012 dataset [14] contains 1.28 million training images and 50K validation images from IK classes. Networks are trained on the training set and the Top-1/-5 errors are reported on the validation set.
Setup. For ImageNet, we also conduct experiments with sufficient training settings (300 epochs) under two major backbone designs, i.c., CNNs [29, 26] and Transformers [61]. For training CNN backbone networks, we employ the augmentations described in CutMix [68] to the input images for a fair comparison. All networks are trained using SGD with momentum of 0.9, weight decay of 1 x 10 4, and GPUs with a mini-batch size of 64 on each to optimize models. The initial learning rate is 0.2 with a linear warmup [26] for five epochs and is then decayed following a cosine schedule [46]. To optimize Transformer backbone networks, we use Adam W [47] with a learning rate of 5 x 10-1 momentum of 0.9, weight decay of 5 x 10-2, and 8 GPUs with mini-batch size of 64 on each. We follow PVT [62] and apply random resizing/cropping of 224x224 pixels, random horizontal flipping [55], label-smoothing regularization [56], and random erasing [73] as the standard data augmentations. The hyperparameters for RM on ImageNet are set to a=0.5, w=0.5.
Comparison with State-of-the-art Regularizers. Based on ResNet-50 [26], we compare RM against series of popular regularizers in Table 6 under the 300-epoch training setting, where RM significantly improves the baseline by absolute 2.88% points in Top-1 accuracy and outperforms the strong CutMix and PuzzleMix by 0.6% and 0.44%, respectively.
Performance under Various Network Backbones. Since Transformer models [16, 45, 62, 61] have made significant progress in image classification due the self-attention mechanism, also demonstrate the effectiveness of RM on the popular Transformer network architecture PVTv2-B1 [61] in ImageNet under the 300-epoch training setting in Table 5, where it remains the superiority.
4.3 Transfer Learning
Benefiting from the explicit multi-sscaee/spaae property and spatial semantic learning of the pro- posed RecursiveMix (RM), we suspect that the pretrained model with RM can transfer well to the downstream tasks, e.g.. object detection, instance segmentation, and semantic segmentation, where multi-scale/sspace semantic information plays an important role for identification.
Object Detection and Instance Segmentation. We conduct experiments using the one-stage object detector ATSS [71], GFL [42, 41, 40], and two-stage detector Mask R-CNN [25]. HTC [6]
7

---

Segmentor
Pretrain Backhone
mloU
mAcc
a/cc
Detector
CL
AP
APse
APra
RexNet-50 [26]
40.90
79:52
41.1
59.4
44.5
ATSS [71]
CutMix [68]
40.96
51.16
79.93
41.5
59.9
45.1
PSPNet (72)
RM (ours)
41.73
52.47
80.01
GFL GFL [42]
41.4
44.9
PVTv2-B1 [6]]
36.48
46.26
76.79
41.9
45.6
CuiMix [68]
37.99
48,70
77.50
(a) Comparisons on objoct detection.
RM (ours)
38.67
49.40
77.93
RexNet- 50 [26]
40.40
51,00
79.54
Segmentor
CL
mloU
mAcc
nAcc
CutMix [68]
41.24
51.79
79.69
PSPNet [72]
41.09
51,72
79,99
RM (ours)
42.30
52.61
80.14
41.73
52.47
80.01
UperNet [67]
PVTv2-B1 [6]]
39.94
50.75
79.02
UperNet [67]
41.88
52.79
79,94
42.30
52.61
80.14
CutMix [68]
41.73
52.99
80.02
RM (ours)
43.26
54.21
80.36
(b) Comparisons on semantic sogmentation.
Table 9: Semantic segrmentation fine luned on ADE20K [75] lor
Table 10: Comparisons on transferring down-
iterations by PSPNet [72] and UperNet [67].
stream tasks w/ and we consistency (CL).
ResNer-50 50 (300 spochs)
Memory
Flops
#P (train)
#P (deploy)
Hours
Top-1 (9))
Baseline
5832.0 MB
4.12G
25.56
25.56 M
73.0
23.68
Mixup [70]
5870.0 MB
4.12G
25.56 M
25.56 M
73.5
22.58
CutMix [68]
5832.0 MB
4.12G
25.56
25.56 M
73.8
21.40
RM (nurs)
5887.0 MR
4.12G
27.61 M
25.56 M
73.8
20.80
Table 11: Comparisons of the training efficiency by hours, ovaluated on 8 TITAN Xp GPUs "#P" denotes the number of parameters.
in COCO [43] dataset with comparisons to the vanilla model and CutMix [68]-pretrained model
based on ResNet-50 [26] and PVTv2-B1 [61]. The training protocol follows the standard 1x (12 epochs) setting described in MMDetection [7]. In Table 7, it is observed that under ATSS, the
RM-pretrained models significantly improve baseline by +2.1 1 AP points with ResNet-50 and +3.0 AP points with PVTv2-B1, 1, while outperforming CutMix [68] pretrained models by +1.4 AP and 0.5 AP. Table 8 indicates the superiority of RM in instance segmentation under Mask R-CNN and HTC, where RM boosts the bbox AP and mask AP by ~2.8 and 2.0 points, respectively.
Semantic Segmentation. Next, we experiment on ADE20K [75]] using two popular algorithms, i.c., PSPNet [72] and UperNet [67] following the semantic segmentation code of MMSegmentattin [12]. Under ResNet-50 [26] and PVTv2-B1 [61] backbone networks, we fine-tune 80k iterations on ADE20K with a batch size of 16 and AdamW [47] optimization algorithm. Table 9 shows that among various pretrained models RM outperforms the baseline and CutMix [68]. Notably, we improve the ResNet-50 by 1.9 and 1.1 mloU compared to baseline and CutMix under UperNet. Also, there an improvement of 3.3 mloU over the baseline under PVTv2-B1 based on UperNet.
4.4
Analyses
Class Activation Mapping Visualization. To demonstrate the benefits of RM, we visualize the class activation map through CAM [74]. We select images that originally contain multiple categories and create CAM heatmaps for each (Fig. 6) to demonstrate the superiority of RM in spatial semantic representation. The visualization shows that RM locates the corresponding objects more accurately than Baseline and CutMix. Thanks to the recursive paradigm of RM, the algorithm can generate inputs that have multiple objects (>2), which increases the diversity of inputs in data space and enriches each instance with multi-scale and spatial-variant views. Thus, the model is more capable of recognizing complex images containing multiple categories or objects that differ greatly in size. Furthermore, the model trained under RM explicitly learns dense semantic representations and can locate the meaningful regions more accurately via the designed consistency loss. More visualization can be found in the Supplementary Material.
RM Enables Efficient Self-distillation. self-distillation by explicitly restricting two spatial representations of one instance. However, it is worth mentioning that the proposed recursive augmentation paradigm and the consistency loss are not isolated from each other. Only in the form of the "recursive mix", can we achieve self-distillation in convenient, efficient, and memory- -frinnyy way. Because in the recursive paradigm, the historical part in the current iteration can be supervised directly by the predictions of the former iteration as they contents. delicately fits the logic of local self-distillation, at minimal cost of storing only the outputs of the last iteration. On the contrary, to implement local self-distillation without the recursive operation, one needs to forward one input through the networks repeatedly [64, 52, 4, 10, 24, 19, 11, 21, 57] or record the information of the whole data
Generally, the consistency loss resembles a variant of local
share identical but hifferentialllyscale
8

---

hatund Semplees Shitiple
Hoop
Cutkk
Nos josent
Eutis
Orsgge
Agple
Da
Back
Puddet France
Coach
Pilaa
Lamp
Starcaer
Hillang ttalions
Milliy Riae
Figure 6: CAM [74] visualization comparing RM with Mixup [20] and CutMix [68] on natural samples with multiple labels.
in an entire epoch [66, 35, 38, 65], which either requires a tremendous computation cost or large memory consumption. Experimentally, RM with consistency loss enhances accuracy by 0.16% over that without in Table 2. We also verify the effectiveness consistency loss on downstream tasks in Table 10. With the models pretrained with consistency loss, GFL and ATSS improve 0.5mAP on COCO while PSPNet and UperNet improve 0.7 mloU on ADE20K. The results show that the introduced consistency loss enhances the performance the downstream tasks via learning the spatial-correlative semantic representation.
Effective Class Number. To illustrate the diversity in the pervision signals of RMM. we denote the number of classes in the regularized target whose supervision values are higher than a given threshold as effective classes. We then record the average effective number over 3 runs under 100 iterations RM and other meth- ods in the ImageNet training set. Note that the label smoothing threshold is roughly set to "10-4". We suspect that if signal value above the label smoothing threshold can be used as effective supervision, then RMM can achieve learning signal containing average of about 4-5 objects in a single image, which confirms the diversity of its supervision (Fig. 7).
Baseline
Mixup
Effective Classes
5
CutMix
(ours)
1
10-3
10-1
10
10-2
10
Threshold
Efficiency. We improve performance with negligible additional memory cost, shown in Table 11. Notably, the training time and computation complexity are competitive with existing methods, which are evaluated on TITAN Xp GPUs.
Figure Comparisons effective during training under different thresholds.
Limitations. Although our method shares the same parameters and FPS as the baseline model during employment, due to the introduction of an additional classification head and the operation of RolAlign, we inevitably increase small number of parameters only during the training process. In addition, it is unclear how to effectively adopt the recursive paradigm to Mixup [70] since the images would be cluttered and unidentifiable via iterative mixup operations.
Conclusion
In this paper, we propose novel regularization method termed RecursiveMix (RM), which leverages the rarely exploited aspect of the historical information: the input- input-prediction-ae prediction-label triplets, to enhance the generalization of deep vision models. RM shares several good properties according to its historical mechanism, and it consistently improves the recognition accuracy competitive vision benchmarks with considerably negligible additional budgets. We hope RM can serve as a simple yet effective baseline for the community.
Acknowledgments and Disclosure of Funding
This paper is supported by the National Key R&D Plan of the Ministry of Science and Technology (Project 20.2000AAA0004400), the Young Scientists Fund of the National Natural Science Founda- tion of China (Grant No. No.62206134) and the National Science Fund of China (Grant No. No.U17132088,

---

References
[1] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views, arXiv preprint arXiv:1906.009100 2019.
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to -uperuupe learning. arXiv preprint AKY::995.002244 2019. 3
[3] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. NeurlPS, 1993. 3
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- self-supervised vision transformers, In ICCV, 2021. 3, 4, 8
[5] Jie-Neng Chen. Shuyang Sun, Ju He, Philip Tomr Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. arXiv preprint urXiv:2111. 098333, 2021. 3
[6] Kai Chen, Jiangmiao Pang, Jiaqi Wang. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In CVPR, 2019. 7
[7] Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu. Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 0906699066.300 2019. 8
[8] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmask data augmentation. arXiv preprint arXiv:2001. 2020.
[9] Ting Chen, Simon Komblith, Mohammad Norouzi, and Genffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI,, 2020.
[10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 042977 2020. 3,8
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised visual transformers. In ICCV, 2021. 3,
[12] MMSegmentation Contributors. MMScgmentation: Openmmlab semantic segmentation toolbox and
benchmark. https://github. wwtccccm/oooccccecccccccccccc open mm1 /bm.m//mmmtttttainnnnn 2020. 8
[13] Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, and Nasser M Nasrabadi. Supermix: Supervising the mixing data augmentation. CVPR, 2021, 1,3
[14] Jia Deng, Wci Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
[15] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: I 1708. 2017. 1, 5,
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010.11922 2020. 7
[17] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2011. 3
[18] Mojtaba Faramarzi, Mohammad Amini, Akilesh Badrinaaraayanan, Vikas Verma, and Sarath Chandar. Patchup: A regularization technique for convolutional neural networks. arXiv preprint 47X19:2006.7794. 2020. 3,
[19] Weixin Feng. Yuanjiang Wang, Lihua Ma. Ye Yuan, and Chi Zhang. Temporal knowledge consistency for unsupervised visual representation learning. In ICCV, 2021. 3, 8.8
[20] Golnaz Ghiasi, Tsung-Yi Yi Lin, and Quoe Le. Dropblock: A regularization method for convolutional networks. arXiv preprint 1810 10310.12890 /2890, 2018. 5,
[21] Jean- Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your latent: A new approach to self-supervised learning. In NeurIPS, 2020. 3,
10

---

[22] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR. 2006. 3,
[23] Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In CVPR, 2017. 5, 6
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 8,8.
[25] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 1, 4, 7
[26] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 1, 2, 5, 6, 8
[27] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang. Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In CVPR, 2019.
[28] Minui Hong, Jinwoo Choi, and Gunhee Kim. Stylemix: Separating content and style for enhanced data augmentation. CVPR, 2021. 3,5
[29] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. 1, 6,
[30] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In ECCV, 2016. 1, 2, 6,
[31] Shaoli Huang, Xinchao Wang, Dacheng Tao. Snapmix: Semantically proportional mixing for augmenting fine-grained data. AAAI, 2021.
[32] Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. Pretraining with contrastive sentence objectives improves discourse performance of language models. arXiv preprint arXiv:2005. 10389, 2020. 3
[33] Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity, arXiv preprint
2021. 3
[34] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In ICMI, 2020. 2,
[35] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In /CCV, 2021. 3,9
[36] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[37] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009. 5
[38] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017. 3,
[39] Boyi Li, Felix Wu, Ser-Nam Lim, Serge Belongic, and Kilian Q Weinberger. On fcature normalization and data augmentation. In CVPR, 2021. 3, 5, 6
[40] Xiang Li, Chengqi Lv. Wenhai Wang, Gang Li. Lingfeng Yang. and Jian Yang. Generalized focal loss: Towards efficient representation learning for dense object detection. TPAMI, 2022.
[41] Xiang Li, Wenhai Wang, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. In CVPR, 2021. 7
[42] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li. Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. NeurlPS, 2020.
I, 7,8
[43] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 8
[44] Liyuan Liu, Haoming Jiang, Pengeheng Hc, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In /CLR, 2020. 3
[45] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 2021. 7

---

[46] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv:1608.0300 03983, 2016. 6,
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 3, 7,8
[48] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NeurlPS, 2013. 3
[49] Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint arXiv: 19Xivv1906.000 02629, 2019. 5.5
[50] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 10t.iv:807..37777 2018.
[51] Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim. Spatially consistent representation learning. In CVPR. 2021. 3
[52] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko, Adversarial dropout regularization. arXiv preprint arxiv:/71 2017. 3,8
[53] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout:
simple way to prevent neural networks from overfitting. JMLR, 2014. 2,
[54] Ilya Sutskever, James Martens. George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. ICML, 2013. 3
[55] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich, Going deeper with convolutions. In CVPR, 2015. 7
[56] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architecture for computer vision. CVPR, 2016.
[57] Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurlPS, 2017. 3, 3,8
[58] AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bac, et al. Saliencymix: saliency guided data augmentation strategy for better regularization. arXiv preprint arXiv:2006.01911 2020. 3
[59] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICMI., 2019. 1,
2, 3, 5, 6
[60] Devesh Walawalkar, Zhiqiang Shen, Zechun Liu, and Marios Savvides. Attentive cutmix: An enhanced data augmentation approach for deep learning based image classification. arXiv preprint 20xv:20033313445 2020. 1.3
[61] Wenhai Wang, Enze Xie, Xiang Li. Deng- Ling-Ping&& Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. arXiv preprint a00i 13797, 2021. 2, 6, 7,8
[62] Wenhai Wang, Enze Xie, Xiang Li, Deng- Ling-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2 102 /2/22, 2021. 1,7
[63] Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R Scott. Cross-batch memory for embedding learning. In CVPR, 2020. 3
[64] Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. R-drop: regularized dropout for neural networks. NewrIPS, 2021. 3, 8
[65] Zhirong Wu, Alexei A Efros, and Stella X Yu. Improving generalization via scalable neighborhood component analysis. In ECCV, 2018. 3,9
[66] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised fcature learning via non- paramaarettic instance discrimination. In CVPR, 2018. 3,
[67] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018. 1,8
12

---

[68] Sangdoo Yun. Dongyoon Han, Seong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019. 2, 4, 5, 6. 8,
[69] Matthew D Zeiler. Adadelta: adaptive learning rate method. arXiv preprint Xrv::
/212.5000 2012. 3
[70] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2017. 2, 3, 4, 6, 8,
[71] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor- based and anchor-free detection via adaptive training sample selection. In CVPR, 2020. 2, 7,
[72] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiay Jia. Pyramid scene parsing network. In CVPR, 2017. 1,8
[73] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In AAAI, 2020. 7
[74] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. CVPR, 2016. 8,
[75] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the adc20k dataset. IJV,, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the broader impacts in the Supplementary Material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions nceded to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Results are reported over 3 runs on the CIFAR dataset (see Fig. 4 and Fig. 5).
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] They are publicly available for
research purpose.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] There are no new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] They are publicly available for research purpose.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] They are publicly available for research purpose.
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "uSJSXFHPgh", "reviewer": "Reviewer_1VFL", "summary": "The authors of this paper propose to increase the model generalizability via historical mix-up. Specifically, the compositional training batch from previous time-step is stored and ultilized in the training process at current time-step.", "questions": "Please refer to Sec.2.", "limitations": "Please refer to Sec.2.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses:\n\n1. Is there any experimental comparison with hierarchical mixed-up in large-size batch? Specifically, the comparison with hierarchical mixed-up within single batch, rather than using the last time-step information. Since during the training process of large datasets, the number of batch in each epoch could be large, which may cause two problems: \na) Over-memorizing the information(images) appears in the early batches; \nb) Unexpected semantic dependencies caused by certain information co-occurance.\n\n2) Line38. The proposed recursive-mix improves the inner-data diversity within a batch, while decrease the intra-data diversity within an epoch by information paste.\n\n3) Although the increase of RM's model complexity compared with others is small, is there any possible that the performance gain comes from the increased model complexity? Is there any experiment to verify this assumption?\n\n4) In Eq(5). a) Since H and H' are with different weights, which part of the model is actually optimized for distribution alignment? The encoder ot the projector H?\nb) The network is updated after processed each batch, will the KL-term affacts/prevent the optimization process? Another possible is that it will result in updating momentum appears in contrative learning methods, is there any possible that the performance gain actually comes from the momentum rather than the mix-up process?\nc) The information loss is ignored during the mix-up process. Even an instance can still be observed at current stage, it is hard to ensure that the information loss is aligned with the scale loss. For example, without the tiny boarders, a billboard will be same with the wall, thus its key information may disappear within few time-steps.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jJZ4y760R2E", "reviewer": "Reviewer_3KBU", "summary": "This paper proposes an interesting idea on data augmentation via mixture of samples. Unlike typical ways of mix-based data augmentation methods, this work performs iterative mixtures, which is able to reuse the augmented mixed samples and thereby generates more diverse samples. The idea is simple yet effective, which is validated by extensive experiments.", "questions": "1. Is such recursive mixture operation performed the same times of iterations of the whole optimization? Or is there an upper-bound on the recursive times? Intuitively, more recursive times leads to richer and more diverse semantics in the augmented samples, however, incorporating too many  classes of semantics may degenerate the performance. Maybe it is better to conduct experiments to investigate the effect of recursive times of mixture operations.\n\n2. It is claimed that the introduced consistency loss can help the model to learn the spatial-correlative semantic representation, which potentially benefit the downstream tasks like object detection or semantic segmentation. Thus, it is better to conduct such ablation study to verify the effect of the consistency loss on these two tasks. ", "limitations": "Refer to the questions posed above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of recursive mixtures of samples is novel.\n2. Extensive experiments demonstrates the effectiveness of the method.\n3. The paper is written well and easy to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bnP5qpblAct", "reviewer": "Reviewer_HH8z", "summary": "This paper introduces a recursive data mixing augmentation called RecursiveMix. It pastes a historical image patch onto the current training sample to promote the data diversity for image mixing. Furthermore, a consistency loss is introduced to force the local pasted patch prediction is invariant to the background. Extensive evaluation on image classification, object detection, and semantic segmentation shows that RM largely improves performance.", "questions": "the proposed method needs an additional task head, which inevitably adds on the training computation. However,  the authors only compare the cost of the classification task, with a linear head. In contrast, the task head in detection and segmentation is extremely large compared to the backbone. Could the author please compare the computation cost overhead on detection and segmentation?", "limitations": "The limitation is discussed in Line 313-319.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n1. The method is well-motivated and finely designed. The two key components are tightly coupled: recursive data mixing promotes image diversity, while local patch consistency calls for representation invariance regardless of background. \n2. The experiments are truly comprehensive. The author not only testifies the performance on a wide range of mainstream visual tasks but also provides an extensive ablation study on each component of the method.\n3. The proposed method shows compelling improvements compared with existing methods.\n4. The paper is clearly written and easy to follow.\n\n# Weakness\n1. The current method seems not to provide striking new insights. The current RM design is mainly a improved version of exiting data mixing pipeline, by considering the historical data information. I do not mean the derived approach is \"not novel\". I just want to see more fundamental insights or rigorous discussion on \"why the current design is reasonable?\" and \"How does it improve the performance?\". Good performance itself just indicates the results, but tell nothing about the reason.\n2. No ablation on $\\lambda$. In lines 127-131, the author claims that \"$\\lambda$ denotes the proportion of the historical images, which is not suggested to be quite large\". However, no ablation is provided on resizing ratio $\\lambda$. I strongly suggest providing the results. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_1p5Mi4FO64", "reviewer": "Reviewer_oBX3", "summary": "This paper presents RecursiveMix that applies CutMix iteratively. The proposed method is efficient and effective and shows strong generalization ability for different tasks including image classification, object detection, instance and semantic segmentation.", "questions": "See weaknesses", "limitations": "See weaknesses", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n\n1. The idea is simple and effective, which makes it easy to follow.\n\n2. The performance it achieves is impressive. It can lead to consistent gain for four different tasks and all of them were evaluated on challenging benchmarks. Meanwhile, it only requires marginal computational cost during training and is completely cost-free during inference.\n\n3. The experiments and ablation studies are comprehensive and extensive. The authors evaluate their methods on a wide range of tasks and datasets, which can help to attract more audiences with different backgrounds.\n\n\n### Weaknesses\n\n1. RecursiveMix changes the previous crop-based CutMix to resize-based CutMix and keeps the original label assigning strategy in CutMix, which seems counter-intuitive. In this way, the foreground area of the crop in the mixture will be always kept since it will resize the whole image and keep all content there. But the crop can cover the foreground area of the other image, which means the valid object can sometimes disappear. Intuitively, the label assignment under such circumstances should always keep the label of the cropped image as 1 since the foreground object in the crop always exists, and the label confidence for the other image in the mixture should be lower. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9DE_H4KUnG2", "reviewer": "Reviewer_kceg", "summary": "This paper proposes a novel data augmentation strategy (RecursiveMix). The proposed augmentation recursively resizes the historical input and then fills it into the current batch. Experiments demonstrate that the proposed RecursiveMix consistently outperforms the popular augmentations such as Mixup and CutMix. ", "questions": "Please address the above weaknesses.", "limitations": "The authors have discussed the limitations. No potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed augmentation is novel and interesting. \n2. Abundant experiment results are provided. According to the experiment results, RecursiveMix can consistently improve the performance.\n\nWeaknesses:\n1. The discussion subsection 3.2 is not informative. Most information has been included in the introduction.\n\n2. The paper also lacks an experiment in which several basic augmentations, e.g. cropping, scaling and flipping, have been applied and then the authors further add and compare RecursiveMix to advanced augmentations, e.g. mixup and cutmix. In other words, the authors should consider the joint usage of multiple augmentations.\n\n3. It will be better to give the standard deviation of multiple experiments. \n\n4. The proposed method involves two hyper-parameter \\alpha and \\omega. The ablation study shows that the performance is sensitive to the hyper-parameters considering that the improvements over Mixup and CutMix are usually less than 1%. \n\nSmall problems:\n1. The font in figures can be larger for reading easily.\n2. The interval between bars in Figure 1 can be uniform. \n3. Syntax error in Line 129 “Because …”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept