SlowGuess/NeurIPS_2023_OCR huggingface dataset:
generalizes (1) and the weight vector is defined as  $s = \mathbf{1}_m^\top M^j \in \mathbb{R}^n$ . This is exactly the vector used to normalize the columns of  $M^j$  in (11). The motivation is that the columns of  $M^j$  are the original interpolation vectors weighted by attention: A small  $\ell_1$  norm indicates that for the given position  $j$ , we are sampling from examples of low attention, hence the loss is to be discounted. The total number of loss terms per mini-batch is now  $nr$ .
We use a mini-batch of size  $b = 128$  examples in all experiments. Following manifold mixup [51], for every mini-batch, we apply MultiMix with probability 0.5 or input mixup otherwise. For MultiMix, the default settings are given in subsection 4.6. We use PreActResnet-18 (R-18) [21] and WRN16-8 [63] as encoder on CIFAR-10 and CIFAR-100 datasets [29]; R-18 on TinyImagenet [60] (TI); and Resnet-50 (R-50) and ViT-S/16 [13] on ImageNet [44]. To better understand the effect of mixup in ViT, we evaluate MultiMix and Dense MultiMix on ImageNet without using strong augmentations like Auto-Augment [9], Rand-Augment [10], random erasing [67] and CutMix [62]. We reproduce TransMix [7] and TokenMix [35] using these settings.
4.2 Results: Image classification and robustness
Image classification In Table 2 we observe that MultiMix and Dense MultiMix already outperform SoTA on all datasets except CIFAR-10 with R-18, where they are on par with Co-Mixup. Dense MultiMix improves over vanilla MultiMix and its effect is complementary on all datasets. On TI for example, Dense MultiMix improves over MultiMix by  $1.33\%$  and SoTA by  $1.59\%$ . We provide additional analysis of the embedding space on 10 classes of CIFAR-100 in subsection 4.4.
DATASET
NETWORK	CIFAR-10	CIFAR-100	TI
R-18	W16-8	R-18	W16-8	R-18
Baseline†	95.41±0.02	94.93±0.06	76.69±0.26	78.80±0.55	56.49±0.21
Manifold mixup [51]†	97.00±0.05	96.44±0.02	80.00±0.34	80.77±0.26	59.31±0.49
PuzzleMix [26]†	97.04±0.04	97.00±0.03	79.98±0.05	80.78±0.23	63.52±0.42
Co-Mixup [25]†	97.10±0.03	96.44±0.08	80.28±0.13	80.39±0.34	64.12±0.43
AlignMixup [48]†	97.06±0.04	96.91±0.01	81.71±0.07	81.24±0.02	66.85±0.07
ζ-Mixup [1]*	96.26±0.04	96.35±0.04	80.46±0.26	79.73±0.15	63.18±0.14
MultiMix (ours)	97.07±0.03	97.06±0.02	81.82±0.04	81.44±0.03	67.11±0.04
Dense MultiMix (ours)	97.09±0.02	97.09±0.02	81.93±0.04	81.77±0.03	68.44±0.05
Gain	-0.01	+0.09	+0.22	+0.53	+1.59
Table 2: Image classification on CIFAR-10/100 and TI (TinyImagenet). Mean and standard deviation of Top-1 accuracy (\%) for 5 runs. R: PreActResnet, W: WRN.  $\star$ : reproduced,  $\dagger$ : reported by AlignMixup, Bold black: best; Blue: second best; underline: best baseline. Gain: improvement over best baseline. Additional comparisons are in subsection A.2.
ATTACK	FGSM	PGD
DATASET NETWORK	CIFAR-10	CIFAR-100	TI	CIFAR-10	CIFAR-100
R-18	W16-8	R-18	W16-8	R-18	R-18	W16-8	R-18	W16-8
\( Baseline {}^{ \dagger } \)	\( {88.8} \pm {0.11} \)	\( {88.3} \pm {0.33} \)	\( {87.2} \pm {0.10} \)	\( {72.6} \pm {0.22} \)	\( {91.9} \pm {0.06} \)	\( {99.9} \pm {0.0} \)	\( {99.9} \pm {0.01} \)	\( {99.9} \pm {0.01} \)	\( {99.9} \pm {0.01} \)
\( \text{Manifold mixup}\left\lbrack {51}\right\rbrack {}^{ \dagger } \)	\( {76.9} \pm {0.14} \)	\( {76.0} \pm {0.04} \)	\( {80.2} \pm {0.06} \)	\( {56.3} \pm {0.10} \)	\( {89.3} \pm {0.06} \)	\( {97.2} \pm {0.01} \)	\( {98.4} \pm {0.03} \)	\( {99.6} \pm {0.01} \)	\( {98.4} \pm {0.03} \)
\( \text{PuzzleMix}\left\lbrack {26}\right\rbrack {}^{ \dagger } \)	\( {57.4} \pm {0.22} \)	\( {60.7} \pm {0.02} \)	\( {78.8} \pm {0.09} \)	\( {57.8} \pm {0.03} \)	\( {83.8} \pm {0.05} \)	\( {97.7} \pm {0.01} \)	\( {97.0} \pm {0.01} \)	\( {96.4} \pm {0.02} \)	\( {95.2} \pm {0.03} \)
\( Co-Mixup\left\lbrack {25}\right\rbrack {}^{ \dagger } \)	\( {60.1} \pm {0.05} \)	\( {58.8} \pm {0.10} \)	\( {77.5} \pm {0.02} \)	\( {56.5} \pm {0.04} \)	-	\( {97.5} \pm {0.02} \)	\( {96.1} \pm {0.03} \)	\( {95.3} \pm {0.03} \)	\( {94.2} \pm {0.01} \)
\( \text{AlignMixup}\left\lbrack {48}\right\rbrack {}^{ \dagger } \)	\( {54.8} \pm {0.03} \)	\( {56.0} \pm {0.05} \)	\( {74.1} \pm {0.04} \)	\( {55.0} \pm {0.03} \)	\( {78.8} \pm {0.03} \)	\( {95.3} \pm {0.04} \)	\( {96.7} \pm {0.03} \)	\( {90.4} \pm {0.01} \)	\( {92.1} \pm {0.03} \)
\( \zeta -Mixup [1]{}^{ \star } \)	\( {72.8} \pm {0.23} \)	\( {67.3} \pm {0.24} \)	\( {75.3} \pm {0.21} \)	\( {68.0} \pm {0.21} \)	\( {84.7} \pm {0.18} \)	\( {98.0} \pm {0.06} \)	\( {98.6} \pm {0.03} \)	\( {97.4} \pm {0.10} \)	\( {96.1} \pm {0.10} \)
\( MultiMix (ours) \)	\( {54.1} \pm {0.09} \)	\( {55.3} \pm {0.04} \)	\( {73.8} \pm {0.04} \)	\( {54.5} \pm {0.01} \)	\( {77.5} \pm {0.01} \)	\( {94.2} \pm {0.04} \)	\( {94.8} \pm {0.01} \)	\( {90.0} \pm {0.01} \)	\( {91.6} \pm {0.01} \)
\( Dense MultiMix (ours) \)	\( {54.1} \pm {0.01} \)	\( {53.3} \pm {0.03} \)	\( {73.5} \pm {0.03} \)	\( {52.9} \pm {0.04} \)	\( {75.5} \pm {0.04} \)	\( {92.9} \pm {0.04} \)	\( {92.6} \pm {0.01} \)	\( {88.6} \pm {0.03} \)	\( {90.8} \pm {0.01} \)
Gain	+0.7	+2.7	+0.6	+2.1	+3.3	+2.4	+3.5	+1.4	+1.3
In Table 3 we observe that on ImageNet with R-50, vanilla MultiMix outperforms all methods except AlignMixup. Dense MultiMix outperforms all SoTA with both R-50 and ViT-S/16, bringing an overall gain of  $3\%$  over the baseline with R-50 and  $2.2\%$  with ViT-S/16. The gain over AlignMixup with R-50 is small, but it is impressive that it comes with only linear interpolation. To better isolate the effect of each method, we reproduce TransMix [7] and TokenMix [35] with ViT-S/16 using their official code with our settings, i.e., without strong regularizers like CutMix, Auto-Augment, Random-Augment etc. MultiMix is on par, while Dense MultiMix outperforms them by  $1\%$ .
Training speed Table 3 also shows the training speed as measured on NVIDIA V-100 GPU including forward and backward pass. The vanilla MultiMix has nearly the same speed with the baseline, bringing an accuracy gain of  $2.49\%$  with R-50. Dense MultiMix is slightly slower, increasing same for all methods.
Table 4: Robustness to FGSM & PGD attacks. Mean and standard deviation of Top-1 error (%) for 5 runs: lower is better.  $\star$ : reproduced,  $\dagger$ : reported by AlignMixup. Bold black: best; Blue: second best; underline: best baseline. Gain: reduction of error over best baseline. TI: TinyImagenet. R: PreActResnet, W: WRN. Comparison with additional baselines is given in subsection A.2.
NETWORK METHOD	RESNET-50	ViT-S/16
SPEED	ACC	SPEED	ACC
\( Baseline {}^{ \dagger } \)	1.17	76.32	1.01	73.9
Manifold mixup \( {\left\lbrack {51}\right\rbrack }^{ \dagger } \)	1.15	77.50	0.97	74.2
PuzzleMix \( {\left\lbrack {26}\right\rbrack }^{ \dagger } \)	0.84	78.76	0.73	74.7
Co-Mixup \( {\left\lbrack {25}\right\rbrack }^{ \dagger } \)	0.62	-	0.57	74.9
TransMix [7]*	-	-	1.01	75.1
TokenMix [35]*	-	-	0.87	75.3
AlignMixup \( {\left\lbrack {48}\right\rbrack }^{ \dagger } \)	1.03	79.32	-	-
MultiMix (ours)	1.16	78.81	0.98	75.2
Dense MultiMix (ours)	0.95	79.42	0.88	76.1
Gain	+0.1	+1.2
Table 3: Image classification and training speed on ImageNet. Top-1 accuracy (%): higher is better. Speed: images/sec  $(\times 10^{3})$  higher is better.†: reported by AlignMixup;‡: reproduced. Bold black: best; Blue: second best; underline: best baseline. Gain: improvement over best baseline. Comparison with additional baselines is given in subsection A.2.
Robustness to adversarial attacks We follow the experimental settings of AlignMixup [48] and use  $8/255 l_{\infty}$ $\epsilon$ -ball for FGSM [16] and  $4/255 l_{\infty}$ $\epsilon$ -ball with step size 2/255 for PGD [38] attack.
AlignMixup
Figure 3: Embedding space visualization for 100 test examples per class of 10 randomly chosen classes of CIFAR-100 with PreActResnet-18, using UMAP [39].
MultiMix (ours)
Dense MultiMix (ours)
In Table 4 we observe that MultiMix is already more robust than SoTA on all datasets and settings. Dense MultiMix also increases the robustness and is complementary.
The overall gain is more impressive than in classification according to Table 2. For example, against the strong PGD attack on CIFAR-10 with W16-8, the SoTA Co-Mixup improves the baseline by  $3.8\%$  while Dense MultiMix improves it by  $7.3\%$ , which is double. MultiMix and Dense MultiMix outperform Co-Mixup and PuzzleMix by  $3 - 6\%$  in robustness on CIFAR-10, even though they are on-par on classification. There is also a significant gain over SoTA AlignMixup by  $1 - 3\%$  in robustness to FGSM on TinyImageNet and to the stronger PGD.
4.3 Results: Transfer learning to object detection
We evaluate the effect of mixup on the generalization ability of a pre-trained network to object detection as a downstream task. Following the settings of CutMix [62], we pre-train R-50 on ImageNet with mixup methods and use it as the backbone for SSD [36] with fine-tuning on Pascal VOC07+12 [15] and Faster-RCNN [43] with fine-tuning on MS-COCO [34].
In Table 5, we observe that, while MultiMix is slightly worse than AlignMixup on Pascal VOC07+12, Dense MultiMix brings improvements over the SoTA on both datasets and is still complementary. This is consistent with classification results. Dense MultiMix brings a gain of  $0.8\mathrm{mAP}$  on Pascal VOC07+12 and  $0.35\mathrm{mAP}$  on MS-COCO.
DATASET DETECTOR	VOC07+12	MS-COCO
SSD	SPEED	FR-CNN	SPEED
Baseline†	76.7	9.7	33.27	23.6
Input mixup†	76.6	9.5	34.18	22.9
CutMix†	77.6	9.4	35.16	23.2
AlignMixup†	78.4	8.9	35.84	20.4
MultiMix (ours)	77.9	9.6	35.93	23.2
Dense MultiMix (ours)	79.2	8.8	36.19	19.8
Gain	+0.8	+0.35
Table 5: Transfer learning to object detection. Mean average precision (mAP, %): higher is better.†: reported by AlignMixup. Bold black: best; Blue: second best; underline: best baseline. Gain: increase in mAP. Speed: images/sec: higher is better.
4.4 Analysis of the embedding space
Qualitative analysis We qualitatively analyze the embedding space on 10 CIFAR-100 classes in Figure 3. We observe that the quality of embeddings of the baseline is extremely poor with severely overlapping classes, which explains its poor performance on image classification. All mixup methods result in clearly better clustered and more uniformly spread classes. AlignMixup [48] yields five somewhat clustered classes and five moderately overlapping ones. Our best setting, i.e., Dense MultiMix, results in five tightly clustered classes and another five somewhat overlapping but less than all competitors.
Quantitative analysis We also quantitatively assess the embedding space on the CIFAR-100 test set using alignment and uniformity [52]. Alignment measures the expected pairwise distance of examples in the same class. Lower alignment indicates that the classes are more tightly clustered. Uniformity measures the (log of the) expected pairwise similarity of all examples using a Gaussian kernel as a similarity function. Lower uniformity indicates that classes are more uniformly spread in the embedding space.
Figure 4: Ablation study on CIFAR-100 using R-18. (a) Interpolation layers (R-18 block; 0: input mixup). (b) Number  $n$  of interpolated examples per mini-batch with  $m = b$  (Default) and  $m = 2$  (Standard). (c) Number  $m$  of examples being interpolated, with  $n = 1000$  (Default) and  $n = 100$  (Standard). (d) Fixed value of Dirichlet parameter  $\alpha$ .
On CIFAR-100, we obtain alignment 3.02 for baseline, 2.04 for AlignMixup, 1.27 for MultiMix and 0.92 for Dense MultiMix. We also obtain uniformity -1.94 for the baseline, -2.38 for AlignMixup [48], -4.77 for MultiMix and -5.68 for Dense MultiMix. These results validate the qualitative analysis of Figure 3.
4.5 Manifold intrusion analysis
Manifold intrusion [17] can occur when mixed examples are close to classes other than the ones being interpolated in the embedding space. To evaluate for manifold intrusion, we define the intrusion distance (ID) as the minimum distance of a mixed embedding to the clean embeddings of all classes except the ones being interpolated, averaged over a mini-batch:  $\mathrm{ID}(\widetilde{Z}, Z) = \frac{1}{|\widetilde{Z}|} \sum_{\widetilde{z} \in \widetilde{Z}} \min_{z \in Z} \| \widetilde{z} - z \|^2$ . Here,  $\widetilde{Z}$  is the set of mixed embeddings in a mini-batch and  $Z$  is the set of clean embeddings from all classes other than the ones being interpolated in the mini-batch. Intuitively, a larger  $\mathrm{ID}(\widetilde{Z}, Z)$  denotes that mixed embeddings in  $\widetilde{Z}$  are farther away from the manifold of other classes in  $Z$ , thereby preventing manifold intrusion.
Averaged over the training set of CIFAR-100 using Resnet-18, the intrusion distance is 0.46 for Input Mixup, 0.47 for Manifold Mixup, 0.45 for AlignMixup, 0.46 for MultiMix and 0.47 for Dense MultiMix. This is roughly the same for most SoTA mixup methods. This may be due to the fact that true data occupy only a tiny fraction of the embedding space, thus generated mixed examples lie in empty space between class-specific manifolds with high probability. The visualization in Figure 3 indeed shows that the embedding space is sparsely populated, even in two dimensions. This sparsity is expected to grow exponentially in the number of dimensions, which is in the order of  $10^{3}$ .
4.6 Ablations
All ablations are performed using R-18 on CIFAR-100. We study the effect of the layer where we interpolate, the number  $n$  of generated examples per mini-batch, the number  $m$  of examples being interpolated and the Dirichlet parameter  $\alpha$ . More ablations are given in the supplementary.
Interpolation layer For MultiMix, we use the entire network as the encoder  $f_{\theta}$  by default, except for the last fully-connected layer, which we use as classifier  $g_{W}$ . Thus, we interpolate embeddings in the deepest layer by default. Here, we study the effect of different decompositions of the network  $f = g_{W} \circ f_{\theta}$ , such that interpolation takes place at a different layer. In Figure 4(a), we observe that mixing at the deeper layers of the network significantly improves performance. The same behavior is observed with Dense MultiMix, which validates our default choice.
It is interesting that the authors of input mixup [65] found that convex combinations of three or more examples in the input space with weights from the Dirichlet distribution do not bring further gain. This agrees with the finding of SuperMix [11] for four or more examples. Figure 4(a) suggests that further gain emerges when mixing in deeper layers.
Number  $n$  of generated examples per mini-batch This is important since our aim is to increase the amount of data seen by the model, or at least part of the model. We observe from Figure 4(b) that accuracy increases overall with  $n$  and saturates for  $n \geq 1000$  for both variants of MultiMix. The improvement is more pronounced when  $m = 2$ , which is standard for most mixup methods.
Our best solution, Dense MultiMix, works best at  $n = 1000$  and  $n = 10,000$ . We choose  $n = 1000$  as default, given also that the training cost increases with  $n$ . The training speed as a function of  $n$  is given in the supplementary and is nearly constant for  $n \leq 1000$ .
Number  $m$  of examples being interpolated We vary  $m$  between 2 (pairs) and  $b = 128$  (entire mini-batch) by using  $\Lambda' \in \mathbb{R}^{m \times n}$  drawn from Dirichlet along with combinations (subsets) over the mini-batch to obtain  $\Lambda \in \mathbb{R}^{b \times n}$  with  $m$  nonzero elements per column in (6),(7). We observe in Figure 4(c) that for both MultiMix and Dense MultiMix the performance increases with  $m$ . The improvement is more pronounced when  $n = 100$ , which is similar to the standard setting ( $n = b = 128$ ) of most mixup methods. Our choice of  $m = b = 128$  brings an improvement of  $1 - 1.8\%$  over  $m = 2$ . We use this as our default setting.
Dirichlet parameter  $\alpha$  Our default setting is to draw  $\alpha$  uniformly at random from  $[0.5, 2]$  for every interpolation vector (column of  $\Lambda$ ). Here we study the effect of a fixed value of  $\alpha$ . In Figure 4(d), we observe that the best accuracy comes with  $\alpha = 1$  for most MultiMix variants, corresponding to the uniform distribution over the convex hull of the mini-batch embeddings. However, all measurements are lower than the default  $\alpha \sim U[0.5, 2]$ . For example, from Table 2(a) (CIFAR-100, R-18), Dense MultiMix has accuracy 81.93, compared with 81.59 in Figure 4(d) for  $\alpha = 1$ .
The take-home message of this work is that, instead of devising smarter and more complex interpolation functions in the input space or intermediate features, it is more beneficial to use MultiMix, even though its interpolation is only linear. In this work, we combine three elements:
1. Increase the number  $n$  of generated mixed examples beyond the mini-batch size  $b$ .
2. Increase the number  $m$  of examples being interpolated from  $m = 2$  (pairs) to  $m = b$ .
3. Perform interpolation in the embedding space rather than the input space.
Figure 4 shows that all three elements are important in achieving SoTA performance and removing any one leads to sub-optimal results. We discuss their significance and interdependence here.
Increasing the number  $n$  of generated examples The expected risk is defined as an integral over the underlying continuous data distribution. Since that distribution is unknown, the integral is approximated by a finite sum, i.e., the empirical risk. A better approximation is the vicinal risk, where a number of augmented examples is sampled from a distribution in the vicinity of each training example, thus increasing the number of loss terms per training example. Input mixup [65] is inspired by the vicinal risk. However, as a practical implementation, all mixup methods still generate  $b$  mixed examples and thus incur  $b$  loss terms for a mini-batch of size  $b$ . As discussed in section 2, previous works have used more loss terms than  $b$  per mini-batch, but not for mixup.
Our hypothesis for the significance of element 1 is that more mixed examples, thus more loss terms by interpolation, provide a better approximation of the expected risk integral. Dense interpolation further increases the number of loss terms, thus further improving the quality of approximation.
Increasing the number  $m$  of examples being interpolated As discussed in section 2, previous works, starting from input mixup [65] have attempted to interpolate  $m > 2$  examples in the input space by sampling  $\lambda$  from Dirichlet or other distributions but have found the idea not effective for large  $m$ . Our finding is that element 2 becomes effective only by interpolating in the embedding space, that is, element 3.
Interpolating in embedding space Element 3 is originally motivated by Manifold Mixup [51], where "interpolations in deeper hidden layers capture higher level information [64]." ACAI [4] explicitly studies interpolation in the latent space of an autoencoder to produce a smooth semantic warping effect in data space. This suggests that nearby points in the latent space are semantically similar, which in turn improves representation learning. Mixed examples generated by sampling in the embedding space lie on the learned manifold. We hypothesize that the learned manifold is a good surrogate of the true, unknown data manifold.
A natural extension of this work is to settings other than supervised classification. A limitation is that it is not straightforward to combine the sampling scheme of MultiMix with complex interpolation methods, unless they are fast to compute in the embedding space.
This work was in part supported by the ANR-19-CE23-0028 MEERQAT project and was performed using the HPC resources from GENCI-IDRIS Grant 2021 AD011012528.