SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shashanka Venkataramanan<sup>1</sup>
Ewa Kijak<sup>1</sup>
Laurent Amsaleg<sup>1</sup>
Yannis Avrithis²
<sup>1</sup>Inria, Univ Rennes, CNRS, IRISA
$^{2}$ Institute of Advanced Research on Artificial Intelligence (IARAI)
Mixup refers to interpolation-based data augmentation, originally motivated as a way to go beyond empirical risk minimization (ERM). Its extensions mostly focus on the definition of interpolation and the space (input or embedding) where it takes place, while the augmentation process itself is less studied. In most methods, the number of generated examples is limited to the mini-batch size and the number of examples being interpolated is limited to two (pairs), in the input space.
We make progress in this direction by introducing MultiMix, which generates an arbitrarily large number of interpolated examples beyond the mini-batch size, and interpolates the entire mini-batch in the embedding space. Effectively, we sample on the entire convex hull of the mini-batch rather than along linear segments between pairs of examples.
On sequence data we further extend to Dense MultiMix. We densely interpolate features and target labels at each spatial location and also apply the loss densely. To mitigate the lack of dense labels, we inherit labels from examples and weight interpolation factors by attention as a measure of confidence.
Overall, we increase the number of loss terms per mini-batch by orders of magnitude at little additional cost. This is only possible because of interpolating in the embedding space. We empirically show that our solutions yield significant improvement over state-of-the-art mixup methods on four different benchmarks, despite interpolation being only linear. By analyzing the embedding space, we show that the classes are more tightly clustered and uniformly spread over the embedding space, thereby explaining the improved behavior.
Mixup [65] is a data augmentation method that interpolates between pairs of training examples, thus regularizing a neural network to favor linear behavior in-between examples. Besides improving generalization, it has important properties such as reducing overconfident predictions and increasing the robustness to adversarial examples. Several follow-up works have studied interpolation in the latent or embedding space, which is equivalent to interpolating along a manifold in the input space [51], and a number of nonlinear and attention-based interpolation mechanisms [62, 26, 25, 46, 7]. However, little progress has been made in the augmentation process itself, i.e., the number  $n$  of generated examples and the number  $m$  of examples being interpolated.
Mixup was originally motivated as a way to go beyond empirical risk minimization (ERM) [47] through a vicinal distribution expressed as an expectation over an interpolation factor  $\lambda$ , which is equivalent to the set of linear segments between all pairs of training inputs and targets. In practice however, in every training iteration, a single scalar  $\lambda$  is drawn and the number of interpolated pairs is limited to the size  $b$  of the mini-batch  $(n = b)$ , as illustrated in Figure 1(a). This is because if
METHOD	SPACE	TERMS	MIXED	FACT	DISTR
Mixup [65]	input	b	2	1	Beta
Manifold mixup [51]	embedding	b	2	1	Beta
ζ-Mixup [1]	input	b	25	1	RandPerm
SuperMix [11]	input	b	3	1	Dirichlet
MultiMix (ours)	embedding	n	b	n	Dirichlet
Dense MultiMix (ours)	embedding	nr	b	nr	Dirichlet
Table 1: Interpolation method properties. SPACE: Space where interpolation takes place; TERMS: number of loss terms per mini-batch; MIXED: maximum number  $m$  of examples being interpolated; FACT: number of interpolation factors  $\lambda$  per mini-batch; DISTR: distribution used to sample interpolation factors; RandPerm: random permutations of a fixed discrete probability distribution.  $b$  : mini-batch size;  $n$  : number of generated examples per mini-batch;  $r$  : spatial resolution.
interpolation takes place in the input space, it would be expensive to increase the number of pairs per iteration. To our knowledge, these limitations exist in all mixup methods.
In this work, we argue that a data augmentation process should increase the data seen by the model, or at least by its last few layers, as much as possible. In this sense, we follow manifold mixup [51] and generalize it in a number of ways to introduce MultiMix.
First, we increase the number  $n$  of generated examples beyond the mini-batch size  $b$ , by orders of magnitude  $(n \gg b)$ . This is possible by interpolating at the deepest layer, i.e., just before the classifier, which happens to be the most effective choice. To our knowledge, we are the first to investigate  $n > b$ .
(a) mixup
(b) MultiMix (ours)
Figure 1: Data augmentation in a mini-batch  $B$  of  $b = 10$  points in two dimensions. (a) mixup: sampling  $n = b$  points on linear segments between  $b$  pairs of points using the same interpolation factor  $\lambda$ . (b) MultiMix: sampling  $n = 300$  points in the convex hull of  $B$ .
Second, we increase the number  $m$  of examples being interpolated from  $m = 2$  (pairs) to  $m = b$  (a single tuple containing the entire mini-batch). Effectively, instead of linear segments between pairs of examples in the mini-batch, we sample on their entire convex hull as illustrated in Figure 1(b). This idea has been investigated in the input space: the original mixup method [65] found it non-effective, while [11] found it effective only up to  $m = 3$  examples and [1] went up to  $m = 25$  but with very sparse interpolation factors. To our knowledge, we are the first to investigate  $m > 2$  in the embedding space and to show that it is effective up to  $m = b$ .
Third, instead of using a single scalar value of  $\lambda$  per mini-batch, we draw a different vector  $\lambda \in \mathbb{R}^m$  for each interpolated example. A single  $\lambda$  works for standard mixup because the main source of randomness is the choice of pairs (or small tuples) out of  $b$  examples. In our case, because we use a single tuple of size  $m = b$ , the only source of randomness being  $\lambda$ .
We also argue that, what matters more than the number of (interpolated) examples is the total number of loss terms per mini-batch. A common way to increase the number of loss terms per example is by dense operations when working on sequence data, e.g. patches in images or voxels in video. This is common in dense tasks like segmentation [41] and less common in classification [27]. We are the first to investigate this idea in mixup, introducing Dense MultiMix.
In particular, this is an extension of MultiMix where we work with feature tensors of spatial resolution  $r$  and densely interpolate features and targets at each spatial location, generating  $r$  interpolated features per example and  $nr > n$  per mini-batch. We also apply the loss densely. This increases the number of loss terms further by a factor  $r$ , typically one or two orders of magnitude, compared with MultiMix. Of course, for this to work, we also need a target label per feature, which we inherit from the corresponding example. This is a weak form of supervision [70]. To carefully select the most representative features per object, we use an attention map representing our confidence in the target label per spatial location. The interpolation vectors  $\lambda$  are then weighted by attention.
Table 1 summarizes the properties of our solutions against existing interpolation methods. Overall, we make the following contributions:
1. We generate an arbitrary large number of interpolated examples beyond the mini-batch size, each by interpolating the entire mini-batch in the embedding space, with one interpolation vector per example. (subsection 3.2).
2. We extend to attention-weighted dense interpolation in the embedding space, further increasing the number of loss terms per example (subsection 3.3).
3. We improve over state-of-the-art (SoTA) mixup methods on image classification, robustness to adversarial attacks, object detection and out-of-distribution detection. Our solutions have little or no additional cost while interpolation is only linear (section 4).
4. Analysis of the embedding space shows that our solutions yield classes that are tightly clustered and uniformly spread over the embedding space (section 4).
Mixup: interpolation methods In general, mixup interpolates between pairs of input examples [65] or embeddings [51] and their corresponding target labels. Several follow-up methods mix input images according to spatial position, either at random rectangles [62] or based on attention [46, 26, 25], in an attempt to focus on a different object in each image. Other follow-up method [35] randomly cuts image at patch level and obtains its corresponding mixed label using content-based attention. We also use attention in our dense MultiMix variant, but in the embedding space. Other definitions of interpolation include the combination of content and style from two images [23], generating out-of-manifold samples using adaptive masks [37], generating binary masks by thresholding random low-frequency images [18] and spatial alignment of dense features [48]. Our dense MultiMix also uses dense features but without aligning them and can interpolate a large number of samples, thereby increasing the number of loss-terms per example. Our work is orthogonal to these methods as we focus on the sampling process of augmentation rather than on the definition of interpolation. We refer the reader to [30] for a comprehensive study of mixup methods.
Mixup: number of examples to interpolate To the best of our knowledge, the only methods that interpolate more than two examples for image classification are OptTransMix [71], SuperMix [11],  $\zeta$ -Mixup [1] and DAML [45]. All these methods operate in the input space and limit the number of generated examples to the mini-batch size; whereas MultiMix generates an arbitrary number of interpolated examples (more than 1000 in practice) in the embedding space. To determine the interpolation weights, OptTransMix uses a complex optimization process and only applies to images with clean background;  $\zeta$ -Mixup uses random permutations of a fixed vector; SuperMix uses a Dirichlet distribution over not more than 3 examples in practice; and DAML uses a Dirichlet distribution to interpolate across different domains. We also use a Dirichlet distribution but over as many examples as the mini-batch size. Similar to MultiMix, [6] uses different  $\lambda$  within a single batch. Beyond classification,  $m$ -Mix [66] uses graph neural networks in a self-supervised setting with pair-based loss functions. The interpolation weights are deterministic and based on pairwise similarities.
Number of loss terms Increasing the number of loss terms more than  $b$  per mini-batch is not very common in classification. In deep metric learning [40, 28, 53], it is common to have a loss term for each pair of examples in a mini-batch in the embedding space, giving rise to  $b(b - 1) / 2$  loss terms per mini-batch, even without mixup [49]. Contrastive loss functions are also common in self-supervised learning [8, 5] but have been used for supervised image classification too [24]. Ensemble methods [55, 14, 19, 3] increase the number of loss terms, but with the proportional increase of the cost by operating in the input space. To our knowledge, MultiMix is the first method to increase the number of loss terms per mini-batch to  $n \gg b$  for mixup. Dense MultiMix further increases this number to  $nr$ . By operating in the embedding space, their computational overhead is minimal.
Dense loss functions Although standard in dense tasks like semantic segmentation [41, 20], where dense targets commonly exist, dense loss functions are less common otherwise. Few examples are in weakly-supervised segmentation [70, 2], few-shot learning [33, 31], where data augmentation is of utter importance, attribution methods [27] and unsupervised representation learning, e.g. dense contrastive learning [42, 54], learning from spatial correspondences [59, 57] and masked language or image modeling [12, 58, 32, 69]. To our knowledge, we are the first to use dense interpolation and a dense loss function for mixup.
3.1 Preliminaries and background
Problem formulation Let  $x \in \mathcal{X}$  be an input example and  $y \in \mathcal{Y}$  its one-hot encoded target, where  $\mathcal{X} = \mathbb{R}^D$  is the input space,  $\mathcal{Y} = \{0,1\}^c$  and  $c$  is the total number of classes. Let  $f_{\theta}: \mathcal{X} \to \mathbb{R}^d$  be an encoder that maps the input  $x$  to an embedding  $z = f_{\theta}(x)$ , where  $d$  is the dimension of the embedding. A classifier  $g_W: \mathbb{R}^d \to \Delta^{c-1}$  maps  $z$  to a vector  $p = g_W(z)$  of predicted probabilities over classes, where  $\Delta^n \subset \mathbb{R}^{n+1}$  is the unit  $n$ -simplex, i.e.,  $p \geq 0$  and  $\mathbf{1}_c^\top p = 1$ , and  $\mathbf{1}_c \in \mathbb{R}^c$  is an all-ones vector. The overall network mapping is  $f := g_W \circ f_{\theta}$ . Parameters  $(\theta, W)$  are learned by optimizing over mini-batches.
Given a mini-batch of  $b$  examples, let  $X = (x_{1},\ldots ,x_{b})\in \mathbb{R}^{D\times b}$  be the inputs,  $Y = (y_{1},\dots ,y_{b})\in \mathbb{R}^{c\times b}$  the targets and  $P = (p_{1},\dots ,p_{b})\in \mathbb{R}^{c\times b}$  the predicted probabilities of the mini-batch, where  $P = f(X)\coloneqq (f(x_1),\dots ,f(x_b))$ . The objective is to minimize the cross-entropy
$$
H (Y, P) := - \mathbf {1} _ {c} ^ {\top} (Y \odot \log (P)) \mathbf {1} _ {b} / b \tag {1}
$$
of predicted probabilities  $P$  relative to targets  $Y$  averaged over the mini-batch, where  $\odot$  is the Hadamard (element-wise) product. In summary, the mini-batch loss is
$$
L (X, Y; \theta , W) := H (Y, g _ {W} (f _ {\theta} (X))). \tag {2}
$$
The total number of loss terms per mini-batch is  $b$ .
Mixup Mixup methods commonly interpolate pairs of inputs or embeddings and the corresponding targets at the mini-batch level while training. Given a mini-batch of  $b$  examples with inputs  $X$  and targets  $Y$ , let  $Z = (z_{1},\ldots ,z_{b})\in \mathbb{R}^{d\times b}$  be the embeddings of the mini-batch, where  $Z = f_{\theta}(X)$ . Manifold mixup [51] interpolates the embeddings and targets by forming a convex combination of the pairs with interpolation factor  $\lambda \in [0,1]$ :
$$
\widetilde {Z} = Z (\lambda I + (1 - \lambda) \Pi) \tag {3}
$$
$$
\widetilde {Y} = Y (\lambda I + (1 - \lambda) \Pi), \tag {4}
$$
where  $\lambda \sim \mathrm{Beta}(\alpha, \alpha)$ ,  $I$  is the identity matrix and  $\Pi \in \mathbb{R}^{b \times b}$  is a permutation matrix. Input mixup [65] interpolates inputs rather than embeddings:
$$
\widetilde {X} = X (\lambda I + (1 - \lambda) \Pi). \tag {5}
$$
Whatever the interpolation method and the space where it is performed, the interpolated data, e.g.  $\widetilde{X}$  [65] or  $\widetilde{Z}$  [51], replaces the original mini-batch data and gives rise to predicted probabilities  $\widetilde{P} = (p_1,\dots ,p_b)\in \mathbb{R}^{c\times b}$  over classes, e.g.  $\widetilde{P} = f(\widetilde{X})$  [65] or  $\widetilde{P} = g_W(\widetilde{Z})$  [51]. Then, the average cross-entropy  $H(\widetilde{Y},\widetilde{P})$  (1) between the predicted probabilities  $\widetilde{P}$  and interpolated targets  $\widetilde{Y}$  is minimized. The number of generated examples per mini-batch is  $n = b$ , same as the original mini-batch size, and each is obtained by interpolating  $m = 2$  examples. The total number of loss terms per mini-batch is again  $b$ .
3.2 MultiMix
Interpolation The number of generated examples per mini-batch is now  $n \gg b$  and the number of examples being interpolated is  $m = b$ . Given a mini-batch of  $b$  examples with embeddings  $Z$  and targets  $Y$ , we draw interpolation vectors  $\lambda_{k} \sim \mathrm{Dir}(\alpha)$  for  $k = 1, \dots, n$ , where  $\mathrm{Dir}(\alpha)$  is the symmetric Dirichlet distribution and  $\lambda_{k} \in \Delta^{m-1}$ , that is,  $\lambda_{k} \geq 0$  and  $\mathbf{1}_{m}^{\top} \lambda_{k} = 1$ . We then interpolate embeddings and targets by taking  $n$  convex combinations over all  $m$  examples:
$$
\tilde {Z} = Z \Lambda \tag {6}
$$
$$
\widetilde {Y} = Y \Lambda , \tag {7}
$$
where  $\Lambda = (\lambda_1, \dots, \lambda_n) \in \mathbb{R}^{b \times n}$ . We thus generalize manifold mixup [51]:
1. from  $b$  to an arbitrary number  $n \gg b$  of generated examples: interpolated embeddings  $\widetilde{Z} \in \mathbb{R}^{d \times n}$  (6) vs.  $\mathbb{R}^{d \times b}$  in (3), targets  $\widetilde{Y} \in \mathbb{R}^{c \times n}$  (7) vs.  $\mathbb{R}^{c \times b}$  in (4);
Figure 2: Dense MultiMix (subsection 3.3) for the special case  $m = 2$  (two examples),  $n = 1$  (one interpolated embedding),  $r = 9$  (spatial resolution  $3 \times 3$ ). The embeddings  $\mathbf{z}_1, \mathbf{z}_2 \in \mathbb{R}^{d \times 9}$  of input images  $x_1, x_2$  are extracted by encoder  $f_\theta$ . Attention maps  $a_1, a_2 \in \mathbb{R}^9$  are extracted (9), multiplied element-wise with interpolation vectors  $\lambda, (1 - \lambda) \in \mathbb{R}^9$  (10) and  $\ell_1$ -normalized per spatial position (11). The resulting weights are used to form the interpolated embedding  $\tilde{\mathbf{z}} \in \mathbb{R}^{d \times 9}$  as a convex combination of  $\mathbf{z}_1, \mathbf{z}_2$  per spatial position (12). Targets are interpolated similarly (13).
2. from pairs  $(m = 2)$  to a tuple of length  $m = b$ , containing the entire mini-batch:  $m$ -term convex combination (6),(7) vs. 2-term in (3),(4), Dirichlet vs. Beta distribution;
3. from fixed  $\lambda$  across the mini-batch to a different  $\lambda_{k}$  for each generated example.
Loss Again, we replace the original mini-batch embeddings  $Z$  by the interpolated embeddings  $\widetilde{Z}$  and minimize the average cross-entropy  $H(\widetilde{Y},\widetilde{P})$  (1) between the predicted probabilities  $\widetilde{P} = g_W(\widetilde{Z})$  and the interpolated targets  $\widetilde{Y}$  (7). Compared with (2), the mini-batch loss becomes
$$
L _ {M} (X, Y; \theta , W) := H (Y \Lambda , g _ {W} (f _ {\theta} (X) \Lambda)). \tag {8}
$$
The total number of loss terms per mini-batch is now  $n \gg b$ .
3.3 Dense MultiMix
We now extend to the case where the embeddings are structured, e.g. in tensors. This happens e.g. with token vs. sentence embeddings in NLP and patch vs. image embeddings in vision. It works by removing spatial pooling and applying the loss function densely over all tokens/patches. The idea is illustrated in Figure 2. For the sake of exposition, our formulation uses sets of matrices grouped either by example or by spatial position. In practice, all operations are on tensors.
Preliminaries The encoder is now  $f_{\theta} : \mathcal{X} \to \mathbb{R}^{d \times r}$ , mapping the input  $x$  to an embedding  $\mathbf{z} = f_{\theta}(x) \in \mathbb{R}^{d \times r}$ , where  $d$  is the number of channels and  $r$  is its spatial resolution—if there are more than one spatial dimensions, these are flattened.
Given a mini-batch of  $b$  examples, we have again inputs  $X = (x_{1},\ldots ,x_{b})\in \mathbb{R}^{D\times b}$  and targets  $Y = (y_{1},\dots ,y_{b})\in \mathbb{R}^{c\times b}$ . Each embedding  $\mathbf{z}_i = f_\theta (x_i) = (z_i^1,\dots,z_i^r)\in \mathbb{R}^{d\times r}$  for  $i = 1,\dots ,b$  consists of features  $z_{i}^{j}\in \mathbb{R}^{d}$  for spatial position  $j = 1,\dots ,r$ . We group features by position in matrices  $Z^{1},\ldots ,Z^{r}$ , where  $Z^{j} = (z_{1}^{j},\dots,z_{b}^{j})\in \mathbb{R}^{d\times b}$  for  $j = 1,\dots ,r$ .
Attention In the absence of dense targets, each spatial location inherits the target of the corresponding input example. This is weak supervision, because the target object is not visible everywhere. To select the most reliable locations, we define a level of confidence according to an attention map. Given an embedding  $\mathbf{z} \in \mathbb{R}^{d \times r}$  with target  $y \in \mathcal{Y}$  and a vector  $u \in \mathbb{R}^d$ , the attention map
$$
a = h \left(\mathbf {z} ^ {\top} u\right) \in \mathbb {R} ^ {r} \tag {9}
$$
measures the similarity of features of  $\mathbf{z}$  to  $u$ , where  $h$  is a non-linearity, e.g. softmax or ReLU followed by  $\ell_1$  normalization. There are different ways to define vector  $u$ . For example,  $u = \mathbf{z} \mathbf{1}_r / r$  by global average pooling (GAP) of  $\mathbf{z}$ , or  $u = W y$  assuming a linear classifier with  $W \in \mathbb{R}^{d \times c}$ , similar to class activation mapping (CAM) [68]. In case of no attention,  $a = \mathbf{1}_r / r$  is uniform.
Given a mini-batch, let  $a_{i} = (a_{i}^{1},\ldots ,a_{i}^{r})\in \mathbb{R}^{r}$  be the attention map of embedding  $\mathbf{z}_i$  (9) for  $i = 1,\dots ,b$ . We group attention by position in vectors  $a^1,\ldots ,a^r$ , where  $a^j = (a_1^j,\dots ,a_b^j)\in \mathbb{R}^b$
for  $j = 1, \dots, r$ . Figure 6 in the supplementary shows the attention obtained by (9). We observe high confidence on the entire or part of the object. Where confidence is low, we assume the object is not visible and thus the corresponding interpolation factor should be low.
Interpolation There are again  $n \gg b$  generated examples per mini-batch, with  $m = b$  examples being densely interpolated. For each spatial position  $j = 1, \ldots, r$ , we draw interpolation vectors  $\lambda_k^j \sim \mathrm{Dir}(\alpha)$  for  $k = 1, \ldots, n$  and define  $\Lambda^j = (\lambda_1^j, \dots, \lambda_n^j) \in \mathbb{R}^{m \times n}$ . Since input examples are assumed to contribute according to the attention vector  $a^j \in \mathbb{R}^m$ , we scale the rows of  $\Lambda^j$  accordingly and normalize its columns back to  $\Delta^{m-1}$  to define convex combinations:
$$
M ^ {j} = \operatorname {d i a g} \left(a ^ {j}\right) \Lambda^ {j} \tag {10}
$$
$$
\hat {M} ^ {j} = M ^ {j} \operatorname {d i a g} \left(\mathbf {1} _ {m} ^ {\top} M ^ {j}\right) ^ {- 1} \tag {11}
$$
We then interpolate embeddings and targets by taking  $n$  convex combinations over  $m$  examples:
$$
\tilde {Z} ^ {j} = Z ^ {j} \hat {M} ^ {j} \tag {12}
$$
$$
\widetilde {Y} ^ {j} = Y \hat {M} ^ {j}. \tag {13}
$$
This is similar to (6),(7), but there is a different interpolated embedding matrix  $\widetilde{Z}^j\in \mathbb{R}^{d\times n}$  as well as target matrix  $\widetilde{Y}^j\in \mathbb{R}^{c\times n}$  per position, even though the original target matrix  $Y$  is one. The total number of interpolated features and targets per mini-batch is now  $nr$ .
Classifier The classifier is now  $g_W: \mathbb{R}^{d \times r} \to \mathbb{R}^{c \times r}$ , maintaining the same spatial resolution as the embedding and generating one vector of predicted probabilities per spatial position. This is done by removing average pooling or any down-sampling operation. The interpolated embeddings  $\widetilde{Z}^1, \ldots, \widetilde{Z}^r$  (12) are grouped by example into  $\widetilde{\mathbf{z}}_1, \ldots, \widetilde{\mathbf{z}}_n \in \mathbb{R}^{d \times r}$ , mapped by  $g_W$  to predicted probabilities  $\widetilde{\mathbf{p}}_1, \ldots, \widetilde{\mathbf{p}}_n \in \mathbb{R}^{c \times r}$  and grouped again by position into  $\widetilde{P}^1, \ldots, \widetilde{P}^r \in \mathbb{R}^{c \times n}$ .
In the simple case where the original classifier is linear, i.e.  $W \in \mathbb{R}^{d \times c}$ , it is seen as  $1 \times 1$  convolution and applied densely to each column (feature) of  $\widetilde{Z}^j$  for  $j = 1, \ldots, r$ .
Loss Finally, we learn parameters  $\theta, W$  by minimizing the weighted cross-entropy  $H(\widetilde{Y}^j, \widetilde{P}^j; s)$  of  $\widetilde{P}^j$  relative to the interpolated targets  $\widetilde{Y}^j$  again densely at each position  $j$ , where
$$
H (Y, P; s) := - \mathbf {1} _ {c} ^ {\top} (Y \odot \log (P)) s / \left(\mathbf {1} _ {n} ^ {\top} s\right) \tag {14}
$$