SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zicheng Liu $^{1,2,*}$
Siyuan  $\mathbf{L}\mathbf{i}^{1,2,*}$
Ge Wang1,2
Chen Tan1,2
Lirong Wu<sup>1,2</sup>
Stan Z. Li $^{2, \dagger}$
AI Lab, Research Center for Industries of the Future, Hangzhou, China;
$^{1}$ Zhejiang University;  $^{2}$ Westlake University;
{liuzicheng; lisiyuan; wangge; tanches; lirongwu; stan.zq.li}
@westlake.edu.cn
Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. Recently, dynamic mixup methods have improved previous static policies effectively (e.g., linear interpolation) by maximizing target-related salient regions in mixed samples, but excessive additional time costs are not acceptable. These additional computational overheads mainly come from optimizing the mixed samples according to the mixed labels. However, we found that the extra optimizing step may be redundant because label-mismatched mixed samples are informative hard mixed samples for deep models to localize discriminative features. In this paper, we thus are not trying to propose a more complicated dynamic mixup policy but rather an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM). The primary effect is that DM can adaptively utilize those hard mixed samples to mine discriminative features without losing the original smoothness of mixup. As a result, DM enables static mixup methods to achieve comparable or even exceed the performance of dynamic methods without any extra computation. This also leads to an interesting objective design problem for mixup training that we need to focus on both smoothing the decision boundaries and identifying discriminative features. Extensive experiments on supervised and semi-supervised learning benchmarks across seven datasets validate the effectiveness of DM as a plug-and-play module. Source code and models are available at https://github.com/Westlake-AI/openmixup.
Deep Learning has become the bedrock of modern AI for many tasks in machine learning [3] such as computer vision [19, 18], natural language processing [12]. Using a large number of learnable parameters, deep neural networks (DNNs) can recognize subtle dependencies in large training datasets to be later leveraged to perform accurate predictions on unseen data. However, models might overfit the training set without constraints or enough data [53]. To this
CAM of Squirrel
CAM of Panda
mnnnne
Figure 1: visualization of hard mixed sample mining by class activation mapping (CAM) [49] of ResNet-50 on ImageNet. From left to right, CAM of top-2 predicted classes using mixup cross-entropy (MCE) and decoupled mixup (DM) loss.
Mixed Cross-entropy
Decoupled Mixup Loss
*Equal contribution. †Stan Z. Li (Stan.ZQ.Li@westlake.edu.cn) is the corresponding author.
(1.0 Panda)
$x_{a}$
(1.0 Squirrel)
x
Figure 2: Illustration of the two types of hard mixed samples in CutMix with 'Squirrel' and 'Panda' as an example. Hard mixed samples indicate that the mixed sample contains salient features of a class, but the value of the corresponding label is small. MCE loss fails to leverage these samples.
(0.3 Squirrel 0.7 Panda)
Hard Mixed Sample for Squirrel
(0.7 Squirrel 0.3 Panda)
Hard Mixed Sample for Panda
end, regularization techniques have been deployed to improve generalization [61], which can be categorized into data-independent or data-dependent ones [16]. Some data-independent strategies, for example, constrain the model by punishing the parameters' norms, such as weight decay [40]. Among data-dependent strategies, data augmentations [51] are widely used. The augmentation policies often rely on particular domain knowledge [58] in different fields.
Mixup [77], a data-dependent augmentation technique, is proposed to generate virtual samples by a linear combination of data pairs and the corresponding labels with the mixing ratio  $\lambda \in [0,1]$ . Recently, a line of estimizable mixup methods are proposed to improve mixing policies to generate object-aware virtual samples by optimizing discriminative regions in the data space to match the corresponding labels [56, 23, 22] (referred to as dynamic methods). However, although the dynamic approach brings some performance gain, the extra computational overhead degrades the efficiency of mixup augmentation significantly. Specifically, the most computation of dynamic methods is spent on optimizing label-mismatched samples, but the question of why these label-mismatched samples should be avoided during the mixup training has rarely been analyzed. In this paper, we find these mismatched samples are completely underutilized by static mixup methods, and the problem lies in the loss function, mixed cross-entropy loss (MCE). Therefore, we argue that these mismatched samples are not only not static mixup disadvantages but also hard mixed samples full of discriminative information. Taking CutMix [74] as an example, two types of hard mixed samples are shown on the right of Figure 2. Since MCE loss forces the model's predictions to be consistent with the soft label distribution, i.e., the model cannot give high-confidence predictions for the relevant classes even if the feature is salient in hard mixed samples, we can say that these hard samples are not fully leveraged.
From this perspective, we expect the model to be able to mine these hard samples, i.e., to give confident predictions according to salient features for localizing discriminative characteristics, even if the proportion of features is small. Motivated by this finding, we introduce simple yet effective Decoupled Mixup (DM) loss, a mixup objective function for explicitly leveraging the hard samples during the mixup training. Based on the standard mixed cross-entropy (MCE) loss, an extra decoupled regularizer term is introduced to enhance the ability to mine underlying discriminative statistics in the mixed sample by independently computing the predicted probabilities of each mixed class. Figure 1 shows the proposed DM loss can empower the static mixup methods to explore more discriminative features. Extensive experiments demonstrate that DM achieves data-efficiency training on supervised and semi-supervised learning benchmarks. Our contributions are summarized below:
- Unlike those dynamic mixup policies that design complicated mixing policies, we propose DM, a mixup objective function of mining discriminative features adaptively.
- Our work contributes more broadly to understanding mixup training: it is essential to focus not only on the smoothness by regression of the mixed labels but also on discrimination by encouraging the model to give reliable and confident predictions.
- Not only in supervised learning but the proposed DM can also be easily generalized to semi-supervised learning with a minor modification. By leveraging the unlabeled data, it can reduce the conformation bias and significantly improve performance.
- Comprehensive experiments on various tasks verify the effectiveness of DM, e.g., DM-based static mixup policies achieve a comparable or even better performance than dynamic methods without the extra computation.
Mixup Augmentation. As data-dependent augmentation techniques, mixup methods generate new samples by mixing samples and corresponding labels with well-designed mixing policies [77, 57, 69, 64]. The pioneering mixing method is Mixup [77], whose mixed samples are generated by linear interpolation between pairs of samples. ManifoldMix variants [57, 14] extend Mixup to the latent space of DNNs. After that, cut-based methods [74] are proposed to improve the mixup for localizing important features, especially in the vision field. Many researchers explore using nonlinear or estimizable sample mixup policies to generate more reliable mixed samples according to mixed labels, such as PuzzleMix variants [23, 22, 45], SaliencyMix variants [56, 60], AutoMix variants [38, 31], and SuperMix [11]. Concurrently, recent works try to generate more accurate mixed labels with saliency information [20] or attention maps [5, 9, 7] for Transformer architectures, which require prior pre-trained knowledge or attention information. On the contrary, the proposed decoupled mixup is a pluggable learning objective for mixup augmentations. Moreover, mixup methods extend to more than two elements [22, 11] and regression tasks [70]. Some researchers also utilize mixup augmentations to enhance contrastive learning [8, 21, 28, 50, 31] or masked image modeling [33, 6] to learn general representation in a self-supervised manner.
Semi-supervised Learning and Transfer Learning. Pseudo-Labeling [27] is a popular semi-supervised learning (SSL) method that utilizes artificial labels converted from teacher model predictions. MixMatch [2] and ReMixMatch [1] apply mixup on labeled and unlabeled data to enhance the diversity of the dataset. More accurate pseudo-labeling relies on data augmentation techniques to introduce consistency regularization, e.g., UDA [65] and FixMatch [52] employ weak and strong augmentations to improve the consistency. Furthermore, CoMatch [29] unifies consistency regularization, entropy minimization, and graph-based contrastive learning to mitigate confirmation bias. Recently proposed works [62, 4] that improve FixMatch by designing more accurate confidence-based pseudo-label selection strategies, e.g., FlexMatch [76] applying curriculum learning for updating confidence threshold dynamically and class-wisely. More recently, SemiReward [30] proposes a reward model to filter out accurate pseudo labels with reward scores. Fine-tuning a pre-trained model on labeled datasets is a widely adopted form of transfer learning (TL) in various applications. Previously, [13, 44] show that transferring pre-trained AlexNet features to downstream tasks outperforms hand-crafted features. Recent works mainly focus on better exploiting the discriminative knowledge of pre-trained models from different perspectives. L2-SP [35] promotes the similarity of the final solution with pre-trained weights by a simple L2 penalty. DELTA [34] constrains the model by a subset of pre-trained feature maps selected by channel-wise attention. BSS [68] avoids negative transfer by penalizing smaller singular values. More recently, Self-Tuning variants [67, 54] combined contrastive learning with TL to tackle confirmation bias and model shift issues in a one-stage framework.
3 Decoupled Mixup
Mixed Cross-Entropy Underutilizes Mixup Let us define  $y \in \mathbb{R}^C$  as the ground-truth label with  $C$  categories. For labeled data point  $x \in \mathbb{R}^{W \times H \times C}$  whose embedded representation  $z$  is obtained from the model  $M$  and the predicted probability  $p$  can be calculated through a Softmax function  $p = \sigma(z)$ . Given the mixing ratio  $\lambda \in [0,1]$  and  $\lambda$ -related mixup mask  $H \in \mathbb{R}^{W \times H}$ , the mixed sample  $(x_{(a,b)}, y_{(a,b)})$  can be generated as  $x_{(a,b)} = H \odot x_a + (1 - H) \odot x_b$ , and  $y_{(a,b)} = \lambda y_a + (1 - \lambda) y_b$ , where  $\odot$  denotes element-wise product,  $(x_a, y_a)$  and  $(x_b, y_b)$  are sampled from a labeled dataset  $L = \{(x_a, y_a)\}_{a=1}^{n_L}$ . Note that superscripts denote the index; subscripts indicate the type of data, e.g.,  $x_{(a,b)}$  represents a mixed sample generated from  $x_a$  and  $x_b$ ;  $y^i$  indicates the label value on  $i$ -th position. Since the mixup labels are obtained by somehow  $\lambda$ -based interpolation, the standard CE loss weighted by  $\lambda$ ,  $\mathcal{L}_{CE} = y_{(a,b)}^T \log \sigma(z_{(a,b)})$ , is typically used as the objective in the mixup training:
$$
\mathcal {L} _ {M C E} = - \sum_ {i = 1} ^ {C} \left(\lambda \mathbb {I} \left(y _ {a} ^ {i} = 1\right) \log p _ {(a, b)} ^ {i} + (1 - \lambda) \mathbb {I} \left(y _ {b} ^ {i} = 1\right) \log p _ {(a, b)} ^ {i}\right). \tag {1}
$$
where  $\mathbb{I}(\cdot) \in \{0,1\}$  is an indicator function that values one if and only if the input condition holds. Noticeably, these two items of Equation 1 are classifying  $y_{a}$  and  $y_{b}$  while keeping the linear
consistency with mixing coefficient  $\lambda$ . As a result, DNNs with this mixup consistency prefer relatively less confident results in high-entropy behaviour [46] and longer training time in practice. The main reason is that in addition to  $\lambda$  constraint, the competing relationships defined by Softmax in  $\mathcal{L}_{MCE}$  are the main cause of the confidence drop, which is more obvious when dealing with hard mixed samples. Precisely, the competition between the mixed class  $a$  and  $b$  in Equation 1 can severely affect the prediction of a single class; that is, interference from other classes prevents the model from focusing its attention. This typically causes the model to be insensitive to the salient features of the target and thus undermines model performance, as shown in Figure 1. Although the dynamic mixup alleviates this problem, the extra time overhead is unavoidable if only focusing on mixing policies on the data level. Therefore, the key challenge is to design an ideal objective function for mixup training that maintains the smoothness of the mixup and can simultaneously explore the discriminative features without any computation costs.
3.2 Decoupled Regularizer
To achieve the above goal, we first dive into the  $\mathcal{L}_{MCE}$  and propose the efficient decoupled mixup.
Proposition 1. Assuming  $x_{(a,b)}$  is generated from two different classes, minimizing  $\mathcal{L}_{MCE}$  is equivalent to regress corresponding  $\lambda$  in the gradient:
$$
\left(\nabla_ {z _ {(a, b)}} \mathcal {L} _ {M C E}\right) ^ {i} = \left\{ \begin{array}{l l} - \lambda + \frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {c} \exp \left(z _ {(a , b)} ^ {c}\right)}, & i = a \\ - (1 - \lambda) + \frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {c} \exp \left(z _ {(a , b)} ^ {c}\right)}, & i = b \\ \frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {c} \exp \left(z _ {(a , b)} ^ {c}\right)}, & i \neq a, b \end{array} \right. \tag {2}
$$
Softmax Degrades Confidence. As we can see from Proposition 1, the predicted probability of  $x_{(a,b)}$  will be consistent with  $\lambda$ , and the probability is computed from the Softmax directly. The Softmax forces the sum of predictions to one (winner takes all), which is undesirable in mixup classification, especially when there are multiple and non-salient targets in mixed samples, e.g., hard mixed samples, as shown in Figure 2. The standard Softmax in  $\mathcal{L}_{MCE}$  deliberately suppresses confidence and produces high-entropy predictions by coupling all classes. As a consequence,  $\mathcal{L}_{MCE}$  makes many static mixup methods require longer epochs than vanilla training to achieve the desired results [57, 73]. Based on previous analysis, a novel mixup objective, decoupled mixup (DM), is raised to remove the Coupler and thus utilize the hard mixed samples adaptively, finally improving the performance of mixup methods. Specifically, for mixed data points  $z_{(a,b)}$  generated from a random pair in labelled dataset  $L$ , an encoded mixed representation  $z_{(a,b)} = f_{\theta}(x_{(a,b)})$  is generated by a feature extractor  $f_{\theta}$ . A mixed categorical probability of  $i$ -th class is attained:
$$
\sigma \left(z _ {(a, b)}\right) ^ {i} = \frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\sum_ {c} \exp \left(z _ {(a , b)} ^ {c}\right)}. \tag {3}
$$
Decoupled Softmax. where  $\sigma(\cdot)$  is standard Softmax. Equation 3 shows how the mixed probabilities are computed for a mixed sample. The competition between  $a$  and  $b$  is the main reason that results in low confidence of the model, i.e., the sum of semantic information of hard mixed samples are larger than "1" defined by Softmax. Therefore, we propose to simply remove the competitor class in Equation 3 to achieve decoupled Softmax. The score on  $i$ -th class is not affected by the  $j$ -th class:
$$
\phi \left(z _ {(a, b)}\right) ^ {i, j} = \frac {\exp \left(z _ {(a , b)} ^ {i}\right)}{\exp \left(z _ {(a , b)} ^ {j}\right) + \sum_ {c \neq j} \exp \left(z _ {(a , b)} ^ {c}\right)}. \tag {4}
$$
where  $\phi(\cdot)$  is the proposed decoupled Softmax. In Equation 4, by removing the competitor, compared with Equation 1, the decoupled Softmax makes all items associated with  $\lambda$  become -1 in gradient, the derivation is given in the A.1. Our Proposition 2 verifies that the expected results are achieved with decoupled Softmax.
Proposition 2. With the decoupled Softmax defined above, decoupled mixup cross-entropy  $\mathcal{L}_{DM}$  can boost the prediction confidence of the interested classes mutually and escape from the  $\lambda$ -constraint:
$$
\mathcal {L} _ {D M} = - \sum_ {i = 1} ^ {c} \sum_ {j = 1} ^ {c} y _ {a} ^ {i} y _ {b} ^ {j} \left(\log \left(\frac {p _ {(a , b)} ^ {i}}{1 - p _ {(a , b)} ^ {j}}\right) + \log \left(\frac {p _ {(a , b)} ^ {j}}{1 - p _ {(a , b)} ^ {i}}\right)\right). \tag {5}
$$
Figure 3: Results illustration of applying decouple mixup. Left: taking MixUp as an example, our proposed decoupled mixup cross-entropy, DM(CE), significantly improves training efficiency by exploring hard mixed sample; Middle: Acc vs. cost on ImageNet-1k; Right: Top-2 acc is calculated when the top-2 predictions equal to  $\{y_a, y_b\}$ .
The Decoupled Mixup. The proofs of Proposition 1 and 2 are given in the Appendix. In practice, the original smoothness of  $\mathcal{L}_{MCE}$  should not be lost, and thus the proposed DM is a regularizer for discriminability. The final form of decoupled mixup can be formulated as follows:
$$
\mathcal {L} _ {D M (C E)} = - \big (\underbrace {y _ {(a , b)} ^ {T} \log (\sigma (z _ {(a , b)}))} _ {\mathcal {L} _ {M C E}} + \eta \underbrace {y _ {[ a , b ]} ^ {T} \log (\phi (z _ {(a , b)})) y _ {[ a , b ]}} _ {\mathcal {L} _ {D M}} \big).
$$
where  $y_{(a,b)}$  indicates the mixed label while  $y_{[a,b]}$  is two-hot label encoding,  $\eta$  is a trade-off factor. Notice that  $\eta$  is robust and can be set according to the character of mixup methods (see Sec. 5.4).
Practical consequences of such simple modification on mixup and the performance:
Make What Should be Certain More Certain. As we expected, mixup training with a decoupling mechanism will be more accurate and confident in handling hard mixed samples with our artificially constructed hard mixed samples by using PuzzleMix. Figure 3 right demonstrates the model trained with decoupled mixup mostly doubled the top-2 accuracy on these mixed samples, which also verifies the information contained in mixed samples is beyond the "1" defined by standard Softmax. More interestingly, this advantage of decoupled mixup, i.e., higher confidence and accuracy, can be further amplified in semi-supervised learning due to the uncertainty of pseudo-labeling.
Enhance the Training Efficiency. It is straightforward to notice that there is no extra computation cost when using DM in vanilla mixup training, and the performance we can achieve is the same or even better than estimizable mixup policies, i.e., PuzzleMix, CoMixup, etc. Figure 3 left and middle show decoupled mixup unveils the power of static mixup for more accurate and faster.
4 Extensions of Decoupled Mixup
With the high-accurate nature of decoupled mixup for mining hard mixed samples, semi-supervised learning is a suitable scenario to propagate the accurate label from labeled space to unlabeled space by using asymmetrical mixup. In addition, we can also generalize the decoupled mechanism into the binary cross-entropy for boosting the multi-classification task.
4.1 Asymmetrical Strategy for Semi-supervised Learning
Based on labeled data  $L = \{(x_{a},y_{a})\}_{a = 1}^{n_{L}}$ , if we further consider unlabeled data  $U = \{(u_{a},v_{a})\}_{a = 1}^{n_{U}}$  decoupled mixup can be the strong connection between  $L$  and  $U$ . Recall the confirmation bias [67] problem of SSL: the performance of the student model is restricted by the teacher model when learning from inaccurate pseudo-labels. To fully use the  $L$  and strengthen the teacher model to provide more robust and accurate predictions, the unlabeled data with large  $\lambda$  can be used to mix with the labeled data to form hard mixed samples. With these hard mixed samples, we can employ decoupled mixup into semi-supervised learning effectively. Since only the label of  $L$  is accurate, we need to make a little asymmetric modification to the decoupled mixup, called Asymmetrical Strategy(AS). Formally, given the labeled and unlabeled datasets  $L$  and  $U$ , AS builds reliable connection by generating hard mixed samples between  $L$  and  $U$  in an asymmetric manner  $(\lambda < 0.5)$ :
$$
\hat {x} _ {(a, b)} = \lambda x _ {a} + (1 - \lambda) u _ {b}; \quad \hat {y} _ {(a, b)} = \lambda y _ {a} + (1 - \lambda) v _ {b}.
$$
Due to the uncertainty of the pseudo-label, only the labeled part is retained in  $\mathcal{L}_{DM}$ :
$$
\hat {\mathcal {L}} _ {D M} = y _ {a} ^ {T} \log (\phi (z _ {(a, b)})) y _ {b},
$$
where  $y_{a}$  and  $y_{b}$  are one-hot labels from  $L$ . AS could be regarded as a special case of DM that only decouples on labeled data. Simply replacing  $\mathcal{L}_{DM}$  with  $\hat{\mathcal{L}}_{DM}$  can leverage the hard samples and alleviate the confirmation bias in semi-supervised learning.
4.2 Decoupled Binary Cross-entropy Loss
Binary Cross-entropy Form of DM. Different from Softmax-based classification, we can also build decoupled mixup in multi-label classification tasks (1- vs-all) by using mixup binary cross-entropy (MBCE) loss [63]  $(\sigma(\cdot)$  denotes Sigmoid rather Softmax in this case). Proposition 2 demonstrates the decoupled CE can mutually enhance the confidence of predictions for the interested classes and be free from  $\lambda$  limitations. Similarly, for MBCE, since it is not inherently bound to mutual interference between classes by Softmax, we have to preserve partial consistency and encourage more confident predictions, and thus propose a decoupled mixup binary cross-entropy loss, DM(BCE).
Figure 4: Rescaled label of different  $\lambda$  value.
To this end, a rescaling function  $r: \lambda, t, \xi \to \lambda'$  is designed to achieve this goal. The mixed label is rescaled by  $r(\cdot): y_{mix} = \lambda_a y_a + \lambda_b y_b$ , where  $\lambda_a$  and  $\lambda_b$  are rescaled. The rescaling function is defined as follows:
$$
r (\lambda , t, \xi) = \left(\frac {\lambda}{\xi}\right) ^ {t}, \quad 0 \leq t, 0 \leq \xi <   1, \tag {6}
$$
where  $\xi$  is the threshold,  $t$  is an index to control the convexity. As shown in Figure 4, Equation 6 has three situations: (a) when  $\xi = 0$ ,  $t = 0$ , the rescaled label is always equal to 1, as two-hot encoding; (b) when  $\xi = 1$ ,  $t = 1$ ,  $r(\cdot)$  is a linear function (vanilla mixup); (c) the rest curves demonstrate  $t$  is the parameter that changes the concavity and  $\xi$  is responsible for truncating.
Empirical Results. In the case of interpolation-based mixup methods (e.g., Mixup, ManifoldMix, etc.) that keep linearity between the mixed label and sample, the decoupled mechanism can be introduced by only adjusting threshold  $t$ . In the case of cutting-based mixing policies (e.g., CutMix, etc.) where the mixed samples and labels have a square relationship (generally a convex function), we can approximate the convexity by adjusting  $\xi$ , which are detailed in Sec. 5.4 and Appendix C.5.
We adopt two types of top-1 classification accuracy (Acc) metrics (the mean of three trials): (i) the median top-1 Acc of the last 10 epochs [52, 38] for supervised image classification tasks with Mixup variants, and (ii) the best top-1 Acc in all checkpoints for SSL tasks. Popular ConvNets and Transformer-based architectures are used as backbone networks: ResNet variants including ResNet [19] (R), Wide-ResNet (WRN) [75], and ResNeXt-32x4d (RX) [66], Vision Transformers including DeiT [55] and Swin Transformer (Swin) [37].
5.1 Image Classification Benchmarks