DAIR: DATA AUGMENTED INVARIANT REGULARIZATION
Year: 2021

Abstract:
While deep learning through empirical risk minimization (ERM) has succeeded at achieving human-level performance at a variety of complex tasks, ERM generalizes poorly to distribution shift. This is partly explained by overfitting to spurious features such as background in images or named entities in natural language. Synthetic data augmentation followed by empirical risk minimization (DA-ERM) is a simple and widely used solution to remedy this problem. In addition, consistency regularization could be applied to further promote model performance to be consistent on the augmented sample and the original one. In this paper, we propose data augmented invariant regularization (DAIR), a simple form of consistency regularization that is applied directly on the loss function rather than intermediate features. Through extensive empirical experiments, we show that DAIR consistently performs well in a variety of settings. We apply DAIR to multiple real-world learning problems, namely robust regression, visual question answering, robust deep neural network training, and neural task-oriented dialog modeling. Our experiments show that DAIR consistently outperforms ERM and DA-ERM with little marginal cost and sets new state-of-the-art results in several benchmarks.

heading: 1 INTRODUCTION
text: Deep neural networks are widely used in various applications ranging from computer vision to language processing. While deep learning has surpassed human-level performance in numerous tasks, neural networks are extremely vulnerable to overfitting to spurious correlations and therefore fail to generalize even under slight perturbations of the test distribution (Arjovsky et al., 2019). This observation motivated the research community to tackle the problem of domain generalization (see (Ribeiro et al., 2020) for a detailed literature review). Recent benchmark datasets, such as Rotated MNIST (Arjovsky et al., 2019), Colored MNIST (Arjovsky et al., 2019), PACS (Li et al., 2017), VLCS (Fang et al., 2013), Office-Home (Venkateswara et al., 2017), Terra Incognita (Beery et al., 2018) and DomainNet (Peng et al., 2019), have shown difficulties for the generalization of deep neural network models under distribution shifts, and have sparked invention of many new algorithmic frameworks to address domain generalization.
A standard approach for improving out-of-distribution performance is to guarantee that learned models are invariant to certain transformations. For example, trained models for computer vision should generally be invariant to rotations, changes in color, or background.
Geometric deep learning bakes such invariances into the neural network architecture. For example, convolutional layers (Lecun et al., 1998) are fundamentally preserving translations. There are other specifically designed networks to maintain invariances: Zaheer et al. (2017) studied the problem of designing models for machine learning tasks defined on sets and characterized the permutation invariant functions. Bloem-Reddy & Teh (2020) obtained generative functional representations of probability distributions that are invariant under the action of a compact group. Finzi et al. (2021) provided an algorithm for solving for the equivariant layers of matrix groups.
Data augmentation promotes invariances in models by curating synthetic examples that exhibit the desired invariances. Tensmeyer & Martinez (2016) showed simple image transformations affect the CNN representations. Mixup (Zhang et al., 2017), CutMix (Yun et al., 2019) and Cutout (DeVries & Taylor, 2017) showed linear combination and random blocking features improves generalization of state-of-the-art neural network architectures. Volpi et al. (2018); Zhou et al. (2020) showed data augmentation with adversarial images could make the label classifier more robust to unknown domain shifts. Cubuk et al. (2018); Lim et al. (2019) introduced a procedure which automatically searches for improved data augmentation policies. Zhou et al. (2020) showed data augmentation with adversarial images could make the label classifier more robust to unknown domain shifts. Nam et al. (2021)
improved domain generalization by reducing the intrinsic style bias of CNNs through training a separate network for randomizing the style of images and generating augmented data during training.
Consistency regularization can be further applied on top of data augmentation to enhance invariance by enforcing similarities on the model. Engstrom et al. (2018); Kannan et al. (2018); Zhang et al. (2019) utilized consistency regularization to train robust neural networks against adversarial attacks. This has been applied to unsupervised learning (Sinha & Dieng, 2021), self-supervised learning (Chen et al., 2020; von Kügelgen et al., 2021), and semi-supervised learning to exploit unlabeled data (Bachman et al., 2014; Laine & Aila, 2016; Sohn et al., 2020; Xie et al., 2020).
Besides the directions mentioned above, researchers have proposed numerous algorithmic solutions to impose invariance and improve domain generalization such as DANN (Ganin et al., 2016), IRM (Ghifary et al., 2015), DRO (Sagawa et al., 2019), MLDG (Li et al., 2018a), CORAL (Sun & Saenko, 2016), MMD (Li et al., 2018b) and CDANN (Li et al., 2018c) and REx (Krueger et al., 2021). The approaches listed above are more complex than simple training mechanisms such as empirical risk minimization (ERM) and hence they cannot be readily applied to involved tasks with non-trivial model architectures. For example, in generative language models imposing a constraint on the intermediate data representations is non-trivial, which is required by CORAL (Sun & Saenko, 2016). Recently, Gulrajani & Lopez-Paz (2020) demonstrated that ERM may even outperform many such complex methods in real-world scenarios, while ERM itself is known to generalizes poorly to distribution shift. For example, in learning neural dialog models, Qian et al. (2021) showed up to 29% performance drop due to the memorization of named entities. Ribeiro et al. (2020) showed that both commercial and state-of-art language models fail on up to 76.4% of the generalization tests.
In this paper, we propose a consistency regularization technique, called data augmented invariant regularization (DAIR). DAIR is applicable when data augmentation results in pairs of data samples expecting consistent performance, it specifically penalizes the inconsistency of loss on augmented samples with respect to the original ones. This is in contrast to many feature consistency regularizers that apply on an intermediate embedding space. As a result, DAIR only requires marginal additional cost on top of data augmentation, and is simple and broadly applicable to a wide host of supervised and unsupervised learning tasks, including generative models. We introduce the DAIR formulation, motivate it, and theoretically prove some of its properties in Section 2. We empirically evaluate DAIR on a variety of problem setups ranging from defense against adversarial attacks to domain generalization in the presence of environment shift in Section 3, where our experimental results show that DAIR is competitive with or even outperforms state-of-the-art algorithms specifically designed for imposing invariance in these problems.

heading: 2 DAIR: DATA AUGMENTED INVARIANT REGULARIZATION
text: For a data sample z = (x, y), let ℓ(z; θ) be its parametric loss function, where θ is the set of model parameters (e.g., network weights). The popular Empirical Risk Minimization (ERM) framework trains the model by minimizing the expected value of the following loss over the training data:
fERM(z; θ) = ℓ(z; θ). (ERM)
We assume that we have access to a (potentially randomized) data augmenter function A(·). Examples for A include (random) rotation, change of background, or change of entity names. Such augmenters aim at capturing the transformations against which we wish to be invariant to. Given a sample z, let z̃ = (x̃, ỹ) = A(z) denote an augmented sample. Previous work has used both original and augmented examples during training, which leads to the following standard objective function, called Data Augmented Empirical Risk Minimization (DA-ERM):
fDA-ERM(z, z̃; θ) = 1
2 ℓ(z; θ) +
1 2 ℓ(z̃; θ). (DA-ERM)
While DA-ERM has been successful in many applications, one natural question is whether we can further improve upon it using the knowledge that the performance on augmented samples should be consistent with the original ones. Consistency regularization further penalizes DA-ERM for any such inconsistency at the feature/loss level: fConsistency,D,λ(z, z̃; θ) = fDA-ERM(z, z̃; θ) + λD(z, z̃; θ), where D(z, z̃; θ) is a proper divergence between the original sample representation and the augmented sample representation, and where the goal of the regularizer applied at some intermediate feature
space is to maintain the performance of the model on z and z̃ consistent. In this paper, we focus on a specific type of such regularization, called data augmented invariant regularization (DAIR):
fDAIR,R,λ(z, z̃; θ) = fDA-ERM(z, z̃; θ) + λD(z, z̃; θ)
= 1
2 ℓ(z; θ) +
1 2 ℓ(z̃; θ) + λR(ℓ(z; θ), ℓ(z̃; θ)), (DAIR)
where the regularization is directly applied to the loss. The idea behind DAIR is to simply promote ℓ(z; θ) ≈ ℓ(z̃; θ), and ignore the features or even the rest of the possible outcomes of y and simply focus on the current sample’s loss. Hence, DAIR is a relatively weak form of consistency regularization only enforcing an original sample and an augmented one to be equally likely under the learned model (assuming loss is a log-likelihood function). This weaker form of consistency is suitable for problems where feature consistency may not be conceptually meaningful. For instance, in language modeling when a pair of sentences differ in their corresponding named entities, it is not clear why we should enforce their embeddings to be similar, however, loss consistency is still meaningful promoting the probability of label given input to be the same on the original and the augmented samples.
We remark that DAIR requires pairing information between original and augmented samples, which may not always be available (e.g., DomainBed (Gulrajani & Lopez-Paz, 2020)). However, we show that this simple approach is still broadly applicable to various real-world problems regardless of model architecture, and is indeed competitive with state-of-the-art methods for imposing invariance. As it turns out, we are particularly interested in a particular form of the DAIR regularizer:
Rsq(ℓ(z; θ), ℓ(z̃; θ)) := (√ ℓ(z; θ)− √ ℓ(z̃; θ) )2 , (SQ Regularizer)
and we call this variant DAIR-SQ. Note that Rsq has the same scale as the loss function ℓ, making it easier to tune λ. Empirically we observe that the optimal λ for all the experiments mentioned later in the paper falls in [0.2, 100], across various tasks (from regression to sequence-to-sequence generative modeling). Further justification on DAIR-SQ will be provided through the rest of this section.
Finally, in most (real-world) applications performance is measured through 0-1 metrics other than the loss function. For example, we are usually concerned with accuracy in image classification while we optimize cross-entropy loss. Let F (z; θ) ∈ {0, 1} denote a 0-1 evaluation performance metric of interest, e.g., accuracy. Given the sample z (or z̃), the model performance is captured by F (z; θ) (or F (z̃; θ)). For any z such that F (z; θ) = 1, we define the corresponding consistency metric as:
CM(z, z̃; θ) = I{F (z̃; θ) = 1 | F (z; θ) = 1}. (Consistency Metric)
Notice that similarly to the original performance metric, which is only used for model evaluation, we use the consistency metric at evaluation time only.

heading: 2.1 WHAT DOES DAIR OFFER BEYOND DA-ERM?
text: To motivate DAIR, we consider a toy example through which we demonstrate that DAIR can fundamentally outperform DAERM, even in the limit of infinite training samples (no overfitting due to finite samples). Consider a linear regression problem where at the training time the input is xtrain = (x, s = y) and the label y, i.e., ztrain = (xtrain, y). Here, x ∼ N (0, σ2x), and y = x+ε, where ε is independent of x and ε ∼ N (0, σ2ε). In this example, the target is explicitly provided as a spurious feature to the learner at the training time. At test time, the spurious feature is absent, i.e., xtest = (x, s = 0).
Clearly, in this toy example, the optimal regressor is w⋆ = (w⋆1 , w ⋆ 2)
⊤ = (1, 0)⊤. However, absent the knowledge of the spurious feature vanilla ERM will learn wERM ≈ (0, 1)⊤, completely overfitting the spurious feature. We assume that the learner has access to a data augmentation module that generates z̃ = A(z; a, σ2n) = (xaug, y), such that xaug = (x, s = ay+n) where n ∼ N (0, σ2n). The augmented data will encourage the learned model to become invariant to the spurious feature. In Figure 1, we perform simulations with a = 0.5, σ2x = 1, σ 2 ε = 0.25, σ 2 n = 0.1 and plot four lines associated with each regressor with the slope of their respective w1. We ignore w2 as the second spurious feature
is absent at test time and hence w2 does not impact test performance. The optimal regressor is shown as the blue line, with a slope of 1. ERM (red line) completely fails due to the overfitting to the spurious feature. DA-ERM (orange line) significantly improves over ERM but still is far from optimal performance. DAIR-SQ (purple line) almost recovers the optimal solution. This is not a coincidence. We prove that DAIR-SQ is optimal for a class of linear regression problems, while DA-ERM does not approach optimal performance even in the limit of infinite samples. In other words, DAIR can lead to better generalizing models beyond simply offering better sample complexity. Proposition 1. Consider the class of linear regression problems described above with a spurious feature (highly correlated with the output). Assume that the learner has access to a data augmentation module that perturbs the spurious feature. Then, for any value of a and σn, DAIR-SQ achieves optimal test error as number of samples grows and λ → ∞. On the other hand, DA-ERM cannot recover optimal performance even in the limit of infinite training data unless σn → ∞.
The proof of Proposition 1 is relegated to Appendix A. One can show that simple data independent regularization methods (e.g. weight decay) cannot help close the gap between the performance of DA-ERM and DAIR (see Proposition 2) in Appendix A. While we only analyzed DAIR-SQ, we believe the content of this proposition extends to other variants of DAIR as well. Note that when σn → ∞, DA-ERM could also recover w⋆. One can interpret that as σn → ∞, the augmenter becomes stronger and forces w2 to vanish. On the other hand, DAIR recovers w⋆ with a much weaker augmenter. This is crucial since in real-world applications, designing strong augmentation schemes requires careful design. We will expand on this in Section 2.2.

heading: 2.2 VARIANTS OF DAIR VS OTHER CONSISTENCY REGULARIZATION TECHNIQUES
text: In this section, we empirically compare ERM, DA-ERM and some variants of consistency regularization, including two DAIR variants on two classification tasks using CNNs. Let q(z; θ) be the output of the model right after the softmax layer. If we treat the loss function as (un-normalized) negative log-likelihood of the output distribution, and let q(z; θ) ∝ e−ℓ(z;θ). In addition to DAIR variants, we consider the regularizer to be any proper divergence between the output distributions q(z; θ) and q(z̃; θ), such as L2 distance or KL divergence, which will promote q(z; θ) ≈ q(z̃; θ). Rotated MNIST (Ghifary et al., 2015) is a dataset where MNIST digits are rotated. We work with two different sets of degrees of rotation for Rotated MNIST. The first one is Weak Rotation where the digits are rotated uniformly at random [0, π6 ) radians. In Strong Rotation the digits are rotated uniformly at random [0, 2π) radians. To evaluate the robustness of the methods, we further add label noise at training time where the label is replaced with a digit chosen from {0,. . . ,9} uniformly at random with a certain probability. No label noise is added at test time. Detailed setup is in Appendix D.
In the first experiment, we use Weak Rotation for data augmentation while at test time we use Strong Rotation. Thus, some test time rotations have not been observed at training time. Figure 2 shows the test performance of all algorithms (averaged over three runs) as a function of λ. As can be seen, ERM (with no data augmentation) does not generalize to rotated test images and performs poorly. DA-ERM offers significant performance improvement over ERM. When λ is very small all variants of consistency regularization are virtually the same as DA-ERM. DAIR-SQ and KL regularizer outperform other regularizers and are the only two variants that offer improvement over DA-ERM as λ increases. As the label noise level becomes larger, DAIR-SQ is more robust than KL and offers the best performance.
Besides DAIR-SQ and KL regularizer, it is noteworthy that the other consistency regularization variants did not offer improvement over DA-ERM and they converged to poor local minima with
10% test accuracy (random) for large λ. We were not able to remedy this by tuning of their step size. See Section 2.3 for further justification of this phenomenon. We also observe that the performance of both DAIR-SQ and KL regularizer achieves a sweet spot for some finite λ, i.e., the performance starts to drop for large values of λ. This is not theoretically expected and can be attributed to the practical issues with solving the consistency regularization problem. We further investigate this phenomenon in Appendix B and provide some explanations.
The setup for the second experiment is the same as the first one, except we also use Strong Rotation in training for augmentation, so there is no distribution shift for DA-ERM. As can be seen in Figure 3, data augmentation achieves very good performance in this case and none of the DAIR regularizers offer any improvement beyond data augmentation. We suspect this to be true in general; if the data augmentation is well-devised and optimized the resulting model could become invariant to the desired transformations at test time. This also agrees with findings of Section 2.1, where observed that with strong augmentation, DA-ERM could potentially result in similar performance as DAIR. Additional experiments on consistency metric can be found in Appendix E.1.
Colored MNIST (Arjovsky et al., 2019) is a binary classification task built on the MNSIT dataset. Digits 0-4 are labeled 1; whereas digits 5-9 are labeled 0. Additionally, 25% label noise is added, i.e., the labels are flipped with probability 0.25, both at train and test time, capping the achievable test accuracy to 75%. In this dataset, each digit is RGB colored. During training, label 1 is given the color green with probability 0.9 and red with probability 0.1. On the other hand, label 0 is given red color with probability 0.9 and green with probability 0.1. This introduces a high degree of spurious correlation between color and the label. Thus, ERM is expected to significantly overfit to color for predicting the label.
At test time, the correlation with color is reversed for digits. Hence, vanilla ERM is expected to perform worse than 50% coin flip at test time. We explore two data augmentation schemes in this experiment. For the Adversarial Augmentation (Adv. Aug.) setup, the augmented images will have their color flipped (from red to green or vice versa) with probability 0.1. For the Random Augmentation (Rnd. Aug.) setup, the augmented images are colored uniformly at random. Detailed description of the setup and additional experiments can be found in Appendix D and Appendix E.1, respectively.
Figure 4 suggests that DAIR-SQ and KL consistency regularization achieve ∼ 72% test accuracy using both augmentation schemes, outperforming the stateof-the-art 68% test accuracy reported by invariant risk minimization (IRM) (Arjovsky et al., 2019), and almost reaching the 75% cap. We note however that this comparison may be unfair because IRM does not have access to any pairing information between the original and the augmented samples. As we observe in the next section, such information is readily available in several real-world benchmarks and DAIR can
exploit it to achieve new state-of-the-art results. We also notice that neither variant of DA-ERM achieves test performance better than 50% coin flip in this experiment, while Adversarial Augmentation seems to fare better than Random Augmentation.
Following the experiments, we conclude that DAIR-SQ is more stable and robust than other ones followed by KL divergence consistency regularization. Additionally, DAIR-SQ enjoys the simplicity and computational efficiency, especially when the cardinality of the output is large, e.g., language
models where output vector dimension is the same as the vocabulary size. As opposed to KL divergence, DAIR-SQ is also readily applicable to regression with uncountable output.

heading: 2.3 FURTHER JUSTIFICATION OF DAIR-SQ AND PRACTICAL CONSIDERATIONS
text: While we have already compared DAIR-SQ with several consistency regularization alternatives, we want to specifically focus on a closely related DAIR variant called DAIR-L1, i.e., R1(ℓ(z; θ), ℓ(z̃; θ)) = |ℓ(z; θ) − ℓ(z̃; θ)|. As we already observed in Section 2.2, DAIR-L1 either outright failed or was unstable on majority of the experiments we have performed so far. The following lemma further investigates the discrepancy between DAIR-SQ and DAIR-L1: Lemma 1. For any non-negative loss function ℓ,
R1(z, z̃; θ)−Rsq(z, z̃; θ) = 2 √ min{ℓ(z; θ), ℓ(z̃; θ)}Rsq(z, z̃; θ) ≥ 0.
Thus, R1(z, z̃; θ) ≥ Rsq(z, z̃; θ) with equality iff ℓ(z̃; θ) = 0 or ℓ(z; θ) = 0 or ℓ(z̃; θ) = ℓ(z; θ).
The proof of Lemma 1 appears in Appendix A. The difference is depicted in Figure 5. This suggests that Rsq(z, z̃; θ) incurs a much smaller penalty when ℓ(z; θ) is large. On the other hand, when ℓ(z; θ) ≈ 0 the regularizer is much stronger and almost equivalent to R1. Why does this matter? At the beginning of training when the network is not yet trained, the loss values on the original samples are large, and the Rsq regularizer is weak letting the training to proceed towards a good solution for the original samples. As the network is being trained on original samples and their loss is vanishing, the regulairzer starts to force the network to become invariant on the augmented samples.
We empirically verify this conjecture on Colored MNIST with Adversarial Augmentation. Figure 6 depicts the classification loss and
regularization of the first 10 and last 140 iterations. One observes that at the beginning of training, regularization term of DAIR-SQ impacts the training dynamics less while DAIR-L1 starts optimizing the regularizer right away, which dominates the entire training procedure and therefore leads the model to a poor local minimum. The left panel of Figure 6 confirms that the classification loss of DAIR-L1 remains large and unchanged (that of a random classifier).
This same property of DAIR-SQ also weakens the regularizer on training samples with high losses at the later stages of training. These samples are likely noisy, which makes DAIR-SQ more robust to noisy samples, as we already observed in Section 2.2.
2.4 THE IMPACT OF PARTIAL AUGMENTATION
We explore the impact of partial augmentation, where we only augment a certain fraction of the training samples. The experiment revisits noiseless Rotated MNIST with weak rotation data augmentation and Colored MNIST with Adversarial augmentation. This experiment emulates situations where an augmentation function is only applicable to certain examples or where augmentation is expensive and we would like to decrease the augmentation cost.
In Figure 7, we report the experiment results for DA-ERM and DAIR-SQ by applying augmentation only {10%, 20%, 30%, 50%, 100%} of the training samples, averaged on three runs. In Rotated MNIST experiment, as can be seen, DAIR-SQ with augmentation on only 20-30% of the samples performs similar to full augmentation. On the other hand, DA-ERM is more sensitive to partial augmentation and is subject to a steeper performance drop. This could be viewed as further evidence that DAIR-SQ could reach its best performance using weak augmenter functions. It is also noteworthy that in this example, DAIR-SQ with only 10% partial augmentation still outperforms DA-ERM with 100% augmentation. One can draw similar conclustion in the Colored MNIST experiment as only 10% augmentation gives comparable performance to full augmentation.

heading: 3 EXPERIMENTS ON REAL-WORLD TASKS

heading: 3.1 ROBUST REGRESSION: SIMULTANEOUS DOMAIN SHIFT AND LABEL NOISE
text: In this experiment, we consider a regression task to minimize the root mean square error (RMSE) of the predicted values on samples from the Drug Discovery dataset. The task is to predict the bioactivities given a set of chemical compounds (binary features). We follow the setup of Li et al. (2021) to introduce random noise to corrupt the targets. Furthermore, similar to Colored MNIST, we add a spurious binary feature to the original setup. At training time, the spurious feature is set to 1 if a particular target is above the median of the all the targets in the training samples, and 0 otherwise. At test time, this condition is reversed leading to poor generalization. We compare using ERM, DA-ERM and DAIR-SQ formulations under 0%, 20% and 40% noise levels on three baselines: L2 loss, Huber loss, and negatively tilted loss (Li et al., 2021), which is called tilted empirical risk minimization (TERM) and is designed for robust regression. For each of these baselines, we perform data augmentation by randomly assigning the spurious feature as 0 or 1 with equal probability. Finally, we apply the DAIR-SQ regularizer to each of these loss functions with λ = 10.
The results of this experiment are reported in Table 1. In the last column of the table we report results on the clean dataset without any spurious features for comparison purposes. As can be seen, without data augmentation all methods fall prey to spurious features and perform poorly, especially as the noise level is increased. It is noteworthy that while TERM is not designed for domain shift, it slightly outperforms the other baselines in the presence of spurious features showing that TERM has some inherent robustness to the domain shift. By adopting data augmentation, testing error decreases but is still quite large as compared to the Clean ERM setup for high values of noise. Notably, DAIR is able to reduce the testing error across all objectives and noise levels with the gap between DAIR and other approaches increasing with the degree of noise. For the 0% noise setup, DAIR is able to almost recover the Clean ERM accuracy for all three objectives. The gains achieved with DAIR are prominent for L2 and Huber, but marginal for TERM. Finally, data augmentation/DAIR combined with TERM can simultaneously handle domain shift and noisy labels as can be seen in this table.

heading: 3.2 INVARIANT VISUAL QUESTION ANSWERING
text: Visual Question Answering (VQA) has diverse applications ranging from visual chatbots to assistants for the visually impaired. In such real-world settings, it is desirable for VQA models to be robust to variations in the input modalities. In this spirit, recent works (Agarwal et al., 2020; Shah et al., 2019; Ray et al., 2019) have studied the robustness and consistency of VQA models under linguistic and visual variations. In this paper, we focus on the InVariant VQA (IV-VQA) dataset which contains semantically edited images corresponding to a subset of the original images from VQA v2 (Goyal et al., 2017). For each image in this subset, IV-VQA contains one or more edited images constructed by removing an object which is irrelevant to answering the question. A robust model should be invariant to such edits by making the same predictions on the edited image.
We choose the attention based SAAA (Kazemi & Elqursh, 2017) model to match the original setup from Agarwal et al. (2020). Using DAIR, we enforce consistency in predictions between the original and edited samples. Wherever the edited image is not available, the DAIR formulation reduces to ERM. We use the standard VQA accuracy along with the consistency metrics proposed in Agarwal et al. (2020) to compare our results against the ERM setup and the DA-ERM approach discussed in Agarwal et al. (2020).
The results are reported in Table 2. We measure the accuracy on the original VQA v2 ‘val’ set and the consistency metrics across edited IV-VQA instances and their corresponding real instances from VQA v2 ‘val’ set. The consistency metrics measure the three types of flips namely, pos → neg, neg → pos and neg → neg. A pos → neg
flip indicates that the answer predicted with the original image was correct but was wrong with the corresponding edited image. A neg → neg flip indicates that the answer changes from original to edited image but is wrong for both. The accuracy of DAIR on the VQA v2 ‘val’ set is higher as compared to others, while improving over all baselines by a minimum of 1.3% under the ‘Predictions flipped’ metric which is the sum of the three types of flips. This improvement is significant given that the model needs to predict the answer correctly from 3000 candidate answers. While applying DAIR to this task, we observe a trade-off between the VQA accuracy on ‘val’ and the ‘Predictions flipped’ percentage controlled by the λ parameter. By increasing λ, the ‘Predictions flipped’ percentage decreases, and drops to as low as 7-8% when λ is at 10, albeit sacrificing the VQA accuracy by 5-6%. Thus, for moderate values of λ, DAIR is able to maintain the predictive power while enforcing consistency across variations in the visual space.

heading: 3.3 TRAINING ROBUST DEEP NETWORKS AGAINST ADVERSARIAL ATTACKS
text: In this section, we demonstrate that our regularizer can be applied to train robust neural networks and it achieves comparable or better results than baseline models from state-of-the-art approaches which are specifically designed for this task. In our approach, the augmented examples z̃ can be generated by a certain strong attack, such as Projected Gradient Descent (PGD) (Madry et al., 2018) or CW (Carlini & Wagner, 2017).We conduct our experiments on CIFAR-10 dataset and compare our approach with several other state-of-the-art baselines.
The performance of our algorithm against FGSM and variants of PGD, is summarized in Table 3, which shows that our results are competitive with the baselines. We report the performance of DAIR-SQ in Table 3 based on the configurations that give the best Clean accuracy (row 3) and the best Robust accuracy against PGD20 (row 6). The trade-off curve shown in Figure 8 suggests that by sweeping the value of λ, DAIR-SQ can achieve a better clean accuracy but a slightly lower PGD20 accuracy, and dominates most of the baseline, while it achieves a similar performance with TRADES. Note that the formulation in TRADES is equivalent to consistency regularization with KL divergence between the logits of the original and adversarial images. As opposed to our setup, the regularizer term in TRADES is also used in solving the maximization problem to generate adversarial images, whereas we only use the original loss for generating the adversarial examples.
We also report the accuracy consistency metric (CM) in this experiment in Table 3. CM captures the consistency of accuracy on PGD20 attack compared to clean examples. We observe that DAIR-SQ outperforms all baselines, which is in line with its best generalization to different attacks.

heading: 3.4 NEURAL TASK-ORIENTED DIALOG MODELING
text: Virtual digital assistants that engage in conversations with human users are rapidly gaining popularity. These devices require the modelling of task-oriented dialog systems that can communicate with users through natural language to accomplish a wide range of tasks. One of the main objectives in task-oriented dialog systems is the Dialog State Tracking (DST), which refers to keeping track of the user goals as the conversation progresses. Among task-oriented dialog datasets, MultiWOZ (Budzianowski et al., 2018) has gained the most popularity owing to the availability of 10k+ realistic dialogs across 8 different domains, and has been improved several times (Wu et al., 2019; Eric et al., 2019; Zang et al., 2020; Han et al., 2021; Qian et al., 2021).
Recently, SimpleTOD (Hosseini-Asl et al., 2020) achieved state-of-the-art results on MultiWOZ using a neural end-to-end modeling approach. However, Qian et al. (2021) observed that the performance of SimpleTOD drops significantly when the test set named entities (which are places in the UK) are replaced with new ones never observed during training (with new entities all based in the US), perhaps due to the memorization of named entities during training. We leverage DAIR-SQ to promote invariance of the dialog policy to named entities in the dialog flow. Here, the data augmentation scheme is a simple one. We replace named entities in the training set with their randomly scrambled version. For example, “cambridge” could be turned into “bmcedrgia.” Details on training data, augmentation schemes and hyper-parameters can be found in Appendix H.
The results are presented in Table 4, where performance is measured in Joint Goal Accuracy (JGA). JGA is a binary metric, and is equal to 1 if the predictions of all dialog states in a turn are correct. As such it is a difficult metric to get right too. As can be seen, both DA-ERM and DAIR outperform SimpleTOD (Hosseini-Asl et al., 2020) on MultiWOZ 2.2 w/ SGD entities (Qian et al., 2021). Perhaps, more surprisingly, DAIR also outperforms SimpleTOD on the original MultiWOZ 2.2 test set with no distribution shift, which we attribute to better robustness to the named entity memorization problem observed by Qian et al. (2021). Finally, we also observe that DAIR significantly improves the JGA consistency metric compared to the DA-ERM baseline.

heading: 4 CONCLUSION
text: In this paper, we proposed a simple yet effective consistency regularization technique, called data augmented invariant regularization (DAIR). DAIR is applicable when data augmentation is used to promote performance invariance across pairs of original and augmented samples, and it enforces the loss to be similar on the original and the augmented samples. As such, DAIR requires access to pairs of original and augmented examples. We also provided motivation and justification for DAIR, and particularly showed that it can recover the optimal solution in a certain regression task where data augmentation alone is insufficient. We also compared DAIR with several other consistency regularizers on several toy problems and showed that it is more stable and results in better performance. We empirically evaluated DAIR in four real-world machine learning tasks, namely robust regression, invariant visual question answering, training robust deep neural networks, and task-oriented dialog modeling. This is a major benefit of DAIR as some of other consistency regularizers cannot be applied broadly. Empirically, DAIR performed well on all these tasks and set new state-of-the-art results in these benchmarks.
Several problems remain open for future research: An in-depth theoretical understanding of the properties of DAIR that lead to its superior empirical performance on broad applications is an important open question. Further, automated hyperparameter tuning techniques for the strength of the regularizer is another avenue for future research. Finally, while we showed that DAIR boosts existing performance metrics, such as accuracy, the interplay of DAIR with other metrics, especially group fairness, is another important area for future research.

heading: A PROOFS
text: Proof of Proposition 1. First let us present the DA-ERM solution: fDA-ERM(w) =E [ (w1x+ w2y − y)2 + (w1x+ w2(ay + n)− y)2 ] (1)
=E [ w21x 2 + (w2 − 1)2y2 + 2w1(w2 − 1)xy ]
+ E [ w21x 2 + (w2a− 1)2y2 + w22n2 ]
+ E [2w1(w2a− 1)xy + 2w1w2xn+ 2w2(w2a− 1)yn] (2) =w21σ 2 x + (w2 − 1)2(σ2x + σ2ε) + 2w1(w2 − 1)σ2x
+ w21σ 2 x + (w2a− 1)2(σ2x + σ2ε) + w22σ2n
+ 2w1(w2a− 1)σ2x (3) =(w1 + w2 − 1)2σ2x + (w2 − 1)2σ2ε + (w1 + w2a− 1)2σ2x + (w2a− 1)2σ2ε + w22σ2n. (4)
Hence, the solution of w⋆DA-ERM = argminw fDA-ERM(w) is given by 2w⋆1 + (1 + a)w ⋆ 2 − 2 = 0,
(w⋆1 + w ⋆ 2 − 1)σ2x + (w⋆2 − 1)σ2ε + a(w⋆1 + w⋆2a− 1)σ2x + a(w⋆2a− 1)σ2ε + w⋆2σ2n = 0. (5)
Subsequently,
w⋆DA-ERM =  a2(σ2x+σ 2 ε)−2a(σ 2 x+σ 2 ε)+σ 2 x+σ 2 ε+2σ 2 n a2(σ2x+2σ 2 ε)−2aσ2x+σx+2(σ2ε+σ2n)
2(a+1)σ2ε a2(σ2x+2σ 2 ε)−2aσ2x+σ2x+2(σ2ε+σ2n)  . (6) w⋆DAIR = argmin
w fDAIR(w)
= argmin w
E [ (w1x+ w2y − y)2 + (w1x+ w2(ay + n)− y)2 ] + [ λ(|w1x+ w2y − y| − |w1x+ w2(ay + n)− y|)2 ] .
When λ → ∞, we have w⋆DAIR,2 = 0 and hence:
w⋆DAIR =
( 1
0
) .
We then evaluate the testing loss assuming the spurious feature is absent, i.e., xtest = (x, s = 0).
ℓDAIR(xtest;w ⋆ DAIR) = E [ (w⋆DAIR ⊤xtest − y)2 ]
= E [ (x− (x+ ε))2 ] = σ2ε .
ℓDA-ERM(xtest;w ⋆ DA-ERM) = E [ (w⋆DA-ERM ⊤xtest − y)2 ]
= E
[( a2(σ2x + σ 2 ε)− 2a(σ2x + σ2ε) + σ2x + σ2ε + 2σ2n
a2(σ2x + 2σ 2 ε)− 2aσ2x + σx + 2(σ2ε + σ2n)
x− (x+ ε) )2]
= σ2ε + (a+ 1)4σ4εσ 2 x
(a2(σ2x + 2σ 2 ε)− 2aσ2x + σx + 2(σ2ε + σ2n))2
≥ ℓDAIR.
Proposition 2. It is not hard to check that even using the weight decay regularizer γ2 (w 2 1+w 2 2) would not close the gap between the performance of DA-ERM and DAIR. In particular, this regularizer would result in
w⋆DA-ERM-WD =  a2(σ2ε+σ 2 x)−2a(σ 2 ε+σ 2 x)+2γ+σ 2 ε+2σ 2 n+σ 2 x a2(γ(σ2ε+σ 2 x)+2σ 2 ε+σ 2 x)−2aσ2x+γ2+γ(σ2ε+σ2n+σ2x+2)+2σ2ε+2σ2n+σ2x
(a+1)(γ(σ2ε+σ 2 x)+2σ 2 ε)
a2(γ(σ2ε+σ 2 x)+2σ 2 ε+σ 2 x)−2aσ2x+γ2+γ(σ2ε+σ2n+σ2x+2)+2σ2ε+2σ2n+σ2x
)  , which is not equal to w⋆ = (1, 0) unless σ2n → ∞ and γ = 0.
Proof of Proposition 2. The proof follows the same idea of Proposition 1 and therefore it is omitted here.
Proof of Lemma 1. We proceed as follows: R1(z, z̃; θ)−Rsq(z, z̃; θ) = 2 √ min{ℓ(z; θ), ℓ(z̃; θ)} ∣∣∣√ℓ(z̃; θ)−√ℓ(z; θ)∣∣∣ ,
We break it into two cases: if ℓ(z̃; θ) > ℓ(z; θ): R1(z, z̃; θ)−Rsq(z, z̃; θ) = ℓ(z̃; θ)− ℓ(z; θ)− ( √ ℓ(z̃; θ)− √ ℓ(z; θ))2
= ℓ(z̃; θ)− ℓ(z; θ)− ℓ(z̃; θ)− ℓ(z; θ) + 2 √ ℓ(z̃; θ) √ ℓ(z; θ)
= −2ℓ(z; θ) + 2 √ ℓ(z̃; θ) √ ℓ(z; θ)
= 2 √ ℓ(z; θ)( √ ℓ(z̃; θ)− √ ℓ(z; θ)).
If ℓ(z̃; θ) ≤ ℓ(z; θ): R1(z, z̃; θ)−Rsq(z, z̃; θ) = ℓ(z; θ)− ℓ(z̃; θ)− ( √ ℓ(z̃; θ)− √ ℓ(z; θ))2
= ℓ(z; θ)− ℓ(z̃; θ)− ℓ(z̃; θ)− ℓ(z; θ) + 2 √ ℓ(z̃; θ) √ ℓ(z; θ)
= −2ℓ(z̃; θ) + 2 √ ℓ(z̃; θ) √ ℓ(z; θ)
= 2 √ ℓ(z̃; θ)( √ ℓ(z; θ)− √ ℓ(z̃; θ)).
If we combine the two cases, we have: R1(z, z̃; θ)−Rsq(z, z̃; θ) = 2 √ min{ℓ(z; θ), ℓ(z̃; θ)} ∣∣∣√ℓ(z̃; θ)−√ℓ(z; θ)∣∣∣ .

heading: B PRACTICAL CONSIDERATIONS WHEN DAIR IS USED IN TRAINING
text: In this section, we investigate the reason why we see a sweet spot for the performance of DAIR-SQ as a function of λ. As shown in Figure 2, we see a sweet spot for λ, where the performance takes its maximum and starts to decrease for larger values of λ. There are a few explanations for this performance degradation.
1. It is observed that a large λ requires a relatively longer time for convergence. To show empirically this is true, we added another example in Appendix B.1. Theoretically, this is in line with the classical results in the optimization literature where larger Lipschitz constants (resulting from adding a regularizer) slows down the convergence rate. Thus, as we are training all models for a certain number of epochs, we will end up with underfitting.
2. A larger λ is more likely to guide the optimization trajectory towards a spurious poor local minimum with poor generalization performance, when the optimization trajectory is non-convex. We have experimentally verified this in Section 2.3 (Figure 6) as the reason for the poor performance of DAIR-L1 in Figure 2.
3. With a finite number of samples our regularizer does not necessarily lead to the best possible performance in the infinite sample setting (with weak domain shift). Hence, we might expect to observe the classical approximation-estimation tradeoff. This is especially true in real-world scenarios where one might expect that the difficulty of the example may not necessarily be preserved through data augmentation, and hence forcing the loss to be equal on both samples might be detrimental to the overall performance, which may lead to a practical sweet spot for λ.
We dig into the experiment in Figure 2 specifically and try to understand which case is the responsible for the sweet spot in Figure 2. We extend the number of training epochs from 40 to 160, and report the accuracy for λ ∈ {1.43, 8.85, 16.23, 100}.1 Table 5 suggests that, when we increase the number of training epochs, the sweet spot of λ moves from 8.85 to 16.23 and in fact we can achieve an even better performing model with accuracy 89.22 as compared to the previously reported 85.89, while the performance does not change much for the smaller values of λ. We also observe a big performance boost for larger values of λ. This suggests that in this experiment the sweet spot for λ is caused by capping the training epochs to a finite value. Having said that, we believe that we are practically interested in using DAIR with marginal computational overhead over ERM and hence we would expect to observe such sweet spot in performance in practice as λ → ∞.
B.1 ADDITIONAL EVIDENCE ON GROWING COST OF TRAINING WITH THE REGULARIZATION STRENGTH
We also provide further evidence for the growing cost of training with λ on a toy problem where we can reliably measure the gradient norm and ensure convergence. We study the following simple binary logistic classification problem which mirrors the MNIST experiments: at the training time the input is xtrain = (x, s = 2y − 1 + t1) and the label y, i.e., ztrain = (xtrain, y). Here, x ∼ N (0, σ2x), and P (y = 1|x) = 11+e−x , where t1 is independent of x and t1 ∼ N (0, σ 2 1). In this example, we intentionally provide feature s which is highly correlated with the label during training. Again, clearly, w⋆ = (1, 0)⊤, but w⋆ERM will converge to (0, 1)
⊤ due to the overfitting to the spurious feature. We introduce an augmenter which generates the augmented example such as xaug = (x, s = 2y − 1 + t1 + t2) where t2 ∼ N (0, σ22). We use this data augmenter for DAIR training and test on
1Note these values comes from log10 sweeping of λ.
xtest = (x, s = 1 − 2y). We summarize the steps need for convergences and the testing accuracy in Table 6 as well. We can find that the required number of iteration to convergence increases as λ increase.
For this tiny toy example, there is a factor of 10x increase in the required number of iterations when λ is chosen to be 10,000 as opposed to 0.5. Note that this is using ADAM and the gap is significantly larger if we use vanilla gradient descent; as we were not able to even converge in 108 steps. This is provided as further evidence for the practical sweet spot for DAIR as λ → ∞.

heading: C MODEL ARCHITECTURE AND TRAINING PARAMETERS FOR MNIST EXPERIMENTS
text: We use a Convolutional Neural Network (CNN) with three convolutional layers followed by two fully connected layers. The last layer output size for Colored MNIST experiments is set to 1, and 10 for the Rotated MNIST experiments. For training we follow a two stage schedule with a learning rate of 0.005 for the first 20 epochs and a learning rate of 0.0005 for the next 20. We choose a batch size of 64 for all experiments. The architectural details and training parameters can be found in Table 7 and Table 8.

heading: D COLORED MNIST & ROTATED MNIST SETUP
text: We apply the proposed loss function (DAIR) on the following two datasets: Colored MNIST and Rotated MNIST. We compare the performance of DAIR with plain data augmentation, and invariant risk minimization (IRM) as a strong baseline. One crucial difference between our work and IRM is is the motivation. IRM is designed to take two examples from two different environments and learn representations that are invariant to the environment, e.g., in cases where we are aggregating multiple datasets. On the other hand, we are interested in promoting invariance when we have a single dataset. As such, we artificially generate the second environment in IRM using data augmentation. For a given example z, we design an augmenter A(·) and use it to generate additional samples that adhere to the invariance we have in mind. Hence, IRM will be applied in the same way that examples from different environments are augmenting pairs.
Our Colored MNIST is an extension of the original Colored MNIST Arjovsky et al. (2019). The label is a noisy function of both digit and color. The digit has a correlation of 0.75 with the label and a certain correlation with the label depending on the color scheme. Besides the two colors in the original dateset, we introduce fully random colored scheme to the dateset, which is the best augmenter one can think of. The three color schemes are detailed in Table 9.
Our Rotated MNIST is a variant of the original Rotated MNIST (Ghifary et al., 2015). The original dataset contains images of digits rotated d degrees, where d ∈ D ≜ {0, 15, 30, 45, 60, 75}. Similarly, we introduce the random degree scheme here to serve as the best possible augmenter. To further exploit the potential of the proposed algorithm, we make this dataset more difficult by introducing more challenging degree scheme; The rotation schemes are summarized in Table 10.
Note all the augmented images are generated on the fly. Examples of images from some transformation schemes are shown in Figures 9 to 14.
Setup: We train a model consisted of three convolutional layers and two fully connected layers with 20,000 examples. For each dataset we are defining several different schemes on how the dataset could be modified: Table 9 (Colored MNIST) and Table 10 (Rotated MNIST). Then, we define several setups. Each setup is consisted of one original dataset, one augmentation dataset, and one test dataset, each of which is selected among the defined schemes. These setups are provided in Table 11 (Colored MNIST) and Table 12 (Rotated MNIST). For each setup, we train the model with the following four algorithms and compare their performances: ERM, DA-ERM, DAIR and Invariant Risk Minimization (IRM). Each experiment is repeated for 10 times; the mean and the standard derivation are reported. The value of λ are chosen base on the validation results. Detailed architectures and training parameters can be found in Appendix C.
D.1 COLORED MNIST
We conduct two sets of experiments for this dataset: Adversarial Augmentation Setup (Table 11) follows the exact same color schemes from the original Colored MNIST Arjovsky et al. (2019). For Random Augmentation Setup, we train the model with the strongest possible augmenter: uniformly random color. The entire procedure is summarized in Table 11.
D.2 ROTATED MNIST
We start with the strongest augmenter case. One may notice that there is a chance that the augmented images bear the same rotation degrees as the testing set. To make the task more difficult, we will use R6 as the augmented test to test how the trained model generalize to entirely unseen domain. The training procedure is summarized in Table 12.

heading: E ADDITIONAL RESULTS ON COLORED MNIST & ROTATED MNIST
text: E.1 COLORED MNIST
We show additional results on Colored MNIST and Rotated MNIST in Tables 13 and 14. Note that each algorithm has been tuned for best performance. As mentioned in Section 2.2, DAIR outperforms DA-ERM, ERM and other baseline models on classification accuracy. For accuracy consistency, we use the training scheme as the original scheme and the testing scheme as the augmentation scheme. We further compare DAIR with IRM (Arjovsky et al., 2019), DRO (Sagawa et al., 2019), and REx (Krueger et al., 2021). In doing so, we feed all original examples as one environment and all augmented examples as a second environment to these baselines. While we can see that DAIR outperforms all baselines, we caution that the comparison may not be fair in that DAIR exploits pairing information between original and augmented samples, which is not used by the other baselines.
E.2 ROTATED MNIST
We report the accuracy consistency on Rotated MNIST (weak augmentation) in Table 15. The original training scheme here is Scheme R4 (Table 10), i.e., 90◦ and 270◦ rotated images, and the augmentation scheme for training is R6 (weak rotation). At test time, we test with R1 (no rotation) and we also use the augmentation scheme of 180◦ rotation to test the accuracy consistency metric. Note that neither the un-rotated or 180◦ rotated images have been observed at training time. Hence, the setup is difficult for ERM which struggles to generalize. As can be seen, since the digit 0 is “almost” circularly symmetric, ERM actually does a decent job at classifying 0, however it significantly struggles with all other digits. We see that DAIR outperforms ERM and DA-ERM by a large margin. We observe that digits 6 and 9 are challenging to get right (as one would expect for them to be difficult to tell apart). While we see 2− 3% drop on the consistency for digits 6 and 9 (when rotating them by 180◦), the drop is smaller than expected perhaps due to the fact that the neural network learns to classify these digits based on features that are harder to get for humans.

heading: F SETUP AND ADDITIONAL RESULTS FOR VISUAL QUESTION ANSWERING
text: Table 16 indicates a tradeoff between the accuracy on the VQA v2 ‘val’ set and the consistency metrics. As the λ value increases, the consistency between the predictions increases, while the accuracy on original examples decreases. For instance, A λ value of 10 strongly boosts consistency thus lowering the ‘Predictions flipped’ percentage to only 7.9% but sacrifices the predictive power causing the accuracy to drop to 51.3%.

heading: G DETAILS ON TRAINING ROBUST NEURAL NETWORKS
text: For all algorithms reported in Table 3, we use Pre-Activation ResNet-18 (He et al., 2016), with a last-layer output size of 10 as the classification model. For training the DAIR model, the adversarial examples are generated by L∞ based PGD attack with 11 iterations, ε (attack strength) set to 8/255 and attack step size to 2/255. We evaluate all the models against the standard FGSM attack and PGD attack with 20 iterations of same perturbation sizes.

heading: H DETAILS ON NEURAL TASK-ORIENTED DIALOG MODELING
text: We provide details on the benchmark that we used in this experiment. Qian et al. (2021) proposed a new test set for MultiWOZ 2.2, called MultiWOZ 2.2 with SGD entities, where named entities are replaced with those from Schema Guided Dialog dataset (Rastogi et al., 2020) and showed that SimpleTOD (Hosseini-Asl et al., 2020) endures more than 8% performance drop on the new test set. Examples from the dataset are shown in Table 18. To address this problem, we define a new data augmentation scheme for DAIR and DA-ERM by replacing the named entities from the MultiWOZ 2.2 training set with randomly scrambled versions of the named entities. For example, “warkworth house” could be turned into “easrtokow hhrwu” (see Table 18). In all of our experiments, we utilize the SimpleTOD model (Hosseini-Asl et al., 2020) and we apply DAIR to enforce invariance between the named entities in the training examples and the scrambled entities from their corresponding augmented samples. The model is trained with ParlAI (Miller et al., 2017) fine-tuned with the pre-trained BART (Lewis et al., 2019). Training hyper-parameters can be found in Table 17.

Review summaries:
< Summary >   Many machine learning models depend on empirical risk minimization (ERM), vicinity risk minimization (VRM), and ERM with data augmentation. Original ERM and VRM show a good performance in many tasks, but they fail to generalize the out-of-distribution dataset. This paper proposes a new regularization method, data augmented invariant regularization (DAIR) methods. DAIR is a kind of regularizer that penalizes loss inconsistency. This paper validates the superiority of DAIR on drug discovery, visual question answering, and CIFAR-10 

The authors propose a conceptually simple regularizer which forces consistency of loss between the true samples and their corresponding augmentations, termed DAIR-SQ. The authors demonstrate the qualitative characteristics of DAIR-SQ with a clean toy example, and a supporting theoretical result of asymptotic consistency on the same. For real-world datasets, the authors demonstrate the robustness induced by DAIR-SQ under domain shift, label noise, and adversarial attacks.

This paper studies a new learning method that makes a model be invariant under a data augmentation (implicitly, to achieve high test performance when the test data distribution is different from the training data distribution). The authors propose a new objective function, called DAIR, characterized by a regularized function where a regularizer maintains the performance of the model on data augmentations. Some numerical experiments by varying regularizers and datasets are provided (in Sections 2 and 3), and some theoretical results are presented. The authors claim that DAIR consistently outperforms the baseline methods including ERM and DA-ERM.

In this paper, the authors proposed a regularizer (DAIR) that penalize the difference between the model performance on original data and augmented data. The authors argued that the new regularizer imposes invariance and effectively prevented the model from overfitting to spurious features. By applying the regularizer on various types of machine learning tasks, the authors analyzed the experiments results and explained the empirical novelty.