Robustness through Data Augmentation Loss Consistency
Authors: Tianjian Huang
Shaunak Ashish Halbe
Chinnadhurai Sankar
Pooyan Amini
Satwik Kottur
Alborz Geramifard
Meisam Razaviyayn
Ahmad Beirami
Venue: Accepted by TMLR ()

Abstract: While deep learning through empirical risk minimization (ERM) has succeeded at achieving human-level performance at a variety of complex tasks, ERM is not robust to distribution shifts or adversarial attacks. Synthetic data augmentation followed by empirical risk minimization (DA-ERM) is a simple and widely used solution to improve robustness in ERM. In addition, consistency regularization can be applied to further improve the robustness of the model by forcing the representation of the original sample and the augmented one to be similar. However, existing consistency regularization methods are not applicable to covariant data augmentation, where the label in the augmented sample is dependent on the augmentation function. For example, dialog state covaries with named entity when we augment data with a new named entity. In this paper, we propose data augmented loss invariant regularization (DAIR), a simple form of consistency regularization that is applied directly at the loss level rather than intermediate features, making it widely applicable to both invariant and covariant data augmentation regardless of network architecture, problem setup, and task. We apply DAIR to real-world learning problems involving covariant data augmentation: robust neural task-oriented dialog state tracking and robust visual question answering. We also apply DAIR to tasks involving invariant data augmentation: robust regression, robust classification against adversarial attacks, and robust ImageNet classification under distribution shift. Our experiments show that DAIR consistently outperforms ERM and DA-ERM with little marginal computational cost and sets new state-of-the-art results in several benchmarks involving covariant data augmentation. Our code of all experiments are available at: https://github.com/optimization-for-data-driven-science/DAIR.

Published in Transactions on Machine Learning Research (11/2022)
Robustness through Data Augmentation Loss Consistency
Tianjian Huang
tionjion Busc. edu
University of Southern California
Shaunak H
shalbe:9@ggteech.ede edu
Georgia Institute of Technology
Chinnadhurai Sankar
chinnadhurus@mmetts.com
Meta AI
Pooyan Amini
pamini@meta.com com
Meta AI
Satwik Kottur
skottur@meta.com com
Meta AI
Alborz Geramifard
alboray@meta. com
Meta AI
Meisam Razaviyayn*
razaviga edu
University of Southern California
Ahmad Beiramit
beirami@@google.com
Google Research
Reviewed on OpenReview: https: //te:///ppeeee openreview. net/ forum? id= almeaRylbl
Abstract
While deep learning through empirical risk minimization (ERM) has succeeded at achieving human-level performance variety of complex tasks, ERM is not robust to distribution shifts adversarial attacks. Synthetic data augmentation followed by empirical risk mini- mization (DA-ERM) is simple and widely used solution to improve robustness in ERM. In addition, consistency regularization can be applied to further improve the robustness of the model by forcing the representation of the original sample and the augmented one to be similar. However, existing consistency regularization methods are not applicable to covoriant data augmentation, where the label in the augmented sample dependent on the augmentation function. For example, dialog state covaries with named entity when augment data with a named entity. In this paper, propose data augmented loss invariant regularization (DAIR), simple form of consistency regularization that is applied directly at the loss level rather than intermediate features, making it widely applicable to both invariant and covariant data augmentation regardless of network architecture, problem setup, and task. We apply DAIR to real-world learning problems involving covariant data augmentation: robust neural task-oriented dialog state tracking and robust visual question answering. We also apply DAIR to tasks involving invariant data augmentation: robust regression, robust classification against adversarial attacks, and robust ImageNet classification under distribution shift. Our experiments show that DAIR consistently outperforms ERM and DA-ERM with little marginal computational cost and sets new state-of- of--oh---t-ttttt results in several benchmarks involving covariant data augmentation. Our code of all experiments are available al: betps://gitthule -ttpiaaat..oootttttttaaaaaaooo-o data-oo------riiiea sttaarddvvvveniieeeeeeAAAAe
"The work of TH, SH, and MR was partially supported with funding from the USC-Meta Center for Research and Education in Al and Learning (REAL@USC) and with gift funding from 3M. The work of SH was done the University of Southern California and the work of AB done at Meta AL.

---

Published in Transactions on Machine Learning Research (11/2022)
1 Introduction
Deep neural networks are widely used in various applications ranging from computer vision to language processing, While deep learning has surpassed human-level performance in numerous tasks, neural networks fail under small adversarial perturbations of the lest samples (Goodfellow et al. 2015) or natural shifts of distribution at deployment time (Arjovsky et al., 2019). These issues have motivated the research community to invest in a varicty of methods for evaluation and mitigation of robustness in deep learning. This includes introduction of new robustness benchmarks, e.g., Rotaled/ Kotaled/Colloed /Colored MNIST (Arjovsky et al. 2019 DomainNet (Peng et al.] 2019), ImageNet-RR (Hendrycks et al., 2021). ImageNet-9 (Xiao et al.) 2020 for robustness to natural distribution shift; and Fast Gradient Sign Method (FGSM) (Goodfellow et al. and Projected Gradient Descent (PGD) (Madry et al. 2018 for adversarial robustness.
2014
Researchers have also proposed numerous algorithmic solutions to improve robustness to distribution shift Ganin et al. 2016 Ghifary et al. 2015 Sagawa et al. 2019 Li et al.
2018a Sun & Saenko 2016 Li
ct al. 2018b c Krueger et al.] 2021 Zhang et al. 2021 | Robey et al.
2022 and adversarial attacks (Madry
et al.
2018 et al. 2020 Zheng et al. 2020 Zhang et al. 2019
Tack et al.,
2021p. p. These approaches
are usually more complex than conventional empirical risk minimization (ERM) and hence they cannot be
readily applied to involved lasks with non-trivial model architectures. For example, in generative language modeling imposing a constraint on the intermediate data representations is non-trivial which is required
by CORAL (Sun & Sacnko 2016). In addition, recently Gulrajani & Lopez-Paz (2020) demonstrated that vanilla ERM, if tuned properly, remains competitive with (or may even outperform many such complex methods in real-world scenarios involving robust inference.
Data augmentation can be employed to improve the robustness of ERM by curating synthetic examples that exhibit a desired invariance/ covarriiccco In this paper, invariant data augnentation refers to the case where the features are perturbed to obtain a synthetic augmented example that preserves the original label. For example, in image classification task where the aim is to classify the foreground of an image, create augmented data points by keeping the foreground and changing the background. Under this data augmentai ation process, the label of data (i.e., the image class) does not change and hence we call it invariant data augmentation. On the other hand, covariant data augmentation refers to the case where perturbation of the features results in the label to covary with the features. For example, in a dialog state tracking task where data augmentation is performed by varying named entities in the input data, the model output (label) of the augmented data also needs to covary to contain the new named entity.
can
Data augmentation techniques abound in the literature: Tensmeyer & Martinez (016) and Cutout (Derries & Taylor 2017) curate invariant image transformations to improve image representations. Mixup (Zhagg
et al. 2017) and CutMix (Yun et al.) 2019) curate covariant data augmentations via linear combination of features between different classes to learn more robust representations. Volpi et al.] (2018); Zhou ct al. 2020
perform data augmentation with adversarial images to improve robustness. Finally, Cubuk el al. (2018) Lim
let al. (2019) introduce a procedure which automatically searches for improved data augmentation policies.
While simple, data augmentation remains an effective and universal solution to improve model robustnoss.
Consistency regularization can be further applied on top of data augmentation to enhance robustness by enforcing the desired invariances on the model. Engstrom et al. (2018); Kannan et al. (0018) Zhang let al.
2019 : TTck et al.] (2021) utilize consistency regularization at an embedding layer to train robust
neural networks against adversarial attacks. Various forms of consistency regularization have been applied
to unsupervised learning (Sinha Dieng 2021). ) self-supervised learning (Chen et al., 2020: von Küggolgen
et al. 2021), and semi-supervised learning to exploit unlabeled data (Bachman al, 2014 Laine & Aila,
Miyato et al. 2018 Sohn et al. 2020 Xie et al. 2020). Standard consistency regularization forces intermediate features to be similar among all inputs variations and hence is only applicable to invariant data augmentation, where data augmentation keeps the label of the augmented sample intact. Such consistency regularization may even hurt performance in the face of covariant data augmentation, where the label for the augmented sample may change. For example, in dialog state tracking where the dialog state covaries with named entities dialog context, standard consistency regularization hurts overall performance as it forces the model to not depend on the named entity for its prediction. See Scction 2.1 for a more detailed explanation and Section for experiments that confirm this.
2016
2

---

Published in Transactions on Machine Learning Research (11/2022)
In this paper, we propose simple form of consistency regularization, called data augmented loss invariant regularization (DAIR), that is directly applied at the loss level. While existing consistency regularization methods can only be applied to invariant data augmentation, DAIR is applicable to both invariant/ araarian/covariann cooariantt data augmentation when a pair of data samples expecting consistent performance. This is vastly in contrast to existing feature consistency regularizers that apply on an intermediate embedding space or the very last. layer of the model (logits). As a result, DAIR only requires marginal computational cost on top of data augmentation, and is broadly applicable to wide host of learning tasks, including generative models with covariant data augmentation. We theoretically prove some of the properties of DAIR in Section 2 and empirically evaluate DAIR on variety of problems ranging from defense against adversarial attacks to domain generalization in the presence of environment shift in Sections 3 and 4 Our experimental results show that DAIR is compelitive with state-of-the-art algorithms specifically designed for these problems.
2 DAIR: Data Augmented Loss Invariant Regularization
For a data sample (zx, y), let {(z; be its parametric loss function, where 0 is the set of model parameters (e.g., network weights). The popular Empirical Risk Minimization (ERM) framework trains the model by minimizing the expected value of the following loss over the training data:
(2 ();
(ERM)
We assume that we have access to a (potentially randomized) data augmenter function A(-). Examples for A include (random) rotation, change of background, change of entity names. Such augmenters capture the transformations against which we wish to be invariant. Given a sample let 전 = () = A(=) denote an augmented sample. Previous work has used both original and augmented examples during training, which leads to the following standard objective function, called Data Augmented Empirical Risk Minimization (DA-ERM):
} fDA-ERM(2, 2,0) - 0( ((;6)).
(DA-ERM)
While DA-ERM has been successful in many applications, one natural question is whether we can further improve upon it using the knowledge that the performance on augmented samples should be consistent with the original ones. Consistency regularization further penalizes DA-ERM for any such inconsistency at the feature/loss level: (Comuistency,,DAe ž; 0) SDA-ERM(2, 2; )) + XD(2, ž; f), where D(2, 2) is a proper divergence between the original sample representation and the augmented sample representation, and where the goal of the regularizer applied at some intermediate feature space to maintain the performance of the model on z and ž consistent. In this paper, we focus on a specific type of such regularization, called data augmented loss invariant regularization (DAIR)
(DA-ERM(2,2 + (), 1 (=; + _ [( 0) XR(((=; ((:0))
(DAIR)
2
where the regularization is directly applied to the loss. This "loss-level" regularization approach is different from existing "feature-level" consistency regularization approaches discussed in section [ For example, consider classification task with a neural network F(,,0) Id + R where f denotes the weights of the network. This network lakes a d-dimensional input data point and outputs a k-dimensional logil/softmaax output. A popular training approach is to consider {(2; = C(F(x, (xx 8). where C(.-)) is the cross-entrops
loss and 2 = (x. y). Existing regularization mochanisms, such as Engstrom et al. 20188): Kannan et al. (018);;
Zhang et al. (2019); : Tack al. (2021): : Sinha & Dieng] (2021): von Kügelgen et al. (2021) Miyato et al.
2018
to name just few, operate at. the feature level, i.e., they are of the form R (F 0), F(r, 0)). However,
DAIR operates at the loss level as demonstrated in eq. (DAIR). The idea behind DAIR is to simply promote
E(2; 0) ~ 6), and ignore the features or even the rest of the possible outcomes of and simply focus on the
'We performed our initial experiments by weighting the original and augmented examples differently, and empirically observed that the weight had significant effect the performance. Hence, fixed equal weight on both original and augmented samples for rase of tuning.
3

---

Published in Transactions on Machine Learning Research (11/2022)
current sample's loss. While DAIR is relatively weaker form of consisteney regularization, it is suitable for problems where feature consistency may not be conceptually meaningful (See Section 2.1 for more detailed discussion). For instance, in language modeling when a pair of sentences differ in their corresponding named entitics, it is not clear why we should enforce their embeddings to be similar unless the label is desired not to depend on such named entity. However, loss consistency still meaningful promoting the probability of label given input to be the same on the original and the augmented samples.
We remark that DAIR requires pairing information between original and augmented samples, which may not always be available (e.g., DomainBed (Gulrajani & Llopzz-Paz 2020)). However, show that this simple approach is still broadly applicable to various rcal-world problems regardless of model architecture, and is indeed competitive with state-of-the-art methods for imposing invariance. As it turns out, we are particularly interested in a particular form of the DAIR regularizer:
Rz = -
(DAIR-SQ)
and we call this variant DAIR-SQ. Note that Reg has the same scale as the loss function l, making it casier to tune A. Empirically we observe that the optimal X for all the experiments mentioned later in the paper falls in [0.2, 100], across various tasks (from regression to sequence-to-sequenn generative modeling). Further justification on DAIR-SQ will be provided through the rest of this section.
Finally, in most (real-world) applications performance is measured through 0-1 metrics other than the loss function. For example, we are usually concerned with accuracy in image classification while we optimize cross-entropy loss. Let H(>; 0) E {0, denote a 0-1 evaluation performance metric of interest, e.g., accuracy. Given the sample 2 (or 2), the model performance is captured by H(2;0) (or H)²; For any & such that H(z; 0) = 1, we define the corresponding consistency metric as:
CM(2, (M), I()()))))= 0)z;0) = 1}.
(Consistency Metric)
Notice that similarly to the original performance metric, which only used for model evaluation, we use the consistency metric at evaluation time only.
2.1 Why DAIR at the loss level?
As we mentioned in Section [ consistency regularization has been extensively studied in the literature. However, regularization at loss level has been relatively unexplored. We propose DAIR at the loss level, making it broadly applicable when pairing information is available between original and augmented samples even under the cases where the respective labels are different. Consider the following two examples: visual question answering (Section 3.2) and dialog state tracking (Section 3.1 b in which the labels of the augmented examples covary with the augmented features. In these setups, feature consistency regularization is not conceptually meaningful as the embedding of the image with zebra removed should not be the same as the original image (Figure D or the embedding of the dialog state with named entity changed from airport to bus station should not remain unchanged (Figure 2). In fact, forcing the embeddings to be exactly the same will remove vital information needed for performing the task and will incorrectly force the model to provide the same output for the original and augmented samples. On the other hand, we can enforce the loss value at the augmented sample and the original sample to be the same, which implies p(ū 35; 0) Rs p(ylæ;0) 0) when loss is viewed as likeliioddo function.
To contextualize DAIR, consider a classification task using a function approximator (c.g., a deep neural network) followed by a softmax layer. Let F(z, 0) be the output of the model right before the sofumax layer. Hence, F(z, () x =(((,=00) for all possible outcomes In addition to two DAIR variants, we consider the regularizer to be any proper divergence between the output distributions F(x, f) and F(ū, 0), such as KL divergence, which will promote F(z, 0) x F(G.8) f). In addition to DAIR-SQ we define the following regularizers that we use throughout the paper:
4

---

Published in Transactions on Machine Learning Research (11/2022)
Q: How many zebras are there in the picture?
A:2 2
zebra removed A: 1
book to the port?
book to the hus station?
low many people?
bow many people? it's just for me.
it's just for me..
tani-dest: adrport
tani-dest:: bus station kuoi-bookpeople:
luxii- bookkoopppepp:e:
Figure 1: VQA: Answer changes after augmentation.
Figure 2: DST: dialog state changes after augmenta- tion.
Image taken from Agarwal et al. (2020).
1.0
1.0
0.9
0.9
0.8
0.8
Jeurery
Aucuruy
0.7-
0.7
DAIR
DAIR
DAIR-LI
0.6
DAIR
XL
0.5
DA-ERUM
0.5
DA-ERM
ERM
0.4
10-1
10"
10'
10
20
10
Figure 3: Test accuracy for 0 = 45 rotated im- ages as a function of x (strength of the regularizer). Best validation À is marked with solid circles.
Figure 4: Test accuracy as function of test-time rotation 0 E [09, 60%). We report the performance of each mothod with their best validation A.
RL1(2,2;0) = ((2;0)
- 0)
(DAIR-L1)
(z, (L(
(KL Feature Consistency)
Notice that the KL feature consistency regularizer is oblivious to and remains the same for covariant data augmentation where 7/ + y.
Toy Experiment: Rotated 6 & 9. Consider a binary classification problem which is derived from MNIST with only digits and 9. The goal is to explore test-time distribution shifts in the form of rotation. We consider the augmentation function to be 180 rotations. Since the digits and 9 are similar to each other respectively after such rotation, we naturally flip the augmented label, making this covariant data augmentation. The rotation schemes for training, augmentation and lesting are detailed in Table D
In Figure 3 we benchmark the performance of KT, feature consistency, DAIR-LI, and DAIR-SQ against ERM and DA-ERM, when the test rotation is 8= 45%, as function of regularizer strength. As can be seen, DAIR-SQ (blue) is the only approach that outperforms the ERM baseline (green). This is not a coincidence. Our 180" rotation augmentation covariant and can be seen that vanilla DA-ERM does not help but even hurts the performance, which may be attributed the fact that the test rotations are novel and unscen. KL feature consistency (yellow) slightly outperforms DA-ERM for small X but struggles to outperform ERM. This is covariant data augmentation case, and hence feature consistency was expected to fail. We the performance of KL degrades as À gets larger. It is noteworthy that DAIR-LI (red) starts to degenerate to a random classifier baseline for large values of À and fails to produce a model that outperforms ERM. We will investigate this issue more deeply in the next section.
To further explore how the performance of these baselines are affected by the degree of test-time distribution shift, we consider variety of test sets with the degree of rotation 9 swept in [0P, 60%). In Figure 4 we report
A similar L1 regularization idea has been explored Garg al. 2019). however, our proposed DAIR regularizer differs suhstantially by imposing consistency at the loss level.
5

---

Published in Transactions on Machine Learning Research (11/2022)
Split
Rotation
Example Input
Example Label
Train
0
6
Augmentation
180
9
Test
E 10*60")
Table 1: The rotation scheme used in the Rotated 6 & 9 experiment.
the performance of each algorithm for different. values of 0. We see that DA-ERM hurts in-distribution accuracy (0" rotation), and generally hurts test accuracy for various degrees of rotation. For relatively large 0 (representing relatively large distribution shift), only DAIR-SQ consistently outperforms the ERM baseline. Further, since the data augmentation scheme was weak and did not match the test-time distribution, it was expected that vanilla data augmentation is not sufficient to help the model to generalize to the test-time shift. It is noteworthy that perhaps surprising DAIR-SQ outperforms ERM (by a large margin for big distribution shifts) even though DA-ERM is underperforming ERM.
Through this toy experiment, we motivated the significance of loss-level regularization through DAIR-SQ. To theoretically analyze why/how DAIR-SQ works, we also conduct a simple toy linear regression experiment in Appendix A followed by a multi-dimensional extension in Appendix B where we provide formal proofs to show that DAIR-SQ is guaranteed to outperform DA-ERM, even in the regime infinite data or when using weight decay regularization.
2.2 Practical considerations for DAIR
Next, we discuss a more detailed comparison of DATR-SQ with DAIR-L,1, and its dependency on X, where we explain the rationale for settling on DAIR-SQ for the experiments in Section
2.2.1 Why does DAIR-SQ significantly outperform DAIR-L1?
While we have already compared DAIR-SQ with several consistency regularization alternatives, we want to specifically focus on a closcly related DAIR variant called DAIR-L1, which has already appeared in the literature for invariant data augmentation (Garg et al., 2019). As we observed in Section 2.1 DAIR-LI either outright failed or was unstable on the toy example. The following lemma further investigates the discrepancy between DAIR-SQ and DAIR-L1:
Lemma 1. For any non-negative loss function e
R2, 2) (2, 0,
with equality i0 ((in(((;))) -(z 0.
The proof of Lemma Daammaapppee appears Appendix C.1 The difference is depicted in Figure 5 This suggests that (2 2; 8) incurs much smaller penalty when E(2; 8) is large. On the other hand, when E(2; 8) 0 the regularizer is much stronger and almost equivalent to RR11 Why does this matter? At the beginning of training when the network is not yet trained, the loss values on the original samples are large, and Reg regularizer is weak letting the training to proceed towards a good solution for the original samples. As the network being trained on original samples and their loss is vanishing, the regulairzer starts to force the network to become invariant the augmented samples. The above hypothesis is also empirically verified on Colored MNIST with Adversarial Augmentation (see Appendix 3
1.5
0
2
2
1
((2:8)
0
1(2;0)
Figure 5: The plot of (2, (=,
We also explore the impact of partial augmentation, where we only
augment certain fraction of the training samples. DAIR shows stable performance compared with DA-ERM when the number of augmented examples are limited. The results are presented in Figure 18 (Appendix G
6

---

Published in Transactions on Machine Learning Research (11/2022)
2.2.2 Dependence of DAIR-SQ the regularization strength À
While we have already seen that practically the optimal À lies in the range [0.2, 100), in this section we relate to the quality the solution.
Definition 2 (Empirical Expectation). S.
We use E denote the empirical expectation over a set of examples
1 [S]
E. ((=)
((zi).
iES
Proposition 3. Let 0 E mig (2, and denote any perfectly invoriant solution, i.c., (((()))),((E0)) = 0. We have:
R, { (200001::033::01:000) < B,
[(200) + E(2; 0)
2A
Proposition bounds the value of equation DAIR-SQ inversely proportional to A. Consider the example of classification task with K classes where the number of samples in different classes are the same. When the weights are zero, i.e., 0 =, 0, we have a perfectly invariant solution. Moreover, for this choice, we have {(; 0) log(K) for all if cross entropy loss is used. The above lemma implies that Eg (e(( 8%), ([))) s log K In other words, we can impose invariance by increasing À but we don't need very large X, reconfirming that À < 100 sufficed in all of our experiments.
Although we have shown above that À needs not to be very large, a natural question is that could we choose a large À anyway since when + oc, the resulting model perfectly invariant. In the following theorem, we show DAIR-SQ leads convergent algorithms when optimized by popular methods such as gradient descent and the convergence rate is affected by A. We remark that the proof of Proposition can be extended to cover the setup of DAIR-L1 well.
Theorem 4. Consider a classification problem with logistic loss, where x, is the imput, y is the output and denotes model parameters. Assume ||x||, ||x]|| S Da, and ||0|| S Do. After iterations of gradient descent have
algoritim (Algorillum in Appendir C.3
(1+ (
VT
Theorem 4 shows that DAIR-SQ penalizes the convergence rate for solving the problem to c-gradient accuracy D,,D, Do factor as À increases. The penalty grows linearly with X, which is consistent with the observation in Appendix F.2. However, we observe that the additional complexity is negligible in practice as we usually do not solve the optimization problems to stationarity but rather stop after certain number of iterations. For all experiments, we solve each task for a certain number of epochs, which is chosen the same as the ERM baseline. While Theorem 4 is established for the gradient descent, it can be extended the stochastic settings (Li 2019 Theorem 2) based on the smoothness of the DAIR-SQ loss established in the proof of Theorem
by AvD,,
3 Experiments on covariant tasks
Thus far, we observed that DAIR-SQ is practically stable variant of DAIR with some theoretical motivations and guarantees (such as convergence). In the rest of the paper, when we refer to DAIR without only postfix, we mean DAIR-SQ. As we empirically evaluate the performance of DAIR, we emphasize that the only hyperparameter that we tune for DAIR is (chosen via grid search on validation set). The rest of the hyperparameters, such as step-sizo, batch-size, and number of training epochs, are only tuned for the ERM bascline and chosen to be exactly the same for DAIR. In this section, we continue with covariant tasks where feature-level regularization expected to hurt the performance.
7

---

Published in Transactions on Machine Learning Research (11/2022)
3.1 Neural task-oriented dialog modeling
Virtual digital assistants that engage in conversations with human users are rapidly gaining popularity. These devices require the modeling of task-oriented dialog systems that can communicate with users through natural language to accomplish a wide range of tasks. One of the main objectives in task-oriented dialog systems is the Dialog State Tracking (DST), which refers to keeping track of the user goals as the conversation progresses. Among task-oriented dialog datasets, MultiWOZ (Budzianowski et al. 2018) has gained the most popularity owing to the availability of 10k+ realistic dialogs across 8 different domains, and has been improved several
times (Wu et al. 2019; Eric et al. 2019 Zang et al. 2020 Han et al. 2021 Qian et al. 2021).
Recently, SimpleTOn (Hosscini-Asl et al.
2020) achieved state-of-the-art results on MultiWOZ using neural end-to-end modeling approach. However, Qian et al. (2021) observed that the performance of SimpleTTOD drops significantly when the test set named entities (which are places in the UK) are replaced with new ones never observed during training (with new entities all based in the US), perhaps due to the memorization of named entities during training. We leverage DAIR to promote invariance of the dialog policy to named entities the dialog flow. More importantly, we show that standard consistency regularization on feature space simply does not work. Here, the data augmentation scheme is a simple one. We replace named entitics in the training set with their randomly scrambled version. For example, "cambridge" could be turned into "bmcedrgia." Details on training data, sugmentation schemes and hyper-parameters can be found in A Appendix
MultiWOZ 2.2 Test JGA
MultiWOZ 2.2 Test JGA
CM
w/ SGD entities
SimpleTOD (Hosseini-Aad et 2020 SOD + DA
0.5483
0.4844
0.8206
0.5915
0.5311
0.8854
SimpleTOD + KT. feature consistency
0.5124
0.4053
0.82298
SImpleTOD + DAIR
0.5998
0.5609
0.8902
Table 2: DAIR achieves state-of-the-art Joint Goal Accuracy (JGA) on both the original MultiWOZ 2.2 test set Zang
et al. 2020) and well as the MultiWOZ 2.2 test set w/ named entities replaced with SGD QQian et al. 2021
The results are presented in Table 2 where performance is measured in Joint Goal Accuracy (JGA). JGA is a binary metric, and is equal to 1 if the predictions of all dialog states in a turn are correct. As such it is a difficult metric to get right too. As can be seen, both DA-ERM and DAIR outperform SimpleTOn (Hosseini-As) et al. 2020 on MultiWOZ 2.2 w/ SGD entities (Qian et al., 2021p. More surprisingly, DAIR also outperforms SimpleT TOD on the original MultiWOZ 2.2 test set with no distribution shift, which we altribute to better robustness to the named entity memorization problem observed by Qian et al.] (2021): Cho et al. (2022). We also observe that DAIR significantly improves the JGA consistency metric compared to the DA-ERM baseline It worth nothing but the simple regularizer provides non-trivial performance increase which takes researchers several years of explore. Finally, we show that standard consistency regularization (KT.) is not well suited to this task and results in performance degradation (see Section 2.1 for more explanation on why).
3.2 Covariant Visual Question Answering
Visual Question Answering (VQA) has diverse applications ranging from visual chatbots to assistants for the visually impaired. In such real-world settings, it is desirable for VQA models to be robust to variations in the input modalitics. In this spirit, recent works (Agarwal et al.] 2020 Shah ct al., 2019 Ray et al. 2019) have studied the robustness and consistency of VQA models under linguistic and visual variations. In this paper, focus on the Invariant and Covariant VQA (IV/CV-VQA) dataset which contains semantically edited images corresponding to a subset from VQA v2 (Goyal et al., 2017). CV-VQA contains images constructed by removing an object which is relevant to answering the question and leads to a different answer. A robust. model should make predictions that are consistent to such edits.
We choose the attention basod SAAA (Kazmmi & Elqursh (017)) model to match the original sctup from Agarwal et al. (2020). Using DAIR, we enforce consistency in predictions between the original and edited samples. Wherever the edited image is not available, the DAIR formulation reduces to ERM. We use the
8

---

Published in Transactions on Machine Learning Research (11/2022)
standard VQA accuracy along with our consistency metric to compare results against the ERM and DA-ERM setups discussed in Agarwal et al. (2020). In addition we compare against KT. feature regularizer to highlight the strengths of our approach.
Algorithm
CV-VQA test 1
CM f
ERM (Kazemi & Elquish
2017
45.89
0.5792
DA-ERMI [Agarwal et 2020
48.32
0.5631
KL Feature Consistency
48.20
0.3479
DAIR
49.75
0.7161
Table 3: Accuracy and Consistency metries on CV-VQA test set
The results for the CV-VQA in Table 3 DAIR achieves a higher accuracy compared to all baselines. This improvement is significant given that the model needs to predict the answer correctly from 3000 candidate answers. As against this, applying KL, for feature consistency catastrophicall fails on the CV-VQA lask achieving significantly lower CM scores than ERM. We describe the Invariant VQA experiment is Section 1.3
4 Experiments on invariant tasks
Now that we have established the effectiveness of DAIR covariant tasks, we also benchmark its performance on invariant tasks where more baselines are available. As in the previouss section, we only tune for À via grid search and all other hyperparameters remain the same as the ERM baseline.
4.1 Colored MNIST
Colored MNIST (Arjovsky et al. 2019 is binary classification task built on the MNSIT dataset. Digits 0-4 are labeled 1; whereas digits 5-9 are labeled 0. Additionally, 25% label noise is added, i..., the labels are flipped with probability 0.25, both at train and test time, capping the achievable test accuracy to 75%. In this dataset, each digit is RGB colored. During training, label 1 is given the color green with probability 0.9 and red with probability 0.1. On the other hand, label 0 is given red color with probability 0.9 and green with probability 0.1. This introduces
Adv. Aug.
Rud. Aug.
100
100
SQ
Cinstsanry
L] Cansisanny
50
50
L2 Consistency
DAIB-LI
IRM
FRM
0
10" 10
10a
10" 10
10a
Figure 6: Test accuracy vs À on Colored MNIST for Adversarial Color aug. and Random Color aug. a high degree of spurious correlation between color and the label. Thus, ERM is expected to significantly overfit to color for predicting the label.
At test time, the correlation with color is reversed for digits. Hence, vanilla ERM is expected to perform worse than 50% coin flip at test time. We explore two data augmentation schemes in this experiment. For the Adversarial Augmentation (Adv. Aug.) setup, the augmented images will have their color flipped (from red to green or vice versa) with probability 0.1. For the Random Augmentation (Rnd. Aug.) setup, the augmented images are colored uniformly at random. Detailed description of the setup and additional experiments can be found in Appendix D and Appendix [ respectively.
Figure 0 suggests that DAIR-SQ and KL consistency regularization achieve ~72% test accuracy using both augmentation schemes, outperforming the state-of-the-art 68% test accuracy reported by IRM (Arjovsky et al. 2019). and almost reaching the 75% cap. We also compare to more recent baselines in Table 15 (Appendix 1. We note however that this comparison may be unfair because IRM docs not have access to any pairing information between the original and the augmented samples. As we observe in the next section, such information is readily available in several real-world benchmarks and DAIR-SQ can exploit it to achieve new state-of-tho-art results. We also notice that neither variant of DA-ERM achieves test performance better than 50% coin flip in this experiment, while Adversarial Angmentation seems to fare better than Random Augmentation.
9

---

Published in Transactions on Machine Learning Research (11/2022)
0% Noise
20% Noise
40% Noise
80% Noise
100
100
100
100
DAIR SQ
KL. Consistency
L1 Cousistancy
50
50
50
50
L2 Conmisteucy
DAIR-LI
DA-ERM
ERM
10-
100
10a
10
100
10a
10-3
100
10
10-3
100
10a
Figure Test accuracy as function of À for different label noise levels for Weak Rotation augmentation.
0% Noise
20% Noise
40% Noise
80% Noise
100
100
100
100
DAIR-SQ
KL Consistencyy
Consisteacy
50
50
50
50
Cousistency
DAIR-LI
DA-ERM
ERM
100
10a
10-3
100
10a
10-3
100
10a
10-3
100
10a
10-
Figure 8: Test accuracy as function for different label noise levels for Strong Rotation augmentation.
4.2 Rotated MNIST
Rotated MNIST (Ghifary et al. 2015) is a dataset where MNIST digits are rotated. We work with two different sets of degrees of rotation for Rotated MNIST. The first one is Weak Rotation where the digits are rotated uniformly at random [0, ) radians. In Strong Rotation the digits are rotated uniformly at random [0, 2T) radians. To evaluate the robustness of the methods, we further add label noise at training time where the label is replaced with a digit chosen from {0....,9) uniformly at random with a certain probability. No label noise is added test time. Detailed setup is in Appendix(] D Model architecture and training parameters are detailed Appendix D
In the first experiment, Rotation. Thus, some test time rotations have not been observed al training lime. Figure 7 shows the test performance of all algorithms (averaged over three runs) as a function of X. As can be seen, ERM (with no data augmentation) does not gencralize to rotated test images and performs poorly. DA-ERM offers significant performance improvement over ERM. When À is very small all variants of consistency regularization are virtually the same as DA-ERM. DAIR-SQ, and KT. regularizer outperform other regularizers and are the only two variants that offer improvement over DA-ERM as À increases. As the label noise level becomes larger, DAIR-SQ is more robust than KL regularizer and offers the best performance.
Weak Rotation for data augmentation while test time use Strong
Besides DAIR-SQ and KL regularizer, it is noteworthy that the other consistency regularization variants did not offer improvement over DA-ERM and they converged to poor local minima with 10% test accuracy (random) for large A. We were not able to remedy this by tuning of their step size. See Section 2.2.1 for further justification this phenomenon. We also observe that. the performance of both DAIR-SQ and KI. regularizer achieves a sweet spot for some finite X, i.c., the performance starts to drop for large values of X. This not theoretically expected and can be attributed to the practical issues with solving the consistency regularization problem. We further investigate this phenomenon in Appendix and provide some explanations.
The sctup for the second experiment is the same as the first one, except we also use Strong Rotation in training for augmentation, so there is no lest-time distribution shift for DA-ERM. As can be seen in Figure data augmentation achieves very good performance in this case and none of the DAIR regularizers offer any improvement beyond data augmentation. We suspect this be true general; if the data augmentation is well-devised and optimized the resulting model could become invariant the desired transformations test time. Additional experiments consistency metric can be found in Appendix F.1
10

---

Published in Transactions on Machine Learning Research (11/2022)
4.3 Invariant Visual Question Answering
In this experiment, we focus on the IV-VQA dataset which contains one or more edited images constructed from orignal images by removing an object which is irrelevant to answering the question. A robust model should make consistently predict the same answer for the original and corresponding edited images.
Similar to the setup CV-VQA, We use the SAAA (Kazemi & Elqursh 2017 model and compare performance against baselines using accuracy and consistency metrics.
Algorithm
VQA v2 val f
CM f
Predictions Flipped 4 pos + neg |
neg +
pas |
neg +
neg
FRM (Kzzemi & Elqursh. 2017
64.18
0.9456
8.64
3.45
3.00
2.2
DA-ERTM [Agarwal et al. 2020
64.66
0.9543
7.47
2.92
2.73
2.73
KL Feature Consistency
64.37
0.9582
7.07
2.73
2.50
1.84
DAIR
64.75
0.9606
6.33
2.54
2.22
1.57
Table 4: Accuracy and Consistency metrics on VQA v2 val & IV-VQA test
The results are reported in Table In addition to our CM score, we borrow the consistency metrics from |Agarwal et al. 2020 that measure three lypes of flips namely, pos + neg, neg + pos and neg + neg. A pos -> neg flip indicates that the answer predicted with the original image was correct but was wrong with the corresponding edited image. A neg - neg flip indicates that the answer changes from original to edited image but is wrong for both. DAIR achieves a higher accuracy as compared all baselines across both datasets, while improving under CM score and the iPredictions flipped' metric which the sum of the three types of flips. While applying DAIR to this task, observe trade-off between the VQA accuracy and the consistency metrics controlled by the À parameter. By increasing X, the consistency score rises to high 98%, albeit sacrificing the VQA accuracy by 6-7%. For moderate values of X, DAIR maintains classification accuracy of the model while enforcing consistency across variations in the visual space. See Appendix for more details.
4.4 Training robust deep networks against adversarial attacks
Neural networks have been widely used in various applications, espe- cially in computer vision. However, neural networks are vulner- able to adversarial attacks, such Fast Gradient Sign Method (FGSM) [Goodfellow et al.) 2014 and Projected Gradient Descent (PGD) (Madry et al.) 2018)), where small adversarial perturbations in the input are designed to significantly alter the output prediction.
55
PGD
50
DAIR-SQ TRADES ATTA
TRADES
45
APART
PGD Training
40
In this section, we consider training robust neural networks against adversarial attacks and compare with state-of-the-arl baseline models which are specifically designed for this task. We evaluate the perfor- mance of each algorithm against PGD attacks as well as the clean (no attack-free) accuracy. In our approach, the augmented examples E can be generated by certain strong attack, such as Projected Gradient Descent (PGD) CW (Carlini & We Wagner 2017). We conduct our experiments on CIFAR-10 dataset and compare our approach with several other state-of-the-art baselines.
75
80
85
90
Clean
Figure 9: PGD20//lean Acc. trade-off by sweeping X.
The performance of our algorithm against the Fast Gradient Sign Method (FGSM) and variants of PGD, is summarized in Table 5 which shows that our results are competitive with the baselines. We report the performance of DAIR in Table 5 based the configurations that give the best Clean accuracy followed by the best Robust accuracy against PGD20 in parenthesis afterward. The trade-off curve shown in Figure g suggests that by sweeping the value of X, DAIR can achieve a better clean accuracy but a slightly lower PGD20 accuracy, and dominates most of the baseline, while it achieves a similar performance with TRADES. Note that the formulation in TRADES is equivalent to consistency regularization with KL divergence between the logits of the original and adversarial images. As opposed to our sctup, the regularizer term in TRADES is also used in solving the maximization problem generate adversarial images, whereas we only use the original loss for generating the adversarial examples.
11

---

Published in Transactions on Machine Learning Research (11/2022)
Algoritim
Cleun (%)
FGSM (%)
PGD20 (%)
CM (%)
PGD Training (aadry et al. 2018
82.89
55.38
48.40
APART (Ti et al. 2020
82.45
55.33
48.95
60.05
DAIR (A
6]
83,04
57.51
50.68
62.66
4 TRADES ATTA Zheng et al.
2020
78.98
55.58
52.30
60.56
TRADES Zhang al. 2019
81.67
57.78
52.90
63.14
DAIR (A 16.7)
81.29
58.58
53.37
67.51
Table 5: C-FAR-10 test accuracies under no attack (clean); FGSM, and PGD20 attacks, and accuracy consistency
metric between original and PGD20 attack.
We also report the accuracy consistency metric (CM) in this experiment in Table 5
CM captures the
consistency of accuracy on PGD20 attack compared to clean examples. We observe that DAIR outperforms all baselines, which is in line with its best generalization to different attacks.
Lastly, we compare DAIR with more recent baselines (namely Tack et al. (2021) on a different robust classification against adversarial attacks setup, where we observe that DAIR offers competitive performance improvements compared to the state-of-the-ar baselines (sce Appendix K
4.5 Robust regression: simultaneous domain shift and label noise
In this experiment, we consider a regression task to minimize the root mean square error (RMSE) of the predicted values samples from the Drug Discovery dataset. The task is to predict the bioactivities given a set of chemical compounds (binary features). We follow the setup Li et al. (2021) to introduce random noise to corrupt the targets. Furthermore, similar to Colored MNIST, we add a spurious binary feature to the original setup. At training time, the spurious feature is set to if the larget is above a threshold (the median of all the targets in the training samples), and O otherwise. At test time, this condition is reversed lcading to poor generalization. We compare using ERM, DA-ERM and DAIR formulations under 0%, 20% and 40% noise levels on three baselines: L2 loss, Huber loss, and negatively tilted loss (Li et al. 2021], which is called tilted empirical risk minimization (TERM) and is designed for robust regression. For each of these baselines, perform data augmentation by randomly assigning the spurious feature as 0 or 1 with equal probability. Finally, we apply the DAIR regularizer to each of these loss functions with X = 10.
Test RMSE (Drug Discovery datuset)
Algorithus
0% Noise
20% Noise
40% Noise
Clean
ERM
DA- E--ERM
DAIR
ERM
DA-ERM
DAIR
ERM
DA-ERM
DAIR
ERM
Co loss
1.97 (0.00)
1.36 (9.00.
1.23 (0.00)
4.33 10.
2.52 m0s
2.04 (5.00)
5.30 (uos:
3.17 (0.0r)
2.99 o..0
1.23 (3.00)
Huber
Huther 1964
1.27 (a.00)
1.24 po
2.93 mo)
1.50 moo
1.39 mm
4.40 cwor:
2.18 (noa))
1.70 oxo))
2000
TERM
2021
1.74 (0.00)
1.26 (0.00)
1.25 (0.0))
1.87 o00))
1.27 (0:00)
1.27 (0:0)
2:01 c0.00:
1.33
1.31 (1.01)
1.23 (8.00)
Table Test RMSE for varying degrees of label noise for ERM, DA-ERM, and DAIR using different losses.
The results of this experiment are reported in Table G In the last column of the table we report results on the clean dataset without any spurious features for comparison purposes. As can be seen, without data augmentation all methods fall prey to spurious features and perform poorly, especially as the noise level is increased. It is noteworthy that while TERM is not designed for domain shift, it slightly outperforms the other baselines in the presence of spurious features showing that TERM has some inherent robustness to the domain shift. By adopting data augmentation, testing error decreases but is still quite large as compared to the Clean ERM setup for high values of noise. Notably, DAIR is able to reduce the testing error across all objectives and noise levels with the gap between DAIR and other approaches increasing with the degree of noise. For the noiseless setup, DAIR is able to almost recover the Clean ERM accuracy for all three objectives. The gains achieved with DAIR are prominent for L2 and Huber, but marginal for TERM. Finally, DAIR combined with TERM can simultaneously handle domain shift and noisy labels as can be seen in this table.
12

---

Published in Transactions on Machine Learning Research (11/2022)
4.6 ImageNet-9 9 background challenge
Deep learning models have outperformed human-level performance in many applications of which the most. prominent is image classification. However, these models are extremely vulnerable to overfitting to spurious correlations such as background features. ImageNet-9 Background Challenge (Xiao et al., 2020 was proposed to test the background robustness of image classification models. In this challenge, seven variations of images such as background/forre /forerruudd removal, are provided to measure the extent to which models rely on the background. For example, variations Ouly-BG-B and Ouly-BG-T remove the backgrounds and therefore the test accuracy is expected to be low for a model which does not learn spurious background features. See (Xiao et al. 2020] Figure 1) for example images of each variation. X et al. (2020) choose to train model on Mixed-Rand variation, which is the most powerful and comprchensive augmentation scheme, and demonstrated that the resulting model is more robust. We choose the Mixed-Rand variation our augmentation scheme and compare the test accuracy on all seven variations of the models trained by ERM, Xiao et al. 2020 DA-ERM, DAIR and KL.
Table 7 summarizes the results. We see that DAIR oulperforms ERM and DA-ERM by a large margin and is similarly competitive as KL feature consistency regularization (see the third column). In particular, DAIR outperforms DA-ERM on all metrics for which higher test accuracy is more desirable. DAIR improves performance on the variations which include domain shift, such as Mixed-Same, Mixed-Next and Only-FG. In particular, it also helps Original and the Mixed-Rand variations which are seen during training as well. In this experiment, similar to Section 3.1 DAIR not only enhances out-of-domain generalizability wawlliizaabiiityrrrrrraer cullity/robustness /robustness but also gives the best the in-distribution performance (original). The detailed training setup can be found in Appendix
Original f Mixed-Rand 1 Mixed-Same t
Mixed-Next 1
Only-BG-B J
Only-BG-T
No-FG
Only-FG
ERM (Original)
69.43
38.57
59.63
34.27
25.83
31.06
37.04
44.00
Xian et al. 2020
49.41
51.98
51 31.85
50.48
13.41
13.80
20.05
52.22
DA-ERM
64.02
58.05
58.81
48.17
16.91
22.27
28.44
56.62
Feature Consistency
70.84
65.36
68.79
64.07
13.23
20.81
26.77
67.48
DAIR
72.91
63.48
69.16
62.02
17.19
24.02
31.88
66.15
Table 7: ImageNet-9 Backgrounds Challenge test accuracy on different shifted test sets. We use Mixed-Rand as augmentation during training for DA-ERM, DATR, and KT. feature consistency. Arrows next to the heading of each column indicate the desired direction of the metric. For example, the accuracy on Original images should be as high as possible but the accuracy on Only-BG-B should be as low possible.
5 Conclusion
In this paper, we proposed a simple yet effective consistency regularization technique, called data augmented loss invariant regularization (DAIR). DAIR is applicable when data augmentation is used to promote performance invariance across pairs of original and augmented samples, and it enforces the loss to be similar on the original and the augmented samples. We provided motivation and justification for DAIR, and particularly showed that it can recover the optimal solution in a certain regression task where data augmentation alone insufficient. We also compared DAIR with other consistency regularizers and showed that existing consistency regularization techniques cannot handle covariant data augmentation. On the other crovaian/covaariant
hand, DAIR is broadly applicable to tasks involving invariant /covariant data augmentation. We empirically evaluated DAIR in five real-world machine learning lasks, namely (ask-oriented dialog modeling, visual question answering, robust regression, robust classification against adversarial attacks, and ImageNet-9 background challenge. Our experiments confirmed a major benefit of DAIR as some other consistency regularizers cannot be applied broadly. Empirically, DAIR set new state-of-the-ar results in dialog state tracking and VQA benchmarks which involved covariant data augmentation, and provided competitive results in all other benchmarks. This demonstrated the effectiveness of DAIR despite its simplicity.
Several problems remain open for future research: An in-depth theoretical understanding of the properties of DAIR that lead to its superior empirical performance on broad applications is an important open question.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Further, automated hyperparameter tuning techniques for the strength of the regularizer is another avenue for future research. Finally, discussion on limitations and broader impact appears in Section 6
6 Limitations & Broaddr Impact
Firstly, while we demonstrated the success of DAIR when the pairing information between original and augmented training samples is known, the applicability of DAIR remains limited in only sctups where such pairing information is available. We also remark that DAIR incurs double the computational cost compared with algorithms which only consider one sample for backpropagation (e.g., only an augmented examples) rather than the pair.
Secondly, applying DAIR arbitrary tasks requires domain knowledge about the nature of the desired invariances to be promoted which is expressed by choosing appropriate augmented samples. In particular, we did not address devising good data augmentation procedures, but rather argued that if data augmentation is already employed (i.e. DA-ERM is used), DAIR can lead to remarkable gains (almost) with marginally added cost via further regularization of the losses. It remains to examine whether DAIR could be used as a component for solving domain gencralization for arbitrary domain shifts where data augmentation pairing cannot be performed a Auraigh/forwart fashion.
Thirdly, we demonstrated the effectiveness of DAIR on a variety of supervised tasks involving multimodal, generative and regression models. However, the applicability of DAIR to semi-supervised or self-supervised learning remains to be seen.
Finally, while we showed that DAIR boosts existing performance metries, such as accuracy, the interplay of DAIR with other important socially consoquential metrics, such as group fairness and privacy, was not explored in this paper. It remains to be seen whether DAIR may have any positive or negative consequences on these other Responsible AT metrics.
References
Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards causal VQA: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedliggs of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXio:1907.0280 2019.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. Advances in neural information processing systems, 27:3365 3373, 2014.
Pawel Budzianowski, Tsung- Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. Multiwoz a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling, arXiv preprint arXie:1810.0027 2018
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks, 2017.
Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-traie self-training avoids using spurious features under domain shift. arXiv preprint XrXiv:2066. 10032, 2020.
Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Ram Sadagopan, Shahin Shayandeh, Asli Celikyilmaz, Jonathan May, and Ahmad Beirami. Know thy strengths: Comprehensive dialogue state tracking diagnostics. EMNLP Findings, 2022.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 18X5.0880000900 2018.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. Xio preprint orXio:170s 708.04452 04552, 2017.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Logan Engstrom, Andrew Ilyas, and Anish Athalye. Evaluating and understanding the robustness of adversarial logit pairing. arXiv preprint (tt:1801 807.10277, 2018.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani- Tur. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXie preprint arXiv:1907.01669 2019.
Yaroslav Ganin, Evgeniya. Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, (7(1):2096-208) 2016.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Aukur Taly, Ed H Chi, and Alex Beutel. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 219-226, 226, 2019.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551 2559, 2015.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ar preprinl Xrx: 141.6572, 2014.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Flevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. ICLR, 2020.
Ting Han, Ximing Liu, Ryuichi Takanobu, Yixin Lian, Chongxuan Huang, Dazhen Wan, Wei Peng, and Minlie Huang. Multiwoz 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation, 2021.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks, 2016.
Dan Hendrycks, Steven Basart, Norman MU, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340 8349, 2021.
Ehsan Hosseini-Aso Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue. NewrIPS, 2020.
Peter J. Huber. Robust estimation of location parameter. Annals of Mathematical Statistics, 35:492 518, 1964.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv:1803.0634 2018.
arXio preprint
V. Kazemi and A. Elqursh. Show, ask, attend, and answer: A strong baseline for visual question answering. ArXio, abs/1704.03162, 2017.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (REx). In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5815-5826. PMLR, 18-24 Jul 2021. URL https::/proceedings. mlr. wttpppsssseeeeeeeppppppp wwtl://////ee21...... html
15

---

Published in Transactions on Machine Learning Research (11/2022)
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.0220 022222, 2016.
Yunwen Lei, Ting Hu, Guiying Li, and Ke Tang, Stochastic gradient descent for nonconvex learning without bounded gradient assumptions. IEEE transactions on neural networks and learning systems, 31 31(1)): 4394 4400, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Tuke Zettlemoyer. Bart: Denoising sequence-to-ssqquenc pre-training for natural language generation, translation, and comprehension. arXiv preprint wrXi:1110013446: 2019.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 540005409,, 2018b.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. ICLR, 2021.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018c.
Zichao Li, Liyuan Liu, Chengyu Dong, and Jingbo Shang. Overfitting or underfittin?? understand robustness drop in adversarial training, 2020.
Sungbin Lim, Todoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32:6665 6675, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial atlacks. In International Conference on Learning Representations, 2018.
H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. Parlai: A dialog research software platform. arXiv preprint arxiv:1705 (tt::775.004 06.06647,6 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regular- ization method for supervised and semi-supervised learning. IEEE transactions pattern analysis and machine intelligence, 41 1879 2018.
Yurii Nesterov. Introductory lectures on concer optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406-1415, 2019.
Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. Annotation inconsistency and entily bias in MultiWOZ. The 22nd Annual Meeling of the Special Interest Group on Discourse and Dialogue (SIGDIAL), July 2021.
Abhinay Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The -chemd dialogue dataset, 2020.
Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lee, and Giedrius Burachas. Sunny and dark outside?! improving answer consistency in vqa through entailed question generation. In FMNLP/IJCNTPP, 2019.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Alexander Robey, Luiz FO Chamon, George J Pappas, and Hamed Hassani. Probabilistically robust learning: Balancing average-and worst-case performance. arXiv preprinl arXin:2202.00 011000000113, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Perey Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. Xrx preprinl
arXis:1911.08791, 2019.
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-consistency for robust visual question answering. In 2019 Conference on Computer Vision and Pattern Recognation (CVPR), 2019.
Samarth Sinha and Adji B Dieng. Consistency regularization for variational auto-encoders. arXiv preprint arXiv:2105.14559, 2021.
Kihyuk Sohn, David Berthelot, Chun-Tiang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprinl wxXic:2001.06666 2020.
Baochen Sun and Kate Sacnko. Deep CORAL: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 443 448. Springer, 2016.
Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Ju Hwang, and Jinwoo Shin. Consistency regularization for adversarial robustness. arXiv preprint arXiv:2133.0440 2021.
Christopher Tensmoyer and Tony Martinez. Improving invariance and equivariance propertics of convolutional neural networks. 2016.
Riccardo Volpi, Hongscok Namkoong, Ozan Soner, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. arXiv preprint 1KKi:1805.12113 2018.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Beruhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. arXiv preprint Xiv:22 0et:2106604661 04699, 2021.
Yisen Wang, Difan Zou, Jinleng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Imppoving adversarial ro- bustness requires revisiting misclassified examples. In International Conference on Learnóng Representations, 2019.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. Transferable multi-domain state generator for task-oriented dialogue systems, 2019.
Kai Xiao, Logan Engstrom, Andrew Tyas, and Alcksander Madry. Noise or signal: The role of image backgrounds in object recognition, 2020.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoe V Le. telf-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023 033-6032,22 2019.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. Multiwoz 2.2 A dialogue dataset with additional annotation corrections and state tracking baselines, 2020.
Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. Quantifying and improving transfcrability domain generalization. Advances in Neural Information Processing Systems, 34, 2021.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, 2019.
17

---

Published in Transactions on Machine Learning Research (11/2022)
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. orXiv preprinl Xr::::1710.0000 12, 2017.
Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adversarial training with transferable adversarial examples. In Proceedings of the IEFF Conference Computer Vision and Pottern Recognition (CVPR), 2020.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang, Deep domain--dveesarrie image generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. -3002,, 2020.
18

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
In this appendix, we provide additional linear regression example with thcoretical analysis (Appendix A and its extension (Appendix B include proofs of the analyses from Section 2 (Appendix C details on MNIST
experiments (Appendices D to practical considerations when DATR used in training (A Appendix ; the im-
pact of partial augmentation (Appendix G); training details on experiments Sections and 4 (Appendices
Lb. We provide table of contents below for casier navigation.
Contents
A
Additional linear regression example with theoretical analysis
20
B
Multi-dimensional extension of linear regression example (Appendix A
22
Proofs
24
C.1
Proof of relation between DAIR-SQ and DAIR-L1
24
C.2
Proof of dependence of DAIR-SQ on À (Proposition 3
24
C,3
Proof of convergence of DAIR-SQ (Theorem
25
D
Model architecture and training parameters for MNIST experiments
28
Colored MNIST
30
22
Rotated MNIST
30
Additional results on Colored MNIST Rotated MNIST
31
Colored MNIST
Rotated MNIST
Practical considerations when DAIR used training
33
Explanations a practical sweet spot for the performance of DAIR-SQ as function
Additional evidence on growing cost of training with the regularization strength
33
Compare DAIR-SQ and DAIR-L1 on avoidance of spurious local min
34
The impact of partial augmentation
35
H Details on neural task-oriented dialog modeling
36
Setup and additional results for visual question answering
Additional details robust regression
38
K Details on training robust neural networks
39
K.1 Setups for the main results Section
39
K.2 Additional results with new baselines
39
Details on training ImageNet-9
40
19

---

Published in Transactions on Machine Learning Research (11/2022)
A Additional linear regression example with theoretical analysis
In this section, we answer the question "What does DAIR offer beyond DA-ERM?" both theorotically and empirically through a simple toy example. In this example, we remonstrate that DAIR can fundamentally outperform DA-ERM, even in the limit of infinite training samples (no overfitting due to finite samples). Consider linear regression problem where at the training time the input is Xtrain = (xt
Opiinna
1
FRM DA-ERM -DAERMM"
DAIR
y) and the label y, i.C., Strain = (Xerain V). Here, I ~ NN(0, o2), and y x + e, where s is independent of x and ε ~ oN), In this example, the target is explicitly provided as a spurious feature to the learner at the training
-1
-2
-1
1
2
time. At test time, the spurious feature is absent, i.e., Xtest = (x. 8 = 0).
Figure 10: The plot of the optimal,
Clearly, in this toy example, the optimal regressor is w' = (wi, w²) (1,0)T. However, absent the knowledge of the spurious feature vanilla ERM will learn WERM R (0, completely overfitting the spurious feature. We assume that the learner has access to data augmentation
ERM, DA-ERM and DAIR-SQ (A 100) regressons for the toy example of Appendix A
module that generates _ = A(z; a, ) = (Xaug, y), such that Xang = (2, 8 ay + n) where n ~ ~N(00, The augmented data will encourage the learned model to become invariant to the spurious feature. In Figure Io we perform simulations with a = 0.5, o² = 1, of 0.25, of 0.1 and plot four lincar regressors associated with the slope of their respective w1. We ignore W2 as the second spurious feature absent at test time and hence w2 does not impact test. performance. The optimal regressor is shown the blue line, with a slope of 1. ERM (red line) completely fails due to the overfitting to the spurious feature. DA-ERM (orange line) significantly improves over ERM but still is far from optimal performance. DA-ERM" (purple line) which is solely trained on augmented examples (ignoring the original examples) slightly outperfors DA-ERM but still significantly overfits to the spurious feature. DAIR-SQ (green line) almost recovers the optimal solution. This is not a coincidence. We prove that DAIR-SQ is optimal for a class of linear regression problems, while DA-ERM does not approach optimal performance even the limit of infinite samples. Here we state the rigorous statement, followed by proof.
Proposition 5. Consider a linear regression problem with training point Ztrain = (Xtrainn v) where Xtrain ( y); denotes label and 8 denotes the spurious feature. Here, x ~ -N, and y = ===+ E, where e is independent of x and E ~ NO,,02) Assume the learner has access to a data augmentation module which generales Zong- = (Xaug: y) where Xaug = (Ir 8 = ay + n). Here n ~ -N(0, oh); and a E R. At lest lime, the spurious feature is absent, i.e., Xtest = (z, = 0). Both DAIR-SQ and DA DA-ERM are applied to solve this problem: DAIR-SQ achieves optimal test error number of samples grows and X + co. On the other hand, DA-ERM cannol generully recover optimal performance even in the limit of infinile training dala.
Proof. First let us present the DA-ERM solution:
SDA-ERM (w) =E [[w] + way - y)² + (wia + wrsaa + m) y)
(1)
=E Iuft² + (w2 - 1)²y² + 2wn (w2 Jayy
E [wf=² + (wza- 1)² 0² + w3n?
+ E [2w1 (w2a 1) 1 + 2w w2an + 2wzz(wza 1)ym]
(2)
=wio² + (w² 10² + o²)) 2wn (w2 1)02
+ + + (wza- 1) 2 o.) + wāO
+ +w (wza - 1)0²
(3)
(wi + W22 1) + (wy - 1)²0²
(w) + Wya 1) + (wya 1)²²² 2 + wžO
(4)
Hence, the solution of w EAA = arg min, fDA-ERM(()) is given by
2wi (1 a)wi 2 0,
(wi wi 1)02 + (wi 1)o2 + a(wi + wwa 1)o2 + a(wja 1)o2 + wi O.
(5)
20

---

Published in Transactions on Machine Learning Research (11/2022)
Subsequently,
o'(o2+0?) 20(0, +o a=(02 + 2202) 200 ta 2(07+
EDA-ERM
(6)
a 2
"DAI arg min (DAIR(U))
arg min [wi + w2y y)² + (wix + w2(aa + n) y)²l
w
+ A(1ur + way yl - | + w2laay + n) - y
When Ả 0o, we have "DAIR.2 and hence:
DAIR
We then evaluate the testing loss assuming the spurious feature is absent, i.e., Xtest (xr, 0).
( AAIR = EE (wbAI Xtest - y)2²
= EE (x - (x + =))
o
(DA-ERM(XIest. WDA-ERM) =E (WDA-ERM Xtest - y)²
(a"(02 + o2) - 20 + 0 + 0 +0 20²
27
=E EE'
a²(o² (~ + 20 200
2 +
x
(x +e)
(a + 10
= oe + (a2 + +0o?) 2002 + 02 2(02 o2))2
2 DAIR.
completing the proof.
One can show that simple data independent regularization methods (e.g. weight decay) cannol help close the gap between the performance DA-ERM and DAIR (see Proposition 6). While the toy example presented an extreme case with a spurious feature equal to the output, we prove theoretically that the same conclusion holds as long a subset of features have different correlation patterns with the output at training and test time (see Proposition 7 for the general multi-variate linear regression setup). Note that in this toy example when on + co, DA-ERM could also recover w*. One can interpret that as on 0o, the augmenter becomes stronger and forces W2 to vanish. On the other hand, DAIR recovers w" with a much weaker augmenter. This is crucial since in real-world applications, designing strong augmentation schemes requires careful design. See the example of Weak Rotation setup in Section 4.2 for details.
Proposition 6. Consider the case in which weight decay regularizer [w + w/) added to the DA-ERM, the resulting solution is the following:
(a22a 2n(o aP +2 +x:) 200
220 +
W DA- D-A-EEM--NR
(a ( 4200
a- +23)
-
21

---

Published in Transactions on Machine Learning Research (11/2022)
Proof of Proposition 0 The proof follows the same idea of Proposition and therefore it is omitted here. ]
Proposition shows that even using the weight decay regularizer would not close the gap between the performance of DA-ERM and DAIR. In other words WDA-ERM-WD o w* = -1,0) unless oî - Do and 7 = 0.
B Multi-dimensional extension of linear regression example (Appendix A
Consider multt-diimensional extension of the example in Appendix A During training, the input is Xtrain [x: Strain] E IR+ and = 1 E, where ~ (0, R NO, o.), Strain = yerrain + Mtrain Utrain E R and Ahrain ~ ~O, a2 Raram E TR. Similar to the toy example, we introduce spurious feature Strain to mislead the model. ERM based approach should overfit to use the information in Strain if o nasage is small. Suppose the learner also has access to the augmented datapoints of the form Xaug = [a Sangl E RR++ where Saug = yuaug + Raug: Mang E R$, and naug (O o P E The augmented data points will encourage the model to be invariant to the spurious feature during training. The testing data is Xtest = [ar 0] E IRd++ Again, the optimal w is [1 O] and we compare the performances of DA-ERM and DAIR. One can think of this is a simplified setup that is aimed at emulating a case with some features, e.g., background, bearing no impact on the labels, whereas they may be (highly) corrclated with the label during the training. One may guess the results of comparison as this is the extension of the toy example in the main body of the paper: DAIR mostly works better than DA-ERM. We introduce the formal proposition below.
Proposition 7. Consider the linear least squares regression problem for predicting the target variable in the problem described above. Assume that the learner has access to a dala augmentation module that perturbs the spurious feature, as described above. Consider the population level loss (L.e. number of samples is infinity)- Then, for any value of on " Utrain and Naugs DAIR-SQ achieves optimal test error as - oc. On the other hand,
canmot obtain optimal performance (other than for only certain corner cases such 02 DC and/ov /o o2 Baria ).).
Proof. Assuming the linear least squares regression fit, the objective function of DA-ERM can be written
2E [R)) = EE' [w T Xtrain y)² + (w Xaug - T
-
E [w]] x + w Strain 4)²² + (wT w Saug r)
EF [w]] x + w (Vtrain 1/ + Ttraii) - y) + (w) x + Wiy (Waugl) + +augg) - y)
= E [(w] x + wh] (Vtrata (1) T x + E) + -ersin)) 1 x - =]
+ E [(we] x: + wi (Maug (1 TIT e) Maugg) 1 x )]
= (x [w] + 11 W2 - 1) + s(w] Utrain - 1) + 72 T W2
+ E [xx (wi + 1u W2 - 1) + =(w2 Maug
1) + ang Wy
T
=o [[w + Iu W22 1 ( Uerain
1)2 + Peratn |
+ o2 [| + 1u W2 1 12 + a² [ Uaug
1)2 + o nann
122122
22

---

Published in Transactions on Machine Learning Research (11/2022)
where [w] w2 with wi E R and W2 R Expanding the norms will result in
2R SEAAAA.EMMMM of(w" Tw 2w To 21 w Uerain I
I Turaiin W
21 1 train
[1
+ + Üerain Vurai - 2w Ütrain + 1] + +o masin w Tw
" TTw 22 "unng w- 21 w + w Vaugg 1u w 21 18 1mang w 1'
=f "ang Haug"aug" 2w Hang 1] 0 name w Tww
w [o + Verainn 1 + 00 train + +o Berain I
I 1'1 + o Otrain Derain + on Nw
(-20 Brrain
Utrain
1 Berain)
w T o Maug 1 + 0²² 12 + o Hang 1 1 + oz Uraug H + o Nw
(-20 1 20 Mang i 20 dang) Tw
2 2 1 1 +0
0 0
0 0
I
0 0
where Uerain
00
train
T
Dang
10
" ang
T
0
1
10T 1J
and
i= [
By optimality condition, have:
QWiDAERM
(7)
where
Q a (Vrran 1 + 1uraan + Lang1 1 + 1uaug + Srran 1 Torrat + Hang I (usuus)
2021 I (ousrim onnng oe (Utrain Oerain + thing Uaug)
o..--2 + Etrain Vang 1 1) + o. Utrain Wang).
1
One can check that WDA-ERM *
for generic choice of Vtrains Maug. 02 02 o2 o This is because
0
we have more free variables than the number of equations equation 7 Thus, "DA-ERM cannot recover the optimal regressor for the generic choice of parameters. More specifically, only in certain corner cases A
would recover the optimal regressor
For example, when o2 Peraii +00, have Teraann
T and
b- O. Thus, in this corner case, the optimality condition equation is asymptotically satisfied. Pirnin
On the other hand, in the presence of DAIR regularizer, when À + oc, the loss function in equation DAIR remains finite if and only if R(E(2;w), ((2; ()) = O, almost everywhere. Equivalently, the objective in equa- tion DAIR remains finite (when X + oo) if and only
(g T w1 Irain w2)2 (y x wi sung W2)
for almost all realizations data, which implies W2 = 0. Thus, when + co, W - [W.. w DAIR with WIDDAIR O. In other words, the cocfficient of the spurious features vanishes as X oo and hence the
1
regression will recover the groundtruth regressor
0
23

---

Published in Transactions on Machine Learning Research (11/2022)
C Proofs
C.1 Proof of relation between DAIR-SQ and DAIR-L1 (Lemma !
Proof of Lemma 7
We proceed as follows:
R1,(2, Raq (2, 2,0) 2 minn(((2;0));(33000) v(E50)- v((z;))
We break it into two cases: if ((2;0) (zz;)):
R2, () Rag 0) ((;00) (2;0)) (V((E,)) -(zz)))))((())
{(Z;0) 0) (12; 0) - -(::0(())))))))))) 22
=-20(z; 0) 2V(EE0)
=2/((=:0) v((z))).
If (( {z(;0);
(2,12,2;00) -Rag (2, [(2;0) ((2:0) (V - v((z;)))
= =((;0) ((2 0) (x: +)
--2((2;0) +2((E0))
=2/=(2:))((/(((00) v((2;)).
If we combine the two cases, we have:
R2,(2,2;00) R2n 2;0) 2 ((;](((())))(((3)) V((E0)) v((2;))
C.2 Proof of dependence of DAIR-SQ on À (Proposition 3
Proof of Proposition
We start the proof with the objective value.
{(2 (5) + ((2;00)
((2; 0) + ((2:0)
E
S E,
XR ([(x)
(8)
2
2
((2208 03) + ((;
((2; 0) + ((2:0)
E.
XRa ((2;))
12_
(9)
2
2
((2; 0) + ((2)0)
[(2;
+ (();(( (2)01)
E,
(2)
E
(10)
2
2
((2; 0) + ((200)
R
(11)
2
[(2;; 0) + ((2:0)
E,
Rog (((2))))),
E.
(12)
2A
Note equation holds since 0% is the minimizer of the fDAIR. > (2, 2)6). equation and equation since R(.)) and ((.) are non-negative respectively.
hold
24

---

Published in Transactions on Machine Learning Research (11/2022)
C.3 Proof of convergence of DAIR-SQ (Theorem 4
We first present DAIR applied to training neural networks with Gradient Descent (GD) and followed by proof of Theorem !
Algorithm 1 Training Neural Networks with GD
Input: Number of steps T, Training sot S, Learning Rate 77, Initialized Parameter 8
for t 1, 2,... ...,, T do
Compute V E fDAIR, 2, Set @++1 = 0 - 1/ É (Z
4:
end for
Remark 8. Algorithm 1 shows DAIR applied to training neural networks with GD. Note Algorithm can also exlended to the stochastic selling, which the variant we used in the experiments of Sections
Proof of Theorem
The objective function of DAIR is the following:
(x, (x, (0000))++((xx,x)x) v: - Ve(X, v: 2
(Ve((x,8:)))
(f(x, ys + g( (V((x, v v:
Substituting logistic loss function, the above equation reduces
[xAIR(X,,, (log(1 + exp(((i(x y, 0))) + log(1 + 2x,
((()))(((1))
2
Mog(1 + exp(GI(x. 0))) Vlg(1 exp((2 y. 8)))
(00(0)=
where < = (-)()) (2()) (x,,8,0) -y0 x and = In order to obtain the convergence rate of gradient descent, we need to compute the Lipschitz constant of the gradient of fDAIR- To this end, we need to bound the Hessian of (f). Applying chain rule to this function, we obtain:
0g (0)+ dg vZ6(()=
1/X
y/X
= Vecco)))
VEg(C)
a0,
-
TxT
Where the last incquality is due to the fact that (0) ViCa(0)
0 since (1()) and (2(+) are lincar
functions in 0. Recall g({)= (Vlog(1 exp(Ci)) - V Tog(1 + exp(C2)) 2
which implies:
25

---

Published in Transactions on Machine Learning Research (11/2022)
expÇi (22 (exp + 1) log (exp + 1) exp / (exp + 1) 2oo (exp + 1)
(V(g(S))
2 (exp(( + 1)2 log² (exp<i 1)
exp (<i Ca)
(Veg(e)) 2 (exp Gi +) +
log (exp + log (exp C2 1)
(V(o()))
exp (Ci (2)
2 (exp Çi + 1) (exp 1) v log (exp<i+ log (exp C2
ex log (exp Ca + 1) V log (exp Ci + 1) + exp v log (exp + 1) + 2log (exp + 1)
(V(g(())
2 (2 + 1)a log (exp Ce + 1)
The Lipschitz constant the gradient is equal to the spectral norm of the Hessian. To bound that, we use the fact that the spectral norm bounded by the Frobenius norm and hence we need to bound each individual entry of Vgg(C). To this end, we will leverage the following inequality throughout our process:
< log x S 11
Vo O.
(13)
We now bound (Vzg()))11 Notice that, using equation
13
have:
0 ()xp<))²
(exp
Veli
S 1,
(exp (1 + 1)² log (exp<++1) (exp<1 + 1)²
Vesi +
(1- 1 1+expci
exp <1 log (exp <i + 1)
(exp
Vesi
50
Vesi +1 S
(exp (1 + 1)a log (exp S1 + +1)
(exp C1 1)²
(1
+ exp e
exp Ç1
and 0 - (exp Ci 1)2 S 1. Putting these pieces together, we obtain - -Voo (exp S2 + S (Vg(())) x
log (exp C2 1) + 3: which term implies
(Véglt)) = O(VD.Do)
Similarly, we can obtain (Vgg(C))22 O(VD,Dg). We now bound (Vāg(C))12 and (V))(())21- By equation 13 have:
expCi
exp
0 0
(1 ex+e exp()
(1 + exp (1) / I+expCi
exp<i
(1)² (1 exp(1) exp C1
+ expx(eexppG 1
Therefore, we have (Vz0()))12 S and (Vzg(C))2 S } Given the bounds of the four entries of V²9(C) above, we have
Vor(s))1a DDDD..))
26

---

Published in Transactions on Machine Learning Research (11/2022)
- 1/X
yx
Recall (0)
VEg(e)
and the boundedness assumption and x, finally
yxx
-xx
have:
-yxx
yx (S)(l)
(0)122 -
= =O(DiVD,Do)
yx
yx T
We now find [V)l()) using similar approach. Notice that
exp S1 (exp + 1)2
0
VOI(0)= = -yx
yx
(2h(c)
and Van(C)
yxX T
yxx
exp C2 (exp C2 + 1)2
Thus, [V2h(C)1l2 = O(1) and therefore [[ = O(D2). Recall that
(X,IR(x,3,311 = fDA-EEM((X y: + A (v [ v) vx) 2L(() + XRxg (0).
Thus, using the computed bounds R (0)]l2 ODD and [Vi((0)|22 O(D2), we have
X 0)1 x O((1 + D.DDDD)
Now that have shown that the Lipschitz constant of the gradient is bounded, by classic gradient descent results (Nesterov 2003 Theorem 1.2.4), we know that after T iterations of gradient descent with stepsize ( [+AV/DDD.)D we have
NofDAn,RAA()2=-0 (1LLA--DD
VT
which completes the proof.
27

---

Published in Transactions on Machine Learning Research (11/2022)
D Model architecture and training parameters for MNIST experiments
We use a Convolutional Neural Network (CNN) with three convolutional layers followed by two fully connected layers. The last layer output size for Colored MNIST experiments is set to 1, and 10 for the Rotated MNIST experiments. For training we follow a two stage schedule with a learning rate of 0.005 for the first. 20 epochs and a learning rate of 0.0005 for the next 20. We choose a batch size of 64 for all experiments. The architectural details and training parameters can be found in Table and Table
Layer Type
Shape
Convolution ReLU
x 4
Max Pooling
2 x 2
Convolution + ReLU
4 x 4 X 16
Max Pooling
2 x 2
Convolution + ReLU
x 4 x 96 64
Fully Connected + ReLU
Fully Connected
C
Table 8: Model Architecture, C for Colored MNIST and C = 10 for Rotated MNIST.
Parameter
Value
Learning Rate
0.005
0.0005
Epochs
First. 20
Second 20
Batch-size 64
Table Training parameter of MNIST experiments.
We apply the proposed loss function (DIRR) on the following two datasets: Colored MNIST and Rotated MNIST. We compare the performance of DAIR with plain data augmentation, and invariant risk minimization (IRM) as strong baseline. One crucial difference between work and IRM is is the motivation. IRM is designed to take two examples from two different environments and learn representations that are invariant. to the environment, e.g., in cases where we are aggregating multiple datasets. On the other hand, we are interested promoting invariance when we have a single dataset. As such, we artificially generate the second environment. IRM using data augmentation. For a given example 2, we design an augmenter A(-) and use it to generate additional samples that adhore to the invariance we have in mind. Hence, IRM will be appliod in the same way that examples from different environments are augmenting pairs.
Our Colored MNIST is an extension of the original Colored MNIST Arjovsky et al. 20199) The label is noisy function of both digit and color. The digit has a correlation of 0.75 with the label and a certain correlation with the label depending on the color scheme. Besides the two colors in the original dateset, we introduce fully random colored scheme to the dateset, which is the best augmenter one can think of. The three color schemes are detailed in Table [T]
Our Rotated MNIST is variant of the original Rotated MNIST Ghilary al. 2015 The original dataset contains images of digits rotated d degrees, where d D & {0, 15, 30, 45, 60, 75). Similarly, introduce the random degree scheme here to serve as the best possible augmenter. To further exploit the potential of the proposed algorithm, we make this dataset more difficult by introducing more challenging degree scheme; The rotation schemes are summarized in Table
Note all the augmented images are generated on the fly. Examples of images from some transformation schemes are shown in Figures| to
28

---

Published in Transactions on Machine Learning Research (11/2022)
Scheme
2
Color =
with p 0.8, 2 y
Red
C1
with p 0.2,
1
Green
with p 0.9, = y
Red
C2
with p 0.1, 2 = 1
Green
with p = 0.1, 2 = y
Red
C3
with p = 0.9, 2 =
Green
C4
2 2
Random
Table 10: Color schemes in Colored MNIST. Random color means that the value of each channel of the image is uniformly random chosen from to 255.
Scheme
Rotation
RI
0
R2
90º
09, 180
R4
90% 270
R5
[0 360%
R6
22.55" 67.5%), [202.5 247.5%)
Table 11: Rotation schemes in Rotated MNIST. (a, b] means that degrees are unformly random chosen between and
Setup Name
Train
Aug
Test
Adv. Aug.
C1
C2
1000
Rnd. Aug.
C1
C4
C3
100
Table 12: Training procedure of Colored MNIST.
Setup
Train
Aug
Test.
Strong Aug.
R.1
R5
R2
1
Weak Aug.
R4
R6
R3
10
Table 13: Training procedure Rotated MNIST
5
9
5
5
3
6
7
C
7
3
7
3
4
3
5
?
h
7
6
2
9
0
4
e
Figure 11: C2
Figure 12: C3
Figure 13: C4
Figure 14: R4
Figure 15: R5
Figure 16: R6
Sctup: We train model consisted of three convolutional layers and two fully connected layers with 20,000 examples. For each dataset we are defining several different schemes on how the dataset could be modified:
Table 10 (Colored MNTST) and Table 11 (Rotated MNIST). Then, define several selups. Fach setup
29

---

Published in Transactions on Machine Learning Research (11/2022)
is consisted of one original dataset, one augmentation dataset, and test dataset, each of which is selected among the defined schemes. These setups are provided in Table 12 (Colored MNTST) and Table 13 (Rotated MNIST). For each setup, we train the model with the following four algorithms and compare their performances: ERM, DA-ERM, DAIR and Invariant Risk Minimization (IRM). Each experiment is repeated for 10 times; the mean and the standard derivation are reported. The value of À are chosen base on the validation results. Detailed architectures and training paramcters can be found in Appendix D
D.1 Colored MNIST
We conduct two sets of experiments for this dataset: Adversarial Augmentation Setup (Table 12 follows the exact same color schemes from the original Colored MNIST Arjovsky et al. (2019). For Random
Augmentation Setup, we train the model with the strongest possible augmenter: uniformly random color. The entire procedure is summarized in Table 12
D.2 Rotated MNIST
We start with the strongest augmenter case. One may notice that there chance that the augmented images bear the same rolation degrees the testing set. To make the task more difficult, we will R6 the augmented test to test how the trained model generalize to entirely unseen domain. The training procedure is summarized in Table 13
30

---

Published in Transactions on Machine Learning Research (11/2022)
E
Additional results on Colored MNIST & Rotated MNIST
E.1
Colored MNIST
We show additional results on Colored MNIST and Rotated MNIST in Tables 14 and 15 Note that each algorithm has been tuned for best performance. As mentioned in Section 4.1 DAIR outperforms DA-ERM, ERM and other baseline models on classification accuracy. For accuracy consistency, we use the training scheme the original scheme and the testing scheme as the augmentation scheme. We further compare DAIR with IRM (Arjovsky et al., 2019), DRO (Sagawa et. al. 2019), and REx (Krueger al. 2021). In doing so, feed all original examples
environment and all augmented examples second environment to these baselines. While we can see that DAIR outperforms all basclines, we caution that the comparison may not be fair in that DAIR exploits pairing information between original and augmented samples, which is not used by the her baselines.
Algorithm
Accuracy
CM
ERM DA-ERM DAIR
32.70 +: 0.45
77.76 +: 1.01
40.91 0.45
84.60 + 0.60
72.58 + 0.11
99.39 + 0.11
TRM Arjovsky et al.
2019
66.90
DRO
Sagawa et al.
2019
37.40
REx
Krueger et al.
2021
68.70
Table 14: Accuracy and Accuracy Consistency Metric (CM) on Colored MNIST with Adversarial Augmentation.
Algorithm
Accuracy
CM
ERM
32.70 + 0.45
63.50 + 1.92
DA-ERM
29.61 + 0.80
88.15 0.18
DAIR
73.10 + 0.12
99.88 0.01
Table 15: Accuracy and Accuracy Consistency Metrie (CM) on Colored MNIST with Random Angmentation.
E.2
Rotated MNIST
We report the accuracy consistency on Rotated MNIST (weak augmentation) in Table 16 The original training scheme here is Scheme R4 (Table I1,, , i.e., 90% and 270" rotated images, and the augmentation scheme for training is R6 (weak rotation). At test time, we test with (no rotation) and we also use the augmentation scheme of 180 rotation to test the accuracy consistency metric. Note that neither the un-rotated or 180% rotated images have been observed at training time. Hence, the setup is difficult for ERM which struggles to generalize. As can be seen, since the digit 0 is "almost" circularly symmetric, ERM actually does decent job at classifying 0, however it significantly struggles with all other digits. We see that DAIR outperfomms ERM and DA-ERM by a large margin. We observe that digits 6 and 9 are challenging to get right (as one would expect for them to be difficult to tell apart). While we see 2 - 3% drop on the consistency for digits 6 and 9 (when rotating them by 180%), the drop is smaller than expected perhaps due to the fact that the neural network learns classify these digits based on features that are harder get for humans.
31

---

Published in Transactions on Machine Learning Research (11/2022)
ERM
DA-ERM
DAIR
Digit
Acc.
CM
Acc.
CM
Acc.
CM
0
86.19 = 01.48
94.95 = 01.53
95.61 + 00.66
98.43 00.21
98.44 00.07
99.31 00.15
1
00.15 + 00.08
11.11
82.79 + 03.38
98.54+000.3
96.09 + 00.71
+ 01.28
+
2
29.84 + 00.51
57.91 = 02.76
76.68 + 03.54
03.27
86.21 + 00.82
93.21 01.32
3
00.63 + 00.53
76.47 23.53
7 +: 02.60
01.26
86.60 + 02.24
94.26 00.36
01.97 = 00.90
23.38 13.49
51.09 03.30
79.67 = 01.26
92.42 00.41
05.53 = 00.32
39.91 = 04.59
1.02::0..0
84.68 + 03.71
83.26 = 02.51
95.11 01.46
00.66 = 00.37
51.79 = 25.13
67.43 #
83.41 05.74
84.79 = 01.17
92.78 = 01.71
16.67 = 02.75
18.28 06.65
56.29 07.26
81.67 06.90
78.11 = 02.10
95.03 = 01.21
10.92 = 05.47
22.54 05.46
74.50 01.10
90.55 + 01.13
95.35 = 00.47
9
17.08 + 07.70
11.56 00.62
69.54 + 04.18
86.78 + 01.08
80.84 + 01.18
93.21 = 01.39
All
16.85 = 1.08
64.14 + 2.69
71.98 + 1.70
88.28 0.27
86.57 = 0.55
94.98 + 0.29
Table 16: Rotated MNIST with 90" or 270 rotated original images and Weak Augmentation during training. The test scheme is un-rotated original images. Consistency metric (CM) is computed between un-roated images and ones with 180 rotation. It can seen that CM is relatively small for and but the drop smaller than expected suggesting that CNNs learn from features different from how humans perceive the digits.
32

---

Published in Transactions on Machine Learning Research (11/2022)
F Practical considerations when DAIR is used in training
F.1 Explanations for a practical sweet spot for the performance of DAIR-SQ as a function of À
In this section, we investigate the reason why we see a sweet spot for the performance of DAIR-SQ as a function of A. As shown Figure] 7 we see a sweet spot for X, where the performance takes its maximum and starts decrease for larger values of X. There are a few explanations for this performance degradation.
1. It is observed that large À requires a relatively longer time for convergence. To show empirically this is true, we added another example in Appendix F.2 Theoretically, this is in line with the classical results in the optimization literature where larger Lipschitz constants (resulting from adding a regularizer) slows down the convergence rate. Thus, as we are training all models for a certain number of epochs, we will end up with underfitting
2. A larger À is more likely to guide the optimization trajectory towards a spurious poor local minimum with poor generalization performance, when the optimization trajectory is non-convex. We have experimentally verified this in Section 2.2.1 (Figure 17 as the reason for the poor performance of DAIR-L1 in Figure 7
3. With a finite number of samples regularizer does not necessarily lead to the best possible performance in the infinite sample setting (with weak domain shift). Hence, we might expect to observe the classical opproximationn-stiiattoo tradeoff. This is especially true in real-world scenarios where one might expect that the difficulty of the example may not necessarily be preserved through data augmentation, and hence forcing the loss to be equal on both samples might be detrimental to the overall performance, which may lead to a practical sweet spot for
We dig into the experiment in Figure 7 specifically and try to understand which case is the responsible for the sweet spot in Figure 7 We exlend the number of training epochs from 40 to 160, and report the accuracy for E {1.43, 8.85, 16.23, 100} Table 17 suggests that, when we increase the number of training epochs, the sweet spot of moves from 8.85 to 16.23 and in fact we can achieve an even better performing model with accuracy 89.22 as compared to the previously reported 85.89, while the performance does not change much for the smaller values of X. We also observe big performance boost for larger values of X. This suggests that in this experiment the sweet spot for X is caused by capping the training epochs to finite value. Having said that, we believe that we are practically interested in using DAIR with marginal computational overhead over FRM and hence we would expect to observe such sweet spot in performance in practice as À - co.
À
Acc Epoch 40
Acc at Epoch 160
1.43
79.09
80.19
8.85
85.89
86.60
16.23
82.66
89.22
100
46.95
69.37
Table 17:
Testing accuracy of Rotated MNIST, Weak Augmentaion. We see the accuracy increases as we extend the number of training epochs.
F.2 Additional evidence on growing cost of training with the regularization strength
We also provide further evidence for the growing cost of training with À toy problem where we reliably measure the gradient norm and ensure convergence. We study the following simple binary classification problem with logistic regression, which mirrors the MNIST experiments: at the training time the input is Xtrain (z, 2y 1 + f1) and the label y, i.e., Zuain = (Xtrain: y). Here, x ~ o0), and P(y = 1|z) = i where f is independent of x and t ~ NN(0 of). In this example, we intentionally provide feature which is highly correlated with the label during training. Again, clearly, w² (1, but
Note these lues comes from 1og sweeping
33

---

Published in Transactions on Machine Learning Research (11/2022)
WERM will converge to (0,1) due to the overfitting to the spurious feature. We introduce augmenter which generates the augmented example such as Xaug (Ir, 2y - 1 4 + 12) where l2 ~ (0, oz) We use this data augmenter for DAIR training and test on Xtest = (z, = 1 - 2y). We summarize the steps need for convergences and the testing accuracy in Table 18 as well. We can find that the required number of iteration to convergence increases as À increase.
For this tiny toy example, there is a factor of 10x increase in the required number of iterations when À is chosen to be 10,000 opposed to 0.5. Note that this is using ADAM and the gap is significantly larger if we use vanilla gradient descent; were not able to even converge in 10 steps. This is provided as further evidence for the practical sweet spot for DAIR as À + 00.
A
Iterations to Converge
0.5
81.35 6.07
1
91.05 + 2.53
2
89.10 = 2.41
5
101.65 + 2.87
10
107.70 + 5.77
100
151.75 + 4.28
1,000
195.85 + 4.54
10,000
802.60 + 7.58
Table 18: Iteration needed for the logistic model to converge with different A. The model is converged when the Lz norm of the gradient less than 10-7.
F.3 Compare DAIR-SQ and DAIR-L1 on avoidance of spurious local min
We empirically verify this hypothesis on Colored MNIST with Adversarial Augmentation. Figure 17 depicts the classification loss and regularization of the first 10 and last 140 iterations. One observes that at the beginning of training, regularization term of DAIR-SQ impacts the training dynamics less while DAIR-L1 starts optimizing the regularizer right away, which dominates the entire training procedure and therefore leads the model to poor local minimum. The left panel of Figure [7 confirms that the classification loss of DAIR-L1 remains large and unchanged (that of random classifier).
SDA-ERM
AR(-)
0.8
100
0.6
DAIR-SO DAIR-LI DA-ERM
0.4
10
10
5
10 500 550 600
0
5
10
500 550 600
Iterations
Iterations
Figure 17: Training DA-ERM loss and (DATR-SQ) for first and last 140 iterations Colored MNIST with Adv Aug for DAIR (X = 100). The regularizer loss on DA-ERM grows large as it is uncontrolled. DAIR-LI is optimizing an regularizer, but for unified illustration we evaluate it using (DAR--QQ.
This same property of DAIR-SQ also weakens the regularizer on training samples with high losses at the later stages of training. These samples are likely noisy, which makes DAIR-SQ more robust to noisy samples, as we already observed in Section 4.2
For the experiments in Sections 3 and we tune in DAIR regularizer on top of the ERM basclines. In other words, all experiments we use the same set hyper-parameters including step-size, batch-size, etc., from literature and only tune À on the validation set.
34

---

Published in Transactions on Machine Learning Research (11/2022)
G The impact of partial augmentation
We explore the impact of partial augmentation, where we only augment a certain fraction of the training samples. The experiment revisits noiseless Rotated MNIST with weak rotation data augmentation and Colored MNIST with Adversarial augmentation. This experiment emulates situations where an augmentation function is only applicable to certain examples or where augmentation is expensive and we would like to decrease the augmentation cost.
In Figure 18 report the experiment results for DA-ERM and DATR-SQ by applying augmentation only {10%, 20%, 30%, 50% 100%} of the training samples, averaged on three runs. In Rotated MNIST experiment, as can be seen, DAIR-SQ with augmentation on only 20-30% of the samples performs similar to full augmentation. On the other hand, DA-ERM is more sensitive to partial augmentation and is subject to a stceper performance drop. This could be viewed as further evidence that DATR-SQ could reach its best performance using weak augmenter functions. is also noteworthy that in this example, DAIR-SQ with only 10% partial augmentation still outperforms DA-ERM with 100% augmentation. One can draw similar conclustion in the Colored MNTST experiment only 10% augmentation gives comparable performance to full augmentation.
Rotated MNIST
Colored MNIST
100
100
Testing Accuracy
75
75
50
50
25
DAIR-SQ
25
Augmentation Portion
Figure 18: Test accuracy vs fraction of augmented samples Rotated MNIST.
35

---

Published in Transactions on Machine Learning Research (11/2022)
H Details on neural task-oriented dialog modeling
We provide details on the benchmark that we used in this experiment. Qian et al. (2021) proposed a new test set for MultiWOZ 2.2, called MultiWOZ 2.2 with SGD entities, where named entities are replaced with those from Schema Guided Dialog dataset Rastogi et al. 2020) and showed that SimpleTOD (Hosseini-Asl et al., 2020) endures more than 8% performance drop on the new test set. Examples from the dataset are shown in Table 20, To address this problem, we define a new data augmentation scheme for DAIR and DA-ERM by replacing the named entities from the MultiWOZ 2.2 training set with randomly scrambled versions of the named entities. For example, "warkworth house" could be turned into "casrtokow hhrwu" (sec Table 20 In all of our experiments, we utilize the SimpleTOD model (Hosseini-Asl et al. 2020) and we apply DAIR to enforce invariance between the named entities in the training examples and the scrambled entities from their corresponding augmented samples. The model is trained with ParlAT MMiller et al. 2017) fine-tuned with the
pre-trained BART (Lewis et al. 2019). Training hypeer-arameeerss can be found in Table 19 The optimal À was luned by grid search in (0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99, 1.03.
Parameter Epochs
Value
0.5
4
Batchsize
6
Optimizer
AdamW
Learning rate
10-5
Table 19: Hyper-parametee parameters used in training SimpleTTOD.
User: can you help me book reser- vation at the clarion inn &
User: can you help book a
User:
you help book
reservation at the warkworth house hotel?
reservation the eaartokow hhrwu hotel?
suites atlanta downtown ho- tel?
A Agent:
yes rould! how many people are staying, and what days
Agent: yes could! how many people are staying, and what days
Agent:
yes i could! how many people are staying, and what days would fyou like to stay?
would fyou like stay?
would fyou like stay?
User:
' just for me, and i'll he staying for three nights start- ing from tuesday.
User:
it's just for me, and staying for three nights start- ing from tuesday.
User:
it's just for me, and i'll be staying for three nights start- ing from tuesday.
botel-bookdagp tucsday hotel- botel-bookpeoppe:
hotel- bookkay tucaday hotel- boote-bookkeeppe:: 1 hotel- book-bookstay:: 3 hotel-name: hdrrwu
hotel-bookdayy tucaday hotel- bookk-o-oopppple::: 1
DS:
DS:
hotel- bookstay: 3
DS:
hotel-bookstay::
hotel-name:
warkworth
easrtokow
hotel-name: clarion irrr &
house
suites allanta downtoou
Table 20: Left: sample from the original MuluiOZ dataset. Middle: augmented sample generated by scrambling. Right: synthetic sample with name entities from SGD. Comparing left and the middle example, are generating new named entities (marked in red) by scrambling. Comparing left and the right example, the only difference is the named entity from different dataset, which is marked in red. Note that the SGD named entities not exposed to the model during training. Only the original named entities and scrambled named entities from MultiWOZ are used during training.
36

---

Published in Transactions on Machine Learning Research (11/2022)
Setup and additional results for Visual Question Answering
All the approaches included in this paper use the original VQA v2 train split for training. along with the IV-VQA and CV-VQA train splits for augmentation in the DAIR and DA-ERM(Agarwal et al. 2020) settings. The ERM setup (Kazemi & Elqursh 2017j, represents a vanilla SAAA model trained on the VQA v2 train split. For the data augmentation methods, an image from VQA v2 contains multiple edited versions IV-VQA/CV-VQA, we randomly select one of them to serve as an augmented sample during training. We modify the official code released by [Agawall et. al. (2020) to suit. formulation. All the methods are trained for 40 epochs with learning rate of 0.001 and batch size of 48. The bascline approaches that compare against are trained and evaluated by us, using the same training setup DAIR.
A
VQA val (%)
CM
Predictions flipped (%) pos
nog (%)
neg
pos (%)
nog
nog (%)
0.72
64.89
95.89
6.67
2.64
2.38
1.65
1
64.75
96.06
6.33
2.54
2.22
1.57
1.68
63.90
96.19
5.78
2.20
1.95
1.64
2.68
62.51
96.63
5.23
1,88
1.86
1.49
5.18
60.03
97.22
4.45
1.63
1.59
1.22
10
57.70
97.67
3.91
1.33
1.37
1.21
Table 21: Accuracy-Consistency Tradeoff VQA v2 val and IV-VQA test set controlled by
Table 21 indicates a tradcoff between the accuracy on the VQA v2 val and the consistency metrics. The optimal À value is determined by grid search over uniformly chosen set of size in log space 10) with the corresponding performance the validation set. As the À value increases, the consistency between the predictions increases, while the accuracy on original examples decreases. For instance, A X value of 10 strongly boosts consistency thus lowering the 'Predictions flipped' percentage to only 3.91% but sacrifices the classification accuracy causing to drop to 57.7%.
37

---

Published in Transactions on Machine Learning Research (11/2022)
J Additional details on robust regression
For the robust regression task, we determine the optimal À by performing grid search over a uniformly chosen set of size in log space [1, 10 and the best performing À on validation set is used for reporting the results on the test set. We set the learning rate to 0.01 for all these experiments. Following the convention from Li et al. (2021), we set the tilting factor 't' to -2 for all experiments that use the TERM objective.
38

---

Published in Transactions on Machine Learning Research (11/2022)
K Details on training robust neural networks
K.1 Setups for the main results in Section 4.4
For all algorithms reported in Table we use Pre-Activation ResNct-18 (He et al. 2016 with last-layer output size of 10 as the classification model and their original hyper-parameleess. For training the DAIR model, the adversarial examples are generated by Lse based PGD attack with iterations, (attack strength) set 8/255 and attack step size to 2/255. We train the model for 120 epochs with initial step size 0.0001 and uses CosineAnnealin PGD attack with 20 iterations of same perturbation sizes. The optimal À by performing grid search uniformly chosen set in log space [10-1, 10a] with 10 points.
scheduler. We evaluate all the models against the standard FGSM attack and
K.2 Additional results with new baselines
We also compare DAIR with some recent basclines such Tack et al. 2021 which utilizes Jensen-Shannon consistency regularization the features. To be detailed, pair of images with attacks are fed into the model and the Jensen-Shannon distance between the resulting output logits are computed afterward as the regularizer. The regularizer then is added to existing algorithms such as (Madry et al.) 2018), Zhang et al. (2099) to boost their performances. The results are summarized in Table 22 It can be seen that DAIR is also comparable with new baseline. worth mentioning that the performance of DAIR is better in Table 22 than in Table 5 The reason that the training and the tuning sctups are different. We follow the exact. setup of Tack al. (2021) and obtain the results in this subsection.
Method
Clean
PGD-20
ERM (Madry et al. 2018)
84.57 (83.43)
45.04 (52.82)
MART (Wang et al. 2019
82.63 (77.00)
51.12 (54.83)
TRADES [Zhang et al. 2019
82.87 (82.13)
50.95 (53.98)
JS Consistency (Tack et al. 2021
86.45 (85.25)
56.51 (57.53)
DAIR-SQ
86.16 (85.24)
56.68 (57.22)
Table 22: DAIR vs Tack (202 Accuracios the parenthesis are from models tuned for PGD-20 while accuracies the left of the parenthesis from models tuned for clean images.
39

---

Published in Transactions on Machine Learning Research (11/2022)
L Details on training ImageNet-9
We use ResNet-50 provided by Torchvision but replace the last layer with 9 outputs. We train the model for 175 epochs with batchsize 128, initial learning rate of 0.1 and decay of 0.1 at 30, 70, 110, 150 epochs.
40


Reviews:
[{"review_id": "kCljtOL6m1", "reviewer": "Reviewer_cKSA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well written, and the proposed method is easily understood.\n- This paper performed thorough experiments on various problems ranging from robust classification against adversarial attacks to domain generalization in the presence of environmental shifts.\n- This paper conducted a theoretical analysis of the optimality of DAIR-SQ versus DA-ERM in linear regression and theoretical analysis of the convergence of DAIR-SQ.\n- While conventional consistency regularization methods can only be applied to invariant data augmentations, DAIR-SQ can be used for both invariant and covariant data augmentations.\n\n\nWeaknesses\n- Theoretical analysis and experiments in this paper focus only on DAIR-SQ, a special form of DAIR, and do not cover DAIR in general.\n- The proposed method requires pairing information of the original data and its augmented data.\n- The current method for determining the hyperparameter $\\lambda$ is ad hoc, such as grid search.\n- Theoretically, the value of the hyperparameter $\\lambda$ can be set to infinity to obtain the optimal test error, while convergence becomes linearly worse as the value of $\\lambda$ increases.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a simple and versatile consistency regularization method called DAIR (data augmented invariant regularization). While conventional consistency regularization evaluates consistency at the intermediate feature level, the proposed method evaluates consistency at the loss level. Therefore, conventional consistency regularization methods can only be applied to invariant data augmentations, but the proposed method is applicable for both invariant and covariant data augmentations. In particular, this paper focuses on DAIR-SQ ($(\\sqrt{l(z;\\theta)} - \\sqrt{l(z;\\theta)} )^2$), a special form of DAIR, and provides some theoretical proof of this property. Furthermore, DAIR is experimentally evaluated for various problems ranging from robust classification against adversarial attacks to domain generalization in the presence of environmental shifts. It is shown to achieve comparable or better performance than empirical risk minimization plus data augmentation (DA-ERM) and conventional consistency regularizations.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The reviewer enjoyed reading this well-written paper. However, to better understand this research, please answer the following?\n\n- In Sec. 2.4, this paper states, \"At the beginning of training when the network is not yet trained, the loss values on the original samples are large, and $\\mathcal{R}_{sq}$ regularizer is weak letting the training to proceed towards a good solution for the original samples.\" If this is true, can we use DAIR-L1 to obtain similar results by using a smaller value of $\\lambda$ in the early stages of learning and increasing the value of $\\lambda$ as the learning progresses?\n\n- Is it possible to evaluate the effectiveness of the proposed method as the strength of correlation between inputs and outputs changes? For example, in Colored MNIST, the effectiveness of DAIR-SQ  will be measured according to changes in the probability of color since the strength of the proposed method is that DAIR-SQ is applicable to covariate data augmentation. \n\n- In the experiment on the real dataset, the value of $\\lambda$ is determined by grid search on the validation data. Is it possible to find an appropriate value of $\\lambda$ on the colored MNIST dataset described in section 2.3? The reviewer is interested to see the difference between the optimal value in Fig.6 and the ones from grid search. In the realistic case, the distribution of validation data is assumed to be the same as the training data.\n\n- In the case of neural task-oriented dialog modeling, SimpleTOD is used, and in VQA, SAAA is used as the base model. Can we get the same effect by using other base models?\n", "broader_impact_concerns": "There are no broader impact concerns.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NHpoPLOktE", "reviewer": "Reviewer_mkiw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n* The paper is well written and easy to follow. All contents are well organized and well presented.\n* The toy example givein in Sec 2.2 is interesting and insightful.\n* Lots of experiments on multiple datasets with good diversity.\n\n[Weakness]\n\nOne major concern is how to compare DAIR to Contrastive Learning (CL). The authors claimed that the loss-level consistency (as in DAIR) is better than feature-level consistency (as in CL). However, this key argument is not well supported by the theoretical analysis and the numerical experiments:\n* From theoretical side, it is unclear why DAIR is better than the popular CL approach. The toy example in Sec 2.2 did not show that CL-based method is also guarantee to fail.\n* From empirical side, there is no experiment to compare DAIR to CL on large-scale dataset and on large models. For example, it is better to compare DAIR to MOCO on ImageNet-1k.\n\nIt is unclear what Prop. 4 indicates. For example, how can we choose the $\\lambda$ in practice according to Prop. 4?\n\nTheorem 5 seems trivial. It follows the classical online learning bound for excess risk. Can authors explain why Theorem 5 is emphasized here?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposed a new loss function for deep learning termed DAIR. For input z and its augmented input z1, the DAIR enforces the loss-level consistency between the two input. Proposition 1 proves that even in infinite data regime, the proposed DAIR is able to outperform naive linear regression methods when the input features depends on the regression target. The authors validate the performance of DAIR on multiple datasets, including computer vision tasks and NLP tasks.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* More analysis on why DAIR is better than feature-level consistency methods, especially, Contrastive learning.\n* Numerical experiments of comparing DAIR to CL methods, such as MOCO for example, on ImageNet. Consider using ResNet-50 and ViT as backbone.\n* NLP experiments for DIAR v.s. CL are suggested as well. However, I am not an NLP expert so cannot give detailed suggestions.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cFWHShpWxg", "reviewer": "Reviewer_P15o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths\n1. The paper is well organized, which is easy to follow.\n2. The experimental results are somewhat good compared with the baselines.\n3. The idea of DAIR-SQ is interesting.\n\n\n*Weaknesses\n1. The contribution of this paper is somewhat incremental and limited. The authors claim that “However, regularization at loss level has been relatively unexplored”.  The reviewer cannot agree with this point. In fact, the consistency regularization no matter at feature level nor loss level has been widely used in different fields, such as semi-supervised learning, knowledge distillation and adversarial training [1][2][3]. There is no essence different from DAIR with the existing works.\n2. Because the covariant data augmentation will make the label to be somewhat changed, why the proposed DAIR could deal with such a kind of data augmentation?\n3. Although DAIR-SQ is interesting and indeed effective, the motivation or inspiration of proposing DAIR-SQ is not clear.\n4. In the experimental parts, the authors only use the name of DAIR. Is it in fact DAIR-SQ?\n5. What’s the meaning of “KL feature consistency” in Table 1? How about using KL in the loss level like KL in Figures 4-6? \n6. The authors only compare DAIR with the basic DA-ERM in the experimental parts, making the experimental results less convincing. The reviewer recommend the authors to compare DAIR with closely related works especially the regularization terms used in Figures 4-6.\n\n\n[1] Adversarial Logit Pairing. NeurIPS 2018. \n\n[2] Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. TPAMI 2019.\n\n[3] Theoretically Principled Trade-off between Robustness and Accuracy. ICML 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes data augmented invariant regularization (DAIR) to constrain the consistency between different augmentations of the same sample from a loss level perspective. Besides, an interesting variant of DAIR-SQ as well as some theoretical analyses are also proposed. Extensive experiments have been conducted to verify the proposed method.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please see the above weaknesses.", "broader_impact_concerns": "The authors have provided this part in the paper, and there is no obvious ethical implication.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gnrIcso1p7", "reviewer": "Reviewer_TWyi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ The paper is well written and easy to follow although I did not fully understand the choice of presenting some experiments in sec.2 and others in sec.3\n\n\\+ It is appealing to have a consistency loss that can work also for the case of covariant data augmentation. I wonder if someone already tried the same approach for unsupervised pre-training\n\n\\+ I like that the authors presented some more theoretical analysis of the proposed approach\n\n\\- Data augmentation is simpler and in many cases the difference in performance is small. Additionally, an often used scheme for data augmentation considers only the augmented samples and not the original one. In this case, adding the consistency loss would increase also the computational cost of the training as it would require the evaluate each sample only once while consistency loss would require two evaluations per sample.\n\n\\- Section 2.2 does not add much to the understanding of the proposed innovation. The proposed task is a toy example and the only reason that DAIR works better than DA is that it has a lambda value that allows the method to tune the importance of the augmentation in the final loss. Also a more classic costinsency loss based on KL on logits with a large lambda would make the job. Finally even DA in which the augmented sample loss has higher weight than the original samples would make the job. Thus, in my understanding this toy example does not bring anything to the method and it should be removed.\n\n\\- In most of the cases the performance of DAIR and KL on logits is very similar for invariant tasks. In my understanding enforcing the logits to be similar is like imposing a stronger constraint than imposing the losses to be similar. This might be the reason why for high noise (in Fig. 4) DAIR performs better than KL on logits. \n\nMinor comments:\n- Why Data augmented invariant regularisation? Finally the main contribution is the ability to deal with covariant augmentations. \n- Why the authors presented DA as the combination of original samples and augmented samples? This is not what it is done in many cases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method that adds to a normal training with data augmentation a consistency loss that compares the augmented sample with the original one. \nThe main contribution of the paper is to propose a consistency loss computed on the losses between the original and the augmented sample. This enables the method to work also on problems in which the data augmentation could be covariant, i.e. the label of the augmented sample can vary with the augmentation.\n     ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- I am not fully convinced that DAIR is better that consistency loss on logits for invariant DA. Can the authors find more examples in which DAIR is better that consistency on logits?\n\n- In the contributions, at the end of sec.1 the authors mention that DAIR can also reduce the computational cost compared to KL on logits. Can you explain why and how? I did not find a clear example/results in the manuscript \n\n- I think sec. 2.2 is not really adding much to the understanding of the method and the example try to give some additional capability to DAIR, which is not really there in my opinion. Thus, I would remove it.\n\n- I do not fully understand the structure of the paper. I would like the authors explain more about this structure in which experiments are mixed with some theoretical analysis in sec. 2.\n\n", "broader_impact_concerns": "I am happy with the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the issues that appear when applying regularization via data augmentation to problems in which enforcing the labels of augmented examples being same seems less meaningful, as the motivating examples of covariant tasks shown in Fig.1 and Fig. 2 in the paper. To address the issue, the paper argues for data-augmented invariant regularization (DAIR) at the LOSS level, rather than the conventional feature consistency regularization. 

The paper presents a few algorithm variants of DAIR at the loss level, including DAIR-SQ and DAIR-L1 (Garg et al., 2019), and further shows that DAIR-SQ lower bounds DAIR-L1 (Lemma 1 in the paper), leading to better regularization solutions when training with DAIR-SQ as the regularizer. The paper further analyzes how the penalty parameter \lambda of the DAIR-SQ regularizer affects performance by showing that DAIR-SQ leads to convergent algorithms when optimized by gradient descent, where classification with logistic loss is assumed (Proposition 3 and Theorem 4 in the paper). 

The paper presents experiments of both covariant and invariant tasks; results confirm the effectiveness of DAIR at the loss level. 

The paper receives mixed reviews from 4 reviewers. While all of them admit some merits of the proposed issue of DAIR at the loss level, they point out main concerns such as the lack of comparisons with contrastive learning, no use of pairs of samples both of which are augmented ones, the lack of large-scale experiments (e.g., on the ImageNet), etc. 

The biggest concern is, however, how the paper can be positioned in the literature. The paper claims that regularization at loss level has been relatively unexplored, however, reviewer P15o points out “consistency regularization no matter at feature level nor loss level has been widely used in different fields, such as semi-supervised learning, knowledge distillation and adversarial training [1][2][3]. There is no essence different from DAIR with the existing works.”

AE agrees. In fact, the key novelty of the paper is its improvement of DAIR-SQ over DAIR-L1 (Garg et al., 2019, as already discussed in the paper), and also Proposition 3 and Theorem 4 that additionally support the improvement. As such, the paper should be structurally reformed; for example, the paper may first present the rich literature of DAIR at the LOSS level, in particular those techniques (e.g., DAIR-L1, the regularization methods presented in [1], and many others) closely related to the proposed DAIR-SQ, and then discuss how these existing techniques are limited or suboptimal in addressing covariant tasks; the real contribution of this paper about DAIR-SQ and its supporting analyses then follows; as such, DAIR-SQ should be the real method proposed in the paper, rather than DAIR. Please revise the paper following the above suggestions, including other minor issues pointed out by the reviewers. 

[1] Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. TPAMI 2019.

Decision: Accept (poster)